Critical.
Authoritative.
Strategic.
Subscribe to CIO Magazine »

Microsoft beats data-sorting record with new approach

A new approach, called Flat Datacenter Storage, pushes computational sorting to each data server

Besting a record set by Yahoo in 2009, the research arm of Microsoft have deployed a new technique for quickly sorting large amounts of data, called Flat Datacenter Storage (FDS).

The researchers will discuss their work at an Association for Computing Machinery conference dedicated to databases this week in Scottsdale, Arizona. They are also implanting their data-sorting techniques in Microsoft's Bing search engine, where it could boost response times to user queries.

"Improving big-data performance has a wide range of implications across a huge number of businesses," said Microsoft Research project leader Jeremy Elson, in an online entry describing the work. "Almost any big-data problem now becomes more efficient, which, in many cases, will be the difference between the work being economically feasible or not."

In tests conducted under the MinuteSort benchmark, the system set up by Elson and his colleagues was able to sort 1,401Gb of data in a minute, which beat Yahoo's previous record of 500GB in the same time. Microsoft also boasted of sorting the data using fewer resources: The system used 1,033 disks in 250 machines while Yahoo required 5,624 disks across 1,406 machines to complete their operation.

FDS starts with a similar approach as Google's MapReduce -- as it is implemented in Apache Hadoop -- by moving the computational sorting to each individual data server. Unlike Hadoop, however, every server trades information with all the other server in the sorting cluster. The researchers used an additional Microsoft networking technology, called full bisection bandwidth networks, to boost the bandwidth, allowing each computer to both send a receive send up to 2GB per second.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

More about: Apache, Google, IDG, Microsoft, Yahoo
References show all
Comments are now closed.
Related Whitepapers
Latest Stories
Community Comments
Latest Blog Posts
Whitepapers
  • How to Successfully Select an ERP System
    An Enterprise Resource Planning (ERP) system is a series of software applications that collect and compiles data from different departments to enhance collaboration and co-ordination within the business. If you’re looking to implement your first ERP system, or to upgrade from an existing system, this whitepaper offers eight simple steps for selection that will lead to long-term strategic success.
    Learn more »
  • IDC MarketScape Excerpt: Worldwide Client Virtualization Software Assessment
    The rise of BYOD is creating governance and regulatory nightmares while providing end users with unprecedented flexibility and agility. While IT is still intrigued by the possibility of a better desktop management model and the operational savings client virtualization software could deliver, it is the increased governance and the ability to deliver desktops, applications, and data to any device that are driving today's purchases.
    Learn more »
  • The CIO Paradox
    As there are timeless leadership principles underlying IT value, there are unfortunately also timeless challenges that thwart the IT organisations efforts and make for a rocky path to CIO success. These are the inherent contradictions we call the CIO Paradox.
    Learn more »
All whitepapers
rhs_login_lockGet exclusive access to Invitation only events CIO, reports & analysis.
Latest Jobs
Salary Calculator

Supplied by

View the full Peoplebank ICT Salary & Employment Index

Recent comments

Computerworld
ARN
Techworld
CMO