Microsoft beats data-sorting record with new approach
- 23 May, 2012 04:56
Besting a record set by Yahoo in 2009, the research arm of Microsoft have deployed a new technique for quickly sorting large amounts of data, called Flat Datacenter Storage (FDS).
The researchers will discuss their work at an Association for Computing Machinery conference dedicated to databases this week in Scottsdale, Arizona. They are also implanting their data-sorting techniques in Microsoft's Bing search engine, where it could boost response times to user queries.
"Improving big-data performance has a wide range of implications across a huge number of businesses," said Microsoft Research project leader Jeremy Elson, in an online entry describing the work. "Almost any big-data problem now becomes more efficient, which, in many cases, will be the difference between the work being economically feasible or not."
In tests conducted under the MinuteSort benchmark, the system set up by Elson and his colleagues was able to sort 1,401Gb of data in a minute, which beat Yahoo's previous record of 500GB in the same time. Microsoft also boasted of sorting the data using fewer resources: The system used 1,033 disks in 250 machines while Yahoo required 5,624 disks across 1,406 machines to complete their operation.
FDS starts with a similar approach as Google's MapReduce -- as it is implemented in Apache Hadoop -- by moving the computational sorting to each individual data server. Unlike Hadoop, however, every server trades information with all the other server in the sorting cluster. The researchers used an additional Microsoft networking technology, called full bisection bandwidth networks, to boost the bandwidth, allowing each computer to both send a receive send up to 2GB per second.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
Trust issue looms large for tech companies capitalizing on personal data
5 women who've made it in IT
Five trends affecting legal CIOs
CIO Roundtable: The changing face of security
Bitcoin malware count soars as cryptocurrency value climbs
Evolving Threats Demand New Approaches to Security
As the world becomes increasingly hyperconnected, the opportunities for innovation are virtually limitless. At the same time, the complexity and risk associated with those opportunities is great. Security threats have the potential for enormous ramifications, but so does deploying a security strategy that compromises the user experience, performance, and the ability to innovate online. This paper will profile the emerging disruptive players, and identifies the essential steps to establishing a secure environment without compromising performance or experience.
Taking Managed Security Services to the Next Level
In this white paper, we will outline the changing landscape of security; the importance of a high-security posture; the elements of effective Web, endpoint, user and mobile protection; and the ways in which Webroot helps partners increase the value and profitability of their security practices.
‘A Little Extra Service’ Raises Customer Satisfaction and Lowers Costs
Companies are responding to the digital generation’s preference for online support, with new channels like Live Chat and Email Management. These mobile-friendly solutions give customers the right answers at the right time, when self-service just isn’t enough, and phone calls are undesirable. Read about these new touch points and the importance of a personalized web self-service.