Critical.
Authoritative.
Strategic.
Subscribe to CIO Magazine »

Why big data means a big year for Hadoop

You can't have a conversation in today's business technology world without touching on the topic of big data.

Simply put, it's about data sets so large-in volume, velocity and variety-that they're impossible to manage with conventional database tools. In 2011, our global output of data was estimated at 1.8 zettabytes (each zettabyte equals 1 billion terabytes). Even more staggering is the widely quoted estimate that 90 percent of the data in the world was created within the past two years.

Behind this explosive growth in data, of course, is the world of unstructured data. At last year's HP Discover Conference, Mike Lynch, executive vice president of information management and CEO of Autonomy, talked about the huge spike in the generation of unstructured data. He said the IT world is moving away from structured, machine-friendly information (managed in rows and columns) and toward the more human-friendly, unstructured data that originates from sources as varied as e-mail and social media and that includes not just words and numbers but also video, audio and images.

Given the rise of big data, I'm sure you're hearing the buzz around Apache Hadoop, the software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes (a thousand terabytes) of data. It certainly looks like the Holy Grail for organizing unstructured data, so it's no wonder everyone is jumping on this bandwagon. A quick Web search will show you that in just the past few months, companies including EMC, Microsoft, IBM, Oracle, Informatica, HP, Dell and Cloudera (to name a few) have adopted this software framework.

What I find even more notable is that companies such as Yahoo, Amazon, comScore and AOL have turned to Hadoop to both scale their businesses and lower storage costs.

According to some recent research from Infineta Systems, a WAN optimization startup, traditional data storage runs $5 per gigabyte, but storing the same data costs about 25 cents per gigabyte using Hadoop.

That's one number any CEO will remember.

So get ready for Hadoopalooza 2012. I'd love to hear what you're doing to tackle big data storage, so please drop me a line anytime.

Michael Friedenberg is the president and CEO of CIO magazine's parent company, IDG Enterprise. Email him at mfriedenberg@cio.com.

Read more about data management in CIO's Data Management Drilldown.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

More about: Amazon, AOL, Apache, Autonomy, Dell, Dell Computer, EMC, Hewlett-Packard, HP, IBM, IBM Australia, IDG, Informatica, Microsoft, Oracle, Yahoo
References show all

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
Users posting comments agree to the CIO comments policy.
Login or register to link comments to your user profile, or you may also post a comment without being logged in.
Related Coverage
Related Whitepapers
Latest Stories
Community Comments
Tags: applications, Applications | Data Management, Autonomy, Big Data, data management, hadoop, software, storage
Latest Blog Posts
Whitepapers
  • Consolidated Storage for Virtualised Server Environments
    This research brief is based on a recent Tech Target survey with more than 200 storage administrators and IT professionals in mid-sized and enterprise-class companies, and focuses on how these decision-makers view the storage-related challenges that result from server virtualisation. See the results.
    Learn more »
  • Unified Communications Strategy Guide
    Articles include: How to ensure a successful UC project; Five reasons to set up unified communications; Unified communications: Is your network ready?; How to get the most from unified communications. Read this Computerworld Strategy Guide.
    Learn more »
  • Providing effective endpoint management at the lowest total cost
    Endpoints, otherwise known as servers, workstations, laptops, mobile devices, and virtually any other network-connected device, are critical components that enable business to be transacted. Properly implemented, endpoint management ensures continuous compliance with IT policies, regardless of where the machines are located and what type of network they are connected to.
    Learn more »
All whitepapers
rhs_login_lockGet exclusive access to Invitation only events CIO, reports & analysis.