Analytics in the cloud: 5 lessons learned

Every company-from the smallest start-up to the largest firm-needs to be agile in today's market to respond to changing dynamics and new competition. But these days it's often the smaller companies who are better positioned to adapt: as the barriers to entry have decreased, emerging companies now have access to data streams-and techniques for analyzing them-that used to be the exclusive province of the largest companies. At the same time, the CIOs of larger organizations now find themselves as much bound by their legacy systems and data as they are empowered by them. The costs of managing these legacy systems are getting in the way: too much of the budget goes to maintenance, and not enough is left over for new development and technologies.

Nowhere is this dynamic more apparent than with Business Intelligence (BI). As BI once again rises to the top of priority and wish lists, CIOs are struggling with the costs of meeting internal demands while keeping within their budgets, and still finding time for innovation. The costs of proprietary servers and storage devices, as well as the space and energy to manage them, are off the charts and highly visible to every CFO, CTO and procurement professional. Proliferating copies of data into multiple one-off analytical systems-seemingly one for every question to be asked-only adds to the costs, and even new "data appliances" can cost in the tens of millions to scale up as requirements grow.

Clearly, new approaches are needed to cost-effectively scale BI systems while meeting the demand for information on the front lines. Here are some examples of how forward-looking organizations are doing large-scale analytics in the cloud to break the logjam.

1. Hold the line with commodity hardware.

Most new analytic data engines run on inexpensive commodity hardware, transforming IT cost models and conventional wisdom about the costs of new systems. As Mark Dunlap, a consultant with Evergreen Technologies and a veteran of massive data warehouse projects at Amazon and Fox Interactive, puts it, "If you're using proprietary hardware, you're in a losing battle. Sooner or later, whatever company's developing that technology will not be able to keep up. We've seen it over and over and over again-they won't keep pace with what commodity systems are doing."

2. Buy capacity when you need it, not according to a closed appliance size

Clint Johnson, VP of Business Intelligence at Zions Bancorporation, says he's avoiding locked-in purchase models as they tackle massive data challenges. "We like the ability to add hardware easily, incrementally," says Johnson. "Specialized appliances we looked at scaled in very specific size increments." Not only are those new purchases large, they may be substantially greater than near-term needs-but payment is not scaled to usage, it's by total capacity.

3. Unused server power is a priceless resource -- use it.

Typical capacity utilization rates on distributed servers used for BI applications or data marts are often at 20 percent or below, leaving substantial system power unused. Newer software can harness that power with effective provisioning strategies. Brian Dolan, Director of Research Analytics at the Fox Audience Network, says, "With my Greenplum [cloud-based] database, I get to share 40 nodes with the production system. I use them when I need them, and then I give them back." Building "sandboxes" as needed-mapping servers (or cores) and data stores into the form needed-addresses the task at hand efficiently. A well-designed server pool, with the right software for flexible provisioning, becomes your internal "cloud."

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

• Bookmark this page
• Share this article
  •  
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email CIO
• Follow CIO on twitter

• Share
  Share this article

  •  google+
  • facebookfacebook
  • slashdotslashdot
  • diggdigg
  • RedditReddit
  • stumbleuponstumbleupon
  • linkedinlinkedin
  • twittertwitter
• print
• email
• Bookmark