Analytics in the cloud: 5 lessons learned

Analytics in the cloud: 5 lessons learned

In this opinion column, consider some lessons learned from IT leaders who are crunching large data sets in the cloud using Greenplum's technlogy.

4. Keep asking, keep changing -- and keep the data.

New practices in BI are echoing the "agile" methodologies programmers are finding effective. Complex techniques, statistical analyses, and new analytical models emerge and disappear. For example, Ryan Hawk, T-Mobile Director of Information Management, and his analytics team needed to build models of telecom usage-propensity to churn, revenue generation, and more- but were challenged because "data is a business case-we have to decide what we can afford to store on our MPP systems," Hawk says. "The hardest thing is having to purge data every 60 days-you can't do much trending."

By shifting their data warehouse into an agile, virtualized infrastructure, T-Mobile now has flexible access to more data and can analyze and rethink at will. Like Fox, they're able to build analytical "sandboxes" on-demand to discover new questions. Grab data to explore those questions. Tear it down, and do it again. Data is the other element of the "cloud;" keep it where you need it, and use as appropriate.

5. Run programs "close to the data."

Dolan's team at Fox might work with two weeks' worth of data: 100 billion lines, 10s of terabytes. Exporting, transforming, moving and distributing that data in chunks (extract, transform and load: ETL style), constrained by bandwidth and system load factors, used to take 3 to 4 days. Rebuilding all the joins, indexes and other structures within the data would consume another day or two. But with new in-database analytics technology, Fox can run programs directly in the database, eliminating the bottlenecks standing between his team and business insights. According to Dolan, "Inside our Greenplum database, setting up two weeks' worth of data takes us about 20 minutes."

Exploiting information has become an imperative for all businesses, and this continues to become even more important as data growth accelerates and new streams of precious information emerge. Supporting the teams who will provide agile response is a competitive necessity.

Merv Adrian is principal at consulting firm IT Market Strategy.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags Cloudanalytics

More about Amazon Web ServicesetworkEvergreen TechnologiesT-MobileT-MobileZions Bancorporation

Show Comments
<img height="1" width="1" style="border-style:none;" alt="" src="//"/>