The promise and lie of big data

The term big data is everywhere these days, there’s no doubt about it. When I meet with CIOs, big data is often named as one of the top trends shaping their IT agendas for 2012, along with cloud and mobility.

So what exactly is big data and why is it big news now?

The best formal definition of big data I have seen is data that takes more than one server to store or process. For me, when I think of big data, I think of a world of possibilities to dig deeper into information to bring additional insight to the business. For example, imagine analysing a data set that includes public transport statistics with your own Web store to determine if people shop on their mobile phone during peak hour commuting.

Like the term ‘cloud’, technology vendors have realised the marketability of ‘big data’ and have applied it to almost any solution they can. From big data databases and big data backup services, to cloud based big data analysis systems, not a week goes past without a new solution being promoted as helping companies enable big data.

The truth is that at the moment unleashing the essence of big data into small, medium and even enterprise businesses is hard. Most big data jobs are spawned from internal business intelligence projects where the business is trying to understand purchasing habits of their customers, work out the most effective shipping routes to get stock to stores, or trying to predict the stock market. Businesses have been analysing data in this way for many years but due to the cloud — and a few other technology movements such as NoSQL — the velocity of data growth has placed a new perspective on this.

The failure of big data at the moment is still in what it takes to get there. Many businesses believe that they can purchase a new Web 2.0 Big Data solution and in a few days have shiny reports and graphs walking them through the spending habits of their customers — which is a picture that is currently being painted by the marketing teams of most big technology vendors. However, what they fail to tell you is that data analysis and big data have spawned a new role which some IT teams need to fill; the data scientist. These people have been around for many years, writing the algorithms that power everything from our traffic lights, how our air traffic control system prioritises landing schedules through to the systems that power background checks for new mortgages.

To get to this promised land of big data where our dashboards, reports and business decisions are being made from solid data, we need to bring data scientists onboard, help them to understand the business and how it works, and then let them loose on all the data they can get their hands on. From there they may use tools from the major vendors (e.g. EMC, Microsoft, Oracle), or they may use tools that have been developed by open source companies (Hadoop, Google BigQuery), or they might look at moving these workloads to the cloud (think Amazon, Rackspace, or others).

Big data isn’t a technology or strategy that is implemented by the IT team, it needs to be thought about from the highest levels within the business starting with the question: ‘How can we better utilise the data we have to make better business decisions?’

This style of project needs executive backing, both in support and budget to ensure it gets off the ground and continues over time.

There are many businesses out there that spend their time developing dashboards, reports and data analysis systems that can help and really ensure that goals are reached, and IT budgets aren’t depleted by purchasing a ‘big data ready product’.

Big data is new and for many business (and IT teams) it’s still an unfulfilled promise that the business may be driving them towards. Before you embark on a big data project, have a really good look at what you’re buying into and ensure it’s something that can actually help your business — not drain time and budget.

Rhys Evans is national practice manager Enterprise Information Systems at Thomas Duryea Consulting.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

• Bookmark this page
• Share this article
  •  
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email CIO
• Follow CIO on twitter

• Share
  Share this article1

  •  google+
  • facebookfacebook
  • slashdotslashdot
  • diggdigg
  • RedditReddit
  • stumbleuponstumbleupon
  • linkedinlinkedin
  • twittertwitter
• print
• email
• Bookmark