Every day, organisations are producing and capturing enormous amounts of information about their customers, suppliers and operations. Add to this the information now available from multimedia, smart phones and social networking sites, and we are faced with more data than ever before.
Traditional data warehousing capabilities are unable cope with the sheer volume of information, but Big Data technology enables us to access and use these highly valuable, large scale data sets for increasingly sophisticated data analytics and better business decision making.
Big Data is definitely here to stay. Gartner analysts claim that information volume is growing at a minimum rate of 59 per cent annually. IDC’s latest Digital Universe study estimated that the total volume of data being stored in the world will reach 35 zettabytes by 2020 (one zettabyte equals one trillion gigabytes).
Big Data will change the Business Intelligence (BI) landscape and provide a valuable data resource for organisations. The following is a list of steps that CIOs should follow to successfully incorporate Big Data into their BI program.
- Find the right project — Arguably the most important step is to identify the right project on which to trial Big Data. It should be a commercial business problem that needs solving rather than a technical issue. Ensure the project provides a direct benefit or advantage to the business that can’t be met with existing infrastructure, and you’re on the right track to gaining executive support.
- Gain executive support — Big Data complements your existing investment in data warehouse technology. Executive support will be based on an acceptance of the value of evidence based strategy (i.e. they will already be using data warehousing and probably data mining extensively within the organisation).
- Get the right people — You will need people with a very specific skill set; those who can manage large, distributed data sets and the hardware that comes with it. Next are the people who can make sense of all the data and can then put that into a business context; think data scientists as opposed to existing data analysts and data miners.
- Embrace open source — Big Data means thinking differently about toolsets and getting comfortable with open source quickly. Traditional vendors are not necessarily the answer here; most Big Data tools are open sourced. The innovators in this space are communities made up of the smartest people from companies including Google, Yahoo, Apple and Facebook.
- Don’t start from scratch — The most widely accepted Big Data tool is Hadoop, an open source technology which you can get from Cloudera or EMC. Hadoop is intended to ease the complexities of performing large-scale batch operations on data and is managed within the Apache project framework; it will provide the basic tools you need. The major BI vendors announcing support for, or solutions using, Big Data technology.
- Prepare for changing architecture and hardware — Big Data works in data lakes and not only runs analysis on this mass scale information but also becomes a source for data warehouses. You will need to rely less on small numbers of large machines and look more at large numbers of commodity hardware and cloud resources.
- Buy capacity from small standard units — Infrastructure as a Service (IaaS) vendors and cloud resources provide massive time-to-market and timeliness advantages to those organisations capable of taking advantage. Security issues and concerns are common blockers here, but can be overcome. See cloudsecurity.org for further information.
- Find a data source you don’t use — Look at the data you collect from say, your company website. This can give you information on the popularity of web pages, the busiest times of day for the site and which ISP your customers use. Investigate the potential for using this information for marketing and sales.
- Look at visualisation — Think about new ways of presenting data. Due to the volume, some Big Data analysis simply won’t make sense using tables or graphics. Edward Tufte and Stephen Few are the preeminent authors in this field.
- Manage expectations — Big Data is good for large scale analytics and long-term strategic direction. Ensure your users know it won’t deliver monthly management reporting or ad-hoc queries over structured data.
Conrad Bates and Cameron Wall are co-founders and directors of C3 Business Solutions.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.