In today’s inter-connected world of transactions and interactions, data has become a key factor in the production process. The ability to take advantage of this factor is what makes a smart organisation different from one that is not so smart.
Traditionally, most of the business intelligence that is of interest to an organisation comes from data that originates and/or is managed within the organisation. Also, in the past, the sole focus had been on structured data despite the fact that most of the information is generated in unstructured form. With more data important to an organisation’s activities being generated – and not necessarily in structured form – outside an organisation, these organisations have started to take an interest in big data.
In addition, with the increased use of sensors (RFIDs, smart meters, etc), we generate a lot of data that can be used to gain competitive advantage, drive productivity and make operational improvements.
This phenomenon has also been enabled by technological advances such as in-memory analytics.
Capgemini’s 2012 global survey on big data, commissioned the Economic Intelligence Unit, covered 607 executives around the world (25 per cent in the Asia-Pacific region), and found that the majority of respondents (54 per cent) say that management decisions based purely on intuition or experience are increasingly regarded as suspect.
This view is held even more firmly in the manufacturing, energy and government sectors, and 65 per cent assert that more and more management decisions are based on ‘hard analytic’ information.
The research shows that organisations are increasingly moving towards evidence-based decision making, but at the same time, face significant challenges in managing and leveraging the ever-increasing volumes of data not only from a technology perspective but also as an organisation.
With regard to the most valuable big data sets, there has been significant consensus across industries: 69 per cent of the participants agree that business activity data (sales, PoS data, purchases, etc) were the most valuable, followed by office documentation (emails, document stores, etc).
However, health care and pharmaceuticals differ, citing social media as the second most valuable big data set, possibly because of sentiment analyses reflecting consumer views and trends as keys to health care products.
With unstructured data seen as the second most valuable big data set, more than 40 per cent of respondents said that unstructured content was too difficult to interpret and manage to be used in support of decision making.
Interestingly, the survey also showed that many of the organisations that described themselves as being data driven felt that they could have made better decisions if the right data had been available.
With regard to impediments to effective decision making using big data, organisational silos topped the list of 56 per cent of the respondents. This seems to apply more to large organisations (with more than $10 billion in revenues), the response rate being 72 per cent vis-a-vis smaller firms (with less than $500 million in revenues) at 43 per cent.
The research shows that organisations are also struggling with the enormous volumes of data as well as their poor quality, together with ‘silo’ed’ nature of dealing with data to drive decision making.
The other major impediments included a shortage of skilled people to analyse the data properly, the time taken to analyse, the difficulties in interpreting the unstructured content of big data, and the high cost of storing and manipulating large data sets.
Until a few months ago, when organisations (apart from a few pioneering enterprises/groups) in Australia talked about big data it was pretty much a matter of “give me a business case for big data and we can put it up for project funding”.
However, in the last few months, there has been an increased awareness of the big data conundrum and the discussion has now come to focus on “let us take up a use case (hypothesis-based, if feasible) and do a short proof of value”.
If the proof of value (PoV) is promising, the scope can be further expanded iteratively. The key to this is to be able to keep the PoV succinct, employ support of external organisations like Systems Integrators, big data forums to exchange experiences, to obviate the need for upfront investment in the technology/ tool set and others.
We have often seen technological advances raising productivity; more often this happens over a large time frame of many years. Organisations looking for productivity gains, potential benefits even in short and medium term are possible through big data initiatives.
The aspects of big data that are more predominantly covered relate to customer sentiment analysis available through social media. Of late, we are observing this extended beyond customer satisfaction to other facets viz. Top line growth, profitability improvement and risk management. One notable exception to this is the energy utilities sector. In Victoria, driven by the regulatory requirements, installation of household smart meters provides a big data platform (where the expected meter readings could be once every 30 minutes or so).
The need here is the ability to leverage the investments made, in the form of smart meters, to achieve operational improvements for these enterprises.
Some of our clients also raised the question of possible replacement of their traditional Enterprise Data Warehouse(s), Data Mart environments with big data technology.
When you consider the sizeable investments that have been made and the maturity of these applications in supporting business needs, it is unlikely that this would happen.
However, in cases where no sizeable incumbent applications are in place, but where further investments are required, there is real potential to look at the big data solutions. We believe that the traditional warehouses will co-exist with the big data platforms for some time to come. When it comes to big data investments, the key point to remember is that the analytical insight it offers together with acting on that insight make up the value proposition.
Some of the popular metaphors adopted with regards to big data, such as ‘finding the nuggets of gold’, capture only part of the picture. Australian organisations will do well by not succumbing to the hype but by focusing on the complete picture that drives the business value.
Paul Thorley is CEO of Capgemini.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.