Big data - Part 2
- 20 September, 2011 10:12
A second technology making a significant impact on solving Big Data problems is in-memory computing, which takes workloads that were traditionally resident on disk-based storage and moves them into main memory. This delivers a performance improvement many times above that which has been possible previously.
The beauty of these technologies, however, is that they enable organisations to analyse events in their transactional systems in real time, rather than having to extract and translate data into a relational data warehouse.
Read Part 1 of Big Data.
The global general manager for business analytics at SAP, Steve Lucas, says the company has 25 active beta users on HANA today and has strong interest from close to another thousand organisations.
“We’re seeing some large consumer packaged goods companies that are looking to do massive analysis on very Big Data, like on-shelf availability and trade promotion effectiveness,” Lucas says. “And we are seeing things like financial services companies that want to do fraud analytics.
“There are companies looking at using HANA to do real time analysis on what TV shows consumers are watching. We’re seeing companies that are looking to do some unexpected types of activities with HANA.”
The advantage these companies are seeking is speed. In-memory technology and the database optimisation it enables allows organisations to perform analytics functions at speeds which are in some cases thousands of times greater than has been possibly previously. They can handle much larger and more complex data sets without losing the granularity of individual records.
While Big Data is creating new opportunities for some businesses, it is also creating entirely new business models. New companies such as US-based Infochimps and Factual have emerged to gather and provide access to numerous data sets (on both a paid and unpaid basis), in a model that is becoming known as data-as-a-service (DaaS).
Both are increasingly being used by developers to create new applications, particularly through calling in geolocation data held within their repositories.
Palantir Technologies is another US company specifically created to help organisations to integrate, visualise and analyse multiple data types, including structured, unstructured, relational, temporal, and geospatial data.
Some of the service providers are not so new, but Big Data tools are enabling them to become more sophisticated in what they offer. The Melbourne-based doubleIQ has been solving customers’ data problems for a decade by conforming and matching up data types for analysis, such as providing an integrated view of customer behaviour.
Cofounder, Dennis Claridge, says new possibilities are opening up from the volumes of data now available for processing. He says the channels for data have increased tenfold, but the volumes of data have increased a hundredfold, to the point where it might now be analysing billions of transactions.
Read more about Big Data.
“As a result of things being digitised there is more data that can be generated — a lot of Web data, such as Web logs and buying behaviour,” Claridge says. “There is a lot more transaction data and a lot more fragmentation in terms of channels that are being used.
“That’s great for us because it give us more and more opportunity to remix those, repackage those, and re-aggregate them back together to make sense of it.”
Also working in doubleIQ’s favour has been the plummeting cost of the hardware required for analytics, with Greenplum being run on commodity hardware for much of the data crunching. This is enabling doubleIQ to offer a wider range of service to a broader range of companies.
“Ten years ago the cost would have been prohibitive,” Claridge says. “It’s possible for us to develop quite a reasonable capability at a low cost now. Because there is more data available there are more possibilities across different industries, and more breadth of how it could be applied and used within a particular industry.”
Claridge says the long term vision is for doubleIQ to set itself up as a service, where it would host customer data in its own cloud and perform analysis, then publish access back to the client’s users. Should it be able to do so, Claridge says the company will one step further along in its goal of helping customers get maximum value from their data assets — something that many have failed to do.
“They don’t make as much use of their information as they could, and it gets lost over time,” Claridge says. “Data’s never valued correctly in an organisation.”
In the era of Big Data, that lost value is only likely to increase.
Read Part 1 of Big Data.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
Why change management doesn’t work
Larry Page wants to see your medical records
Dual-Persona Smartphones Not a BYOD Panacea
After two-year hiatus, EFF accepts bitcoin donations again
CIOs struggle to deliver timely mobile business apps: survey
BYOD and Beyond - Implementing a Unified Access Solution
The rise of BYOD programs is the single most radical shift in the economics of client computing for business since PCs invaded the workplace. Whether you are contemplating the creation of a BYOD program or currently trying to establish one, this fact cannot be overstated. Find out how to overcome these challenges.
Russian Underground 101
This research paper intends to provide a brief summary of the cybercriminal underground and shed light on the basic types of hacker activity in Russia. It discusses fundamental concepts that Russian hackers follow and the information they share with their peers. It also examines prices charged for various types of services, along with how prevalent the given services are in advertisements. The primary features of each type of activity and examples of associated service offerings are discussed as well. Read this paper.
Managing the Rapid Rise in Database Growth: 2011 IOUG Survey on Database Manageability
As the era of “Big Data” marches on unabated, data is coming from an ever wider range of sources, including transactional systems, mobile devices, sensors, streaming media, and social networks. Businesses are looking for innovative ways to better leverage terabytes—and for some, petabytes—of information. Read more.