Big data - Part 2
- 20 September, 2011 10:12
A second technology making a significant impact on solving Big Data problems is in-memory computing, which takes workloads that were traditionally resident on disk-based storage and moves them into main memory. This delivers a performance improvement many times above that which has been possible previously.
The beauty of these technologies, however, is that they enable organisations to analyse events in their transactional systems in real time, rather than having to extract and translate data into a relational data warehouse.
Read Part 1 of Big Data.
The global general manager for business analytics at SAP, Steve Lucas, says the company has 25 active beta users on HANA today and has strong interest from close to another thousand organisations.
“We’re seeing some large consumer packaged goods companies that are looking to do massive analysis on very Big Data, like on-shelf availability and trade promotion effectiveness,” Lucas says. “And we are seeing things like financial services companies that want to do fraud analytics.
“There are companies looking at using HANA to do real time analysis on what TV shows consumers are watching. We’re seeing companies that are looking to do some unexpected types of activities with HANA.”
The advantage these companies are seeking is speed. In-memory technology and the database optimisation it enables allows organisations to perform analytics functions at speeds which are in some cases thousands of times greater than has been possibly previously. They can handle much larger and more complex data sets without losing the granularity of individual records.
While Big Data is creating new opportunities for some businesses, it is also creating entirely new business models. New companies such as US-based Infochimps and Factual have emerged to gather and provide access to numerous data sets (on both a paid and unpaid basis), in a model that is becoming known as data-as-a-service (DaaS).
Both are increasingly being used by developers to create new applications, particularly through calling in geolocation data held within their repositories.
Palantir Technologies is another US company specifically created to help organisations to integrate, visualise and analyse multiple data types, including structured, unstructured, relational, temporal, and geospatial data.
Some of the service providers are not so new, but Big Data tools are enabling them to become more sophisticated in what they offer. The Melbourne-based doubleIQ has been solving customers’ data problems for a decade by conforming and matching up data types for analysis, such as providing an integrated view of customer behaviour.
Cofounder, Dennis Claridge, says new possibilities are opening up from the volumes of data now available for processing. He says the channels for data have increased tenfold, but the volumes of data have increased a hundredfold, to the point where it might now be analysing billions of transactions.
Read more about Big Data.
“As a result of things being digitised there is more data that can be generated — a lot of Web data, such as Web logs and buying behaviour,” Claridge says. “There is a lot more transaction data and a lot more fragmentation in terms of channels that are being used.
“That’s great for us because it give us more and more opportunity to remix those, repackage those, and re-aggregate them back together to make sense of it.”
Also working in doubleIQ’s favour has been the plummeting cost of the hardware required for analytics, with Greenplum being run on commodity hardware for much of the data crunching. This is enabling doubleIQ to offer a wider range of service to a broader range of companies.
“Ten years ago the cost would have been prohibitive,” Claridge says. “It’s possible for us to develop quite a reasonable capability at a low cost now. Because there is more data available there are more possibilities across different industries, and more breadth of how it could be applied and used within a particular industry.”
Claridge says the long term vision is for doubleIQ to set itself up as a service, where it would host customer data in its own cloud and perform analysis, then publish access back to the client’s users. Should it be able to do so, Claridge says the company will one step further along in its goal of helping customers get maximum value from their data assets — something that many have failed to do.
“They don’t make as much use of their information as they could, and it gets lost over time,” Claridge says. “Data’s never valued correctly in an organisation.”
In the era of Big Data, that lost value is only likely to increase.
Read Part 1 of Big Data.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
Why IT projects really fail
Queensland government to provide 200 services online by 2015
Call Centers Suffer From Big Data Overload
CIO 100: Carsales wins top gong for innovation
How to secure passwords and other critical numbers
Securing Virtual Desktop Infrastructure
Today’s enterprises are rapidly adopting desktop virtualisation as a means to reduce operating costs, enable workplace flexibility, increase business agility and bolster their information security and compliance posture. Actually realising these benefits, however, depends upon ensuring the security and availability of the virtual desktop infrastructure. Find out how you can not only preserves the benefits promised by virtual desktops, but how it can maximise them. Click to download!
Best Practices to Make BYOD Simple and Secure
As consumerisation continues to transform IT, organisations are moving quickly to design strategies to allow BYOD in the workplace. This paper provides IT executives with guidance to develop a complete BYOD strategy which gives people optimal freedom of choice while helping IT adapt to consumerisation - at the same time addressing requirements for security, simplicity and cost reduction. Find out how device ownership eases IT burdens in endpoint procurement and management. Click to download!
Is your data centre growing too complex for your backup?
Backing up data today is growing more complex - and in an era of virtualisation, big data and cloud deployments, it can be difficult to maintain control over your data, resulting in loss and downtime. This hour-long webcast features expert commentary on navigating the complexity of backing up a heavily virtualised infrastructure; simplifying your backup software and hardware ecosystem; reducing the cost of backing up your organisation’s data, and modernising your backup infrastructure with integration. The presentations will conclude with an interactive Q&A session.