Traditional security and log-management tools attempt to provide insight into the chaos, but they typically require users to write rules to detect anomalies. Writing those rules requires pre-existing understanding of the data-you need to know what you're looking for before you can perform a search to find it.
"CIOs don't care about the logs. CIOs care about the events those logs represent. They care about anomalies." --Sanjay Sarathy, CMO of Sumo Logic
It's Humanly Impossible to Know Everything About Your Data
"The first challenge is not just that there's a vast amount of data, but the fact that typical analysis of machine data typically relies on search as the fundamental mechanism to investigate what's going on," says Sanjay Sarathy, chief marketing officer (CMO) of machine data analytics specialist Sumo Logic.
"The challenge with search is that you fundamentally need to know what you're searching on. Given the explosion of data, it's humanly impossible to know everything about your data," says Sarathy.
"CIOs don't care about the logs. CIOs care about the events those logs represent," he adds. "They care about anomalies. The traditional way of getting to those anomalies and events is writing rules. But the challenge you have is actually to write those rules. Given the amount of data, it's impossible to write rules for every event."
Anomaly Detection Uses Machine Learning, Statistical Analysis to Detect Events
Sumo Logic's answer is Anomaly Detection, a major architectural enhancement to its Log Management and Analytics service based on its LogReduce technology.
Anomaly Detection combines machine learning, statistical analysis and human knowledge from your domain experts to analyze streams of machine data, detect events in the stream and provide alerts on those events, allowing you to remediate issues before they affect business services.
"Basically, we reduce log lines into a set of patterns," Sarathy says. "That allows us to figure out the root cause of issues. We don't need to know anything about that data in advance to be able to come up with any of those patterns. You, as the domain expert, help us understand which patterns are relevant and which aren't. We're building on that pattern recognition technology to provide an automated way to do anomaly detection."
"When you get an alert, you as a human go in and see what contributes to that anomaly," he adds. "We don't know how relevant that is to you as a human. You can
Thor Olavsrud covers IT Security, Open Source, Microsoft Tools and Servers for CIO.com Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at email@example.com
Read more about big data in CIO's Big Data Drilldown.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.