Data Mining: The State of the Art

The US government's data mining projects fall into two broad categories: subject-based systems that retrieve data that could help an analyst follow a lead, and pattern-based systems that look for suspicious behaviours across a spread of activities. Most data mining experts consider the former a version of traditional police work - chasing down leads - but instead of a police officer examining a list of phone numbers a suspect calls, a computer does it.

One subject-based data mining technique gaining traction among government practitioners and academics is called link analysis. Link analysis uses data to make connections between seemingly unconnected people or events. If you know someone is a terrorist, you can use link analysis software to uncover other people with whom the suspect may be interacting. For example, a suspicious link could be a spike in the number of e-mail exchanges between two parties (one of which is a suspect), cheques written by different people to the same third party, or plane tickets bought to the same destination on the same departing date. Many experts believe that the NSA project analyzing millions of domestic phone records is this kind of link analysis system.

Finding the Hidden Linkages

However, link analysis projects are useful only if they have a narrow scope, says Valdis Krebs, an IT consultant who famously developed a map showing the connections among the 9/11 hijackers - after the fact. Successful link analysis requires a reliable starting point - a known terrorist, for example, or a phone number associated with one. Link analysis becomes less effective when it's used in an attempt to spot anomalous behaviour. "If you're just looking at the ocean, you'll find a lot of fish that look different," says Krebs. "Are they terrorists or just some species you don't know about?" If the US government searched for only the activities mentioned above - e-mails, cheques and plane tickets - without the added insight that one of the network's members was a terrorist, investigators would be more likely to uncover a high school reunion than a terrorist plot, says Krebs. If the government casts the net too wide, he adds, the projects could cost more, take longer and raise the risk of "false positives", such as the high school reunion example.

One example of the government applying a more realistic scope to a data mining project is a system the US DoD is currently testing that sifts through the data the agency has on everyone with a security clearance, looking for patterns that could identify spies. These patterns might include purchases that are out of line with someone's pay grade, unreported foreign travel or e-mail exchanges with a person known to work for a foreign government, says a counterintelligence official involved with the project who requested anonymity. The parameters for these searches are developed by counterintelligence officers, based on their experience of what suspicious activity looks like. As the technology improves, the US DoD hopes to rely on artificial intelligence to decide which patterns warrant attention and which do not.

However, even systems that have more limited scope, such as the US DoD's security clearance system, are sending out mixed signals. "Right now, it's information overload," says the counterintelligence official. "With the rules we have now, we would have a ton of false positives." His goal is to refine the system and eventually show that the concept works. This, he hopes, will encourage people to share more data.

His project isn't yet a success, nor has it been deemed a failure. He doesn't anticipate getting usable results for three or four years. The factors that will determine its future are the same as with any IT project: how well the technology performs, the problems the US DoD uses the system to solve and what it does with the results it gets.

Projects Get the Axe

If antiterrorism data mining is going to improve, the business rules aren't the only aspect that needs to change. After all, a system is nothing without good data. Sometimes law enforcement has a detailed profile of a terrorist suspect. But in other cases all they have is a name. "Names alone are not a helpful way to match people," says Jeff Jonas, data mining's acknowledged superstar, who made his name protecting Las Vegas casinos from cheats. Jonas, for example, shares his name with at least 30 other Americans. This is one of the reasons why Yusuf Islam (aka folk singer Cat Stevens) was detained in a Maine airport in 2004.

After 9/11 the government began replacing the Computer Assisted Passenger Pre-Screening (Capps) system - which only tracked passenger data collected from the airlines (names, credit card numbers, addresses) - with Capps II, which would add information culled from data brokers. Capps II first gained notoriety in 2003, when reports surfaced that Northwest Airlines and JetBlue gave passenger records to the US Transportation Security Administration (TSA) so it could test the new system. Critics asked about privacy safeguards, which were virtually nonexistent according to public records, and in response to the outcry Congress withheld funds for Capps II until the GAO completed a study on how exactly the TSA intended to protect privacy.

In August 2004, the TSA pulled the plug on its $US100 million-plus investment in Capps II in favour of a new system called Secure Flight. Secure Flight and its predecessor share many characteristics, most notably combining passenger records with data purchased from commercial databases. (According to a recent government audit, DHS and the US Department of Justice spent more than $US25 million in 2005 buying data for fighting crime and preventing terrorism.)

In September 2005, the Secure Flight Working Group, a collection of data mining and privacy experts who the TSA asked to review the project, completed a nine-month analysis and filed a confidential report that was highly critical of the system. Within a week, the report was on the Internet. It read: "First and foremost, TSA has not articulated what the specific goals of Secure Flight are". It went on to say: "Based on the limited test results presented to us, we cannot assess whether even the general goal of evaluating passengers for the risk they represent to aviation security is a realistic or feasible one or how TSA proposes to achieve it".

Bruce Schneier, a security expert who was a member of the working group, sees Capps II and Secure Flight as primary examples of how the lack of proper scope has damaged antiterror IT efforts. Even if you managed to design a data mining system that could comb through phone records or credit card transaction and spot terrorists with a 99 percent success rate, it still would not be a good use of investigative resources, argues Schneier. For example, if the approximately 300 million Americans make just 10 phone calls, purchases or other quantifiable events per day, that would produce 1 trillion pieces of data a year for the government to mine. Even 99 percent accuracy would produce a billion false positives a year, or about 27 million a day. And 99 percent accuracy would still mean missing some trans­actions that might actually be terrorists. And no one wants to consider the price of missing another attack. That's why Schneier wasn't surprised when he read a January article in [ital]The New York Times[end] reporting that hundreds of FBI agents were looking into thousands of data mining-generated leads every month, almost all of which turned out to be dead ends. "It's a waste of money," he says. "[Data mining] is a lousy way to fight terrorism."