Peter Clarke, head of Scotland Yard's anti-terrorist branch, told the BBC that the surveillance involved in foiling the plot to blow up trans-Atlantic planes had been "unprecedented". Officials haven't said if it involved data mining, but experts say that data mining can be a valuable tool in the war on terror if it is applied properly. The question is: Is it?

On the evening of September 27, 2001, Howard Rubin, a computer science professor at City University of New York who had advised the Clinton administration on technology issues, was home observing Yom Kippur, the holiest day on the Hebrew calendar.

Observant Jews don't work, drive or use appliances on Yom Kippur, but Rubin had a strong feeling he should pick up the phone when it rang that night."

"My wife didn't want me to answer it," he recalls. But he did.

On the other end of the line was one of the most senior members of the previous administration. He wanted to know if Rubin knew of any technologies the government could use to help catch terrorists.

Rubin's answer has since become a technology mantra among members of the intelligence community: data mining, he told the official.

Data mining is a relatively new field within computer science. In the broadest sense, it combines statistical models, power­ful processors, and artificial intelligence to find and retrieve valuable information that might otherwise remain buried inside vast volumes of data. Retailers use it to predict consumer buying patterns, and credit card companies use it to detect fraud. In the aftermath of September 11, the government concluded that data mining could help it prevent future terrorist attacks.

A Proliferation of Projects

Experts say that the US government, and in particular the intelligence community, has come to rely heavily on data mining. A 2004 Government Accountability Office report found that US federal agencies were actively engaged in or planning 199 data mining projects. Of these, 14 focused explicitly on catching terrorists and preventing attacks, a total that does not include projects at seven agencies (such as the CIA and the National Security Agency) that did not respond to the GAO survey. Over the past year, The New York Times, ]USA Today and other media outlets have uncovered top-secret programs within those agencies that collect and look for patterns in phone records, e-mail headers and other personal information. When these programs were made public, President Bush and other members of his administration defended them as critical to the war on terrorism.

Given the administration's commitment to programs using these data mining tools and the pressure on everyone to prevent another attack, it comes as no surprise that these projects are being approved by agency heads in the US almost as fast as they are being conceived, experts say. "There is a real fear of not going down this path, because if there is value you don't want to be on the side that opposed [a data mining project]," says Robert Popp, who was deputy director of the Information Awareness Office at the Defence Advanced Research Projects Agency. Of course, US government officials also have a straightforward reason for pursuing data mining projects, says Robert Gourley, CTO of the Defence Intelligence Agency: "We want to protect our country and our way of life . . . "

No Scope, No Budget, No End

But some experts are beginning to question whether an IT strategy of unlimited scope, budget and schedule will best serve that end. It's a conundrum CIOs face every day. IT projects, no matter how vital, tend to fail when controls don't exist or those controls fall away in the face of a time crunch or crisis. Lack of oversight is the chief cause of project failures, according to the Standish Group, an analyst firm that tracks IT success rates. It leads to overly ambitious projects, an unwillingness to change the original vision and inattention to signs that something isn't working. "It doesn't matter if it is a supply chain project, an ERP system or data mining - those things need to be considered," says Jim Johnson, the Standish Group's chairman.

"No one [in the US government] has looked at data mining from an IT value perspective," says Steve Cooper, former CIO of the US Department of Homeland Security (DHS). "I couldn't figure out [the value of data mining] when I was in DHS, and I can't figure it out now. But that didn't stop us from using it."

In other words, according to Cooper, no one has done a business case analysis to determine whether the government is getting a return on its investment. Instead, a rationalization is usually sufficient: If a project has a chance to catch just one terrorist, then it is worth it.

Given that the US government's track record on IT project management is particularly poor, a lack of typical IT project analysis, prioritization and management controls could backfire. Badly. Experts worry that projects could drag on for years and that good projects could be thrown out with the bad because of privacy and civil liberties issues. (In fact, the US Congress has already halted a number of data mining projects, including the Department of Defence's Total Information Awareness project, an ambitious 2003 attempt to create a massive database containing just about everything and anything that could be used to identify possible terrorists.)

Experts are also concerned that in its zeal to apply technology to antiterrorism, the government could disrupt the crime-fighting processes of the agencies that are charged with finding and stopping terrorists before they act. As any good CIO knows, if users see a system as an obstacle to getting their jobs done effectively, they will rebel or simply ignore it - in this case, with potentially disastrous consequences.

Among data mining experts, there is a growing sense that the US government needs to apply the same kind of analysis to its antiterrorism IT strategy that CIOs in the private sector use to keep their projects from spinning out of control. "These projects have perfectly reasonable goals," says Fred Cate, director of the Centre for Applied Cybersecurity Research at the University of Indiana. (Cate was counsel for the Technology and Privacy Advisory Committee created in 2003 by US Secretary of Defence Donald Rumsfeld to study his agency's use of data mining.) "But there's no oversight procedure," he says.