Late at night, a system administrator performed a routine check of a crashed server, one of 48 systems comprising a major online infrastructure that generated about US$4 million per month in revenue. He was a bit surprised that the system had gone down, as it had been humming for months without any indication of being prone to crashing. The check uncovered three encrypted files. The administrator called on MANDIANT to analyse them.

What MANDIANT found was that an unauthorized kernel modification had caused the system to become unstable, and that the modification had compromised the system's security as well. To determine the extent of the breach, each of the 48 servers needed to be taken offline, booted in a controlled environment, and analyzed for three to five hours each. About half had the crack installed, forcing the company to assume that all credit card information had been compromised. What had first seemed routine resulted in a financial nightmare -- one that many companies are leaving themselves exposed to, unaware of the increasing pervasiveness of rootkits.

Every organization is aware of the importance of securing core systems, networks, and end-user equipment in an increasingly mobile and malware-saturated world. But what most may not realize is the growing threat of malicious software intended to keep its presence hidden from administrators and traditional anti-virus software. Termed after early Unix packages designed to replace commands that would otherwise alert admins to the presence of intruders who had "root" or admin access to systems, rootkits are on the rise among those seeking to steal corporate and personal information for financial gain.

Rootkits alone, of course, are not inherently malicious. But when packaged with malware, they can facilitate deeply compromising security breaches undetected, especially as they become increasingly popular for attacks on non-Unix systems, specifically Windows. And with Forrester Research recently estimating that security breaches cost companies between US$90 and US$305 for each record lost, who can afford to turn a blind eye to what may invisibly be leaching sensitive data from their network?

The rise of rootkits

Rootkits date back to the earliest years of the Internet, when crackers created cloaked variants of Unix commands to ensure their deeds on compromised systems would go undetected. A concern mainly of system administrators for Net-connected Unix systems, rootkits remained relatively low-profile for many years, until Sony BMG Music Entertainment's Windows rootkit DRM (digital rights management) boondoggle of 2005.

In an attempt to enforce copyright protection, Sony BMG developed a rootkit that surreptitiously installed XCP (Extended Copy Protection) or MediaMax CD-3 software when music CDs were played on a PC. Poorly designed, the software opened holes in the Windows OS, facilitating infection by viruses and causing other system problems. Mark Russinovich, now a technical fellow at Microsoft, discovered the rootkit's behavior, which he then announced on his blog. The resulting furor and further illustrations of the fallout of the rootkit led Sony BMG to recall the CDs and issue a removal program. Unfortunately, the removal program was equally poorly designed, leading to additional privacy and security concerns, as documented by Russinovich.

This incident awoke two groups to the potency of Windows rootkits: crackers and professional criminals who break into computers on the one side, and the companies who create software to protect systems on the other. Already entrenched in a high-stakes battle over malware, the two camps now had a new, potentially more damaging front on which to contend. The Computer Economics 2005 Malware Report, the organization's latest, put the cost of malware in 2005 at $US14.2 billion. The ability of malware authors to hide their scripts from anti-virus software's capability of automatically detecting, protecting, and eradicating most malware would only serve to escalate the stakes, especially as malware authors' motivation "continued to shift from a general desire to inflict damage to an intent to gain financially, through theft of personal information such as credit card data or by gaining access to financial accounts," according to the survey.

The greater emphasis on mobility in the enterprise has certainly contributed to the increasing likelihood of infection with cloaked malware. So too are the various unpatched security holes in Microsoft Windows and related products, which provide access for automated rootkit installation. The proliferation of rootkits -- which are used to cloak files on disks, system hooks, and processes running on systems -- is alarming, as spyware developers and malware authors are creating bot networks that use rootkits to evade detection, hiding not only the malware but also what information is being obtained. Some of the more sophisticated rootkits even modify and corrupt Windows APIs. (For more detailed information on rootkits, visit rootkit.com or read Greg Hoglund and Jamie Butler's Rootkits: Subverting the Windows Kernel.)

Part of what's fueling the proliferation of rootkits is the ease with which they can be implemented.

"It has definitely ramped up over the last year and a half to two years," says Butler, principal software engineer at MANDIANT. "It has gotten very easy for malware authors to cut and paste these technologies into their code set to maintain a presence on the machine."

For the time being, malware rootkit use remains crude. "Many of the attacks are unsophisticated," Butler says. "We're not seeing leading-edge rootkit technologies." But the dynamics of intrusion and response that are the hallmarks of the security industry are fast pushing the use of rootkits in innovative directions.