The front lines of rootkit defense

Rootkits employ a variety of methodologies to conceal themselves. Some overwrite kernel structures to replace the hooks normally used by Windows commands. Others create files within the file system that are effectively invisible. Still others capture hooks in Windows commands to corrupt their outputs. Many hook into addresses used for kernel services, changing the address of the table entry so the rootkit gets called before the real Windows system call is performed. Extensive details on current approaches to concealment are available at rootkit.com and other Internet sites. One recent methodology posted on rootkit.com involves loading a drive in place of the Windows null.sys dummy driver. The same post outlines three other methods for hiding drivers and offers the code for null.sys replacement.

In terms of defending against infection, Microsoft Windows Vista 64-bit resource protection and Software Restriction Policies in Windows XP provide some assurance, but developers of rogue software have proven their ability to find new ways to hide code on compromised machines. In fact, the rootkit front is fast transforming into an arms race, with each side innovating in response to developments the other camp pushes forward. Keeping on top of the latest modes of prevention is essential, especially if you are responsible for a fleet of computers running any variations of Microsoft Windows.

As for the big security players, a number are appropriating the traditional approach to viruses, using signature-based searches to track down known rootkits and applying related fixes. Two of the major vendors, Symantec and Trend Micro, however, are taking unique tacks in combating rootkits.

Symantec is leveraging mapping technology to discover rootkits on compromised systems. Oliver Friedrichs, director of emerging technologies for security response at Symantec, believes rootkit eradication requires a stable, reliable design that minimizes false positives and mitigates system instability during rootkit removal. To make good on this mission, Symantec has employed the expertise and technology brought on board during the Veritas acquisition. Using VxMS (Veritas Mapping Service), Symantec's Norton Internet Security 2007 maps data on the hard drive, compares it with the Windows file structure, and isolates any discovered mismatches in an effort to repair potential problems. In effect, VxMS enables Norton to compare file systems with the raw data on the disk. Differences are immediately suspect.

For example, say Windows Explorer shows five files in a directory, whereas VxMS shows 10. Clearly, the additional five files are cloaked. Norton sends the suspicious files to Symantec for analysis, eradication occurs during reboot, and the discovered rogue is removed from other systems worldwide as a result.

Trend Micro takes a different approach. Using experience gained in its security labs, the company developed a complete library -- the RCM (Rootkit Common Module) -- to replace the Windows APIs, says Geoff Grindrod, solution product manager at Trend Micro. According to Grindrod, the library includes double encryption to avoid spoofing, and its proxy for API calls is constructed as a special kernel module. With the RCM, the system sees hidden processes, hidden registry keys, and hidden files. As the RCM has matured, it has been integrated into more and more Trend products and is now a core component of anti-spyware and other Trend Micro products, Grindrod says.

Discovering rootkits, however, is only half the battle, as excising them can result in its own set of problems.

"Rootkits are so embedded in the operating system," Mandiant's Butler says. "Plus, we're seeing firmware attacks and survivable rootkits installing themselves in the BIOS. Removing rootkits can also make the system unstable while it's running."

Admins should be aware of the implications of rootkit removal before lunging headlong into the endeavour, says Ron O'Brien, senior security analyst at Sophos, one of the first security vendors to offer a rootkit removal tool.

"Rootkits are not 'bad,' but they have developed a reputation for being bad," O'Brien says. "They are really just a form of hidden files" that may have legitimate uses. Ripping rootkits out before establishing their purpose can prove detrimental to overall system health, he adds.