Frank Boldewin had seen a lot of malicious software in his time, but never anything like Rustock.C.

Used to infect Windows PCs and turn them into unwitting spam servers, Rustock.C is a rootkit that installs itself on the Windows operating system and then uses a variety of sophisticated techniques that make it nearly impossible to detect or even analyze.

When he first started looking at the code earlier this year, it would simply cause his computer to crash. There was driver level encryption, which had to be decrypted, and it was written in assembly language, using "spaghetti code structure" that made it extremely hard for Boldewin to figure out what the software was actually doing.

Analyzing a rootkit is typically an evening's work for someone with Boldewin's technical skills. With Rustock.C, however, it took him days to figure out how the software worked.

Because it is so hard to spot, Boldewin, a security researcher with German IT service provider GAD, believes that Rustock.C had been around for nearly a year before antivirus products began detecting it.

This is the story with rootkits. They're sneaky. But are they a major threat?

In late 2005, Mark Russinovich discovered the most famous rootkit. A windows security expert, Russinovich was baffled one day when he discovered a rootkit on his PC. After some sleuthing, he eventually discovered that copy protection software used by Sony BMG Music Entertainment actually used rootkit techniques to hide itself on computers. Sony's software wasn't designed to do anything malicious, but it was virtually undetectable and extremely difficult to remove.

Sony's rootkit became a major PR disaster for the company, which spent millions in legal settlements with users who were affected by the software.

Three years later, Russinovich, a technical fellow with Microsoft, still considers it the rootkit that caused the most trouble for computer users.

But the Sony rootkit presaged problems for the antivirus vendors too. The fact that none of them had even noticed this software for about a year was a serious black eye for the security industry.

Though they got their start on Unix machines years earlier, at the time of the Sony fiasco, rootkits were considered the next big threat for antivirus vendors. Security researchers explored the use of virtualization technology to hide rootkits and debated whether a completely undetectable rootkit could someday be created.