Background Vulnerabilities Come to the Fore

The initial reaction to slammer was confusion on a Keystone Kops scale. "It was difficult to know just what patch applied to what and where," says NTBugtraq's Cooper, who's also the "surgeon general" at vendor TruSecure.

Slammer hit at a particularly dynamic moment: Microsoft had released Service Pack 3 for SQL Server days earlier. It wasn't immediately clear if SP3 would need to be patched (it wouldn't), and Microsoft early on told customers to upgrade their SQL Server to SP3 to escape the mess.

Meanwhile, those trying to use MS02-061 were struggling mightily with its kludginess, and those who had patched - but got infected and watched their bandwidth sucked down to nothing - were baffled. At the same time, a derivative SQL application called MSDE (Microsoft Desktop Engine) was causing significant consternation. MSDE runs in client apps and connects them back to the SQL Server. Experts assumed MSDE would be vulnerable to Slammer since all of the patches had applied to both SQL and MSDE users.

That turned out to be true, and Cooper remembers a sense of dread as he realised MSDE could be found in about 130 third-party applications. It runs in the background; many corporate administrators wouldn't even know it's there. Cooper estimated it could be found in half of all corporate desktop clients. In fact, at Beth Israel Deaconess Hospital in Boston, MSDE had caused an infestation although the network SQL Servers had been patched.

When customers arrived at work on Monday and booted up their clients, which in turn loaded MSDE, Cooper worried that Slammer would start a reinfestation, or maybe it would spawn a variant. No one knew what would happen. And while patching thousands of SQL Servers is one thing, finding and patching millions of clients with MSDE running is another entirely. Still, Microsoft insisted, if you installed SQL Server SP3, your MSDE applications would be protected.

It seemed like reasonable advice.

Then again, companies take more than a week to stick a service pack into a network. After all, single patches require regression testing, and service packs are hundreds of security patches, quality fixes and feature upgrades rolled together. In a crisis, upgrading a service pack that was days old wasn't reasonable. Cooper soon learned that Best Software's MAS 500 accounting software wouldn't run with Service Pack 3. MAS 500 users who installed SP3 to defend against Slammer had their applications fall over. They would have to start over and reformat their machines. All the while everyone was trying to beat Slammer to the workweek to avoid a severe up-tick in Slammer infections when millions of machines worldwide were turned on or otherwise exposed to the worm that, over the weekend, remained blissfully dormant.

"By late Sunday afternoon, Microsoft had two rooms set up on campus," says Cooper. "Services guys are in one room figuring out what to say to customers. A security response team is in the other room trying to figure out how to repackage the patches and do technical damage control.

"I'm on a [mobile] phone, and there's a guy there running me between the two rooms." Cooper laughs at the thought of it.