How to Build a Monster

Patching is, by most accounts, old as software itself. Unique among engineered artefacts, software is not beholden to the laws of physics; it can endure fundamental change relatively easily even after it's been "built". Automobile engines, by contrast, don't take to piston redesigns once they roll off the assembly line nearly so well.

This unique characteristic of software has contributed to a software engineering culture that generally regards quality and security as obstacles. An adage among programmers suggests that when it comes to software, you can pick only two of three: speed to market, number of features, level of quality. Programmer's egos are wrapped up in the first two; rarely do they pick the third (since, of course, software is so easily repaired later, by someone else).

Such an approach has never been more dangerous. Software today is massive (Windows XP contains 45 million lines of code) and the rate of sloppy coding (10 to 20 errors per 1000 lines of code) has led to thousands of vulnerabilities. CERT published 4200 new vulnerabilities last year - that's 3000 more than it published three years ago. Meanwhile, software continues to find itself running evermore critical business functions, where its failure carries profound implications. In other words, right when quality should be getting better, it's getting exponentially worse.

Patch and Pray

Stitching patches into these complex systems, which sit within labyrinthine networks of similarly complex systems, makes it impossible to know if a patch will solve the problem it's meant to without creating unintended consequences. One patch, for example, worked fine for everyone - except those unlucky users who happened to have a certain Compaq system connected to a certain RAID array without certain updated drivers. In which case the patch knocked out the storage array.

Tim Rice, network systems analyst at Duke University, was one of the unlucky ones. "If you just jump in and apply patches, you get nailed," he says. "You can set up six different systems the same way, apply the same patch to each, and get one system behaving differently."

Raleigh Burns, former security administrator at St. Elizabeth's Medical Centre, agrees. "Executives think this stuff has a Mickey Mouse GUI, but even chintzy patches are complicated."

The conventional wisdom is that when you implement a patch, you improve things. But Wynn isn't convinced. "We've all applied patches that put us out of service. Plenty of patches actually create more problems - they just shift you from one vulnerability cycle to another," Wynn says. "It's still consumer beware."

Yet for many who haven't dealt directly with patches, there's a sense that patches are simply click-and-fix. In reality, they're often patch-and-pray. At the very least, they require testing. Some financial institutions, says Shawn Hernan, team leader for vulnerability handling in the CERT Coordination Centre at the Software Engineering Institute (SEI), mandate six weeks of regression testing before a patch goes live. Third-party vendors often take months after a patch is released to certify that it won't break their applications.

All of which makes the post-outbreak admonishing to "Patch more vigilantly" farcical and, probably to some, offensive. It's the complexity and fragility - not some inherent laziness or sloppy management - that explains why Slammer could wreak such havoc 185 days after Microsoft released a patch for it.

"We get hot fixes everyday, and we're loath to put them in," says Frank Clark, former senior vice president and CIO of Covenant Health, whose six-hospital network was knocked out when Slammer hit, causing doctors to revert to paper-based care. "We believe it's safer to wait until the vendor certifies the hot fixes in a service pack."

On the other hand, if Clark had deployed every patch he was supposed to, nothing would have been different. He would have been knocked out just the same.