Good project management isn't enough; CIOs need a worst-case contingency plan for the business

Reader ROI

• How minor glitches in HP's ERP rollout snowballed into a $US160 million disaster
• The limitations of IT project management to address business problems stemming from project snafus
• Why business contingency plans are necessary for enterprise IT projects

It goes against human nature to always expect the worst. But with IT projects, pessimism - otherwise known as contingency planning - is the only way to keep small technology problems from becoming full-blown business disasters.

Too bad no one can bring themselves to do enough of it.

Christina Hanger had little reason to be pessimistic in May 2004, when she was moving one of Hewlett-Packard's biggest North American divisions onto a centralized ERP system from SAP. As the leader of an IT consolidation project rooted in HP's acquisition of Compaq two years earlier, Hanger, HP's senior vice president of Americas operations and IT, had an unbroken record of success migrating five product groups within the two former companies onto one of two SAP systems.

Hanger had every reason to believe that the sixth would go well too. Even so, she knew to be prepared for problems. At approximately $US7.5 billion in annual revenue, the division involved with this latest project, Industry Standard Servers (ISS), is much larger than any of the others that Hanger had migrated to SAP to that point. So Hanger took the contingency plan that her team had developed for the other five migrations and adjusted it to accommodate the ISS division's larger sales volume. She planned for three weeks of IT snafus, mostly focused on what might happen as a result of tweaking a legacy order-entry system to work with the new SAP system. The contingency plan addressed business impacts too. HP banked three weeks' worth of extra servers and took over an empty portion of an HP factory in Omaha to stand by for any overflow of orders that needed special configurations (for example, an unusual component or software combination) and could not be stockpiled ahead of time.

But the plan wasn't pessimistic enough.

Starting when the system went live at the beginning of June and continuing throughout the rest of the month, as many as 20 percent of customer orders for servers stopped dead in their tracks between the legacy order-entry system and the SAP system. As IT problems go, this wasn't too big: Some data modelling issues between the legacy system and the SAP system prevented the SAP system from processing some orders for customized products. These programming errors were fixed within four or five weeks. But Hanger and her business colleagues from the ISS division who were on the project steering committee never envisioned the degree to which these programming glitches would affect the business.

Orders began to backlog quickly and HP did not have enough manual workarounds to keep servers flowing fast enough to meet customer demand. Angry customers picked up the phone and called HP - or worse, arch-competitors Dell and IBM. In a commodity market such as servers, customer loyalty is built upon a company's ability to configure products to order and get them delivered on time. HP could do neither for much of the northern summer. In a third-quarter conference call on August 12, HP chairman and CEO Carly Fiorina pegged the financial impact at $US160 million: a $US120 million order backlog that resulted in $US40 million in lost revenue. That's more than the cost of the project itself, which AMR Research estimates to be $US30 million.

The headlines all claimed an IT disaster, but in fact, HP's disaster resulted from a few relatively small problems in IT that snowballed into a much bigger problem for the business: the inability to cope with the order backlog. This was a disaster that could have been prevented - not by trying to eliminate every possibility for error in a major IT system migration, which is virtually impossible, but by taking a much broader view of the impact that these projects can have on a company's supply chain.

CIOs don't run the supply chain in most companies, so they have trouble envisioning what will happen to sales if a critical system doesn't function as expected for a few days or weeks. Businesspeople, meanwhile, have trouble imagining an IT programming glitch getting past the walls of the data centre and causing hundreds of millions of dollars' worth of lost sales. The chasm between cause and effect is almost too vast to contemplate.

But if CIOs want to stop being held liable for hundreds of millions of dollars in losses for relatively small IT problems, they have to convince business leaders of the vastly increased risks that major enterprise software projects pose to businesses with high-volume supply chains. They must use that awareness to build - with full support and cooperation from the business - a business contingency plan for IT projects that is as robust as the project plans they create for the new software.

"The potential benefits to the supply chain are much bigger than the IT costs in projects like this," says Bill Swanton, vice president of research for AMR Research. "But the potential risk to the supply chain is also much bigger." If business contingency planning continues to play a secondary role to IT project management, the problem is only going to get worse as computer systems become more powerful, integration methods improve, and companies consolidate their critical business processes on fewer and fewer systems.

In retrospect, HP CIO and executive vice president of Global Operations Gilles Bouchard does not see the data modelling problems between the legacy and SAP systems as the source of the $US160 million impact. He focuses on HP's inability to keep pace with orders in the supply chain once the problems were discovered. "It was mostly capacity issues, material issues and factory issues," he says. "We had a series of small problems, none of which individually would have been too much to handle. But together they created the perfect storm."