The story of a botched CRM upgrade that cost the telco thousands of new customers and an estimated $US100 million in lost revenue. Hard lessons learned.

Jack Lee maked November 24, 2003, on his calendar because that was the day he would finally be able to change his mobile phone carrier without losing his phone number - thanks to a US Federal Communications Commission ruling. But Lee, president of Tangara Technologies, a company that develops software for forms, decided to wait a day before switching to AT&T Wireless to let the chaos of "number porting" die down a little. Little did he know that the chaos was just beginning.

Lee ordered his new phone on November 25. When he went to the AT&T Wireless Web site to check the status of his order a day later, he was greeted with a message: "We could not find a porting request for this number in the system. Please contact Customer Care." It was the beginning of a two-month odyssey in which Lee estimates he made 15 to 20 calls to AT&T Wireless, sent nearly as many e-mails and spent 60 hours on the phone dealing with customer service representatives or waiting on hold - with the line often going dead when AT&T Wireless's customer service lines became overloaded.

After being routed all over the company, Lee finally discovered what was going on. A major CRM system had crashed during an upgrade, and customer service representatives could not set up or access new accounts. The system breakdowns, which continued through February 2004, swamped other AT&T systems, gridlocked customer service phone banks and sent furious customers scurrying to other providers.

The breakdown couldn't have come at a worse time for AT&T Wireless. It deprived the telco of thousands of potential new customers and cost the company an estimated $US100 million in lost revenue. But that wasn't all. The failure so damaged AT&T Wireless's reputation that many analysts believe it hastened its sale to Cingular in February for $US41 billion, or $US15 per share, which was just under half the value of AT&T Wireless's shares when it went public in April 2000. While an AT&T Wireless spokesman says the company would have been sold regardless of the fiasco because "it was the right thing to do", the crash and resulting confusion could not have helped AT&T cut a good deal. "The system problems made AT&T look like a wounded provider, and the sharks smelled blood and circled," says Roger Entner, a wireless and mobile services analyst at The Yankee Group, a research company.

AT&T Wireless's mistakes offer valuable lessons for CIOs. For one, it's unwise to freight major system upgrades with external complications. AT&T Wireless's CRM upgrade was hamstrung from almost the very beginning by rumours of outsourcing deals and future lay-offs. These rumours generated pervasive morale problems that hurt the productivity of project staff.

Second, it should be understood that complex projects require flexible deadlines. AT&T Wireless undertook a difficult upgrade that affected roughly 15 systems just before it was faced with an immovable deadline - the federally mandated November 24 number portability date.

Finally, it always pays to have a plan B. Without one, AT&T Wireless was forced to move forward even as it became apparent that its upgrade would not be completed in time.