Hurricane Katrina proves why it most definitely pays to have a disaster recovery plan in place

"Monday was not a good day." That's how Ray Johnson, not one for hyperbole, remembers August 29, 2005 - the day Hurricane Katrina roared ashore on the US Gulf Coast. But there's a reason for the understatement. Johnson, CIO of New Orleans-based Entergy, would see darker days to follow.

It's now two and a half weeks after the killer storm left a devastated landscape in its wake. Johnson has had little time to reflect on his company's efforts not only to restore power to its customers affected by the hurricane but also to prevent severe business interruption despite a virtual attack on its own physical and human infrastructure. But he spoke to CIO on September 15 to discuss Entergy's preparation for and response to the worst disaster in the company's history.

Entergy's recovery efforts can be traced back to long before Katrina hit. The $US10 billion energy company has a disaster recovery plan that's tested and revised once a year. Last year, that "test" came in the form of Hurricane Ivan. Entergy activated its storm command centre and disaster recovery processes in response to the approaching storm which ultimately made landfall further east in Pensacola, Florida. And just this April, the company conducted an extensive storm drill that simulated what corporate executives thought of as a worst case scenario - a major hurricane followed by extreme flooding.

"This is not a unique event for us," explains Johnson, "I wish it was. But we've got our disaster plan nailed. We review it at least once a year and either we conduct a drill or get to test it when a hurricane threatens us and then misses."

On Friday, August 26, it looked like Johnson would have another opportunity to test the disaster recovery plan. Katrina was crossing over Florida and beginning to churn in the Gulf. But early in the day, the hurricane was predicted to hit several hundred miles east of New Orleans.

As Friday evening wore on, the situation changed. This was not a test. "We started making our storm calls and made the decision to activate our disaster recovery plan," which calls for some preliminary actions to be taken beginning 72 hours before a category 3 hurricane is scheduled to make landfall," recalls Johnson.

Saturday morning 5am, Johnson sent his first "away" team to the active disaster recovery site Entergy has in Little Rock, Arkansas. Entergy's primary data centre is located in Gretna, Louisiana, across the river from Entergy's corporate headquarters. The backup generators supporting that data centre had never failed before, but Johnson knew there could be one of many gloomy firsts he might encounter this time out. "By late in the day [on Saturday] it became more and more obvious as Katrina tracked closer and closer that New Orleans was going to be a primary target." Once the first team arrived safely, a second was deployed.

Around that time the decision was made to do a full implementation of the disaster recovery plan and Entergy's core restoration team flew into action, led by Randy Helmick, vice president of customer service and support during normal operations but given the title of "storm boss" when an emergency is declared.