Critical.
Authoritative.
Strategic.
Subscribe to CIO Magazine »

Inside American Eagle Outfitter's 8-Day website nightmare

What happens when a business suffers an 'unprecedented outage'

HANG IN THERE. That was the first plea American Eagle Outfitters offered its nearly 1.5 million Facebook friends three days after its trio of e-commerce sites had crashed.

The online retail outlets - ae.com, aerie.com and 77kids.com, which target teens and kids-s uffered what American Eagle Outfitters (AEO) termed a "major incident" on July 19. "This has caused all of our sites to go down," a Facebook update said on July 22, at 1:26 p.m. "We're working together 24/7 to get them up and running as soon as possible. Hang in there....we'll make it up to you. Check back soon."

Those back-to-school shoppers who checked back too soon at the sites received this message: "We're making updates to our sites." This was no planned maintenance, however. Behind the scenes, AEO and its external hosting provider IBM were scrambling for several days as the sites stayed dark.

The reasons for the four-day outage (and subsequent four more days of technical aftershocks) related to backup and disaster-recovery technologies and processes that failed, according to several articles on retail tech site StorefrontBacktalk.com.

As the articles describe, atypical and concurrent failures with IBM's hosting servers and backup plans as well as with Oracle's Data Guard utility program ultimately proved to be the sources of problems. (Stephanie Oschwald, a spokesperson for Pittsburgh-based AEO, did not return several messages seeking comment. IBM and Oracle declined to comment for this article.)

[ Read about e-tailer Zappos.com's $1.6 million pricing mistake ]

On July 23, at 5:39 p.m., AEO announced on Facebook that the sites were A-OK. "We had a few glitches but we're back! We missed you while we were gone. Free shipping now thru Monday, July 26 11:59 PST. Automatic on your entire purchase (US and Canada only)."

However, noted StorefrontBacktalk.com, four more days of technological healing lie in wait for the AEO site, "when it limped along with various parts not functioning until Tuesday afternoon, July 27th."

An Unprecedented Outage

The shock of a 100-hour e-commerce outage now over and the damage done to their critical back-to-school sales period, executives of the $3 billion AEO have to be left scratching their heads in frustration--and wondering just how much "teeth" they had written into their service-level agreements (SLAs) with IBM. (For the record, StorefrontBacktalk.com reported that AEO had recently filled the position of "Manager--Business Continuity & Disaster Recovery," and not a moment too soon.)

AEO's websites weren't the first to go down unplanned, nor will they be the last. Dating back to the mid-1990s, history is littered with tales of site outages, complex IT fixes and lost revenue--embarrassing for the companies and career killers for those in charge. Any Amazon.com unplanned downtime, for instance, is still reported with as much zeal as a Jennifer Aniston "new boyfriend" sighting.

But the duration of AEO's site outages could be considered unprecedented for an e-retailing brand.

"In my career, I can't remember a mission-critical application that was down for four days," says Paula Rosenblum, an analyst at RSR Research and former retail CIO. "It's almost impossible to conceive--and I say that not as an analyst but as a former CIO."

Deals, Discounts and Disappointment

The timing couldn't have been worse for AEO, at the outset of its "back to school" summer push. In its stores, AEO's July performance was a "disappointment," according to a Zacks Investment Research report on retailers. The retailer had also lowered its quarterly earnings outlook in July.

The company not only lost online revenue during the outage, but also attempted to make up for the downtime by offering its customers deep promotions--such as this deal promoted on its Facebook page on July 30: "Thanks for hanging in there while our site was down...this Friday & Saturday (7/30 & 7/31) online only take 25% off your entire order & we'll throw in free shipping, too!... Thanks again for hanging in there!"

Brian Walker, a Forrester Research principal analyst who follows e-commerce, doesn't term the AEO outages unprecedented, but unusual. "Sites routinely will become slow and the customer experience and conversion rate may be affected," he says via e-mail, "but to be down for four days has a significant business and customer impact."

AEO's "direct businesses" (those three websites) rung up more than $340 million in sales in fiscal 2009, which accounted for approximately 12 percent of the retailer's overall sales.

Outsourcing E-Commerce Ops: Deal or No Deal?

Outsourcing e-commerce operations in retail is not a new proposition--and IBM and Oracle, for that matter, are not fly-by-night operations. "Using managed services or SaaS solutions for all or key parts of the e-commerce platform is routine today," says Forrester's Walker. "On average, eight hosts will be involved in supporting a typical e-commerce transaction."

Even with the site outages, the outsourcing proposition is still sound, Walker says. "There is no guarantee that doing this in-house is any better," he opines, "and in many ways can be much harder to do as resources, budgets and skillsets may be harder to have at the right levels."

However, it's important that a company outsourcing its operations not abdicate every decision and responsibility over to the hosting company. "Often in these situations there will be some shared responsibility between the business and IT-support staffs," Walker says. Trouble happens, he says, when something as simple as a failure to communicate an upcoming big promotion or event that will likely result in a spike in online traffic big enough to take down a website.

Interestingly, during the sites' outages, AEO's mobile site was still functional, according to StorefrontBacktalk.com. RSR Research's Rosenblum says that's a "good news / bad news" situation. The good news: The mobile site was up, and AEO could have done a customer redirect from the online site that was down to the mobile site, if they chose. The bad news: It would appear that AEO is running separate, siloed systems (for customer databases and queries, for instance) for what are, essentially, similar e-commerce channels, Rosenblum says.

"Our research shows us that too many retailers are in a similar situation," she says. "They spend an inordinate amount of scarce resources--IT and human--forcing multiple channels into synch. The opportunity costs associated with these activities are incalculable."

Forrester's Walker advises other retailers to use AEO's outage as a wake-up call. "Get your responsibilities, processes and systems together now, test them, and correct as needed," he says. "Having a clear plan in and of itself will pay huge dividends in the event something happens, and with the technology and architectures we have available today this can be done at a reasonable cost."

"Consider the downside of having the business offline for a few days during the holidays," he adds, "and the ROI is pretty clear."

Do you Tweet? Follow me on Twitter @twailgum. Follow everything from CIO.com on Twitter @CIOonline.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

More about: Amazon, Amazon.com, American Eagle Outfitters, Facebook, Forrester Research, IBM, IBM Australia, Oracle, Salesforce.com, Yahoo
References show all
Comments are now closed.
Related Coverage
Related Whitepapers
Latest Stories
Community Comments
Tags: e-commerce, Utilities, backup, American Eagle Outfitters, software, internet
Latest Blog Posts
Whitepapers
  • Stop Paying the Earth for Global Roaming
    Why do we continue to pay the earth for global roaming? With Telstra increasing global roaming charges by 100-500% in over 180 countries, bill shock can only get worse. This paper investigates why, what and how your company can address the need for global coverage.
    Learn more »
  • How to Successfully Select an ERP System
    An Enterprise Resource Planning (ERP) system is a series of software applications that collect and compiles data from different departments to enhance collaboration and co-ordination within the business. If you’re looking to implement your first ERP system, or to upgrade from an existing system, this whitepaper offers eight simple steps for selection that will lead to long-term strategic success.
    Learn more »
  • Rebranded Quadmark revamps its IT solutions with Google Apps
    The Singapore office was using Exchange as its email server but encountered various issues such as storage capacity limitations and difficulty in managing spam. Adding new users to the server was also a hassle that often required a third party vendor, resulting in a waste of time and resources. Quadmark also experienced email performance issues that slowed down their employees’ response time, leading to frustration among staff and clients. Quadmark’s management felt that it was unacceptable to continue it’s current solution and thus decided to streamline its IT infrastructure alongside its rebranding plans. The business wanted a unified and consolidated email service for its various offices. Quadmark also wanted to be able to house files and documents on the cloud.
    Learn more »
All whitepapers
rhs_login_lockGet exclusive access to Invitation only events CIO, reports & analysis.
Latest Jobs
Salary Calculator

Supplied by

View the full Peoplebank ICT Salary & Employment Index

Recent comments

Computerworld
ARN
Techworld
CMO