Inside American Eagle Outfitter's 8-Day website nightmare
- 10 August, 2010 08:19
HANG IN THERE. That was the first plea American Eagle Outfitters offered its nearly 1.5 million Facebook friends three days after its trio of e-commerce sites had crashed.
The online retail outlets - ae.com, aerie.com and 77kids.com, which target teens and kids-s uffered what American Eagle Outfitters (AEO) termed a "major incident" on July 19. "This has caused all of our sites to go down," a Facebook update said on July 22, at 1:26 p.m. "We're working together 24/7 to get them up and running as soon as possible. Hang in there....we'll make it up to you. Check back soon."
Those back-to-school shoppers who checked back too soon at the sites received this message: "We're making updates to our sites." This was no planned maintenance, however. Behind the scenes, AEO and its external hosting provider IBM were scrambling for several days as the sites stayed dark.
The reasons for the four-day outage (and subsequent four more days of technical aftershocks) related to backup and disaster-recovery technologies and processes that failed, according to several articles on retail tech site StorefrontBacktalk.com.
As the articles describe, atypical and concurrent failures with IBM's hosting servers and backup plans as well as with Oracle's Data Guard utility program ultimately proved to be the sources of problems. (Stephanie Oschwald, a spokesperson for Pittsburgh-based AEO, did not return several messages seeking comment. IBM and Oracle declined to comment for this article.)
[ Read about e-tailer Zappos.com's $1.6 million pricing mistake ]
On July 23, at 5:39 p.m., AEO announced on Facebook that the sites were A-OK. "We had a few glitches but we're back! We missed you while we were gone. Free shipping now thru Monday, July 26 11:59 PST. Automatic on your entire purchase (US and Canada only)."
However, noted StorefrontBacktalk.com, four more days of technological healing lie in wait for the AEO site, "when it limped along with various parts not functioning until Tuesday afternoon, July 27th."
An Unprecedented Outage
The shock of a 100-hour e-commerce outage now over and the damage done to their critical back-to-school sales period, executives of the $3 billion AEO have to be left scratching their heads in frustration--and wondering just how much "teeth" they had written into their service-level agreements (SLAs) with IBM. (For the record, StorefrontBacktalk.com reported that AEO had recently filled the position of "Manager--Business Continuity & Disaster Recovery," and not a moment too soon.)
AEO's websites weren't the first to go down unplanned, nor will they be the last. Dating back to the mid-1990s, history is littered with tales of site outages, complex IT fixes and lost revenue--embarrassing for the companies and career killers for those in charge. Any Amazon.com unplanned downtime, for instance, is still reported with as much zeal as a Jennifer Aniston "new boyfriend" sighting.
But the duration of AEO's site outages could be considered unprecedented for an e-retailing brand.
"In my career, I can't remember a mission-critical application that was down for four days," says Paula Rosenblum, an analyst at RSR Research and former retail CIO. "It's almost impossible to conceive--and I say that not as an analyst but as a former CIO."
Deals, Discounts and Disappointment
The timing couldn't have been worse for AEO, at the outset of its "back to school" summer push. In its stores, AEO's July performance was a "disappointment," according to a Zacks Investment Research report on retailers. The retailer had also lowered its quarterly earnings outlook in July.
The company not only lost online revenue during the outage, but also attempted to make up for the downtime by offering its customers deep promotions--such as this deal promoted on its Facebook page on July 30: "Thanks for hanging in there while our site was down...this Friday & Saturday (7/30 & 7/31) online only take 25% off your entire order & we'll throw in free shipping, too!... Thanks again for hanging in there!"
Brian Walker, a Forrester Research principal analyst who follows e-commerce, doesn't term the AEO outages unprecedented, but unusual. "Sites routinely will become slow and the customer experience and conversion rate may be affected," he says via e-mail, "but to be down for four days has a significant business and customer impact."
AEO's "direct businesses" (those three websites) rung up more than $340 million in sales in fiscal 2009, which accounted for approximately 12 percent of the retailer's overall sales.
Outsourcing E-Commerce Ops: Deal or No Deal?
Outsourcing e-commerce operations in retail is not a new proposition--and IBM and Oracle, for that matter, are not fly-by-night operations. "Using managed services or SaaS solutions for all or key parts of the e-commerce platform is routine today," says Forrester's Walker. "On average, eight hosts will be involved in supporting a typical e-commerce transaction."
Even with the site outages, the outsourcing proposition is still sound, Walker says. "There is no guarantee that doing this in-house is any better," he opines, "and in many ways can be much harder to do as resources, budgets and skillsets may be harder to have at the right levels."
However, it's important that a company outsourcing its operations not abdicate every decision and responsibility over to the hosting company. "Often in these situations there will be some shared responsibility between the business and IT-support staffs," Walker says. Trouble happens, he says, when something as simple as a failure to communicate an upcoming big promotion or event that will likely result in a spike in online traffic big enough to take down a website.
Interestingly, during the sites' outages, AEO's mobile site was still functional, according to StorefrontBacktalk.com. RSR Research's Rosenblum says that's a "good news / bad news" situation. The good news: The mobile site was up, and AEO could have done a customer redirect from the online site that was down to the mobile site, if they chose. The bad news: It would appear that AEO is running separate, siloed systems (for customer databases and queries, for instance) for what are, essentially, similar e-commerce channels, Rosenblum says.
"Our research shows us that too many retailers are in a similar situation," she says. "They spend an inordinate amount of scarce resources--IT and human--forcing multiple channels into synch. The opportunity costs associated with these activities are incalculable."
Forrester's Walker advises other retailers to use AEO's outage as a wake-up call. "Get your responsibilities, processes and systems together now, test them, and correct as needed," he says. "Having a clear plan in and of itself will pay huge dividends in the event something happens, and with the technology and architectures we have available today this can be done at a reasonable cost."
"Consider the downside of having the business offline for a few days during the holidays," he adds, "and the ROI is pretty clear."
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
- Incompatible Browser : Facebook
- StorefrontBacktalk » Blog Archive » Oracle Backup Failure Major Factor In American Eagle 8-Day Crash
- How Much Is a Simple User Interface Worth? : CIO - Blogs and Discussion
- Salesforce.com Nailed for Downtime: But Your In-House CRM and ERP Record May Be Worse : CIO - Blogs and Discussion
- Colorado Rockies Computer Crash Provides Lessons in Redundant Systems and Contingency Planning : CIO - Blogs and Discussion
- Amazon.com unplanned downtime
- mission-critical application
- Retail Systems Research - Home
- Zacks Analyst Blog Highlights: Abercrombie &amp; Fitch, Gap, Macy, American Eagle Outfitters and J.C. Penney - Yahoo! Finance
- American Eagle Outfitters - Investor Relations - Press Release
CIOs need to get their house in order, CFO panel says
Is Data Complexity Blinding Your IT Decision-Making?
Why IT projects really fail
CIOs say cost, complexity impede true mobile gains in enterprise
The enlightened CIO’s guide to running projects
Multi-Factor Authentication; Current Usage and Trends
In this digital age, validating identities and controlling access is vital, which is why multifactor authentication has become such a fundamental requirement in so many organisations. This survey looks at the authentication landscape in Europe, the Middle East, and Africa, and offers insights into how it is expected to change in the coming years.
Efficient Data Management in Three Simple Steps
Gartner reports that Business Intelligence, Mobile Technologies and Cloud Computing rank 1-2-3 as the 2013 Global CIO Technology Priorities. These three trends, labelled the “Perfect Storm” of new technologies, are transforming every link in the IT value chain, promising to deliver more efficient, responsive and dynamic IT operations. But this also means massive shifts in the way IT applications and services are created, deployed and maintained. This whitepaper aims to help you begin the journey to efficient modern data management
Deliver Enterprise Mobility with Security and Performance
Mobility and the consumerisation of IT pose key challenges for IT around scalability, security and application visibility. In this whitepaper, we look at complete, integrated and scalable solutions that deliver apps and data to any device with full security and a high-performance user experience. Learn more!