How AmEx disciplined unruly software updates

The once-sprawling software development groups at American Express have been reined in. Developers, testers and deployment staff are still part of business units around the United States and overseas, but now they work according to one repeatable process.

Unifying test and deployment methods across the company has sped up the pace of software updates and helped make agile development a success. That discipline has also helped ensure that the 100,000 application builds the company supports every month are stable, says Paul Stoxen, director of the development center and application development security at the $27.8 billion financial-services company. Builds can include application compiling, test execution, vulnerability scans and other changes that are being done almost continuously, Stoxen says.

At a large company tweaking applications nearly all the time, repeatable processes help improve reliability and stability, says Jeffrey Hammond, an analyst at Forrester. The more IT tests and deploys software updates, the better the group gets at it, Hammond says. "If I only practice guitar once a quarter, I'm not going to be very good."

The effort to unify software testing and deployment is part of a larger initiative Stoxen and his group launched a few years ago to strip waste from application-development work. Previously, development and deployment groups in one business unit might work on three- or four-week cycles, using their own set of homegrown processes for managing their work, while counterparts in another area of the company might work on quarterly cycles with entirely different processes. The result was that the software updates were rolled out at inconvenient times for business units, and IT staff assigned to one area couldn't be easily redeployed because they had to learn new, sometimes convoluted, ways of doing things, Stoxen says.

To coax developers and others to change their habits, Stoxen and his team heavily marketed their ideas-at meetings and presentations and in informal conversations. Stoxen also created a logo for the development center to help brand the new initiative.

Now, with frequent updates that flow into production via predictable release cycles, American Express is able to roll out changes demanded by business units faster, he says. The change also allows more staffing flexibility: Developers, testers and other staff can move quickly between business units and teams because they already know how things are done.

The rise of agile development is pushing corporate test and deployment groups to pick up the pace, says Hammond. Agile processes usually allow developers to deliver code every two weeks, but deployment isn't moving that fast at many companies, he says. The resulting clog frustrates developers and end users waiting for upgrades. "Your release schedule must be in tempo with the development cycle," Hammond says.

Tools to help create standard deployment templates and track release cycles include those from Cfengine, MaestroDev, Nolio and Puppet Labs. American Express uses a variety of products, Stoxen says, but he declines to name them.

Next, Stoxen says he would like to develop business analytics systems to identify patterns in how the development center staff makes software changes, to find the most efficient methods. "Consistency is what will make us more efficient," he says. "At this size, that's where you see the best gains."

Follow Senior Editor Kim S. Nash on Twitter: @knash99.

Read more about development in CIO's Development Drilldown.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

• Bookmark this page
• Share this article
  •  
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email CIO
• Follow CIO on twitter

• Share
  Share this article

  •  google+
  • facebookfacebook
  • slashdotslashdot
  • diggdigg
  • RedditReddit
  • stumbleuponstumbleupon
  • linkedinlinkedin
  • twittertwitter
• print
• email
• Bookmark