Why Vendors Hate It

Attribute the sharpness of the bite to the emergence of benchmarking companies around 1998. Benchmarkers had been evaluating IT shops' internal operations since the late 80s, but when the outsourcing boom hit they came up with an enticing pitch for CIOs: Let us invoke the benchmarking clause, and we guarantee to cut your prices. With benchmarks costing anywhere from $100,000 to a million dollars, depending upon scope, it was hard for benchmarking firms to seal the deal without pledging to get at least some of that money back. But that promise poisoned their objectivity, some say. "When you say we guarantee to save you 20 percent, you're not really being objective," says Adam Strichman, senior partner at research and benchmarking company Nautilus Advisors and former director of outsourcing strategies at Meta Group.

The outsourcers screamed foul. The whole benchmarking process, they contended, was a crock: Service provider cost models are complicated; there are financial and operational dependencies among different services; outsourced services are rarely commodities but rather are sold in varying combinations upon a wide array of terms. For example, the price for mainframe services can vary by plus or minus 40 percent, says Stewart of EDS, depending on what software is running on the machines. "You have to understand the client environment, service levels and the financial assumptions that went into the deal to come up with an accurate benchmark," he says. "And benchmarkers don't know the deal specifics."

Furthermore, say the outsourcers, the benchmarkers' attempts to account for the variations among companies — what is euphemistically known as "normalization" in the industry — isn't an adequate substitute for real numbers and knowledge. "The vendors complain benchmarking is more art than science," says Robinson of EquaTerra. "And a black art at that."

The Benchmarking Business

Benchmarkers, whose overall market now exceeds $US200 million per year, according to one industry analyst who asked not to be named, concede some of these points. "There is an art to it," admits Strichman. "Benchmarking is far more than statistical analysis and number-crunching." Benchmarkers say that over the years they have refined their processes to reconcile unlike data and adjust for differences such as the client environment and service levels.

But while benchmarking may require some apples-to-oranges comparisons and fact-finding to adjust for different environments, it's the only method currently available for CIOs to ensure the competitiveness of the prices they're paying. "There's value in the benchmarking clause. It gives a customer security when signing a long-term contract with a single provider," says Neil Barton, Hewlett-Packard Services' benchmarking manager for Europe.

And while outsourcers complain that the benchmarkers' numbers don't add up, they are loath to admit that their own numbers don't add up either. Service providers typically grant customers a great price on IT services on day one but backload their costs to recover initial investments later on. Compass reports seeing year-one savings of as much as 18 percent (usually 10 percent to 15 percent) turn into costs in excess of 23 percent above market rate by year three and more than 35 percent in longer deals. Outsourcers aren't likely to open up their books and show you how they've arrived at your charges, so benchmarking is an important tool. "Whatever its limitations," says Kimball, "benchmarking can be an effective catalyst for renegotiations that raise service levels and reduce charges."