Bursting the CMM Hype

CIOs want to do business with offshore companies with high CMM ratings. But some outsourcers exaggerate and even lie about their Capability Maturity Model scores.

As soon as she walked into the meeting, Jane Smith knew that the executive on the other side of the desk wanted to buy something that Smith wasn't supposed to sell: a trumped up rating for the executive's software development division so that his company could qualify to bid on contracts from the United States Department of Defence.

Smith (not her real name) is one of a select group of experienced IT pros, called lead appraisers, who go into companies and assess the effectiveness of their software development processes on a scale from 1 (utter chaos) to 5 (continuously improving) under a system known as the Capability Maturity Model, or CMM. The company she was visiting wanted to move up to Level 2, but based on some initial discussions, Smith knew that the company was a 1. Level 1 describes most of the software development organizations in the world: no standard methods for writing software, and little ability to predict costs or delivery times. Project management consists mostly of ordering more pizza after midnight.

After a few initial niceties, the executive leaned across the table to Smith and another lead appraiser who had accompanied her to the meeting and asked: "How much for a Level 2?"

"That's when I got up and left the room," Smith recalls. "The other appraiser stayed. And the company got its rating."

The stakes for a good CMM assessment have got only higher since Smith's close encounter with corruption some 10 years ago. Today, many US government agencies in addition to the DoD insist that companies that bid for their business obtain at least a CMM Level 3 assessment - meaning the development organization has a codified, repeatable process for an entire division or company. CIOs increasingly use CMM assessments to whittle down the lists of dozens of unfamiliar offshore service providers - especially in India - wanting their business. For CIOs, the magic number is 5, and software development and services companies that don't have it risk losing billions of dollars worth of business from American, European - and increasingly - Australian corporations.

"Level 5 was once a differentiator, but now it is a condition of getting into the game," says Dennis Callahan, senior vice president and CIO of Guardian Life Insurance. "Having said that, there are some Level 3 or 4 start-ups that we might consider, but they have a lot more convincing to do before I would do business with them. They would be at a disadvantage."

With CIOs increasingly dependent on outside service providers to help with software projects, some have come to view CMM (and its new, more comprehensive successor, CMM Integration, or CMMI) as the ultimate seal of approval for software providers. Yet CIOs who buy the services of a provider claiming that seal without doing their own due diligence could be making a multimillion-dollar, career-threatening mistake.

That's because software providers routinely exaggerate their assessments, leading CIOs to believe that the entire company has been assessed at a certain level when only a small slice of the company was examined. And once providers have been assessed at a certain level, there is no requirement that they test themselves ever again - even if they change dramatically or grow much bigger than they were when they were first assessed. They can continue to claim their CMM level forever.

Worse, some simply lie and say they have a CMM assessment when they don't. And appraisers say they occasionally hear about colleagues who have had their licences revoked because of poor performance or outright cheating in making assessments.

Join CIO, the CIO Executive Council & IDC on 6 October at Australia’s premier Melbourne event for senior IT executives – the CIO Summit 2010. Find out more or register now.

• Bookmark this page
• Share this article
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email CIO
• Follow CIO on twitter

Comments