Wanted: Metric Sceptics

Accepting vague or irrelevant numbers for project value can lead to bad IT spending decisions.

I confess. I'm a numbers kind of guy. But I'm concerned. I see too many numbers, with too little meaning, cavalierly factored into too many IT value discussions. Metrics are often management's single most important determinant of value. So allowing anything less than top quality numbers - and the correct context for these measurements - leads to wrong decisions about which IT investments to select, which implementation paths to choose or what portfolios to manage.

There are three common ways that people misuse metrics: tolerating ambiguity, discouraging guess­timates and overlooking available sources. Let's take a look at some solutions for avoiding these missteps.

Tolerating Ambiguity

Numbers falsely convey a mystique of truth, by their very existence. But rarely do numbers actually mean what they seem to imply. That is because a metric is not "the result". It is shorthand for a story about the result. If the metric user leaves out the story portion of the metric, we are left to make potentially erroneous assumptions to fill in the gaps. For example, if someone states: "We have a 25 per cent customer retention rate", he is providing us with an ambiguous metric. Because the story behind the metric is missing, his message has no infallible meaning. In this example, a good metric story might say: "Our internal audit committee confirmed last April that our Southern division has experienced a 25 per cent customer retention rate, for those customers who have bought at least $10,000 annually from us, for the past three years in a row. Sue Smith, vice president of finance, has stated that this retention rate is too low to allow us to realise the gross margin goals for this year." Armed with that explanation, we are less likely to misuse the metric's meaning and implications, and can make an informed decision.

Discouraging Guesstimates

Another misuse of numbers occurs when management refuses to accept guesstimates (informed estimates) as legitimate metrics. This problem occurs whenever someone says: "If you can't prove it, we won't use it." In this mistaken worldview, only hard metrics from factual situations are valid. The clear message is that guesses don't count. The reality is that informed guesses are what makes the world work. If business investment strategies were 100 per cent fact-based, we wouldn't need high-price executives to guide the company. Computers could probably do the job via fancy calculations. The whole discipline of risk management is ultimately grounded in probabilities, which themselves are estimates. The medical profession begins and ends with the reliability of doctor diagnoses, none of which are 100 per cent correct. But they get enough right that we don't ban their profession. A similar situation exists with our economy's reliance on meteorologists. They don't hit the mark 100 per cent of the time. But their track record is good enough for planes to fly safely through troubling storms, not to mention properly attiring our families for school and work. The key to accepting informed guesses as valid metrics is to make sure they: A. Come from knowledgeable, experienced and dependable people; and B. Are accompanied by clear explanations of the guesstimate's premises, assumptions and logic.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

• Bookmark this page
• Share this article
  •  
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email CIO
• Follow CIO on twitter

• Share
  Share this article

  •  google+
  • facebookfacebook
  • slashdotslashdot
  • diggdigg
  • RedditReddit
  • stumbleuponstumbleupon
  • linkedinlinkedin
  • twittertwitter
• print
• email
• Bookmark