Everything is Measurable

Everything is Measurable

The "IT value measurement problem" is ubiquitous in companies today. Business executives often hear a mantra from their IT departments that goes something like this: "We can't measure the value of IT. It's too intangible." If the line works, and IT leaders persuade business executives that IT investments are somehow fundamentally different from other types of business investment, IT is relieved of the responsibility of attaching dollar values to those investments.

Some high-profile CIOs have gotten extended mileage from this approach. Paul Strassmann (the CIO guru) often recounts the story of being put on the spot to justify the costs of IT at Xerox, where he was corporate director of worldwide computing. Since that uncomfortable experience, he has argued that evaluating an IT investment in the same manner that most other departments evaluate their efforts is nearly impossible. After all, he points out, accounting departments are never required to justify their budgets. IT is just as basic to the business as accounting, right? So the IT budget likewise should be exempt from the messiness of justification.

As one might imagine, this position is popular among some IT executives because it gets them off the hook. There is only one problem. Many business executives just aren't buying it. And they shouldn't. Unlike the accounting department, the IT department typically has a large and rapidly growing budget. In addition, most IT departments have had at least a few high-profile failures, causing business executives to be somewhat suspicious of IT's value.

Other IT executives believe that only pseudo-measurements are possible. A method misleadingly dubbed Information Economics (I defy anyone to find any real economics in this method) was developed by Marilyn Parker and Robert Benson.

It is basically a subjective scoring procedure that computes a composite score based on a completely arbitrary formula. It is, in effect, a non quantitative method masquerading as a quantitative one by using numbers and a simple formula.

The first problem is that results have no meaning when compared to other investments: What would you rather have, a retooling of a plant with a 62 per cent ROI or an IT investment with a "score" of 95? The second problem is that, to date, there is no empirical evidence that this method improves decisions.

So how should CIOs deal with the problem of measuring IT value? They should start by considering the possibility that the value of IT actually is measurable. Of course, this flies in the face of much of the last two decades of IT dogma, but then it wouldn't be the first time dogma proved wrong.

In seminars that I give and conferences where I speak I always challenge the audience to come up with the most difficult, even impossible, IT measurement problems they can think of. Then we determine how to derive a measurement. So far no suggested intangible, no matter how difficult it originally seemed to measure, has ever lasted 20 minutes in this analysis.

The rather radical-sounding position I take is that all "immeasurability" is just an illusion caused by three basic types of misunderstanding about measurement problems:* The object of measurement that is, the thing being measured) is not understood* The concept or the meaning of measurement is not understood* The methods of measurement - proven techniques used by science - generally are not well understoodOnce IT executives are coached in each of these areas, they seem quite capable of solving their own measurement problems.

THE OBJECT: Understanding what you're measuringUnderstanding the object of measurement simply requires being specific. When we use terms like employee empowerment, customer relationship, or strategic alignment, do we really know what we mean? Often we don't. IT project managers or business sponsors label these things intangibles, relegate them to the intangibles column in their cost/benefit analysis and think no more of it. But I find that usually more specific, unambiguous measurables underlie these ambiguous concepts.

For example, suppose someone says that information availability is an intangible that would supposedly result from a better document management system. What is meant by information availability? Perhaps it means that people spend less time looking for information. That is a measurable quantity for which we can calculate an economic value. Perhaps it means that information is lost less often. That is also measurable; if this information affects some routine business decision (like the approval/disapproval of an insurance claim) then we can find the economic value by measuring the effect that not losing that information has on reducing costly errors. Perhaps it really means that losing information causes extra work to re-create data (like redoing engineering drawings because the originals could not be found). Whatever is meant by "information availability" can be boiled down to something more concrete and therefore measurable.

Some useful tools can help you identify the clarified tangibles that exist beneath your intangible. They are used in what I refer to as The Clarification Chain. The objective of this process is to analyse and define intangibles so that they can be replaced with more clearly understood tangibles. The Clarification Chain:* If something is better, then it is different in some relevant way.

* If it is different in some relevant way, then it is observable.

* If it is observable, then it can be counted.

* If it can be counted, then it is measurableIs it logical to say that more X (an intangible) is better than less, but it is in no way different or observable? Then in what way is X "better"? If you believe X is a good thing, then you must also believe that it is somehow different from not having X. And if it is different in a way that is relevant, then it must be observable. So ask what the observable consequence is. Once you have identified an observable consequence, thinking of a way to measure it is pretty easy.

Here is another useful exercise. Create a thought experiment in which you imagine you have cloned an entire organisation into twin organisations, A and B. The two are identical in every way except for one thing: Organisation A has more of intangible X than organisation B. Now, imagine that you are an objective observer standing outside these organisations looking in. What do you imagine you observe to be different between A and B? If X is such a desirable thing, then there has to be some difference. What is it? Are certain things getting done cheaper or faster? Are the customers of A likely to come back for more business than the customers of B? Is employee turnover lower? Are mistakes of some type less frequent? Just think it through - and be specific.

After thinking through the intangible, we often find that our only barrier to measurement was that we didn't understand what we wanted to measure. Once we have clarified the ambiguously defined intangible we can address some other problems with the perception of immeasurability.

THE CONCEPT: Understanding measurement

Measurement, believe it or not, is a widely misunderstood concept. It's often mistaken for a process that produces an exact number. If you are told that something cannot be measured because there is no way to put an exact number on it, then you know that the problem is a misunderstanding of the measurement concept. The way scientists see it, measurement is the reduction of uncertainty about a quantity through observation.

The key element here is reduction of uncertainty - which is not necessarily (in fact, almost never) the elimination of uncertainty. If you tell me more than I knew before, then you have reduced my uncertainty. If you have told me that orders would be filled 10 to 30 per cent faster with some proposed system, then you have reduced my uncertainty if I previously thought the range was 0 to 60 per cent.

Actually, if a process produces an exact number (such as an accounting formula), that is a good indication that it is not a measurement at all. It's just a calculation. Measurements are pragmatic observations and observation never eliminates uncertainty. This is why all realistic measurements in science, engineering, actuarial science, economics, and so on are expressed as "probability distributions." Simply put, a probability distribution is a range of possible outcomes and their possibilities. If you can reduce the range of possible values by a bit, you still have a measurement. In fact, that small change in uncertainty might be enough to sway the decision.

METHODS: Understanding ways to measure

After intangibles have been clarified by developing unambiguous definitions and measurement itself is better understood, many more things appear to be measurable. But a few perceived immeasurables could remain because CIOs may not be aware of the variety of methods used to measure.

How often do IT execs conduct simple random samples or controlled experiments in order to measure some quantity of interest? The answer is not nearly so often as in other fields - including other areas of business. If I may be so bold, perhaps the problem is that computer science is not really taught as an empirical method like other sciences. Methods of fact finding in this culture generally do not include basic tools of direct observation. In Joint Requirements Planning (JRP) workshops, analysts and users spend considerable time talking about the business, and the analysts generally take the users' word for quantities such as orders processed per day, error rates, etc.

Analysts ask the users how long an order takes to process instead of finding a way to observe it directly.

IT executives seem rarely to consider simple scientific observation as a solution, though basic elements of the scientific method are widely used in market research, actuarial science, product development (both technical and non-technical), operations research and even human resources.

Remember, no matter how difficult your measurement problem seems, the first step in measurement is to change an assumption you may have been making for a very long time. Instead of assuming that it is a fundamentally immeasurable intangible, assume that it is measurable. Then the only question becomes whether you are clever enough to figure out how.

Douglas Hubbard is director of applied information economics with DHS & Associates in the US.

He can be reached at dhubbard@

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about EmpiricalLogicalOperations ResearchStrassmannXerox

Show Comments