Getting the Metrics Right

When designing a scorecard, start with the most basic question: "Who is it for?" Two demand-side possibilities are senior business management and business customers (business users and operational managers). Three supply-side possibilities are the IS management team, individual IS staff members and ESPs.

Should everyone see the same scorecard? Generally, stakeholders do not see the same scorecard because the focus and level of detail appropriate for one group are not appropriate for another. But the measures used in all scorecards need to be related. If they are not, the different scorecards risk driving behaviours in different directions, usually not the goal.

Meaningful measures present information on areas important to the stakeholder. Measuring the performance of individual IS components - such as WAN and server availability - may be important to the operations staff because they are on the supply side. But the measures don't address the demand side because they don't tell how well a business service, such as a public portal or an order-entry application, is working.

The best demand-side measures describe the availability of business services because these metrics relate to what the demand side receives, not what the supply side provides. They look through the lens of the recipient.

Choosing and defining measures and appropriate targets must be collaborative. The most time-consuming activity in developing a scorecard is creating data definitions that everyone understands and agrees with. Many of the Excel scorecards we saw allow users to click on a cell and see its definition and underlying formula.

In the majority of case histories, the CIOs noted that iterations were needed first with IS stakeholders, then with IS staff.

There is no such thing as a perfect scorecard. There will always be areas that can't be measured satisfactorily. The advice from our case-history enterprises is to start where you can. Choose "good enough" proxy measures, if need be, to garner early successes. Evolve from there.

To make value-based decisions, management needs metrics that link IS activities to business outcomes. Even better are predictive metrics. Most measurement systems report on the long past and recent past, but not on the future.

Forecasting is notoriously difficult; but for scorecards there is a better way: find a surrogate measure for inferring some outcome. For example, one firm has found that "customer participation in key business projects" is a lead indicator because, in its experience, projects with high levels of customer participation deliver better results than those with low levels of participation.

Similarly, the capability maturity model level of the applications group can be a present-day indicator of future improvements in developer productivity or quality.

Evolve Your Scorecard

Scorecards can drive desirable behaviour and continuous improvement. There are numerous continuous improvement techniques, such as total quality management (TQM). Perhaps one of the best known is the "Plan, Do, Check, Act" cycle, which lays out activities designed to drive continuous improvement and was popularized by quality guru Edwards Deming.

For the IS organization, this approach translates into using the scorecard to observe what is happening, orient itself based on its interpretation of the signals, decide what to do and take corrective action. Then start the loop over again.

When designing a scorecard, several rules of thumb apply. First, use fewer than seven result areas. A second rule of thumb is to produce a scorecard that resembles performance communications already in use, such as familiar reports and regular announcements. Try not to introduce a unique or new format.

Support your scorecard with appropriate technology. Most scorecards reviewed for our report were spreadsheets.

It's time to review how you currently demonstrate value, enhance alignment, focus IS attention on what's important and improve IS performance, so you don't risk losing your war for the want of a scorecard. Andrew Rowsell-Jones is vice president and research director for Gartner's CIO Executive Programs