IT projects are usually judged successes or failures when they go live. But look back and you will see the real judgment comes later - and that requires a new set of value criteria

There's a tradition in IT project management that says if a project comes in on time, on budget and with the required features and functions, it is considered successful. Meeting that standard is difficult enough. A continuing study of thousands of IT projects by The Standish Group consultancy found that in 2004, just 29 percent of projects were successful.

But this definition of success needs revising. In performing 72 IT project retrospectives at 57 organizations since 1999, graduate students in the Master of Science in the Management of Information Technology program at the University of Virginia's McIntire School of Commerce have discovered that projects that were found to meet all of the traditional criteria for success - time, budget and specifications - may still be failures in the end because they fail to appeal to the intended users or because they ultimately fail to add much value to the business.

When Success Is Failure

We call these "failed successes". For example, a real estate management company successfully completed a two-year project to develop a Lotus Notes-based CRM application that was designed to provide strategic advantage in leasing vacant space. The project met all the project specifications but didn't successfully integrate with the company's business processes. So in the end, no one used it. The company's CTO put it best: "The application is 100 percent effective at 100 percent participation and zero percent effective at 95 percent participation."

Similarly, projects considered failures according to traditional IT metrics may wind up being successes because despite cost, time or specification problems, the system is loved by its target audience or provides unexpected value. For example, at a financial services company, a new system to enable rapid development, testing, deployment and measurement of collections strategies and to improve collections performance was six months late and cost more than twice the original estimate (final cost was $US5.7 million). But the project ultimately created a more adaptive organization (after 13 months) and was judged to be a great success - the company had a $US33 million reduction in write-off accounts, and the reduced time-to-value and increased capacity resulted in a 50 percent increase in the number of concurrent collection strategy tests in production.

New Criteria for Success

Indeed, the true impact of a project may not be felt until long after installation is complete. The study data showed that beyond the traditional metrics of time, cost and product (an acceptable system of reasonable quality that meets specifications), the following additional criteria need to be applied to projects to determine long-term success.

• Use: The project's resultant products or services were used by its intended constituents.

• Value: The project led directly to the organization's improved efficiency or effectiveness (common metrics included NPV, IRR, EVA and the Balanced Scorecard).

• Learning: The project increased stakeholder knowledge and better prepared the organization for future challenges.

Interestingly, the retrospective data showed that across all stakeholders (project managers, their teams, project sponsors, top management and users), the top three criteria by which stakeholders judged a project's success were product, use and value, respectively. Meanwhile, cost was ranked lowest overall. None of the groups ranked learning (preparing for the future) in their top three criteria, although all of them suggested that learning was of at least moderate importance.