CIO

The Meaning of Success

Part 3 of a Three-Part Examination of Project Management Missing Links

As companies become wiser about recognizing and adopting successful project management approaches, they face the challenge of creating an environment that fosters success — but that means first defining what success means to the organization

If the project manager says a project will take a year, and it takes two instead, should we see that as a project management failure? If a project was supposed to cost $2 million, and ends up costing $4 million, is the project a dead duck?

When we talk about a project failing when it cost more than somebody said it would cost, or takes twice as long, how can we be sure those estimates were anywhere near the right ball park in the first place?

These might sound as ethereal as other philosophical conundrums like whether a tree falling in a deserted forest still makes a sound, or the number of angels that can dance on the head of a pin.

However, Cutter Consortium Fellow and project management guru Tom DeMarco believes it is time many organizations rethought their definitions of project success to acknowledge that plenty of so-called "failed" projects have led to development of successful products. And anyway, he asks, when we talk about a project failing when it cost more than somebody said it would cost, or takes twice as long, how can we be sure those estimates were anywhere near the right ball park in the first place?

"If I say this project will be done in a year and you run the project and it takes two years, is that a failed project?" he asks. "I think you could argue very reasonably that it was just as likely to have been a failed estimate: I set out to tell when I wanted it done, not set a goal for you. In addition, the projects that fail in that respect are the ones that have the relatively large number of unknowns in them, and they are typically projects that are trying to do something which is worth doing.

"So you have this quandary: projects that are failures in that sense very often deliver something which is very useful." DeMarco says in assessing the relative success of a project, many organizations omit this most important part of the equation: consideration of whether the resultant product is a success or failure.

If a project fails on two criteria — it cost a lot more than you thought, and also ends up being useless — few would argue the project was a failure and it should be labelled as such. But DeMarco points to a middle ground, where a project raises the ire of management and the frustration levels of staff by taking longer than expected, or where cost overruns are rife, but where the product ultimately delivered is useful and highly esteemed by the people that receive it. Are we going to call that project a success or a failure? DeMarco has no doubt that we should call it a success.

What is missing from many assessments, he says, is a reluctance to acknowledge the unpredictability of software development, and the fact that projects intended to produce something worthwhile are inherently hard to predict. "We hear these studies all the time about how many projects are failures," he says. "I've never seen a study about how many products are failures, have you?"

It is normal for project managers to focus on generating data and delivering projects on time and on cost, Valense managing partner Michel Thiry says. But since IT is a fairly turbulent environment, dynamic and fast moving, it is just as normal for the goalposts to change. Obviously if we focus on time and on cost, we will be seen to miss the target, but that may well be to miss the point. Better to focus on benefits to the organization, Thiry says.

"If instead of focusing on time, costs and the technical aspects of the project, they could focus on the benefits that the project will provide to the organization, it would then be much easier to say there might be changes in cost, there might be delays, but as long as we achieve our expected benefits we will call that a success. Take the example of the Sydney Opera House, which 40 years ago was absolutely over cost but today is a major landmark of Sydney. As a project if you consider it only in terms of time and cost, it was a failure, but as a benefit to Sydney it was a success, although it took more time than planned to get there," Thiry says.

Dimension Data Group executive: services Scott Petty thinks part of the difficulty lies with the way many organizations measure their projects. Typically when a team wants to do an IT project they write a business case and define desired business outcomes. But in many organizations, once the project is under way those outcomes, while not forgotten, get little consideration. Most project managers are "goaled" on whether they get the project done on time and on budget, he says. As any good project manager will tell you, the focus is on keeping the scope well-defined, looking for variations and pushing back on any changes around those areas that would help keep the project running on time and on budget.

"But because they lose that link to the business outcome, often the end result that's delivered doesn't actually achieve the business goals that were originally intended in the business case when they started that project," Petty says. "So I tend to think that there's a missing link created in that area because those business outcomes aren't driven into the fabric of the way the project is run, so pretty much everyone working on the project focuses on the way the project is going to be measured, which is on time, on budget."

Page Break

Why Bother?

If project management failure rates are as high as we are led to believe, why try to do projects at all? asks Margo Visitacion, vice president in Forrester's Application Development Infrastructure research group. In a May 2005 paper called 'What Successful Organizations Know About Project Management', Visitacion answers herself by saying that in fact, Forrester, too, believes the state of project management is far better than the headlines suggest. Like DeMarco, Visitacion does not accept it is possible to pigeonhole any project's outcome as simply successful, challenged or failed; outcomes are too specific, she says, and inseparable from the situational reality of the organizations that implement them.

"As business becomes increasingly digitized, the old way of doing projects — business making a request and walking away to wait for delivery — doesn't work any more. To be successful, a company must toss aside preconceived notions of what makes up project delivery as well as the criteria for measuring successful projects," Visitacion writes.

"Companies are making gains in project delivery and are doing so by reconsidering the elements of project management; not by throwing out traditional practices, but by adapting processes that better fit changing business models. Accomplishing such change requires a major culture shift; organizations are rethinking how the business and IT must share responsibility and accountability for project delivery and what practices must be put in place to achieve greater chances of success," she writes.

To help draw such conclusions, Forrester interviewed 32 organizations at various levels of maturity and experiencing different levels of success across North America, Europe and the Asia Pacific region about their project management practices.

Undoubtedly the goals for successful delivery should always be on time, on budget and meet expectations, Visitacion says, but situational realities are just as important. And it is those situational realities that she argues may allow the organization to declare success even if only one of those requirements is met. "Companies that are successful project organizations are flexible enough to first determine which of the requirements are a necessity on a project-by-project basis, then base their metrics on the agreed-upon measurement," she says.

What brands a project a success all comes down to a question of value, which will almost always trump timeliness. As Visitacion points out, delivering what the customer ordered is likely to matter far more than when it is delivered, unless failure to meet the time frame is going to incur a significant penalty. Organizations need to be much more flexible in thinking about projects, she suggests.

"Shifting schedules is unacceptable if caused by things such as poorly defined business cases, inaccurate estimates and poor project management. But external or organizational business changes, changing customer requirements or compliance changes can be acceptable if they will result in a more useful and effective project. Change management is the critical component that ensures successful delivery — consistent communication practices, management and execution keep all stakeholders informed and involved in every decision-making process.

"What does success mean? If a company were to dissect any recently completed project, it would find a multitude of measures to demonstrate success or failure and many shades of grey in between," she writes. "Identifying these elements can enable an organization to put practices in place that prevent a project from running off track or create the right environment to support consistent successful delivery."

Page Break

What's It Worth?

It is one thing branding a project that was supposed to cost $100,000 a failure if it ultimately costs $200,000, but what if the final product is worth $1 million, or even $10 million? Not only will this drastically alter any assessments of product success, but it should also influence how the organization runs the project, says John Thorp, vice president in the Strategic Consulting Practice at Fujitsu Consulting in Canada and author of The Information Paradox (revised 2003).

Thorp thinks too many organizations still suffer from "silver bullet thinking", with executives who still believe you can just plug in the technology and watch the benefits flow. In other words, as has been repeated with monotonous regularity, project management failures are a management, not a technical problem.

"If we are to understand, and more importantly fix this problem, it is important to step back and think about what success would look like," he says. "Several years ago, I asked a number of senior project managers of a very large US organization how they defined a successful IT project. The answers included on time, on budget, delivering the expected functionality and 'getting out with my skin intact'. 'Wrong,' I told them. 'A successful project is one that results in benefit to the organization.' But the simple fact is that none of them had ever been asked to think like this."

It is not that bringing projects in as specified, on time and on budget, is unimportant, Thorp says. It is that it is necessary but not sufficient. Organizations must not only manage the delivery process but also the benefits realization process. That means understanding the results that the business expects and the way the project will contribute to these results as well as other changes the business will have to make in order to achieve the results.

In a world of much higher project development complexity, he says, traditional project management will not do. Organizations must adopt a new approach focused on realizing the benefits of investments in IT-enabled change.

"The adoption of such a benefits realization approach has become a business imperative, and requires fundamental shifts in management mind-sets," he wrote in The Information Paradox. "Organizations must shift beyond managing technology projects in isolation to managing business programs. Programs clearly focused on business benefits, including organizational, process and people change projects in addition to technology projects. They must elevate decision making from the technology level to the business level, and from the business 'silo' view to the enterprise view. Multiple business programs need to be evaluated, the best ones chosen and then managed as a portfolio of programs. Finally, they must recognize that decision making is not a one-time event. Programs must be proactively managed through to the full delivery of value."

Page Break

Upside Down View

Yet despite growing awareness of the importance of such an approach we can see how little this thinking has meaningfully penetrated most organizations right here at home. For the past two years Australian consultancy Capability Management has been probing the attitudes of 27 of Australia's top companies and their success with project benefits management.

"Companies have an upside down view of benefits," Capability Management executive chairman Jed Simms announced when introducing the findings. "Rather than seeing benefits realization as the core focus of their projects, they tend to see it as either an assumed by-product or as an optional, hoped-for extra. However, they still accept that realization of benefits is why they authorize projects in the first place."

The research identified four root causes of poor benefits realization:

1. Projects are not set up to deliver the available benefits

2. The business uses dollar realization as a proxy for benefits realization, and then hopes they will be delivered

3. There is no end-to-end process for benefits tracking and measurement

4. The common, conventional approaches used to deliver projects and their benefits inadvertently destroy more value than they deliver.

"We were surprised that none of the 59 projects we reviewed in depth were found to be set up to deliver the expected benefits," notes Alex Chapman, CEO of Capability Management. "All sponsors and project managers interviewed really believed that their project would deliver the promised projects, but when analyzed, the connection between the business case and the project was completely non-existent."

"But the situation is worse than just not delivering the expected benefits, as the business cases were found to miss many benefits," Simms adds. "Just redoing the business cases on several projects doubled the expected benefits — and all of this value is currently being missed. Add up the missed, lost and destroyed project value and you get to over 50 percent of a project's potential value lost. This represents billions of dollars a year."

"One common reaction we got was that it was all too hard to track benefits with so many projects going on," Chapman says, "So we used a simple and proven benefits management process to show it could be done. You can identify, quantify, track and measure benefits both during and after the project's completion. This is not rocket science, it just takes the right approach."

Page Break

Sheeted Home

In many organizations, says Thiry, the need to reshape thinking on projects sheets responsibility straight home to the CIO, who must focus the attention of CEOs or the business on benefits rather than time or cost considerations.

First off, that means meeting with key stakeholders at the project's inception and finding out what they will consider a success. Then, as the project proceeds, it means breaking the project down to enable early and phased delivery of benefits, concentrating first on "low hanging fruit", then marketing those early benefits to stakeholders.

"Take the case of the Opera House," Thiry says. "Obviously if you had said: 'Your solution will be delivered in 25 years', people would have said: '25 years is too late — I want something now'. But if you pace the benefits that you are delivering in a way that you can identify the quick wins, the low hanging fruit, and how you can deliver on those first, the stakeholders can see that they will get some benefits although not all the expected benefits early. Then even if it takes a bit longer in terms of time, and in terms of cost, they will accept that because they already see some results."

However, since clients or customers will inevitably change their mind, or have their expectations raised by the early delivery of benefits, it also means creating different levels of success criteria or expected benefits. High-level benefits will be more abstract, and not expected to change often. More precisely defined benefits will, on the other hand, be subject to greater change, Thiry says.

"One of the things in managing expectations is you have to make stakeholders understand the high-level expectations will be achieved in the long term. Now the low-level expectations might change, and therefore you can only promise to fulfil them if they can be achieved within finite time limits," he says.

And speaking of time limits brings us nicely to another concern raised by Fujitsu Consulting's Thorp: the dangers in allowing an obsession with time pressures to undermine your project. People cannot think any faster under deadline pressure, he says, and while pressure sends a message that the project is important because someone cares about the result, the reality can be very different. In fact, he says, all too many projects face pressure specifically because the product is not highly valued.

"Say you're a political animal in a company and you're looking around for a new project because you're a project manager and you see something that is on the backburner that nobody has been willing to take on because frankly, it isn't very essential," he says. "Well it isn't essential if you build it over the course of two years with 25 people, because its value is lower than the cost of two years of those 25 people. But if you — because you're looking for something to do to make your mark on — say: 'I'll build it in one year with six people', the business might think it a winner. So you start this project off under tremendous pressure to get a lot done in a short period of time with not enough people, and the reason you're under pressure is not because the thing is so valuable but because it's useless.

"Now you think that situation is so bizarre as not to be real world, but I would put half of all projects in that category. Years ago I would have thought if people were under pressure it was because the product is really valuable and people really care about it but my experience has been the opposite," Thorp says.

The answer, he says, is to use value metrics to evaluate projects to ensure the project owner is accountable.

"If some sales manager says: 'If I had this, I could double my sales', then you are going to stand back and say: 'Okay, double your sales'. If instead of that he says: 'This is essential infrastructure, I will be grievously inconvenienced if I don't have it', that is a value statement that has no accountability whatsoever attached to it. So we end up with all kinds of projects being justified as essential infrastructure as opposed to being justified by 'This will increase our profit margin by 62 percent or add 110 percent to our sales'."

Thorp says in some organizations project sponsors or team leaders are obliged to make a value statement and their budgets are adjusted automatically to reflect the assumed savings. Such an approach is not hard to adopt, and delivers real results, he says. It is just that most organizations have not yet become "super disciplined" because in the early days of software development the values were inevitably much greater than the costs, and any project anyone dreamed up was so obviously justified that it was hardly worth doing the maths. Now that organizations have built all the systems that were so obviously valuable that it was not worth doing the maths, they lack the discipline needed to reflect the new reality.

Page Break

Lifting the Game

There is always plenty of talk about improving the software process, but Thorp thinks it is the project chartering process that most needs re-engineering. There are also other areas where project teams must lift their game. We all understand we do not know how to predict how long a piece of software will take to develop, he says; any company will tell you so. What almost none of them do then is to take the obvious next step of trying to figure out how to get better at it.

"Some companies have really solved the problem: they build metric databases, they measure each piece using some level of function metric before they build it, they predict from the estimates of the function metrics and extrapolate from the database to say how long it will take to build, how much effort and so on. That is something that the best companies do, and interestingly, the companies that are called systems integrators, because they have to deal directly with the market are very good at that — the Lockheed Martins, the Computer Sciences. The systems integrators in general have solved that problem the way the whole world would have, I thought by this time. Other companies just put a wet finger in the wind but do not really do their homework. They are not able to predict how long software will take, because they haven't built the metric databases, they haven't built the skills," Thorp says.

"Now that's the surface analysis of the problem. The deeper analysis of the problem is they don't want to do that because they believe that it isn't really germane. What they do is they pick some date that clearly is not doable, and announce that to be the schedule, and they think that is good enough because that guarantees the project will be under a lot of pressure.

"But they're not dealing with the damage you can do to your project by starting it off on an unrealistic schedule. And that is so damaging. If you set out to build a project in one year and it takes two, then I want to hold your nose to the awful fact that if you'd started out on a more realistic schedule, you might have built it in a lot less than two. If you had scheduled 16 months you might well have finished it in 16 months. And you set people up as losers. So there is an area where most companies aren't on their game."

Going back to benefits realization, Capability Management's Simms puts it this way: "A benefits realization approach is about making the right decisions about the right things, and then managing the decisions you make. A client of mine once said: 'A lot of consultants say the same thing — start with the end in mind'. A benefits realization approach does not just start with the end in mind, it manages value continually with the end in mind. It's a process that goes right through to the full realization of value, that is 'from concept to cash'."

Adopting a benefits realization approach involves a long-term, sustained change effort in how organizations think, manage and act. New processes and organizational structures will be needed to operationalize the new mind-set. Major changes will be required in the areas of accountability, measurement and the process of change itself. Project managers can and must play a leadership role in driving and implementing this change.

Until most organizations get there, we might as well debate whether falling trees make a sound if no one is around to hear them as whether or not any particular project succeededor failed.

Page Break

SIDEBAR: Home and Hosed?

Why post-project reviews are crucial

By Bart Perkins

Nearly every major systems implementation methodology includes some type of post-project review (PPR). A PPR provides an opportunity to review a completed project's strengths and weaknesses as well as propose improvements for future projects.

A PPR should be undertaken for every large project, especially unsuccessful ones. Outsourced projects particularly benefit from the structured analysis and communication required by a review.

Changes in your IT organization, like changes in your life, require time for reflection. A PPR is an opportunity to learn from past mistakes and successes and to improve project development practices. Unfortunately, such reviews are frequently perfunctory or skipped entirely. Major companies perform comprehensive reviews on less than 20 percent of their large projects, according to Stuart Orr at Vision 2 Execution.

Objections to PPRs are myriad. Many IT organizations find it difficult to get project staffers to focus on PPRs, which are often viewed as time-consuming overhead. Because of resource shortages, key project members are frequently reassigned to other projects before a review is conducted. In addition, reviews of projects that were plagued by political problems or contentious disagreements can reopen old wounds.

Despite these objections, comprehensive project reviews are worth doing because they provide significant benefits. For example, they help IT organizations do the following:

Reassess business benefits. Most IT projects are part of a business program. While the benefits of the overall program may not be known for several years, the PPR provides an opportunity to assess whether the projected business benefits (from both the project and the overall program) will be realized. Use the review to reset management expectations if necessary.

Improve estimation accuracy. The PPR provides an opportunity to compare the original (and revised) estimates with the actual time and resources consumed. The objective is to make future estimates more accurate, not to criticize.

Most large organizations have progressed from "back of the envelope" estimates to more rigorous bottom-up estimates based on interfaces, screens, data fields and other tools. But even sophisticated estimating tools must be frequently recalibrated based on your organization's actual project experience. Use the PPR to do this.

Evaluate what went well. The PPR is an opportunity to evaluate the effectiveness of project management methodology, executive sponsor participation, risk mitigation and political support. Use it to determine how each can be better leveraged on future projects.

Identify areas for improvement. PPRs help minimize problems on future projects. Every project encounters some problems, such as unexpected functionality changes, missed deadlines, technology snafus and marketplace changes. Use the PPR to evaluate the project team's effectiveness at identifying, responding to and resolving problems.

Also assess the success of the adopted solutions. To improve future planning, review the original project documents to determine whether the problems actually encountered were initially identified as risks. Anticipated problems are much easier to address than unexpected ones. Finally, brainstorm about what could be improved on future projects to prevent similar problems.

Capture and encapsulate project experience. Project history tends to get rewritten or forgotten altogether if it is not captured in a reasonable amount of time. Written records ensure that the lessons learned — sometimes painfully — don't get lost. In addition, PPR records may be invaluable in future negotiations or during litigation related to contentious projects.

Get management support. A PPR is an opportunity for management to listen and learn from team members' experiences. Reviews often uncover ways to improve your organization for future projects. If management fails to act on changes proposed through the PPR process, however, staffers will become cynical.

Use the review to get management's commitment for the improvements and to hold management accountable for implementing the changes.

Acknowledge contributions. Use the PPR to publicly identify heroic efforts and thank all team members for their contributions to the project.

Although PPRs take time and effort, the insights gained can be invaluable to IT efforts to improve project delivery capability. A comprehensive PPR is an opportunity to leverage the lessons from past projects into improvements that will enhance the planning, delivery and success of future projects.

Most IT organizations recognize that comprehensive post-project reviews provide valuable information and develop suggestions for improvement. Properly undertaken, the PPR provides an opportunity for team members to reflect and report on their experiences in a way that helps the organization learn constructive lessons. Without feedback, few organizations are fully aware of their potential to improve. Capitalize on past experience; you have already paid for it!

Bart Perkins is managing partner at Leverage Partners Incorporated, a group which helps organizations invest well in IT

This project management series included: "Part 2" (December 06/January 07), which explored the importance of change management and business outcomes. "Part 1" (November 2006) looked at the human aspects of project management.