Time It Right

When to conduct an audit, just as who should lead it, is a matter of debate. When to start depends on the type of system or application deployed, the amount of time it will take before the application begins generating some results or data, and the amount of time it takes staffers to get acclimated to the new system and new processes. Generally the audit should take place well within a year of implementation.

"When we started the audit a month after the implementation, we didn't have enough data to see if the system was successful or not," says Michael Baker's Higgins. "It's really important to make sure the system is up and running long enough to have enough data in the system that you can analyse."

Ken Cunneen, IT leader for technology and integrated supply chain systems in Honeywell Aerospace's Aviation Aftermarket Services division, says if a company implements a financial system that only gives results once a quarter, the audit team should wait at least three to six months until the system has generated enough data before performing a PIA.

On the other hand, the sooner you start the audit, the fresher the data and the easier it is to cull lessons learned. Which is why PIAs should not be a onetime event, says PwC's Christian.

Sun Life Financial's project office begins PIAs within a month of project completion to do a post-mortem on the implementation process and to get feedback from users to make enhancements to the system. "When we have benefits that are realised over a period of years after the project is implemented, we do follow-up reviews," says Sun Life Financial's Esposito.

Higgins plans to take that iterative approach with Michael Baker's ERP implementation, which is scheduled to be completed by the end of this year. "We will begin to audit the effectiveness of the new system within a couple of months after we finish implementing it," says Higgins. "Then we'll probably start to look at the ROI six to eight months after we go live so that we have enough data to perform a good analysis."

Maintain Meticulous Documentation

Another key to successful PIAs is good documentation, which takes many forms. It includes the business case that outlines the system's expected cost, benefits and ROI; the project's time line, including key metrics and milestones; a breakdown of the system's technical requirements; a record of the security and financial controls that have been put inside the system; and a compendium of all the changes that have been made to the system or project plan. But getting good documentation is one of the biggest challenges CIOs and IT departments face in conducting PIAs, particularly if they're doing one for the first time.

Sharon Thompson, director of IT audit for the AARP (American Association of Retired People), says that if an auditor discovers that the implementation team does not have a thorough business case, a detailed project time line or meticulous records of changes and security controls, the auditor should note that discovery in the audit report and include a recommendation for the implementation team to keep better records.

To make life easier for himself and his post-implementation auditors, Higgins uses a Web-based collaboration tool called MPOWR - developed by members of his IT department to do version control and to archive changes made to the project's scope, templates, time line and budget. When it comes time to do a PIA, Higgins sends the auditors a link to the MPOWR site with a user name and password.

Sun Life Financial's IT project office also developed a project portfolio management system in-house that the project manager uses to report on a monthly basis changes to a project's cost, milestones and schedule.