Right-the-First-Time Metrics

One issue offshore vendors are notoriously stubborn about is performance metrics. If left to their own devices, many would just as soon stick with the metrics they brought to the table on day one, particularly if those benchmarks make them look good. But the metrics often offered up by offshore vendors - simple cost or man-hour figures, ratios of onsite to offsite staff, errors per thousand lines of code - may not be useful. Over time, it's the customer who must push for new, more meaningful metrics. "Trying to figure out what's the right metric to use is the area where we spent the most hours," says Vinod.

The majority of work Sierra Atlantic does for Vinod's manufacturing company is in the area of application support. Throughout the day, a series of tickets are opened as Vinod's users report problems with applications (anything from a password that needs changing to a program that malfunctions). Those tickets are passed to the offshore team. They look at the problem and make an attempt to resolve it. The metrics Sierra Atlantic has used all along to measure its application-support effectiveness were things like how long an open ticket sat in a technician's queue or how many hours that technician worked on the problem. And according to those numbers, the vendor was doing a bang-up job.

But on the other side of the world, Vinod was seeing the backlog of new tickets inch up every day. "The Sierra Atlantic team thought they were doing a great job. They were publishing this report that showed they were squeaky clean," he says. "But their metrics didn't mean anything at all. None of these metrics helped drive the only goal - ticket closure with a satisfied user." And since there was an increasing number of tickets being entered into the system, Vinod suspected problems were not being resolved on the first or even second try.

The offshore support team had no way of knowing whether the solution they tried actually resolved the original problem (they worked during the day in India, while it was night back in the United States) and, with the performance metrics Sierra Atlantic had in place, the support staffers had no impetus to follow up and find out the net result. So Vinod brought the entire offshore support team (at considerable cost) to his headquarters in Ohio, where he thought they'd feel more connected to the company and more accountable to users. And sure enough, the backlog decreased. "We got the numbers back, and they were fantastic," he says. "Once in the US, they were held to the only metric that was important to us - two-day closure of every ticket." Of course, Vinod can't keep the entire support team in Ohio full time. He's still working with Sierra Atlantic to figure out the right mix of offshore and onshore vendor staff and new processes to make it work.

Drouin says his team has also had a tough time figuring out what numbers will paint a more meaningful picture. His offshore management team recently added a number of metrics to track resources, projects and network availability, which are delivered monthly to Satyam's offshore project managers and TRW's project champions. More importantly, they're working to finalize a next generation of metrics whose inspiration comes from the world of manufacturing. Drouin calls them right-the-first-time metrics, an IT corollary to the manufacturing metric "first-time yield".

"Rather than the number of bugs per line of code, we want to figure out how many times we get something that's just right out of the box from the vendor, or close enough to just right that we don't have to kick it back to them," Drouin explains.

The Truth About Turnover

Lately, Drouin has been focused on turnover metrics. Satyam itself tracks when an employee leaves the company. But for Drouin, it's when a Satyam employee leaves the TRW account that he feels the pain, even if that employee is still working for the vendor.

And like most CIOs who have been outsourcing offshore for more than three years, Drouin has been feeling that pain more than ever lately. He was aware of the well-publicized turnover rates in India, sometimes nearing 25 percent or 30 percent. "That's bad enough," he says. "But you can have a specific project team and experience 100 percent turnover overnight. That's a tremendous impact, and projects can grind to a halt."

Indeed, one of TRW's biggest offshore projects came to a dead stop twice last year because the entire project team on a product data management (PDM) system to support TRW's engineering work left overnight. "They literally walked across the street to join another vendor to work on some giant ERP project," Drouin says.

It was particularly costly because of the type of project. "If it's a SAP project, we know that Satyam has a whole host of SAP talent they can bring to bear," Drouin says. "But if it's something more specialized like PDM - they didn't have a wealth of resources in that area. And it took them time to go outside and find people." The defection led to lengthy delays in project completion that made TRW's VP of engineering none too happy. "It significantly slowed down a project aimed at increasing the efficiency of our engineers," says Drouin. "It also impacted our credibility with the business, who began to doubt our ability to deliver the project."