Carol Schillat's pilot project was a flop.
That's what made it a success.
Schillat's Internet-based ordering and forecasting system for Heineken USA was supposed to cut into the eight weeks it took to get beer to the company's distributors. After testing the new system with Heineken USA's Florida distributors for six months, the time it took to get the beer to the distributors was . . . essentially the same.
So why was Schillat pleased?
Because the pilot project, called Hops (Heineken online purchasing system), had a much larger impact on Heineken USA and its parent, the Amsterdam-based beer giant, than she ever could have imagined. When Heineken USA's ordering, forecasting and logistics processes were laid bare, it became clear that the reason for the lag in delivery time was not to be found within Heineken USA but within Heineken Netherlands, which produces and bottles the beer, and Heineken Export, which delivers it to all the company's divisions outside Holland. No matter how streamlined the American process became, it still took the same amount of time to get the brew out of Holland.
"We're now working on a supply chain project with Heineken Netherlands, Heineken Export and Heineken USA to create an integrated system," says Schillat, director of information technology for the New York, company. "I don't think we would have been able to do that if we didn't have the experience we gained with Hops." Today Hops has become a trusted portal for Heineken's 450 US distributors.
More important, though, was the strategic lesson that came from a relatively small, cheap experiment within a subsidiary. And lessons like these can be had by any company that begins to view pilot projects as a window on the future rather than as a tightly controlled test run of new software.
Think big. Once upon a time, launching a pilot project meant that a few hand-picked employees would peck at keyboards in a conference room for a couple of days using bogus customer profiles. If the software worked in the conference room, it was considered a successful pilot.
Today people like Schillat have learned that the humble pilot can be something much more ambitious -- and risky. But it is the sort of risk that companies must take to stay competitive. Enterprise resource planning (ERP) software provides the best evidence of the dangers of narrow piloting. In the early '90s companies jumped at ERP's promise of automating core business processes before gauging the impact these huge, complex software packages would have on people's jobs and the existing computing infrastructure. Stories abound of companies that could not make the software work and aborted two- and three-year efforts costing tens of millions of dollars.
These hard lessons have made pilot projects indispensable disasto-meters for any large-scale software or business process change.
What follows are 10 insights from those who have captained risky pilots and lived to tell about it.
Understand the old before ringing in the new. Jumping into the new work without fully understanding the old processes is taboo. The new processes just might be addressing problems that never really existed.
At American Express Financial Advisor's Client Services Organisation in Minneapolis, a pilot team was assembled in the brokerage group in 1998 to create a workflow process to enter, track and manage new account applications.
The old process was messy, unproductive and time consuming. Paper applications came into the group's mailroom and were distributed by hand, often sitting for days unnoticed in people's inboxes. When account representatives called in to check the status of a new application, no one had an answer.
The pilot team immediately jumped to an electronic solution. It assumed that the whole process would become smoother if the applications were scanned into an electronic document management system as soon as they arrived in the mailroom and were distributed electronically. Fortunately, the technical team, led by Deborah Cashin, president of Minnesota-based consultancy Digital Agility, conducted interviews with everyone involved in the application process and observed them in action to learn how they worked.
Cashin discovered that no one needed to see what was in the applications at this early stage; they simply wanted to know where the applications were. And new applications make for fat digital files, which would have tied up the network. But by logging information about the application into the system rather than the application itself, the reps could find out whether a new application had arrived and when it would be acted on.
Scanning was pushed to a later point in the process, when the group actually had to act on the information. In this way, good piloting -- learning about the business process before jumping in -- enabled Cashin and her group to avoid a catastrophe.
Sign a project agreement and let everyone know about it. Cashin's team prepared a project agreement, which clearly stated the pilot's scope and goals.
"Everyone has a concept of how the workflow should look and how it will change their lives," says Cashin. But if the pilot's more limited goals aren't documented, people may feel let down when their expectations are not met.
Cashin says there are still people waiting for their workflow to be automated even though the agreement said that they were out of the project's scope.
Looking back at the original agreement, Cashin sees how it was flawed. "The project agreement went out only to certain management people, and the people in the line didn't really understand what was in there," she recalls. "The communication of the agreement needs a very broad audience through staff meetings and team leader meetings. We were going to prepare a presentation and go within each of the groups that would be affected by the pilot and tell them this is what you can expect to get and this is what you won't get. We didn't take the time to do that, and I regret it." Choose your pilot group from the best and brightest. A small, fringe business unit that is not involved in the most important day-to-day work of the company won't be regarded as a good test bed for software and processes intended to be used by everyone in the company. "Demonstrate that you're involving enough different people so that the effects aren't contained within one small niche," says Carl Frappaolo, executive vice president of the Boston-based Delphi Group.
The business unit in which the pilot is being conducted should be high profile, have a reputation for relevance and importance to the rest of the company and have people who are respected by their peers.
At Amex, Cashin chose the brokerage group because it was a fast-growing unit that had been underserved by the IS department in the past. Also important, the group size and the number of transactions it handled were both right for a pilot. The business -- brokerage services -- changed rapidly and the group itself had an appetite for change.
If no particular group jumps out as a pilot candidate, some companies try to cherry-pick from all the groups that will be affected. At the Santa Clara, California office of Siemens Information and Communications Networks' Enterprise Networks Division (END), the pilot implementation team went to the area vice presidents of all the division's sales organisations and asked them to choose people who were leaders in their group to take part in a pilot of a new sales-force automation software package. The project goal, to increase sales productivity by automating the handoff from the marketing department's call centres, which generated leads, to the salespeople in the field (a process that basically consisted of scribbled notes and voice mail) was not welcome.
Sales representatives from END's national accounts division had boycotted the last attempt at automation and were still feeling pretty good about their Daytimers and pens. That's why the call went out -- to get buy-in from the area VPs, who would coax their people into using the software. "We also did our own choosing," says Susan Dale, manager of sales-force automation, "and we didn't leave out people who showed resistance." What the people chosen to participate in the pilot had in common was that they were all good salespeople. "We looked for people who, if we were hiring more people, we would hire them," says Dale.
Start small, be careful and don't rush. Before Cashin and her team began the workflow pilot that would eventually involve 300 participants, she put the new software on the machines of 12 people and had them work for two weeks before gradually adding more people. The slow start helped verify that there were no major problems lurking after all the planning and discussing and software development. "We needed that final check-in from business people saying, 'Yes, we see the new functionality, and yes, it is doing what it was designed to do,'" says Cashin.
This gradual beginning also helped Cashin control the waves of problems and suggestions that she knew would come rolling in once people actually began using the system to do their jobs. Similarly, Siemens capped participation in its pilot at 45 out of a total sales force of 450. The percentage here is less important than the number. Pilot project managers find that the sweet spot for participants usually peaks somewhere between 40 and 60. More than that and evaluating the project can get too complex.
By getting some major problems out of the way with the small group, Cashin could focus the rest of the pilot participants on suggesting enhancements rather than identifying bugs. A companywide obsession with bugs could easily have done serious damage to the project.
Remember that you're not working in a vacuum. You can only do so much to make a process work better internally if the biggest bottleneck lies outside. Even as Cashin and her team worked to eliminate the paper trail in the new account application process, she found that the rules of the National Association of Securities Dealers (NASD) began to get in the way. The NASD requires a high-level securities executive to sign off on new accounts, which meant that Cashin had to keep the original paper application flowing through the process pipe simply to get the required signature.
Just as Schillat took her case to Holland to try to resolve Heineken's home office bottleneck, Cashin and her team went to the NASD with a proposal for electronic signatures. By designing a second tier of security and passwords so that only the appropriate executives could access the applications and create an electronic signature, Amex satisfied the NASD's concerns and ended the paper chase.
Test everything, including the humble PCs. Most IS departments understand the need to simulate the expected loads that the new software will put on the network (several software-based testing tools are available for this) as well as test for any network software incompatibilities. But many IS groups fail to make sure that the people using the new system will be able to get it to work on their own computers. At Amex, Cashin's team laboratory-tested the variety of PCs and laptops the participants would use during the pilot to make sure the new software was compatible with what was already loaded on the machines.
Feedback is good; timely feedback is better. One of the great advantages of pilot testing new processes and applications is the ability to modify the system while the issues are still fresh in everyone's mind. While piloting the new sales-force automation system at Siemens, Dale's team provided a place on the screen to click so that participants could offer suggestions and opinions.
Sometimes the comments would be: "I don't remember what you told me about how to do this part," or "This part is confusing; I don't understand it." "We realised that if they weren't remembering something from month to month it was too complex," Dale says, adding that this kind of learning comes only when piloters depend on the system to get their jobs done.
Stock your pilot project with parachutes -- just in case. When Heineken USA's customers in Florida began using Hops, Schillat had two things working against her. One was their comfort level with the old way of ordering -- sitting down with the local Heineken representative to figure out how many types of beers to order and which ones were on special promotion, then massaging the order so that it would fit neatly into one of Heineken's shipping containers coming from Holland. Weaning customers from that time-consuming but cosy approach wasn't easy, and the glitches in the new system didn't help.
"In the beginning, it was very rocky," says Schillat. About six months after we started we had to do an entire server upgrade because the system was moving too slowly." In the meantime, Heineken didn't abandon the old process. And although Heineken didn't encourage its distributors to use the fax machine, simply knowing that they could fax in the order as a backup made them comfortable enough to give the new system a try.
"At first, everyone looked for workarounds," recalls software developer and consultant Frank Fuerst, president of B2B Technologies LLC in Atlanta, who helped Heineken develop the system. "The first month just about everyone faxed in their orders after they made them online," says Fuerst. "The second month, most of them did it exclusively online." Make your test group feel special. During the Hops project, no request from the distributors was too large or too small. "We would go down to Florida with a team of two or three people and do whatever it took to get a distributor hooked up to the Internet," says Schillat. "We'd even do things completely unrelated to the project, like troubleshoot their PCs.
Schillat also pumped up the distributors' enthusiasm by inviting them to headquarters, all expenses paid, to chat with the big shots and view presentations. As Hops took hold among the distributors, Schillat began cutting back the perks. "But we never stopped telling them, and we were careful to demonstrate that we were grateful for their participation," she adds.
Never leave your test group alone. Like Siemens, PeopleSoft -- a maker of ERP software -- struggled to get employees to accept a new sales-force automation tool which, like Amex's, was designed to improve handoffs between marketing and sales as well as to increase sales-cycle efficiency and decrease reporting. To facilitate communication with the pilot participants, the IS project team created a "buddy system." Each team member was assigned four or five salespeople whom they had to call at least once a week, updating them on the overall progress of the project and checking to see what specific problems they had with the new system. The project team tried to keep these chats upbeat. "If we wanted them to be enthusiastic about the project, we had to show enthusiasm ourselves," says Jim Alexander, PeopleSoft's director of information systems.
Every Monday at the team's staff meeting, each member would report on what the buddies had to say. The team member whose buddies had the highest usage at the end of the pilot phase won a free weekend getaway.
The buddies also took the group's focus off IT and helped Alexander and his colleagues get a much clearer idea of how the business worked.
"This was an opportunity to talk to people in the business about the business as opposed to the system," says Alexander. "You can say a system does A, B and C, but until you know what the business does with the system, you haven't gotten a full understanding of it." Looking back on her experience with Hops, Schillat says a little naivety goes a long way. By pushing the goals of the pilot beyond that of simply testing new forecasting software, she shook Heineken's supply chain to its core and moved the big brewery toward a new strategic direction.
As the stakes continue to rise in software implementations, companies need to give pilot projects the same kind of breathing room that Heineken gave to Schillat's effort. Pilots aren't just for testing anymore; they're for learning -- about your company, about your company's future. When a pilot flies high, the view can be breathtaking.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.