How do you know if a vendor will deliver on its promise? When checking references, you'd better ask the right questions, or you might end up with the wrong solution.
You say you've narrowed your list of vendors, and now you're ready to make some decisions? No one knows better than you: you've made a significant investment in time and money - for starters - and have a lot on the line. How can you better ensure the success of the project?
Checking vendor references is one critical step toward making a final decision. But remember: not every reference is reliable.
Salespeople wouldn't knowingly give you a reference that would say something negative about them. Beyond that, sometimes vendors give discounts to companies that agree to be a reference, which jeopardises the unbiased opinion of the vendor and solution you seek. Consequently, you should automatically be sceptical of any testimonial. If you look at the client lists from several competing software vendors, you'll probably see an overlap. That's because different divisions use different products. Ask the salesperson why the rest of the company hasn't bought the software you're considering.
Ask for references that closely match your situation. If you are going to have 500 users on your system, try to avoid talking to a reference with only 20. If the vendor doesn't have references that have requirements similar to yours, it isn't necessarily a bad sign. You just need to be aware of that to make an informed decision.
When you contact the reference, ask for the person who was responsible for choosing the system and the person who maintains it; they may be different people. You need to make these distinctions because the reference may be a manager who had little to do with the actual decision-making process. Also, be sure to talk to the users; they will be more open to discussing the system's problems.
If possible, make an onsite visit to the recommended company, especially if the system is critical to your business and you're making a substantial investment. It's best to see the software as it's being used. Even better, ask the user to give you a demonstration of the software. In addition to calling references, contact writers from your industry's trade publications and ask them about the vendor. Many have written product reviews and will gladly share the information with you.
Anyone who's been asked to recommend a product wants to help the prospective customer. Trust that most references won't want to see you make a mistake. They'll tell you what you need to know if you ask the right questions. Here is a list of questions to ask references before you make a purchasing decision:
Are you still using the system? And if not, why not? This should be your first question because the reference might not be using the system. You need to know why. Although a salesperson will usually make sure that the reference is using the system before recommending you contact the company, you should still ask this question. You never know.
What exactly are you using? Which modules? What version? Make sure that you're comparing apples to apples. Many large systems have several modules. It is unlikely that a reference is using all of them. Clients rarely use more than 50 per cent of a system's features and functionality. You want to talk to someone who is using the features and modules that you want to use. If the reference is using a system that is several versions old, the new system may have bugs in it. If they've upgraded, ask if the transition was easy.
How are you using the system? This might be useful for you to know - it could be applicable to your business. Or, the reference could be using the system in a way that is not applicable to your situation; you might need another reference to replace this one. Find out what operating system and hardware are being used. The environment can affect performance.
How long has the system been up and running? You want to know if the reference is a long-time user or a company that just implemented the system and hasn't experienced any problems.
How many simultaneous users do you have? The number of simultaneous users can affect system performance. How is the performance when all of the users are on the system?
Why did you choose the system? What are its strengths? What are its limitations? Basically, you want to know why the company made the decision to go with the vendor. You might be surprised by what the reference says. One told me that he chose the system because he had no choice - a simple ultimatum from his boss! You could also discover things that you didn't know the system could do. What one company considers strengths - or weaknesses - could be irrelevant to your situation.
What are some of the bugs you've seen? If the company has used the system for a while, it has probably seen everything. Find out how the vendor has addressed such issues.
How long did the implementation take? Implementations can be nightmares. You need to know what to expect. Ask how you could make the implementation smoother. If you have a hard start date, you need to know if you'll be able to make it. Also, based on the date your system will go "live", a software salesperson may know who will implement your system. If so, ask to talk to references that used the same implementation team to find out if there were any glitches.
What additional modules are you planning to use in the future? This question could reveal to what extent the company is putting its future in the system. You could find out that it's going to scrap the system and get a new one next year.
What about the vendor's customer service? Ask the reference if it is currently under a support contract. Ask about the responsiveness of the vendor's help desk or if it's ever had any major problems and how they were handled. You are not just buying a solution, you are betting your success on a vendor. You need to know that the company's customer service is acceptable, especially if the system is critical to the success of your department or business.
How was the training? Was it helpful? Is it worth getting in-house customised training if it is offered?
Do you attend the user-group meetings? Many vendors have user groups. Find out if the reference attends the user-group meetings and what the company's experience has been. You should also try to attend the next user-group meeting to hear from companies that are not references. You will get a chance to hear the good, the bad and the ugly.
Lastly, don't just call the references supplied by the salesperson. Randomly call companies on the vendor's client list and explain that you are considering buying the system. Tell them that you want to talk to someone who can give you an honest answer. You'll find most people are more than willing to help you avoid being the victim of vapourware.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.