Like many CIOs, Steve Pickett is happy to serve as a reference for his favourite vendors. If he spends a couple of hours talking to a prospective customer, the vendor might not bill his company the next time he needs a service visit not covered in his contract. He doesn't benefit personally from the deal; the perks go to his company, which approves of the practice, so why not?

But when the time comes for Pickett to make a decision about buying his own software, he doesn't place much stake in what his colleagues have to say - and he doesn't always think that the CIOs who call him should either.

"Some of the companies that have called me have taken [the reference process] much too lightly," says Pickett, vice president and CIO for Penske, a Detroit-based transportation services company. "If they're calling me to find out whether a particular supplier is doing a good job, without knowing the complexities within my company, I'm not sure that they're learning anything. I may be giving them misleading information because their business is different from mine."

Pickett says he's up front about this. Nevertheless, he admits there is a natural tendency to avoid the negatives if things are going well. "If you have a good relationship with the supplier, you're not going to bad-mouth them," he says, though he's careful to point out that if he's receiving bad service from a vendor, he'll say so.

All of which is to say, customer references come with baggage - big, unwieldy baggage. Vendors' long-held practice of offering enticements to customers to speak to potential new customers is widespread and widely accepted by CIOs. Yet despite the fact that reference checks are possibly the most important step in the software selection process, CIOs generally know little about the references they call and even less about the terms of the arrangement between the reference and the vendor.

Companies that serve as references are likely to get things that average customers never get: preferential treatment, influence on the development cycle - even cash rebates. These customers aren't average in any way and might not reflect the reality of your situation unless you become a reference too. And even if the reference isn't getting special treatment, most software projects have become so big and complex that no two companies are likely to have the same experience anyway. All this means that customer references - especially in the realm of big enterprise software projects like ERP and CRM - have ceased to have much real meaning.

Worse, serving as a reference yourself can have tragic consequences for your company and your career. Accepting personal gifts or money for being a reference is unethical and can, of course, get you fired. But even if the gifts and special treatment benefit only your enterprise, not you, they can still violate your company's code of ethics or jeopardise your company's fortunes. For example in the US, a CIO saying good things about bad or problematic software can send up a red flag with Wall Street analysts, who now track software projects more closely and downgrade company stocks at the slightest whiff of trouble. The reference game is so much of an ethical quagmire that the US federal government has banned its IT executives from making endorsements of any kind .

The government may be on to something. The benefits that companies gain from acting as references for vendors may no longer justify the risks - both personal and companywide - that are part of the package. In fact, the only viable reason for agreeing to be a reference for a supplier may turn out to be the simplest one of all: because you think it's good.