Vendors sell products. Whatever their sales hype and pitch, they make money selling products and collecting ongoing maintenance and licence fees. Only things that increase their sales, revenue and margin are of real interest to them.
So, don't expect your vendor to really be concerned as to whether you do or don't realize the benefits you were expecting, hoping for or promised by the vendor! Unless they are on a performance fee basis, that's your problem. While they usually cannot afford for you to completely fail (bad publicity), most projects fail to deliver to expectations so if yours is deemed unsuccessful, it is just another one.
So you cannot rely on your supplier to work earnestly for your success. So you need to make sure they don't undermine your success. Not in political or such terms, but product performance terms.
Projects are problematical enough without product deficiency problems. You should expect your vendor's products to perform to specification.
Now, you need to understand what 'to specification' means to the vendor. For example, for many years Microsoft Word was designed for documents up to 100 pages in length. Beyond 100 pages the software often went berserk and the document took on a life of its own! To Microsoft, only handling documents up to 100 pages was 'to specification' -- but not a specification that was widely publicised.
So, you need to understand what your product was designed for, what it was not designed for and what the vendor would not recommend. When confronted with being asked, "What should we not do with your product? What could we do that would be beyond its capabilities or design constraints?" most vendors will give you honest answers albeit couched in sales terms. So, you have to listen very, very carefully to what is said and not said.
When they say, "It can do x", does this mean it cannot do "y"? When they say, it can do "x" transactions per minute, what happens at "x+1" and "x+100"? What assumptions are there when claiming this throughput volume? (Like, is it assumed that it is the only system on the machine?)
What you're trying to do is discover the vendor's quality profile for the product. All products have limitations and constraints. They are built to a specification and expected usage. Within these quality boundaries they should work well. You first need to understand what these quality boundaries are.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.