How to Choose an AI Vendor Without Wasting Your Money
Seven criteria for evaluating AI vendors, the questions to ask before signing, red flags, and how to run a pilot on real data before paying in full.
Three AI vendors pitched in the same week. Each had an impressive demo, beautiful slides and promises of huge time savings. Six months after signing with the cheapest one, the team found the system handled Thai poorly on the company's real documents, integration cost extra, and if they wanted to leave, the data they had built up was hard to get out. This is preventable if you have a clear way to evaluate AI vendors before the first meeting.
What is AI vendor evaluation?
AI vendor evaluation means checking whether an AI provider or product solves your real business problem, is secure and good value, and does not create unexpected new risks. NIST AI RMF 1.0 addresses this in GOVERN 6, which says organisations should have policies and procedures for AI risks arising from third-party software and data, including contingency plans for when a third party's systems fail.
Start from the problem, not the demo
Before talking to the first vendor, write down the problem, how you will measure success and how you handle it today. For example: "Reduce the time to answer customer emails about order status, with answers that match the back-office system." Without a clear problem, you will be pulled towards exciting features instead of what you need.
Seven criteria for evaluating AI vendors
1. Results on your real data
Ask for a pilot or proof of concept using your company's real data and real Thai text, not just a prepared demo.
2. Data handling
Where is data stored, for how long, is it used to train models, can it be deleted, and does it comply with the PDPA, including the rules on overseas transfers? Read more in PDPA and AI.
3. Security
Are there access controls, encryption and testing for vulnerabilities such as prompt injection, which OWASP ranks as the top risk in its Top 10 for LLM Applications 2025? Some vendors may reference AI management system standards such as ISO/IEC 42001. That is useful supporting information, but it does not replace your own testing.
4. The model underneath
Whose model does it use, can the model be changed, and how will your quality be affected if the upstream model changes?
5. The full pricing structure
Per user or per usage? Are there setup fees, integration fees or overage charges? Calculate at your realistic volume over the next 12 months.
6. Integration and support
How well does it connect to your existing systems? Is there Thai-language support, or support in your time zone?
7. Your exit route
Can you export your data, prompts and settings, and in what format? How long is the lock-in period?
Questions to ask in the first meeting
- Do you have customers in our industry or of our size, and can we speak to a reference customer?
- If the system gives a wrong answer that causes harm, how is responsibility shared?
- How much visibility will we have into logs and quality reports?
- What is on next year's roadmap, and which features are still only plans?
Record every vendor's answers in the same table. Differences become much clearer than when relying on memories of the pitch.
Red flags
- Claims of 100 percent accuracy, or that it never makes mistakes
- Refusing a trial on real data before a long-term contract
- Vague answers on data and security, with no documentation
- Pressure to decide quickly with a promotion that expires soon
A good demo shows the vendor prepared well. Only a good pilot shows the system works for your job.
Watch out for
Do not decide on price alone. A cheap system that staff have to correct every time can cost more in the long run. Have your legal team review the contract, especially liability, data protection and termination terms. This article is general information; consult a professional on contracts.
How to get started
- Write the problem and success metrics on one page.
- Shortlist two or three vendors and send them the same questions based on the seven criteria.
- Run a four- to eight-week pilot with real data in a limited scope.
- Measure against your current approach: quality, time and total cost.
- Negotiate a contract with clear review points and an exit route.
Before choosing a vendor, assess the project's risk with Assessing AI risk before deployment, and plan long-term spend with Controlling AI costs.