Before a small business buys an AI product, it should be able to explain the problem the tool will solve, what information it will handle, what it can do, and how the business can stop using it. A short vendor review helps uncover hidden dependencies and makes it easier to compare options without relying on a sales demo.
This checklist is about evaluating an AI supplier and service. It complements our existing business software vendor scorecard with AI-specific questions, and our AI-agent adoption checklist covers additional safeguards when software can take actions.
Start with the job, not the product
Write a one-sentence use case, such as “draft internal meeting summaries from approved notes for a staff member to review.” Specify who will use it, what information it needs, what a correct result looks like, and what it must never do. A vague goal like “use AI to improve productivity” cannot guide a meaningful evaluation.
Define a baseline before a trial: time spent per task, current error or rework rate, and any review time. Set a stop condition too. For example, stop if the tool exposes data, creates a material error that is not detected, or requires more correction than the current process.
12 questions to ask an AI vendor
- What exactly does the product do? Ask which features use generative AI, which actions are automated, and what requires a human click.
- Which model and sub-processors are involved? Ask whether the underlying model or hosting providers can change, and where to find the current sub-processor list.
- What happens to prompts, files, and outputs? Request clear retention and deletion terms, including backups and logs.
- Is customer data used to train or improve models? Get the answer for the specific plan and configuration in writing; do not infer it from a marketing page.
- Where is data processed? Confirm available regions and whether the arrangement meets your customer commitments and applicable requirements.
- How is access controlled? Check user roles, administrator controls, multifactor authentication, audit logs, and account recovery options.
- How does the service handle errors? Ask what sources it uses, whether it can cite them, and what happens when it lacks a reliable answer.
- Can you review outputs before actions occur? Confirm whether messages, edits, payments, or record changes can be held for approval.
- What integrations and permissions are requested? List the data and actions each connection allows; prefer the smallest permission set needed.
- How are incidents reported? Ask how the provider notifies customers of security incidents and where to report a suspected issue.
- What are the total costs and exit terms? Include seats, usage, setup, support, storage, cancellation, and the cost of exporting or removing data.
- What documentation supports the claims? Request current product documentation, terms, privacy details, security information, and an explanation of limitations.
Run a controlled pilot
Use a small group and non-sensitive test material first. Prepare a representative test set with ordinary requests, ambiguous cases, incorrect assumptions, and questions the system should decline. Have a subject-matter reviewer score accuracy, clarity, source support, and the amount of editing required. Include the time spent checking and correcting responses when calculating value.
Do not connect live customer data or grant write access merely to make the trial impressive. If integration is needed, use a sandbox or read-only setup where possible, and document who can approve expanded access. Compare at least one alternative using the same test cases and criteria.
Decision record and red flags
Record the use case, tools considered, evidence reviewed, data categories, permissions, trial results, costs, owner, and reasons for approval or rejection. Revisit the decision if the vendor changes material terms, models, data handling, or capabilities.
Pause if the supplier cannot explain data retention, refuses to clarify training use, asks for broad permissions without a clear reason, makes unqualified accuracy or security promises, or cannot describe how customers leave. A small business does not need to accept uncertainty just because a product is popular.
Frequently asked questions
Should a small business ask for an AI security certificate?
Ask what independent assurance and technical documentation are available, what scope they cover, and when they were issued. A certificate or report is evidence to evaluate, not proof that every risk is addressed.
Can we rely on a vendor's privacy page?
Use it as a starting point, then check the terms for your plan, settings, integrations, and contractual commitments. Keep a copy of the materials reviewed and confirm important answers in writing.
How long should an AI pilot run?
Long enough to test realistic cases and observe the review workload, but bounded by a defined decision date and stop conditions. The right duration depends on how often the task occurs.
Conclusion
Good AI procurement begins with a well-defined task and ends with evidence that the tool is useful, manageable, and appropriate for the data it handles. Ask the same questions of every vendor, test real workflows safely, and keep a record of the decision. Features, contract terms, and privacy obligations vary; confirm them with the supplier and qualified advisers where needed.