Buyer checklist

AI vendor due diligence checklist

Most AI contracts are signed on a demonstration, and most AI regrets are about clauses nobody read, so this is the list to work through before the pen comes out.

Buying AI is not like buying software, because the thing you are buying changes underneath you. The model version moves, the retrieval corpus grows, accuracy drifts, and your data may or may not be feeding something else. The clauses below are the ones that turn out to matter, roughly in the order they cause problems.

Data rights

• Are inputs or outputs used to train, fine tune or improve any model, including in aggregate or anonymised form? Get an explicit no in the contract, not a link to a policy page that can change.
• Which subprocessors see the data, in which countries, and are you notified before that list changes?
• Retention. How long are prompts and outputs kept, and can you set it to zero?
• On termination, what is deleted, when, and how is that evidenced?

Evidence of performance

• Ask for the evaluation set. What was the system measured on, how many cases, and who labelled them? "It performs well" without a dataset is a claim, not evidence.
• Ask for accuracy on inputs that look like yours rather than on a public benchmark.
• Ask what happens when the model is unsure. A system with no abstention behaviour will guess, confidently.
• Ask how regressions are detected when the underlying model version changes.

Traceability

Demand a single traced example end to end: the input, the retrieved sources, the generated output, the tool calls made, the human who approved it and where that approval is stored. A vendor who cannot produce one trace has a demonstration rather than an auditable system, and you will not be able to answer a regulator or an internal auditor with it.

Cost and lock-in

• How is it metered, and what does the bill look like at three times current volume? Model your own worst month.
• Are there caps or alerts, and who can set them?
• Can you export the prompts, evaluation sets, retrieval index and embeddings in a usable format?
• What is the notice period, and is there a transition assistance obligation?

Security and compliance

• Current penetration test summary and a named security contact.
• Data processing agreement, transfer mechanism, and a data protection impact assessment where the use case requires one.
• For EU deployments, the vendor's own position on their AI Act risk classification and what documentation they provide to support yours.
• Incident notification timelines, in hours, written down.

Two questions that settle most of it

Ask who is accountable for accuracy in month four, and ask to speak to a reference running the same use case in production rather than a pilot. The answers to those two will tell you more than the rest of the process. If you want a second pair of eyes on a vendor response, Digiton reviews them as part of an AI audit.

Frequently asked questions

Related