Buyer guide

12 questions to ask an AI agency before you hire one

Most AI vendor calls are decided by whoever demos best, which selects for the wrong thing. These twelve questions, from Digiton's enterprise engagements in Europe, select for a firm that ships.

By Brandon Da Costa, Founder, Digiton Dynamics. Reviewed 8 September 2026.

The twelve questions, and what a bad answer sounds like

Read the table first. Then take the four or five that matter most for your situation into the call, because asking twelve in one hour turns a conversation into an interrogation and you learn less.

1. Show me a system you have in production right now. Who uses it and how often?

This is the question that ends most conversations honestly. A firm that has shipped will answer in specifics inside thirty seconds, because the system is a thing they think about. A firm that has not will reach for a demo, a pilot, or a client who cannot be named for reasons that turn out to be vague.

Push once on frequency. Daily use by a team of forty is a different engineering problem from a system three people open in a quarter, and only one of them has met real failure modes.

2. Who gets paged when it breaks at 3am, and what is your incident process?

An AI system is a running service. It drifts, it costs money per call, and it fails in ways a normal web application does not, because a model provider can change a response shape or deprecate a version with a few weeks of notice. Somebody has to own that.

The useful follow-up is to ask for the last incident they handled and what changed afterwards. A firm that has operated anything will have one. The answer also tells you whether they run their own software or only build other people's.

3. Who owns the code, the prompts and the evaluation set at the end?

The code is the easy part and almost everybody concedes it. The prompts and the evaluation set are where lock-in actually lives. An evaluation set built from your own historical failures is the single most valuable artefact of the engagement, because it is what lets the next team change anything without breaking what already works.

Get it named in the contract as a deliverable, not as a by-product.

4. What is the scope tied to numerically, and who measures it after launch?

Hours removed, cycle time, error rate, queue length, revenue. Pick one. The baseline has to be measured before anything is built, because retrofitting a baseline afterwards produces whatever number the firm needs it to produce.

Then ask who reads it in month six. If the answer is the agency, the number is marketing. If the answer is somebody in your own operation, it is a metric.

5. How do you handle GDPR erasure when the data is inside embeddings and agent memory?

This is the question that separates people who have shipped a retrieval system in Europe from people who have read about one. A deletion request has to reach every derived copy: the vector store, the cached retrieval context, the agent's conversation memory, and any evaluation set built from production traffic.

Most builds handle the first and quietly miss the rest. The engineering detail is on GDPR-compliant RAG, and it is worth reading before the call so you can tell a real answer from a confident one.

6. Do your deployed systems meet Article 50 transparency duties, which have applied since 2 August 2026?

The deadline that moved is not this one. Regulation (EU) 2026/1744 pushed Annex III high-risk obligations to 2 December 2027, and it left Article 50 exactly where it was. Transparency duties went live on 2 August 2026 and they bind today, on ordinary systems, including the chat assistant somebody put on a website last spring.

A firm that answers this question with the December 2027 date has misread the regulation in the direction of its own convenience. There is also a second date to know: systems placed on the market before 2 August 2026 have until 2 December 2026 for content marking. Full timeline on EU AI Act readiness for enterprises.

7. What is your agent governance model: scoped identity, least privilege, human approval on irreversible actions, audit log, kill switch?

An agent with a shared API key and write access to production is an incident waiting for a bad afternoon. Each of the five has a specific shape. Scoped identity means the agent authenticates as itself and not as a service account somebody else also uses. Least privilege means read access by default and write access granted per action. Human approval covers anything irreversible, which includes sending mail, moving money and deleting records.

The audit log is the one to press on, because it is the artefact an auditor reads. Inputs, retrieved context, model version, output, approver. More detail on AI agent governance.

8. What does the handover look like if we bring this in-house in year two?

The answer matters much less than the reaction. A firm confident in its work will describe the handover cheerfully, because it has done a few and because clients who leave well come back. A firm whose commercial model depends on you never leaving will handle the question differently and you will feel it.

Ask for the handover package to be a deliverable in phase one, produced as the build goes, rather than a document somebody writes in a hurry at the end.

9. Which parts of this are your own code and which are a vendor you resell?

There is nothing wrong with reselling. There is something wrong with not being told, because a resold component brings its own pricing changes, its own outage history, and its own exit problem. If forty percent of the delivered system is somebody else's platform, your renewal negotiation is with them and not with the firm you signed.

Ask what happens to your data inside the vendor's system, and where it sits.

10. What did the last engagement that went badly look like, and what did you change?

Everybody who has delivered enough has a bad one. The useful signal is whether the lesson turned into a process. "We now measure the baseline before the build because we could not prove the value on that project" is a good answer. "The client changed their mind" is a firm blaming a client to a prospective client, which tells you what they will say about you.

11. How do you evaluate model output, and can I see the golden dataset?

Without an evaluation set, nobody can tell whether a prompt change made the system better or quietly worse on the cases that matter. The set should grow from production: every correction a human makes becomes a case, so the system is measured against its own history rather than against a benchmark somebody published.

Ask when it last ran. Ask what the score was. A firm that cannot answer is shipping on vibes.

12. What is your answer to a client security questionnaire asking for model provenance and data residency?

Enterprise procurement now asks this regardless of regulation, and the question has moved down-market fast. Model provenance means which model, from which provider, hosted in which region, and whether your data touches training. Data residency means where the vectors, the logs and the conversation history physically sit.

A firm selling into enterprises has been asked this a dozen times and has a document. A firm that has not been asked is telling you something about its client base. The vendor-side view is on AI vendor due diligence.

The three questions that changed in 2026

Nine of the twelve above would have been on this list two years ago. Three would not.

Article 50 became a live obligation. Transparency duties under the EU AI Act applied from 2 August 2026 and were untouched by the Omnibus deferral. Disclosure when a person is talking to an AI, machine-readable marking of synthetic media, deepfake notices. This binds ordinary systems, not only high-risk ones, and most vendors have not updated their answer.

The December 2027 deferral became a way to spot who reads primary sources. Regulation (EU) 2026/1744 entered into force on 27 July 2026 and moved Annex III standalone high-risk obligations from 2 August 2026 to 2 December 2027, with Annex I product-embedded systems going to 2 August 2028. It is not eighteen free months. Classification is the prerequisite for every later obligation, and a firm that treats the deferral as a reprieve will hand you the same problem in a year with less time to fix it.

The security questionnaire arrived early. Model provenance and data residency questions used to appear at contract stage in large enterprises. They now appear in the first procurement pass at mid-market firms, because those firms are being asked the same questions by their own customers. A vendor without a written answer sheet costs you weeks.

How to score the answers

Give each question three points for a specific answer with a mechanism in it, one point for a plausible answer with no mechanism, and zero for deflection. Questions 1, 2, 4 and 6 are the load-bearing ones, so double them.

Above 32 out of 48 and you are talking to a firm that has shipped. Between 20 and 32, you are talking to a good team who have not yet operated anything, which is fine for a first build if you have the internal capacity to run it afterwards. Below 20, you are buying a pilot.

One caution about the scoring. A firm can rehearse answers, and the better ones will have. What cannot be rehearsed is the follow-up: ask for the incident, the number, the audit log schema, the evaluation score from last week. Specifics are the whole test.

Where our own answers sit

Digiton's content and citation systems produced 365 Copilot citations across 30 days from 249 indexed pages, measured in Bing Webmaster Tools, which is what a production content system looks like when somebody keeps it running. The same applies to the client work: we build and then operate, which is why questions 2, 8 and 11 are on this list rather than the easier ones.

If you are working through a shortlist, how to vet an AI agency covers the reference-checking side, and enterprise AI consulting in Europe sets out what production delivery looks like once the shortlist is down to two.

Frequently asked questions

Related