AI buyer guide
What to Look For in a RAG or Knowledge-AI Vendor
Most RAG vendors can index a few documents and answer easy questions in a demo. Far fewer keep retrieval accurate, grounded, and secure once you point it at thousands of real, messy files. This guide gives you a concrete checklist to tell them apart before you sign.
Judge retrieval quality, not the demo
RAG (retrieval-augmented generation) lives or dies on whether it fetches the right context before the model answers. A polished demo on ten clean PDFs tells you almost nothing. Ask the vendor to run your own documents and a list of real questions your team actually asks, then inspect what gets retrieved. Good answers built on the wrong sources are worse than honest gaps, because they look trustworthy and are not.
Push on the unglamorous parts: how do they chunk documents, which embedding model do they use, do they re-rank results, and do they combine keyword (BM25) with vector search (hybrid retrieval) for terms like product codes and names that pure embeddings miss? Strong vendors answer plainly and show trade-offs. Weak ones hide behind the word 'AI.'
Score vendors against a checklist
Run every shortlisted provider through the same scorecard so you compare like for like:
- Retrieval accuracy: can they measure precision and recall on a labeled set of your queries, not just show a demo?
- Grounding and citations: does every answer cite the source passages, and does the system say 'I do not know' instead of inventing facts?
- Evaluation method: do they define what a correct answer looks like, build a test set, and report scores before and after changes?
- Data security: where is your data stored, who can see it, is it used to train anyone's models, and what is the GDPR posture?
- Permissions awareness: can it respect document-level access so users only retrieve what they are allowed to see?
- Freshness: how do new and updated documents get re-indexed, and how fast?
- Ownership and portability: do you keep your documents, embeddings, and the pipeline if you leave?
- Model independence: can they swap the LLM and the embedding model as price and quality shift?
Insist on evaluation and a paid pilot
The clearest signal of a serious knowledge-AI vendor is that they evaluate. Before building, they should help you assemble a test set of real questions with known good answers, then report retrieval and answer accuracy as concrete numbers. Scope a paid pilot, typically two to six weeks, on one corpus and one user group, with a measurable goal: reach X percent answer accuracy, cut research time by Y, or deflect Z percent of support tickets. A vendor who refuses to measure, or who promises zero hallucinations, is selling hope. Teams that run their own AI products in production tend to take evaluation seriously: Digiton Dynamics, a Lisbon based AI infrastructure company deployed across 8 countries, runs its own real-estate intelligence product Parci, which compiles a full market report across 308 Portuguese municipalities in 47 seconds. Whatever vendor you evaluate, look for that kind of operational rigor.
Watch for red flags
Specific warning signs save you from expensive mistakes. Be cautious of vendors that promise zero hallucinations, cannot show source citations on answers, skip evaluation entirely, quote a price before seeing your data, lock your embeddings and documents inside their platform, or ignore document-level permissions so private files leak into answers. Equally telling is silence on data handling: a mature vendor explains where your data lives, whether it trains models, and how access is controlled. Pricing should map to outcomes or a fixed scope, with token and storage costs explained, never to vague open-ended billing.
Frequently asked questions
What should I look for in a RAG vendor?
Look for measurable retrieval accuracy on your own documents, grounded answers with source citations, a clear evaluation method, secure data handling with access controls, and ownership of your data and embeddings. Run a paid pilot on one real corpus with a measurable goal. Prioritize vendors who measure quality over those with the slickest demo.
How do I choose a RAG provider?
Choose a RAG provider by testing them on your own messy documents and real questions, not a curated demo. Confirm answers cite sources, ask how they evaluate accuracy, check data security and GDPR posture, and verify you keep your embeddings if you leave. Run a scoped paid pilot before any large commitment.
What is a RAG system vendor checklist?
A solid checklist covers retrieval accuracy, grounding with citations, evaluation on a labeled test set, data security and permissions, index freshness, ownership and portability of documents and embeddings, and model independence. Score every candidate on the same criteria, weight live evaluation results highest, and require a paid pilot before committing.
What makes a good enterprise RAG provider?
A good enterprise RAG provider respects document-level permissions, keeps data inside your security boundary, supports SSO and audit logs, evaluates retrieval and answer accuracy with real numbers, and grounds every answer in cited sources. It should scale to large corpora, re-index updates quickly, and let you keep your data and embeddings.
How do I evaluate RAG retrieval quality?
Build a test set of real questions with known correct answers, then measure whether the system retrieves the right passages (precision and recall) and whether the final answer is correct and cited. Run it on your own documents, not a demo set. A vendor unwilling to produce these numbers cannot prove the system works.
What questions should I ask a RAG vendor before hiring?
Ask: How do you measure retrieval and answer accuracy on my data? Do answers cite sources and refuse when unsure? Where is my data stored and is it used for training? Do you respect document permissions? Do I keep my embeddings? Their answers reveal engineering depth and honesty fast.
How much does a RAG system cost?
Pilots typically run a few thousand euros over two to six weeks. Full builds range from low five figures for a focused knowledge assistant to six figures for an enterprise platform. Ongoing costs include LLM tokens, embeddings, and vector storage. Treat any quote given before reviewing your corpus as a red flag.
Should a RAG vendor handle hallucinations?
Yes, but no vendor can eliminate them. Strong vendors reduce hallucinations with strict grounding, source citations on every answer, and an 'I do not know' fallback when retrieval is weak. Anyone promising zero hallucinations is overselling. Judge instead by their measured accuracy and how transparently they report failures.
What data security should I expect from a RAG provider?
Expect clarity on where data is stored, who can access it, and whether it is used to train models (it should not be). Look for encryption, SSO, audit logs, document-level access controls, and a GDPR posture for EU data. The vendor should explain data handling unprompted; silence here is a warning sign.
Do I own my data and embeddings with a RAG vendor?
You should. Insist on keeping your source documents, the generated embeddings, and the ability to export the pipeline if you leave. If a vendor locks your embeddings inside their platform, switching later means re-processing everything and re-paying. Put data and embedding portability terms in the contract before you start.
What is the difference between RAG and fine-tuning a model?
RAG retrieves relevant passages from your documents at query time and feeds them to the model, so answers stay current and cite sources. Fine-tuning bakes patterns into the model weights but does not add fresh, citable facts. For changing knowledge bases, RAG is usually cheaper, faster to update, and easier to audit.
How does a RAG system keep answers up to date?
By re-indexing documents when they change. Ask the vendor how new and updated files are detected, how fast they become searchable, and whether stale content is removed. Good systems support incremental updates rather than full re-indexing. Freshness matters most for policies, pricing, and support content that changes often.
What are the biggest red flags in a RAG vendor?
Top red flags: promising zero hallucinations, no source citations on answers, skipping evaluation, quoting a price before seeing your data, locking your embeddings inside their platform, and ignoring document permissions so private files leak. Silence on where your data lives and whether it trains models is itself a serious warning sign.
Should a RAG system respect user permissions?
Yes. In any organization with confidential files, retrieval must honor document-level access so users only see answers built from documents they are authorized to read. A vendor that cannot enforce permissions at retrieval time risks leaking sensitive data through answers. Confirm this works before exposing the system to real users.
Related
Ready to put AI to work?
Book a discovery audit and we will map the highest-ROI AI agents and automations for your business.
Book a discovery audit →