Retrieval-augmented generation is now a common enough request that most software vendors will say yes to it. The demo will look good, because RAG demos always do. What separates a partner who ships a system you can trust from one who ships a demo with a contract attached is how they answer a few specific questions, and how comfortable they are being asked.
These are the questions we would ask if we were on the buying side. For each one we describe what a good answer sounds like and the red flags to watch for. You do not need to be an engineer to use them. You only need to listen for specifics.
Should this be RAG at all?
Ask: "Why is RAG the right approach for our problem, and what would you do instead if it is not?"
A good answer starts with your data and your users, not the technology. It covers whether answers must come from documents that change, whether citations matter, and whether a search box, a rules engine or a fine-tuned model would serve better. A partner who understands RAG vs fine-tuning will explain the trade-off in your terms. One who is willing to tell you that SI is the wrong answer for part of the problem is worth more than one who says yes to everything.
Red flag: a proposal that names a model and a vector database before anyone has looked at your documents.
How will you measure whether it works?
Ask: "How will we both know the system is good enough to launch, and how will we know when a change makes it worse?"
This is the most important question on the list. A good answer describes an evaluation set built from real questions your users ask, with known good answers, scored separately for retrieval (did the right passages come back) and generation (was the answer faithful to them and actually useful). It names the RAG evaluation metrics they track, explains how the set grows as real failures are found, and commits to building that harness before tuning starts.
- Red flag: "we test it manually" or "the client handles acceptance testing".
- Red flag: an accuracy figure promised before they have seen your data.
- Red flag: no separation between retrieval quality and answer quality.
- Red flag: evaluation scheduled for the end of the project.
What will you do with our documents?
Ask: "Walk me through how our documents become something the system can search."
Listen for ingestion detail. How do they handle PDFs, scans, tables and your messiest formats? Do they chunk along the structure documents already have, or by character count? What metadata (source, date, section, access level) travels with each chunk? How does the index stay fresh when documents change or are deleted? Most production RAG problems start here, which is why the anti-patterns we keep finding are mostly ingestion and retrieval failures, not model failures.
Red flag: a single sentence about "uploading your documents into a vector database", with no plan for updates and deletions.
When an answer is wrong, how do you find out why?
Ask: "When the system gets something wrong, how do you tell whether retrieval failed or the model did?"
A good team answers without hesitation, because they debug this routinely. Expect to hear about hybrid retrieval (keyword search alongside vector similarity, for exact terms like product codes and names), re-ranking, metadata filters, and logging the retrieved passages for every answer. They should be able to show you, for a failed question, exactly which passages the model saw.
Red flag: every quality problem is answered with "we will use a bigger model" or "we will tune the prompt".
How do you handle access control and sensitive data?
Ask: "If a user is not allowed to see a document, how do you guarantee its contents never appear in their answer?"
The right answer enforces permissions at retrieval time, filtering what can be retrieved by the user’s access rights before anything reaches the model. Asking the model to hide content is not access control. Also ask where data is stored and processed, whether model providers retain or train on your traffic, how personal data is handled, and what logging exists for audits. Expect concrete practices, described in terms like SOC 2-aligned, GDPR ready or HIPAA-grade, and be wary of broad assurances with nothing written behind them.
What happens when the answer is not there?
Ask: "What does the system say when the answer is not in our documents?"
A good system declines, says what it searched, and points the user somewhere useful. Every answer should carry citations to the passages it used, so users can check claims and your team can debug wrong ones. Ask to see refusal behaviour live, with a question you know the documents cannot answer.
Ask to see the system fail. A team that has engineered for failure will show you gladly. A team that has only built a demo will change the subject.
Red flag: a demo that only runs on questions the vendor chose.
Who owns it after launch?
Ask: "What do the first months after launch look like, and what do we own at the end?"
Good answers cover monitoring in production (answer quality, retrieval misses, latency and cost per query), a process for turning user-reported failures into new evaluation cases, and a plan for model upgrades, since the model you launch on will not be the one you run on later. You should own the code, the prompts, the evaluation set and the ingestion pipeline, with documentation your own team can operate.
- Red flag: a proprietary platform you cannot leave without rebuilding.
- Red flag: no mention of monitoring cost per query.
- Red flag: prompts and evaluation data held by the vendor.
- Red flag: "maintenance" defined as fixing bugs on request.
How do you scope and stage the work?
Ask: "What will we have at the end of the first phase, and how will we decide whether to continue?"
Strong partners propose a short first phase on a narrow slice of your documents and real questions, ending with measured results against the evaluation set and an honest recommendation. That gives you a decision point based on evidence rather than enthusiasm. It is how we structure our own RAG development engagements, and it is a fair standard to hold any partner to. Weak proposals commit to the full build up front, with milestones defined by features delivered rather than quality reached.
Who will actually build it?
Ask: "Can we speak to the engineers who will work on our system, before we sign?"
The people in the sales meeting are often not the people who will write the ingestion pipeline. A good partner will put a senior engineer on a call with you, and that engineer will be able to talk through your documents, your access rules and the evaluation plan without reaching for slides. Ask them about a RAG system they shipped that went wrong, and what they changed afterwards. Specific, slightly uncomfortable stories are a good sign.
Red flag: the technical team is introduced only after the contract, or the answer to every technical question is "our engineers will figure that out".
Listen to how they answer
The content of the answers matters, but so does the tone. Teams that have shipped RAG to production talk about failure modes freely, ask you hard questions about your data and users, and are specific about what they do not know yet. Teams that have not tend to talk about models, speed and features. If a partner gets defensive at any question on this list, you have learned what you needed to.
We build RAG systems with the evaluation harness, citations and access controls in place from the first phase, and we are glad to be asked every question here. If you are comparing partners, our RAG development service explains how we scope that first phase, and we will answer these questions about your own documents on a first call.