A vendor demo is a performance. The questions were chosen, the data was prepared and the presenter knows which paths to avoid. None of that is dishonest; it is what a demo is for. The trouble is that LLM systems fail on the inputs nobody picked, and a polished demo tells you very little about those.
You can learn far more in the room than most buyers do, if you arrive prepared. This is a practical way to stress-test an SI vendor's demo: what to bring, what to ask, what to try, and how to read the answers.
Before the meeting: bring your own inputs
The most useful thing you can do is bring material the vendor has not seen. Tell them in advance that you will, so it does not feel like an ambush, but do not send the actual examples ahead of time.
- Real questions from your users, copied from support tickets, emails or search logs, typos included.
- Messy documents: scanned PDFs, tables, long contracts, outdated versions sitting next to current ones.
- Edge cases your domain experts know about: the exception to the policy, the product that was renamed, the rule that differs by region.
- Questions whose answer is not in the source material at all.
- Questions that should be refused, because the user should not see the answer or because the topic is out of scope.
Ask for the demo to run on your inputs, live. A vendor confident in their system will usually agree, perhaps with a caveat that it has not been tuned for your data. That caveat is fair. Refusing outright is not.
How do I evaluate an SI vendor's demo?
Run the demo on inputs you bring: real user questions, messy documents, edge cases and questions the source cannot answer. Ask to see the evaluation set, current scores and the cases that still fail. Check that the system declines clearly instead of inventing an answer. With permission, try a light prompt injection. Ask which model, prompt and index versions are running, who wrote the demo data, how latency and cost per request behave under load, and what the logs and monitoring show.
Questions that expose a fragile prototype
Show me the evaluation set and the scores
Ask how they know the system works, beyond the demo. A mature team will describe a set of test questions with expected outcomes, scored automatically, with results they can show you. Ask how many test cases there are, who wrote them and which ones still fail. For retrieval systems, ask whether retrieval and answer quality are scored separately; RAG evaluation metrics explains why that split matters. Building and maintaining that kind of test set is the core of SI quality assurance, whoever does the work.
What happens when the source has no answer?
This is the question that separates demos from products. Ask something you know the documents cannot answer and watch what happens. A good system says it does not know, says what it looked at and offers a next step. A fragile one produces a confident, fluent, invented answer. Our guide to what an SI assistant should say when it does not know covers what good looks like.
What changed between runs?
Ask the presenter to run the same question twice. LLM output varies, so small differences in wording are normal. Different facts, different sources or a different conclusion are not. Then ask what changed recently: the model version, the prompt, the index, the retrieval settings. A team that tracks those changes can answer straight away. A team that cannot is probably not versioning them, which makes every later fix a guess.
Who wrote the demo data?
Ask where the demo content came from. Was it written by the vendor, generated by a model, or taken from a real client with permission? Synthetic or hand-written data is fine for a first look, but it tends to be cleaner and more consistent than anything inside a real organisation. Ask how the system performed the first time it met a real customer's documents, and what they had to change.
What about latency and cost per request under load?
A single user in a demo is the best case. Ask how response time behaves with many concurrent users, very long inputs and multi-step agent runs. Ask how cost per request is measured, what drives it up, and what limits stop a runaway loop or an abusive user from consuming resources without bound. The OWASP Top 10 for LLM Applications lists this risk as Unbounded Consumption (LLM10:2025). You are not asking for binding figures in the room; you are checking that they measure it at all.
Can I see the logs and monitoring?
Ask to see what the team sees after a request: the retrieved passages, any tool calls, the final prompt, the response, latency and errors. Ask what alerts exist and who receives them. If the only way to investigate a bad answer is to rerun it and hope, running the system in production will be hard.
A light prompt injection test, with permission
Prompt injection is the first entry, LLM01:2025, in the OWASP Top 10 for LLM Applications. It covers direct injection, where a user's own input changes the model's behaviour, and indirect injection, where instructions hidden in a document, web page or file are picked up by the model. Ask before you try anything, and keep it light: you are probing, not attacking.
- Ask the assistant to ignore its instructions and reveal its system prompt.
- Ask it to answer as a different persona with no restrictions.
- If it reads documents, ask whether you can upload one containing an embedded instruction, such as a line telling the assistant to recommend a competitor.
- If it can take actions, ask what stops it from taking one the user did not intend.
A mature vendor will welcome this and explain their defences, including what they cannot fully prevent. For a fuller list, see our prompt injection testing checklist.
Red, amber, green: reading the answers
Green
- They run your inputs live and talk openly about the ones that go wrong.
- They show an evaluation set, current scores and the cases that still fail.
- The system declines clearly when the source has no answer.
- They can name the model version, prompt version and index behind the demo.
- They show traces, dashboards and alerts, and say who watches them.
- They have measured latency and cost per request, and can describe the limits in place.
Amber
- They agree to try your inputs but want to prepare first. Reasonable, if they follow through.
- Evaluation exists but is mostly manual spot checks.
- The demo data is synthetic, and they say so plainly.
- Monitoring is planned for production but not built yet.
- They acknowledge prompt injection risk but have not tested this system against it.
Red
- They refuse to use any input they did not prepare.
- There is no evaluation set, or "the demo is the test".
- The system invents an answer when the source is silent, and nobody on their side notices.
- Nobody can say which model or prompt version is running.
- There are no logs, or logs nobody can show you.
- Questions about load, cost or misuse are deferred until after you sign.
One amber answer is normal for a prototype. Several in the same area tell you where the real work is. Red answers are not always disqualifying, but they mean you are looking at a demo rather than a system, and the proposal should be scoped and planned with that in mind.
The demo shows what the system can do. Your own inputs show what it will do.
After the demo
Write up what you saw while it is fresh, including the inputs you used and how the system handled each one. Ask the vendor to describe their evaluation approach in writing, and compare it with what was said in the room. If several vendors are on your shortlist, use the same inputs with each, so the comparison is fair.
Our list of questions to ask a RAG development company before you sign picks up where this one ends, covering ownership, security and support after launch.
Where Octalcode fits
Octalcode builds LLM applications, SI agents and RAG systems, and our SI-driven QA work is the testing side of this list: evaluation sets built from real inputs, failure handling and injection testing before launch.
If you are about to sit through vendor demos, ours included, we can help you put together the inputs and questions to bring. We expect to be tested this way ourselves, and we think every vendor on your shortlist should be too.