Most teams add guardrails to an LLM feature the same way: one moderation call on the way in, perhaps another on the way out, and a hope that this covers it. It rarely does. The incidents that hurt in production tend to come from places a single filter never sees: an instruction hidden in a retrieved document, an agent calling a tool with arguments nobody checked, or unsafe text that has already streamed to the screen before the output check runs.
This guide maps where each check belongs at runtime and what it costs. It is a companion to our prompt injection testing checklist, which covers how to attack your own system before launch. Here the question is what stays in place after launch, on every request.
What counts as a guardrail?
A guardrail is any check that can block, change or flag a request or response. They come in four kinds, and a good design uses all of them:
- Deterministic checks: schema validation, length limits, allow lists, patterns for secrets or account numbers. Fast, predictable and easy to test.
- Classifier models: small models trained to label content. Meta's Llama Guard 4, for example, is a 12-billion-parameter classifier that labels both prompts and responses as safe or unsafe against a hazard taxonomy based on MLCommons categories.
- LLM checks: a separate model call that judges the input or output against a written policy. Flexible, but slower and less predictable.
- Policy in code: permissions, approvals and rate limits enforced by your application and downstream systems, not by the model.
Open-source frameworks package these checks. NeMo Guardrails, an Apache 2.0 library from NVIDIA, defines five types of rails: input, dialog, retrieval, execution (around tool calls) and output. Guardrails AI, also Apache 2.0, runs input and output guards built from validators, many of them published on its Guardrails Hub. Either can sit in your stack, and neither replaces the decisions below about where checks go.
Where should guardrails go in a production LLM application?
Place checks at five points. On input, limit size and rate, mask personal data and screen for injection. On retrieved content, filter by user permission and treat document text as untrusted data. On tool calls, validate arguments, give the agent the fewest permissions possible, enforce authorisation in the downstream system and require human approval for high-impact actions. On output, validate structure, encode before rendering and screen for leaks or unsafe content. When streaming, decide whether each chunk is checked before or after the user sees it.
What should you check on input?
Input checks run before the model call, so they are the cheapest place to stop obvious problems.
- Size and rate: cap prompt length, attachments and requests per user. The OWASP Top 10 for LLM Applications lists unbounded consumption as a risk in its own right.
- Personal and secret data: detect and mask it if the model does not need it.
- Scope: reject requests far outside what the product is for, with a helpful reply rather than a bare refusal.
- Injection screening: a classifier or rules for known jailbreak patterns. Useful, but never the only defence, because attackers rephrase faster than lists update.
- Identity: attach the authenticated user and their permissions to the request now, so every later check can use them.
How do you guard retrieved content?
OWASP defines indirect prompt injection as what happens when an LLM accepts input from external sources such as websites or files. In a RAG system or a browsing agent, every retrieved chunk is that kind of input. A user can ask an innocent question and still receive an answer steered by text an attacker placed in a document.
- Filter by permission before retrieval, not after generation. A chunk the user cannot open should never reach the prompt. Our guide to per-user access control in RAG covers the patterns.
- Mark retrieved text as data. Put it in clearly delimited sections and tell the model it may contain instructions it must not follow. OWASP lists labelling untrusted content among its mitigations.
- Screen chunks at ingestion and at retrieval for instruction-like text, hidden characters and links, and record which source each chunk came from.
- Mask sensitive fields inside chunks when the answer does not need them. NeMo Guardrails' retrieval rails exist for exactly this: rejecting or altering chunks before they reach the prompt.
Which checks matter most for agent tool calls?
For agents, this is where the real damage happens. A wrong sentence is embarrassing; a wrong refund, deletion or email is an incident. OWASP's guidance on excessive agency is the best checklist we know for this layer:
- Offer only the tools the task needs, and only the functions each tool needs. A summariser that reads mail should not be able to send it.
- Prefer narrow tools to open-ended ones such as a shell or a general HTTP client.
- Run each action with the user's own permissions, not a shared service account with broad access.
- Enforce authorisation in the downstream system, not by asking the model whether an action is allowed.
- Validate every argument against a schema and business rules (amount limits, allowed recipients, record ownership) before execution.
- Require a person to approve high-impact actions. Our guide to human-in-the-loop approval patterns covers how to do this without approval fatigue.
- Rate-limit actions and log every call, so a misbehaving loop is caught and contained.
What should output checks catch?
Treat model output as untrusted input to whatever consumes it next. OWASP calls the failure here improper output handling.
- Structure: validate JSON or other formats against a schema in code, and retry or fail cleanly when it does not match.
- Encoding: escape output before it is rendered as HTML, used in SQL or passed to a shell.
- Leaks: check for personal data, secrets and fragments of your system prompt, which OWASP lists separately as system prompt leakage.
- Safety and policy: a classifier or policy check for content your product must not produce.
- Grounding, for RAG: whether the answer is supported by the retrieved sources, and whether cited sources were actually retrieved.
Decide what the user sees when a check blocks something. A clear, honest message beats a silent failure; our piece on what an SI assistant should say when it does not know applies here too.
How do guardrails work with streaming?
Streaming creates a real trade-off: you cannot check a whole response that has not been written yet. NeMo Guardrails' documentation shows the two options clearly. Output rails run on buffered chunks of tokens, with some context carried over from the previous chunk so a problem spanning a boundary is still seen. Then you choose: send each chunk to the user before it is checked, which keeps latency low but means text already shown cannot be recalled, or check each chunk first, so blocked text never reaches the user but every chunk waits for the check.
- Check before sending for routes where one bad sentence is costly: regulated advice, personal data, anything shown to the public.
- Send then check for low-risk routes, and design the interface to replace a partial answer with a clear message if a later chunk is blocked.
- Never stream tool calls into execution. Arguments are validated whole, after the model finishes producing them.
How much latency do guardrails add?
Every check adds time, so placement and order matter more than the number of checks. Run cheap deterministic checks first and stop early. Run independent checks in parallel; NeMo Guardrails, for example, can run input or output rails concurrently so the first one to block ends the request. Prefer a small classifier to a large LLM judge where it performs well enough on your own test set. And move checks that only need to inform, not block, off the request path into asynchronous review. Measure the added time per route under real load rather than trusting a benchmark.
What should you log?
For every check, record the request ID, the guardrail's name and version, its decision, any score, the rule or category that fired, and the time it took. Log a reference to the content rather than the raw text where personal data may be involved; our guide to logging from an LLM app without storing personal data covers how. Then watch block rates per route. A sudden rise means an attack or a broken check; a steady trickle of complaints about refusals means false blocks.
How do you test guardrails?
- Keep two test sets: attacks that must be blocked, and harmless requests that look similar and must pass. Measure both missed attacks and false blocks.
- Re-run both on every change to the model, prompts, retrieval or guardrail configuration, and treat a drop as a failed release.
- Test each layer on its own, including a poisoned document in the retrieval index and malformed tool arguments.
- Test the streaming path separately from the non-streaming one, since the checks run differently.
- Decide what happens when a guardrail service is down: fail closed for high-risk routes, and know which routes fail open.
Where Octalcode fits
Octalcode builds LLM applications, SI agents and RAG systems, and guardrails are part of how we test them rather than an add-on at the end. The aim is a set of checks you can explain, measure and change without fear.
If you want an outside view of where your checks sit today, our SI quality and testing work can map your request path, test each layer and leave you with a written list of gaps, whoever ends up fixing them.