Most companies that want SI features already have software that works: a web app, an operations console, a back office, a mobile app with a real backend. The question is not whether to rebuild it around a language model. It is how to add model-powered features without destabilising a system people rely on every day.
The good news is that you almost never need a rewrite. The patterns below are the ones we use to put LLM features into existing products while keeping the old system boring, predictable and safe.
Treat the model as a dependency, not a platform
The rewrite instinct comes from treating SI as the new centre of the system. Resist it. The model should sit behind an interface your existing code calls, the same way it calls a payment provider or a search service. Your domain logic, data model and permissions stay where they are.
From your architecture’s point of view, a language model is a slow, probabilistic, external dependency. You already know how to integrate those.
That framing gives you three properties straight away. The existing system keeps working if the model is slow or down. You can swap models without touching business code. And you can test the integration like any other dependency. Everything else in this guide follows from it.
Latency deserves its own budget too. A model call is far slower than a database query, so decide per feature how long a user can reasonably wait, stream responses where the interface allows it, and cache answers to questions that repeat.
Four integration patterns that avoid a rewrite
Which pattern fits depends on where the feature lives and how fast it must respond. Most products end up using two or three of them together.
- Sidecar service. A small, separate service owns everything model-related: prompts, retrieval, tool calls and output validation. The existing app calls it over an internal API. New dependencies, new languages and fast-changing prompt code stay out of your core codebase.
- LLM gateway. One internal gateway sits between your services and every model provider. It handles keys, routing between providers, retries, timeouts, rate limits, logging and redaction in one place. Even a thin version pays off as soon as more than one team calls a model.
- Feature flags. Every SI feature ships behind a flag scoped by user, account or environment. Flags let you turn a feature on for internal users first, compare it with the old path, and switch it off in seconds if something goes wrong.
- Async jobs. Anything that does not need an answer inside the request (summarising a document, classifying a ticket, drafting a reply, enriching a record) goes on the job queue you already run. Nobody waits on a model call, retries are simple, and failures land somewhere you already monitor.
The sidecar plus gateway combination is the default architecture for the SI applications we build on top of existing products, because it keeps the blast radius of every change small. The async pattern is the most underrated of the four. Many of the most valuable SI features are not chat at all; they are background enrichments that make an existing screen smarter, and a weak output can be held for review before anyone sees it.
Data access: give the model less than you think
The hardest part of integration is rarely the model. It is deciding what data the model can see, and making sure it never sees more than the user who is asking. These are the rules we hold to:
- The model never gets database credentials. It reads data through the same service layer and permission checks your application already uses, ideally a narrow set of read endpoints built for the feature.
- Permissions are enforced before retrieval, not after generation. If a user cannot see a record in the app, it must never be retrieved into their prompt.
- Unstructured content such as documents, tickets and notes goes through a retrieval pipeline that carries metadata for access control and freshness.
- Sensitive fields are redacted or masked at the gateway unless the feature genuinely needs them.
- Writes are tools, not free text. If the model can change anything, it does so through explicit, validated functions with the same authorisation as a human user.
If the feature needs semantic search, choosing a vector database is a smaller decision than it looks, and the database you already run can often handle it. The moment a feature can write data or trigger actions, though, you are building an agent and the design questions change. We cover scoping, approvals and audit trails for that case in agents that don’t go rogue.
Guardrails at the boundary
Models produce text. Your system expects valid data. The integration layer is where that gap gets closed, and it should be strict.
Use structured outputs where your provider supports them, and validate every response against a schema before it touches application code. Treat model output as untrusted input, exactly like anything a user types, because content retrieved from documents or emails can carry instructions aimed at the model. That is prompt injection, and the defence is architectural: limit what the model can do, not just what it is told.
Set hard timeouts and design a fallback for every feature: a cached answer, the old non-SI path, or a clear message. The existing product should degrade gracefully and never break because a provider had a bad hour. Log every call with its prompt version, model, inputs, outputs and latency. When something goes wrong, those logs are the difference between a quick diagnosis and a week of guessing.
Evals before rollout, and on every change
Integration tests tell you the plumbing works. They do not tell you the feature is any good. For that you need an evaluation set: real inputs from your own system, with known good outputs or a scoring rubric, run automatically against every prompt and model change.
On our copilots embedded in an engineering-ops console, a custom eval harness runs regression suites on every model change. That is what makes it safe to upgrade models inside a live product: you know before users do whether the new version is better, worse, or different in ways that matter.
Build the set from your own logs and support history rather than invented examples, and include the awkward cases: empty fields, very long records, mixed languages, the customer who writes in capitals. Score structured outputs with exact checks and free text against a rubric, and keep a human spot-check for the cases the automated score is least sure about.
Rolling out without drama
Feature flags make a staged rollout cheap, so use every stage:
- Shadow mode: the feature runs on real traffic, but its output is logged rather than shown. Compare it with what people or the old system did.
- Internal users: turn the flag on for your own team, who will report rough edges honestly.
- A small slice of customers: selected accounts or a small share of traffic, with an easy way to send feedback.
- Wider release: only when eval scores, error rates and latency hold steady at each earlier stage.
Keep the old path alive until the new one has earned the right to replace it. Many features never fully replace anything. They sit alongside the existing workflow as an assist, which is often where SI adds the most value with the least risk.
Where to start
Pick one feature with a clear owner and a measurable outcome, ideally an async one. Put a thin gateway in front of the model, build the sidecar, wire one flag, and write the eval set before the prompt. That first feature sets the pattern every later one follows, so it is worth getting the boring parts right.
This is the shape most of our SI application work takes when a client already has a working product, and it pairs naturally with our custom software engineering when the existing system needs some groundwork first. If you want SI features in your product without betting the product on them, we are happy to talk through where they would fit.