A bank's model risk function was built for credit scorecards, pricing models and stress tests: mathematical models with stable code, known inputs and outputs you can check against outcomes. An LLM feature that drafts credit memos, answers customer questions or summarises complaints fits that function awkwardly. The core model belongs to a vendor, the same input can give different outputs, and a one-line prompt edit can change behaviour more than a retrain.
This post is for heads of product, model risk leads and engineering managers at banks and lenders bringing an LLM feature through model risk management. It covers where the guidance stands, inventory and tiering, what makes validation hard, the evidence to produce, monitoring, vendor risk and change control. Logging is covered separately in what an LLM audit trail in fintech should record, so we do not repeat it here.
Where the guidance stands in 2026
In April 2026 the OCC, Federal Reserve and FDIC issued revised interagency guidance on model risk management (OCC Bulletin 2026-13), replacing the 2011 guidance that many bank model risk policies were built on. It is not an enforceable standard. It covers model development and use, validation and monitoring, governance and controls, and vendor and third-party products, and the agencies expect it to be most relevant to larger banking organisations.
The detail that matters for this post: the agencies describe generative and agentic models as novel and rapidly evolving, and state that they are not within the scope of the guidance. They said they plan to issue a request for information on model risk management that looks in particular at banks' use of these models. When we checked the Federal Register in late October 2026, it had not been published. A separate interagency proposal on third-party risk management guidance was published in September 2026, with comments due by 16 November 2026.
So the new guidance does not govern LLM features, and nobody should claim it does. In practice, banks still apply model risk principles and their own model risk policies to these features, because the risks (wrong outputs, poor data, misuse, vendor dependence) are the same kind of risk the function already manages. Expect separate supervisory work on generative models to follow.
Outside the US, two points are worth knowing. In the UK, the PRA's supervisory statement SS1/23 sets model risk principles for banks with internal model approval, covers vendor models, and addresses risks from techniques such as machine learning. In the EU, the EU AI Act lists systems that evaluate the creditworthiness of individuals or set their credit score as high-risk, with an exception for fraud detection. A 2026 amendment moved the application date for these Annex III systems to 2 December 2027.
How should a bank apply model risk management to an LLM feature?
Treat the whole feature as the model: the provider's model version, prompt templates, retrieval sources, tools and guardrails. Record it in the model inventory and tier it by decision impact, autonomy and customer exposure. Validate with task-specific test sets run repeatedly, since outputs vary, and with adversarial and subgroup tests. Monitor quality, overrides and provider version changes in production. Assess the vendor's versioning, data use and change notice, and route every prompt, model or retrieval change through change control with fresh evaluation evidence.
Inventory and tiering: register the whole feature
The first mistake is to register only the vendor model. What produces the output is a system, and each part can change it. One inventory entry per feature should list:
- the provider and the exact model version in use, not only an alias that can move;
- the prompt templates and system instructions, by version;
- retrieval sources and the index or corpus version, for RAG features;
- tools the model can call, and what each tool can change;
- guardrails, output validation and any fallback path;
- the business owner, the intended use, and uses that are explicitly out of bounds.
Then tier it. Useful factors are the decision the output feeds (a credit or AML decision sits far above an internal search tool), whether customers see the output directly, how much the feature acts on its own versus drafting for a person, the sensitivity of the data, and volume. Tiering decides how deep validation goes and who signs off, so agree the criteria once rather than arguing each feature.
Why LLMs strain traditional validation
Validators who know statistical models will recognise the goals: conceptual soundness, outcomes analysis, ongoing monitoring. Four properties of LLM features make those goals harder to meet.
- Non-determinism. The same input can produce different outputs, even with sampling settings turned down. A single test run proves little; you need repeated runs and measures of how much results vary.
- Vendor models you cannot inspect. You will not see the training data, weights or full development process. Conceptual soundness shifts from reviewing the model to reviewing the use: is this task suitable for this kind of model, and is the evidence strong enough for this tier?
- Prompt changes are model changes. Rewording an instruction, adding an example or changing retrieval settings can shift behaviour as much as a new model. If prompts can be edited outside change control, the validated model is not the one in production.
- Drift when the provider updates. Hosted models are updated and older versions retired on the provider's schedule. Behaviour can change without any change on your side, so pin versions where the provider allows it and plan for forced migrations.
What evaluation evidence should look like
Validation needs evidence a reviewer outside the build team can examine. For an LLM feature, that means a documented evaluation, not a demo.
- A task-specific test set built with domain experts, covering normal cases, edge cases and cases the feature should refuse or escalate.
- Metrics tied to failure modes that matter to the bank: factual errors against source documents, missing required content, wrong refusals, and policy breaches, not only an overall score.
- Repeated runs, so the report shows the spread of results as well as the average.
- A comparison with the current process, usually the human baseline the feature is meant to support.
- Adversarial testing for prompt injection, data leakage and attempts to push the feature outside its intended use.
- Subgroup testing where the feature touches customers, to check that quality does not differ across groups it should treat alike.
- Pass and fail thresholds agreed before testing, so the result is not argued after the fact.
An eval-first release process produces most of this as a by-product: every release already has a test suite and a report. For retrieval-based features, add retrieval quality metrics, since a correct answer from the wrong document is still a finding.
Ongoing monitoring
Pre-release testing tells you how the feature behaved on known cases. Monitoring tells you whether that still holds. Set thresholds and named owners for each signal:
- sampled human review of live outputs, scored against the same criteria as the test set;
- override and edit rates, where a person reviews the output;
- refusal, escalation and fallback rates, which often move first when something changes;
- shifts in the kinds of input users send, compared with what the test set covers;
- the model version the provider actually returned, with an alert when it changes;
- a scheduled rerun of the evaluation suite, so slow drift shows up as a trend.
The monitoring data has to come from somewhere, which is where the audit trail and observability logs do their work. Keep the two designs separate, as the audit trail post explains.
Third-party and vendor model risk
Most banks will run LLM features on a provider's model, so vendor risk and model risk overlap. Before approval, get clear answers on:
- Versioning: can you pin a model version, how much notice comes before a version is retired, and how are changes announced?
- Data use: are prompts and outputs retained, for how long, and are they ever used for training?
- Location: in which regions is data processed, and can you restrict it?
- Documentation: what does the provider publish about evaluation, known limitations and safety testing?
- Incidents: how quickly will they tell you about an outage, a security incident or a behaviour change?
- Exit: could you move to another model, and has the feature been evaluated on an alternative?
The last point is about concentration as much as quality. A feature that only works on one provider's model inherits that provider's roadmap.
Documentation and change control
Model documentation for an LLM feature should cover the intended use and its limits, the full component list from the inventory, the evaluation design and results, known weaknesses, the monitoring plan and the conditions for withdrawing the feature. Write it so a validator who did not build the feature can follow it.
Change control is where LLM features most often slip. Define which changes count as material for each tier: a new model version or provider, a prompt template change, a new tool, a change to retrieval sources. Each one reruns the evaluation suite against agreed thresholds, and material changes go back to validation before release. Keep prompts in source control with reviews, not in an admin screen anyone can edit. The engineering behind this, versioned prompts, pinned models and automated evaluation in the release pipeline, is standard MLOps practice applied to a new kind of model.
Where Octalcode fits
Octalcode builds LLM applications and the MLOps and SI infrastructure around them: evaluation suites, version pinning, monitoring and release pipelines that produce the evidence a model risk team asks for. With fintech teams, we would start from the inventory and tiering questions above.
We do not replace your model risk function or your validators, and we would not want to. If you are preparing an LLM feature for review, we can help you assemble the documentation and evaluation evidence before it reaches them.