LLM Model Migration Checklist: What to Do When Your Model Is Retired

Every major model provider retires older models on a published schedule. When the model your product depends on is deprecated, requests to it will eventually fail, and the replacement the provider recommends will not behave exactly the same. Answers get longer or shorter, refusals shift, output formats drift and agents pick different tools. Very little of that shows up in a quick manual test.

This checklist is for the engineering lead or CTO who owns that migration. It works for any provider, and it assumes you would rather do this calmly, well ahead of the deadline, than on the last day.

How do providers announce model retirements?

OpenAI, Anthropic and Google each publish a deprecation page, and they use similar but not identical terms. Read your provider's page directly rather than relying on a summary, including this one, because the policies are updated.

• OpenAI's deprecations page uses deprecation for the announcement and shutdown for the date a model stops working, and marks models as legacy when they no longer receive updates. It says generally available models get at least six months' notice, specialised variants at least three, and preview models can be retired at much shorter notice, with faster timelines possible for safety reasons. Customers actively using a model are emailed, and the models API returns a shutdown date you can check in code.
• Anthropic's model deprecations page describes four states: active, legacy, deprecated and retired. It promises at least sixty days' notice before retirement for publicly released models, notifies customers with active deployments by email, and explains how to export usage by API key and model from the Console. Its dates apply to Anthropic-operated platforms; Amazon Bedrock and Google-managed Claude on Google Cloud set their own schedules.
• Google's Gemini API documentation separates stable, preview, latest and experimental models, and recommends pinning a stable model in production. Preview models can be deprecated with at least two weeks' notice, and the latest aliases are swapped to each new release after an email notice. Its deprecations page lists shutdown dates as the earliest a model might be retired, and notes that requests to some deprecated models are routed automatically to a replacement.

Two practical points follow. If you reach a model through a cloud marketplace rather than the provider's own API, the schedule that matters is the marketplace's. And with some providers a request to a deprecated model can be served by a different model instead of failing, so the risk is a silent change in behaviour, not only an outage.

What should I do when an LLM provider retires the model my product uses?

Treat it as a planned release, not a config edit. Find every place the model identifier is used, including judges, embeddings and fine-tunes, and confirm the retirement date on the provider's own deprecation page. Pin the replacement to a dated version, run your behavioural regression suite on old and new models side by side, and fix prompts, parsers and token limits where it fails. Then shadow live traffic to it, roll it out in stages by risk, and keep the old model as a rollback until its shutdown date.

Start with an inventory of every model you call

Most teams know the main model in their product. Fewer know every other place a model identifier lives. List them before a deadline forces you to.

• The main generation model, and any fallback models in retry logic.
• Smaller models used for routing, classification, summarisation or query rewriting.
• Embedding models. Moving to a new embedding model usually means re-embedding the whole corpus, because vectors from different models are not comparable.
• Fine-tuned models. OpenAI's deprecations page notes that fine-tuned versions follow the shutdown schedule of their base model.
• LLM judges in your evaluation suite, which are easy to forget because they never touch users.
• Model identifiers in notebooks, scheduled jobs, internal tools and other teams' services that share your API keys.

Your provider's usage export or your own request logs, grouped by API key and model, are the quickest way to find the calls nobody remembers.

Pin versions and assign someone to watch the pages

This is basic MLOps hygiene, and it is what turns a retirement notice from an emergency into a planned piece of work.

• Call dated snapshots or stable version identifiers in production, not aliases that move to the newest model. Aliases are handy in development; in production they turn a provider's release into your release.
• Keep model identifiers in configuration, not scattered through code, so a change is one reviewed edit.
• Name an owner who reads the deprecation pages regularly and receives the provider's notices. Make sure those emails reach a shared address, not an engineer who has since left.
• Add an automated check: where the provider exposes retirement dates through its API, fail a scheduled job when any model in your configuration has one.
• Put each announced retirement date in your release calendar, and plan the migration well before it, not at it.

Run the behavioural regression suite before anything else

The replacement model is a new dependency, so it gets the same test as any other change: your regression suite, run against the old and new model side by side on the same day. If you do not have one, build it first. Eval-first engineering describes the approach, and for retrieval systems RAG evaluation metrics covers scoring retrieval and answers separately.

Look past the headline score. The useful output is the list of cases that changed, grouped by user segment, with both answers side by side. Anthropic's own guidance says the same in plainer terms: test your applications with newer models well before the retirement date of your current one.

What usually breaks when you switch models?

• Parsers. Output that used to be clean JSON arrives wrapped in prose or code fences, a field is renamed or dropped, or list formatting changes. Use the provider's structured output features where they exist, validate every response against a schema, and treat a parse failure as a test failure.
• Request parameters. Newer models and API versions can reject parameters older ones accepted. Anthropic's deprecations page, for example, lists temperature, top_p and top_k as removed in its newest API version, where sending them returns an error. Read the provider's migration guide for your target model line by line.
• Token use. Tokenizers differ between models. Anthropic's migration guide for its current Opus model notes that a newer tokenizer can use more tokens for the same text, and tells developers to re-tune output limits and re-benchmark latency and spend. Check context limits, truncation and budgets with real inputs.
• Behaviour. Longer or shorter answers, more or fewer refusals, a different reading of the same instruction, and different tool choices in agents.
• Latency. Time to first token and total response time change, which matters for streaming interfaces and for timeouts set around the old model.
• Safety. Re-run your adversarial and prompt injection cases. A model that is better on average can still be weaker on a case you care about.

Re-tune prompts one change at a time

Prompts are often tuned to the quirks of one model. On the replacement, some instructions become unnecessary and some become counterproductive. Start by running the existing prompt unchanged on the new model, so you know what the model change alone did. Then change the prompt in small, versioned steps and rerun the suite after each one.

Keep the old prompt and model pairing intact. It is your baseline for comparison and, until the shutdown date, your rollback.

Shadow traffic, staged rollout and a real rollback

Shadow traffic

Once the suite passes, mirror a sample of live requests to the new model without showing its output to users, and compare its responses with production. Shadow traffic finds the inputs your test set missed. It also sends real user data to a second model, so confirm the provider terms and data handling are the same, and keep the comparison logs free of personal data, as described in what to log from an LLM app without storing personal data.

Staged rollout

Move real traffic in steps: internal users first, then a small share of low-risk traffic, then wider segments, holding at each step until quality, error rate, latency and token use look right. Route by a flag in configuration so each step is a reviewed change, and leave the highest-risk workflows until last.

Rollback

Keep the old model reachable through the same flag, and make rollback a single configuration change, not a redeploy. The rollback path ends on the shutdown date, which is the strongest reason to start early. In regulated settings, record the switch as a model change with its test evidence; our post on model risk management for LLM features in banking covers what change control expects.

The migration checklist

1. Confirm the retirement date and recommended replacement on the provider's own deprecation page, or your marketplace's if you use one.
2. Inventory every model identifier, including embeddings, fine-tunes, judges and fallbacks.
3. Pin the replacement to a dated or stable version, held in configuration.
4. Read the provider's migration guide for removed parameters and changed defaults.
5. Run the regression suite on old and new models side by side, and review every changed case.
6. Fix parsers, token limits and timeouts, then re-tune prompts in small versioned steps.
7. Run shadow traffic and compare it with production.
8. Roll out in stages by risk, with a one-change rollback.
9. Remove the old model from configuration before its shutdown date, and update the inventory.

The deadline is set by your provider. How calm the migration is depends on whether you started before it.

Where Octalcode fits

Version pinning, deprecation tracking, regression suites and staged rollout are the plumbing that lets a product survive its provider's release cycle. Octalcode builds and maintains that plumbing for LLM applications, SI agents and RAG systems as part of our MLOps and SI infrastructure work.

If a retirement notice has just landed and you are not sure what depends on that model, the inventory and a side-by-side regression run are the right first steps, whoever does them. We reply to enquiries within 24 hours and sign an NDA before touching any client data.