Every Flutter team adding an LLM feature reaches the same fork early: run a small model on the phone, or call a larger one in the cloud. The choice shapes your privacy story, whether the feature works offline, how fast it feels, what it does to the battery, and how quickly you can fix it once it ships.
Flutter adds a twist. The most capable on-device options live in platform SDKs and community plugins, not in Flutter itself, so the decision is also about how much native code and plugin risk you are willing to carry. This post is a decision list for that choice. If you are still deciding which feature to build, start with our broader guide on how to add SI features to a mobile app.
What your options look like from Flutter today
Cloud inference is the simple path. Any provider works through your own backend, and Google's Firebase AI Logic ships an official Flutter plugin for Gemini models. If the app calls a provider directly, Firebase's own docs treat App Check as a critical safeguard, and a thin backend of your own is still the safer default for keys, quotas and logging.
On the device, there are three realistic routes, and none of them is a single official Flutter API.
- Models built into the operating system. Apple's Foundation Models framework gives Swift code access to the on-device model behind Apple Intelligence, on devices that support it and with Apple Intelligence turned on. On Android, ML Kit's GenAI APIs expose Gemini Nano on a published list of supported devices. Flutter reaches both through platform channels or Pigeon, which means writing and maintaining some Swift and Kotlin.
- Open models you ship or download, run through community plugins that wrap runtimes such as LiteRT-LM, llama.cpp or ONNX Runtime. Google's own LiteRT-LM documentation points Flutter developers to a community-maintained package rather than an official one, and that package has since been renamed and split into separate engine packages. Budget for that kind of churn.
- Hybrid SDKs that route between device and cloud for you. Firebase documents hybrid inference for Android, Apple platforms and the web, but at the time of writing its docs do not list Flutter, so in a Flutter app you build the routing yourself.
This area moves quickly. Check each option against its current documentation before you commit, and prefer the route with the fewest moving parts that still meets your requirements.
Should a Flutter app run an LLM on-device or in the cloud?
Run the model in the cloud by default, and on the device only when the feature must work offline, handles data that should not leave the phone, or runs too often for a network call each time. Cloud inference gives you stronger models, access to your business data, and prompt or model changes without an app release. From Flutter, it is reached through your own backend or an official plugin such as Firebase's. On-device options are less direct: operating system models such as Apple's Foundation Models framework or Gemini Nano through ML Kit need platform channels, and open models run through community plugins that change often. Device support is uneven, so many apps combine both. Define one Dart interface with a local and a cloud implementation, route by sensitivity, connectivity and model availability, check availability on each call, and score both against the same evaluation set.
Privacy: what is allowed to leave the phone?
Privacy is the strongest argument for on-device inference. If a feature reads health notes, financial records or photos of documents, keeping the processing on the phone removes a whole class of questions about third-party processors and data regions.
On-device is not automatically private, though. If your analytics, crash reporting or debug logging captures prompts and outputs, the data leaves anyway. And cloud inference can be acceptable for sensitive data when you send the minimum, strip identifiers, choose providers that do not train on your traffic and pick a suitable region. In a regulated product, such as the insurance app Oneplan that we built end to end in Flutter, the useful exercise is to decide field by field what may leave the device, then pick the inference location that fits.
Offline use, device reach and model size
Offline use and device reach
If the feature must work with no connection, on a plane, in a basement clinic or on a farm, only on-device inference will do. That is a hard requirement and it settles the question quickly.
The catch is reach. Operating system models exist only on some devices, and even there they can be unavailable. Apple's framework reports when a device is not eligible or the model is not ready yet, for example while it is still downloading, and a user can turn Apple Intelligence off. Android's Prompt API publishes device lists that vary by Gemini Nano version and by API. A downloaded open model avoids that dependency but brings its own: storage, memory and a first-run download. Either way, an on-device feature needs an answer for the phones it cannot run on.
Model size against the device
Small on-device models are good at narrow, well-defined tasks: classification, extraction, short summaries, rewriting and smart replies. They are weak at broad reasoning and know nothing about your business data. Larger models do more, but they cost more memory, more storage and a longer download, and some plugin documentation notes that big models on iOS need extra memory entitlements.
- Test on the lowest-end device you support, not on the flagship in your pocket. Memory pressure shows up there first, often as the operating system killing your app.
- Do not bundle a large model in the app binary. Download it after install, on Wi-Fi by default, with resume, a storage check and a way for the user to delete it.
- Pick the smallest model that passes your evaluation set for the task, then stop. A bigger model that is rarely needed is a cost every user pays in storage and battery.
Latency and battery
On-device inference skips the network round trip, which makes short tasks feel instant once the model is loaded. Loading is the part teams forget: the first request after a cold start can be slow, so warm the model when the relevant screen opens, not when the user taps send. Long generations on mid-range phones can be slower than a cloud model that streams tokens over a decent connection.
Battery and heat are the quieter costs. Sustained work on the GPU or neural engine drains power and warms the device, and phones throttle when they get hot. On-device models suit short bursts better than long conversations. Measure on real devices with each platform's profiling tools, under realistic use, before you decide a feature is cheap to run locally.
Updates: prompts, models and app releases
Cloud inference wins clearly on change. You can adjust a prompt, swap a model or roll back a bad change on the server, run your evaluation suite first, and reach every user without waiting for store review or for people to update the app.
On-device features are harder to change. A downloaded model can be replaced, but you need versioning, a migration path and a fallback if the download fails. Operating system models are updated by the platform on its own schedule, so behaviour can shift under you without a release of yours. Two habits help: keep prompts and feature flags in remote configuration rather than in the binary, and run the same evaluation set against every model version you ship.
The hybrid fallback pattern
Most production apps end up with both. A small local model handles the common case quickly and privately, and the cloud takes the requests the device cannot. This is the pattern we use in Flutter.
- Define one Dart interface for the task, such as a summariser, with an on-device implementation and a cloud implementation, so the UI never knows which one answered.
- Route by explicit rules: data sensitivity, connectivity, whether the device model is available and ready, and how large or hard the request is.
- Check availability on every call, not once at startup, because a model can still be downloading or the user can switch the feature off.
- Decide the fallback for each case: the cloud where the data is allowed to leave, otherwise a non-SI path or a clear message saying what is unavailable and why.
- Score both implementations against the same evaluation set, so you know the quality gap and can tell users honestly when an answer came from the smaller model.
- Log which route served each request, without logging the content, so you can see how often the fallback runs and on which devices.
Firebase's Android hybrid API describes the same choices as routing modes: prefer on-device, prefer cloud, or use only one. Even if you build the routing yourself in Dart, those modes are a useful vocabulary for writing the rules down.
A quick way to decide
- Choose cloud when the feature needs your business data, broad reasoning, frequent prompt changes, or must work the same on every phone you support.
- Choose on-device when it must work offline, handles data you have decided must not leave the phone, or runs so often that a network call per use makes no sense.
- Choose hybrid when both lists apply, which is common, and accept the extra engineering of two implementations, one interface and shared evaluations.
- Choose neither yet if you cannot write down the job the feature does for the user in one sentence.
If you are still weighing the framework itself, our comparisons of Flutter vs React Native for a startup MVP and Flutter vs native for a startup cover the trade-offs that sit underneath this decision.
Octalcode builds Flutter apps together with the backend, routing and evaluation work that SI features need. If you want a second pair of eyes on where your model should run, our custom software team is happy to look at your feature and devices and give you a straight answer, including when the answer is a simpler non-SI feature.