Running Language Models on Your Phone: The llamafu Experiment

This article was revised in October 2026 to withdraw device benchmarks, quality scores and battery figures that had not been published with a reproducible method.

The Premise

We wanted to answer a straightforward question: what does it take to run a real language model entirely on a phone, with no cloud fallback? Not a toy model. Not a keyword matcher dressed up as AI. A genuine large language model, running inference locally, processing natural language and generating coherent responses — all within the memory, thermal and battery constraints of a device that fits in your pocket.

llamafu is our vehicle for that question. It is a Flutter plugin that runs quantised GGUF models on Android and iOS through llama.cpp, so that any Flutter app can generate text on the device without an API key or a network call. This article sets out the constraints that decide what is feasible, what llamafu provides, and what we would measure to turn the remaining intuitions into evidence.

The Setup

llamafu uses llama.cpp as its inference engine. llama.cpp is a C/C++ implementation of LLM inference built for consumer hardware: it supports aggressive quantisation, runs on CPUs without requiring a GPU, and has been heavily optimised for ARM processors.

llamafu is the layer between that engine and a Flutter app. A native C++ layer wraps llama.cpp, Dart FFI bindings expose it, and a high-level async Dart API sits on top. Flutter was chosen for reach: one codebase targets both Android (API 21 and up) and iOS (12.0 and up), which matters when you want to try the same model across a range of hardware.

Beyond plain text generation, the plugin exposes streaming, chat with conversation history, embeddings, vision models such as LLaVA and Qwen2-VL, tool calling, schema-constrained JSON output, GBNF grammar-constrained generation and LoRA adapters. The minimal use looks like this:

import 'package:llamafu/llamafu.dart';

void main() async {
  // Load a GGUF model from local storage
  final llamafu = await Llamafu.init(
    modelPath: '/path/to/model.gguf',
    threads: 4,
    contextSize: 2048,
  );

  final result = await llamafu.complete(
    prompt: 'Summarise the following note:',
    maxTokens: 256,
    temperature: 0.7,
  );

  print(result);

  // Free native memory when done
  llamafu.close();
}

Two parameters in that snippet — contextSize and threads — are where most of the practical tuning happens, and both trace back to the same constraint.

Memory: The Hard Constraint

Memory is the binding constraint on mobile. It determines which models you can load and how much context you can keep.

The arithmetic is unforgiving. A 7-billion-parameter model at 16-bit precision needs two bytes per weight: about 14 GB for the weights alone, more than the total RAM of most phones before the operating system and other apps take their share. Quantisation is not optional — it is mandatory.

At a little under five bits per weight, which is where 4-bit k-quant formats such as Q4_K_M land once their scaling factors are counted, the same model’s weights come to around 4 GB. On a phone with 8 GB of RAM, of which the OS will let an app use only part, that is tight. On a phone with less, it is out of reach, and models in the 1-3 billion parameter range become the realistic choice.

Weights are not the whole bill. The attention mechanism’s key-value cache grows linearly with context length: two tensors (keys and values) per layer, each as wide as the model’s key-value dimension, per token. For a Llama-2-7B-shaped model — 32 layers, a 4096-wide key-value dimension, a 16-bit cache — that is about half a megabyte per token, so a 2048-token context costs around 1 GB on top of the weights. Models that use grouped-query attention share key-value heads and need a fraction of that. Either way, the context window you configure is a memory decision, which is why llamafu’s own guidance is to start small and increase it only if the task needs it.

When a mobile OS runs short of memory, it does not slow the app down politely; it kills it. On-device inference therefore needs a memory budget worked out before the model is loaded, not discovered when the process disappears mid-conversation.

Quantisation Trade-Offs

Quantisation reduces memory at the cost of model quality. The useful question is how much quality you lose, on which tasks.

The broad shape is well established in the llama.cpp community: 8-bit quantisation is close to the original model, and quality falls off progressively as the bit-width drops, with tasks that need precise multi-step reasoning or exact numbers degrading before summarisation and conversational tasks do. llamafu’s documentation recommends Q4_K_M as the best quality-for-size trade-off on mobile, Q4_0 where speed matters most, and Q8_0 where quality matters most and the memory is there.

What we have not done is publish our own measurements of that trade-off for llamafu, and we would rather say so than quote figures that cannot be checked. “Q4 is fine for most things” is folklore until someone measures it on a stated task set, with a stated model, against a stated baseline.

What We Would Measure, and How

To turn the constraints above into evidence, these are the measurements we consider worth making, and the method each needs.

Throughput, split into prefill and decode. Prompt processing and token generation behave differently and should be reported separately, in tokens per second, for each model size and quantisation on named devices. A single headline number hides the part users feel most: the wait before the first token appears on a long prompt.

Peak memory and survivable context. The resident memory at load and at full context, and the largest context that runs without the OS terminating the app, per device. This is the number that decides whether a feature ships.

Quality per quantisation. A fixed, published task set — summarisation, question answering, short reasoning problems, code — scored identically for the same model at Q8_0, Q4_K_M and Q4_0, with the full-precision model as the reference where it fits on some test machine.

Sustained behaviour. Phones throttle under sustained load. Generation speed measured over several minutes of continuous inference, not just the first burst, together with battery drain over a fixed workload, says more about real use than a cold-start benchmark.

Every result needs the device model, OS version, llama.cpp revision, thread count and context size recorded beside it. Without those, mobile inference numbers are anecdotes.

What On-Device Inference Is Good For

Measurements aside, the use cases follow from the architecture.

Privacy-sensitive applications. Medical questions, personal journal analysis, financial document summarisation — any task where the user would prefer their data never leave the device. This is the strongest argument for on-device inference: not that it is better than cloud inference, but that it is private by construction.

Offline operation. On a plane, in a rural area, in a country with unreliable connectivity. Once the model file is on the device, inference does not depend on a network at all.

No per-call cost. There is no API bill and no rate limit. For features that run often on small inputs — classification, extraction, short rewrites — that changes what is affordable.

Structured, narrow tasks. Grammar-constrained generation and schema-constrained JSON make small models far more useful than free-form chat suggests, because the output space is restricted to valid answers.

Where It Struggles

Complex reasoning. Smaller, heavily quantised models produce plausible-sounding but wrong answers on multi-step problems more often than large hosted models. A phone gives you a capable assistant for drafting and summarising, not a reliable reasoner.

Long context. Memory caps the context window. Analysing a long document or keeping a long conversation coherent hits that ceiling, and when context is truncated the model simply loses track of what came earlier.

Older hardware. Generation speed scales with the processor and memory bandwidth, and on older devices it can be slow enough that users accustomed to cloud assistants will not wait.

Current knowledge. The model’s knowledge is frozen at its training cutoff. Without retrieval, it cannot check anything.

Heat and battery. Sustained generation is sustained compute, and phones respond with heat, throttling and battery drain. Occasional queries are cheap; long sessions are not.

Honest Conclusions

On-device LLM inference is real and useful today, and it is meaningfully limited compared with cloud inference. Most of the limits are not software bugs; they are memory and thermal constraints that will ease as mobile hardware improves, but will not vanish.

The right framing is not “on-device instead of cloud” but “on-device when it matters”: privacy-sensitive tasks, offline scenarios, and frequent narrow tasks where availability and cost matter more than peak quality. For complex reasoning, long-context analysis and tasks that need large models, cloud inference remains the better answer.

llamafu’s job is to make the on-device side of that choice cheap to try from a Flutter app. The job of the measurements above is to make the choice an informed one, and publishing them — with the device, the build and the method beside every number — is the work still ahead.