Small LLMs on-device: what a 0.5B model can do
Large language models usually mean an API call to someone else’s servers. For healthcare data that is often the sticking point: the text you want processed is exactly the text you cannot send out. So we tested the other end of the scale. Five of our Playground demos run a 0.5-billion-parameter language model, entirely inside your browser, with no server involved.
This is what that taught us about when a small local model is enough.
What we run
The model is Qwen2.5-0.5B-Instruct, compressed to 4-bit precision so it can run in a browser through WebAssembly. These demos use it:
- Speech to SOAP note. Speech is transcribed first with a small speech model, then the language model turns the transcript into a structured note.
- Invoice and receipt processing. Text detection reads the page, then the language model structures it into fields such as vendor, date and totals.
- The three retrieval demos (plain, chunk overlap and reranking). The model answers a question from passages retrieved out of your document.
The speech-to-note demo downloads about 340 MB the first time, which is why we mention it. After that, nothing leaves the device.
What it does well
All of these jobs have something in common: the answer is already in the text we hand the model. Its work is to rearrange, extract or summarise, not to know things. For that kind of task a small model is genuinely useful, and we make it easier on the model in three ways:
- Tight prompts. The retrieval demo’s instruction is to answer only from the numbered passages, and to say “I could not find that in the document” if the answer is not there.
- Low randomness. The note-drafting demo does not sample at all, so the same transcript produces the same draft. The other demos use a low temperature.
- Short outputs. Each task caps how much the model may write.
Where it falls short
- Knowledge. A 0.5B model knows little. Ask it something that is not in the text you gave it and it will guess fluently.
- Consistency of format. Small models drift: a heading missing here, an extra line there. We treat the structure as a draft that code and a person check.
- Long or multi-step reasoning. It loses the thread. Keep each call narrow.
That is why the speech-to-note demo tells you to treat the result as a rough draft, and why production engagements use larger, evaluated models for the same job.
What happens when you ask a small model to do the whole job
In our HCC coding work we ran a small general-purpose medical model (MedGemma 4B) directly on the task. On HCC capture it reached 31% recall and 62% precision, and 13% of the ICD-10 codes it returned were not valid codes. A pipeline that used small models only for narrow steps, with rules and a code catalogue around them, reached 77% recall and 74% precision with no invalid codes. The lesson is not that small models are bad. It is that they do better with a narrow job and structure around them. We describe the full approach in our note on small fine-tuned models.
When we would use one
A small local model fits when:
- The input text contains the answer
- The data cannot leave the device or the building
- The output is reviewed by a person or checked by code
- Volume is high enough that per-call API costs matter
Step up to a larger model, or add something around it, when:
- You need real-world knowledge rather than rearranged input
- Errors are costly and no one reviews the output
- The output must follow a strict format every time
What we don’t claim
We have not benchmarked these demos, and we do not publish accuracy figures for them. They exist to show what runs locally and what its limits look like. If you are weighing a small model for a real workload, build a small test set of your own documents with the answers you expect, and measure it before you trust it.
One practical note on privacy: running locally means your text stays on your device, but the browser does still download the model files from a third-party host. We say so in our privacy policy.