Small LLMs on-device: what a 0.5B model can do
A small local language model is a good fit for narrow, text-in text-out jobs where the answer is already in the input. It is a poor source of knowledge, and it needs a person checking its output.
Using large and small language models where they earn their place: cost, privacy, running them locally, and where they fall short.
Language models are useful, expensive and easy to over-trust. This area is about using them where they earn their place: which tasks suit a large hosted model, which suit a small one running on your own hardware, and where a model should not be doing the job at all.
The notes cover cost, privacy and running models locally, with examples from our own work: a 0.5-billion-parameter model that drafts notes in the browser, and a healthcare coding task where a pipeline of small models beat repeated calls to a large one. They also say plainly where small models fall short.
Each topic carries a stance: use it, try it, watch it, or avoid it. The stance is what we would do today, with the evidence behind it.
A small local language model is a good fit for narrow, text-in text-out jobs where the answer is already in the input. It is a poor source of knowledge, and it needs a person checking its output.
Working on something in this area? Tell us the problem and we will tell you what we would do.
View AllSend us the paragraph you'd normally write to a colleague. We'll get back to you within 24 hours.
hello@tathastha.com