How we cut a healthcare partner's LLM spend to $0 per chart
FINE-TUNING

NER: general vs fine-tuned, and when to switch

Named entity recognition (NER) finds the pieces of a text that name something and says what kind of thing each one is. In “Maria Alvarez visited Boston General on 3 May”, a good NER model returns a person, an organisation or place, and a date. It is the first step in a surprising number of pipelines: extracting facts, redacting names, linking records to a catalogue, counting mentions, routing documents.

This note explains how to choose between a general model and a fine-tuned one. We use clinical text as the worked example because it shows the gap clearly, but the same reasoning applies to contracts, invoices, support tickets, legal filings or part numbers.

A model only knows the labels it was trained on

A NER model predicts a fixed set of entity types. Many popular general-purpose models are trained on news text and know four of them: person, organisation, location and miscellaneous (the standard CoNLL-2003 data set). They are very good at those, and they say nothing about anything else.

That silence matters. A missing entity does not look like an error. The model simply returns fewer results, and unless you count what should have been found, you will not notice.

Worked example: two models, two clinical notes

Our Playground has two NER demos that run in your browser. We ran each on its own sample note.

The general model, on this note:

Patient is a 58-year-old male, John Carter, DOB 03/14/1967, presenting with chest pain radiating to the left arm and shortness of breath. History of hypertension and type 2 diabetes, currently on metformin and lisinopril. Seen by Dr. Angela Reyes in the cardiology department at Meridian General Hospital on April 2, 2026. Contact number 555-014-2231.

FoundType it gave
John CarterPerson
Angela ReyesPerson
Meridian General HospitalLocation

It found the two people and the hospital, though it filed the hospital as a location, where most people would say organisation. It found none of the medical content: no hypertension, no type 2 diabetes, no metformin or lisinopril. It also left the birth date, the visit date and the phone number untouched.

The biomedical model, a small model fine-tuned to recognise diseases, on a second note:

Patient presents with a known history of type 2 diabetes mellitus, chronic obstructive pulmonary disease, and a prior myocardial infarction. No evidence of pneumonia or acute kidney injury on this visit. Continues to be monitored for early-stage chronic kidney disease.

It returned six diseases: diabetes mellitus, chronic obstructive pulmonary disease, myocardial infarction, pneumonia, acute kidney injury and chronic kidney disease. That is the coverage the general model lacked.

It also shows what NER alone does not do. Two of the six, pneumonia and acute kidney injury, are conditions the note says the patient does not have. The model reported them with confidence scores of 93.7% and 95.1%. And its span for the first disease was “diabetes mellitus”, not “type 2 diabetes mellitus”, because the model has to decide where an entity starts and stops.

Finding an entity is half the job

For most real uses you need more than a list of spans:

  • Assertion and negation. Is the condition present, absent, in the past, hypothetical, or someone else’s? In risk adjustment coding, for example, only current, supported conditions count, and “history of”, “rule out” and “denies” do not. A separate rules layer or a second model has to decide this.
  • Normalisation. Turning the text “chronic kidney disease” into a code from a catalogue, so downstream systems can use it.
  • Boundaries. Whether “type 2” belongs inside the span is a decision you have to make and write down.
  • Thresholds. Our demo ignores predictions below 60% confidence. Where you cut changes what you miss and what you wrongly include.

In our HCC coding pipeline, NER is one step, a compact biomedical model that marks every disease mention. Rules for negation and history, a code catalogue and a decision model do the rest, and we measure the whole pipeline against professional coders. You can read how that fits together in the case study.

When a general model is enough, and when to fine-tune

A general model is usually enough when the things you need are the standard ones (people, organisations, places, dates) and your text looks like the news or general web writing.

Fine-tune a small model when:

  • Your entities are specific to a field: conditions, drugs, contract clauses, product codes
  • The general model’s errors are systematic, such as always missing one type
  • You need it to run privately or cheaply, on a CPU or inside your own network
  • You can get enough labelled examples to train and, more importantly, to test: a held-out set you trust matters more than the model

How we would decide

  1. Write down the entity types you need, with a definition and examples for each, including the awkward cases like the “type 2” boundary.
  2. Run a general model on a few dozen real texts and look at what it missed, not only what it got wrong.
  3. Label a held-out set and measure precision and recall for each type, with a clear rule for what counts as a match: the exact span, or any overlap.
  4. Fine-tune only if the gap matters, and compare against the same held-out set every time you change something.
  5. Keep a person on the edge cases and feed their corrections back in.

Pitfalls

  • Labels drift. Two annotators will not agree on every boundary unless the guidelines settle it.
  • Domain shift. A model fine-tuned on one hospital’s notes can stumble on another’s.
  • Confidence is not correctness. The disease model was most confident about conditions the note rules out.
  • De-identification is harder than extraction. Using NER to remove names and dates from text demands far stricter testing. In the example above, the general model would have left the birth date, the visit date and the phone number in place.

Try it

Run the same note through both models. Our clinical NER demo uses the general model, and our disease detection demo uses the biomedical one. Paste your own text and see what each one misses.

Sources

KEEP READING

Related topics

Use Fine-tuning

Small fine-tuned models vs per-chart LLM calls

For a high-volume, privacy-sensitive coding workflow, a pipeline of small, task-specific models beat repeated LLM calls on cost, speed, privacy and recall. We would not assume that holds for every task.

Try LLMs

Small LLMs on-device: what a 0.5B model can do

A small local language model is a good fit for narrow, text-in text-out jobs where the answer is already in the input. It is a poor source of knowledge, and it needs a person checking its output.

READY TO BUILD WHAT'S NEXT?

Not sure where to start? Start with the problem.

Want this applied to your data? Tell us the problem and we will tell you what we would do.

View All
Contact us

Have a problem worth getting right?

Send us the paragraph you'd normally write to a colleague. We'll get back to you within 24 hours.

hello@tathastha.com