Fine-tuning BERT: labelled data, not instructions

The data is labels, not conversations

This is the whole difference and it trips people coming from LLM work. An encoder model does not generate text, so there is no prompt, no completion, no chat template, no loss masking and no stop token. There is a text input and a label.

For classification: text plus a class index. For token classification such as named entity recognition: tokens plus a label per token. For sentence-pair tasks: two texts plus a relationship label. For extractive question answering: a passage, a question, and character offsets of the answer span.

That makes the data much cheaper to produce and much easier to validate. A label is either right or wrong; there is no judging whether a generated answer was good. It also makes evaluation trivially decidable, which is the property our golden eval set guide spends most of its time trying to engineer for generative tasks.

# sequence classification
{"text": "The delivery arrived three days late.", "label": 2}

# token classification (NER) - one label per token, aligned
{"tokens": ["Acme", "Ltd", "filed", "in", "Delaware"],
 "ner_tags": [3, 4, 0, 0, 5]}

# sentence pair
{"text_a": "The policy covers flood damage.",
 "text_b": "Flooding is excluded.",
 "label": 2}                      # MNLI convention: 0 entailment, 1 neutral, 2 contradiction

# extractive QA - offsets, not generated text
{"context": "...", "question": "When was it filed?",
 "answers": {"text": ["March 2024"], "answer_start": [412]}}

Why an encoder is often still the right choice

For a fixed classification or extraction task, an encoder model frequently beats a fine-tuned generative model on the metrics that matter, and beats it decisively on the ones nobody puts in the paper.

Latency and cost. A base-sized encoder is a fraction of the size of even a small generative model and produces a label in a single forward pass with no decoding loop. At volume, the cost difference is orders of magnitude.

Constrained output. It cannot produce an invalid label. A generative model asked to classify can return something off-schema, and defending against that is work you do not have to do here.

Calibrated scores. You get a probability per class, which you can threshold, rank and route on. Getting trustworthy confidence out of a generative model is substantially harder.

The case for a generative model is when you need the output to be text, when the task benefits from reasoning before answering, or when you want one model doing many tasks rather than a classifier per task. Those are real reasons. “It is what everyone uses now” is not.

How much data, and the label-quality question

Encoders need labelled examples per class, and the useful rule of thumb is low hundreds per class for a workable classifier on a reasonably distinct set of categories, scaling with how subtle the distinctions are.

Class balance matters more here than in instruction tuning. A classifier trained on heavily imbalanced data learns the prior and predicts the majority class — so check your per-class metrics rather than overall accuracy, which can look excellent while the model never predicts your minority classes at all.

Label quality is the ceiling. Two annotators disagreeing on your taxonomy means the model cannot do better than that disagreement rate, and the usual cause is an underspecified label definition rather than careless annotators. Measure inter-annotator agreement on a sample before scaling up a labelling effort; it is the cheapest way to discover that your categories need rewriting.

Practical notes

Encoder fine-tuning is full fine-tuning by default and is cheap enough that adapters are usually unnecessary — a base-sized model fine-tunes comfortably on a single modest GPU.

Typical settings differ from LLM conventions: learning rates in the 2e-5 to 5e-5 range rather than 2e-4, and two to four epochs. Maximum sequence length is a hard architectural limit rather than a budget choice, which matters if your documents are long — you will need to truncate, chunk and aggregate, or choose a long-context encoder variant.

Token classification has one specific pitfall: subword tokenisation splits words, so your word-level labels must be aligned to subword tokens, conventionally by labelling the first subword and masking the rest. Getting this wrong degrades results without erroring, which is the familiar pattern — check a decoded, aligned example before training.

Questions people ask