Three ways to teach a model

The idea in one line: You can teach a model by training it from scratch on broad data (pre-training), by training an existing model a little more on your examples (fine-tuning, cheaply with LoRA), or by putting examples in its input and changing nothing (in-context learning); which one is the 10x choice depends on how many labels you have, how often you will call the model and how unusual your task is.

The three ways cost wildly different amounts, and they pay off in different places:

• Pre-training TabPFN v2 took about two weeks on a node of eight GPUs and about 130 million synthetic datasets (Hollmann et al., 2025-01-08).
• Fine-tuning a few-example text classifier with SetFit took 30 seconds on one GPU for $0.025 (Hugging Face, 2022-09-26).
• In-context learning costs nothing to set up, then charges you on every call. On one news-topic dataset, adding examples to a GPT-4o prompt cost $903.50 per million requests against $276 without them, and was still less accurate than a fine-tuned RoBERTa at $10 (Valdes Gonzalez, 2026-02-06).

That last line has a 90-fold cost gap with the accuracy going the other way. Choosing well between these three is one of the cheapest 10x wins in the field.

Recall from what a model is that a model’s behaviour lives in its parameters, its dials. The three ways differ in which dials move, if any:

Where the new knowledge goes

  Pre-training   broad data ------> every dial, starting from random
  Fine-tuning    your labels -----> every dial, nudged a little
    with LoRA    your labels -----> small add-on dials only;
                                    the original dials stay frozen
  In-context     your examples ---> the input only;
                                    no dial moves at all

Pre-training is the first, big training run: start from random dials and train on broad data until the model has learned general patterns. For text, the usual task is predicting the next or a hidden word, so no human labels are needed; the text labels itself.

Its cost follows the rule from the earlier lessons: about 6 × N × D FLOPs for a model with N parameters trained on D tokens (Kaplan et al., 2020-01-23). The range across real models is enormous:

(All from the in-context models research notes, which cite each paper.)

A twist that matters for bootstrappers: pre-training does not have to use real data. TabPFN learns from synthetic tables drawn from a “prior”, a recipe for random but realistic datasets. The current leader on the TabArena table benchmark, NVIDIA’s Kumo-Tabular, was pre-trained only on synthetic tables (NVIDIA, 2026-09-29). The research notes draw the consequence: the “rights to the data” problem applies to evaluation and later training, not to pre-training, if you can write a good generator.

Fine-tuning continues training a pre-trained model on your own labelled examples, with a small learning rate, so it keeps what it knew and adds your task. BERT made this standard in 2018: its abstract says it “can be fine-tuned with just one additional output layer” (Devlin et al., 2018-10-11).

How many labels do you need? Fewer than people think:

• On four social-science classification tasks, a fine-tuned RoBERTa-large passed a zero-shot classifier after about 200 training examples, with a “sweet spot” at 200-500 (Bucher & Martini, 2024-06-12).
• On one of those tasks, German anger detection, GPT-4 zero-shot scored 0.20 accuracy, below the 0.71 you would get by always guessing the most common class. Fine-tuned models scored 0.87-0.89 (same source). Frontier models can collapse on label schemes that are unusual or specific to one group of people.
• A RoBERTa-base model fine-tuned on 1,811 sentences labelled by an open LLM reached 94% accuracy, matching GPT-4. Fine-tuning cost under $1 and took 15 minutes. Running it cost about $2.70 per million sentences against $3,061 for GPT-4 (Laurer, Hugging Face, 2024-02-16).

That last pattern, labelling with a big model and fine-tuning a small one on its labels, is called distillation. It is the backbone of distill a decision model.

Full fine-tuning moves every dial, so you need memory for the model plus its training state, and you store a full copy per task. LoRA (low-rank adaptation) freezes the original dials and learns a small correction next to each big weight matrix (Hu et al., 2021-06-17).

The trick is that the correction is stored as two thin matrices multiplied together. A tiny example: instead of learning a 4 × 4 update (16 numbers), learn a 4 × 1 column and a 1 × 4 row (8 numbers) and multiply them:

  [1]                       [1  0  2  1]
  [2]   ×  [1  0  2  1]  =  [2  0  4  2]
  [3]                       [3  0  6  3]
  [4]                       [4  0  8  4]

  4 + 4 = 8 numbers learned   16 numbers in the update

The width of the thin matrices is the rank (1 here). Real layers are much wider, so the saving is much bigger:

# LoRA in miniature: a low-rank update stores far fewer numbers.
import numpy as np

d, r = 1024, 8                       # layer width and LoRA rank
full_update = d * d                  # every dial in one weight matrix
lora_update = d * r + r * d          # B (d x r) plus A (r x d)
print(full_update, lora_update, full_update // lora_update)
# 1048576 16384 64

# The 4 x 4 example from the text: 8 numbers make 16.
B = np.array([[1], [2], [3], [4]])   # 4 x 1
A = np.array([[1, 0, 2, 1]])         # 1 x 4
print(B @ A)                         # the update added to the frozen W

The LoRA paper reports that, compared with fully fine-tuning GPT-3 175B, LoRA can cut the trainable parameters “by 10,000 times” and GPU memory “by 3 times”, while matching or beating full fine-tuning on the models it tested and adding no inference delay, because the correction can be merged back into the weights. QLoRA went further by storing the frozen model in 4 bits: it fine-tuned a 65B-parameter model on a single 48 GB GPU, and its best model trained in about 24 hours on one GPU (Dettmers et al., 2023-05-23).

LoRA is how most open decision models are built in 2026, per the landscape research notes:

• decider-2b, one person’s open Jev replica: Qwen3.5-2B-Base plus a rank-64 LoRA, trained on 39.5K text and 18K image rows, on one B300 GPU in 56 minutes. It drew about 95,000 downloads in a month (HF model card, read 2026-10-28).
• Cloudflare’s Clef: a frozen backbone plus a rank-256 LoRA and a routing head, trained with label-smoothed cross-entropy and a Brier term on synthetic data, then reinforcement learning.

In-context learning means giving the model examples, or the user’s own data, as part of its input, and letting it work out the task with no change to its dials. Two words you will hear constantly:

• Zero-shot: the model handles a task with no examples of it, only an instruction or a list of label names.
• Few-shot: the same, with a handful of worked examples in the input.

GPT-3 (175B parameters, 2020) made this famous. Its abstract says it was “applied without any gradient updates or fine-tuning”, with “tasks and few-shot demonstrations specified purely via text” (Brown et al., 2020-05-28).

One warning about words. In the table and time-series world, “zero-shot” means no training on your task, but you still supply context: labelled rows for TabPFN, the history for TimesFM-3. Jev is zero-shot in the language sense: you send a state and a typed question, and “the same weights serve every account” (TypeSafe docs: Models, read 2026-10-28).

This is a real research question, and the answers are worth knowing (from the in-context models research notes):

1. It learns to be a learning algorithm. Garg and colleagues trained transformers on prompts made of input-output pairs from a fresh random function each time. For linear functions, the trained model predicted about as well as the textbook best method, least squares, on functions it had never seen (Garg et al., 2022-08-01). Train on enough problems, not just enough examples, and the network becomes a solver.
2. It is doing Bayesian inference, roughly. One theory treats the examples in a prompt as evidence about a hidden “concept”, which the model weighs much as Bayes’ rule weighs evidence in probability is the product (Xie et al., 2021-11-03).
3. You can build that on purpose. A prior-data fitted network (PFN) is trained on millions of datasets sampled from a prior, to predict held-out points from the rest. Its training loss equals, up to a constant, the gap to the exact Bayesian answer under that prior, so the trained network does approximate Bayes in one forward pass (Müller et al., 2021-12-20). That is TabPFN’s recipe.

The dissent is worth teaching too. Below a certain number of distinct training tasks, transformers just memorize those tasks; above it, they generalize and can even beat the “Bayes with the training prior” answer (Raventós et al., 2023-06-26). The lesson the notes draw: variety in the training tasks matters more than model size. The full story is in in-context learning and PFNs.

• Prompts are fragile. Changing the prompt format and the order of examples moved few-shot GPT-3 accuracy “from near chance to near state-of-the-art” (Zhao et al., 2021-02-19).
• The cost moves to every call. The model rereads the examples each time, unless the provider caches them. On TabArena, TabPFN-3.5 “trains” (one forward pass over your table) in 1.84 s per 1,000 rows, against 417-1,346 s for tuned LightGBM or CatBoost; but it predicts about 19 times slower than default CatBoost (in-context models research notes, from the TabArena results, 2026-10-27).

Sources: GLiNER2, 2025-07-24 for the first two rows (GLiNER2’s own paper); Valdes Gonzalez, 2026-02-06 for the rest.

Read it as three facts. Zero-shot, a frontier model beats a small one by about 10 points on general tasks. Few-shot prompting roughly triples the cost without closing the gap to fine-tuning. A fine-tuned small model wins on accuracy and cost once you have a few hundred labels.

The last row is how Jev-like models are made. The research notes trace a clear lesson through FLAN (60+ datasets), Super-NaturalInstructions (1,616 tasks), Flan (1.8K tasks) and Laurer’s universal classifier (33 datasets, 389 classes, +9.4% zero-shot): a model generalizes to new questions in proportion to how many distinct question types it saw in training, not how many examples of each (Laurer et al., 2023-12-29; Wei et al., 2021-09-03). Breadth of tasks is the moat; you will see it again in designing the questions.

And it is fine to combine them. TypeSafe’s own cookbook trains a classical CatBoost model on top of Jev’s probabilities (in-context use first, then a small trained layer for one task), per the Jev research notes.

• A fourth way: reinforcement learning after training. Instead of showing correct answers, you reward good ones. RLVR (reinforcement learning with verifiable rewards, named in Tulu 3, 2024-11-22) rewards answers a program can check, and it is what made today’s reasoning models strong. It has a cost: reward only right or wrong and models learn to guess. Reinforcement fine-tuning cut refusal rates by more than 80% in one study (Song et al., 2025-05-20). Adding a Brier term to the reward fixes calibration at no accuracy cost (Damani et al., 2025-07-22). This is the subject of verifiable rewards and evals.
• An expert caveat. For a model that outputs a whole distribution in one pass, the decision-architectures notes argue (as their own analysis) that RL with a proper-scoring-rule reward on real outcomes has the same optimum as plain supervised training on those outcomes. RL earns its keep when something is sampled, such as a reasoning trace or a tool call.
• Is zero-shot really zero-shot? If a benchmark leaked into pre-training, a “zero-shot” score is partly memory. Synthetic-only pre-training (TabPFN, Kumo-Tabular) is one defence; testing on data made after the training cut-off is another.
• Open: the research notes found no published model that combines TabPFN-style labelled rows in context with natural-language questions for typed decisions. A “language-conditioned PFN” is an open direction.

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021): the method behind almost every cheap fine-tune.
• Language Models are Few-Shot Learners (Brown et al., 2020): GPT-3 and the zero-, one- and few-shot vocabulary.
• What Can Transformers Learn In-Context? (Garg et al., 2022): the clearest experiment showing a model learning to learn.
• Transformers Can Do Bayesian Inference (Müller et al., 2021): the PFN idea that TabPFN is built on.
• Fine-Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models (Bucher & Martini, 2024): the learning curves behind “a few hundred labels”.
• QLoRA (Dettmers et al., 2023): fine-tuning a 65B model on one 48 GB GPU.