In-context learning and PFNs

The idea in one line: train a network once on millions of made-up tasks and it learns to be a learning algorithm: hand it a new user’s examples as input and it predicts for them in one forward pass, with no training step.

Fine-tuning a model for each customer means a training job, a hyperparameter search, a GPU and a stored model per customer. This lesson’s approach replaces all of that with one frozen network and an input. On the TabArena benchmark, TabPFN-3.5 “trains” (really, reads the rows) in a median 1.84 seconds per 1,000 rows. CatBoost, tuned and ensembled, needs 1,346 seconds: about 700x more (TabArena data, 2026-10-27). In the 2025 Nature paper, TabPFN v2 took 2.8 seconds to beat ensembles tuned for 4 hours, about 5,140x faster (Nature, 2025-01-08). “An ML team per task” becomes “one endpoint”.

The second 10x is about data. These networks are pretrained on synthetic datasets only. As of October 2026 the top model on TabArena, NVIDIA’s Kumo-Tabular, was “pretrained only on artificial data” (NVIDIA, 2026-09-29). The data rights you need move from pretraining to evaluation. The honest catch: the cost moves to prediction time, as you’ll see below.

Recall from three ways to teach a model: in-context learning (ICL) means the model receives example inputs and answers in its input and predicts for new inputs, with its weights frozen. An LLM does it when you paste three labelled emails into a prompt. Why should that work at all?

An analogy first. A doctor who has seen ten thousand patients doesn’t relearn medicine for the next one. They read the symptoms and recognise which kind of case it is. The real mechanism is close to that. Garg, Tsipras, Liang and Valiant trained transformers from scratch on prompts made of (x, f(x)) pairs, drawing a fresh function f for every prompt. For linear functions, the trained network predicted unseen functions about as well as least squares, the textbook optimal method. For sparse linear functions, two-layer neural nets and decision trees, it “matches or exceeds task-specific learning algorithms” (Garg et al., 2022-08-01).

In plain words: train a network on enough problems, not just enough examples, and it learns to be a learning algorithm. Two follow-ups looked inside:

• Akyürek and colleagues proved transformers can implement gradient descent (the step-by-step weight nudging used in training) and ridge regression (least squares with a penalty on large coefficients). Trained in-context learners tracked one or the other depending on depth and noise (arXiv 2211.15661, 2022-11-28).
• von Oswald and colleagues showed one linear self-attention layer can reproduce a gradient-descent step. Trained models behave like “mesa-optimizers”: optimizers running inside the forward pass (arXiv 2212.07677, 2022-12-15).

A second reading says the model is doing Bayes. Recall Bayes’ rule from probability is the product: the posterior is proportional to prior times likelihood.

Xie, Raghunathan, Liang and Ma modelled pretraining text as a mixture of hidden Markov models, each with a hidden “concept”. To predict a coherent document, the model must infer its concept. At test time your in-context examples are evidence for a shared concept, so prediction is implicit Bayesian inference: weigh every concept by how well it explains the examples, then predict. They found ICL emerges even when prompts don’t look like pretraining data, improves with scale at equal pretraining loss, is sensitive to example order, and that zero-shot sometimes beats few-shot (Xie et al., 2021-11-03).

A toy world where every task follows one of two rules, each with prior probability 0.5:

• Rule A: the label is 1 when x > 5.
• Rule B: the label is 1 when x < 5.
• Labels are noisy: each one is flipped with probability 0.1.

Your context holds two examples, (x = 8, label 1) and (x = 2, label 0). Your query is x = 7.

prior        P(A) = 0.5                P(B) = 0.5
likelihood   P(data | A) = 0.9 x 0.9 = 0.81
             P(data | B) = 0.1 x 0.1 = 0.01
posterior    P(A | data) = 0.81 / (0.81 + 0.01) = 0.988
             P(B | data) = 0.01 / 0.82          = 0.012
predict x=7  P(label 1)  = 0.988 x 0.9 + 0.012 x 0.1 = 0.89

Both examples agree with rule A, so A gets almost all the weight. Rule A says x = 7 is a 1 with probability 0.9 (the noise flips the rest), rule B says 0.1, and the weighted average is 0.89.

That 0.89 is a posterior predictive distribution (PPD): the Bayesian answer to “given these examples, how likely is each label for this new input?”, averaged over every hypothesis and weighted by how well it explains the examples. With two rules you can sum by hand. With “every table that every possible causal graph could produce” you can’t: there are far too many hypotheses. The trick in the next section is to train a network to output the 0.89 directly.

The Bayesian story is a lens on what ICL does, not a settled fact. Three results push back:

• ICL can beat the Bayes answer. Raventós, Paul, Chen and Ganguli varied how many distinct regression tasks a transformer saw in pretraining. Below a threshold of task diversity, it behaved like the Bayes estimator whose prior is just those pretraining tasks, and it failed on new ones. Above the threshold, it matched ridge regression, generalised to unseen tasks and outperformed that Bayes estimator (arXiv 2306.15063, 2023-06-26).
• Memory, not size, is the limit. Kirsch and colleagues found the same memorise-then-generalise switch as the number of meta-training tasks grows. The limiting factor was the state (memory) available per prediction, not the parameter count (arXiv 2212.04458, 2022-12-08).
• A frequentist reading. Nagler explains why PFNs get more accurate with more data as variance shrinking, with no guarantee that bias vanishes (ICML 2023, arXiv 2305.11097).

The practical lesson all three share: the diversity of training tasks matters more than model size. That is why tabular foundation model papers spend most of their effort on the data generator.

A prior here is a recipe for generating plausible datasets, written before you see any real data. A prior-data fitted network (PFN) is a transformer trained on datasets sampled from a prior, so that given a context set it outputs the PPD for a query point. Müller and colleagues introduced the recipe in 2021 (PFNs, arXiv 2112.10510). Here it is with a toy prior.

1. Write a prior you can sample from. Toy version: pick a threshold t from 1 to 9, pick rule A (x > t) or rule B (x < t) with a coin flip, and flip each label with probability 0.1. Real versions use Gaussian processes, Bayesian neural networks or structural causal models.

2. Sample one dataset. Draw 10 random x values between 0 and 10 and label them with the rule you picked.

3. Hide some labels. Show the network 8 labelled points as context, plus the other 2 x values as queries.

4. Score its guesses. The network outputs a probability for each hidden label. The loss is cross-entropy, the log loss from Part 1.

5. Repeat with a fresh dataset every step, millions of times. The network never sees the same task twice, so it can’t memorise answers. It has to learn how to infer the rule from the context.

6. Use it. Feed a real user’s labelled rows as context and their unlabelled rows as queries. One forward pass returns probabilities, and no gradient step touches the weights.

Here is the sampler for that toy prior. Everything else is an ordinary transformer and an ordinary training loop.

# Illustrative sketch, not a library API: the toy prior above.
import numpy as np

rng = np.random.default_rng(0)

def sample_task(n=10, noise=0.1):
    t = rng.integers(1, 10)              # threshold, 1..9
    rule_a = rng.random() < 0.5          # A: x > t, B: x < t
    x = rng.uniform(0, 10, size=n)
    y = (x > t) if rule_a else (x < t)
    flip = rng.random(n) < noise         # 10% label noise
    y = np.where(flip, ~y, y).astype(int)
    return x, y

x, y = sample_task()
context, queries = (x[:8], y[:8]), x[8:]   # labels y[8:] are hidden
# Train: model(context, queries) -> P(hidden labels); minimise
# cross-entropy; draw a fresh task at every step.

Why this lands on Bayes. Müller and colleagues showed that this loss equals the expected KL divergence (a measure of how far one distribution is from another) between the true PPD and the network’s output, up to a constant. Minimising it pushes the network toward exact Bayesian prediction under that prior (PFN paper, HTML). For numeric targets, the network outputs a “Riemann” (bar) distribution: buckets of equal prior probability, which with enough buckets can approximate any such density.

Notice what the prior does to the answer. Under the two-rule prior, the right output for x = 7 was 0.89. Under the random-threshold prior in the steps, the same two examples give about 0.72 (my arithmetic, summing all 18 rule-and-threshold pairs). The drop happens because a threshold of 7 also fits both examples, and it labels x = 7 as 0. A well-trained PFN outputs whatever its prior implies. The prior is the model’s worldview, and a prior that doesn’t match the world is a source of error you can’t fix with more parameters.

The payoff is speed. In the original paper, for Gaussian processes with hyper-priors, PFN inference was more than 200x faster than MLE-II and 1,000-8,000x faster than NUTS (a standard sampling method), while approximating the PPD more closely. These are per-dataset times after a one-time training cost (PFN paper). That is amortized inference: pay a large training bill once, so each later Bayesian answer costs one forward pass.

Real tables need a richer prior. A structural causal model (SCM) generates data by sampling a random directed graph of variables, where each variable is a random function of its parents plus noise. One sample might look like this (the functions are made up for illustration):

  X1 = noise
   |
   v
  X2 = tanh(2 * X1) + noise      X3 = noise
   |                              |
   +-------------+----------------+
                 v
  Y  = 1 if X2 + 0.5 * X3 > 0.3, else 0

Sample a new graph, new functions and new noise for every dataset, and you get endless varied tables. TabPFN v2 trained on about 130 million synthetic datasets (2,000,000 steps of 64 datasets each). Its prior used preferential-attachment graphs, small neural nets, trees, discretisation and noise on the edges, plus feature warping, quantisation and random missing values (Nature, 2025-01-08). Kumo-Tabular saw about 35M, 71M and 137M synthetic tables for its Small, Medium and Large sizes (NVIDIA, 2026-09-29).

The training set is the prior. Nothing real is consumed: no scraping, no licences, no privacy review and, in principle, no test-set contamination. The flip side is that the prior is the moat. Prior Labs did not release TabPFN v2’s generator with the Nature paper. TabICLv2 released its generator and pretraining code (arXiv 2602.11139, 2026-02-11).

Real data still helps at the frontier, and the evidence is mixed:

• TabDPT trains on real tables and reports “significantly faster training” and better generalisation (arXiv 2410.18164).
• Prior Labs says Real-TabPFN-2.5, fine-tuned on real data, beats its synthetic-only base (Prior Labs report).
• In time series, Amazon’s synthetic-only Chronos-2-Synth scores within about 2% of the real-data Chronos-2 on the GIFT-Eval benchmark (0.496 versus 0.485, lower is better; the course’s research recomputed both from raw result files).

The message that started this course described Google’s TimesFM-3 this way: “Training happens once, and each new user brings their own history as context.” In this field, zero-shot means no training or fine-tuning on the user’s task; the user still supplies context, such as labelled rows for a table or history for a forecast. Here is what that changes compared with fine-tuning.

ONCE, by the lab                     MANY TIMES, by each user
----------------                     ------------------------
prior sampler                        user's labelled rows or
  |  millions of fake datasets       history (the context)
  v                                    |
transformer training                   v
  |  days on GPUs            +--> one forward pass
  v                          |         |
frozen weights --------------+         v
                                 probabilities for the
                                 user's queries, in
                                 seconds, no training

The left column is paid once, by whoever trains the model. The right column runs for every user, and nothing in it touches the weights.

The same idea at the C4 system-context level (see how to read the diagrams):

C4 · L1 System context · PFN prediction service

+------------------+  sends rows,   +-----------------------+
| [Person]         |  gets odds     | [Software system]     |
| Analyst with a   |--------------->| PFN prediction        |
| small table      |                | service (frozen net)  |
+------------------+                +-----------------------+
                                                ^
                                 ships weights  |
                                 once           |
                                    +-----------------------+
                                    | [External system]     |
                                    | Pretraining run on    |
                                    | a synthetic prior     |
                                    +-----------------------+

The analyst only ever talks to the service. The pretraining run is outside the system: it happened once, somewhere else, and its only output is a weights file.

Where the cost goes instead. The model rereads the whole context for every query unless you cache it. TabArena labels these models “FitsAtPredict” for that reason. In the Nature paper, one prediction took 0.2 seconds on a GPU (0.6 on a CPU) for a 10,000-row, 10-feature table, against 0.0002 seconds for CatBoost: about 1,000x slower. Caching cut a CPU run from 32 seconds to 0.1 (Nature). Prior Labs says TabPFN-3’s cache takes about 7 GiB per estimator at 1M rows, with 8 estimators by default. So the rule of thumb: many small tasks with few predictions each favour ICL; a high-volume scoring path wants a cache, batching, or a small model distilled from the big one (see distill a decision model).

Here is what has been disclosed, with rental prices from RunPod’s pricing page, fetched 2026-10-28: an RTX 4090 at $0.34 per hour (community cloud), and an H100 from $1.99 (PCIe, community cloud) to $3.99 per hour (SXM, secure cloud) (RunPod).

Sources: nanoTabPFN (arXiv 2511.03634, 2025-11-05), which is about 160,000x less compute than v2; the PFN paper, where no PFN trained longer than a few days on one GPU (arXiv 2112.10510); TabPFN v1 (arXiv 2207.01848); TabICLv2 (arXiv 2602.11139). TabICLv2 is the striking one: untuned, it beats the tuned, real-data fine-tuned Real-TabPFN-2.5 on TabArena, so near-frontier pretraining now fits a bootstrapper’s budget.

As of October 2026, eight of the top 12 entries on TabArena are pretrained in-context models, and the other four are systems built around such models. The best tuned and ensembled gradient-boosted tree sits about 600 Elo below the leader (TabArena). Elo is a chess-style rating from pairwise wins, where a 400-point gap means about 10:1 odds, so 600 points is roughly 30:1. The next lesson walks through those models.

PFNs have spread well beyond tables:

• PFNs4BO for Bayesian optimisation (arXiv 2305.17535).
• LC-PFN for learning-curve extrapolation: trained on 10M synthetic curves, more than 10,000x faster than MCMC, with 2-6x early-stopping speedups (arXiv 2310.20447).
• Do-PFN, which predicts the effect of interventions from observational data without being given the causal graph (arXiv 2506.06039).
• ForecastPFN, TabPFN-TS (11M parameters) and TempoPFN (38M, trained only on synthetic generators) for time series.

The open problems:

1. Is it really Bayes? Xie and Müller say approximately yes, under the prior. Raventós shows useful non-Bayesian behaviour, and Nagler offers a frequentist account. Unresolved.
2. Does real data matter? TabDPT says yes; Kumo-Tabular, Mitra-v2 and TabICLv2 reach the top synthetic-only; Chronos-2 versus Chronos-2-Synth suggests real data adds about 2%.
3. Scale. TabArena’s datasets top out around 150,000 rows. Claims of superiority at 1M rows rest on Prior Labs’ internal datasets.
4. Prediction cost. Caches and distillation help, but a forward pass over a million-row context is not free.
5. The next “PFN for X”. The course’s research suggests (its own analysis, not a sourced finding) the pattern needs a domain with a simulator or mechanistic model to sample from, small private data per user, and decisions that need calibrated probabilities. Candidates: manufacturing yield, battery degradation curves, assay dose-response curves, time-to-failure with censoring, and software build outcomes, which the case study picks up.

• Müller et al., prior-data fitted networks (arXiv 2112.10510): the original recipe and the proof that the loss targets the posterior predictive.
• Accurate predictions on small data with a tabular foundation model (Nature, 2025): TabPFN v2, its SCM prior and its honest limits.
• Garg, Tsipras, Liang and Valiant (arXiv 2208.01066): the cleanest experiment showing a transformer learning to be a learning algorithm.
• Raventós et al. (arXiv 2306.15063): the task-diversity threshold where ICL stops looking Bayesian.
• nanoTabPFN (arXiv 2511.03634) and its code: train your own PFN in about a minute.