The big debates

The idea in one line: Almost every argument about specialized models is settled by the same unglamorous test: a fair head-to-head on data the model has never seen, against a tuned simple baseline, counting the cost of the whole workflow.

Researchers judge you by whether you can state the other side’s best case. These debates also hide order-of-magnitude facts. The 65-318x cost gap between a decision model and a frontier LLM at equal accuracy is real (the market map). A forecasting model’s “win” can vanish against a tuned statistical ensemble. A calibration number can swing tenfold between two tasks on the same model. Knowing which claim survives which test is worth more than memorizing any leaderboard.

Each debate below has the same shape: the strongest case for, the strongest case against, the question that settles it, and where the evidence points as of October 2026.

Worked example: “Chronos beats the baselines.” Chronos is Amazon’s open time-series model. Its own paper says it “significantly outperforms” on datasets inside its training corpus, but is only “comparable and occasionally superior” zero-shot on new ones (arXiv 2403.07815). Nixtla’s ensemble of four statistical methods beat Chronos-T5-Large by 10% on error over 50,000+ classic competition series, about 5x faster on a CPU; note that Nixtla is a competitor (Nixtla experiment). Yet on today’s broad GIFT-Eval benchmark, the best single foundation models score an error ratio of about 0.44-0.46 against seasonal naive, where 1.0 means “no better than repeating last season” and AutoARIMA scores 0.91.

All three statements are true. Foundation models dominate short, varied, covariate-rich series; tuned classical ensembles stay competitive on long, clean, single-frequency business series. The lesson: before you argue, ask on which data, against which baseline, after which cut-off date?

For the generalist. Generalists absorb specialists over time. Medprompt, GPT-4 with general prompting, beat the specialist Med-PaLM 2 on all nine MultiMedQA benchmarks (arXiv 2311.16452). Single-pass decision models keep only their base model’s knowledge and fail at counting, arithmetic and date comparison, as TypeSafe’s own documentation admits. In Every’s test, Fable 5.1 caught 7 of 7 planted defects against Jev’s 6. Vals put Jev last of 12 on a LegalBench slice (73.0%) (Vals AI, 2026-10-06). And “specialized” is a thin layer: the decision models are post-trained generalists such as Qwen3.5-9B and Gemma-4-12B.

For the specialist. On closed-set, high-volume decisions, specialists match the frontier within noise at under 2% of the cost (Arize, 2026-10-07). One estimate puts a self-hosted 150M encoder at 350-16,000x cheaper per decision than Sonnet-class LLMs, though only 10-800x cheaper than the cheapest frontier tier. An NVIDIA paper argues small models are “sufficiently powerful, inherently more suitable, and necessarily more economical” for repetitive agent calls (arXiv 2506.02153). Removing the LLM from LLM-based forecasters did not hurt accuracy in 26 of 26 Time-LLM cases, and the ablation had 0.2M parameters against 6.6B (Tan et al., NeurIPS 2024). GPT-4o reaches 74.9% on the MMAD industrial-inspection benchmark, which its authors call “far short of industrial requirements”.

Ask: what do cost and accuracy look like per completed workflow, fallbacks included?

Where it points: a hybrid. A decision model with a threshold handles the bulk, and an LLM takes what it is unsure about, as in Browserbase’s deployment (debate 6).

A baseline is the simple method any new model must beat: seasonal naive (repeat last season), ETS (exponential smoothing), gradient-boosted trees, or a linear model.

The sceptic, time series. A one-layer linear model beat sophisticated Transformer forecasters “in all cases” on nine datasets (Zeng et al., 2022). Chronos’s in-corpus versus new-data gap is the cost of leakage made visible.

The sceptic, single-cell biology. Five foundation models and two other deep models did not beat deliberately simple baselines at predicting the effect of gene perturbations (Ahlmann-Eltze, Huber and Anders, Nature Methods, 2025-08-04). Zero-shot Geneformer and scGPT “often underperform compared to simpler methods” (Kedzierska et al.). In Arc’s 2025 Virtual Cell Challenge (1,200+ teams), a one-person ridge regression on averaged data ranked 38th overall but first on Pearson correlation and, by one participant’s account, beat Arc’s own State model; the scoring clipped negative scores to zero, so one metric “stopped influencing optimization” (GEN, 2025-12-08).

The optimist. Newer benchmarks were built to answer these critiques. GIFT-Eval has 23 datasets, 144k series and a non-leaking pretraining set (arXiv 2410.10393); fev-bench has 100 tasks and bootstrapped confidence intervals (arXiv 2509.26468). There, the best models roughly halve seasonal naive’s probabilistic error and reach a fev-bench skill of about 0.50 against 0.21 for a statistical ensemble. On TabArena the best tuned gradient-boosted tree sits about 600 Elo (a chess-style rating from head-to-head wins) below the leading tabular foundation model (TabArena, 2026-10-27). TabPFN’s result is peer-reviewed in Nature.

Ask: did it beat a tuned baseline, on data generated after its training cut-off, with confidence intervals?

Where it points: forecasting and tabular models now win on broad, varied benchmarks and cold starts. Few-shot is the deployable product: zero-shot TimesFM had 9.25% day-ahead solar error on ERCOT data, 3.71% fine-tuned. Single-cell models have not yet beaten linear baselines; Arc’s 2026 zero-shot challenge on newly generated data is pending. TabPFN, TimesFM and friends and vertical foundation models have the detail.

Calibration, a reminder: when a model says 80% on 100 cases, about 80 should be right (calibration). ECE (expected calibration error) is the average gap between stated confidence and actual accuracy.

No, not by default. Post-training breaks it: GPT-4’s base model had ECE 0.007 on an MMLU subset, and 0.074 after RLHF (reinforcement learning from human feedback), about 10x worse (GPT-4 report). Verbalized confidence is overconfident (Xiong et al.), and RLHF reward models favour confident answers whatever their quality (Leng et al.). Reasoning fine-tuning degraded abstention by 24% on average across 20 models. Even good decision models are fragile: Jev’s ECE was 0.011 on one Vals task and 0.108 on another, and Liquid d1’s ranking quality (ROC AUC, where 0.5 is a coin flip) flipped to 0.17 when Arize reworded a question.

Yes, by objective. Larger pre-trained models are already well calibrated on multiple choice in the right format (Kadavath et al.), and one temperature (T = 2.5) fixed Anthropic’s 52B RLHF model. A proper scoring rule (log loss or Brier, which reward honest probabilities) in the RL reward took Qwen2.5-7B on HotpotQA from ECE 0.37 to 0.03 at the same accuracy (RLCR, Damani et al.). Decision models train directly on Brier loss and then RL with calibration rewards, and Arize measured 0.023-0.065 across hosted and open models.

Ask: calibrated on whose data, under which wording, measured how? A perfectly calibrated model scores ECE of about 0.03 on 1,000 examples from noise alone.

Where it points: calibration can be trained into a decision head. It cannot be assumed when the data or the wording shifts: RLCR’s out-of-domain ECE was 0.21-0.25 against 0.03 in domain. Recalibrate on the customer’s data, as decider-2b does with per-type temperatures (2.47 for Choice, 2.83 for Score, 3.19 for Noul).

Model collapse is the decay that happens when models train on model outputs.

The risk. Training each generation on the previous one’s outputs makes the tails of the distribution vanish (Shumailov et al., Nature, 2024). The sharpest dissent: “strong model collapse” finds even 1% synthetic data can cause collapse in a supervised regression setting (Dohmatob et al.).

The counter. Collapse comes from replacing real data. Accumulating real plus synthetic keeps test error bounded (Gerstgrasser et al.). The strongest specialists are synthetic-first. NVIDIA’s Kumo-Tabular, trained only on synthetic tables, is #1 on TabArena. Amazon’s synthetic-only Chronos-2-Synth lands within about 2% of the real-data Chronos-2 on GIFT-Eval (0.496 against 0.485). And a designed prior or physics simulator is not a model eating its own outputs, provided a verifier checks it.

Ask: does the synthetic data replace real data or add to it, and what checks it?

Where it points: avoidable. Real data is the anchor, synthetic data is augmentation, and evaluation stays real and time-split (labels from AI and from the world). Open: whether real data still adds at the frontier, where TabDPT says yes and synthetic-only leaders say barely.

Chinchilla (2022) found that, for a fixed training budget, you should double the training tokens whenever you double the parameters; Chinchilla at 70B beat Gopher at 280B on the same compute (Hoffmann et al.). Over-training means training a smaller model on far more tokens than that.

Worked example. Llama 3 8B’s Chinchilla point was about 200B tokens, or 25 tokens per parameter. Meta trained it on 15T, about 75x more, roughly 1,900 tokens per parameter, and saw log-linear gains (Meta, 2024-04-18).

For over-training small. If you expect heavy inference, train smaller and longer; quality keeps improving up to 10,000 tokens per parameter (Sardana et al., ICML 2024). Over-trained models scale predictably: a 1.4B model on 900B tokens was predicted from about 300x less compute (Gadre et al.). In verticals, medium models capture most of the value: ESM C 300M matches ESM-2 650M (a vendor claim), and 30-60M-parameter materials potentials equal a 730M one on Matbench.

The limit. A small model only knows what its base learned. On generic zero-shot classification, GPT-4o averages 0.84 against 0.72 for the 205M GLiNER2 (GLiNER2). Size still buys knowledge.

Ask: how many tokens will this model read in its life, and what is the smallest model that holds accuracy on the task?

Where it points: over-training small models is standard practice. The densing law says capacity per parameter doubles about every three months (Xiao et al.), so a fixed-quality model shrinks about 2x a quarter, and any moat built on “we are cheaper” decays on that clock.

The names come from Kahneman: System 1 is fast, intuitive judgment; System 2 is slow, deliberate reasoning. Jev brands itself a “System One model”.

For fast decisions. Reasoning models overthink easy problems (Chen et al.), and standard models beat reasoning models on low-complexity tasks at equal compute (Shojaee et al., Apple). Jev’s latency barely moves from 1 to 64 questions per request (0.10 s to 0.12 s, per Vals).

For reasoning. The Apple paper’s “collapse” may be a test artefact of token limits and unsolvable puzzles (Lawsen). Reinforcement learning with verifiable rewards (RLVR, named in Tulu 3) is what made reasoning models strong. On JevBench v1.3.0’s hard tier, GPT-5.6 Luna scored 94.5% against Jev’s 74.1%, even though Jev led the composite score (74.4 against 65.9).

The synthesis is routing by difficulty. How to read the diagrams explains the boxes.

C4 · L1 System context · Difficulty router

+-------------------+                +---------------------+
| [Person]          |  asks a typed  | [Software system]   |
| App developer     |--------------->| Decision router     |
| acts on answers   |<---------------| System 1 model plus |
+-------------------+  answer + odds | a threshold         |
                                     +---------------------+
                                       |                |
                        escalates if   |                | checks real
                        p is below     |                | outcomes to
                        threshold      v                v recalibrate
                            +-------------------+ +-------------------+
                            | [External system] | | [External system] |
                            | System 2 LLM      | | Verifier          |
                            | reasons slowly    | | tests, builds     |
                            +-------------------+ +-------------------+

The fast model answers with a probability; cases below the threshold go to a slow reasoning model; a verifier scores real outcomes so the threshold stays honest. The probability is what makes the routing possible.

Ask: what share of cases clears the threshold, and what happens to the rest?

Where it points: the hybrid wins. Browserbase’s 0.7-threshold design cut median latency from 1.97 s to 0.46 s. In the course’s distro case, a decision model alone gives about 1.2-2x, and 2.5-2.9x only inside a loop with a coding agent and a faster build oracle (estimates in is 5x real?).

For closed. Jev, the leader on calibration and latency in Arize’s test, is closed. Closed weights protect the RL recipe, and the frontier’s lead over open models was about a year (5-22 months) as of 2024 (Epoch AI).

For open. Within four weeks, open models (Clef, Decider, d1-3B, mammoth-1, Cygnet) were at or above Jev on several leaderboards. On the Decision Index data of 2026-10-28, 47 entries outscore Jev 1.14, including a stock Gemma-4-31B with an inference trick and no training (Decision Index). Open weights are also the only route into air-gapped buyers.

The middle. Non-commercial weights (TabPFN, TimesFM-3) take the community benefit and keep the revenue. And “open weight” is not “open source” when training data and code stay private, as BenchLM notes for Cygnet.

Ask: open under which licence? Can I deploy it, fine-tune it and sell decisions made with it?

Where it points: in decision models, open caught up in weeks, not a year. The live question is the licence, not the weights (data rights).

Contamination (or leakage) means test items, or data close to them, were in the training set, so “zero-shot” is partly memory.

The sceptic. GPT-4 was shown to be contaminated with AG News, WNLI and XSum (Golchin and Surdeanu, ICLR 2024). GIFT-Eval flags Chronos, Chronos-Bolt, TimesFM 1 and 2, TTM-R2 and Lag-Llama for leakage. On TabArena, issue #648 shows LimiX-2 scoring an error (1 − AUC) of 0.0066 on one dataset where others score 0.15 or more, rising to 0.1536 when the column order was reversed, which suggests memorized labels (tabarena#648); the issue is open. Protein co-folding accuracy tracks similarity to training data. Vendor test sets are unaudited: Microsoft says its 150,000 questions were “kept blind from training”, and TypeSafe’s reference labels came from competitor models.

The constructive side. Leakage-free designs are spreading: GIFT-Eval’s non-leaking pretraining split, Toto 2.0 with no public forecasting data in pretraining, Decision Index 0.4 scoring on private tests, Arc’s 2026 challenge on newly generated data, TabPFN trained only on synthetic data. The remaining risk is subtler: a synthetic prior tuned over many iterations to a benchmark suite overfits by design.

Ask: show me results on data created after your training cut-off, or on my own data.

Where it points: leakage is systemic but now measured. Trust leak-flagged boards and post-cut-off tests, and treat the top of any public board as “best system so far”, not “best model” (verifiable rewards and evals).

• Ahlmann-Eltze, Huber and Anders, Nature Methods — the cleanest case of deep models failing to beat linear baselines.
• Damani et al., RLCR — how a proper scoring rule in the reward restores calibration.
• Gerstgrasser et al., Is Model Collapse Inevitable? — accumulate, don’t replace.
• Sardana et al., Beyond Chinchilla-Optimal — why inference-heavy products should over-train small models.
• GIFT-Eval — a benchmark built with a non-leaking pretraining set.
• Arize: decision model benchmark — the rewording flip test that every calibration claim should face.