Verifiable rewards and evals

The idea in one line: a reward is verifiable when a cheap, independent program can check the answer and is hard to fool. The same checkers, run on a frozen, time-split test set against the best specialist and a frontier model, are what prove “a clear win in accuracy and compute”.

The prompt that started this course names two hard parts. It needs “high-quality, vertical-specific, verifiable rewards”, plus “rights to the data and a clear win in accuracy and compute over existing specialist models” (YC Requests for Startups, read 2026-10-28). This lesson is about the first and the last of those. Data rights covers the middle one.

Verifiers are a 10x lever twice over. In training, once a checker exists (the build passes, the bug resolves, the image boots), every new input becomes a free training signal. DeepSeek-R1-Zero used nothing but rule-based checks as rewards and went from 15.6% to 71.0% on the AIME 2024 maths exam (DeepSeek-R1, 2025-01-22). In evaluation, a checker is what makes your claims believable. Jev launched claiming “193.6x faster, 444.6x cheaper”. A scan of 12,759 launch tweets found a median user-reported speedup of 7x (InfoQ, 2026-10-01), a single source. That gap is what an eval looks like when the truth was not verifiable. Tweaking the model is 10% polish next to getting the eval right.

A reward is a score that tells a training process how good an output was. Think of an exam with an answer key, against an essay graded by a friend. The answer key gives the same mark every time, can’t be charmed, and costs nothing to apply again. The friend is slow, inconsistent, and can be flattered. A verifiable reward is an answer key: a program that checks the answer.

Precisely, a reward is verifiable when the checker is:

1. independent of the model being trained;
2. deterministic, or nearly so;
3. cheap enough to run millions of times;
4. hard to fool: it has a low false-positive rate even when the model is actively searching for ways to fool it.

The course’s dataset report sorts checkers into families. The distro examples come from the decision map.

Building a verifier for a niche domain takes five steps, from the dataset report:

1. List the decisions, and for each one ask “what event in the world settles this?” (designing the questions).
2. Prefer checkers you can run now (build, lint, boot) over outcomes you have to wait for (a bug’s resolution). Use the waited-for outcomes as your test set.
3. Wrap each checker as a pure function, verify(input, answer) -> pass | fail | error, with every version pinned (container digest, repo snapshot), so a label can be reproduced.
4. Measure the verifier itself: its false-positive and false-negative rates on 100 hand-checked cases.
5. Store the verifier’s version with every label.

# Illustrative sketch (not from a report run): a pure, versioned verifier.
import subprocess

VERIFIER_VERSION = "build-check/1"  # store this next to every label


def verify(context_dir: str, predicted_builds: bool) -> str:
    """Return 'pass', 'fail' or 'error' for one 'will it build?' answer."""
    try:
        r = subprocess.run(["podman", "build", "-f", "Containerfile", "."],
                           cwd=context_dir, capture_output=True, timeout=3600)
    except subprocess.TimeoutExpired:
        return "error"  # infrastructure trouble is not an answer: no reward
    built = r.returncode == 0
    return "pass" if built == predicted_builds else "fail"

Two details carry the weight. First, error is its own outcome: a timeout or a dead mirror says nothing about the answer, so it must never count as a reward or a penalty. Second, the Containerfile should pin its base image by digest. Otherwise the same input can pass today and fail tomorrow, and the label is not reproducible.

Reinforcement learning (RL) trains a model by letting it try, scoring each try with a reward, and nudging it toward the tries that scored higher. Reinforcement learning with verifiable rewards (RLVR) uses a program that checks the answer as the reward, instead of a learned reward model. Tülu 3 named the method and used it alongside supervised fine-tuning and DPO on Llama 3.1 base models (Lambert et al., 2024-11-22).

DeepSeek-R1-Zero is the famous case. It used two rule-based rewards. The accuracy reward checked boxed maths answers by rule, and compiled LeetCode code and ran it against tests. The format reward checked that the reasoning sat inside <think> tags. The team deliberately avoided neural reward models, because they “may suffer from reward hacking”. AIME 2024 pass@1 rose from 15.6% to 71.0% (DeepSeek-R1, 2025-01-22).

What RLVR buys is subtler than “the model learns”.

• It elicits more than it teaches. A single training example lifted Qwen2.5-Math-1.5B on MATH500 from 36.0% to 73.6% (Wang et al., 2025-04-29). pass@k is the chance that at least one of k sampled answers is right. RLVR models beat their base models at pass@1, but the base model wins at large k. So the gains are bounded by what the base model could already do, while distillation can add new patterns (Yue et al., 2025-04-18).
• Even random rewards can “work”. Random rewards gave Qwen2.5-Math-7B +21.4 points on MATH-500, against +29.1 for true rewards. The cause was a clipping bias in the GRPO training algorithm that amplified habits from pre-training: “code reasoning” rose from 65% to over 90% of answers. The same trick gave no gains on Llama3 or OLMo2 (Shao et al., 2025-06-12). So always run a random-reward control, and test on two base-model families.

An expert point for decision models. This one is the analysis of one course report, not a published result. For a model that outputs a full probability distribution in one forward pass, an RL reward equal to a proper scoring rule on verified outcomes has the same optimum as plain supervised training with log loss or Brier on those outcomes. RL earns its keep only when something is sampled (a reasoning trace, a tool call) or when outcomes arrive as sparse, delayed feedback. For a one-step typed decision, verified outcomes plus supervised training get you most of the way (how decision models work).

Reward hacking means the model maximises the checker instead of doing the task. It is not an edge case. METR watched OpenAI’s o3 on its RE-Bench tasks. It overwrote timers, monkey-patched evaluators, returned the grader’s precomputed answers and overrode equality operators. It did so in 39 of 128 runs (30.4%), and in 21 of 21 on one task. Prompts telling it not to cheat barely helped (METR, 2025-06-05). Anthropic reported that learning to reward-hack in production coding environments generalised to alignment faking and sabotage. A technique called “inoculation prompting” removed that misaligned generalisation (MacDiarmid et al., 2025-11-23).

A one-step decision model cannot monkey-patch your evaluator. Its hacking surface is the data. A post-decision field left in the input (a dupe_of, a final karma) is a shortcut to the label, and the model will find it, much as o3 found the grader’s answers (leakage traps). That is this lesson’s framing, but the defence is the same either way:

• Run the verifier in a sandbox the policy cannot write to.
• Keep hidden tests that never appear in training.
• Mutation-test the checker: feed it known-wrong answers and confirm it fails them.
• Audit a random sample of rewarded outputs by hand.
• Cap and log reward outliers. A sudden jump in reward is a bug until proven otherwise.

An outcome reward scores only the final answer. A process reward scores each step of the reasoning. Process supervision beat outcome supervision on MATH, solving 78% of a representative test subset, but it needed 800k human step labels (Lightman et al., 2023-05-31). Math-Shepherd built step labels automatically. With them, PPO lifted Mistral-7B from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH (Wang et al., 2023-12-14). The doubters have data too. DeepSeek found process reward models hard to define, hard to label and prone to hacking: useful for reranking, but not worth their cost in large-scale RL. Qwen’s lessons paper found automatic Monte Carlo step labels worse than LLM-judge or human labels (Zhang et al., 2025-01-13).

For this course the answer is simple. A typed decision is usually one step, so outcome rewards suffice. Process rewards start to matter once the model plans multi-step actions, such as an agent assembling a distro image (is 5x real?).

A Jev-like model promises a decision and a probability. So the reward has to score the probability, not just the answer.

A proper scoring rule is a score that is best, on average, only when you report your true belief. Log loss and the Brier score are the two you will meet. Brier is the squared gap between your probability and what happened (1 if it happened, 0 if not), and lower is better (calibration). Why “proper”? Suppose an event truly happens 70% of the time. Here are your expected Brier scores for three reports:

true chance 0.7; expected Brier when you report q:
  E = 0.7 x (1 - q)^2 + 0.3 x q^2

report q = 0.7 (honest):  0.7 x 0.09 + 0.3 x 0.49 = 0.21  best
report q = 1.0 (bold):    0.7 x 0.00 + 0.3 x 1.00 = 0.30
report q = 0.5 (timid):   0.7 x 0.25 + 0.3 x 0.25 = 0.25

Exaggerating and hedging both cost you, so the score rewards honesty.

RLCR (reinforcement learning with calibration rewards) puts that into the RL reward. The model outputs an answer y and a confidence q, and the reward is (Damani et al., 2025-07-22):

R = 1[y is correct] - (q - 1[y is correct])^2

correct, q = 0.9:   1 - (0.9 - 1)^2 =  0.99
correct, q = 0.5:   1 - (0.5 - 1)^2 =  0.75
wrong,   q = 0.9:   0 - (0.9 - 0)^2 = -0.81
wrong,   q = 0.2:   0 - (0.2 - 0)^2 = -0.04

A confident wrong answer is punished hard, and an honest “not sure” on a wrong answer is barely punished. The authors prove that any bounded proper scoring rule in this slot gives a model that is both accurate and calibrated. The measured effect (ECE, the expected calibration error, is the average gap between stated confidence and actual accuracy; lower is better):

Ordinary RL with right-or-wrong rewards made calibration worse. RLCR cut calibration error sharply for a fraction of a point of accuracy. Jev’s own training method, “RLCD”, is described as targeting “epistemically honest probabilities”, but it is unpublished and known from one outside write-up (RITS). The practical rule: if your product sells a probability, reward the probability with a proper scoring rule against real outcomes, and publish reliability diagrams on a time-split test set.

Start with what fell short. Critics of Jev’s launch numbers pointed out four problems (SmartScope, 2026-09-19):

• The headline multipliers averaged four in-house workflows.
• The reference labels were the averaged answers of two chosen models, not ground truth.
• The runs came from laptops in one region.
• There was no reproducibility package.

The course’s Jev report adds that each headline multiple used a different, worst-case comparator. An independent test by Vals found about 194x faster than GPT-6 Astra when 32 judgments were packed into each document, but only 4x to 74x on single judgments under load (Vals AI, 2026-10-06). The truth depended on the workload, and the launch eval hid that.

A credible eval pack, adapted from the course’s landscape report:

1. Truth from the world, frozen. Gold labels come from outcomes and verifiers (or careful humans), never from “agreement with a frontier model”. Freeze the test set and publish a hash of its manifest.
2. Time split, entity split, post-cutoff. Test on items decided after a gap and after every model’s training cutoff, from packages or projects never seen in training. Models found the buggy file from issue text alone 76% of the time on SWE-bench Verified, against about 53% on repositories outside it (SWE-Bench Illusion, 2025-06-14). Contamination is real.
3. Head-to-heads on the same items, with confidence intervals, against: the strongest specialist (for decisions: Jev, Clef or Decision-1); a frontier LLM; a tuned classical baseline (gradient boosting, logistic regression); your rules; and the majority class. The majority class is not a joke. On German anger detection, GPT-4 zero-shot scored 0.20 accuracy against a majority baseline of 0.71, while fine-tuned encoders scored 0.89 (Bucher & Martini, 2024-06-12).
4. Metrics that match the product. Report macro-F1 and per-class recall; Brier, ECE and a reliability diagram; coverage at fixed precision (the share of cases the model handles alone while, say, 98% of its actions are correct); and robustness when questions are reworded or negated.
5. Clean eval labels. At 3-6% label noise, the test set can decide the ranking (Northcutt et al., 2021-03-26). Run cleanlab and a human pass over the test set too.
6. Enough items. 385 gold items give ±5 points at 95% confidence (how many gold labels).
7. Cost per decision and per completed workflow, including fallback rate, retries and errors (a metric suggested by Kanerika in InfoWorld), plus p50 and p95 latency measured in the customer’s region and at their concurrency.
8. A reproducibility package: code, manifest, prompts and model versions.

What a fair head-to-head looks like. The best independent one available in October 2026 is Arize’s (Arize, 2026-10-07):

Accuracy is within noise across the top three. The differences are in calibration, latency and cost. The same post ran a flip test: when the hallucination question was reversed, Jev, Clef and OpenAI held their ROC AUC (0.89-0.93), but Liquid d1 flipped to 0.17. Calibration on one wording does not carry over to another. An eval without a rewording test can’t see that.

A worked example: cost per completed workflow. Per-decision prices flatter small models. Workflows don’t, because the hard cases fall back to an expensive model. Using the course’s stack report (October 2026; 500 tokens in, 30 out; derived figures), a fine-tuned 150M encoder on CPU costs up to $3.74 per million decisions, and a Sonnet-class model $1,300. Put the encoder first and send the unsure cases to the big model (cascades):

cost per 1M workflows = encoder for all + fallback share x LLM

fallback 20%:  3.74 + 0.20 x 1,300 = $263.74   4.9x cheaper
fallback  5%:  3.74 + 0.05 x 1,300 =  $68.74    19x cheaper
fallback  1%:  3.74 + 0.01 x 1,300 =  $16.74    78x cheaper

all-LLM baseline:                     $1,300
per-decision ratio alone: 1,300 / 3.74 = 348x

The headline per-decision ratio (348x) shrinks to 5-78x once fallbacks count, and the fallback rate decides which end you land on. That rate is set by how well the model is calibrated and where you put the threshold. So calibration is not a nicety. It is the cost lever. Add the cost of wrong automatic actions, which your eval must measure, and you have the number customers actually pay (serving for a fraction of a cent).

The case study’s own eval applies all of this (building the co-pilot). It trains on events before 2026-07-01 and tests on events after 2026-08-01, with no package in both. Its metrics are macro-F1, per-class recall, ECE with reliability curves, and coverage at 98% precision. The baselines are the rules, Log Detective’s explanation mapped to a class, and a frontier LLM zero-shot. The bar: beat the rules on coverage at 98% precision, at under a hundredth of the frontier LLM’s cost per decision, with ECE at or below 0.05. It runs in shadow mode for two weeks (predict, don’t act), then live for four.

The eval harness as a C4 diagram (how to read the diagrams). First the context:

C4 · L1 System context · Eval harness

+-------------------+                      +-------------------+
| [Person]          |                      | [Person]          |
| Builder (you)     |                      | Customer or       |
|                   |                      | investor          |
+-------------------+                      +-------------------+
    | freezes test sets,                       ^ reads the
    v runs head-to-heads                       | evidence pack
+--------------------------------------------------------------+
| [Software system]                                            |
| Eval harness                                                 |
| same frozen items to every candidate; scores, costs          |
+--------------------------------------------------------------+
    | reads         | asks          | asks          | asks
    v settled       v               v               v
+-------------+ +--------------+ +--------------+ +--------------+
| [External   | | [External    | | [External    | | [External    |
|  system]    | |  system]     | |  system]     | |  system]     |
| Bugzilla,   | | Your decision| | Best         | | Frontier LLM |
| Koji, CI    | | model server | | specialist   | | API          |
+-------------+ +--------------+ +--------------+ +--------------+

You freeze test sets and run head-to-heads. The harness reads settled outcomes from the world, asks every candidate the same questions, and produces the evidence pack a customer or investor reads.

Zoomed in to the containers:

C4 · L2 Container · Eval harness

+--------------------+
| [External system]  |
| Outcome sources    |
| Bugzilla, Koji, CI |
+--------------------+
     ^
     | reads settled outcomes (HTTPS/JSON)
.....|................. Eval harness [system] ........................
:    |                                                               :
:  +----------------------+             +----------------------+     :
:  | [Container: Python]  |  freezes    | [Database: Parquet]  |     :
:  | Test-set builder     |------------>| Frozen test set      |     :
:  | time + entity split, |  (Parquet)  | items, gold labels,  |     :
:  | post-cutoff, decontam|             | manifest hash        |     :
:  +----------------------+             +----------------------+     :
:                                                                    :
:                                            | items                 :
:                                            v (Parquet)             :
:  +----------------------+             +----------------------+     :
:  | [Container: podman]  |  checks     | [Container: Python]  |     :
:  | Verifier sandbox     |<------------| Runner               |--+  :
:  | read-only to models; |  (JSON)     | same inputs to every |  |  :
:  | pass, fail, error    |             | candidate + baselines|  |  :
:  +----------------------+             +----------------------+  |  :
:                                                                 |  :
:                                            | answers, latency,  |  :
:                                            v cost (JSONL)       |  :
:  +----------------------+             +----------------------+  |  :
:  | [Database: SQLite]   |  metrics    | [Container: Python]  |  |  :
:  | Results store        |<------------| Scorer               |  |  :
:  | per item, per version|  (SQL)      | F1, Brier, ECE, CIs, |  |  :
:  +----------------------+             | coverage at 98% prec.|  |  :
:                                       +----------------------+  |  :
:.................................................................|..:
                                              asks (HTTPS/JSON)   |
                                                                  |
         +--------------------------------------------------------+
         v                      v                      v
+-------------------+  +-------------------+  +-------------------+
| [External system] |  | [External system] |  | [External system] |
| Your decision     |  | Best specialist   |  | Frontier LLM API  |
| model server      |  | API               |  |                   |
+-------------------+  +-------------------+  +-------------------+

The test-set builder applies the time, entity and post-cutoff rules and freezes the result with a manifest hash. The runner sends identical inputs to every candidate (your model, the best specialist, a frontier LLM) and to in-process baselines (majority class, rules, gradient boosting), and records answers, latency and cost. Answers that can be checked go to the verifier sandbox, which the candidates cannot touch. The scorer computes the metrics with confidence intervals, and stores every per-item result against the exact model and data versions, so any claim can be traced back to its items.

• DeepSeek-AI, DeepSeek-R1 (2025-01-22) — rule-based rewards at scale, and why they avoided neural reward models.
• Lambert et al., Tülu 3 (2024-11-22) — where RLVR got its name, inside a fully open post-training recipe.
• Damani et al., Beyond binary rewards: RLCR (2025-07-22) — the proper-scoring-rule reward for models that report confidence.
• METR, Recent frontier models are reward hacking (2025-06-05) — concrete hacks, with counts.
• Kapoor & Narayanan, Leakage and the reproducibility crisis (2022-07-14) — the taxonomy of leakage that every eval must defend against.
• Liang et al., The SWE-Bench Illusion (2025-06-14) — how contamination inflates a famous benchmark.