From probability to automation

The idea in one line: A calibrated probability plus a price list of mistakes tells you when to act, when to ask and when to escalate; automate the confident share, route the rest, and your savings grow as 1 / (1 − share automated).

Accuracy is a 10% lever. Coverage is the 10x one. Coverage is the share of cases you let the model decide on its own. Suppose the big model (or person) costs 100 times the small one and every case tries the small model first. Automating 80% of cases makes the whole pipeline 4.8x cheaper; 95% makes it 16.7x cheaper; 99% makes it 50x cheaper. The order-of-magnitude win sits in the last few points of coverage, and you may only claim those points if the model’s confident cases are verifiably right. That is why the previous lesson mattered: calibration is what lets you certify coverage.

This lesson climbs from one decision to a whole system: the threshold for one case, the abstain option, guaranteed shortlists, cascades, and the arithmetic that ties them together.

Carrying an umbrella costs you 1 unit of hassle. Getting soaked costs 9. If the chance of rain is p, leaving the umbrella costs you 9p on average, and carrying it costs 1 for sure. Carry it when 9p ≥ 1, that is when p ≥ 1/10. You carry an umbrella at a 10% chance of rain, not at 50%, because the mistakes cost different amounts.

Expected cost is each outcome’s cost weighted by its probability (from probability is the product). For a yes/no decision with probability p that the case is positive, a false positive (acting when you shouldn’t) costing C_FP, a false negative (not acting when you should) costing C_FN, and correct calls costing nothing, act when (1 − p) × C_FP ≤ p × C_FN. Rearranged:

act when  p ≥ t* = C_FP / (C_FP + C_FN)

Elkan derived this and showed that moving the threshold is the principled response to a new cost matrix; rebalancing the training data is the indirect equivalent (Elkan, IJCAI 2001). The umbrella: C_FP = 1 (carried it for nothing), C_FN = 9 (soaked), so t* = 1/10.

A model scores each dependency bump for a bootc distro image with p = P(the bump builds and boots). The costs are invented for illustration:

• Merging a bad bump, C_FP = $400 (a broken nightly image, an hour of debugging).
• Rejecting a good bump, C_FN = $40 (redo it later, a delayed security fix).
• A human review, $20, and assume the human is always right.

Two actions (merge or reject): t* = 400 / (400 + 40) = 0.909. Merge only above 0.91.

Three actions (merge, reject or review): merging costs 400(1 − p), rejecting costs 40p, review costs a flat 20. Merge beats review when 400(1 − p) ≤ 20, so p ≥ 0.95. Reject beats review when 40p ≤ 20, so p ≤ 0.50.

One dependency bump: the cheapest action for each p

p:   0.0               0.50                       0.95     1.0
     |------------------|--------------------------|--------|
        auto-reject        a person reviews it       merge
        (40p <= 20)        (a flat $20)         (400(1-p) <= 20)

# Pick the cheapest action for a calibrated p (costs from the example)
C_FP, C_FN, C_REVIEW = 400, 40, 20  # bad merge, wrong reject, a person

def expected_costs(p):
    """p = P(the bump builds and boots); correct calls cost nothing."""
    return {"merge": (1 - p) * C_FP,   # we merged a bad bump
            "reject": p * C_FN,        # we rejected a good bump
            "review": C_REVIEW}        # a person decides, always right

def best_action(p):
    costs = expected_costs(p)
    return min(costs, key=costs.get)

for p in (0.40, 0.80, 0.96):
    print(p, best_action(p))  # 0.4 reject, 0.8 review, 0.96 merge

What miscalibration costs. Say the model reports 0.97, but bumps at that score really succeed 90% of the time. Each auto-merge now costs 400 × 0.10 = $40, twice the $20 review. Automation loses money on every item. If 0.97 really meant 97%, each merge costs 400 × 0.03 = $12, saving $8 against review. Same accuracy, same ranking, opposite business result.

The expected value of perfect information is how much you would pay to know the truth before acting: the expected cost of your best action now, minus the cost if you knew the truth (zero here, since correct calls cost nothing). Without the review option it is min(400(1 − p), 40p):

A check (a person, a frontier model, a CI build) is worth buying when it costs less than that. Far from the threshold, nobody needs to look; near it, information pays most. That is the formal case for cascades: send only the near-threshold band to something expensive. It only works if p is calibrated, because a miscalibrated p puts the band in the wrong place.

Why a calibrated probability beats an accurate label:

1. Costs change and thresholds follow, without retraining. In a security-patch week, C_FN rises; recompute t*. A label hard-codes one threshold, usually 0.5.
2. Accuracy is measured at the wrong threshold. With t* = 0.91, accuracy at 0.5 says little about cost.
3. It sizes the work. The expected number of cases in the review band tells you how many reviewers you need.
4. It composes. Calibrated probabilities can be multiplied through a pipeline and updated with Bayes’ rule when base rates shift.
5. It makes errors auditable. “We automated above 0.95 and saw 2.1% errors against 3% predicted” is a testable contract; “the model said yes” is not.

Selective prediction lets a model abstain. Coverage is the share of inputs it answers, and selective risk is its error rate on those. Sort cases by confidence, accept from the top, and the error rate at each coverage traces a risk-coverage curve.

A worked example with 10 made-up cases:

If you can live with 15% errors, you can automate 80% (threshold 0.70). If you need zero errors, 50% (threshold 0.90). With only 10 cases those numbers are noise, which is why the label counts matter.

Note what drives the curve: only the order of confidences (discrimination), not their calibration. Calibration matters for choosing and predicting the operating point: a calibrated model tells you threshold 0.97 means about 3% errors before you have labelled anything there. The classic result: selective classification held ImageNet top-5 error to 2% “with probability 99.9%” at “almost 60% test coverage” (Geifman & El-Yaniv, 2017-05-23). The area under the curve, AURC, summarises the whole trade-off; lower is better.

# Risk-coverage curve and the largest share you can automate
import numpy as np

def risk_coverage(conf, correct):
    """Accept the most confident first; error rate among accepted."""
    order = np.argsort(-np.asarray(conf, float))
    errs = 1 - np.asarray(correct, float)[order]
    k = np.arange(1, len(errs) + 1)
    coverage, risk = k / len(errs), np.cumsum(errs) / k
    return coverage, risk, risk.mean()  # last value: AURC

def coverage_at_risk(conf, correct, target=0.02):
    """Largest coverage with selective error <= target, and threshold."""
    cov, risk, _ = risk_coverage(conf, correct)
    ok = np.flatnonzero(risk <= target)
    if ok.size == 0:
        return 0.0, None
    return float(cov[ok[-1]]), float(np.sort(conf)[::-1][ok[-1]])

conf = [0.99, 0.97, 0.95, 0.93, 0.90, 0.85, 0.80, 0.70, 0.60, 0.55]
right = [1, 1, 1, 1, 1, 0, 1, 1, 0, 0]
print(coverage_at_risk(conf, right, target=0.15))  # (0.8, 0.7)

The operating loop for any decision API:

1. Label a held-out set from real traffic and compute the risk-coverage curve.
2. Pick the target error rate for automated cases from business costs.
3. Read off the largest coverage that meets it, and its confidence threshold.
4. Confirm on a second held-out set, because a threshold tuned on one set is optimistic on that set.
5. Route the rest to a person, a bigger model, or a cheaper check such as running the build.

Step 4 is not pedantry. Vals AI set a 1% error budget for Jev; it reached 95% coverage but a realised error of 1.6%, over budget. Sonnet 5.5 managed 100% coverage at 0.4% (Vals AI, 2026-10-06).

Instead of one answer, the model hands you a short list, with a promise: “the right answer is on this list 90% of the time”. A list of one means it is sure; a list of three means “one of these”; an empty list means “nothing I know fits”. Conformal prediction builds such lists around any model’s scores, using a small labelled calibration set (Angelopoulos & Bates, 2021-07-15).

A package-triage model labels each package ok, flaky or broken. You have n = 9 labelled calibration packages and want 80% coverage, so the error rate α = 0.2.

1. For each calibration package, take the probability the model gave the true label: 0.95, 0.90, 0.85, 0.80, 0.70, 0.60, 0.55, 0.40, 0.20.
2. Turn each into a nonconformity score, how badly the model fit the truth: s = 1 − p, giving 0.05, 0.10, 0.15, 0.20, 0.30, 0.40, 0.45, 0.60, 0.80.
3. Let k = ⌈(n + 1)(1 − α)⌉ = ⌈10 × 0.8⌉ = 8. The threshold q̂ is the 8th smallest score: 0.60.
4. For a new package, include every label with 1 − p ≤ 0.60, that is p ≥ 0.40.

The guarantee, if new data is exchangeable with the calibration data (roughly, drawn from the same process, order carrying no information):

1 − α  ≤  P(true label is in the set)  ≤  1 − α + 1/(n + 1)

Here that is between 80% and 90% over many packages. But with only 9 calibration points, the coverage you actually get from this calibration draw is spread out: it follows a Beta(8, 2) distribution with mean 0.80 and standard deviation about 0.12. With n = 1,000 and α = 0.1, coverage “typically” lands between 0.88 and 0.92. To be within ±ε of target with probability 0.9 (α = 0.1), you need 22 calibration items for ε = 0.1, 102 for 0.05, 2,491 for 0.01 and 9,812 for 0.005 (Angelopoulos & Bates, Table 1).

# Split conformal prediction sets for any classifier's probabilities
import numpy as np

def conformal_qhat(cal_probs, cal_labels, alpha=0.1):
    """q-hat: the k-th smallest score s = 1 - p(true label)."""
    n = len(cal_labels)
    scores = np.sort(1 - cal_probs[np.arange(n), cal_labels])
    k = int(np.ceil((n + 1) * (1 - alpha)))  # finite-sample rank
    return np.inf if k > n else scores[k - 1]  # too few points: keep all

def prediction_sets(probs, qhat):
    """Boolean mask (n, K): label k is in the set when p_k >= 1 - qhat."""
    return probs >= 1 - qhat

# qhat = conformal_qhat(cal_probs, cal_y, alpha=0.1)
# S = prediction_sets(test_probs, qhat)
# coverage = S[np.arange(len(test_y)), test_y].mean()  # about 0.9
# automate = S.sum(1) == 1                             # one-label sets

On a synthetic 10-class model, α = 0.1 gave 89.3% test coverage, but the average set held 4.7 labels because that model was only 44% accurate (the researcher’s own run). Weak models get valid but useless sets.

Source: Angelopoulos & Bates.

Upgrades to know:

• APS (adaptive prediction sets) adds labels in probability order until their mass passes the threshold, which covers hard cases more evenly (Romano, Sesia & Candès, 2020-06-03). RAPS trims unlikely tail labels and gave sets “often factors of 5 to 10 smaller” than a Platt-scaling baseline on ImageNet (Angelopoulos et al., 2020-09-29).
• Conformal risk control bounds “the expected value of any monotone loss function”, such as the false-negative rate, for when different mistakes cost different amounts (Angelopoulos et al., 2022-08-04).
• For LLMs, conformal factuality backs off an answer until it is correct with probability 1 − α, reaching “80-90% correctness guarantees while retaining most of the model’s original output” (Mohri & Hashimoto, 2024-02-15).

A cascade tries a cheap model first and escalates when it is not confident. A router picks one model per query up front. Both are selective prediction with a bigger model as the fallback, and you can end the chain with a person.

C4 · L2 Container · Decision cascade

  +-----------+
  | [Person]  |
  | Requester |
  +-----------+
        | sends a case (HTTPS/JSON)
        v
..................... Decision cascade [system] ....................
:                                                                  :
:  +----------------------+  scores (gRPC)  +--------------------+ :
:  | [Container: Python]  |---------------->| [Container: ONNX]  | :
:  | Cascade router       |<----------------| Decision model     | :
:  | thresholds per type  |  distribution   | tier 1, cheapest   | :
:  +----------------------+                 +--------------------+ :
:     |     |       |                                              :
:     |     |       | logs case, answer, outcome (SQL)             :
:     |     |       v                                              :
:     |     |  +--------------------+                              :
:     |     |  | [Database: SQLite] |                              :
:     |     |  | Outcome log        |                              :
:     |     |  +--------------------+                              :
:     |     |                                                      :
:     |     |  queues what tier 2    +------------------+          :
:     |     +----------------------->| [Container: Web] |          :
:     |        left unsure (SQL)     | Review queue     |          :
:     |                              +------------------+          :
:     |                                        |                   :
:.....|........................................|...................:
      | escalates the                          | shows cases
      | middle band (HTTPS/JSON)               | (HTTPS)
      v                                        v
+---------------------+                +-------------+
| [External system]   |                | [Person]    |
| Frontier LLM API    |                | Reviewer    |
| tier 2, slower and  |                | tier 3, the |
| pricier             |                | last word   |
+---------------------+                +-------------+

The router asks the small decision model first and acts on confident answers. It sends the near-threshold band to a frontier LLM, and queues whatever is still unsure for a person. Every case and its eventual outcome lands in the log, which becomes the labelled data for refitting the temperature and thresholds. New to these boxes? See how to read the diagrams.

Real examples of the pattern: a community “smart tier” runs Jev first and re-asks a reasoning model only under 0.7 confidence (JevBench results); TypeSafe’s own docs route intents to “deterministic logic, a specialist LLM, or a human” (TypeSafe docs: Patterns, read 2026-10-28).

The guardrail result is the strongest evidence that recalibration is the first step of any cascade: an uncalibrated tier 1 escalates the wrong cases.

• Routing collapse. One source reports that as budgets rise, existing routers send nearly 100% of queries to the strongest model, while an ideal router would need it for fewer than 20%. At the largest budget, 94.9% of queries had a near-zero quality gap between the top two models, so tiny score errors flip choices (When Routing Collapses, 2026-02).
• Savings depend on your traffic. RouteLLM’s own savings range from 35% to 85% across three benchmarks. Savings track how many queries a cheap model can answer, which is a property of your workload, not the router.
• “95% of GPT-4 quality” means losing 5% of the gap, which may be unacceptable when mistakes have lopsided costs.
• Production routers publish no savings. OpenAI’s GPT-5 router picks a fast or a reasoning model by “conversation type, complexity, tool needs, and explicit intent”, with no savings figure given (GPT-5 system card, 2025-08-13).

Every case hits the small model; only the uncovered share goes on to the big one:

cost per case  = c_small + (1 − coverage) × c_big
saving vs big  = c_big / cost per case

With c_big = 100 × c_small:

Check the first row: 1 + 0.2 × 100 = 21 units per case against 100, so 100 / 21 = 4.8x.

# Cascade saving against sending every case to the big model
def cascade_saving(coverage, price_ratio):
    small, big = 1.0, float(price_ratio)    # small model costs 1 unit
    per_case = small + (1 - coverage) * big  # all cases try small first
    return big / per_case

for cov in (0.80, 0.90, 0.95, 0.99):
    print(cov, round(cascade_saving(cov, 100), 1))  # 4.8 9.1 16.7 50.0
print(round(cascade_saving(0.90, 440), 1))  # 9.8: a bigger gap barely helps

The last line is the surprise. On TypeSafe’s own table, Opus 5 costs about 440 times Jev per case ($0.1761 against $0.0004, TypeSafe evals, read 2026-10-28). Yet at 90% coverage the cascade saves only 9.8x, because the escalated 10% dominates the bill. As the price gap grows, the saving approaches 1 / (1 − coverage) and stops there: 10x at 90%, 20x at 95%, 100x at 99%. Coverage, not the price gap, sets the ceiling.

Two reality checks keep this honest:

• Other work dilutes the gain. One source, a scan of 12,759 Jev launch tweets, found a median user-reported speed-up of 7x, far below the headline multiples, because real workflows do more than decide (InfoQ, 2026-10-01). The is 5x real? lesson applies that reasoning to a distro workflow.
• The coverage must be certified. Each extra point of coverage at a fixed error rate needs labels to prove it: about 753 automated items to pin a 2% error rate within ±1 point, or 300 error-free ones to claim at most 1% (calibration).

• Conditional guarantees. Conformal coverage is an average; per-subgroup coverage is impossible in general, and the active research is conditional validity, shift and open-ended generation (Cherian, Gibbs & Candès, 2024-06-14).
• Shift and feedback loops. Once you automate, the cases a person sees change, and so does the data you recalibrate on. Exchangeability, the assumption under every guarantee here, quietly breaks.
• Routers that over-escalate, per the routing-collapse critique, and no published savings from production routers.
• RL that erodes abstention. Reasoning fine-tuning cut abstention by 24% on average across 20 frontier models (AbstentionBench, 2025-06-10), so the big model in tier 2 may itself be bad at saying “I don’t know”.
• Vendor guarantees. No decision-model vendor publishes coverage at a stated error rate on ground-truth labels, as of October 2026. That is a gap a founder can fill.

• Elkan, The Foundations of Cost-Sensitive Learning, IJCAI 2001 — the threshold rule and why thresholds, not retraining, answer new costs.
• Angelopoulos & Bates, A Gentle Introduction to Conformal Prediction, 2021-07-15 — the best practical guide, with code and sample-size tables.
• Geifman & El-Yaniv, Selective Classification for Deep Neural Networks, 2017-05-23 — abstention with a guaranteed error rate.
• RouteLLM (LMSYS), 2024-07-01 — a router with published savings, and how they vary by benchmark.
• Cornacchia et al., Calibrated Deferral Routing for Cost-Efficient Guardrails, ICML 2026 workshop — why recalibration comes before any cascade.
• When Routing Collapses, 2026-02 — the strongest critique of trained routers.