Where the time goes

The idea in one line: Building a distro is mostly waiting for builds, reading failures and choosing between options, not writing code. The mix depends on how much of the distro you build yourself.

This case study asks one question: can a Jev-like model help someone build a new Linux distro 5x faster? You can’t answer that until you know where the hours go. A model that makes one step 10x faster barely moves the total if that step is a small slice of the week. The verdict lesson works through that arithmetic.

The evidence points somewhere surprising. In image-based distros, most failed builds aren’t code problems at all: they’re download mirrors that didn’t answer. And the operator’s own first week went on learning platform facts one failed build at a time. That aims the 10x lever at the build loop (the time from “I changed something” to “I know whether it worked”) and at deleting whole decisions, not at a smarter classifier.

Start with an analogy. You can make bread by growing wheat, by buying flour, or by buying a frozen loaf and adding your own toppings. Each step up saves labour, and it also changes the kind of work that’s left. Distros work the same way.

A distro (distribution) is a curated, tested set of software built from upstream sources, plus the tools to install and update it. Each piece of software is a package, and each package has a build recipe that says how to compile it and where its files go. A build farm is a set of servers that runs those recipes for you.

Here is each kind in plain words.

• From scratch. You compile everything yourself, following the LFS book. LFS measures time in SBUs (standard build units: the time one core takes to build the first pass of binutils). GCC, the biggest package, takes about 5 minutes on the fastest machines and could take days on slow ones (LFS 12.4: About SBUs, read 2026-10-28).
• Source-based. Gentoo compiles packages on your own machine from recipes called ebuilds. USE flags switch optional features on or off. Even Gentoo now offers binary packages, since 2023-12-29, “To speed up working with slow hardware”, as Gentoo puts it (Gentoo news, 2023-12-29).
• Binary, classic. The distro builds packages once and ships the results. RPM is Fedora’s package format, and an RPM spec file is its build recipe. Koji is Fedora’s official build farm. COPR is Fedora’s self-service build service, where anyone can build packages into a personal repository. Arch’s recipe is a shell script called a PKGBUILD, and the AUR (Arch User Repository) holds 123,268 user-submitted ones, counted from its metadata dump on 2026-10-28. Debian keeps its recipe in a debian/ folder.
• Optimized rebuild. CachyOS recompiles Arch packages for newer CPU levels (x86-64-v3, v4, Zen 4 and 5). It claims about 5%-20% more speed for v3; that’s the vendor’s own figure (CachyOS wiki, updated 2026-10-17).
• Declarative. In nixpkgs you describe packages and whole systems as code. Hydra, Nix’s build farm, builds the lot. One evaluation on 2026-10-28 held 215,592 build jobs and took 774 seconds just to evaluate, before any compiling (Hydra eval 1830781, read 2026-10-28).
• Image-based. bootc ships the whole operating system, kernel included, as an OCI image: the same container format Docker and Podman use. A machine updates by switching to a new image in one step, and can roll back (bootc docs, read 2026-10-28). Universal Blue (Bluefin, Aurora, Bazzite) is a layer of customization on top of Fedora, built by GitHub Actions (universal-blue.org, read 2026-10-28).
• Embedded. You cross-build a small system for a device. Yocto’s quick-build guide asks for at least 140 GB of disk, 32 GB of RAM and 16 cores (Yocto quick build, updated 2026-10-28).

The pattern that matters. The more derivative the distro, the less time goes into compiling and the more goes into selecting, configuring and verifying. A bootc derivative mostly picks packages, sources and defaults, then checks that the image builds and boots. A from-scratch distro mostly writes and debugs recipes. Picking and checking are typed decisions, the thing a decision model is good at. Writing recipes is generation.

This course’s operator builds two distros. They make the case concrete.

terra-fedora is a Fedora 44 bootc desktop, shipped as two images: a base image for most GPUs, and an NVIDIA image with the open kernel modules for Turing cards and newer (tsaqib/terra-fedora/README.md:1-12). The Containerfile (the build script for a container image) starts from a minimal Fedora root filesystem, then runs 33 feature scripts in order. A separate stage builds and signs the NVIDIA driver, so the signing key and the compilers never reach the final image (Containerfile:23,34-41).

C4 · L1 System context · terra-fedora image build

                    +-----------------------+
                    | [Person]              |
                    | Operator              |
                    | picks, fixes, tests   |
                    +-----------------------+
                         |            ^
          edits, starts  |            | pass or fail, logs,
                         v            | a VM boot
                    +-----------------------+
                    | [Software system]     |
                    | terra-fedora build    |
                    | Containerfile plus    |
                    | 33 feature Taskfiles  |
                    +-----------------------+
                                |
          +---------------------+---------------------+
          | pulls RPMs          | runs on             | pushes image
          v                     v                     v
+-------------------+ +-------------------+ +-------------------+
| [External system] | | [External system] | | [External system] |
| Fedora 44 repos,  | | GitHub Actions    | | ghcr.io           |
| RPM Fusion, Terra | | 180-min timeout   | | image registry    |
+-------------------+ +-------------------+ +-------------------+

The diagram shows one person, one build system and three outside services (how to read the diagrams). Every change the operator makes has to travel down through the package repositories and a CI runner before an answer comes back up.

The NVIDIA stage exists because of three pieces of plumbing:

• Secure Boot is a firmware feature that only loads kernel code signed with a trusted key.
• akmods builds out-of-tree kernel modules, such as NVIDIA’s driver, and signs them with a local key.
• That key has to be enrolled through the MOK (Machine Owner Key) screen at the next boot, by typing a one-time password (RPM Fusion: Secure Boot, read 2026-10-28).

The repository writes its decisions down as data. There are 187 package rows across the 33 features, each a line of name | source | function. 164 come from Fedora, 11 from RPM Fusion, 5 from Terra and the rest from GitHub releases, COPR and one vendor repository. The rules say “Prefer Fedora packages, then RPM Fusion or Terra, then upstream releases” (CLAUDE.md:11) and “Weak deps are off” (CLAUDE.md:10). A weak dependency is a package that another package only recommends: with weak dependencies off, you get it only if you name it. There is also a ledger of 25 rejected features (CLAUDE.md:40).

mycachy turns a bare CachyOS install into the operator’s machine with just recipes. Its whole NVIDIA setup is one line: install <kernel>-nvidia-open for each installed CachyOS kernel (tsaqib/mycachy/modules/nvidia/mod.just:7-8). That works because CachyOS ships prebuilt NVIDIA modules for each kernel. Hold on to that contrast. The signing pipeline terra-fedora maintains by hand doesn’t exist at all on mycachy.

Nobody keeps a stopwatch on distro maintainers, but build farms keep receipts. Here is what they say.

In the 12 months to 2026-10-28, Fedora’s Koji ran 5,774,084 successful and 284,204 failed per-architecture build tasks. That’s counted from Koji’s public XML-RPC hub on 2026-10-28. Check the arithmetic yourself:

failure rate = 284,204 / (5,774,084 + 284,204) = 0.047, so 4.7%
per day      = 284,204 / 365                   = about 780

Now a what-if, not a measurement. Suppose a human spent just 5 minutes on each failure. That’s 780 × 5 = 3,900 minutes a day, or 65 hours: eight people doing nothing but reading logs. Many of those tasks are scratch builds nobody reads, so the real figure is lower. The point stands, though: nobody can look at every failure.

A mass rebuild rebuilds every package against a new toolchain. Release engineering bumps the release, builds in the background, lists failures by owner and tags the successes (Fedora 45 Mass Rebuild, read 2026-10-28). FTBFS (“fails to build from source”) is the distro word for a package that no longer builds.

Source: the kojipkgs mass-rebuild pages, read 2026-10-28. Other distros look the same. Ubuntu’s GCC 16 test rebuild, started 2026-03-20, listed 2,272 source packages failing on at least one architecture, 1,150 of them on amd64 (report page, counted 2026-10-28). Debian’s continuous rebuild shows 799 FTBFS out of 41,617 source packages in unstable, about 1.9% (reproducible builds, updated 2026-10-29).

This is the most useful fact in the lesson. When Fedora tried GCC 15 on the whole archive, 1,438 of 8,711 builds failed. People classified 914 of the failures by hand (Fedora wiki: F42 GCC 15 mass prebuild, classified 2025-01-10):

C23 is the newer C language standard, and GCC 15 made it the default. In C23, an empty parameter list () means “no arguments”, the same as (void). Words like bool, true and false also became reserved. Old code that relied on either now fails. One subcategory, function declarations (c23/fndecls), held 499 failures by itself: 55% of everything classified.

Why this matters for a model: when three quarters of a campaign is one class, a classifier that recognizes that class plus one fix template clears most of the pile. Failure distributions like this, where a few classes hold most of the cases, are called heavy-tailed, and they’re what make triage worth automating.

Universal Blue builds its images on GitHub Actions, so every run is public. Here is the failure share over each repository’s most recent 1,000 runs, counted from the GitHub REST API on 2026-10-28:

Now the telling part. In akmods (the repo that prebuilds kernel modules), 47 of the last 1,000 runs were re-run, and 40 of those, 85%, then passed. The research read the failed-step logs of nine “Test Image” jobs. Seven were Librepo error: Cannot download … All mirrors were tried. One was an HTTP 403 on a download, and one was envsubst: command not found. None was a compile error. Nine logs is a tiny sample, so treat this as a strong hint, not a law. Still, most of the “NVIDIA broke the build” pain in this CI is really “did the mirror answer?”

• Keeping up. Being out of date is normal. On 2026-10-28, Repology counted 25.9% of Fedora Rawhide’s projects and 31.3% of the AUR’s as having a newer release upstream (Repology).
• Security. A CVE is a public identifier for a vulnerability. The kernel alone produced 8,100 CVEs in 2026 up to 2026-10-28, about 27 a day, counted from OSV’s Linux data. Each one asks: does this affect our kernel and config?
• Support. Bazzite had 542 issues opened in the 90 days to 2026-10-28, about six a day (GitHub API count).
• Hardware QA. A VM can’t emulate an NVIDIA GPU or the MOK screen. No public oracle says “this image boots on a Turing laptop with Secure Boot on”. Hardware truth only arrives through users.

Put together, each activity is dominated by a different kind of work:

The operator’s git history makes the abstract table personal. terra-fedora went through four delivery designs in six days (git log, 20 commits on main):

10-10 20:09  a chezmoi source for the desktop         0a44fbd
10-14 23:27  Ansible plus Nix home-manager            8e020c6
10-15 08:57  Ansible only, after one VM run           5051a9b
10-15 16:48  a Fedora 44 bootc image                  88edfb0

Almost every fix in between was a fact about Fedora 44 that only a failed build or a VM boot revealed:

Each row cost at least one lap of this loop:

  edit a feature
       |
       v
  build the image ----> lint ----> boot a VM ----> read the result
  (20-180 min in CI)                                     |
       ^                                                 |
       +--------- one Fedora 44 fact learned per lap ----+

So the binding constraint is build-loop latency, not judgement. Here’s an illustration with made-up round numbers. If a lap takes 60 minutes, twenty facts cost 20 hours of waiting, whatever model you use. Cut the lap to 6 minutes and the same twenty facts cost 2 hours. That’s a 10x change, and no classifier gets you it.

One more signal: 16 of the 20 commits carry trailers from coding agents, 10 from Cursor and 6 from Claude. The operator already drives generative agents. So the real question for the co-pilot is what a deciding model adds to that loop.

A search for a time-use study of distro maintainers found none. A known maintainer survey (Tidelift’s 2024 report) couldn’t be text-extracted, so it’s an open gap. What exists is outcome data: failures, queues, out-of-date counts, CVE volumes.

Outcomes are a weak stand-in for hours. A 4.7% failure rate could mean one minute per failure or one hour, and the data can’t tell you which. So every time share in the verdict lesson is the researcher’s estimate, reasoned from this evidence and labelled as such. The single most useful next step for anyone serious about the 5x question is boring: log every maintainer touch for a month.

What already works. The most successful time-saver in distro land is plain automation. nixpkgs’ update bot r-ryantm opened 45,153 pull requests in the 12 months to 2026-10-28, and 39,422 were merged. That’s 41% of all 95,265 merged nixpkgs pull requests that year (GitHub search API count), and it uses no language model. Fedora’s mass rebuild, likewise, is a calendar with scripts.

Where AI has started. Fedora runs an LLM that explains failed builds (Log Detective) and a small model that predicts build durations (rpmeta). The decision map covers both.

Open problems:

• No time data, so every speedup claim rests on estimates.
• No public dataset links image versions to boot success on specific GPUs and firmware.
• History is vanishing. GitHub keeps Actions logs for public repos for at most 90 days (GitHub docs), so CI-failure datasets have to be collected continuously or lost.
• Anti-bot walls (Anubis) now front Fedora’s web pages, the AUR and Buildroot. The plain APIs still answer, as the sourcing lesson discusses.

• Fedora wiki: F42 GCC 15 mass prebuild: the hand classification behind the 73% figure, and a model of how a toolchain campaign runs.
• kojipkgs mass-rebuild failure pages: raw failure lists by owner for each Fedora release.
• ublue-os/akmods: how Universal Blue prebuilds and signs kernel modules at scale, including the kernel pin.
• bootc documentation: what it means to ship an OS as a container image.
• RPM Fusion: Secure Boot: the akmods and MOK enrolment steps that terra-fedora automates.
• Repology statistics: how far behind upstream every major distro is, refreshed daily.