Sourcing from the internet

The idea in one line: for a vertical decision model, the open web is the cheap long tail. The core is structured sources that already record what happened next, such as project APIs, issue trackers, event archives and your own logs. The real work is extraction, filtering, deduplication and decontamination, in that order.

The crawl is no longer the expensive part. Common Crawl gives away billions of pages a month. Running a small quality classifier over a million documents costs about 0.3 H100-hours. At late-October-2026 rental prices of $1.99-3.99 per H100-hour, that is roughly $0.60-1.20 (arithmetic on figures from the course’s dataset and stack reports). The expensive parts are data rights and gold labels (labels).

Two choices here change outcomes by an order of magnitude. The first is where you get the data. A JSON API that returns a bug’s resolution as a field gives you a label per row for free. Scraping the same bug’s HTML gives you text you still have to label. The second is how you turn pages into text. In one controlled experiment, swapping the text extractor moved a 1B model’s benchmark score from 20.7 to 24.5, with nothing else changed (DCLM, 2024-06-17). Tuning filter thresholds, by contrast, is 10% polish.

Picture three shelves.

• The library: the open web. Huge, general, and mostly unlabelled. You get it from crawls and dumps.
• The ledger: a project’s own records. Bug trackers, update systems, build farms, mailing lists, code hosts. These are smaller, structured, and full of outcomes: closed as duplicate, build failed, update pulled.
• The diary: your own logs. Every request to your product, and what happened afterwards.

For a decision dataset, the ledger and the diary matter most, because they record what happened next, and that is a label. The library supplies context and the long tail. The rest of this lesson takes each shelf in turn, then the cleaning steps that every source needs.

Common Crawl is a free, monthly crawl of the public web. The August 2026 crawl (CC-MAIN-2026-34, crawled 7-20 August) holds 2.14 billion pages, 360 TiB uncompressed, from 40.2 million hosts and 33.1 million registered domains (Common Crawl, 2026-08-24). It ships in three formats:

• WARC: the raw HTTP responses, HTML and all.
• WAT: metadata about each response, such as headers and links.
• WET: plain text that Common Crawl has already extracted.

It also ships robots.txt captures and URL indexes, on S3 (s3://commoncrawl/crawl-data/...) and over HTTPS.

A worked example: how big is “a million pages”? 360 TiB divided by 2.14 billion pages is about 185 kB of raw response per page. So a million pages is roughly 185 GB uncompressed: a laptop disk, not a cluster. For a niche domain you rarely need whole crawls anyway. The columnar URL index lets you select only the domains you care about and fetch just their WARC records by byte range.

FineWeb is Hugging Face’s cleaned English web corpus. It holds more than 18.5 trillion GPT-2 tokens, from snapshots up to CC-MAIN-2025-26. Its pipeline is the clearest public recipe for turning a crawl into training text. In order (FineWeb card, v1.4.0, 2025-07-11; Penedo et al., 2024-06-25):

1. Block URLs from a block-list.
2. Extract text from the WARC files with trafilatura.
3. Keep English: a fastText language classifier must score at least 0.65.
4. Apply Gopher’s repetition and quality filters.
5. Apply the C4 filters, except the one that requires terminal punctuation.
6. Apply custom filters for list-like and malformed documents.
7. Remove near-duplicates with MinHash (5-word shingles, 14 buckets × 8 hashes), within each snapshot separately.
8. Anonymise PII: replace emails and public IP addresses. Phone numbers are skipped because the detector made too many false positives.

One result runs against intuition: deduplicating each dump separately beat deduplicating all dumps together. More dedup is not automatically better. Test pipeline choices against a downstream model, as FineWeb did, rather than assuming.

The same team extended the pipeline. FineWeb-2 covers over 1,000 languages, about 5 billion documents and 20 TB, from almost 100 snapshots (Penedo et al., 2025-06-26). FinePDFs adds about 3 trillion tokens from 475 million PDFs in 1,733 languages, extracted with Docling and GPU OCR, under the ODC-By licence (InfoQ, 2025-09). The course’s dataset report found FinePDFs’ details in only one source and its blog date reported inconsistently, so treat those numbers as softer.

Extraction quality matters more than people expect. DCLM trained identical 1B models on text from three extractors, with the same filters applied afterwards (DCLM, 2024-06-17):

DCLM chose resiliparse for its speed. Its quality filter was a cheap fastText classifier, trained to tell OpenHermes-2.5 examples and top ELI5 posts apart from random web text. It kept the top 10% of pages. Here is the lesson for a bootstrapper: a cheap classifier trained on examples of what good looks like is the highest-leverage filter you can build. For a vertical, “good” means “relevant to my decision”, so the same trick gives you a relevance filter.

The library behind FineWeb is datatrove. It provides readers and writers over many storage back-ends, extractors (trafilatura is the most used), filters that write out what they exclude, MinHash, sentence-level and exact-substring dedup, statistics and tokenisation blocks. It runs locally, on Slurm, on Ray, or experimentally on HF Jobs (huggingface/datatrove, v0.10.1, 2026-09-30).

For a vertical, the web crawl is the long tail. The core is structured sources. A focused crawl starts from a site’s sitemap.xml and follows only in-domain links, scored by a relevance classifier, instead of crawling everything.

The course’s distro report inventoried what already exists for Linux distributions. A few rows show how much labelled data sits behind open doors (sizes measured in October 2026 unless noted):

One practical rule falls out of that table: public GitHub Actions logs vanish after 90 days (GitHub docs), so start collecting them now.

Your own logs are the best data you will ever have. Each request to a decision API is an input. What happened afterwards (the build failed, the user overrode the decision, the ticket was reopened) is an outcome label. From day one, log the decision, its probability, the model version and a stable item ID, with user consent and a retention policy. This is the data flywheel: more use gives more labelled data, which gives a better model. No competitor can scrape it.

A bot wall is a gate in front of a website that blocks automated clients. Anubis, common on open-source project sites, makes each visitor’s browser solve a small proof-of-work puzzle before it may read the page.

On 2026-10-28 the course’s dataset report tried automated fetches and saw a clear split. Fedora’s docs site, Bodhi’s API documentation page and the ArchWiki all returned Anubis “Access Denied” pages. Meanwhile the JSON APIs of Bodhi and Red Hat Bugzilla answered normally. The distro report found the same pattern elsewhere: the AUR’s web UI and COPR’s stats page sit behind Anubis, while the AUR’s metadata dump and COPR’s api_3 are open. Fedora’s kojipkgs file server let plain, non-browser clients through and challenged clients that claimed to be browsers.

The rule follows directly:

• Build on APIs and official dumps. They are the doors projects built for machines, and they return structured fields.
• Identify yourself honestly. Use a plain user agent that names your tool and a contact address. Never pretend to be a browser to get past a wall.
• Respect published limits, such as Repology’s one request per second.
• Treat a bot wall as an explicit “no” for that content. It is part of your rights record, not a puzzle to solve.

Here is the polite way to use an API. This snippet reproduces the Bugzilla counts used throughout this part. It was run on 2026-10-28 and printed 75,520 duplicates, 12,418 upstream and 7,733 can’t-fix, matching the report.

# Reproduce this part's Bugzilla counts with the public REST API.
import requests

BASE = "https://bugzilla.redhat.com/rest/bug"
# A plain, identified user agent: say who you are and how to reach you.
HEADERS = {"User-Agent": "my-dataset-builder/0.1 (you@example.com)"}


def count(resolution: str) -> int:
    """Number of Fedora bugs closed with one resolution."""
    params = {"product": "Fedora", "resolution": resolution,
              "include_fields": "id"}  # ask only for what you need
    r = requests.get(BASE, params=params, headers=HEADERS, timeout=60)
    r.raise_for_status()
    return r.json()["total_matches"]


for res in ["DUPLICATE", "UPSTREAM", "CANTFIX"]:
    print(res, count(res))

include_fields matters twice. It keeps responses small, and it keeps personal data out (more on that below). For real mining, also record each bug’s last_change_time so you can re-pull what changed, and page through Bodhi with its rows_per_page parameter.

Extraction turns a raw page, PDF or log into the text your model will read. The main tools in October 2026:

• trafilatura (v2.3.1, 2026-10-06) for HTML. It was the best open-source extractor in its author’s own 2021 evaluation, which is a claim by the tool’s author (Barbaresi, ACL 2021).
• resiliparse for HTML at about 8× trafilatura’s speed, with similar downstream results (DCLM).
• ReaderLM-v2, a 1.5B model that turns HTML into Markdown or JSON with a 512K-token context. Its authors say it beats GPT-4o by 15-20% on their own benchmarks, a claim by the makers (Wang et al., 2025-03-03).
• Docling (v2.144.0, 2026-10-28) for PDFs, and olmOCR (v0.4.27, 2026-03-12) for scanned PDFs.
• An LLM with a typed schema turns semi-structured pages straight into rows. It is the most flexible option and the most expensive per page (see the cost table below).

Logs need extraction too. A full build log can be enormous. In Fedora’s own log dataset, full logs reach about 77 million characters, while Jev accepts 64k tokens per request. The case study keeps only snippets of at most about 2k tokens, cut around the first fatal error with Drain, a tool that groups log lines into repeated templates.

Filtering throws away what you don’t want. Cheap filters come first: language identification, repetition and quality heuristics (FineWeb’s steps 3-6). Then comes the high-leverage one: a small classifier trained on examples of what good looks like (DCLM), or of what is relevant to your decision.

PII removal belongs in this stage. PII (personally identifiable information) is anything that identifies a person: names, email addresses, IP addresses. FineWeb replaces emails and public IPs and leaves phone numbers alone because of false positives. Issue trackers are worse than the web. The Bugzilla response the dataset report pulled included real names and email addresses in the cc_detail field. So request only the fields you need with include_fields, hash user IDs, and run a detector such as Microsoft Presidio (2.2.366, 2026-10-27) over free text. Privacy law (GDPR, for people in the EU) applies whatever copyright says.

Why dedup at all? Duplicates waste money and leak answers. In C4, a well-known web corpus, one 61-word sentence was repeated over 60,000 times. Models trained on deduplicated data emitted memorised text 10× less often, and over 4% of standard validation sets overlapped their training data (Lee et al., 2021-07-14). For a decision dataset there is a sharper reason: dedup before labelling. Every duplicate you label is money spent twice and a possible leak between train and test.

Exact duplicates are easy. Normalise the text (lowercase, collapse whitespace, strip boilerplate), hash it, and keep one copy per hash.

Near-duplicates need a measure of overlap. A shingle is a run of n consecutive words. Two documents are near-duplicates when they share most of their shingles. Jaccard similarity measures that: shared shingles divided by all distinct shingles.

A worked example with 3-word shingles (FineWeb uses 5):

A = "the build failed on rawhide today"
B = "the build failed on rawhide again"

A: the build failed | build failed on | failed on rawhide
   | on rawhide today
B: the build failed | build failed on | failed on rawhide
   | on rawhide again

shared = 3, all distinct = 5, Jaccard = 3 / 5 = 0.6

The same computation in code, small enough to check by hand:

# Shingles and Jaccard similarity, small enough to check by hand.
def shingles(text: str, n: int = 3) -> set[str]:
    """All runs of n consecutive words (FineWeb uses n = 5)."""
    w = text.lower().split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}


def jaccard(a: set[str], b: set[str]) -> float:
    """Shared shingles divided by all distinct shingles."""
    return len(a & b) / len(a | b)


a = shingles("the build failed on rawhide today")
b = shingles("the build failed on rawhide again")
print(len(a & b), len(a | b), jaccard(a, b))  # 3 5 0.6

Comparing every pair of a million documents means about 500 billion comparisons, which is far too many. MinHash makes each comparison cheap. Imagine shuffling every possible shingle into a random order, and for each document noting which of its shingles comes first. The chance that two documents share the same “first” shingle is exactly their Jaccard similarity. Repeat with k different shuffles (hash functions, in practice) and keep each document’s k minimums as its signature. The fraction of matching minimums estimates Jaccard. With k = 112 and a true Jaccard of 0.6, one standard error is about sqrt(0.6 × 0.4 / 112) ≈ 0.05.

LSH (locality-sensitive hashing) avoids comparing every pair of signatures too. It cuts each signature into b bands of r rows. Two documents become a candidate pair if all r values match in at least one band:

P(candidate) = 1 - (1 - s^r)^b     s = true Jaccard similarity

FineWeb: b = 14 bands, r = 8 rows (112 hashes per document)

s = 0.5  ->  1 - (1 - 0.0039)^14  =  0.05
s = 0.7  ->  1 - (1 - 0.0576)^14  =  0.56
s = 0.8  ->  1 - (1 - 0.1678)^14  =  0.92
s = 0.9  ->  1 - (1 - 0.4305)^14  =  1.00 (0.9996)

That S-shaped curve is the whole design. Pairs at 0.5 similarity are almost never compared, pairs at 0.9 almost always are, and the 50% point sits near 0.7. If you need to catch 0.7-similar bug reports reliably, use more bands or fewer rows, and pay with more candidate pairs to check. The library version, with datasketch, is in the labelling recipe. Note that datasketch 2.0.0 (2026-07-05) changed its default permutation scheme, fixing a bias that over-estimated similarity. Hash values differ from 1.x, so rebuild stored indexes or pass scheme="legacy" (release notes).

Semantic dedup goes one step further. It embeds each document as a vector, clusters the vectors, and drops near-identical meanings even when the words differ. SemDeDup removed 50% of a LAION image-text subset with minimal loss, roughly halving training time (Abbas et al., 2023-03-16). semhash (v0.5.0, 2026-09-30) packages the idea for text.

Decontamination removes training items that overlap your test set or public benchmarks. Otherwise your eval measures memory, not skill.

The usual tool is n-gram overlap. Cosmopedia, for example, flagged candidates sharing a 10-word sequence with a benchmark, then discarded those with a difflib similarity above 0.5. But n-grams catch copies, not paraphrases. A 13B model trained on rephrased test sets reached GPT-4-level scores on MMLU, GSM8K and HumanEval, and an LLM-based decontaminator found 8-18% of HumanEval overlapping existing pretraining sets (Yang et al., 2023-11-08).

For a decision dataset there is a stronger rule, and it is structural: test items come from a later time window, and from entities (packages, projects, threads) that never appear in training. Then run MinHash between the splits as a final check. A paraphrase can slip past an n-gram filter. It cannot travel back in time into a package you held out. The evals lesson builds this into the harness.

Prices as of October 2026, from the course’s dataset report:

Check the LLM row by hand. Assume 3,000 input tokens and 500 output tokens per page. A million pages is 3 billion input tokens at $0.025 per million ($75), plus 500 million output tokens at $0.20 per million ($100). That makes $175. Google has announced price rises for some newer Flash models from 2027-01-01, so re-check before you budget.

The pattern in the table: getting pages costs nothing to hundreds of dollars per million, and turning them into structured rows with an LLM costs about the same. Neither is where a bootstrapper’s money goes. It goes to rights and to the few hundred gold labels that keep everything else honest.

Here is the sourcing pipeline as a C4 diagram (how to read the diagrams). First the context:

C4 · L1 System context · Sourcing pipeline

+--------------------+
| [Person]           |
| Builder (you)      |
+--------------------+
           | chooses sources, checks samples
           v
+--------------------+  reads dumps     +--------------------+
| [Software system]  |----------------->| [External system]  |
| Sourcing pipeline  |                  | Common Crawl       |
| raw pages and      |                  +--------------------+
| records in; clean, |  calls JSON APIs +--------------------+
| split rows out     |----------------->| [External system]  |
|                    |                  | Bugzilla, Bodhi,   |
|                    |                  | Koji, GH Archive   |
|                    |                  +--------------------+
|                    |  stops at        +--------------------+
|                    |- - - - - - - - ->| [External system]  |
|                    |  bot walls       | Walled websites    |
+--------------------+                  +--------------------+
           | hands over clean rows
           v
+--------------------+
| [External system]  |
| Labelling pipeline |
+--------------------+

You choose the sources. The pipeline reads dumps and APIs, stops at bot walls, and hands clean rows to the labelling pipeline of the next lesson.

Zoomed in to the containers:

C4 · L2 Container · Sourcing pipeline

+------------------+  +------------------+  +------------------+
| [External system]|  | [External system]|  | [External system]|
| Common Crawl     |  | Project APIs     |  | Walled websites  |
+------------------+  +------------------+  +------------------+
       ^                     ^                     x
       | reads WARC          | calls               | stop: never
       | (S3/HTTPS)          | (HTTPS/JSON)        | crossed
.......|.....................|..... Sourcing pipeline [system] ....
:      |                     |                                    :
:  +-------------------------------+ saves   +------------------+ :
:  | [Container: Python]           |-------->| [Database: JSON] | :
:  | Fetchers                      | (JSON)  | Rights snapshots | :
:  | API clients, crawl4ai, plain  |         | robots.txt, ToS, | :
:  | identified user agent         |         | date per source  | :
:  +-------------------------------+         +------------------+ :
:      | writes raw (WARC, JSONL)                                 :
:      v                                                          :
:  +---------------------+  reads   +-------------------------+   :
:  | [Database: files]   |--------->| [Container: datatrove]  |   :
:  | Raw store           | (WARC,   | Extract and filter      |   :
:  | WARC + raw JSON     |  JSONL)  | trafilatura, language,  |   :
:  +---------------------+          | quality classifier      |   :
:                                   +-------------------------+   :
:                                        | text rows (Parquet)    :
:                                        v                        :
:  +---------------------+  rows    +-------------------------+   :
:  | [Container: Python] |<---------| [Container: Python]     |   :
:  | Splitter            |(Parquet) | Dedup and PII scrub     |   :
:  | time + entity split,|          | exact hash, MinHash,    |   :
:  | decontaminate       |          | Presidio                |   :
:  +---------------------+          +-------------------------+   :
:      | writes splits (Parquet)                                  :
:      v                                                          :
:  +---------------------+                                        :
:  | [Database: Parquet] |                                        :
:  | Clean corpus        |                                        :
:  +---------------------+                                        :
:......|..........................................................:
       | hands over clean rows (Parquet)
       v
+------------------+
| [External system]|
| Labelling        |
| pipeline         |
+------------------+

The fetchers save a rights snapshot for every source as they go. Raw responses are kept, so you can re-extract when a better extractor appears. Dedup and PII scrubbing run before the split, and the split is by time and entity, so decontamination is built in rather than bolted on. The order of operations, from the course’s dataset report, is extract, filter, exact dedup, near and semantic dedup, PII scrub, then time and entity split, all before a single label is bought.

• Penedo et al., The FineWeb datasets (2024-06-25) — the full, ablated recipe for turning Common Crawl into training text.
• Li et al., DataComp-LM (2024-06-17) — controlled experiments showing that extraction and a cheap quality classifier move results more than you’d expect.
• Lee et al., Deduplicating training data makes language models better (2021-07-14) — why near-dedup matters, with numbers.
• huggingface/datatrove — the pipeline library behind FineWeb, ready to run on a laptop or a cluster.
• Common Crawl: August 2026 crawl archive (2026-08-24) — what is in a monthly crawl and how to get it.