Corpus
A 20-trillion-token pretraining mixture designed as SQL over public Scry surfaces: web text, public discourse, scholarly full text, the public record, code, and reference — every document time-stamped and source-attributed. Designed and measured in place; nothing is copied.
| Role | Surfaces | Raw | Retained | Weight | Cooldown |
|---|---|---|---|---|---|
| Web | commoncrawl.pagescrawl.pagesinternet.documents | 52T | 12.6T | 47% | 40% |
| Public discourse | internet.textreddit.commentsreddit.poststwitter.tweetssocial.postshackernews.itemsstackexchange.postsforums.postsmailing_lists.messagesquora.answers | 4T | 2.4T | 16% | 12% |
| Scholarly full text | academic.catalogacademic.papersacademic.extractionsopenalex.works | 3.5T | 2.4T | 12% | 18% |
| Public record | courts.china_judgmentscn_enterprise.companiesinternet.documents | 3T | 1.6T | 9% | 8% |
| Code and software history | github.reposgithub.documentspackages.catalog | 1T | 0.8T | 9% | 14% |
| Transcripts | tiktok.transcriptsinternet.documents | 0.1T | 0.08T | 1% | 1% |
| Books and reference | books.cataloginternet_archive.itemsinternet.text | 0.4T | 0.3T | 4% | 5% |
| Markets and structured records | markets.catalogcn_enterprise.companiesevents.records | 0.5T | 0.2T | 2% | 2% |
Raw rows and uncompressed text bytes are read from engine metadata; tokens are estimated at 4 bytes per token and rounded to one significant figure. Retained sizes and weights are design targets.
The same page recurs across crawls and boilerplate rides along, so this slice loses most of its raw mass to exact and fuzzy dedup before quality bands are applied; the top three of five bands train, the lowest two are held for ablation.
Threads are kept whole (post plus replies in order) rather than shredded into rows; reposts, near-duplicate short posts, and deleted or bot-flagged records are dropped. Q&A with accepted answers is concentrated into the cooldown phase.
One text per paper (best extraction wins), references and running headers stripped, abstracts used where full text is absent. Upweighted in cooldown.
Long, formulaic, and heavily templated — dedup of boilerplate sections is the main filter; exhibits and tables are kept as text so the model sees real filings, not summaries.
The thin slice: READMEs, docs, budgeted top-priority source files, and commit messages. A competitive code share at this budget means licensing in a permissively-licensed code corpus; the design leaves that seat marked rather than filled.
Spoken register, kept small and deduplicated by episode; the slice is sized to what is publicly surfaced today.
Small, clean, and seen more than once: the classic multi-epoch reference slice.
Rows serialized into short dated sentences so prices, resolutions, and filings arrive as text with their timestamps — a dense numeracy and calendar signal for little of the budget.
Real documents from the archive: the raw capture on the left, what the model would train on on the right. Every character is quoted from a live surface — nothing is reconstructed.
Extraction and boilerplate removal · commoncrawl.distillate — tier-A HTML to tier-B clean text · mail.python.org distutils-sig archive, 11 April 2018 (public list archive)
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML><HEAD>
<TITLE> [Distutils] How to install examples files?</TITLE>
<LINK REL="Index" HREF="index.html">
<style type="text/css"> pre { white-space: pre-wrap; } </style>
<META http-equiv="Content-Type" content="text/html; charset=us-ascii">
</HEAD>
<BODY BGCOLOR="#ffffff">
<H1>[Distutils] How to install examples files?</H1>
<B>Thomas Kluyver</B>
<A HREF="mailto:distutils-sig%40python.org?Subject=Re%3A...&In-Reply-To=%3C...%3E">thomas at kluyver.me.uk</A><BR>
<I>Wed Apr 11 04:04:19 EDT 2018</I>
<P><UL>
<LI>Previous message (by thread): <A HREF="032142.html">[Distutils] How to install examples files?</A></LI>
<LI>Next message (by thread): <A HREF="032144.html">[Distutils] How to install examples files?</A></LI>
<LI><B>Messages sorted by:</B>
<a href="date.html#32145">[ date ]</a> <a href="thread.html#32145">[ thread ]</a>
<a href="subject.html#32145">[ subject ]</a> <a href="author.html#32145">[ author ]</a></LI>
</UL><HR>
<!--beginarticle--><PRE>If I recall correctly, 'pip install --target' works by installing
into a temporary directory, then copying only the library part ...If I recall correctly, 'pip install --target' works by installing into a temporary directory, then copying only the library part to the target directory. So it will throw away any docs/examples/scripts that would be installed outside the importable package. On Tue, Apr 10, 2018, at 10:08 PM, Michael Schwager wrote: > It looks like the following setup.py will do what I want, but it won't > install the examples when I use --target:
What the stage removed:
8,029 bytes of HTML became 3,981 bytes of text — about half of the capture was markup and navigation the model never needs to see. The same extraction runs on scholarly PDFs, where the cuts are deeper: running headers, hyphenated line-breaks, and reference lists routinely take a raw paper from ~120K characters to ~27K of clean prose.
Public mailing-list archive; a short excerpt shown to illustrate the extraction step.
The same denominators this page reports, reproduced from the unified text surface.
SELECT source, count() AS docs FROM internet.text GROUP BY source ORDER BY docs DESC
Everything in a slice after a cutoff date — the shape of a leakage-free evaluation split.
SELECT id, ts, author, title, left(text, 400) AS excerpt FROM internet.text WHERE source = 'hackernews' AND ts >= '2026-01-01' ORDER BY ts DESC LIMIT 20
A Common Crawl slice narrowed the way the dedup stage sees it: one page text per URL, latest capture wins.
SELECT url, argMax(text, ts) AS text FROM crawl.pages WHERE host = 'lesswrong.com' AND status = 200 GROUP BY url LIMIT 20