Source

Query forum archives with SQL

About 36 million posts and comments across roughly 4,300 forum sites — LessWrong, the EA Forum, DEV, DataSecretsLox, crypto-governance forums, and a long crawl-discovered tail — queryable with read-only SQL by site, thread, author, and text.

Surface

  • Records36.3M+ posts + comments
  • Canonicalforums.posts
  • FreshnessLive
  • Extentoriginal_timestamp 1998-09-12 to 2026-08-27
  • Cadencearchived base refreshed in place by live source-native forum tails (hours)
  • MethodPublic forum pages and source-native post records; archived corpus export plus live tails.
  • StatsLast stats snapshot: 2026-08-27T12:11:07Z

Query surfaces

  • forums.posts
  • embeddings.forum_posts

Public forum pages and source-native post records; archived corpus export plus live tails.

Best for

a community’s whole history in one relation — enumerate site_key, then scope
LessWrong and EA Forum posts ranked by upvotes, with authors and timestamps
thread trees via parent_post_key / root_post_key

Fields

  • post_key
  • site_key
  • thread_key
  • source
  • external_id
  • external_type
  • kind
  • post_number
  • parent_post_key
  • root_post_key
  • uri
  • payload
  • blake3_hash
  • author_actor_id
  • original_author
  • author_external_id
  • author_handle
  • author_display_name
  • author_profile_url
  • original_timestamp
  • title
  • upvotes
  • comment_count
  • vote_count
  • word_count
  • metadata
  • quality_score
  • quality_issues
  • is_polished
  • entity_id
  • is_deleted
  • created_at
  • updated_at

Read live from the schema registry for forums.posts — 33 columns. Every relation's full contract is served by /v1/scry/schema.

Indexed predicates

  • PruneshasToken(lower(payload), '<lowercase-token>')
  • PruneshasAllTokens(lower(payload), ['<t1>','<t2>'])

Queries

Which forums are here, by size

One aggregate over site_key shows every community, its post count, and its date range.

SELECT site_key, count() AS posts, min(original_timestamp) AS first_post, max(original_timestamp) AS last_post
FROM forums.posts
WHERE is_deleted = 0
GROUP BY site_key
ORDER BY posts DESC
LIMIT 20;

Most-upvoted LessWrong posts

Scope by source and kind, then rank by the source-native vote count.

SELECT post_key, title, original_author, original_timestamp, upvotes
FROM forums.posts
WHERE source = 'lesswrong'
  AND kind = 'post'
ORDER BY upvotes DESC
LIMIT 20;

Gaps

  • HolesNone declared on the live schema entry for forums.posts (declared holes are curation, not a measurement of completeness).
  • Notesite_key is the community identity; source names only the named ingest lanes, and about half the rows carry source = 'manual' — enumerate site_key before scoping a community question
  • Notesome crawl-discovered sites landed without timestamps (original_timestamp is NULL) — filter them out before date aggregates
  • Notetext search runs on the lowercase token index over payload; without a site_key or source filter it spans every site

Related proof pages

Chart a community's drift along a meaning axis with SQL A Scry query pattern that mints two poles as vectors, takes the balanced axis between them, and averages every LessWrong post's projection quarter by quarter.