The calculus of search

Scry’s corpora arrive as relations: hackernews.items, reddit.posts, openalex.works, the historical Twitter archive’s reply and quote graphs. A query over relations yields a relation, so every result is a legitimate input to the next query — a CTE, a subquery, a named set in a fixpoint program. That closure is the entire idea: the unit of work is the construction of new relations from old ones, and each construction is immediately raw material for the next. Below, storytellers is a relation that exists only because a query defined it, queried in the same statement like any base table (1.1 s over 90M rows on the live engine).

WITH storytellers AS (      -- a relation defined by a query
  SELECT original_author AS author
  FROM hackernews.items
  WHERE hn_type = 'story' AND upvotes > 500
  GROUP BY author HAVING count() >= 10
)
SELECT i.title, i.upvotes     -- queried like any base relation
FROM hackernews.items AS i
JOIN storytellers AS s ON i.original_author = s.author
WHERE i.hn_type = 'story'
ORDER BY i.upvotes DESC LIMIT 5

A conjunctive query names every x for which a pattern of facts holds, with intermediate entities that are existential — they must exist, they need not be returned. In SQL the conjunction is a join; in a fixpoint program it is a body, where every atom (rel, edge, filter, in) further constrains one walk. This is the workhorse layer of structured search — relational pattern matching over the corpus, well past keyword retrieval, and the layer the engine optimizes best. The query below runs the pattern as written: every author whose post or comment drew a reply from Hacker News’s moderator, in ~240 ms.

Q(a) ← post(p, a) ∧ reply(r, p) ∧ author(r, 'dang')

SELECT DISTINCT p.original_author
FROM hackernews.items AS p
JOIN hackernews.items AS r ON r.parent_hn_id = p.hn_id
WHERE r.original_author = 'dang' AND p.original_author != ''
LIMIT 12

Real concepts rarely have one structural definition, so the calculus gives a relation several bodies: each body a conjunctive clause, the relation their union. Below, context is defined twice — the papers ResNet cites, and the papers citing it — and the engine returns the union: 131 works, both branches contributing, two metered statements, empty truncations. Each branch stays precise while the union names the concept, which is a different philosophy from embedding similarity, where a concept is whatever lands nearby: a relation defined as a union of explicit patterns is inspectable — read its definition, delete a branch, and know exactly what changed.

{"program": {"relations": {
  "seed":    {"bodies": [[{"ids": ["https://openalex.org/W2194775991"]}]]},
  "context": {"bodies": [
      [{"rel": "seed"}, {"edge": "references"}],
      [{"rel": "seed"}, {"edge": "cited_by"}]
  ], "emit": "counts"}
}, "out": ["context"], "depth": 1}}

Many of the sharpest questions are contrastive — who writes about a field yet sits outside the population that obviously writes about it. SQL carries EXCEPT and NOT EXISTS; programs carry not_in, stratified negation applied while the walk runs, so an excluded node is never expanded and never billed. Stratification is what keeps negation coherent under recursion: the subtracted relation is fully evaluated before the subtraction, so a definition can say “candidates not already established” and can never say “good is whatever is not good”.

novel(x) ← candidate(x) ∧ ¬established(x)   -- established evaluated first

{"bodies": [[{"rel": "candidate"}, {"not_in": "established"}]]}

A relation can be defined in terms of itself: reachable is the seeds, plus everything one edge past what is already reachable. The least fixed point μ is the meaning of that definition — start from the seeds, apply the rule, keep what is new, stop at the round that discovers nothing — so the walk finds its own depth instead of a hop count you guessed. WITH RECURSIVE runs μ over computed sequences; a fixpoint program runs it over the graph corpora semi-naively, expanding only each round’s frontier. The envelope’s truncations list is μ’s honest signature: it names every resource bound that fired, and an empty list means the relation you hold is the true least fixed point of the rules you wrote.

reachable = μX. seed ∪ step(X)

round 1:  seed
round 2:  seed ∪ step(seed)
round 3:  seed ∪ step(seed) ∪ step²(seed)
…until a round adds nothing new

A walked row returns as {id, parent, depth} — a witness to how the walk reached it: the node that discovered it, at which round. Derivations change what an agent can do with results: inspect the path that admitted a row, notice a branch doing the wrong work, subtract it with not_in, re-run — the derivation is the debugging surface for the query itself. Aggregate provenance rides the same envelope: per-depth counts for every output relation, so when the shape of a population is the answer, it arrives whole at zero row egress.

Semantic judgment enters the calculus as ordinary relations. An embedding handle minted at /v1/scry/embed becomes an ann atom — a semantic neighborhood as a seed set — and rank orders a derived set by exact cosine distance to a handle; on the SQL plane the registered vector helpers do the same inside a SELECT. The division of labor is the design: models propose the fuzzy predicates only they can express, and the relational engine composes them exactly — joins, unions, negation, fixpoints — over the corpus. Fuzzy semantics at the leaves, crisp composition above them, and the whole construction stays inspectable.

Each symbol added to the calculus buys a qualitatively new class of questions, and every Scry query is a point on this ladder. Traditional search asks an index a question; a query in this calculus constructs a computation over the index. The operational contract — measured latencies, the twelve registered edges, and the deadline, budget, and depth bounds you set — is on the Turing-complete search page.