What can public data teach when you cross it?
Every named source of public data
Access tiers
| Source | Tier | Section | Notes |
|---|
The Extinct Ward
10,296 pairs, scored. Sets, certified.
Top pairs (fusion)
| # | Pair | Key | Q |
|---|
Gated quality (access + novelty removed)
| # | Pair | Q′ |
|---|
Triangulation channel
| # | Pair | J | T |
|---|
Top triples
| # | Triple | Q |
|---|
The certified optimum: Q*(k)
Spines
A theory of the join
Primitives
The join triangle
| Condition | Question it answers | Kills the join when… |
|---|---|---|
| Co-reference | Are the rows about the same entities? | populations are disjoint (ships × songs; hospitals × foreign agents) |
| Co-resolution | Can codes be aligned at one granularity and vintage? | no crosswalk (CIK↔LEI), granularity slips (tract↔parcel), vintages drift (2010 vs 2020 counties) |
| Complementarity | Do payloads measure different things? | payloads duplicate — you have triangulation or nothing, not fusion |
The seven ways a join lies
Scope algebra — overlap before records
Cell = a-priori population overlap between scopes, computed by the live model (hover a cell for the pair). Diagonals are 1 — except enclave×enclave = 0.05: two vetted enclaves don't link even with themselves' twins, by design.
Keys are institutions
| Key | Registrar | Unique | Coverage mandate | Persistent | Publicly resolvable | Verdict |
|---|---|---|---|---|---|---|
| cnty (FIPS) | US Census | yes | total (US) | decadal drift | yes | gold — the county spine |
| firm (CIK) | SEC | yes | SEC filers only | yes | yes | gold, scoped to public co's |
| firm (LEI) | GLEIF | yes | opt-in (finance) | yes | yes | strong, coverage-limited |
| vessel (IMO) | IMO/IHS | yes | ships ≥100GT | lifetime | yes | gold — enables the dark-fleet cross |
| paper (DOI) | Crossref/DataCite | yes | opt-in, near-total | yes | yes | gold — the paper graph |
| auth (ORCID) | ORCID | yes | opt-in | yes | yes | good, adoption-limited |
| firm (ticker) | exchanges | reused! | listed only | recycled | yes | trap — vintage drift |
| parcel | ~3,100 counties | locally | total, fragmented | varies | tolled ($) | weak nationally — no registrar |
| place (GERS) | Overture (2023–) | yes | growing | promised | yes | an identifier institution being founded in real time |
| work (ISRC/ISWC) | industry bodies | messy | partial | weak | no | why culture crosses score low |
Reference bits — and the privacy duality
| Key | ~population | bits | model strength |
|---|
The duality: data fusion and re-identification are the same operation with different intent. A composite quasi-key (name ≈16b + birthdate ≈13b + ZIP ≈15b ≈ 44 bits) exceeds the ≈28 bits needed to single out one of 335M people — Sweeney's classic 87% result is this arithmetic made empirical. Privacy engineering (k-anonymity, differential privacy, enclaves) is the deliberate practice of keeping assembled bits below the disambiguation threshold. One operation, two professions.
The scaling wall — why only pooling scales
Reference evasion — the adversarial layer
Predictions this theory makes
Field manual for multi-dataset mining
Ten lessons from the practice
Where the research lives
Working bibliography
Seven claims we can't find in the literature
Graded honestly: a new empirical claim · b new operationalization · c new framing of known material.
The Join Audit — how to apply this to a new domain
Worked domain applications
Six domains the census never touched, audited with the same instrument. Each sheet: designation systems → scope traps → best available cross → the missing institution.
Build a cross
Select 2–8 sources to score a cross.
One county, two sources, real numbers
Champaign County, IL — unemployment × active housing listings
| Listings lead (months) | n | r (ΔUR → Δlog listings) |
|---|
Descriptive, one county, no causal claim. The full k=4 protocol pre-registers design, falsifications, and this exact event-window discipline. Access friction measured en route: the NOAA leg sits behind an [R]-tier token a sandboxed agent cannot pass; the dark-fleet cross awaits a free Global Fishing Watch token.
Discoveries
Uncharted, high-value
High-scoring pairs outside the famous set and the worked-to-death blocklist — candidate analyses with no landmark paper yet.
The Prospector
Live log
Field Notes — the shared board
What changed out there
| Source | Finding |
|---|
Extinct Ward — successor pointers
| Source | Died | Where the data went |
|---|
Self-test
The scoring model in this page is a JavaScript port of the Python model that produced every ranking here. Prove it: this runs 85 pair, 40 triple, and 114 set vectors, the IC table, and the pilot correlation tables against Python-computed references.