Add stage-5 unknown-format discovery: header clustering + calibration

Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process
mixture of per-position categoricals over the first 32 header bytes (257-symbol
alphabet) that clusters the binary/ pile by file format, plus signature
extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps
the Dirichlet-multinomial marginal dependency-free).

- src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler
  (phase A), sequential CRP-predictive assignment (phase B core), signatures/
  promotion, and ARI/V-measure calibration metrics.
- bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed
  ground truth and cross-checks a model-free NCD (gzip) baseline.
- FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module.
- Tests for the three DESIGN §10 assertions plus the model primitives.

Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at
ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the
NCD baseline agrees. DESIGN §11 records the results and three assumptions the
data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and
the Bernoulli signature / Occam-penalized restart scoring).

model/DESIGN_clustering.md

@@ -1,8 +1,12 @@
 # Stage-5: Unknown-format discovery by Bayesian header clustering
 
-Status: design (not yet implemented). Product of a design interview; captures the
-decisions and — as important — the assumptions we *rejected* so they don't get
-silently reintroduced.
+Status: **phase A implemented and calibrated** (`src/cluster.jl`,
+`bin/cluster_calibrate.jl`, tests in `test/runtests.jl`); phase-B core scoring
+implemented (`assign_file`), its live batch-process plumbing still to do. Product
+of a design interview; captures the decisions and — as important — the
+assumptions we *rejected* so they don't get silently reintroduced. §11 records
+what building it actually taught us, including three assumptions in this document
+that the data corrected.
 
 ## 1. Goal
 
@@ -220,8 +224,80 @@ This slots in as a batch stage, matching how stages 2/3/4 already work. New
    baseline on the same input; large disagreement is a red flag to investigate
    before trusting the generative model.
 
+## 11. Implementation status & calibration results (v1)
+
+**Shipped.** `src/cluster.jl` — feature extraction (`header_symbols`, 257-symbol
+alphabet), collapsed Gibbs (`gibbs_cluster`, phase A), the sequential
+CRP-predictive rule (`assign_file`, phase B core), signatures/promotion
+(`signature`, `is_promotable`), and calibration metrics (`adjusted_rand_index`,
+`v_measure`). All base-Julia — a base-only Lanczos `loggamma` keeps the
+Dirichlet-multinomial marginal dependency-free (no Manifest churn). Config knobs
+`FS_CLUSTER_*` (§9) added. `bin/cluster_calibrate.jl` runs the §7 grid and the §8
+NCD baseline. Concrete §10 assertions are in the test suite (hermetic synthetic
+corpora, so they need neither `../training_set` nor gzip).
+
+**Calibrated defaults** (grid over the 700-file `training_set`, ranked by ARI
+excluding tar): **n=32, α=1.0, β=0.1, bg_mass=5.0** → ARI **0.77** (0.885 excl.
+tar), V-measure 0.83, homogeneity 0.87. Clusters are clean and promotable:
+`pkzip:197` (docx+zip correctly merged, §7.2 ✓), `gzip:100`, `jpeg`, and several
+`pdf` clusters all self-nominate. The **NCD baseline agrees** (§10.3): on a
+150-file subsample, NCD 1-NN label purity 0.90 vs. the model's same-cluster
+purity 0.987 — the generative header model separates formats at least as well as
+model-free gzip similarity.
+
+### Three assumptions the data corrected
+
+1. **Tar is not in the background here; it merges into ELF.** §4b/§7 assumed
+   tar's `ustar`-at-257 magic is out of window so tars scatter to background. But
+   98/100 tars in the corpus are Hex/Elixir package tarballs whose *first
+   archived file is named `VERSION`* → a constant, strongly-peaked `VERSION\0`
+   prefix at offset 0. They do form a peaked cluster — but it **merges with ELF**,
+   because ELF's ident padding and tar's name-field zero-padding give the two a
+   long shared run of `0x00` in bytes 5–31; they differ in only ~3 magic bytes,
+   and 32 equally-weighted positions let ~20 shared zeros outvote 3 real ones. No
+   β both separates ELF/tar and keeps the other formats whole. The honest v1
+   position: this is the *same* "tar is hard" reality §4b flagged, just wearing a
+   different mask. **Fix (v2):** weight positions by inverse entropy so a
+   low-information shared-zero run stops dominating a few high-information magic
+   bytes — this generalizes beyond tar and is the highest-value next lever.
+
+2. **You cannot cold-start every point in the background.** A natural reading of
+   §4a/§5 is "everything starts in the junk drawer, real clusters condense out."
+   That **deadlocks**: at a format's first file a fresh cluster and the background
+   are *both* uniform, so with the `bg_mass ≥ α` that §4a needs for absorption,
+   the background always wins and no cluster is ever seeded. Fix: **initialize
+   every file in its own singleton**; same-format singletons merge and snowball,
+   while a lone random-blob singleton dissolves on resample and is reclaimed by
+   the (stickier) background. Absorption still works — just not as the *initial*
+   state.
+
+3. **Two pieces of math that look optional but aren't.** (a) Signature peakedness
+   is a **Bernoulli** question ("is this position fixed to byte v?"), so it uses a
+   2-way posterior `(count+β)/(members+2β)`, **not** the 257-way mixture
+   predictive — the alphabet-wide denominator drags even a unanimous position
+   below 0.9 once β<1, which would make promotion *impossible*. (b) Ranking Gibbs
+   restarts needs the **collapsed Dirichlet-multinomial marginal** (with its
+   `loggamma` normalizer / Occam penalty); a plain product-of-predictives score
+   omits the penalty and actively **rewards merging** everything into one blob
+   (observed, then fixed).
+
+### Known v1 limitations (accepted)
+
+- **β=0.1 over-splits** PDF and JPEG into several *pure* sub-clusters (e.g. PDF by
+  version byte). This costs completeness/ARI but not the mission: each sub-cluster
+  still carries valid magic and promotes independently, and a human dedupes
+  overlapping `%PDF-1.x` nominations at the gate.
+- The point partition is the **best of N Gibbs restarts by marginal likelihood**,
+  a MAP-style stand-in for the VI/Binder posterior summary §5 defers — adequate
+  because the formats are strongly separated; revisit if compaction (§5) needs it.
+- Phase B's **live single-owner batch process** (§9) and the durable catalog file
+  are not yet built; `assign_file` is the scoring core they will wrap.
+
 ## Open items (deferred, intentionally)
 
+- **v2, now top priority: inverse-entropy position weighting** (unblocks ELF/tar
+  and any format pair that shares a long constant run — see §11).
+
 - v2: tail-window block; sparse deep-offset probe (tar-class).
 - v3: sub-clustering structureless high-entropy residue (needs entropy/histogram
   feature, not header bytes).