# Stage-5: Unknown-format discovery by Bayesian header clustering

Status: design (not yet implemented). Product of a design interview; captures the
decisions and — as important — the assumptions we *rejected* so they don't get
silently reintroduced.

## 1. Goal

Discover **recurring new file formats** hiding in the `binary/` bucket (the
`:unknown` sink from `classify.jl` → stage-3 triage). A genuinely novel format is
a plausible proxy for a genuinely novel producing application, but we do **not**
try to identify producers directly (see §3). The output is a **growing catalog of
discovered formats**, each with a magic-byte signature that can be promoted into
the classifier's fast path.

Task shape (settled): **unsupervised clustering with an unknown number of
clusters.** Not pairwise "same producer" scoring, not classification against a
fixed label set.

## 2. Two phases — build (A) then run (B)

**(A) Batch, offline — the science.** Cluster the accumulated pile from scratch.
Its job is *not* to be the catalog; it is to (i) prove the header-byte signal
actually separates formats, cross-checked against an NCD baseline (§8), and
(ii) **calibrate hyperparameters** against known formats (§7). Ship this first —
it de-risks (B). If (A)'s clusters are garbage, (B)'s machinery is wasted.

**(B) Online, live — the catalog.** The target deliverable. A persistent catalog
where each discovered format has a **durable, frozen ID** and stored sufficient
statistics. New unknown files are scored against existing clusters; only
genuinely novel ones spawn a new entry. Clusters that accumulate enough evidence
are **nominated for promotion** into the classifier (§6).

## 3. What we are and are NOT clustering

We cluster by **file format**, not by producer. The first-*n* header bytes are
format-mandated and producer-invariant: every valid PNG shares the same magic
regardless of which program wrote it; a PDF's producer string lives deep inside
the file, not in the header. Producer identity, where recoverable at all, is
`exiftool`'s job (stage 2), not this stage's.

Corollary already visible in `../training_set`: extension labels are **not**
header-format labels. `docx` *is* a PK zip; `so`/`o`/`elf`/`out` are all ELF.
Merging those is **correct**, not error (see §7).

## 4. Model: DP mixture of per-position categoricals

A cluster is a **product of independent per-position categorical distributions**
over the first *n* header bytes. Position *i* carries a distribution `θᵢ` over a
**257-symbol alphabet**: byte values `0–255`, plus symbol `256 = "past EOF"`.

- Invariant positions (magic bytes) learn a spiked `θᵢ`; variable positions
  (lengths, timestamps) learn a flat one. A cluster's signature = the vector of
  modal symbols + per-position peakedness. That signature **is a magic-number
  template** — this is the entire reason for the categorical choice.
- `257` alphabet handles short files honestly: a format that is always 20 bytes
  produces a spiked "past-EOF" at positions 20–31, which is real, discriminative
  signal. No zero-padding (would collide `0x00` padding with real `0x00` bytes).

**Priors:** Dirichlet on each `θᵢ` (conjugate to Categorical); **Dirichlet
process (CRP)** over cluster assignments → unknown *k* falls out natively.

**Why categorical, not Euclidean.** Bytes are categorical, not ordinal: `0x89`
and `0x88` are not "close," `0x00` and `0xFF` are not "far." k-means / Gaussian
mixtures over scaled bytes assert a metric that does not exist in header space.
**Do not reuse `model.jl`'s `[0,1]` byte scaling here** — that scaling is correct
for the Lux net and wrong for this model. We need the raw `0–255` byte as a
categorical index.

### 4a. Background component (high-entropy handling)

Add a fixed, **non-adaptive uniform component** (each position uniform over 257)
as the "junk drawer." Compressed/encrypted/structureless blobs are ~uniform after
any magic and would otherwise either (i) mint a singleton per file or (ii)
collapse into one flat cluster that then matches everything. The background
absorbs them cleanly.

Two populations, to be precise:
- **Structured prefix + random tail** (gzip `1f 8b`, PK zip, zstd, most encrypted
  *containers*): peaked at positions 0–3, flat after. These form **real clusters
  for free** — genuine discoveries, no special handling.
- **Uniform from byte 0** (raw encrypted streams, key material): nothing in the
  header to cluster on → absorbed by background.

The background is **never promotable**. But it is **not a silent sink**: its
size / growth / entropy histogram is surfaced as a first-class signal ("12% of
this week's unknowns are structureless"). If sub-clustering the structureless
residue ever matters, that needs a *different* feature (byte histogram / entropy),
a separate v3 model — header bytes genuinely cannot do it.

### 4b. Feature window

**Front-only, `n = 32`** (config knob; try 64 if under-resolved). Magic lives at
offset 0. Tail window **deferred to v2** — a minority of formats have trailers
(ZIP EOCD, ID3v1, PDF `%%EOF`); add as an independent *second block* of positions
only if real trailer-formats show up in the residue.

**Known blind spot: tar.** `ustar` magic is at **offset 257**, outside the
window, so all 100 training tars scatter to background. Accepted for v1 — tar is
already a *known* format, so discovery doesn't need it. General lesson: a minority
of formats put magic at a fixed deeper offset; the fix (if ever needed) is a
**sparse probe window** at that offset (e.g. bytes 257–262 as a third block), not
densely modeling 257 front bytes — that would 8× every cluster's `n×257`
sufficient-stat table to catch one format.

## 5. Inference: different mode per phase (resolves the Bayesian-vs-catalog tension)

A sampler yields a *posterior over partitions*; a catalog needs *one partition
with durable IDs*. Two MCMC gotchas: **label switching** (cluster #3 is not a
stable identity across iterations/runs) and **distribution-not-answer** (1000
partitions, not one). We sidestep both by using two inference modes:

- **Phase (A), offline:** full **collapsed Gibbs** sampler over the
  Dirichlet-Categorical (conjugacy → ~100 lines, no continuous approximation,
  unknown *k* native). Used to validate signal, tune `α` + Dirichlet strength,
  and seed the initial catalog (summarize to a point partition **once**, via a
  VI/Binder loss over the posterior similarity matrix — tolerated because it is
  offline, never in the hot path).
- **Phase (B), live:** **deterministic sequential CRP-predictive assignment.**
  Each catalog cluster stores per-position 257-count vectors (sufficient stats).
  A new file's CRP predictive probability of joining each existing cluster vs.
  the background vs. spawning a new cluster is computed; assign to the argmax.
  A new cluster is minted only if the new-cluster evidence beats the background
  by a margin. **IDs are frozen at birth → no label switching.** This is exactly
  the Gibbs predictive rule with existing assignments held fixed — same math, not
  an ad-hoc hack.
- **Periodic compaction, offline:** re-run Gibbs seeded from the current catalog
  to merge drifted clusters / split bloated ones.

## 6. Promotion (closing the loop to the classifier)

**Layered known-check at ingest** becomes:
1. Match against **promoted signatures** (exact, fast) — runs *before* the net.
2. Else the Lux `:known` / `:unknown` classifier.
3. Else route to `binary/` for this stage.

**Promotion = append a magic-byte signature to a registry.** A cluster's spiked
positions (posterior max-prob `> ~0.9`) become required bytes; flat positions
become wildcards — a libmagic-style signature. This is a **data change, not a
retrain**; interpretable, auditable, reversible. Retraining the Lux net is a
separate, *optional periodic* activity using accumulated signature-labeled files,
never the promotion mechanism itself.

**Nominate automatically, activate by hand.** A cluster crossing thresholds —
`≥ N` members (start `N ≈ 20–50`, loose dial since a human is the backstop) **and**
`≥ ~3` magic positions **and** not the background — is written to a `nominated/`
registry with its signature, member count, and example files. A human glance
promotes it into the active set. Human gate guards the one hard-to-reverse action
(redefining "known"); everything upstream stays automatic.

## 7. Calibration: recover known formats, then trust on unknowns

Do not pick priors blind. We have ground truth: `../training_set` (100 each of
tgz/tar/pdf/docx, 98 zip, 93 jpg, ELF family) and the `data/done` corpus.

1. Run **labeled known files** through the exact clustering pipeline.
2. Ground truth = **magic-collapsed classes**, *not* extensions:
   `{gzip (tgz), PKzip (docx≡zip), ELF (so/o/elf/out/x86_64), JPEG, PDF, tar}`.
   Merging docx+zip and the ELF family is the **correct** answer — scoring
   against raw extensions would penalize correctness and mistune `α`.
3. Measure recovered-vs-truth agreement with **Adjusted Rand Index / V-measure**.
4. **Grid-tune `α` and the Dirichlet pseudocount to maximize agreement** — the
   settings at which the machine rediscovers formats we already know.
5. Freeze, deploy on the `:unknown` pile.

Splitting docx from zip is a **later tier**: the discriminating info
(central-directory filenames like `word/document.xml`) sits at a *variable
offset*, not a fixed position — a different feature problem, deferred.

## 8. Julia package surface

- **Hand-rolled collapsed Gibbs** over Dirichlet-Categorical — recommended. The
  conjugacy makes it short/fast; we own the online + promotion logic; no library
  impedance. `Distributions.jl` for `Dirichlet`/`Categorical` primitives.
- **`CodecZlib`** for the **NCD (Normalized Compression Distance)** baseline —
  model-free gzip-similarity clustering. Excellent at format grouping and a great
  step-(A) sanity check, but O(N²), non-generative (no signature, no online
  scoring, no promotion) → **baseline only, cannot be the catalog.**
- **`Clustering.jl`** — `randindex` / `vmeasure` for the §7 calibration metric,
  plus a throwaway k-modes-ish baseline. **Not** the real model (its k-means is
  the Euclidean trap of §4).
- **`Turing.jl`** — considered and rejected: discrete assignment latents + DP are
  awkward, and we'd still hand-roll the online path. Overkill.

## 9. Architecture: single-owner batch stage, NOT inline inference

The classifier is stateless, immutable, shared read-only across worker threads
(see `classify.jl`). **The catalog is the opposite: mutable, learned, shared** —
every assigned file updates a cluster's counts. It therefore must **not** copy the
classifier's concurrency model (concurrent workers → lock contention, torn reads
of sufficient stats, CRP assignment against stale mass).

Because **promotion is human-gated, nothing here is latency-sensitive.** So:

- Workers stay stateless — they deposit `:unknown` files into `binary/` exactly as
  today. **No catalog access on the hot path.**
- A **separate stage-5 process** (periodic / cron, single-threaded) owns the
  catalog **exclusively**: sweeps newly-arrived `binary/` files, runs sequential
  CRP-predictive assignment, updates sufficient stats, writes nominations.
  **One writer, zero locks, no cross-thread shared mutable state.**
- The catalog is a **durable file** mutated by one process — reuse the stage-2
  **sidecar-first durable-commit** discipline (`commit_enriched!`: temp write →
  fsync bytes → rename → fsync dir) so a crash can't corrupt it or lose a rename.

This slots in as a batch stage, matching how stages 2/3/4 already work. New
`Config` knobs follow the existing `FS_*` env-override convention (e.g.
`FS_CLUSTER_DIR`, `FS_CLUSTER_N`, `FS_CLUSTER_ALPHA`, `FS_CLUSTER_PSEUDOCOUNT`,
`FS_PROMOTE_MIN_MEMBERS`).

## 10. Concrete test assertions (write these first)

1. **Discovers nothing from noise.** Current `data/binary` = 20 small random
   blobs + 1 giant PDF. Correct output: PDF is a singleton that **never promotes**
   (N=1), 20 blobs absorbed by background, **zero promoted clusters.** Any
   promoted cluster from this pile = broken.
2. **Recovers known formats.** On a `../training_set` sample, calibrated settings
   cluster into the ~6 magic-collapsed classes with high ARI (docx+zip merged,
   ELF family merged, tar in background as the accepted blind spot).
3. **NCD agreement.** Step-(A) Bayesian clusters broadly agree with the NCD
   baseline on the same input; large disagreement is a red flag to investigate
   before trusting the generative model.

## Open items (deferred, intentionally)

- v2: tail-window block; sparse deep-offset probe (tar-class).
- v3: sub-clustering structureless high-entropy residue (needs entropy/histogram
  feature, not header bytes).
- Later tier: docx-vs-zip split via variable-offset central-directory names.
- Periodic Lux retrain from accumulated signature-labeled files.