Add design doc for stage-5 unknown-format clustering
This commit is contained in:
229
model/DESIGN_clustering.md
Normal file
229
model/DESIGN_clustering.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# Stage-5: Unknown-format discovery by Bayesian header clustering
|
||||
|
||||
Status: design (not yet implemented). Product of a design interview; captures the
|
||||
decisions and — as important — the assumptions we *rejected* so they don't get
|
||||
silently reintroduced.
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Discover **recurring new file formats** hiding in the `binary/` bucket (the
|
||||
`:unknown` sink from `classify.jl` → stage-3 triage). A genuinely novel format is
|
||||
a plausible proxy for a genuinely novel producing application, but we do **not**
|
||||
try to identify producers directly (see §3). The output is a **growing catalog of
|
||||
discovered formats**, each with a magic-byte signature that can be promoted into
|
||||
the classifier's fast path.
|
||||
|
||||
Task shape (settled): **unsupervised clustering with an unknown number of
|
||||
clusters.** Not pairwise "same producer" scoring, not classification against a
|
||||
fixed label set.
|
||||
|
||||
## 2. Two phases — build (A) then run (B)
|
||||
|
||||
**(A) Batch, offline — the science.** Cluster the accumulated pile from scratch.
|
||||
Its job is *not* to be the catalog; it is to (i) prove the header-byte signal
|
||||
actually separates formats, cross-checked against an NCD baseline (§8), and
|
||||
(ii) **calibrate hyperparameters** against known formats (§7). Ship this first —
|
||||
it de-risks (B). If (A)'s clusters are garbage, (B)'s machinery is wasted.
|
||||
|
||||
**(B) Online, live — the catalog.** The target deliverable. A persistent catalog
|
||||
where each discovered format has a **durable, frozen ID** and stored sufficient
|
||||
statistics. New unknown files are scored against existing clusters; only
|
||||
genuinely novel ones spawn a new entry. Clusters that accumulate enough evidence
|
||||
are **nominated for promotion** into the classifier (§6).
|
||||
|
||||
## 3. What we are and are NOT clustering
|
||||
|
||||
We cluster by **file format**, not by producer. The first-*n* header bytes are
|
||||
format-mandated and producer-invariant: every valid PNG shares the same magic
|
||||
regardless of which program wrote it; a PDF's producer string lives deep inside
|
||||
the file, not in the header. Producer identity, where recoverable at all, is
|
||||
`exiftool`'s job (stage 2), not this stage's.
|
||||
|
||||
Corollary already visible in `../training_set`: extension labels are **not**
|
||||
header-format labels. `docx` *is* a PK zip; `so`/`o`/`elf`/`out` are all ELF.
|
||||
Merging those is **correct**, not error (see §7).
|
||||
|
||||
## 4. Model: DP mixture of per-position categoricals
|
||||
|
||||
A cluster is a **product of independent per-position categorical distributions**
|
||||
over the first *n* header bytes. Position *i* carries a distribution `θᵢ` over a
|
||||
**257-symbol alphabet**: byte values `0–255`, plus symbol `256 = "past EOF"`.
|
||||
|
||||
- Invariant positions (magic bytes) learn a spiked `θᵢ`; variable positions
|
||||
(lengths, timestamps) learn a flat one. A cluster's signature = the vector of
|
||||
modal symbols + per-position peakedness. That signature **is a magic-number
|
||||
template** — this is the entire reason for the categorical choice.
|
||||
- `257` alphabet handles short files honestly: a format that is always 20 bytes
|
||||
produces a spiked "past-EOF" at positions 20–31, which is real, discriminative
|
||||
signal. No zero-padding (would collide `0x00` padding with real `0x00` bytes).
|
||||
|
||||
**Priors:** Dirichlet on each `θᵢ` (conjugate to Categorical); **Dirichlet
|
||||
process (CRP)** over cluster assignments → unknown *k* falls out natively.
|
||||
|
||||
**Why categorical, not Euclidean.** Bytes are categorical, not ordinal: `0x89`
|
||||
and `0x88` are not "close," `0x00` and `0xFF` are not "far." k-means / Gaussian
|
||||
mixtures over scaled bytes assert a metric that does not exist in header space.
|
||||
**Do not reuse `model.jl`'s `[0,1]` byte scaling here** — that scaling is correct
|
||||
for the Lux net and wrong for this model. We need the raw `0–255` byte as a
|
||||
categorical index.
|
||||
|
||||
### 4a. Background component (high-entropy handling)
|
||||
|
||||
Add a fixed, **non-adaptive uniform component** (each position uniform over 257)
|
||||
as the "junk drawer." Compressed/encrypted/structureless blobs are ~uniform after
|
||||
any magic and would otherwise either (i) mint a singleton per file or (ii)
|
||||
collapse into one flat cluster that then matches everything. The background
|
||||
absorbs them cleanly.
|
||||
|
||||
Two populations, to be precise:
|
||||
- **Structured prefix + random tail** (gzip `1f 8b`, PK zip, zstd, most encrypted
|
||||
*containers*): peaked at positions 0–3, flat after. These form **real clusters
|
||||
for free** — genuine discoveries, no special handling.
|
||||
- **Uniform from byte 0** (raw encrypted streams, key material): nothing in the
|
||||
header to cluster on → absorbed by background.
|
||||
|
||||
The background is **never promotable**. But it is **not a silent sink**: its
|
||||
size / growth / entropy histogram is surfaced as a first-class signal ("12% of
|
||||
this week's unknowns are structureless"). If sub-clustering the structureless
|
||||
residue ever matters, that needs a *different* feature (byte histogram / entropy),
|
||||
a separate v3 model — header bytes genuinely cannot do it.
|
||||
|
||||
### 4b. Feature window
|
||||
|
||||
**Front-only, `n = 32`** (config knob; try 64 if under-resolved). Magic lives at
|
||||
offset 0. Tail window **deferred to v2** — a minority of formats have trailers
|
||||
(ZIP EOCD, ID3v1, PDF `%%EOF`); add as an independent *second block* of positions
|
||||
only if real trailer-formats show up in the residue.
|
||||
|
||||
**Known blind spot: tar.** `ustar` magic is at **offset 257**, outside the
|
||||
window, so all 100 training tars scatter to background. Accepted for v1 — tar is
|
||||
already a *known* format, so discovery doesn't need it. General lesson: a minority
|
||||
of formats put magic at a fixed deeper offset; the fix (if ever needed) is a
|
||||
**sparse probe window** at that offset (e.g. bytes 257–262 as a third block), not
|
||||
densely modeling 257 front bytes — that would 8× every cluster's `n×257`
|
||||
sufficient-stat table to catch one format.
|
||||
|
||||
## 5. Inference: different mode per phase (resolves the Bayesian-vs-catalog tension)
|
||||
|
||||
A sampler yields a *posterior over partitions*; a catalog needs *one partition
|
||||
with durable IDs*. Two MCMC gotchas: **label switching** (cluster #3 is not a
|
||||
stable identity across iterations/runs) and **distribution-not-answer** (1000
|
||||
partitions, not one). We sidestep both by using two inference modes:
|
||||
|
||||
- **Phase (A), offline:** full **collapsed Gibbs** sampler over the
|
||||
Dirichlet-Categorical (conjugacy → ~100 lines, no continuous approximation,
|
||||
unknown *k* native). Used to validate signal, tune `α` + Dirichlet strength,
|
||||
and seed the initial catalog (summarize to a point partition **once**, via a
|
||||
VI/Binder loss over the posterior similarity matrix — tolerated because it is
|
||||
offline, never in the hot path).
|
||||
- **Phase (B), live:** **deterministic sequential CRP-predictive assignment.**
|
||||
Each catalog cluster stores per-position 257-count vectors (sufficient stats).
|
||||
A new file's CRP predictive probability of joining each existing cluster vs.
|
||||
the background vs. spawning a new cluster is computed; assign to the argmax.
|
||||
A new cluster is minted only if the new-cluster evidence beats the background
|
||||
by a margin. **IDs are frozen at birth → no label switching.** This is exactly
|
||||
the Gibbs predictive rule with existing assignments held fixed — same math, not
|
||||
an ad-hoc hack.
|
||||
- **Periodic compaction, offline:** re-run Gibbs seeded from the current catalog
|
||||
to merge drifted clusters / split bloated ones.
|
||||
|
||||
## 6. Promotion (closing the loop to the classifier)
|
||||
|
||||
**Layered known-check at ingest** becomes:
|
||||
1. Match against **promoted signatures** (exact, fast) — runs *before* the net.
|
||||
2. Else the Lux `:known` / `:unknown` classifier.
|
||||
3. Else route to `binary/` for this stage.
|
||||
|
||||
**Promotion = append a magic-byte signature to a registry.** A cluster's spiked
|
||||
positions (posterior max-prob `> ~0.9`) become required bytes; flat positions
|
||||
become wildcards — a libmagic-style signature. This is a **data change, not a
|
||||
retrain**; interpretable, auditable, reversible. Retraining the Lux net is a
|
||||
separate, *optional periodic* activity using accumulated signature-labeled files,
|
||||
never the promotion mechanism itself.
|
||||
|
||||
**Nominate automatically, activate by hand.** A cluster crossing thresholds —
|
||||
`≥ N` members (start `N ≈ 20–50`, loose dial since a human is the backstop) **and**
|
||||
`≥ ~3` magic positions **and** not the background — is written to a `nominated/`
|
||||
registry with its signature, member count, and example files. A human glance
|
||||
promotes it into the active set. Human gate guards the one hard-to-reverse action
|
||||
(redefining "known"); everything upstream stays automatic.
|
||||
|
||||
## 7. Calibration: recover known formats, then trust on unknowns
|
||||
|
||||
Do not pick priors blind. We have ground truth: `../training_set` (100 each of
|
||||
tgz/tar/pdf/docx, 98 zip, 93 jpg, ELF family) and the `data/done` corpus.
|
||||
|
||||
1. Run **labeled known files** through the exact clustering pipeline.
|
||||
2. Ground truth = **magic-collapsed classes**, *not* extensions:
|
||||
`{gzip (tgz), PKzip (docx≡zip), ELF (so/o/elf/out/x86_64), JPEG, PDF, tar}`.
|
||||
Merging docx+zip and the ELF family is the **correct** answer — scoring
|
||||
against raw extensions would penalize correctness and mistune `α`.
|
||||
3. Measure recovered-vs-truth agreement with **Adjusted Rand Index / V-measure**.
|
||||
4. **Grid-tune `α` and the Dirichlet pseudocount to maximize agreement** — the
|
||||
settings at which the machine rediscovers formats we already know.
|
||||
5. Freeze, deploy on the `:unknown` pile.
|
||||
|
||||
Splitting docx from zip is a **later tier**: the discriminating info
|
||||
(central-directory filenames like `word/document.xml`) sits at a *variable
|
||||
offset*, not a fixed position — a different feature problem, deferred.
|
||||
|
||||
## 8. Julia package surface
|
||||
|
||||
- **Hand-rolled collapsed Gibbs** over Dirichlet-Categorical — recommended. The
|
||||
conjugacy makes it short/fast; we own the online + promotion logic; no library
|
||||
impedance. `Distributions.jl` for `Dirichlet`/`Categorical` primitives.
|
||||
- **`CodecZlib`** for the **NCD (Normalized Compression Distance)** baseline —
|
||||
model-free gzip-similarity clustering. Excellent at format grouping and a great
|
||||
step-(A) sanity check, but O(N²), non-generative (no signature, no online
|
||||
scoring, no promotion) → **baseline only, cannot be the catalog.**
|
||||
- **`Clustering.jl`** — `randindex` / `vmeasure` for the §7 calibration metric,
|
||||
plus a throwaway k-modes-ish baseline. **Not** the real model (its k-means is
|
||||
the Euclidean trap of §4).
|
||||
- **`Turing.jl`** — considered and rejected: discrete assignment latents + DP are
|
||||
awkward, and we'd still hand-roll the online path. Overkill.
|
||||
|
||||
## 9. Architecture: single-owner batch stage, NOT inline inference
|
||||
|
||||
The classifier is stateless, immutable, shared read-only across worker threads
|
||||
(see `classify.jl`). **The catalog is the opposite: mutable, learned, shared** —
|
||||
every assigned file updates a cluster's counts. It therefore must **not** copy the
|
||||
classifier's concurrency model (concurrent workers → lock contention, torn reads
|
||||
of sufficient stats, CRP assignment against stale mass).
|
||||
|
||||
Because **promotion is human-gated, nothing here is latency-sensitive.** So:
|
||||
|
||||
- Workers stay stateless — they deposit `:unknown` files into `binary/` exactly as
|
||||
today. **No catalog access on the hot path.**
|
||||
- A **separate stage-5 process** (periodic / cron, single-threaded) owns the
|
||||
catalog **exclusively**: sweeps newly-arrived `binary/` files, runs sequential
|
||||
CRP-predictive assignment, updates sufficient stats, writes nominations.
|
||||
**One writer, zero locks, no cross-thread shared mutable state.**
|
||||
- The catalog is a **durable file** mutated by one process — reuse the stage-2
|
||||
**sidecar-first durable-commit** discipline (`commit_enriched!`: temp write →
|
||||
fsync bytes → rename → fsync dir) so a crash can't corrupt it or lose a rename.
|
||||
|
||||
This slots in as a batch stage, matching how stages 2/3/4 already work. New
|
||||
`Config` knobs follow the existing `FS_*` env-override convention (e.g.
|
||||
`FS_CLUSTER_DIR`, `FS_CLUSTER_N`, `FS_CLUSTER_ALPHA`, `FS_CLUSTER_PSEUDOCOUNT`,
|
||||
`FS_PROMOTE_MIN_MEMBERS`).
|
||||
|
||||
## 10. Concrete test assertions (write these first)
|
||||
|
||||
1. **Discovers nothing from noise.** Current `data/binary` = 20 small random
|
||||
blobs + 1 giant PDF. Correct output: PDF is a singleton that **never promotes**
|
||||
(N=1), 20 blobs absorbed by background, **zero promoted clusters.** Any
|
||||
promoted cluster from this pile = broken.
|
||||
2. **Recovers known formats.** On a `../training_set` sample, calibrated settings
|
||||
cluster into the ~6 magic-collapsed classes with high ARI (docx+zip merged,
|
||||
ELF family merged, tar in background as the accepted blind spot).
|
||||
3. **NCD agreement.** Step-(A) Bayesian clusters broadly agree with the NCD
|
||||
baseline on the same input; large disagreement is a red flag to investigate
|
||||
before trusting the generative model.
|
||||
|
||||
## Open items (deferred, intentionally)
|
||||
|
||||
- v2: tail-window block; sparse deep-offset probe (tar-class).
|
||||
- v3: sub-clustering structureless high-entropy residue (needs entropy/histogram
|
||||
feature, not header bytes).
|
||||
- Later tier: docx-vs-zip split via variable-offset central-directory names.
|
||||
- Periodic Lux retrain from accumulated signature-labeled files.
|
||||
Reference in New Issue
Block a user