Add stage-5 unknown-format discovery: header clustering + calibration
Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process mixture of per-position categoricals over the first 32 header bytes (257-symbol alphabet) that clusters the binary/ pile by file format, plus signature extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps the Dirichlet-multinomial marginal dependency-free). - src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler (phase A), sequential CRP-predictive assignment (phase B core), signatures/ promotion, and ARI/V-measure calibration metrics. - bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed ground truth and cross-checks a model-free NCD (gzip) baseline. - FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module. - Tests for the three DESIGN §10 assertions plus the model primitives. Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the NCD baseline agrees. DESIGN §11 records the results and three assumptions the data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and the Bernoulli signature / Occam-penalized restart scoring).
This commit is contained in:
65
README.md
65
README.md
@@ -149,8 +149,9 @@ or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in
|
||||
UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but
|
||||
not a text control byte, so it still reads as binary. A multi-byte character
|
||||
split by the 8000-byte boundary is trimmed before the check so it isn't mistaken
|
||||
for malformed bytes. An empty file is treated as text. `binary/` is terminal;
|
||||
`text/` is handed to stage 4 (`src/content.jl`).
|
||||
for malformed bytes. An empty file is treated as text. `binary/` is terminal on
|
||||
the live path (but is the input the offline **stage-5 discovery** sweeps — see
|
||||
below); `text/` is handed to stage 4 (`src/content.jl`).
|
||||
|
||||
### Language enrichment (stage 4)
|
||||
|
||||
@@ -199,6 +200,50 @@ Like stage 2, the sidecar is committed **before** the file is moved into
|
||||
`data/text_done/`, so the file's presence there always implies its sidecar is
|
||||
present; recovery re-enriches idempotently (`src/language.jl`).
|
||||
|
||||
### Unknown-format discovery (stage 5, offline)
|
||||
|
||||
The `binary/` sink from stage 3 is the pile of genuinely *unrecognized* files.
|
||||
Stage 5 mines it for **recurring new file formats** by clustering files on their
|
||||
header bytes — a growing catalog of discovered formats, each with a magic-byte
|
||||
signature that can eventually be promoted into the classifier's fast path. Unlike
|
||||
stages 1–4 it is **not on the request hot path**: it is a single-owner *batch*
|
||||
process (the catalog is mutable shared state, the opposite of the stateless
|
||||
classifier), and because promotion is human-gated nothing here is
|
||||
latency-sensitive. The full rationale — and the assumptions we deliberately
|
||||
rejected — live in [`model/DESIGN_clustering.md`](model/DESIGN_clustering.md).
|
||||
|
||||
The model (`src/cluster.jl`, base-Julia, no extra deps) is a Dirichlet-process
|
||||
mixture of **per-position categoricals** over the first 32 header bytes, on a
|
||||
257-symbol alphabet (byte `0–255` plus a `past-EOF` symbol so short fixed-length
|
||||
formats are modeled honestly). Bytes are treated as **categorical, not numeric**
|
||||
— `0x89` and `0x88` are not "close" — so this deliberately does *not* reuse the
|
||||
classifier's `[0,1]` byte scaling. A fixed uniform **background** component
|
||||
absorbs structureless (compressed/encrypted) blobs so they don't mint spurious
|
||||
clusters. A cluster's spiked positions become a libmagic-style signature;
|
||||
clusters with enough members and enough fixed positions self-**nominate** for
|
||||
promotion (a human does the one irreversible step, redefining "known").
|
||||
|
||||
**Status:** the offline science (phase A) is implemented and calibrated; the live
|
||||
catalog process (phase B) is designed and its scoring core (`assign_file`) is in
|
||||
place, but its batch-runner plumbing is not yet built.
|
||||
|
||||
Calibration is its own offline script (like training — never in the request
|
||||
path), scored against magic-collapsed ground truth (so `docx`≡`zip` and the whole
|
||||
ELF family count as one format each, which is the *correct* answer, not an error):
|
||||
|
||||
```bash
|
||||
julia --project=. bin/cluster_calibrate.jl [training_set_dir] # defaults to ../training_set
|
||||
```
|
||||
|
||||
It grid-tunes the hyperparameters to maximize Adjusted Rand Index against known
|
||||
formats and cross-checks against a model-free NCD (gzip) baseline. On the 700-file
|
||||
training corpus the calibrated defaults (`n=32`, `α=1.0`, `β=0.1`) recover the
|
||||
known formats at **ARI 0.77** (0.885 excluding tar), with `gzip`, `pkzip`
|
||||
(`docx`+`zip` merged), and `jpeg` forming clean, promotable clusters; the NCD
|
||||
baseline agrees. See `DESIGN_clustering.md` §11 for the full results, including the
|
||||
one known limitation (ELF and these tarballs share a long run of header zero-
|
||||
padding and merge — the v2 fix is inverse-entropy position weighting).
|
||||
|
||||
## The queue seam (→ RabbitMQ later)
|
||||
|
||||
The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and
|
||||
@@ -307,6 +352,13 @@ init, so the artifact is exactly regenerable from the same inputs.
|
||||
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
|
||||
| `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed |
|
||||
| `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed |
|
||||
| `FS_CLUSTER_DIR` | `data/binary` | Stage-5 input: the unknown/binary pile to sweep |
|
||||
| `FS_CLUSTER_N` | `32` | Header bytes modeled per file |
|
||||
| `FS_CLUSTER_ALPHA` | `1.0` | CRP concentration (propensity to spawn new formats) |
|
||||
| `FS_CLUSTER_PSEUDOCOUNT` | `0.1` | Dirichlet pseudocount β (calibrated) |
|
||||
| `FS_CLUSTER_BG_MASS` | `5.0` | Mass of the uniform background component |
|
||||
| `FS_PROMOTE_MIN_MEMBERS` | `20` | Cluster size threshold for promotion nomination |
|
||||
| `FS_PROMOTE_MIN_MAGIC` | `3` | Required fixed signature positions to nominate |
|
||||
|
||||
> To get real parallelism, start Julia with enough threads (`-t N`) to cover all
|
||||
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS`
|
||||
@@ -346,11 +398,14 @@ src/
|
||||
metadata.jl exiftool extraction + normalized sidecar (stage 2)
|
||||
content.jl binary-vs-text sniff for unknown files (stage 3)
|
||||
language.jl natural + programming language enrichment for text (stage 4)
|
||||
cluster.jl header-byte clustering for unknown-format discovery (stage 5, offline)
|
||||
worker.jl parametrized worker loop + classify/enrich/triage/language handlers
|
||||
server.jl HTTP routes/handlers
|
||||
bin/
|
||||
server.jl entry point
|
||||
train.jl offline training script → model/classifier.jld2
|
||||
server.jl entry point
|
||||
train.jl offline training script → model/classifier.jld2
|
||||
cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline
|
||||
model/
|
||||
classifier.jld2 committed trained weights (loaded at startup)
|
||||
classifier.jld2 committed trained weights (loaded at startup)
|
||||
DESIGN_clustering.md stage-5 design rationale + calibration results
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user