Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process
mixture of per-position categoricals over the first 32 header bytes (257-symbol
alphabet) that clusters the binary/ pile by file format, plus signature
extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps
the Dirichlet-multinomial marginal dependency-free).
- src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler
(phase A), sequential CRP-predictive assignment (phase B core), signatures/
promotion, and ARI/V-measure calibration metrics.
- bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed
ground truth and cross-checks a model-free NCD (gzip) baseline.
- FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module.
- Tests for the three DESIGN §10 assertions plus the model primitives.
Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at
ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the
NCD baseline agrees. DESIGN §11 records the results and three assumptions the
data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and
the Bernoulli signature / Occam-penalized restart scoring).
