Update README for stage-5 phase B (durable catalog + sweep runner)

This commit is contained in:
2026-07-03 17:04:37 -04:00
parent 584bad02a7
commit f4e3f5be0b

View File

@@ -223,9 +223,30 @@ clusters. A cluster's spiked positions become a libmagic-style signature;
clusters with enough members and enough fixed positions self-**nominate** for
promotion (a human does the one irreversible step, redefining "known").
**Status:** the offline science (phase A) is implemented and calibrated; the live
catalog process (phase B) is designed and its scoring core (`assign_file`) is in
place, but its batch-runner plumbing is not yet built.
**Status:** both phases are implemented and calibrated. Phase A (offline Gibbs)
is the science; phase B (`src/catalog.jl`) is the live catalog: a durable
single-owner state that sweeps `binary/`, folds each new file into a cluster with
the deterministic CRP-predictive rule, and writes promotion nominations.
The catalog process is run periodically (cron), single-threaded — it is the
**only** writer of the catalog, so it needs no locking:
```bash
julia --project=. bin/cluster_sweep.jl # incremental live sweep of new binary/ files
julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs re-cluster (seed / recompact)
```
The catalog is a single durable file (`FS_CLUSTER_CATALOG`, default
`data/catalog.json`) committed with the same sidecar-first
temp→fsync→rename→fsync-dir discipline as the stage-2 sidecars, so a crash can
neither corrupt it nor lose a write. On the **first** run (empty catalog) the
sweep auto-promotes to a `--compact` pass to seed clusters; later runs assign
incrementally, touching only files they have not seen. A cluster that clears the
member/magic thresholds writes a nomination — a hex magic template, member count,
and example filenames — into `FS_NOMINATED_DIR` (default `data/nominated/`) for a
human to glance at and promote. Under the calibrated `bg_mass > α`, the live
sweep never mints single-file clusters; genuinely new formats surface from the
periodic `--compact` re-clustering of the background residue, not the live path.
Calibration is its own offline script (like training — never in the request
path), scored against magic-collapsed ground truth (so `docx``zip` and the whole
@@ -359,6 +380,8 @@ init, so the artifact is exactly regenerable from the same inputs.
| `FS_CLUSTER_BG_MASS` | `5.0` | Mass of the uniform background component |
| `FS_PROMOTE_MIN_MEMBERS` | `20` | Cluster size threshold for promotion nomination |
| `FS_PROMOTE_MIN_MAGIC` | `3` | Required fixed signature positions to nominate |
| `FS_CLUSTER_CATALOG` | `data/catalog.json` | Durable stage-5 catalog file (single-owner) |
| `FS_NOMINATED_DIR` | `data/nominated` | One JSON per self-nominated cluster (human promote gate) |
> To get real parallelism, start Julia with enough threads (`-t N`) to cover all
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS`
@@ -398,13 +421,15 @@ src/
metadata.jl exiftool extraction + normalized sidecar (stage 2)
content.jl binary-vs-text sniff for unknown files (stage 3)
language.jl natural + programming language enrichment for text (stage 4)
cluster.jl header-byte clustering for unknown-format discovery (stage 5, offline)
cluster.jl header-byte clustering model + Gibbs + scoring core (stage 5, science)
catalog.jl durable single-owner format catalog + sweep + nominations (stage 5, phase B)
worker.jl parametrized worker loop + classify/enrich/triage/language handlers
server.jl HTTP routes/handlers
bin/
server.jl entry point
train.jl offline training script → model/classifier.jld2
cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline
cluster_sweep.jl stage-5 phase-B runner: sweep binary/, update catalog, write nominations
model/
classifier.jld2 committed trained weights (loaded at startup)
DESIGN_clustering.md stage-5 design rationale + calibration results