diff --git a/README.md b/README.md index 0c82cd2..4f1b621 100644 --- a/README.md +++ b/README.md @@ -223,9 +223,30 @@ clusters. A cluster's spiked positions become a libmagic-style signature; clusters with enough members and enough fixed positions self-**nominate** for promotion (a human does the one irreversible step, redefining "known"). -**Status:** the offline science (phase A) is implemented and calibrated; the live -catalog process (phase B) is designed and its scoring core (`assign_file`) is in -place, but its batch-runner plumbing is not yet built. +**Status:** both phases are implemented and calibrated. Phase A (offline Gibbs) +is the science; phase B (`src/catalog.jl`) is the live catalog: a durable +single-owner state that sweeps `binary/`, folds each new file into a cluster with +the deterministic CRP-predictive rule, and writes promotion nominations. + +The catalog process is run periodically (cron), single-threaded — it is the +**only** writer of the catalog, so it needs no locking: + +```bash +julia --project=. bin/cluster_sweep.jl # incremental live sweep of new binary/ files +julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs re-cluster (seed / recompact) +``` + +The catalog is a single durable file (`FS_CLUSTER_CATALOG`, default +`data/catalog.json`) committed with the same sidecar-first +temp→fsync→rename→fsync-dir discipline as the stage-2 sidecars, so a crash can +neither corrupt it nor lose a write. On the **first** run (empty catalog) the +sweep auto-promotes to a `--compact` pass to seed clusters; later runs assign +incrementally, touching only files they have not seen. A cluster that clears the +member/magic thresholds writes a nomination — a hex magic template, member count, +and example filenames — into `FS_NOMINATED_DIR` (default `data/nominated/`) for a +human to glance at and promote. Under the calibrated `bg_mass > α`, the live +sweep never mints single-file clusters; genuinely new formats surface from the +periodic `--compact` re-clustering of the background residue, not the live path. Calibration is its own offline script (like training — never in the request path), scored against magic-collapsed ground truth (so `docx`≡`zip` and the whole @@ -359,6 +380,8 @@ init, so the artifact is exactly regenerable from the same inputs. | `FS_CLUSTER_BG_MASS` | `5.0` | Mass of the uniform background component | | `FS_PROMOTE_MIN_MEMBERS` | `20` | Cluster size threshold for promotion nomination | | `FS_PROMOTE_MIN_MAGIC` | `3` | Required fixed signature positions to nominate | +| `FS_CLUSTER_CATALOG` | `data/catalog.json` | Durable stage-5 catalog file (single-owner) | +| `FS_NOMINATED_DIR` | `data/nominated` | One JSON per self-nominated cluster (human promote gate) | > To get real parallelism, start Julia with enough threads (`-t N`) to cover all > pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS` @@ -398,13 +421,15 @@ src/ metadata.jl exiftool extraction + normalized sidecar (stage 2) content.jl binary-vs-text sniff for unknown files (stage 3) language.jl natural + programming language enrichment for text (stage 4) - cluster.jl header-byte clustering for unknown-format discovery (stage 5, offline) + cluster.jl header-byte clustering model + Gibbs + scoring core (stage 5, science) + catalog.jl durable single-owner format catalog + sweep + nominations (stage 5, phase B) worker.jl parametrized worker loop + classify/enrich/triage/language handlers server.jl HTTP routes/handlers bin/ server.jl entry point train.jl offline training script → model/classifier.jld2 cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline + cluster_sweep.jl stage-5 phase-B runner: sweep binary/, update catalog, write nominations model/ classifier.jld2 committed trained weights (loaded at startup) DESIGN_clustering.md stage-5 design rationale + calibration results