Add stage-5 phase B: durable single-owner format catalog

Wraps the assign_file scoring core (cluster.jl) in the live catalog the
design's phase B calls for (DESIGN §5B/§9):

- src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient
  stats + processed set + examples); sparse, sidecar-first durable
  save/load; incremental catalog_sweep! (deterministic CRP-predictive
  assignment of new binary/ files); offline compact! that seeds on first
  run and recompacts later; write_nominations! emitting one JSON per
  promotable cluster with a hex magic template.
- bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact
  forces a recluster; first run auto-compacts to seed).
- config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG,
  FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs.
- Tests: durable round-trip, incremental sweep growth + idempotency,
  §10.1 nothing-from-noise end-to-end (zero promotions), a recurring
  format self-nominating, seed-then-live-assign (165 pass).
- DESIGN_clustering.md: mark phase B built.
This commit is contained in:
2026-07-03 17:03:33 -04:00
parent d9f32d9aaf
commit 584bad02a7
6 changed files with 631 additions and 9 deletions

View File

@@ -1,8 +1,10 @@
# Stage-5: Unknown-format discovery by Bayesian header clustering
Status: **phase A implemented and calibrated** (`src/cluster.jl`,
`bin/cluster_calibrate.jl`, tests in `test/runtests.jl`); phase-B core scoring
implemented (`assign_file`), its live batch-process plumbing still to do. Product
Status: **phases A and B implemented and calibrated** (`src/cluster.jl` +
`src/catalog.jl`, `bin/cluster_calibrate.jl` + `bin/cluster_sweep.jl`, tests in
`test/runtests.jl`). Phase A (offline Gibbs) is calibrated; phase B's durable
single-owner catalog, incremental sweep, and nomination writer are now built on
top of the `assign_file` scoring core. Product
of a design interview; captures the decisions and — as important — the
assumptions we *rejected* so they don't get silently reintroduced. §11 records
what building it actually taught us, including three assumptions in this document
@@ -291,7 +293,14 @@ model-free gzip similarity.
a MAP-style stand-in for the VI/Binder posterior summary §5 defers adequate
because the formats are strongly separated; revisit if compaction 5) needs it.
- Phase B's **live single-owner batch process** 9) and the durable catalog file
are not yet built; `assign_file` is the scoring core they will wrap.
are implemented in `src/catalog.jl` (the `Catalog` durable state, the
incremental `catalog_sweep!`, the offline `compact!` seed/recompaction, and
`write_nominations!`), driven by `bin/cluster_sweep.jl` (cron/periodic; the
first run auto-compacts to seed, subsequent runs sweep incrementally). The
catalog is persisted with the stage-2 sidecar-first tempfsyncrenamefsync-dir
discipline. As §5B predicts, under the calibrated `bg_mass > α` the live sweep
never mints single-file clusters new formats are discovered by the offline
`compact!` re-clustering the background residue, not by the live path.
## Open items (deferred, intentionally)