Add stage-5 phase B: durable single-owner format catalog
Wraps the assign_file scoring core (cluster.jl) in the live catalog the design's phase B calls for (DESIGN §5B/§9): - src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient stats + processed set + examples); sparse, sidecar-first durable save/load; incremental catalog_sweep! (deterministic CRP-predictive assignment of new binary/ files); offline compact! that seeds on first run and recompacts later; write_nominations! emitting one JSON per promotable cluster with a hex magic template. - bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact forces a recluster; first run auto-compacts to seed). - config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG, FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs. - Tests: durable round-trip, incremental sweep growth + idempotency, §10.1 nothing-from-noise end-to-end (zero promotions), a recurring format self-nominating, seed-then-live-assign (165 pass). - DESIGN_clustering.md: mark phase B built.
This commit is contained in:
@@ -1,8 +1,10 @@
|
||||
# Stage-5: Unknown-format discovery by Bayesian header clustering
|
||||
|
||||
Status: **phase A implemented and calibrated** (`src/cluster.jl`,
|
||||
`bin/cluster_calibrate.jl`, tests in `test/runtests.jl`); phase-B core scoring
|
||||
implemented (`assign_file`), its live batch-process plumbing still to do. Product
|
||||
Status: **phases A and B implemented and calibrated** (`src/cluster.jl` +
|
||||
`src/catalog.jl`, `bin/cluster_calibrate.jl` + `bin/cluster_sweep.jl`, tests in
|
||||
`test/runtests.jl`). Phase A (offline Gibbs) is calibrated; phase B's durable
|
||||
single-owner catalog, incremental sweep, and nomination writer are now built on
|
||||
top of the `assign_file` scoring core. Product
|
||||
of a design interview; captures the decisions and — as important — the
|
||||
assumptions we *rejected* so they don't get silently reintroduced. §11 records
|
||||
what building it actually taught us, including three assumptions in this document
|
||||
@@ -291,7 +293,14 @@ model-free gzip similarity.
|
||||
a MAP-style stand-in for the VI/Binder posterior summary §5 defers — adequate
|
||||
because the formats are strongly separated; revisit if compaction (§5) needs it.
|
||||
- Phase B's **live single-owner batch process** (§9) and the durable catalog file
|
||||
are not yet built; `assign_file` is the scoring core they will wrap.
|
||||
are implemented in `src/catalog.jl` (the `Catalog` durable state, the
|
||||
incremental `catalog_sweep!`, the offline `compact!` seed/recompaction, and
|
||||
`write_nominations!`), driven by `bin/cluster_sweep.jl` (cron/periodic; the
|
||||
first run auto-compacts to seed, subsequent runs sweep incrementally). The
|
||||
catalog is persisted with the stage-2 sidecar-first temp→fsync→rename→fsync-dir
|
||||
discipline. As §5B predicts, under the calibrated `bg_mass > α` the live sweep
|
||||
never mints single-file clusters — new formats are discovered by the offline
|
||||
`compact!` re-clustering the background residue, not by the live path.
|
||||
|
||||
## Open items (deferred, intentionally)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user