Add stage-5 phase B: durable single-owner format catalog
Wraps the assign_file scoring core (cluster.jl) in the live catalog the design's phase B calls for (DESIGN §5B/§9): - src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient stats + processed set + examples); sparse, sidecar-first durable save/load; incremental catalog_sweep! (deterministic CRP-predictive assignment of new binary/ files); offline compact! that seeds on first run and recompacts later; write_nominations! emitting one JSON per promotable cluster with a hex magic template. - bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact forces a recluster; first run auto-compacts to seed). - config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG, FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs. - Tests: durable round-trip, incremental sweep growth + idempotency, §10.1 nothing-from-noise end-to-end (zero promotions), a recurring format self-nominating, seed-then-live-assign (165 pass). - DESIGN_clustering.md: mark phase B built.
This commit is contained in:
19
bin/cluster_sweep.jl
Normal file
19
bin/cluster_sweep.jl
Normal file
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env julia
|
||||
#
|
||||
# Stage-5 phase-B runner (model/DESIGN_clustering.md §9): the single-owner,
|
||||
# periodic/cron process that sweeps `binary/`, folds new files into the durable
|
||||
# format catalog by sequential CRP-predictive assignment, and (re)writes
|
||||
# promotion nominations. Run it single-threaded on a schedule — it is the ONLY
|
||||
# writer of the catalog, so no locking is needed.
|
||||
#
|
||||
# julia --project=. bin/cluster_sweep.jl # incremental live sweep
|
||||
# julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs (seed / recompact)
|
||||
#
|
||||
# On a fresh catalog (nothing processed yet) the incremental sweep would send
|
||||
# every file to background — there are no clusters to match — so the first run
|
||||
# auto-promotes to a compaction pass to seed the catalog. Configure via the
|
||||
# FS_CLUSTER_* / FS_NOMINATED_DIR env vars (see src/config.jl).
|
||||
|
||||
using FileServer
|
||||
|
||||
FileServer.cluster_sweep_cli(ARGS)
|
||||
Reference in New Issue
Block a user