Jeffrey Ward
584bad02a7
Wraps the assign_file scoring core (cluster.jl) in the live catalog the design's phase B calls for (DESIGN §5B/§9): - src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient stats + processed set + examples); sparse, sidecar-first durable save/load; incremental catalog_sweep! (deterministic CRP-predictive assignment of new binary/ files); offline compact! that seeds on first run and recompacts later; write_nominations! emitting one JSON per promotable cluster with a hex magic template. - bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact forces a recluster; first run auto-compacts to seed). - config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG, FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs. - Tests: durable round-trip, incremental sweep growth + idempotency, §10.1 nothing-from-noise end-to-end (zero promotions), a recurring format self-nominating, seed-then-live-assign (165 pass). - DESIGN_clustering.md: mark phase B built.
20 lines
911 B
Julia
20 lines
911 B
Julia
#!/usr/bin/env julia
|
|
#
|
|
# Stage-5 phase-B runner (model/DESIGN_clustering.md §9): the single-owner,
|
|
# periodic/cron process that sweeps `binary/`, folds new files into the durable
|
|
# format catalog by sequential CRP-predictive assignment, and (re)writes
|
|
# promotion nominations. Run it single-threaded on a schedule — it is the ONLY
|
|
# writer of the catalog, so no locking is needed.
|
|
#
|
|
# julia --project=. bin/cluster_sweep.jl # incremental live sweep
|
|
# julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs (seed / recompact)
|
|
#
|
|
# On a fresh catalog (nothing processed yet) the incremental sweep would send
|
|
# every file to background — there are no clusters to match — so the first run
|
|
# auto-promotes to a compaction pass to seed the catalog. Configure via the
|
|
# FS_CLUSTER_* / FS_NOMINATED_DIR env vars (see src/config.jl).
|
|
|
|
using FileServer
|
|
|
|
FileServer.cluster_sweep_cli(ARGS)
|