Add stage-2 metadata enrichment pipeline for known files
Known-classified files now flow to a second queue with its own worker pool that extracts metadata via exiftool and writes a normalized JSON sidecar next to the file in done/, leaving the original bytes untouched. - Two-stage pipeline: spool/ → classify → known/ → enrich → done/; unknowns park in unknown/ as a seam for a future pool - src/metadata.jl: exiftool -json -G -n with timeout, normalized schema (file_type, mime_type, author, created_by, dimensions, ...) + raw dump; degraded sidecar on extraction failure rather than quarantine - Sidecar-first commit so a file in done/ always has its sidecar - Parametrized worker_loop with classify/enrich handlers; blocking backpressure on a full known queue (never drop a classified file) - Stage-aware recovery: spool/ and known/ resume at their correct stage - Ordered drain: close stage-1 and wait its workers (the known queue's only producer) before closing the known queue - exiftool required at startup (fail-fast); new FS_KNOWN_*/FS_UNKNOWN_DIR/ FS_EXIFTOOL_TIMEOUT config knobs; combined-pool thread warning
This commit is contained in:
19
src/spool.jl
19
src/spool.jl
@@ -42,17 +42,22 @@ end
|
||||
const UUID_LEN = 36
|
||||
|
||||
"""
|
||||
recover_spool!(cfg, queue) -> Int
|
||||
recover_dir!(dir, queue) -> Int
|
||||
|
||||
Re-enqueue any files already sitting in the spool directory (left by a crash,
|
||||
a hard shutdown, or an intake that never got processed). This is the payoff of
|
||||
spooling to disk: a restart resumes work instead of stranding it. Returns the
|
||||
number of files recovered.
|
||||
Re-enqueue any files sitting in `dir` (left by a crash, hard shutdown, or an
|
||||
intake that never finished) onto `queue`. This is the payoff of spooling to
|
||||
disk: a restart resumes work instead of stranding it. Stage-aware recovery uses
|
||||
one call per stage — `spool/` → stage-1 queue, `known/` → known queue — so each
|
||||
file re-enters at the correct stage rather than being reclassified from scratch.
|
||||
Returns the number of files recovered.
|
||||
|
||||
Skips `.meta.json` sidecars: those are stage-2 output, not work to redo.
|
||||
"""
|
||||
function recover_spool!(cfg::Config, queue::JobQueue)::Int
|
||||
function recover_dir!(dir::AbstractString, queue::JobQueue)::Int
|
||||
n = 0
|
||||
for path in sort(readdir(cfg.spool_dir; join=true))
|
||||
for path in sort(readdir(dir; join=true))
|
||||
isfile(path) || continue
|
||||
endswith(path, ".meta.json") && continue # sidecar, not a work item
|
||||
fname = basename(path)
|
||||
if length(fname) > UUID_LEN + 1
|
||||
id = fname[1:UUID_LEN]
|
||||
|
||||
Reference in New Issue
Block a user