Add stage-2 metadata enrichment pipeline for known files

Known-classified files now flow to a second queue with its own worker pool
that extracts metadata via exiftool and writes a normalized JSON sidecar
next to the file in done/, leaving the original bytes untouched.

- Two-stage pipeline: spool/ → classify → known/ → enrich → done/;
  unknowns park in unknown/ as a seam for a future pool
- src/metadata.jl: exiftool -json -G -n with timeout, normalized schema
  (file_type, mime_type, author, created_by, dimensions, ...) + raw dump;
  degraded sidecar on extraction failure rather than quarantine
- Sidecar-first commit so a file in done/ always has its sidecar
- Parametrized worker_loop with classify/enrich handlers; blocking
  backpressure on a full known queue (never drop a classified file)
- Stage-aware recovery: spool/ and known/ resume at their correct stage
- Ordered drain: close stage-1 and wait its workers (the known queue's
  only producer) before closing the known queue
- exiftool required at startup (fail-fast); new FS_KNOWN_*/FS_UNKNOWN_DIR/
  FS_EXIFTOOL_TIMEOUT config knobs; combined-pool thread warning
This commit is contained in:
2026-07-02 16:29:08 -04:00
parent 842668b2ac
commit 1c7d7d6cad
6 changed files with 416 additions and 83 deletions

View File

@@ -42,17 +42,22 @@ end
const UUID_LEN = 36
"""
recover_spool!(cfg, queue) -> Int
recover_dir!(dir, queue) -> Int
Re-enqueue any files already sitting in the spool directory (left by a crash,
a hard shutdown, or an intake that never got processed). This is the payoff of
spooling to disk: a restart resumes work instead of stranding it. Returns the
number of files recovered.
Re-enqueue any files sitting in `dir` (left by a crash, hard shutdown, or an
intake that never finished) onto `queue`. This is the payoff of spooling to
disk: a restart resumes work instead of stranding it. Stage-aware recovery uses
one call per stage — `spool/` → stage-1 queue, `known/` → known queue — so each
file re-enters at the correct stage rather than being reclassified from scratch.
Returns the number of files recovered.
Skips `.meta.json` sidecars: those are stage-2 output, not work to redo.
"""
function recover_spool!(cfg::Config, queue::JobQueue)::Int
function recover_dir!(dir::AbstractString, queue::JobQueue)::Int
n = 0
for path in sort(readdir(cfg.spool_dir; join=true))
for path in sort(readdir(dir; join=true))
isfile(path) || continue
endswith(path, ".meta.json") && continue # sidecar, not a work item
fname = basename(path)
if length(fname) > UUID_LEN + 1
id = fname[1:UUID_LEN]