Add stage-2 metadata enrichment pipeline for known files

Known-classified files now flow to a second queue with its own worker pool
that extracts metadata via exiftool and writes a normalized JSON sidecar
next to the file in done/, leaving the original bytes untouched.

- Two-stage pipeline: spool/ → classify → known/ → enrich → done/;
  unknowns park in unknown/ as a seam for a future pool
- src/metadata.jl: exiftool -json -G -n with timeout, normalized schema
  (file_type, mime_type, author, created_by, dimensions, ...) + raw dump;
  degraded sidecar on extraction failure rather than quarantine
- Sidecar-first commit so a file in done/ always has its sidecar
- Parametrized worker_loop with classify/enrich handlers; blocking
  backpressure on a full known queue (never drop a classified file)
- Stage-aware recovery: spool/ and known/ resume at their correct stage
- Ordered drain: close stage-1 and wait its workers (the known queue's
  only producer) before closing the known queue
- exiftool required at startup (fail-fast); new FS_KNOWN_*/FS_UNKNOWN_DIR/
  FS_EXIFTOOL_TIMEOUT config knobs; combined-pool thread warning

src/worker.jl

@@ -1,39 +1,82 @@
-# Worker task: pull jobs off the queue and process them. One of these runs per
-# configured worker, each as its own `Threads.@spawn`'d task.
+# Worker tasks: pull jobs off a queue and process them. The loop scaffolding
+# (dequeue-until-drained, try/catch, quarantine-on-throw) is identical for every
+# stage, so `worker_loop` is parametrized with a `handler` and reused. Today
+# there are two stages:
+#
+#   stage 1  handle_classify_job  spool/  → classify → known/ (+known queue) | unknown/
+#   stage 2  handle_known_job     known/  → exiftool enrich → done/ (+ .meta.json)
+#
+# Adding a stage later (e.g. an unknown-file pool consuming unknown/) is just
+# another queue + pool + handler; the loop below doesn't change.
+
+# How long a stage-1 worker backs off before retrying an enqueue onto a full
+# known queue. Blocking backpressure: a classified file is never dropped, so the
+# stage-1 worker parks until stage 2 makes room. Keeps intake decoupled — the
+# HTTP path's `enqueue!` stays non-blocking; only this worker-to-worker handoff
+# blocks.
+const KNOWN_ENQUEUE_RETRY_SECONDS = 0.05
 
 """
-    handle_job(job, cfg, worker_id)
+    handle_classify_job(job, cfg, worker_id, known_queue)
 
-Do the work for a single job, then move the file to `done/`.
+Stage 1. Classify the spooled file and route it:
+  * `:unknown` → move to `unknown/` (parked for a future pipeline; terminal here).
+  * `:known`   → move to `known/`, then enqueue onto the known queue for stage 2,
+    retrying on a full queue rather than dropping the file.
 
-For now the "work" is just logging the received filename to prove the flow —
-this is the seam where real heavy-lifting will go later.
+Sub-`MIN_FILE_BYTES` files short-circuit to `:unknown` inside `classify`.
 """
-function handle_job(job::Job, cfg::Config, worker_id::Int)
-    # Classify the spooled file (annotate-only for now: the result is logged but
-    # every file still moves to done/ regardless of known/unknown). Sub-32-byte
-    # files short-circuit to :unknown inside classify without touching the model.
+function handle_classify_job(job::Job, cfg::Config, worker_id::Int, known_queue::JobQueue)
     classification = classify(CLASSIFIER[], job.path)
-    @info "received file" worker=worker_id id=job.id name=job.original_name size=job.size classification=classification
-    dest = move_to(cfg.done_dir, job)
-    @info "completed" worker=worker_id id=job.id dest=dest
+    @info "classified file" worker=worker_id id=job.id name=job.original_name size=job.size classification=classification
+
+    if classification === :known
+        # Move first so the file physically lives in known/ before the reference
+        # is visible to stage 2; then hand off. The moved path becomes the job's
+        # new location for the known queue.
+        dest = move_to(cfg.known_dir, job)
+        routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
+        while !enqueue!(known_queue, routed)
+            sleep(KNOWN_ENQUEUE_RETRY_SECONDS)   # known queue full → back off, don't drop
+        end
+        @info "routed to enrichment" worker=worker_id id=job.id dest=dest
+    else
+        dest = move_to(cfg.unknown_dir, job)
+        @info "parked unknown" worker=worker_id id=job.id dest=dest
+    end
     return nothing
 end
 
 """
-    worker_loop(worker_id, cfg, queue)
+    handle_known_job(job, cfg, worker_id)
 
-Consume jobs until the queue is closed and drained. A failure on one job is
-logged and the file is quarantined in `failed/` — it must never kill the
-worker, or the pool would silently shrink.
+Stage 2. Extract metadata (exiftool, with timeout) and enrich: build the
+normalized sidecar and commit both to `done/` sidecar-first. Extraction
+failure/timeout yields a *degraded* sidecar (the file is still a wanted known
+file), so the only way to land in `failed/` is a genuine I/O error writing the
+sidecar or moving the file — handled by `worker_loop`'s quarantine.
 """
-function worker_loop(worker_id::Int, cfg::Config, queue::JobQueue)
+function handle_known_job(job::Job, cfg::Config, worker_id::Int)
+    meta = build_metadata(job, cfg)
+    file_dest, sidecar = finalize_known!(cfg, job, meta)
+    @info "enriched" worker=worker_id id=job.id dest=file_dest sidecar=basename(sidecar) file_type=meta.file_type created_by=meta.created_by degraded=(meta.error !== nothing)
+    return nothing
+end
+
+"""
+    worker_loop(worker_id, cfg, queue, handler)
+
+Consume jobs from `queue` until it is closed and drained, running `handler` on
+each. A failure on one job is logged and the file quarantined in `failed/` — it
+must never kill the worker, or the pool would silently shrink.
+"""
+function worker_loop(worker_id::Int, cfg::Config, queue::JobQueue, handler)
     @info "worker started" worker=worker_id
     while true
         job = dequeue!(queue)
         job === nothing && break   # queue closed and drained → exit
         try
-            handle_job(job, cfg, worker_id)
+            handler(job, cfg, worker_id)
         catch e
             @error "processing failed" worker=worker_id id=job.id name=job.original_name exception=(e, catch_backtrace())
             try