Known-classified files now flow to a second queue with its own worker pool
that extracts metadata via exiftool and writes a normalized JSON sidecar
next to the file in done/, leaving the original bytes untouched.
- Two-stage pipeline: spool/ → classify → known/ → enrich → done/;
unknowns park in unknown/ as a seam for a future pool
- src/metadata.jl: exiftool -json -G -n with timeout, normalized schema
(file_type, mime_type, author, created_by, dimensions, ...) + raw dump;
degraded sidecar on extraction failure rather than quarantine
- Sidecar-first commit so a file in done/ always has its sidecar
- Parametrized worker_loop with classify/enrich handlers; blocking
backpressure on a full known queue (never drop a classified file)
- Stage-aware recovery: spool/ and known/ resume at their correct stage
- Ordered drain: close stage-1 and wait its workers (the known queue's
only producer) before closing the known queue
- exiftool required at startup (fail-fast); new FS_KNOWN_*/FS_UNKNOWN_DIR/
FS_EXIFTOOL_TIMEOUT config knobs; combined-pool thread warning
REST endpoint (Oxygen.jl POST /upload, multipart) that spools uploaded
files to disk, enqueues lightweight references onto a bounded thread-safe
work queue, and hands off immediately (202 + job IDs; 503 when full). A
configurable pool of worker threads pulls jobs off the queue, logs the
received filename (placeholder for real processing), and moves files to
done/ on success or failed/ on error.
- Queue behind an enqueue!/dequeue!/close! seam for a future RabbitMQ swap
- Startup recovery: re-enqueues leftover files in spool/
- Graceful drain on SIGINT and SIGTERM (via atexit)
- Env-var config; filenames sanitized + UUID-prefixed on disk
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
