Address code-review findings on the metadata pipeline:
- finalize_known! now fsyncs the sidecar bytes before the rename and
fsyncs done/ after, so the "file in done/ implies sidecar present"
invariant holds across power loss, not just process crashes. The
docstring previously claimed an fsync the code never performed.
- run_exiftool's timeout escalates SIGTERM -> (2s grace) -> SIGKILL, so
an exiftool that ignores SIGTERM can't pin a worker forever on
wait(proc). Previously the timeout sent only SIGTERM.
- Add test/ (48 tests) covering the correctness-critical paths:
sanitize_filename, normalize_metadata, degraded build_metadata,
real exiftool extraction, finalize_known! end-to-end, recover_dir!.
Each uploaded file is scored by a fixed-structure neural net that labels it
known (resembling the training set) or unknown — novelty detection over the
first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax.
- src/model.jl: shared architecture + byte->feature mapping (trainer + server)
- src/classify.jl: load committed artifact, classify a file at inference
- bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag),
seeded 80/20 split, writes model/classifier.jld2
- worker: classify (annotate-only) and log classification=known|unknown
- config: FS_MODEL_PATH; server fails fast if the artifact is missing
- deps: Lux, JLD2, Optimisers, Zygote
REST endpoint (Oxygen.jl POST /upload, multipart) that spools uploaded
files to disk, enqueues lightweight references onto a bounded thread-safe
work queue, and hands off immediately (202 + job IDs; 503 when full). A
configurable pool of worker threads pulls jobs off the queue, logs the
received filename (placeholder for real processing), and moves files to
done/ on success or failed/ on error.
- Queue behind an enqueue!/dequeue!/close! seam for a future RabbitMQ swap
- Startup recovery: re-enqueues leftover files in spool/
- Graceful drain on SIGINT and SIGTERM (via atexit)
- Env-var config; filenames sanitized + UUID-prefixed on disk
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
