A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.
Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.
Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.
The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.
Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.
Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.
bin/bench.jl reports stage 2 as a single throughput number, which can't
distinguish slow extraction from a slow spawn — and those have opposite fixes.
bin/bench_stage2.jl times each component in isolation, then times the real
handle_known_job end to end. It draws its corpus from real files (default
data/done) because random bytes make exiftool bail out early and understate the
stage by ~10x, and it prices both fork-free alternatives (batched, -stay_open)
so the cost of one-fork-per-file is a measurement rather than a guess.
The benchmark found stage 2 to be ~98% exiftool, and found two problems in
run_with_timeout, which stages 2 and 4 share:
1. The watchdog polled with sleep(0.1) and then joined the polling task, so
every call paid the remainder of an in-flight sleep after the child had
already exited: ~25 ms per file, and a measured 101 ms on a process that
exits instantly. Replaced with a one-shot Timer cancelled when the child
exits. Stage 2 goes from 164.6 ms to 138.3 ms per file (6 -> 8 files/s on one
worker); the wrapper is now within noise of a bare Base.run.
2. Writing the missing tests showed the timeout was never enforceable, in the
old implementation as much as the new. wait(proc) returns only once the
captured stdout pipe closes, and grandchildren inherit that pipe, so
signalling the child alone left the worker blocked until the whole process
tree finished on its own — a `sh -c "trap '' TERM; sleep 30"` child ran the
full 30 s against a 1 s timeout. The child now runs in its own process group
and the timeout signals the group. The trade is that a hard crash of the
server orphans an in-flight child rather than taking it down with it.
Three new tests cover the fast path, the timeout, and the SIGTERM-ignoring
escalation; the second was previously unexercised, which is why the bug stood.
Not addressed here, but measured and documented in the README: Perl interpreter
startup is 76.7 ms of the remaining 135.9 ms call, so a persistent exiftool
(-stay_open, 40.1 ms/file) would cut the stage by roughly another 70%. And
fsync_dir measures 1.75 us, too fast to be a real flush — commit_enriched!'s
durability may not hold on this filesystem, which is a correctness question
left open.
Claude-Session: https://claude.ai/code/session_01Xy9At7HNLHWNmfh1yw71Uy
