Route by queue, not by directory: stop moving files between stages

A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.

Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.

Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.

The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.

Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.

Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.

bin/bench.jl

@@ -14,8 +14,9 @@
 #   2. End-to-end throughput doesn't name the slow stage. The four stages run
 #      concurrently behind their own queues, so the pipeline's rate is the
 #      slowest stage's rate and the others are invisible. Directory polling can't
-#      recover them either — known/, unknown/ and text/ are transient, and a file
-#      can cross one between two samples. So the server keeps per-stage counters
+#      recover them either — every in-flight file sits in spool/ whatever stage it
+#      is at, since stages route by enqueueing rather than by moving bytes. So the
+#      server keeps per-stage counters
 #      (src/stats.jl) and we scrape GET /stats before and after: the deltas give
 #      each stage's throughput, mean service time, and worker utilization, and
 #      utilization is what actually names the bottleneck (see `stage_report`).
@@ -151,12 +152,12 @@ sinkdirs() = (
     failed    = get(ENV, "FS_FAILED_DIR",    "data/failed"),
 )
 
-stagedirs() = (
-    spool   = get(ENV, "FS_SPOOL_DIR",   "data/spool"),
-    known   = get(ENV, "FS_KNOWN_DIR",   "data/known"),
-    unknown = get(ENV, "FS_UNKNOWN_DIR", "data/unknown"),
-    text    = get(ENV, "FS_TEXT_DIR",    "data/text"),
-)
+# One directory, not four. Files no longer move between stages — spool/ holds
+# every in-flight file at every stage, and which stage it has reached lives in
+# the queue holding its reference (src/worker.jl header). So this depth is
+# "files in flight", full stop; per-stage depth comes from /stats, which is the
+# only place that can see it at all.
+stagedirs() = (spool = get(ENV, "FS_SPOOL_DIR", "data/spool"),)
 
 "Count work items in `dir`, ignoring the .meta.json sidecars stages 2/4 write."
 function count_files(dir::AbstractString)::Int
@@ -513,7 +514,7 @@ function main(argv)
     # Leftovers mid-pipeline would land in the sinks during our window and be
     # counted as our throughput, so say so up front rather than quietly skewing.
     pending = total(counts(stages))
-    pending > 0 && @warn "pipeline is not idle: $pending file(s) in the stage dirs; " *
+    pending > 0 && @warn "pipeline is not idle: $pending file(s) still in spool/; " *
                          "throughput will include their completions"
 
     # --- corpus
@@ -570,7 +571,7 @@ function main(argv)
                   "(the server predates src/stats.jl)"
         end
 
-        # --- sampler: RSS curve, stage dir depths, and queue depths.
+        # --- sampler: RSS curve, spool depth, and queue depths.
         stop = Threads.Atomic{Bool}(false)
         rss_samples = Float64[]
         depth_max = Dict(k => 0 for k in keys(stages))
@@ -731,7 +732,7 @@ function main(argv)
             println("\nnote: $(sink_delta.failed) file(s) landed in $(sinks.failed) — check the server log.")
         timed_out &&
             println("\nnote: drain stalled with $(accepted - completed) file(s) outstanding. " *
-                    "Check the server log and the stage dirs; raise --timeout if the pipeline is just slow.")
+                    "Check the server log and spool/; raise --timeout if the pipeline is just slow.")
 
         if opts["json"] !== nothing
             result = (