Route by queue, not by directory: stop moving files between stages

A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.

Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.

Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.

The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.

Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.

Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.

README.md

@@ -22,12 +22,13 @@ enrichment mixes CPU with a subprocess):
                  ▼
         ┌─────────────────┐     stream bytes to disk (never buffered)
         │  HTTP handler    │────────────────────────► data/spool/<uuid>-<name>
-        │  (streaming)     │
+        │  (streaming)     │                          the file's home for its
-        └────────┬─────────┘     enqueue reference (non-blocking)
+        └────────┬─────────┘     enqueue reference    whole time in flight
-                 │                        │
+                 │               (non-blocking)
-                 ▼                        ▼
+                 ▼                        │
-          202 + job IDs        ┌────────────────────┐
+          202 + job IDs                   ▼
-          (503 if full)        │  stage-1 queue     │  classification
+          (503 if full)        ┌────────────────────┐
+                               │  stage-1 queue     │  classification
                                └─────────┬──────────┘
                                          │ dequeue
                      ┌───────────────────┼───────────────────┐
@@ -35,9 +36,10 @@ enrichment mixes CPU with a subprocess):
               classify wkr 1      classify wkr 2   …   classify wkr N
                      │
         ┌────────────┴────────────┐
-    :unknown                    :known
+    :unknown                    :known     the file itself never moves —
-        │  move to data/unknown/,  │  move to data/known/, then
+        │                         │        routing is the enqueue alone
-        ▼  then enqueue (blocking) ▼  enqueue (blocking backpressure)
+        │  enqueue (blocking)     │  enqueue (blocking backpressure)
+        ▼                         ▼
   ┌────────────────────┐        ┌────────────────────┐
   │  unknown queue     │        │  known queue       │  enrichment
   └─────────┬──────────┘        └─────────┬──────────┘
@@ -47,10 +49,10 @@ enrichment mixes CPU with a subprocess):
  unk 1  unk 2 … unk K   known wkr 1  known wkr 2 … known wkr M
    │  binary-vs-text sniff       │  exiftool → normalized sidecar
    ├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name>
-   │   (terminal)                           data/done/<uuid>-<name>.meta.json
+   │   (terminal — moved)                    data/done/<uuid>-<name>.meta.json
    │                                         (sidecar-first commit)
-   │  :text  move to data/text/,  failure  ───────► data/failed/<uuid>-<name>
+   │  :text  enqueue (blocking   failure  ───────► data/failed/<uuid>-<name>
-   ▼  then enqueue (blocking backpressure)
+   ▼          backpressure)
   ┌────────────────────┐
   │  text queue        │  language enrichment
   └─────────┬──────────┘
@@ -63,6 +65,16 @@ enrichment mixes CPU with a subprocess):
                                         (sidecar-first commit)
 ```
 
+A file is written **once**, into `data/spool/`, and stays there for its entire
+time in the pipeline. Stages hand it on by enqueueing its small `Job` reference,
+never by moving bytes — the queue holding the reference *is* the record of which
+stage the file has reached. The only move is the last one: into a terminal sink
+(`data/done/`, `data/text_done/`, `data/binary/`) or into `data/failed/` if a
+worker throws. Stage 1 used to rename each file into `data/known/` or
+`data/unknown/` first, and stage 3 into `data/text/`; those three directories are
+gone, and with them 11.6 µs per file — the equal of the classifier itself, which
+is why stage 1 now runs at ~85k files/s on one worker instead of ~35k.
+
 Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1
 feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language
 enrichment) for every file it sorts as text.
@@ -79,12 +91,24 @@ Key properties:
 - **Backpressure:** each queue is bounded (default 1000). When the *intake* queue
   is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
   the stage-1 worker blocks and retries (a classified file is never dropped).
-- **Crash-resilient:** files survive on disk. On startup, recovery is
+- **Crash-resilient:** files survive on disk. On startup everything left in
-  stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/`
+  `data/spool/` is re-enqueued (`recovered` in the log) and **replays from stage
-  re-enter enrichment, `data/unknown/` re-enter content triage, and `data/text/`
+  1**. That is safe rather than merely tolerable: classification and the
-  re-enter language enrichment (`recovered` / `recovered_known` /
+  binary/text sniff are pure functions of the file's bytes, and the terminal
-  `recovered_unknown` / `recovered_text` in the log), so a file resumes at its
+  commits rename with `force=true`, so a replayed file lands where it would have
-  correct stage instead of restarting from scratch.
+  landed and overwrites its own sidecar. Recovery *blocks* on a full queue rather
+  than dropping the excess, and runs with the worker pools already live, so a
+  backlog larger than one queue's capacity takes longer to re-drive but none of
+  it is abandoned.
+
+  The cost of replay is redoing stages a file had already cleared. That is a
+  property of the **queue**, not of the directory layout: the queues are
+  in-process (`src/queue.jl`), so a crash destroys the only record of how far
+  each file got. Per-stage directories used to stand in for that record, at the
+  price of a rename per file per stage on the hot path — paying a permanent cost
+  on every file to buy a cheaper restart. `src/queue.jl` already defines the seam
+  for the real fix: swap `ChannelQueue` for a broker-backed `JobQueue` and exact
+  resume comes back durably, rather than being inferred from a pathname.
 - **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
   both stop accepting uploads, then drain the stages *in order* — close the
   stage-1 queue and wait out the classify workers (the only producer of the known
@@ -182,21 +206,26 @@ idempotently.
 Files the classifier labels **unknown** are handed to a third pool that sorts
 them into two coarse buckets so downstream tooling can treat them differently:
 
-- **`data/binary/`** — the file looks like binary data.
+- **binary** — the file looks like binary data. This is the end of the live path,
-- **`data/text/`** — the file looks like text.
+  so the file is committed to `data/binary/` (which is also the corpus the
+  offline stage-5 discovery sweep reads).
+- **text** — the file looks like text. Stage 4 is still to come, so nothing
+  moves; the file stays in `data/spool/` and its reference goes onto the
+  stage-4 queue, ending up in `data/text_done/` once enriched.
 
 The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text
 when that window is valid UTF-8 and holds no control bytes outside the text-safe
 set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise
 binary. It's cheap (no full read) and Unicode-aware — unlike the older NUL-byte
 or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in
-`text/` instead of misfiling it, while binary formats — which rarely form valid
+the text bucket instead of misfiling it, while binary formats — which rarely
-UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but
+form valid UTF-8 near their start — still read as binary. A NUL byte is valid UTF-8 but
 not a text control byte, so it still reads as binary. A multi-byte character
 split by the 8000-byte boundary is trimmed before the check so it isn't mistaken
-for malformed bytes. An empty file is treated as text. `binary/` is terminal on
+for malformed bytes. An empty file is treated as text. Binary is terminal on the
-the live path (but is the input the offline **stage-5 discovery** sweeps — see
+live path and is committed to `data/binary/` (which the offline **stage-5
-below); `text/` is handed to stage 4 (`src/content.jl`).
+discovery** sweep reads as its corpus — see below); text is handed to stage 4
+without moving (`src/content.jl`).
 
 ### Language enrichment (stage 4)
 
@@ -367,10 +396,10 @@ tell you "PDF", just "this looks like something I was trained on, or not".
   fast rather than run without classification.
 - **Effect today:** *active routing*. The class is logged
   (`classification=known|unknown`) and drives the pipeline split: `:known` files
-  go to `known/` for metadata enrichment (stage 2), `:unknown` files go to
+  go onto the stage-2 queue for metadata enrichment, `:unknown` files onto the
-  `unknown/` for content triage (stage 3). The class chooses the downstream
+  stage-3 queue for content triage. The class chooses the downstream stage (the
-  stage; what's still unproven is the model's *accuracy*, not whether the routing
+  file itself stays in `data/spool/` either way); what's still unproven is the
-  path runs.
+  model's *accuracy*, not whether the routing path runs.
 
 The architecture and byte→feature mapping are defined once in `src/model.jl` and
 shared by the trainer and the server, so they can't drift apart.
@@ -410,11 +439,8 @@ init, so the artifact is exactly regenerable from the same inputs.
 | `FS_UNKNOWN_QUEUE_CAPACITY` | `1000`       | Max pending triage jobs (then backpressure)  |
 | `FS_TEXT_WORKERS`         | `nthreads()`   | Stage-4 (language enrichment) worker tasks   |
 | `FS_TEXT_QUEUE_CAPACITY`  | `1000`         | Max pending language jobs (then backpressure) |
-| `FS_SPOOL_DIR`            | `data/spool`   | Incoming files (pending classification)      |
+| `FS_SPOOL_DIR`            | `data/spool`   | Every in-flight file, at every stage         |
-| `FS_KNOWN_DIR`            | `data/known`   | Classified-known, awaiting enrichment        |
-| `FS_UNKNOWN_DIR`          | `data/unknown` | Classified-unknown, awaiting content triage  |
 | `FS_BINARY_DIR`           | `data/binary`  | Stage-3 sink: unknown files that look binary  |
-| `FS_TEXT_DIR`             | `data/text`    | Classified-text, awaiting language enrichment |
 | `FS_DONE_DIR`             | `data/done`    | Enriched known files (+ `.meta.json`)        |
 | `FS_TEXT_DONE_DIR`        | `data/text_done` | Enriched text files (+ `.meta.json`)       |
 | `FS_FAILED_DIR`           | `data/failed`  | Files whose processing threw                 |
@@ -462,9 +488,9 @@ Each file in a request becomes its own job. Responses:
 ### `GET /stats` — per-stage counters
 
 The pipeline counts its own work (`src/stats.jl`), because nothing outside it
-can: `known/`, `unknown/` and `text/` are *transient*, so a file can cross one
+can. Every in-flight file sits in `data/spool/` no matter which stage it has
-between two directory polls and an external sampler will miss exactly the stages
+reached — the stage is a property of the queue holding its reference, and only
-you most want to measure.
+the pipeline can see that. There is no directory to poll.
 
 ```jsonc
 {
@@ -560,9 +586,9 @@ Three properties of this design dictate how it measures:
   so `ab`/`hey`/`wrk` would only ever measure intake. The bench instead uploads a
   corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
   `failed/`) until the count stops moving, and reports both numbers separately:
-  intake rate *and* end-to-end completion rate. It also samples the intermediate
+  intake rate *and* end-to-end completion rate. It also samples `spool/`, whose
-  stage dirs, so the peak depth of `spool/`/`known/`/`unknown/`/`text/` shows
+  peak depth is the high-water mark of files in flight — but *which* stage they
-  where work piles up.
+  are waiting on comes from `/stats`, not from disk.
 - **End-to-end throughput doesn't name the slow stage.** The four stages run
   concurrently behind their own queues, so the pipeline's rate *is* the slowest
   stage's rate and the others are invisible in it. The bench scrapes

bin/bench.jl

@@ -14,8 +14,9 @@
 #   2. End-to-end throughput doesn't name the slow stage. The four stages run
 #      concurrently behind their own queues, so the pipeline's rate is the
 #      slowest stage's rate and the others are invisible. Directory polling can't
-#      recover them either — known/, unknown/ and text/ are transient, and a file
+#      recover them either — every in-flight file sits in spool/ whatever stage it
-#      can cross one between two samples. So the server keeps per-stage counters
+#      is at, since stages route by enqueueing rather than by moving bytes. So the
+#      server keeps per-stage counters
 #      (src/stats.jl) and we scrape GET /stats before and after: the deltas give
 #      each stage's throughput, mean service time, and worker utilization, and
 #      utilization is what actually names the bottleneck (see `stage_report`).
@@ -151,12 +152,12 @@ sinkdirs() = (
     failed    = get(ENV, "FS_FAILED_DIR",    "data/failed"),
 )
 
-stagedirs() = (
+# One directory, not four. Files no longer move between stages — spool/ holds
-    spool   = get(ENV, "FS_SPOOL_DIR",   "data/spool"),
+# every in-flight file at every stage, and which stage it has reached lives in
-    known   = get(ENV, "FS_KNOWN_DIR",   "data/known"),
+# the queue holding its reference (src/worker.jl header). So this depth is
-    unknown = get(ENV, "FS_UNKNOWN_DIR", "data/unknown"),
+# "files in flight", full stop; per-stage depth comes from /stats, which is the
-    text    = get(ENV, "FS_TEXT_DIR",    "data/text"),
+# only place that can see it at all.
-)
+stagedirs() = (spool = get(ENV, "FS_SPOOL_DIR", "data/spool"),)
 
 "Count work items in `dir`, ignoring the .meta.json sidecars stages 2/4 write."
 function count_files(dir::AbstractString)::Int
@@ -513,7 +514,7 @@ function main(argv)
     # Leftovers mid-pipeline would land in the sinks during our window and be
     # counted as our throughput, so say so up front rather than quietly skewing.
     pending = total(counts(stages))
-    pending > 0 && @warn "pipeline is not idle: $pending file(s) in the stage dirs; " *
+    pending > 0 && @warn "pipeline is not idle: $pending file(s) still in spool/; " *
                          "throughput will include their completions"
 
     # --- corpus
@@ -570,7 +571,7 @@ function main(argv)
                   "(the server predates src/stats.jl)"
         end
 
-        # --- sampler: RSS curve, stage dir depths, and queue depths.
+        # --- sampler: RSS curve, spool depth, and queue depths.
         stop = Threads.Atomic{Bool}(false)
         rss_samples = Float64[]
         depth_max = Dict(k => 0 for k in keys(stages))
@@ -731,7 +732,7 @@ function main(argv)
             println("\nnote: $(sink_delta.failed) file(s) landed in $(sinks.failed) — check the server log.")
         timed_out &&
             println("\nnote: drain stalled with $(accepted - completed) file(s) outstanding. " *
-                    "Check the server log and the stage dirs; raise --timeout if the pipeline is just slow.")
+                    "Check the server log and spool/; raise --timeout if the pipeline is just slow.")
 
         if opts["json"] !== nothing
             result = (

bin/bench_stage1.jl

@@ -10,7 +10,6 @@
 #   classify        filesize + two 16-byte reads + a 32x1 forward pass
 #     read_features   open, read head, seek, read tail, scale to Float32
 #     Lux.apply       the network on a feature vector already in memory
-#   move_to         rename spool/<f> -> known/<f> or unknown/<f>
 #   enqueue         push a Job reference onto the downstream bounded queue
 #   logging         two @info lines ("classified file", "routed to ...")
 #
@@ -36,8 +35,8 @@
 # Usage:
 #   julia --project=. -t auto bin/bench_stage1.jl [options]
 #
-#   --files N        files per timed pass for consuming benchmarks (default: 2000)
+#   --files N        files per timed pass for whole-corpus benchmarks (default: 2000)
-#   --reps N         calls per timed pass for non-consuming benchmarks (default: 20000)
+#   --reps N         calls per timed pass for per-call benchmarks (default: 20000)
 #   --trials N       timed passes; the minimum is reported (default: 5)
 #   --size SPEC      corpus file size (default: 64k)
 #   --dir PATH       working directory for the corpus (default: a temp dir under data/)
@@ -113,8 +112,8 @@ const SINK = Ref{Any}(nothing)
 
 Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
 (`pass` returns the number of operations it performed). `prepare()` runs before
-each pass and is *not* timed — that is where a consuming benchmark puts the file
+each pass and is *not* timed — that is where a benchmark resets whatever its
-back where it started. `pass` comes first so callers can pass it as a `do` block.
+last pass consumed (today: the queues). `pass` comes first so callers can pass it as a `do` block.
 
 The first pass is thrown away: it pays Julia's JIT compilation, which on calls
 this small is orders of magnitude more than the thing being measured.
@@ -198,26 +197,11 @@ function make_corpus(cfg::FS.Config, n::Int, size::Int, rng)
     return jobs
 end
 
-"""
+# There is no `respool!` step any more. Stage 1 used to consume its input by
-    respool!(cfg, jobs)
+# renaming each file into known/ or unknown/, so every repeated pass had to put
-
+# the corpus back first. It now routes by enqueueing alone and leaves the file in
-Put every corpus file back in `spool/`, wherever the last pass left it (known/,
+# spool/, so the corpus is reusable as-is and the only per-pass reset is draining
-unknown/, or already home). This is the untimed `prepare` step for benchmarks
+# the queues.
-that consume their input by moving it.
-"""
-function respool!(cfg::FS.Config, jobs::Vector{FS.Job})
-    for job in jobs
-        isfile(job.path) && continue
-        for dir in (cfg.known_dir, cfg.unknown_dir, cfg.failed_dir)
-            candidate = joinpath(dir, basename(job.path))
-            if isfile(candidate)
-                mv(candidate, job.path; force = true)
-                break
-            end
-        end
-    end
-    return nothing
-end
 
 "Drain a queue without blocking, so the next pass starts from empty."
 function drain!(q::FS.ChannelQueue)
@@ -263,10 +247,10 @@ end
 """
     component_rows(cfg, clf, jobs, opts) -> Vector
 
-Time each piece of stage 1 on its own. Non-consuming pieces (`filesize`,
+Time each piece of stage 1 on its own. Per-call pieces (`filesize`,
 `read_features`, `Lux.apply`, `classify`, the log lines) run `reps` times over
-the corpus; consuming pieces (`move_to`, the full handler) run once per corpus
+the corpus; whole-corpus pieces (`enqueue_blocking!`, the full handler) run once
-file with an untimed reset between passes.
+per corpus file, with an untimed queue drain between passes.
 """
 function component_rows(cfg::FS.Config, clf::FS.Classifier, jobs::Vector{FS.Job}, opts)
     reps, trials = opts["reps"], opts["trials"]
@@ -317,14 +301,11 @@ function component_rows(cfg::FS.Config, clf::FS.Classifier, jobs::Vector{FS.Job}
             reps
         end))
 
-    # --- move_to: the rename out of spool/. Consuming: reset before each pass.
+    # Stage 1 used to rename each file into known/ or unknown/ before enqueueing
-    push!(rows, (; name = "move_to (rename)", part = "route",
+    # it, and that rename was measured here as its own line item. It is gone:
-        ns = best_of(() -> respool!(cfg, jobs); trials) do
+    # routing is the enqueue alone, and a file does not move until it is
-            @inbounds for job in jobs
+    # committed to a terminal sink (src/worker.jl header). So `route` below is
-                SINK[] = FS.move_to(cfg.unknown_dir, job)
+    # now just the queue handoff.
-            end
-            nfiles
-        end))
 
     # --- enqueue: lock, push, notify on an uncontended, non-full queue
     q = FS.ChannelQueue(nfiles + 1)
@@ -381,7 +362,7 @@ function handler_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
                            (:flush, "handle_classify_job (flush→file)"),
                            (:debug, "handle_classify_job (JULIA_DEBUG)"))
         ns = with_logger_named(which, logfile) do
-            best_of(() -> (respool!(cfg, jobs); drain!(known); drain!(unknown)); trials) do
+            best_of(() -> (drain!(known); drain!(unknown)); trials) do
                 @inbounds for job in jobs
                     FS.handle_classify_job(job, cfg, 1, known, unknown, stats)
                 end
@@ -390,7 +371,7 @@ function handler_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
         end
         push!(rows, (; name = label, part = "total", ns))
     end
-    respool!(cfg, jobs); drain!(known); drain!(unknown)
+    drain!(known); drain!(unknown)
     rm(logfile; force = true)
     return rows
 end
@@ -419,7 +400,7 @@ function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
     base = 0.0
     for k in counts
         ns = with_logger_named(:flush, logfile) do
-            best_of(() -> (respool!(cfg, jobs); drain!(known); drain!(unknown)); trials) do
+            best_of(() -> (drain!(known); drain!(unknown)); trials) do
                 # Static split: each task takes a contiguous slice, so the only
                 # sharing between workers is the state the server also shares —
                 # the classifier, the queues, the logger, the filesystem.
@@ -441,7 +422,7 @@ function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
         k == counts[1] && (base = r)
         push!(rows, (; workers = k, ns, files_per_sec = r, speedup = r / base))
     end
-    respool!(cfg, jobs); drain!(known); drain!(unknown)
+    drain!(known); drain!(unknown)
     rm(logfile; force = true)
     return rows
 end
@@ -481,12 +462,10 @@ function main(argv)
     owned = opts["dir"] === nothing
     cfg = FS.Config(
         spool_dir   = joinpath(root, "spool"),
-        known_dir   = joinpath(root, "known"),
-        unknown_dir = joinpath(root, "unknown"),
         failed_dir  = joinpath(root, "failed"),
         model_path  = modelpath,
     )
-    for d in (cfg.spool_dir, cfg.known_dir, cfg.unknown_dir, cfg.failed_dir)
+    for d in (cfg.spool_dir, cfg.failed_dir)
         mkpath(d)
     end
 
@@ -524,7 +503,7 @@ function main(argv)
         # Stage 1's own log lines are `@debug`, so what the deployed handler pays
         # for them is the disabled-macro cost, not a formatted line.
         accounted = sum(r.ns for r in comps if r.name in
-            ("classify (total)", "move_to (rename)", "enqueue_blocking!", "logging (NullLogger)"))
+            ("classify (total)", "enqueue_blocking!", "logging (NullLogger)"))
         println()
         @printf("accounted: %s of %s (%.0f%%); unaccounted overhead %s\n",
                 human_time(accounted), human_time(total), 100 * accounted / total,

bin/bench_stage2.jl

@@ -17,7 +17,7 @@
 #     JSON3.write         serialize the sidecar payload
 #     write + fsync       durably persist the sidecar bytes to a temp name
 #     mv + fsync_dir      commit the sidecar, then persist the rename itself
-#     move_to             rename known/<f> -> done/<f>, the commit point
+#     move_to             rename spool/<f> -> done/<f>, the commit point
 #   logging             one @info line ("enriched")
 #
 # This script times each of those in isolation, then times the real
@@ -28,7 +28,7 @@
 #   * The corpus must be real files. exiftool's cost depends on what it finds;
 #     random bytes exit early and would understate the stage by a lot. The
 #     default corpus is `data/done` — files that already went through stage 2 on
-#     this machine — copied back into a scratch known/ dir.
+#     this machine — copied back into a scratch spool/ dir.
 #   * Two rows price the *alternatives* to one-fork-per-file, because if the
 #     fork dominates then the only fixes are to stop paying it per file:
 #     `exiftool (batched Nx)` runs the whole corpus through one process, and
@@ -173,7 +173,7 @@ end
 """
     make_corpus(cfg, corpus_dir, n) -> Vector{Job}
 
-Copy up to `n` real files from `corpus_dir` into `known/` and build the `Job`
+Copy up to `n` real files from `corpus_dir` into `spool/` and build the `Job`
 references a stage-2 worker would dequeue for them — the exact input
 `handle_known_job` sees.
 
@@ -198,21 +198,23 @@ function make_corpus(cfg::FS.Config, corpus_dir::AbstractString, n::Int)
         # corpus the file came from.
         id, spooled = FS.spool_path(cfg, @sprintf("s2-%04d-%s", i, basename(name)))
         cp(src, spooled; force = true)
-        dest = joinpath(cfg.known_dir, basename(spooled))
+        # Staged in spool/, not a known/ dir: stage 1 routes by enqueueing and
-        mv(spooled, dest; force = true)
+        # leaves the bytes where intake put them, so this is the exact on-disk
-        push!(jobs, FS.Job(id, basename(name), dest, filesize(dest), time()))
+        # state a stage-2 worker dequeues into (src/worker.jl header).
+        push!(jobs, FS.Job(id, basename(name), spooled, filesize(spooled), time()))
     end
     return jobs
 end
 
 """
-    reknown!(cfg, jobs)
+    respool!(cfg, jobs)
 
-Put every corpus file back in `known/`, wherever the last pass left it (done/ or
+Put every corpus file back in `spool/`, wherever the last pass left it (done/ or
 already home), and delete any sidecar it produced. This is the untimed `prepare`
-step for benchmarks that consume their input by committing it.
+step for benchmarks that consume their input by committing it — stage 2 still
+moves, because its move is the terminal commit, not an inter-stage hop.
 """
-function reknown!(cfg::FS.Config, jobs::Vector{FS.Job})
+function respool!(cfg::FS.Config, jobs::Vector{FS.Job})
     for job in jobs
         base = basename(job.path)
         for dir in (cfg.done_dir, cfg.failed_dir)
@@ -463,8 +465,8 @@ function component_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
         nio
     end)
 
-    # --- move_to: the rename known/<f> -> done/<f>. Consuming: reset each pass.
+    # --- move_to: the rename spool/<f> -> done/<f>. Consuming: reset each pass.
-    add!("move_to (rename)", "commit", best_of(() -> reknown!(cfg, jobs); trials) do
+    add!("move_to (rename)", "commit", best_of(() -> respool!(cfg, jobs); trials) do
         @inbounds for job in jobs
             SINK[] = FS.move_to(cfg.done_dir, job)
         end
@@ -474,17 +476,17 @@ function component_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
     # --- commit_enriched!: the whole sidecar-first commit, extraction excluded.
     committable = [j for (j, _) in good]
     add!("commit_enriched! (total)", "commit",
-        best_of(() -> reknown!(cfg, jobs); trials) do
+        best_of(() -> respool!(cfg, jobs); trials) do
             @inbounds for (k, job) in enumerate(committable)
                 SINK[] = FS.commit_enriched!(cfg.done_dir, job, metas[k])
             end
             length(committable)
         end)
-    reknown!(cfg, jobs)
+    respool!(cfg, jobs)
 
     # --- the one @info line, under each logger.
     job1, meta1 = good[1][1], metas[1]
-    logfile = joinpath(dirname(cfg.known_dir), "bench_stage2.log")
+    logfile = joinpath(dirname(cfg.spool_dir), "bench_stage2.log")
     for (which, label) in ((:null, "logging (NullLogger)"),
                            (:format, "logging (format only)"),
                            (:flush, "logging (flush→file)"))
@@ -513,13 +515,13 @@ running server actually pays.
 function handler_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
     trials = opts["trials"]
     nfiles = length(jobs)
-    logfile = joinpath(dirname(cfg.known_dir), "bench_stage2.log")
+    logfile = joinpath(dirname(cfg.spool_dir), "bench_stage2.log")
     rows = []
     for (which, label) in ((:null, "handle_known_job (NullLogger)"),
                            (:format, "handle_known_job (format only)"),
                            (:flush, "handle_known_job (flush→file)"))
         ns = with_logger_named(which, logfile) do
-            best_of(() -> reknown!(cfg, jobs); trials) do
+            best_of(() -> respool!(cfg, jobs); trials) do
                 @inbounds for job in jobs
                     FS.handle_known_job(job, cfg, 1)
                 end
@@ -528,7 +530,7 @@ function handler_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
         end
         push!(rows, (; name = label, part = "total", ns))
     end
-    reknown!(cfg, jobs)
+    respool!(cfg, jobs)
     rm(logfile; force = true)
     return rows
 end
@@ -556,13 +558,13 @@ function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
         [parse(Int, s) for s in split(String(opts["threads"]), ",")]
     counts = sort(unique(filter(k -> 1 <= k <= Threads.nthreads(), counts)))
 
-    logfile = joinpath(dirname(cfg.known_dir), "bench_stage2.log")
+    logfile = joinpath(dirname(cfg.spool_dir), "bench_stage2.log")
     rows = []
     base = 0.0
     for k in counts
         ns = with_logger_named(:flush, logfile) do
             next = Threads.Atomic{Int}(1)
-            best_of(() -> (reknown!(cfg, jobs); next[] = 1); trials) do
+            best_of(() -> (respool!(cfg, jobs); next[] = 1); trials) do
                 # Shared counter, not a contiguous slice: every worker takes the
                 # next unclaimed file the moment it frees up, exactly as the
                 # server's pool takes the next job off the known queue.
@@ -582,7 +584,7 @@ function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
         k == counts[1] && (base = r)
         push!(rows, (; workers = k, ns, files_per_sec = r, speedup = r / base))
     end
-    reknown!(cfg, jobs)
+    respool!(cfg, jobs)
     rm(logfile; force = true)
     return rows
 end
@@ -622,12 +624,11 @@ function main(argv)
     owned = opts["dir"] === nothing
     cfg = FS.Config(
         spool_dir        = joinpath(root, "spool"),
-        known_dir        = joinpath(root, "known"),
         done_dir         = joinpath(root, "done"),
         failed_dir       = joinpath(root, "failed"),
         exiftool_timeout = opts["timeout"],
     )
-    for d in (cfg.spool_dir, cfg.known_dir, cfg.done_dir, cfg.failed_dir)
+    for d in (cfg.spool_dir, cfg.done_dir, cfg.failed_dir)
         mkpath(d)
     end
 

src/FileServer.jl

@@ -117,18 +117,13 @@ function run(; overrides...)
     DETECTOR[] = LanguageDetector()
     @info "loaded language detector"
 
-    # Stage-aware recovery: re-drive each stage's leftovers onto its own queue so
+    @info "starting file-server" host=cfg.host port=cfg.port workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count capacity=cfg.queue_capacity known_capacity=cfg.known_queue_capacity unknown_capacity=cfg.unknown_queue_capacity text_capacity=cfg.text_queue_capacity
-    # files resume where they were, not from scratch. spool/ → stage-1,
-    # known/ → stage-2, unknown/ → stage-3.
-    recovered         = recover_dir!(cfg.spool_dir, queue)
-    recovered_known   = recover_dir!(cfg.known_dir, known_queue)
-    recovered_unknown = recover_dir!(cfg.unknown_dir, unknown_queue)
-    recovered_text    = recover_dir!(cfg.text_dir, text_queue)
-    @info "starting file-server" host=cfg.host port=cfg.port workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count capacity=cfg.queue_capacity known_capacity=cfg.known_queue_capacity unknown_capacity=cfg.unknown_queue_capacity text_capacity=cfg.text_queue_capacity recovered=recovered recovered_known=recovered_known recovered_unknown=recovered_unknown recovered_text=recovered_text
 
     # Zero the counters here, not at module load: `since` should mean "serving
     # since", so a scrape's totals cover the run, not the minutes spent loading
-    # the classifier. Nothing has been processed yet — recovery only enqueues.
+    # the classifier. It must also come *before* the pools start, since recovery
+    # below runs against live workers — the replayed files are real work and
+    # belong in the totals, but only from this point on.
     reset_metrics!()
 
     # Each pool gets its stage's counters (src/stats.jl); `worker_loop` records
@@ -151,6 +146,20 @@ function run(; overrides...)
                    st.language)
                     for i in 1:cfg.text_worker_count]
 
+    # Recovery runs *after* the pools are live, and blocks rather than dropping
+    # when the queue is full. `spool/` holds every in-flight file now, so a crash
+    # under load can leave far more leftovers than `queue_capacity`; with the
+    # consumers already running the queue drains as we fill it, so a big backlog
+    # takes longer to re-drive but none of it is abandoned. Serving hasn't
+    # started yet, so intake cannot race us for the directory.
+    #
+    # Everything re-enters at stage 1 and replays: the queues are in-process, so
+    # a crash loses the only record of how far each file had got. Safe, because
+    # every stage is a pure function of the file's bytes and every commit is
+    # idempotent — see the header of src/worker.jl for the trade and the fix.
+    recovered = recover_dir!(cfg.spool_dir, queue)
+    recovered > 0 && @info "recovered leftover files; replaying from stage 1" recovered=recovered
+
     register_routes()
     # `handler` replaces Oxygen's root stream handler so POST /upload can read its
     # body incrementally instead of having it buffered into memory first; every

src/config.jl

@@ -13,7 +13,7 @@ Base.@kwdef struct Config
     known_worker_count::Int   = Threads.nthreads()
     known_queue_capacity::Int = 1000
     # Stage 3 (content triage) also gets its own pool + queue: sorting an
-    # unrecognized file into binary/ vs text/ is cheap I/O, tuned independently
+    # unrecognized file into binary vs text is cheap I/O, tuned independently
     # of the classify and enrich pools.
     unknown_worker_count::Int   = Threads.nthreads()
     unknown_queue_capacity::Int = 1000
@@ -22,11 +22,12 @@ Base.@kwdef struct Config
     # github-linguist) is a mix of CPU and process-spawn work, tuned independently.
     text_worker_count::Int   = Threads.nthreads()
     text_queue_capacity::Int = 1000
-    spool_dir::String   = "data/spool"   # files land here on intake (pending classification)
+    # Every in-flight file lives here, at every stage, from intake until it is
-    known_dir::String   = "data/known"   # classified-known, awaiting enrichment (stage 2)
+    # committed to a terminal sink below. Stages route by enqueueing a reference,
-    unknown_dir::String = "data/unknown" # classified-unknown, awaiting content triage (stage 3)
+    # not by moving bytes (src/worker.jl header), so there are no per-stage
+    # directories — the queues are what know how far a file has got.
+    spool_dir::String   = "data/spool"
     binary_dir::String  = "data/binary"  # stage-3 sink: unknown files that look like binary data
-    text_dir::String    = "data/text"    # classified-text, awaiting language enrichment (stage 4)
     done_dir::String    = "data/done"    # fully enriched known files (+ .meta.json sidecars)
     text_done_dir::String = "data/text_done" # fully enriched text files (+ .meta.json sidecars)
     failed_dir::String  = "data/failed"  # files move here if a worker throws
@@ -67,7 +68,7 @@ Recognised variables:
   FS_KNOWN_WORKERS, FS_KNOWN_QUEUE_CAPACITY,
   FS_UNKNOWN_WORKERS, FS_UNKNOWN_QUEUE_CAPACITY,
   FS_TEXT_WORKERS, FS_TEXT_QUEUE_CAPACITY,
-  FS_SPOOL_DIR, FS_KNOWN_DIR, FS_UNKNOWN_DIR, FS_BINARY_DIR, FS_TEXT_DIR,
+  FS_SPOOL_DIR, FS_BINARY_DIR,
   FS_DONE_DIR, FS_TEXT_DONE_DIR, FS_FAILED_DIR, FS_MODEL_PATH,
   FS_UPLOAD_CHUNK_BYTES, FS_EXIFTOOL_TIMEOUT, FS_LINGUIST_TIMEOUT,
   FS_CLUSTER_DIR, FS_CLUSTER_N, FS_CLUSTER_ALPHA, FS_CLUSTER_PSEUDOCOUNT,
@@ -79,8 +80,7 @@ function config_from_env(; host=nothing, port=nothing, worker_count=nothing,
                           known_queue_capacity=nothing, unknown_worker_count=nothing,
                           unknown_queue_capacity=nothing, text_worker_count=nothing,
                           text_queue_capacity=nothing, spool_dir=nothing,
-                          known_dir=nothing, unknown_dir=nothing, binary_dir=nothing,
+                          binary_dir=nothing, done_dir=nothing, text_done_dir=nothing,
-                          text_dir=nothing, done_dir=nothing, text_done_dir=nothing,
                           failed_dir=nothing, model_path=nothing,
                           upload_chunk_bytes=nothing, exiftool_timeout=nothing,
                           linguist_timeout=nothing, cluster_dir=nothing, cluster_n=nothing,
@@ -100,10 +100,7 @@ function config_from_env(; host=nothing, port=nothing, worker_count=nothing,
         text_worker_count      = something(text_worker_count,      parse(Int, get(ENV, "FS_TEXT_WORKERS", string(Threads.nthreads())))),
         text_queue_capacity    = something(text_queue_capacity,    parse(Int, get(ENV, "FS_TEXT_QUEUE_CAPACITY", "1000"))),
         spool_dir              = something(spool_dir,              get(ENV, "FS_SPOOL_DIR", "data/spool")),
-        known_dir              = something(known_dir,              get(ENV, "FS_KNOWN_DIR", "data/known")),
-        unknown_dir            = something(unknown_dir,            get(ENV, "FS_UNKNOWN_DIR", "data/unknown")),
         binary_dir             = something(binary_dir,             get(ENV, "FS_BINARY_DIR", "data/binary")),
-        text_dir               = something(text_dir,               get(ENV, "FS_TEXT_DIR", "data/text")),
         done_dir               = something(done_dir,               get(ENV, "FS_DONE_DIR", "data/done")),
         text_done_dir          = something(text_done_dir,          get(ENV, "FS_TEXT_DONE_DIR", "data/text_done")),
         failed_dir             = something(failed_dir,             get(ENV, "FS_FAILED_DIR", "data/failed")),
@@ -125,9 +122,8 @@ end
 
 "Create all the pipeline-stage directories if they don't already exist."
 function ensure_dirs(cfg::Config)
-    for d in (cfg.spool_dir, cfg.known_dir, cfg.unknown_dir, cfg.binary_dir,
+    for d in (cfg.spool_dir, cfg.binary_dir, cfg.done_dir, cfg.text_done_dir,
-              cfg.text_dir, cfg.done_dir, cfg.text_done_dir, cfg.failed_dir,
+              cfg.failed_dir, cfg.nominated_dir)
-              cfg.nominated_dir)
         mkpath(d)
     end
     return nothing

src/content.jl

@@ -1,8 +1,10 @@
 # Stage-3 content triage for unknown files.
 #
 # A file that stage-1 couldn't recognize is still sorted into one of two coarse
-# buckets so downstream tooling can treat them differently: `text/` for
+# buckets so downstream tooling can treat them differently: text (human-readable)
-# human-readable content, `binary/` for everything else. We sniff only the first
+# and binary (everything else). Only binary is a directory — it is terminal, so
+# the file is committed to `binary/`; a text file has stage 4 still to come, so it
+# stays in `spool/` and only its queue reference moves on. We sniff only the first
 # `CONTENT_SNIFF_BYTES` (no full read) and ask two questions: does the window
 # decode as valid UTF-8, and are any of its control bytes ones that don't belong
 # in text? This is the Unicode-aware successor to the classic "NUL byte" test —

src/language.jl

@@ -1,7 +1,7 @@
 # Stage-4 language enrichment for text files.
 #
-# A file that stage-3 sorted into `text/` is human-readable, but we don't yet
+# A file that stage-3 sorted as text is human-readable, but we don't yet know
-# know *what* it is. This stage answers two questions and records them in a
+# *what* it is. This stage answers two questions and records them in a
 # `.meta.json` sidecar, exactly like the stage-2 known-file enrichment:
 #
 #   * natural language  — via Languages.jl's `LanguageDetector` (a Julia port of

src/queue.jl

@@ -8,6 +8,15 @@
 
 abstract type JobQueue end
 
+# How long a producer backs off before retrying an enqueue onto a full queue.
+# Blocking backpressure: a file that is already on disk is never dropped, so the
+# producer parks until the consumer makes room. Used by the stage-1 and stage-3
+# routing handoffs and by startup recovery (see `enqueue_blocking!`). Intake is
+# deliberately *not* a user: the HTTP path's `enqueue!` stays non-blocking and
+# turns a full queue into a 503, so a saturated pipeline slows uploads down
+# rather than holding request threads hostage.
+const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05
+
 """
     enqueue!(q, job) -> Bool
 

src/spool.jl

@@ -48,7 +48,15 @@ function spool_stream(write_body!, cfg::Config, original_name::AbstractString)::
     return Job(id, String(original_name), path, nbytes, time())
 end
 
-"Move a spooled file into `dir` (done/ or failed/), returning the destination."
+"""
+    move_to(dir, job) -> String
+
+Move a spooled file into `dir`, returning the destination.
+
+This runs once per file, at the end: into a terminal sink (`done/`, `text_done/`,
+`binary/`) or into `failed/` when a handler throws. Files are *not* moved between
+stages — see the header of src/worker.jl.
+"""
 function move_to(dir::AbstractString, job::Job)::String
     dest = joinpath(dir, basename(job.path))
     mv(job.path, dest; force=true)
@@ -64,12 +72,24 @@ const UUID_LEN = 36
 
 Re-enqueue any files sitting in `dir` (left by a crash, hard shutdown, or an
 intake that never finished) onto `queue`. This is the payoff of spooling to
-disk: a restart resumes work instead of stranding it. Stage-aware recovery uses
+disk: a restart resumes work instead of stranding it. Returns the number of
-one call per stage — `spool/` → stage-1 queue, `known/` → known queue — so each
+files recovered.
-file re-enters at the correct stage rather than being reclassified from scratch.
-Returns the number of files recovered.
 
-Skips `.meta.json` sidecars: those are stage-2 output, not work to redo.
+A file's stage is not recorded on disk — it lives in whichever in-process queue
+holds its reference, and a crash loses that (src/worker.jl header). So everything
+still in `spool/` re-enters at stage 1 and replays. Replay is safe rather than
+merely tolerable: classification and the binary/text sniff are pure functions of
+the file's bytes, and the terminal commits rename with `force=true`, so a
+replayed file lands where it would have landed and overwrites its own sidecar.
+
+**This blocks on a full queue**, and that is the whole point: `spool/` now holds
+every in-flight file, so a crash under load can easily leave more leftovers than
+one queue's capacity. Dropping the excess would mean recovery quietly losing the
+work it exists to save. Because the caller spawns the worker pools *before*
+recovering, the consumers are live and the queue drains as we fill it — parking
+here costs latency, never progress.
+
+Skips `.meta.json` sidecars: those are enrichment output, not work to redo.
 """
 function recover_dir!(dir::AbstractString, queue::JobQueue)::Int
     n = 0
@@ -85,7 +105,10 @@ function recover_dir!(dir::AbstractString, queue::JobQueue)::Int
             name = fname
         end
         job = Job(id, name, path, filesize(path), time())
-        enqueue!(queue, job) || @warn "queue full during recovery; leaving file for next start" path
+        # `nothing` stats: recovery is not a stage, so its wait must not be
+        # charged to any stage's blocked_ns (see `enqueue_blocking!`).
+        enqueue_blocking!(queue, job, nothing;
+                          retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
         n += 1
     end
     return n

src/stats.jl

@@ -1,10 +1,12 @@
 # Per-stage pipeline metrics.
 #
 # End-to-end throughput says how fast the pipeline is; it does not say which
-# stage is the reason. The external harness can't answer that either: `known/`,
+# stage is the reason. The external harness can't answer that either, and now
-# `unknown/` and `text/` are *transient* — a file can pass through one between
+# cannot even approximate it: every in-flight file sits in `spool/` regardless of
-# two directory polls — so sampling directories from outside undercounts exactly
+# how far it has got, because stages route by enqueueing a reference rather than
-# the stages we most want to measure. Only the pipeline itself sees every job.
+# by moving bytes (src/worker.jl header). A file's stage is a property of the
+# queue holding it, so only the pipeline itself can see it — there is no
+# directory to poll.
 #
 # So each stage keeps four counters, all incremented in `worker_loop` (the one
 # place every stage's work passes through) and read by `GET /stats`:
@@ -154,15 +156,22 @@ so a full downstream queue means waiting, not failing. Wrapping the retry loop
 here — rather than repeating `while !enqueue! sleep end` at each call site — is
 what makes that wait measurable at all, and keeps the three routing paths from
 drifting into three different backoff behaviours.
+
+`stats` is `nothing` for callers that are not a pipeline stage — startup
+recovery (`recover_dir!`) uses the same never-drop retry, but it runs outside any
+handler, so it has no `busy_ns` to charge against. Billing its wait to a stage's
+`blocked_ns` would make `/stats` utilization — `(Δbusy_ns - Δblocked_ns) / …` —
+go *negative* after a long recovery. Skipping the charge keeps `blocked_ns`
+meaning exactly what its docs say: time a handler spent parked on its successor.
 """
-function enqueue_blocking!(queue::JobQueue, job::Job, stats::StageStats;
+function enqueue_blocking!(queue::JobQueue, job::Job, stats::Union{StageStats,Nothing};
                            retry_seconds::Real)
     enqueue!(queue, job) && return nothing
     t0 = time_ns()
     while !enqueue!(queue, job)
         sleep(retry_seconds)
     end
-    Threads.atomic_add!(stats.blocked_ns, Int(time_ns() - t0))
+    stats === nothing || Threads.atomic_add!(stats.blocked_ns, Int(time_ns() - t0))
     return nothing
 end
 

src/worker.jl

@@ -3,28 +3,48 @@
 # stage, so `worker_loop` is parametrized with a `handler` and reused. Today
 # there are four stages:
 #
-#   stage 1  handle_classify_job  spool/  → classify → known/ (+known queue) | unknown/ (+unknown queue)
+#   stage 1  handle_classify_job  classify → known queue | unknown queue
-#   stage 2  handle_known_job     known/  → exiftool enrich → done/ (+ .meta.json)
+#   stage 2  handle_known_job     exiftool enrich → done/ (+ .meta.json)
-#   stage 3  handle_unknown_job   unknown/ → binary-vs-text sniff → binary/ | text/ (+text queue)
+#   stage 3  handle_unknown_job   binary-vs-text sniff → binary/ | text queue
-#   stage 4  handle_text_job      text/   → language enrich → text_done/ (+ .meta.json)
+#   stage 4  handle_text_job      language enrich → text_done/ (+ .meta.json)
 #
 # Adding a stage later is just another queue + pool + handler; the loop below
 # doesn't change.
-
+#
-# How long a stage-1 worker backs off before retrying an enqueue onto a full
+# A file does not move between stages. It is written once into `spool/` at
-# downstream queue (known or unknown). Blocking backpressure: a classified file
+# intake and stays there for its whole in-flight life; only the small `Job`
-# is never dropped, so the stage-1 worker parks until the next stage makes room.
+# reference travels, and `job.path` is therefore constant from intake until the
-# Keeps intake decoupled — the HTTP path's `enqueue!` stays non-blocking; only
+# file is committed. The stage a file has reached lives in the queue holding its
-# this worker-to-worker handoff blocks.
+# reference, not in which directory the bytes sit — so the intermediate hops
-const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05
+# (spool→known, spool→unknown, unknown→text) are three renames per file that buy
+# nothing on the live path. What remains is one move at the end: into a terminal
+# sink (`done/`, `text_done/`, `binary/`) or into `failed/` on a throw.
+#
+# The cost is on restart. The queues are in-process (src/queue.jl), so a crash
+# loses them, and recovery can only re-drive everything in `spool/` from stage 1.
+# That is safe — classification and the UTF-8 sniff are pure functions of the
+# file's bytes and every commit is idempotent — but it redoes work the old
+# directory-per-stage layout could skip. The durable fix is the queue seam, not
+# the directories: a broker-backed `JobQueue` restores exact resume for free.
+#
+# (`ROUTE_ENQUEUE_RETRY_SECONDS`, the backoff the routing handoffs below use,
+# now lives in src/queue.jl, since startup recovery shares it.)
 
 # Per-file logging in stage 1 is `@debug`, not `@info`, because it is the
 # stage's dominant cost. Measured by bin/bench_stage1.jl (2000 x 64 KiB files,
-# min of 5 trials): the two log lines cost ~71 µs of the ~118 µs
+# min of 5 trials): a formatted pair of log lines costs 71.2 µs per file, against
-# `handle_classify_job` spent per file — roughly 6x the classifier (10.6 µs) and
+# the 11.7 µs the whole handler takes with them switched off — six times the rest
-# 6x the rename (11.6 µs). Nearly all of it is `ConsoleLogger` formatting
+# of the stage put together. Nearly all of it is `ConsoleLogger` formatting
-# (~64 µs); the FlushLogger's per-message flush is only ~8 µs on top. Demoting
+# (64.2 µs); the FlushLogger's per-message flush is only ~7 µs on top. Switching
-# them takes stage 1 from ~8.5k files/s to ~35k files/s on one worker.
+# them back on with `JULIA_DEBUG=FileServer` takes the handler to 106.3 µs, i.e.
+# from 85.4k files/s down to 9.4k on one worker.
+#
+# With logging off the stage is the classifier and nothing else: classify 10.30 µs
+# (of which read_features is 7.82 µs), the queue handoff 0.12 µs, the disabled
+# `@debug` lines 0.29 µs — 10.7 µs of the 11.7 µs total. Removing the inter-stage
+# rename is what left it that way: that rename was 11.6 µs per file, so it was
+# the equal of the classifier, and dropping it took one worker from ~35k to
+# ~85k files/s (16 workers: ~333k files/s).
 #
 # `@debug` is compiled to a min-level check that doesn't evaluate its arguments,
 # so a disabled line costs ~0.15 µs rather than ~36 µs. The messages are still
@@ -38,11 +58,13 @@ const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05
 
 Stage 1. Classify the spooled file and route it to the next stage's queue,
 retrying on a full queue rather than dropping the file:
-  * `:known`   → move to `known/`,   enqueue onto the known queue for stage 2.
+  * `:known`   → enqueue onto the known queue for stage 2.
-  * `:unknown` → move to `unknown/`, enqueue onto the unknown queue for stage 3.
+  * `:unknown` → enqueue onto the unknown queue for stage 3.
 
-In both cases move first so the file physically lives in its stage dir before the
+Routing is the enqueue and nothing else: the file stays where intake wrote it and
-reference is visible downstream; the moved path becomes the routed job's location.
+the *same* `Job` is handed on, so `job.path` still points at it. Reaching stage 2
+is a fact about which queue holds the reference, not about which directory holds
+the bytes.
 
 Sub-`MIN_FILE_BYTES` files short-circuit to `:unknown` inside `classify`.
 """
@@ -53,18 +75,14 @@ function handle_classify_job(job::Job, cfg::Config, worker_id::Int,
     @debug "classified file" worker=worker_id id=job.id name=job.original_name size=job.size classification=classification
 
     if classification === :known
-        dest = move_to(cfg.known_dir, job)
-        routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
         # known queue full → park and retry, don't drop (time charged to blocked_ns)
-        enqueue_blocking!(known_queue, routed, stats;
+        enqueue_blocking!(known_queue, job, stats;
                           retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
-        @debug "routed to enrichment" worker=worker_id id=job.id dest=dest
+        @debug "routed to enrichment" worker=worker_id id=job.id path=job.path
     else
-        dest = move_to(cfg.unknown_dir, job)
+        enqueue_blocking!(unknown_queue, job, stats;
-        routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
-        enqueue_blocking!(unknown_queue, routed, stats;
                           retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
-        @debug "routed to content triage" worker=worker_id id=job.id dest=dest
+        @debug "routed to content triage" worker=worker_id id=job.id path=job.path
     end
     return nothing
 end
@@ -73,7 +91,8 @@ end
     handle_known_job(job, cfg, worker_id)
 
 Stage 2. Extract metadata (exiftool, with timeout) and enrich: build the
-normalized sidecar and commit both to `done/` sidecar-first. Extraction
+normalized sidecar and commit both to `done/` sidecar-first — the file's one and
+only move, straight out of `spool/`. Extraction
 failure/timeout yields a *degraded* sidecar (the file is still a wanted known
 file), so the only way to land in `failed/` is a genuine I/O error writing the
 sidecar or moving the file — handled by `worker_loop`'s quarantine.
@@ -89,10 +108,14 @@ end
     handle_unknown_job(job, cfg, worker_id, text_queue)
 
 Stage 3. Sort an unrecognized file into a coarse content bucket by sniffing its
-first bytes: `binary/` (terminal — no further stage) if it looks like binary
+first bytes.
-data, `text/` otherwise. A text file is then routed onward to the stage-4
+
-language-enrichment queue, retrying on a full queue rather than dropping the file
+The two outcomes are asymmetric, because one is terminal and one is not. Binary
-(the same blocking backpressure stage 1 uses for its downstream queues).
+is the end of the live path, so the file is committed to `binary/` — which is
+also where the offline stage-5 discovery sweep reads its corpus, so the move is
+load-bearing, not bookkeeping. Text has a stage 4 still to come, so nothing moves:
+the same `Job` goes onto the language-enrichment queue, retrying on a full queue
+rather than dropping the file (the blocking backpressure stage 1 also uses).
 """
 function handle_unknown_job(job::Job, cfg::Config, worker_id::Int,
                             text_queue::JobQueue, stats::StageStats)
@@ -100,12 +123,10 @@ function handle_unknown_job(job::Job, cfg::Config, worker_id::Int,
         dest = move_to(cfg.binary_dir, job)
         @info "sorted unknown" worker=worker_id id=job.id name=job.original_name kind=:binary dest=dest
     else
-        dest = move_to(cfg.text_dir, job)
-        routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
         # text queue full → park and retry, don't drop (time charged to blocked_ns)
-        enqueue_blocking!(text_queue, routed, stats;
+        enqueue_blocking!(text_queue, job, stats;
                           retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
-        @info "routed to language enrichment" worker=worker_id id=job.id dest=dest
+        @info "routed to language enrichment" worker=worker_id id=job.id path=job.path
     end
     return nothing
 end
@@ -115,7 +136,8 @@ end
 
 Stage 4. Enrich a text file with its natural language (via `detector`) and
 programming language (via github-linguist): build the sidecar and commit both to
-`text_done/` sidecar-first. Detection failure yields a *degraded* sidecar (the
+`text_done/` sidecar-first — the file's one and only move, straight out of
+`spool/`. Detection failure yields a *degraded* sidecar (the
 file is still wanted text), so the only way to land in `failed/` is a genuine I/O
 error committing — handled by `worker_loop`'s quarantine.
 """

test/runtests.jl

@@ -8,7 +8,8 @@ using JSON3
 using FileServer: Job, Config, ChannelQueue, enqueue!, dequeue!, close!, length,
                   sanitize_filename, recover_dir!, normalize_metadata,
                   build_metadata, finalize_known!, run_exiftool,
-                  is_binary, handle_unknown_job, worker_loop,
+                  is_binary, handle_unknown_job, handle_classify_job, worker_loop,
+                  load_classifier, CLASSIFIER,
                   capacity, StageStats, IntakeStats, Metrics, METRICS, reset_metrics!,
                   record_job!, enqueue_blocking!, stats_snapshot, STAGE_KEYS,
                   detect_natural_language, run_linguist, detect_programming_language,
@@ -70,10 +71,7 @@ end
 function tmp_config(root; kwargs...)
     cfg = Config(;
         spool_dir   = joinpath(root, "spool"),
-        known_dir   = joinpath(root, "known"),
-        unknown_dir = joinpath(root, "unknown"),
         binary_dir  = joinpath(root, "binary"),
-        text_dir    = joinpath(root, "text"),
         done_dir    = joinpath(root, "done"),
         text_done_dir = joinpath(root, "text_done"),
         failed_dir  = joinpath(root, "failed"),
@@ -279,7 +277,7 @@ end
         mktempdir() do root
             cfg = tmp_config(root; exiftool_timeout=5)
             # Point at a nonexistent file → exiftool exits non-zero → degraded.
-            job = Job("id-3", "gone.dat", joinpath(cfg.known_dir, "id-3-gone.dat"), 99, 0.0)
+            job = Job("id-3", "gone.dat", joinpath(cfg.spool_dir, "id-3-gone.dat"), 99, 0.0)
             m = build_metadata(job, cfg)
             @test m.error !== nothing
             @test m.file_type === nothing
@@ -341,7 +339,7 @@ end
         mktempdir() do root
             cfg = tmp_config(root)
             # A real known-stage file to enrich.
-            src = joinpath(cfg.known_dir, "id-9-pixel.png")
+            src = joinpath(cfg.spool_dir, "id-9-pixel.png")
             write(src, PNG_1x1)
             job = Job("id-9", "pixel.png", src, Base.length(PNG_1x1), 0.0)
 
@@ -425,7 +423,7 @@ end
             stats = StageStats()
 
             # A binary file (embedded NUL) lands in binary/ and is NOT enqueued.
-            bpath = joinpath(cfg.unknown_dir, "id-b-blob.dat")
+            bpath = joinpath(cfg.spool_dir, "id-b-blob.dat")
             write(bpath, UInt8[0x00, 0xFF, 0x10])
             bjob = Job("id-b", "blob.dat", bpath, filesize(bpath), 0.0)
             handle_unknown_job(bjob, cfg, 1, text_queue, stats)
@@ -433,19 +431,49 @@ end
             @test !isfile(bpath)
             @test length(text_queue) == 0
 
-            # A text file lands in text/ AND is routed onto the stage-4 queue,
+            # A text file is routed onto the stage-4 queue and *does not move*:
-            # with its path updated to the new text/ location.
+            # stage 4 is still to come, so the bytes stay in spool/ and the same
-            tpath = joinpath(cfg.unknown_dir, "id-t-notes.log")
+            # reference is handed on. Regression guard against re-introducing an
+            # intermediate hop — the queue, not a directory, is what records that
+            # this file has cleared triage.
+            tpath = joinpath(cfg.spool_dir, "id-t-notes.log")
             write(tpath, "just some log text\n")
             tjob = Job("id-t", "notes.log", tpath, filesize(tpath), 0.0)
             handle_unknown_job(tjob, cfg, 1, text_queue, stats)
-            moved = joinpath(cfg.text_dir, "id-t-notes.log")
+            @test isfile(tpath)                        # stayed put
-            @test isfile(moved)
-            @test !isfile(tpath)
             @test length(text_queue) == 1
             routed = dequeue!(text_queue)
             @test routed.id == "id-t"
-            @test routed.path == moved
+            @test routed.path == tpath                 # reference unchanged
+        end
+    end
+
+    @testset "handle_classify_job: routes by enqueue only, never moves the file" begin
+        # Stage 1's entire job is picking a queue. Whichever way the classifier
+        # votes, the bytes must stay where intake wrote them and the *same* Job
+        # must be handed on — the queue records the phase, not a directory.
+        # Asserted against both queues at once so the test doesn't depend on how
+        # the committed model happens to label these bytes.
+        CLASSIFIER[] = load_classifier(joinpath(@__DIR__, "..", "model", "classifier.jld2"))
+        mktempdir() do root
+            cfg = tmp_config(root)
+            known_queue, unknown_queue = ChannelQueue(10), ChannelQueue(10)
+            stats = StageStats()
+
+            path = joinpath(cfg.spool_dir, "id-c-pixel.png")
+            write(path, PNG_1x1)
+            job = Job("id-c", "pixel.png", path, Base.length(PNG_1x1), 0.0)
+
+            handle_classify_job(job, cfg, 1, known_queue, unknown_queue, stats)
+
+            @test isfile(path)                                          # stayed put
+            @test length(known_queue) + length(unknown_queue) == 1      # routed exactly once
+            routed = length(known_queue) == 1 ? dequeue!(known_queue) : dequeue!(unknown_queue)
+            @test routed.path == path                                   # reference unchanged
+            @test routed.id == "id-c"
+            # No intermediate directory was invented alongside spool/.
+            @test !isdir(joinpath(root, "known"))
+            @test !isdir(joinpath(root, "unknown"))
         end
     end
 
@@ -503,7 +531,7 @@ end
             cfg = tmp_config(root)
             d = LanguageDetector()
 
-            src = joinpath(cfg.text_dir, "id-x-script.py")
+            src = joinpath(cfg.spool_dir, "id-x-script.py")
             write(src, join(["# a short program in English prose comment",
                              "import sys",
                              "def greet(name):",
@@ -542,7 +570,7 @@ end
             cfg = tmp_config(root)
             d = LanguageDetector()
 
-            src = joinpath(cfg.text_dir, "id-h-readme.md")
+            src = joinpath(cfg.spool_dir, "id-h-readme.md")
             write(src, "# Project\n\nThis project does something useful and interesting for everyone.\n")
             job = Job("id-h", "readme.md", src, filesize(src), 0.0)
 
@@ -739,6 +767,39 @@ end
         end
     end
 
+    @testset "recover_dir!: blocks on a full queue, recovers every file" begin
+        # The regression this guards is data loss, not slowness. `spool/` now
+        # holds every in-flight file at every stage, so a crash under load leaves
+        # far more leftovers than one queue's capacity — and the old non-blocking
+        # recovery warned and abandoned the excess, which is exactly the work
+        # recovery exists to save. With the pool live, recovery must park until
+        # the consumer makes room and come back with all of it.
+        mktempdir() do root
+            dir = joinpath(root, "spool"); mkpath(dir)
+            n_files = 25
+            for i in 1:n_files
+                write(joinpath(dir, string("id-", lpad(i, 3, '0'), "-f.dat")), "x")
+            end
+
+            q = ChannelQueue(4)          # deliberately far smaller than n_files
+            drained = Job[]
+            consumer = Threads.@spawn begin
+                while Base.length(drained) < n_files
+                    job = dequeue!(q)
+                    job === nothing && break
+                    push!(drained, job)
+                end
+            end
+
+            n = recover_dir!(dir, q)     # would strand 21 files if it didn't block
+            wait(consumer)
+
+            @test n == n_files
+            @test Base.length(drained) == n_files
+            @test Base.length(Set(j.path for j in drained)) == n_files   # no duplicates
+        end
+    end
+
     # Helper: write a "file" of raw bytes into a dir with a UUID-ish unique name,
     # returning its path. Mirrors what stage-3 deposits into binary/.
     function drop_binary(dir, bytes; name=string(rand(UInt128)))
@@ -1012,7 +1073,7 @@ end
                 text_queue = ChannelQueue(1)
                 @test enqueue!(text_queue, Job("filler", "f", "/tmp/f", 1, 0.0))
 
-                p = joinpath(cfg.unknown_dir, "id-t-notes.log")
+                p = joinpath(cfg.spool_dir, "id-t-notes.log")
                 write(p, "plain text\n")
                 job = Job("id-t", "notes.log", p, filesize(p), 0.0)
 
@@ -1024,7 +1085,7 @@ end
                 wait(drainer)
                 @test stats.blocked_ns[] > 50_000_000
                 @test length(text_queue) == 1              # the file did get through
-                @test isfile(joinpath(cfg.text_dir, "id-t-notes.log"))
+                @test isfile(p)                            # and stayed in spool/
             end
         end