jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Route by queue, not by directory: stop moving files between stages

Browse Source 
A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.

Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.

Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.

The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.

Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.

Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.
This commit is contained in:
Jeffrey Ward
2026-08-07 13:21:49 -04:00
parent 341b61f806
commit 4a22123001
13 changed files with 352 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
src/worker.jl
View File

@@ -3,28 +3,48 @@
# stage, so `worker_loop` is parametrized with a `handler` and reused. Today
# there are four stages:
#
# stage 1 handle_classify_job spool/ → classify → known/ (+known queue) | unknown/ (+unknown queue)
# stage 2 handle_known_job known/ → exiftool enrich → done/ (+ .meta.json)
# stage 3 handle_unknown_job unknown/ → binary-vs-text sniff → binary/ | text/ (+text queue)
# stage 4 handle_text_job text/ → language enrich → text_done/ (+ .meta.json)
# stage 1 handle_classify_job classify → known queue | unknown queue
# stage 2 handle_known_job exiftool enrich → done/ (+ .meta.json)
# stage 3 handle_unknown_job binary-vs-text sniff → binary/ | text queue
# stage 4 handle_text_job language enrich → text_done/ (+ .meta.json)
#
# Adding a stage later is just another queue + pool + handler; the loop below
# doesn't change.
# How long a stage-1 worker backs off before retrying an enqueue onto a full
# downstream queue (known or unknown). Blocking backpressure: a classified file
# is never dropped, so the stage-1 worker parks until the next stage makes room.
# Keeps intake decoupled — the HTTP path's `enqueue!` stays non-blocking; only
# this worker-to-worker handoff blocks.
const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05
#
# A file does not move between stages. It is written once into `spool/` at
# intake and stays there for its whole in-flight life; only the small `Job`
# reference travels, and `job.path` is therefore constant from intake until the
# file is committed. The stage a file has reached lives in the queue holding its
# reference, not in which directory the bytes sit — so the intermediate hops
# (spool→known, spool→unknown, unknown→text) are three renames per file that buy
# nothing on the live path. What remains is one move at the end: into a terminal
# sink (`done/`, `text_done/`, `binary/`) or into `failed/` on a throw.
#
# The cost is on restart. The queues are in-process (src/queue.jl), so a crash
# loses them, and recovery can only re-drive everything in `spool/` from stage 1.
# That is safe — classification and the UTF-8 sniff are pure functions of the
# file's bytes and every commit is idempotent — but it redoes work the old
# directory-per-stage layout could skip. The durable fix is the queue seam, not
# the directories: a broker-backed `JobQueue` restores exact resume for free.
#
# (`ROUTE_ENQUEUE_RETRY_SECONDS`, the backoff the routing handoffs below use,
# now lives in src/queue.jl, since startup recovery shares it.)
# Per-file logging in stage 1 is `@debug`, not `@info`, because it is the
# stage's dominant cost. Measured by bin/bench_stage1.jl (2000 x 64 KiB files,
# min of 5 trials): the two log lines cost ~71 µs of the ~118 µs
# `handle_classify_job` spent per file — roughly 6x the classifier (10.6 µs) and
# 6x the rename (11.6 µs). Nearly all of it is `ConsoleLogger` formatting
# (~64 µs); the FlushLogger's per-message flush is only ~8 µs on top. Demoting
# them takes stage 1 from ~8.5k files/s to ~35k files/s on one worker.
# min of 5 trials): a formatted pair of log lines costs 71.2 µs per file, against
# the 11.7 µs the whole handler takes with them switched off — six times the rest
# of the stage put together. Nearly all of it is `ConsoleLogger` formatting
# (64.2 µs); the FlushLogger's per-message flush is only ~7 µs on top. Switching
# them back on with `JULIA_DEBUG=FileServer` takes the handler to 106.3 µs, i.e.
# from 85.4k files/s down to 9.4k on one worker.
#
# With logging off the stage is the classifier and nothing else: classify 10.30 µs
# (of which read_features is 7.82 µs), the queue handoff 0.12 µs, the disabled
# `@debug` lines 0.29 µs — 10.7 µs of the 11.7 µs total. Removing the inter-stage
# rename is what left it that way: that rename was 11.6 µs per file, so it was
# the equal of the classifier, and dropping it took one worker from ~35k to
# ~85k files/s (16 workers: ~333k files/s).
#
# `@debug` is compiled to a min-level check that doesn't evaluate its arguments,
# so a disabled line costs ~0.15 µs rather than ~36 µs. The messages are still
@@ -38,11 +58,13 @@ const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05
Stage 1. Classify the spooled file and route it to the next stage's queue,
retrying on a full queue rather than dropping the file:
* `:known` → move to `known/`, enqueue onto the known queue for stage 2.
* `:unknown` → move to `unknown/`, enqueue onto the unknown queue for stage 3.
* `:known` → enqueue onto the known queue for stage 2.
* `:unknown` → enqueue onto the unknown queue for stage 3.
In both cases move first so the file physically lives in its stage dir before the
reference is visible downstream; the moved path becomes the routed job's location.
Routing is the enqueue and nothing else: the file stays where intake wrote it and
the *same* `Job` is handed on, so `job.path` still points at it. Reaching stage 2
is a fact about which queue holds the reference, not about which directory holds
the bytes.
Sub-`MIN_FILE_BYTES` files short-circuit to `:unknown` inside `classify`.
"""
@@ -53,18 +75,14 @@ function handle_classify_job(job::Job, cfg::Config, worker_id::Int,
@debug "classified file" worker=worker_id id=job.id name=job.original_name size=job.size classification=classification
if classification === :known
dest = move_to(cfg.known_dir, job)
routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
# known queue full → park and retry, don't drop (time charged to blocked_ns)
enqueue_blocking!(known_queue, routed, stats;
enqueue_blocking!(known_queue, job, stats;
retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
@debug "routed to enrichment" worker=worker_id id=job.id dest=dest
@debug "routed to enrichment" worker=worker_id id=job.id path=job.path
else
dest = move_to(cfg.unknown_dir, job)
routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
enqueue_blocking!(unknown_queue, routed, stats;
enqueue_blocking!(unknown_queue, job, stats;
retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
@debug "routed to content triage" worker=worker_id id=job.id dest=dest
@debug "routed to content triage" worker=worker_id id=job.id path=job.path
end
return nothing
end
@@ -73,7 +91,8 @@ end
handle_known_job(job, cfg, worker_id)
Stage 2. Extract metadata (exiftool, with timeout) and enrich: build the
normalized sidecar and commit both to `done/` sidecar-first. Extraction
normalized sidecar and commit both to `done/` sidecar-first — the file's one and
only move, straight out of `spool/`. Extraction
failure/timeout yields a *degraded* sidecar (the file is still a wanted known
file), so the only way to land in `failed/` is a genuine I/O error writing the
sidecar or moving the file — handled by `worker_loop`'s quarantine.
@@ -89,10 +108,14 @@ end
handle_unknown_job(job, cfg, worker_id, text_queue)
Stage 3. Sort an unrecognized file into a coarse content bucket by sniffing its
first bytes: `binary/` (terminal — no further stage) if it looks like binary
data, `text/` otherwise. A text file is then routed onward to the stage-4
language-enrichment queue, retrying on a full queue rather than dropping the file
(the same blocking backpressure stage 1 uses for its downstream queues).
first bytes.
The two outcomes are asymmetric, because one is terminal and one is not. Binary
is the end of the live path, so the file is committed to `binary/` — which is
also where the offline stage-5 discovery sweep reads its corpus, so the move is
load-bearing, not bookkeeping. Text has a stage 4 still to come, so nothing moves:
the same `Job` goes onto the language-enrichment queue, retrying on a full queue
rather than dropping the file (the blocking backpressure stage 1 also uses).
"""
function handle_unknown_job(job::Job, cfg::Config, worker_id::Int,
text_queue::JobQueue, stats::StageStats)
@@ -100,12 +123,10 @@ function handle_unknown_job(job::Job, cfg::Config, worker_id::Int,
dest = move_to(cfg.binary_dir, job)
@info "sorted unknown" worker=worker_id id=job.id name=job.original_name kind=:binary dest=dest
else
dest = move_to(cfg.text_dir, job)
routed = Job(job.id, job.original_name, dest, job.size, job.received_at)
# text queue full → park and retry, don't drop (time charged to blocked_ns)
enqueue_blocking!(text_queue, routed, stats;
enqueue_blocking!(text_queue, job, stats;
retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
@info "routed to language enrichment" worker=worker_id id=job.id dest=dest
@info "routed to language enrichment" worker=worker_id id=job.id path=job.path
end
return nothing
end
@@ -115,7 +136,8 @@ end
Stage 4. Enrich a text file with its natural language (via `detector`) and
programming language (via github-linguist): build the sidecar and commit both to
`text_done/` sidecar-first. Detection failure yields a *degraded* sidecar (the
`text_done/` sidecar-first — the file's one and only move, straight out of
`spool/`. Detection failure yields a *degraded* sidecar (the
file is still wanted text), so the only way to land in `failed/` is a genuine I/O
error committing — handled by `worker_loop`'s quarantine.
"""