Route by queue, not by directory: stop moving files between stages
A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.
Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.
Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.
The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.
Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.
Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.
This commit is contained in:
106
README.md
106
README.md
@@ -22,12 +22,13 @@ enrichment mixes CPU with a subprocess):
|
||||
▼
|
||||
┌─────────────────┐ stream bytes to disk (never buffered)
|
||||
│ HTTP handler │────────────────────────► data/spool/<uuid>-<name>
|
||||
│ (streaming) │
|
||||
└────────┬─────────┘ enqueue reference (non-blocking)
|
||||
│ │
|
||||
▼ ▼
|
||||
202 + job IDs ┌────────────────────┐
|
||||
(503 if full) │ stage-1 queue │ classification
|
||||
│ (streaming) │ the file's home for its
|
||||
└────────┬─────────┘ enqueue reference whole time in flight
|
||||
│ (non-blocking)
|
||||
▼ │
|
||||
202 + job IDs ▼
|
||||
(503 if full) ┌────────────────────┐
|
||||
│ stage-1 queue │ classification
|
||||
└─────────┬──────────┘
|
||||
│ dequeue
|
||||
┌───────────────────┼───────────────────┐
|
||||
@@ -35,9 +36,10 @@ enrichment mixes CPU with a subprocess):
|
||||
classify wkr 1 classify wkr 2 … classify wkr N
|
||||
│
|
||||
┌────────────┴────────────┐
|
||||
:unknown :known
|
||||
│ move to data/unknown/, │ move to data/known/, then
|
||||
▼ then enqueue (blocking) ▼ enqueue (blocking backpressure)
|
||||
:unknown :known the file itself never moves —
|
||||
│ │ routing is the enqueue alone
|
||||
│ enqueue (blocking) │ enqueue (blocking backpressure)
|
||||
▼ ▼
|
||||
┌────────────────────┐ ┌────────────────────┐
|
||||
│ unknown queue │ │ known queue │ enrichment
|
||||
└─────────┬──────────┘ └─────────┬──────────┘
|
||||
@@ -47,10 +49,10 @@ enrichment mixes CPU with a subprocess):
|
||||
unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M
|
||||
│ binary-vs-text sniff │ exiftool → normalized sidecar
|
||||
├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name>
|
||||
│ (terminal) data/done/<uuid>-<name>.meta.json
|
||||
│ (sidecar-first commit)
|
||||
│ :text move to data/text/, failure ───────► data/failed/<uuid>-<name>
|
||||
▼ then enqueue (blocking backpressure)
|
||||
│ (terminal — moved) data/done/<uuid>-<name>.meta.json
|
||||
│ (sidecar-first commit)
|
||||
│ :text enqueue (blocking failure ───────► data/failed/<uuid>-<name>
|
||||
▼ backpressure)
|
||||
┌────────────────────┐
|
||||
│ text queue │ language enrichment
|
||||
└─────────┬──────────┘
|
||||
@@ -63,6 +65,16 @@ enrichment mixes CPU with a subprocess):
|
||||
(sidecar-first commit)
|
||||
```
|
||||
|
||||
A file is written **once**, into `data/spool/`, and stays there for its entire
|
||||
time in the pipeline. Stages hand it on by enqueueing its small `Job` reference,
|
||||
never by moving bytes — the queue holding the reference *is* the record of which
|
||||
stage the file has reached. The only move is the last one: into a terminal sink
|
||||
(`data/done/`, `data/text_done/`, `data/binary/`) or into `data/failed/` if a
|
||||
worker throws. Stage 1 used to rename each file into `data/known/` or
|
||||
`data/unknown/` first, and stage 3 into `data/text/`; those three directories are
|
||||
gone, and with them 11.6 µs per file — the equal of the classifier itself, which
|
||||
is why stage 1 now runs at ~85k files/s on one worker instead of ~35k.
|
||||
|
||||
Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1
|
||||
feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language
|
||||
enrichment) for every file it sorts as text.
|
||||
@@ -79,12 +91,24 @@ Key properties:
|
||||
- **Backpressure:** each queue is bounded (default 1000). When the *intake* queue
|
||||
is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
|
||||
the stage-1 worker blocks and retries (a classified file is never dropped).
|
||||
- **Crash-resilient:** files survive on disk. On startup, recovery is
|
||||
stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/`
|
||||
re-enter enrichment, `data/unknown/` re-enter content triage, and `data/text/`
|
||||
re-enter language enrichment (`recovered` / `recovered_known` /
|
||||
`recovered_unknown` / `recovered_text` in the log), so a file resumes at its
|
||||
correct stage instead of restarting from scratch.
|
||||
- **Crash-resilient:** files survive on disk. On startup everything left in
|
||||
`data/spool/` is re-enqueued (`recovered` in the log) and **replays from stage
|
||||
1**. That is safe rather than merely tolerable: classification and the
|
||||
binary/text sniff are pure functions of the file's bytes, and the terminal
|
||||
commits rename with `force=true`, so a replayed file lands where it would have
|
||||
landed and overwrites its own sidecar. Recovery *blocks* on a full queue rather
|
||||
than dropping the excess, and runs with the worker pools already live, so a
|
||||
backlog larger than one queue's capacity takes longer to re-drive but none of
|
||||
it is abandoned.
|
||||
|
||||
The cost of replay is redoing stages a file had already cleared. That is a
|
||||
property of the **queue**, not of the directory layout: the queues are
|
||||
in-process (`src/queue.jl`), so a crash destroys the only record of how far
|
||||
each file got. Per-stage directories used to stand in for that record, at the
|
||||
price of a rename per file per stage on the hot path — paying a permanent cost
|
||||
on every file to buy a cheaper restart. `src/queue.jl` already defines the seam
|
||||
for the real fix: swap `ChannelQueue` for a broker-backed `JobQueue` and exact
|
||||
resume comes back durably, rather than being inferred from a pathname.
|
||||
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
|
||||
both stop accepting uploads, then drain the stages *in order* — close the
|
||||
stage-1 queue and wait out the classify workers (the only producer of the known
|
||||
@@ -182,21 +206,26 @@ idempotently.
|
||||
Files the classifier labels **unknown** are handed to a third pool that sorts
|
||||
them into two coarse buckets so downstream tooling can treat them differently:
|
||||
|
||||
- **`data/binary/`** — the file looks like binary data.
|
||||
- **`data/text/`** — the file looks like text.
|
||||
- **binary** — the file looks like binary data. This is the end of the live path,
|
||||
so the file is committed to `data/binary/` (which is also the corpus the
|
||||
offline stage-5 discovery sweep reads).
|
||||
- **text** — the file looks like text. Stage 4 is still to come, so nothing
|
||||
moves; the file stays in `data/spool/` and its reference goes onto the
|
||||
stage-4 queue, ending up in `data/text_done/` once enriched.
|
||||
|
||||
The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text
|
||||
when that window is valid UTF-8 and holds no control bytes outside the text-safe
|
||||
set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise
|
||||
binary. It's cheap (no full read) and Unicode-aware — unlike the older NUL-byte
|
||||
or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in
|
||||
`text/` instead of misfiling it, while binary formats — which rarely form valid
|
||||
UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but
|
||||
the text bucket instead of misfiling it, while binary formats — which rarely
|
||||
form valid UTF-8 near their start — still read as binary. A NUL byte is valid UTF-8 but
|
||||
not a text control byte, so it still reads as binary. A multi-byte character
|
||||
split by the 8000-byte boundary is trimmed before the check so it isn't mistaken
|
||||
for malformed bytes. An empty file is treated as text. `binary/` is terminal on
|
||||
the live path (but is the input the offline **stage-5 discovery** sweeps — see
|
||||
below); `text/` is handed to stage 4 (`src/content.jl`).
|
||||
for malformed bytes. An empty file is treated as text. Binary is terminal on the
|
||||
live path and is committed to `data/binary/` (which the offline **stage-5
|
||||
discovery** sweep reads as its corpus — see below); text is handed to stage 4
|
||||
without moving (`src/content.jl`).
|
||||
|
||||
### Language enrichment (stage 4)
|
||||
|
||||
@@ -367,10 +396,10 @@ tell you "PDF", just "this looks like something I was trained on, or not".
|
||||
fast rather than run without classification.
|
||||
- **Effect today:** *active routing*. The class is logged
|
||||
(`classification=known|unknown`) and drives the pipeline split: `:known` files
|
||||
go to `known/` for metadata enrichment (stage 2), `:unknown` files go to
|
||||
`unknown/` for content triage (stage 3). The class chooses the downstream
|
||||
stage; what's still unproven is the model's *accuracy*, not whether the routing
|
||||
path runs.
|
||||
go onto the stage-2 queue for metadata enrichment, `:unknown` files onto the
|
||||
stage-3 queue for content triage. The class chooses the downstream stage (the
|
||||
file itself stays in `data/spool/` either way); what's still unproven is the
|
||||
model's *accuracy*, not whether the routing path runs.
|
||||
|
||||
The architecture and byte→feature mapping are defined once in `src/model.jl` and
|
||||
shared by the trainer and the server, so they can't drift apart.
|
||||
@@ -410,11 +439,8 @@ init, so the artifact is exactly regenerable from the same inputs.
|
||||
| `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) |
|
||||
| `FS_TEXT_WORKERS` | `nthreads()` | Stage-4 (language enrichment) worker tasks |
|
||||
| `FS_TEXT_QUEUE_CAPACITY` | `1000` | Max pending language jobs (then backpressure) |
|
||||
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending classification) |
|
||||
| `FS_KNOWN_DIR` | `data/known` | Classified-known, awaiting enrichment |
|
||||
| `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, awaiting content triage |
|
||||
| `FS_SPOOL_DIR` | `data/spool` | Every in-flight file, at every stage |
|
||||
| `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary |
|
||||
| `FS_TEXT_DIR` | `data/text` | Classified-text, awaiting language enrichment |
|
||||
| `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) |
|
||||
| `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) |
|
||||
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
|
||||
@@ -462,9 +488,9 @@ Each file in a request becomes its own job. Responses:
|
||||
### `GET /stats` — per-stage counters
|
||||
|
||||
The pipeline counts its own work (`src/stats.jl`), because nothing outside it
|
||||
can: `known/`, `unknown/` and `text/` are *transient*, so a file can cross one
|
||||
between two directory polls and an external sampler will miss exactly the stages
|
||||
you most want to measure.
|
||||
can. Every in-flight file sits in `data/spool/` no matter which stage it has
|
||||
reached — the stage is a property of the queue holding its reference, and only
|
||||
the pipeline can see that. There is no directory to poll.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
@@ -560,9 +586,9 @@ Three properties of this design dictate how it measures:
|
||||
so `ab`/`hey`/`wrk` would only ever measure intake. The bench instead uploads a
|
||||
corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
|
||||
`failed/`) until the count stops moving, and reports both numbers separately:
|
||||
intake rate *and* end-to-end completion rate. It also samples the intermediate
|
||||
stage dirs, so the peak depth of `spool/`/`known/`/`unknown/`/`text/` shows
|
||||
where work piles up.
|
||||
intake rate *and* end-to-end completion rate. It also samples `spool/`, whose
|
||||
peak depth is the high-water mark of files in flight — but *which* stage they
|
||||
are waiting on comes from `/stats`, not from disk.
|
||||
- **End-to-end throughput doesn't name the slow stage.** The four stages run
|
||||
concurrently behind their own queues, so the pipeline's rate *is* the slowest
|
||||
stage's rate and the others are invisible in it. The bench scrapes
|
||||
|
||||
Reference in New Issue
Block a user