jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial text cleanup

Browse Source
This commit is contained in:
Jeffrey Ward
2026-08-19 12:55:19 -04:00
parent 819a4cce7a
commit cc84d70d52
26 changed files with 454 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

270
README.md
View File

@@ -3,11 +3,11 @@
A minimal Julia service that receives files over HTTP and hands them off to a
pool of worker threads for processing. The HTTP endpoint does no real work: it
spools each uploaded file to disk, pushes a lightweight reference onto a work
queue, and responds immediately — staying free to accept the next upload.
queue, and responds immediately, staying free to accept the next upload.
The per-file "processing" runs each file through a small neural-network
classifier that labels it **known** (a file type resembling the training set) or
**unknown**, and logs the result. See "File classifier" below.
The per-file processing is deliberately thin. Each file runs through a small
neural-network classifier that labels it **known** (a file type resembling the
training set) or **unknown**, and logs the result. See "File classifier" below.
## Architecture
@@ -36,7 +36,7 @@ enrichment mixes CPU with a subprocess):
classify wkr 1 classify wkr 2 … classify wkr N
│
┌────────────┴────────────┐
:unknown :known the file itself never moves —
:unknown :known the file itself never moves;
│ │ routing is the enqueue alone
│ enqueue (blocking) │ enqueue (blocking backpressure)
▼ ▼
@@ -65,15 +65,15 @@ enrichment mixes CPU with a subprocess):
(sidecar-first commit)
```
A file is written **once**, into `data/spool/`, and stays there for its entire
time in the pipeline. Stages hand it on by enqueueing its small `Job` reference,
never by moving bytes — the queue holding the reference *is* the record of which
stage the file has reached. The only move is the last one: into a terminal sink
A file is written once, into `data/spool/`, and stays there for its entire time
in the pipeline. Stages hand it on by enqueueing its small `Job` reference, never
by moving bytes: the queue holding the reference *is* the record of which stage
the file has reached. The only move is the last one, into a terminal sink
(`data/done/`, `data/text_done/`, `data/binary/`) or into `data/failed/` if a
worker throws. Stage 1 used to rename each file into `data/known/` or
`data/unknown/` first, and stage 3 into `data/text/`; those three directories are
gone, and with them 11.6 µs per file — the equal of the classifier itself, which
is why stage 1 now runs at ~85k files/s on one worker instead of ~35k.
`data/unknown/` first, and stage 3 into `data/text/`. Those three directories are
gone, and with them 11.6 µs per file, the equal of the classifier itself. Stage 1
now runs at ~85k files/s on one worker instead of ~35k.
Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1
feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language
@@ -83,10 +83,10 @@ Key properties:
- **Fast intake:** the queue only ever carries small references; file bytes live
on disk, so memory stays flat regardless of file size. This holds end to end:
intake **streams** each upload from the socket to the spool file a chunk at a
intake streams each upload from the socket to the spool file a chunk at a
time (`FS_UPLOAD_CHUNK_BYTES`, default 64 KiB) rather than buffering the body,
and every worker reads only a bounded prefix. Measured: uploads of 256 MiB,
1 GiB and 2 GiB each grow resident memory by ~20 MiB — a flat line in file
1 GiB and 2 GiB each grow resident memory by ~20 MiB, a flat line in file
size. See "Streaming intake" and "Benchmarking" below.
- **Backpressure:** each queue is bounded (default 1000). When the *intake* queue
is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
@@ -102,15 +102,15 @@ Key properties:
it is abandoned.
The cost of replay is redoing stages a file had already cleared. That is a
property of the **queue**, not of the directory layout: the queues are
property of the *queue*, not of the directory layout: the queues are
in-process (`src/queue.jl`), so a crash destroys the only record of how far
each file got. Per-stage directories used to stand in for that record, at the
price of a rename per file per stage on the hot path — paying a permanent cost
on every file to buy a cheaper restart. `src/queue.jl` already defines the seam
price of a rename per file per stage on the hot path: a permanent cost on
every file, to buy a cheaper restart. `src/queue.jl` already defines the seam
for the real fix: swap `ChannelQueue` for a broker-backed `JobQueue` and exact
resume comes back durably, rather than being inferred from a pathname.
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
both stop accepting uploads, then drain the stages *in order* — close the
both stop accepting uploads, then drain the stages *in order*: close the
stage-1 queue and wait out the classify workers (the only producer of the known
*and* unknown queues), then close those queues and wait out the enrich and
content-triage workers (content triage being the only producer of the text
@@ -123,22 +123,22 @@ Key properties:
The upload endpoint never holds a file in memory. Bytes go socket → spool file in
`FS_UPLOAD_CHUNK_BYTES` chunks, so resident memory per in-flight upload is set by
the chunk size, not the file size — a 2 GiB upload costs about what a 2 KiB one
the chunk size, not the file size. A 2 GiB upload costs about what a 2 KiB one
does. Two pieces make that work, and both are deliberate:
- **`src/multipart.jl` — an incremental multipart parser.** HTTP.jl's
- **`src/multipart.jl`, an incremental multipart parser.** HTTP.jl's
`parse_multipart_form` takes the *complete* body as a byte vector, so using it
means every file in the request is in memory at once (and copied again per
part). The reader here pulls fixed-size chunks and hands each part's bytes
straight to its spool file. Its interface is two calls in a loop —
`next_part!` then `write_part_body!` (or `skip_part_body!`) — so the handler
straight to its spool file. Its interface is two calls in a loop
(`next_part!` then `write_part_body!`, or `skip_part_body!`), so the handler
keeps ordinary control flow instead of inverting into callbacks. The subtle
part is that a boundary delimiter can straddle two chunks, so the buffer always
retains the last `length(delimiter)-1` bytes; the test suite parses the same
body at chunk sizes from 1 byte upward to put that split at every offset.
- **`/upload` bypasses Oxygen's router.** Oxygen's root handler wraps
`HTTP.streamhandler`, which does `request.body = read(stream)` *before*
dispatching — even for an Oxygen `@stream` route, so no route can stream an
dispatching, even for an Oxygen `@stream` route, so no route can stream an
upload. `run` therefore passes its own `handler` to `serve`
(`root_stream_handler`), which intercepts `POST /upload` at the stream level and
delegates everything else to Oxygen unchanged. The trade-off: `/upload` is
@@ -157,28 +157,28 @@ complete) and the event is logged `upload aborted by client`. One cosmetic cavea
like the SIGTERM one below: when a request body is cut short, HTTP.jl's own
`closeread` logs an `EOFError` after the handler returns, because the connection
promised more bytes via `Content-Length` than arrived. It's harmless noise from
inside HTTP.jl — the partial file is already cleaned up and the connection closed.
inside HTTP.jl: the partial file is already cleaned up and the connection closed.
### Metadata enrichment (stage 2)
Files the classifier labels **known** are handed to a second pool that extracts
metadata with [`exiftool`](https://exiftool.org/) (`exiftool -json -G -n`) —
metadata with [`exiftool`](https://exiftool.org/) (`exiftool -json -G -n`),
chosen because no native Julia library comes close to its multi-format coverage.
The output is normalized into a small, stable, documented schema and written as a
JSON **sidecar** next to the file in `data/done/`, e.g.
`data/done/<uuid>-<name>.meta.json`. The original bytes are never modified.
> **Prerequisite:** `exiftool` must be on `PATH` (e.g. `apt install
> libimage-exiftool-perl`). The server **fails fast at startup** if it's missing.
> libimage-exiftool-perl`). The server fails fast at startup if it's missing.
Sidecar top-level fields (all nullable — present only when available), plus the
Sidecar top-level fields (all nullable, present only when available), plus the
complete raw `exiftool` object under `raw`:
| Field | Meaning |
|---|---|
| `id`, `original_name` | job id and client-supplied name |
| `file_type`, `mime_type` | e.g. `PDF` / `application/pdf` |
| `file_size` | bytes (authoritative, from intake — not exiftool) |
| `file_size` | bytes (authoritative, from intake, not exiftool) |
| `created_date`, `modified_date` | content timestamps |
| `author` | person (`Author`/`Artist`/`By-line`) |
| `created_by` | authoring app/tool (`Producer`/`CreatorTool`/`Creator`/`Software`/…) |
@@ -191,12 +191,12 @@ complete raw `exiftool` object under `raw`:
Each normalized field is a coalesce over a priority list of exiftool tags
(`src/metadata.jl`); extend a field by appending tag names. If extraction fails
or `exiftool` times out (`FS_EXIFTOOL_TIMEOUT`, default 30s), the file still
completes to `data/done/` with a **degraded sidecar** — `file_size`/`file_type`
plus an `error` note — rather than being quarantined, because it's still a wanted
completes to `data/done/` with a **degraded sidecar** (`file_size`/`file_type`
plus an `error` note) rather than being quarantined, because it's still a wanted
known file. Only genuine I/O errors (can't write the sidecar or move the file)
send it to `data/failed/`.
The sidecar is committed **before** the file is moved into `data/done/`, so a
The sidecar is committed *before* the file is moved into `data/done/`, so a
file's presence there always implies its sidecar is already present; a crash in
between leaves only a harmless orphan sidecar, and recovery re-enriches
idempotently.
@@ -206,26 +206,23 @@ idempotently.
Files the classifier labels **unknown** are handed to a third pool that sorts
them into two coarse buckets so downstream tooling can treat them differently:
- **binary** — the file looks like binary data. This is the end of the live path,
so the file is committed to `data/binary/` (which is also the corpus the
offline stage-5 discovery sweep reads).
- **text** — the file looks like text. Stage 4 is still to come, so nothing
- **binary:** the file looks like binary data. This is the end of the live path,
so the file is committed to `data/binary/`, which doubles as the corpus for the
offline stage-5 discovery sweep below.
- **text:** the file looks like text. Stage 4 is still to come, so nothing
moves; the file stays in `data/spool/` and its reference goes onto the
stage-4 queue, ending up in `data/text_done/` once enriched.
The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text
when that window is valid UTF-8 and holds no control bytes outside the text-safe
set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise
binary. It's cheap (no full read) and Unicode-aware — unlike the older NUL-byte
or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in
the text bucket instead of misfiling it, while binary formats — which rarely
form valid UTF-8 near their start — still read as binary. A NUL byte is valid UTF-8 but
not a text control byte, so it still reads as binary. A multi-byte character
split by the 8000-byte boundary is trimmed before the check so it isn't mistaken
for malformed bytes. An empty file is treated as text. Binary is terminal on the
live path and is committed to `data/binary/` (which the offline **stage-5
discovery** sweep reads as its corpus — see below); text is handed to stage 4
without moving (`src/content.jl`).
binary. It's cheap (no full read) and Unicode-aware. Where the older NUL-byte and
printable-ASCII heuristics misfiled non-ASCII text, this keeps accents, CJK and
emoji in the text bucket, while binary formats, which rarely form valid UTF-8
near their start, still read as binary. A NUL byte is valid UTF-8 but not a text
control byte, so it too reads as binary. A multi-byte character split by the
8000-byte boundary is trimmed before the check so it isn't mistaken for malformed
bytes. An empty file is treated as text (`src/content.jl`).
### Language enrichment (stage 4)
@@ -233,12 +230,12 @@ Files that stage 3 sorts as **text** are handed to a fourth pool that identifies
their language and writes a `.meta.json` sidecar, mirroring the stage-2
known-file enrichment. Two detectors run per file:
- **natural language** — [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s
- **natural language:** [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s
`LanguageDetector` (a Julia port of the `whatlang` n-gram model) reads a bounded
prefix (up to `LANG_SAMPLE_BYTES`, 64 KiB) and reports the language's English
name, ISO 639-3 code, and a confidence in `[0,1]`. Pure Julia, no subprocess.
The detector is built once at startup and shared read-only across the pool.
- **programming language** — the [`github-linguist`](https://github.com/github-linguist/linguist)
- **programming language:** the [`github-linguist`](https://github.com/github-linguist/linguist)
CLI recognizes source and markup by extension + content heuristics (e.g.
`Python`, `Markdown`). Plain prose reports as `Text` and unrecognized content as
`null`; both collapse to *no programming language*.
@@ -257,42 +254,42 @@ The sidecar schema:
| `error` | set if natural-language detection produced nothing |
> **`github-linguist` and the git-repo quirk:** run against a path *inside* a git
> repository, linguist reads the file's committed git blob, not the on-disk bytes
> — and an untracked file (which everything under `data/` is) has no blob, so it
> crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under
> repository, linguist reads the file's committed git blob, not the on-disk
> bytes, and an untracked file (which everything under `data/` is) has no blob,
> so it crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under
> `/tmp` (outside any repo, preserving the name so extension heuristics still
> fire) and pointing linguist there.
>
> Programming-language detection is **best-effort**: if `github-linguist` is
> Programming-language detection is best-effort: if `github-linguist` is
> missing (a startup warning, not a fatal error, unlike `exiftool`), fails, or
> times out (`FS_LINGUIST_TIMEOUT`, default 30s), `programming_language` is simply
> `null` and the file still completes. Natural-language detection failing produces
> a **degraded sidecar** (with an `error` note) rather than a quarantine, because
> the file is still wanted text.
Like stage 2, the sidecar is committed **before** the file is moved into
Like stage 2, the sidecar is committed *before* the file is moved into
`data/text_done/`, so the file's presence there always implies its sidecar is
present; recovery re-enriches idempotently (`src/language.jl`).
### Unknown-format discovery (stage 5, offline)
The `binary/` sink from stage 3 is the pile of genuinely *unrecognized* files.
Stage 5 mines it for **recurring new file formats** by clustering files on their
header bytes — a growing catalog of discovered formats, each with a magic-byte
Stage 5 mines it for recurring new file formats by clustering files on their
header bytes into a growing catalog of discovered formats, each with a magic-byte
signature that can eventually be promoted into the classifier's fast path. Unlike
stages 1–4 it is **not on the request hot path**: it is a single-owner *batch*
stages 1–4 it is not on the request hot path: it is a single-owner *batch*
process (the catalog is mutable shared state, the opposite of the stateless
classifier), and because promotion is human-gated nothing here is
latency-sensitive. The full rationale — and the assumptions we deliberately
rejected — live in [`model/DESIGN_clustering.md`](model/DESIGN_clustering.md).
latency-sensitive. The full rationale, and the assumptions we deliberately
rejected, live in [`model/DESIGN_clustering.md`](model/DESIGN_clustering.md).
The model (`src/cluster.jl`, base-Julia, no extra deps) is a Dirichlet-process
mixture of **per-position categoricals** over the first 32 header bytes, on a
257-symbol alphabet (byte `0–255` plus a `past-EOF` symbol so short fixed-length
formats are modeled honestly). Bytes are treated as **categorical, not numeric**
— `0x89` and `0x88` are not "close" — so this deliberately does *not* reuse the
classifier's `[0,1]` byte scaling. A fixed uniform **background** component
absorbs structureless (compressed/encrypted) blobs so they don't mint spurious
formats are modeled honestly). Bytes are treated as categorical rather than
numeric (`0x89` and `0x88` are not "close"), so this deliberately does *not*
reuse the classifier's `[0,1]` byte scaling. A fixed uniform *background*
component absorbs structureless (compressed/encrypted) blobs so they don't mint spurious
clusters. A cluster's spiked positions become a libmagic-style signature;
clusters with enough members and enough fixed positions self-**nominate** for
promotion (a human does the one irreversible step, redefining "known").
@@ -302,8 +299,8 @@ is the science; phase B (`src/catalog.jl`) is the live catalog: a durable
single-owner state that sweeps `binary/`, folds each new file into a cluster with
the deterministic CRP-predictive rule, and writes promotion nominations.
The catalog process is run periodically (cron), single-threaded — it is the
**only** writer of the catalog, so it needs no locking:
The catalog process is run periodically (cron), single-threaded. It is the only
writer of the catalog, so it needs no locking:
```bash
julia --project=. bin/cluster_sweep.jl # incremental live sweep of new binary/ files
@@ -313,17 +310,17 @@ julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs re-cluster (se
The catalog is a single durable file (`FS_CLUSTER_CATALOG`, default
`data/catalog.json`) committed with the same sidecar-first
temp→fsync→rename→fsync-dir discipline as the stage-2 sidecars, so a crash can
neither corrupt it nor lose a write. On the **first** run (empty catalog) the
sweep auto-promotes to a `--compact` pass to seed clusters; later runs assign
neither corrupt it nor lose a write. On the first run (empty catalog) the sweep
auto-promotes to a `--compact` pass to seed clusters; later runs assign
incrementally, touching only files they have not seen. A cluster that clears the
member/magic thresholds writes a nomination — a hex magic template, member count,
and example filenames — into `FS_NOMINATED_DIR` (default `data/nominated/`) for a
member/magic thresholds writes a nomination (a hex magic template, member count,
and example filenames) into `FS_NOMINATED_DIR` (default `data/nominated/`) for a
human to glance at and promote. Under the calibrated `bg_mass > α`, the live
sweep never mints single-file clusters; genuinely new formats surface from the
periodic `--compact` re-clustering of the background residue, not the live path.
Calibration is its own offline script (like training — never in the request
path), scored against magic-collapsed ground truth (so `docx`≡`zip` and the whole
Calibration is its own offline script, like training, and never in the request
path. It is scored against magic-collapsed ground truth (so `docx`≡`zip` and the whole
ELF family count as one format each, which is the *correct* answer, not an error):
```bash
@@ -336,15 +333,15 @@ training corpus the calibrated defaults (`n=32`, `α=1.0`, `β=0.1`) recover the
known formats at **ARI 0.77** (0.885 excluding tar), with `gzip`, `pkzip`
(`docx`+`zip` merged), and `jpeg` forming clean, promotable clusters; the NCD
baseline agrees. See `DESIGN_clustering.md` §11 for the full results, including the
one known limitation (ELF and these tarballs share a long run of header zero-
padding and merge — the v2 fix is inverse-entropy position weighting).
one known limitation: ELF and these tarballs share a long run of header
zero-padding and merge. The v2 fix is inverse-entropy position weighting.
## The queue seam (→ RabbitMQ later)
The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and
`close!` on a `JobQueue` (see `src/queue.jl`). Today that's an in-process
`ChannelQueue`. To move to RabbitMQ (or any broker), implement a new `JobQueue`
subtype with those three methods and swap the construction in `run` — no handler
subtype with those three methods and swap the construction in `run`. No handler
or worker code changes.
## Running
@@ -354,7 +351,7 @@ or worker code changes.
julia --project=. -e 'using Pkg; Pkg.instantiate()'
# external tools: exiftool (stage 2, required) and github-linguist (stage 4,
# optional — programming-language detection). e.g. on Debian/Ubuntu:
# optional, for programming-language detection). e.g. on Debian/Ubuntu:
# apt install libimage-exiftool-perl
# gem install github-linguist
@@ -369,12 +366,12 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
- **SIGINT** is caught as an `InterruptException` (we call
`Base.exit_on_sigint(false)`), so shutdown is clean and quiet.
- **SIGTERM** can't be intercepted directly — Julia blocks it on worker threads
- **SIGTERM** can't be intercepted directly: Julia blocks it on worker threads
and handles it in its own runtime, so a user `signal()` handler never fires.
Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path
does run. Caveat: Julia prints its own `signal 15: Terminated` backtrace
*before* `atexit` runs. It's harmless noise — the drain still completes right
after it — but if you want a fully quiet stop under a process manager,
*before* `atexit` runs. It's harmless noise, and the drain still completes
right after it, but for a fully quiet stop under a process manager,
configure it to send SIGINT instead (systemd: `KillSignal=SIGINT`; Docker:
`STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain
in-flight work (systemd: `TimeoutStopSec`).
@@ -383,7 +380,7 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
Each file is scored by a fixed-structure neural network (Lux.jl) that answers a
single binary question: is this file **known** (like the types in the training
set) or **unknown**? It's novelty detection, not exact file-typing — it won't
set) or **unknown**? It's novelty detection, not exact file-typing: it won't
tell you "PDF", just "this looks like something I was trained on, or not".
- **Features:** the first 16 bytes + last 16 bytes of the file, each scaled
@@ -406,14 +403,14 @@ shared by the trainer and the server, so they can't drift apart.
### Training
Training is a separate, offline script — it never runs in the request path:
Training is a separate, offline script. It never runs in the request path:
```bash
julia --project=. bin/train.jl <positives_dir> [negatives_dir]
```
- **positives_dir** — every file in it (≥32 bytes) is a "known" example.
- **negatives_dir** *(optional)* — a grab-bag of *other* real file types used as
- **positives_dir:** every file in it (≥32 bytes) is a "known" example.
- **negatives_dir** *(optional)*: a grab-bag of *other* real file types used as
"unknown" examples. Negatives are generated ~1:1 with positives, split 50/50
between uniform-random byte vectors and grab-bag files. With no grab-bag dir,
negatives are all random (weaker: the net may just learn "high entropy =
@@ -489,8 +486,8 @@ Each file in a request becomes its own job. Responses:
The pipeline counts its own work (`src/stats.jl`), because nothing outside it
can. Every in-flight file sits in `data/spool/` no matter which stage it has
reached — the stage is a property of the queue holding its reference, and only
the pipeline can see that. There is no directory to poll.
reached. The stage is a property of the queue holding its reference, and only
the pipeline can see that; there is no directory to poll.
```jsonc
{
@@ -507,7 +504,7 @@ the pipeline can see that. There is no directory to poll.
}
```
Counters are monotonic since startup, Prometheus-style — rates are the reader's
Counters are monotonic since startup, Prometheus-style. Rates are the reader's
job, so a scrape is stateless and two readers can't disturb each other. Take two
scrapes Δt apart and subtract:
@@ -523,14 +520,14 @@ each pool worked to keep up. The bottleneck sits near 1.0 with its queue backing
up while its neighbours idle.
`blocked_seconds` is what keeps that true. Stages 1 and 3 apply *blocking*
backpressure — a full downstream queue means the handler parks and retries rather
than dropping the file — and that wait happens inside the handler. Counting it as
backpressure (a full downstream queue means the handler parks and retries rather
than dropping the file), and that wait happens inside the handler. Counting it as
busy would pin stage 1 at 1.0 whenever stage 2 is the real jam, making every
stage upstream of a jam look like the jam. Subtracted out, utilization means
"doing its own work", and a high blocked share becomes its own signal: a stage
blocked 90% of the time is naming its successor.
The counters are a handful of atomic adds per file, recorded in `worker_loop` —
The counters are a handful of atomic adds per file, recorded in `worker_loop`:
the one place every stage's work passes through, so a new stage is instrumented
the moment it is wired up, and never on the read path.
@@ -551,13 +548,13 @@ Running all of them from a clean checkout:
```bash
julia --project=. -e 'using Pkg; Pkg.instantiate()' # once
# 1. the model, on its own — no server involved
# 1. the model, on its own; no server involved
julia --project=. -t auto bin/bench_model.jl
# 1b. stage 1 taken apart — also no server
# 1b. stage 1 taken apart; also no server
julia --project=. -t auto bin/bench_stage1.jl
# 1c. stage 2 taken apart — needs a directory of real files, not generated ones
# 1c. stage 2 taken apart; needs a directory of real files, not generated ones
julia --project=. -t auto bin/bench_stage2.jl
# 2. the pipeline. Start the server in one terminal…
@@ -576,18 +573,18 @@ julia --project=. bin/cluster_calibrate.jl ../training_set
The test suite is `julia --project=. -t auto -e 'using Pkg; Pkg.test()'`.
`bin/bench.jl` must run on the same machine as the server (it reads the sink dirs
and `/proc`); otherwise it only speaks HTTP — `/upload`, `/health`, and `/stats`
and `/proc`); otherwise it only speaks HTTP: `/upload`, `/health`, and `/stats`
for the per-stage numbers.
Three properties of this design dictate how it measures:
- **HTTP latency is not throughput.** `/upload` returns `202` once the bytes are
spooled and a reference is enqueued — all four stages run *after* the response,
so `ab`/`hey`/`wrk` would only ever measure intake. The bench instead uploads a
corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
spooled and a reference is enqueued, and all four stages run *after* the
response, so `ab`/`hey`/`wrk` would only ever measure intake. The bench
instead uploads a corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
`failed/`) until the count stops moving, and reports both numbers separately:
intake rate *and* end-to-end completion rate. It also samples `spool/`, whose
peak depth is the high-water mark of files in flight — but *which* stage they
peak depth is the high-water mark of files in flight. But *which* stage they
are waiting on comes from `/stats`, not from disk.
- **End-to-end throughput doesn't name the slow stage.** The four stages run
concurrently behind their own queues, so the pipeline's rate *is* the slowest
@@ -606,8 +603,8 @@ Three properties of this design dictate how it measures:
bottleneck stage 4 (language) at 87.0% utilization of 16 worker(s)
```
(400 mixed files, 16 KiB each, concurrency 16. Stage 4 is the constraint —
`github-linguist` is a process spawn per file at ~850 ms — and the 200 text
(400 mixed files, 16 KiB each, concurrency 16. Stage 4 is the constraint,
`github-linguist` being a process spawn per file at ~850 ms, and the 200 text
files queue up behind it while stages 1 and 3 idle at under 10%.) Read *down*
the util column, not the files/s column: each stage sees a different subset of
the corpus, so a low rate can just mean little work was routed there.
@@ -625,9 +622,9 @@ Three properties of this design dictate how it measures:
| 256 MiB × 4 | 4 | none measurable (peak stayed under the baseline) |
**Read the shape, not the digits.** Growth is flat across an 8× range of file
sizes — 14–31 MiB whether the upload is 256 MiB or 2 GiB — which is the claim
that matters: nothing scales with file size, so nothing is buffering. The
absolute figures are *not* precise to the megabyte. A freshly started server
sizes: 14–31 MiB whether the upload is 256 MiB or 2 GiB. That is the claim
that matters, because nothing scales with file size, so nothing is buffering.
The absolute figures are *not* precise to the megabyte. A freshly started server
settles anywhere in an ~860–985 MiB band, so run-to-run variance in the
baseline is comparable to the growth being measured; the residual is GC churn
from the chunk reads, not retained buffers.
@@ -639,20 +636,20 @@ Three properties of this design dictate how it measures:
(or none at all, as the concurrency-4 row above did).
- **Linear-in-concurrency is not currently demonstrated.** An earlier
measurement put 4 concurrent 256 MiB uploads at 86.9 MiB (21.7 per in-flight
upload), but that run predates a fix to how the baseline was sampled — it was
read *before* the peak counter was reset, against a different origin — and the
re-measurement above cannot reproduce it: the growth sits under the noise
upload), but that run predates a fix to how the baseline was sampled: it was
read *before* the peak counter was reset, against a different origin. The
re-measurement above cannot reproduce it, and the growth sits under the noise
floor. Expect concurrency to cost memory; don't trust a specific coefficient
without a quieter machine or many more runs.
Before intake
was streamed, the same 256 MiB upload grew RSS by ~700 MiB and 4 concurrent ones
pushed a 950 MiB baseline past 2 GiB, so **a `--size` sweep that slopes upward is
the regression signal** — it means something has started buffering bodies again.
Before intake was streamed, the same 256 MiB upload grew RSS by ~700 MiB and 4
concurrent ones pushed a 950 MiB baseline past 2 GiB. So a `--size` sweep that
slopes upward is the regression signal: something has started buffering bodies
again.
Flags: `--files`, `--size` (`8k`/`64m`/`1g`), `--concurrency`, `--kind`
(`binary`/`text`/`mixed` — chooses which stages get loaded), `--corpus DIR` (real
files, the only way to exercise stage 2's exiftool path), `--pid`, `--no-mem`,
(`binary`/`text`/`mixed`, which chooses the stages that get loaded), `--corpus
DIR` (real files, the only way to exercise stage 2's exiftool path), `--pid`, `--no-mem`,
`--no-stats`, `--sample-ms`, `--timeout`, `--json PATH`. Full list in the script
header. The `--json` output carries the per-stage numbers too, so a sweep can be
compared run to run.
@@ -660,7 +657,7 @@ compared run to run.
Two caveats the script reports rather than hides: it counts sink *deltas*, so it
warns if the pipeline isn't idle at the start (in-flight leftovers would be
counted as its own throughput); and because Julia's GC returns memory to the OS
lazily, a second run in the same process starts from an inflated baseline — it
lazily, a second run in the same process starts from an inflated baseline. It
resets the kernel's peak-RSS counter (`/proc/<pid>/clear_refs`) per run and flags
a drifted baseline, but for a clean growth figure restart the server between
memory runs.
@@ -668,7 +665,7 @@ memory runs.
### Stage-1 component benchmark (`bin/bench_stage1.jl`)
`bin/bench.jl` reports stage 1 as one number and `bin/bench_model.jl` takes the
*classifier* apart — but stage 1 is more than the model. Per file it also
*classifier* apart, but stage 1 is more than the model. Per file it also
renames the file into its stage directory, pushes a reference onto the
downstream queue, and logs. `bin/bench_stage1.jl` times each of those in
isolation, then times the real `handle_classify_job` end to end so the parts can
@@ -692,33 +689,32 @@ Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12; 2000 ×
| **`handle_classify_job`** | **28.3 µs** | 100% |
**This benchmark is why stage 1's per-file log lines are `@debug` rather than
`@info`.** As `@info` they cost ~71 µs of the handler's ~118 µs — about 6× the
classifier and 6× the rename — and nearly all of it was `ConsoleLogger`
`@info`.** As `@info` they cost ~71 µs of the handler's ~118 µs, about 6× the
classifier and 6× the rename, and nearly all of it was `ConsoleLogger`
*formatting* (~64 µs), not the `FlushLogger`'s per-message flush (~8 µs on top).
Demoting them took stage 1 from 8.5k files/s to 35.3k files/s on a single worker,
a 4.2× speedup for no algorithmic change. The script still prices a formatted
line, so the cost of turning them back on is visible: running the handler under
`JULIA_DEBUG=FileServer` measures 133 µs per file, a 4.7× slowdown. That is the
trade — per-file tracing is available when you want it, and off by default,
with `GET /stats` giving per-file observability that is counted rather than
formatted.
trade. Per-file tracing is available when you want it and off by default, with
`GET /stats` giving per-file observability that is counted, not formatted.
What's left is evenly split between the rename and the classifier, and neither
has an easy 2×. Two things worth knowing:
- **The rename, not the model, is the single largest component** (11.7 µs), and
it's a plain `mv` within one filesystem. Inside `classify`, the same pattern
holds: 7.9 µs of the 10.7 µs is `read_features` — the `open`, the two reads
and the `seek` — against 2.3 µs of actual inference. Stage 1 is now a
holds: 7.9 µs of the 10.7 µs is `read_features` (the `open`, the two reads
and the `seek`) against 2.3 µs of actual inference. Stage 1 is now a
filesystem-bound stage with a neural network attached, not the reverse.
- **Stage 1 now peaks at ~4 workers.** With the logger removed from the hot path
the sweep reads 35.0k/s at 1 worker, 66.6k/s at 2, **73.3k/s at 4**, then
*falls back* to 60.1k/s at 8 and 53.3k/s at 16 — every worker renaming into the
same two directories contends on the same directory inode. That ceiling
*falls back* to 60.1k/s at 8 and 53.3k/s at 16, because every worker renaming
into the same two directories contends on the same directory inode. That ceiling
coincides with the one `bin/bench_model.jl` finds for inference, so ~4 is the
number from both directions: raising `FS_WORKERS` past it costs throughput.
Reported times are the **minimum** over trials. Flags: `--files`, `--reps`,
Reported times are the minimum over trials. Flags: `--files`, `--reps`,
`--trials`, `--size`, `--dir`, `--model`, `--threads`, `--no-threads`,
`--json PATH`.
@@ -727,7 +723,7 @@ Reported times are the **minimum** over trials. Flags: `--files`, `--reps`,
Stage 2 is the one stage whose cost is dominated by something outside Julia
entirely: it forks `exiftool`, a Perl program, once per file. `bin/bench.jl`
reports the stage as a single throughput number, which can't distinguish "the
extraction is slow" from "the *spawn* is slow" — and those have opposite fixes.
extraction is slow" from "the *spawn* is slow", and those have opposite fixes.
`bin/bench_stage2.jl` times each piece in isolation, then times the real
`handle_known_job` end to end:
@@ -739,14 +735,14 @@ Two things make this benchmark different from the stage-1 one:
- **The corpus must be real files.** exiftool's cost depends on what it finds; a
file of random bytes bails out early and understates the stage by ~10×. The
default corpus is `data/done` — files that already went through stage 2 on this
default corpus is `data/done`, files that already went through stage 2 on this
machine. `--corpus PATH` points it elsewhere.
- **It prices the alternatives to one-fork-per-file**, because if the fork
dominates then the only fixes are to stop paying it per file. `exiftool
(batched Nx)` runs the whole corpus through one process; `exiftool
(-stay_open)` keeps one process alive and feeds it one file at a time over a
pipe — the shape a streaming pipeline could actually adopt. Both are measured,
not assumed.
pipe, which is the shape a streaming pipeline could actually adopt. Both are
measured, not assumed.
Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12,
exiftool 12.40; 150 real files / 102 MiB, minimum of 2 trials):
@@ -763,8 +759,8 @@ exiftool 12.40; 150 real files / 102 MiB, minimum of 2 trials):
| *alt:* `exiftool -stay_open` | 40.1 ms | 29% |
| *alt:* `exiftool` batched 150× | 37.3 ms | 27% |
**Stage 2 is exiftool and nothing else.** Everything the Julia code does —
parsing, normalizing, the durable sidecar-first commit, the log line — sums to
**Stage 2 is exiftool and nothing else.** Everything the Julia code does
(parsing, normalizing, the durable sidecar-first commit, the log line) sums to
about 1.5% of the stage. There is no point optimizing any of it.
**More than half the stage is interpreter startup, not metadata extraction.**
@@ -777,7 +773,7 @@ available win in this stage; it is measured here but not yet implemented.
**This benchmark is also why `run_with_timeout` no longer polls.** The original
watchdog polled with `sleep(0.1)` and then joined the polling task, so every call
paid the remainder of an in-flight sleep *after* the child had already exited —
paid the remainder of an in-flight sleep *after* the child had already exited:
~25 ms per file here, and a measured 101 ms on a process that exits instantly.
Replacing it with a one-shot `Timer` took the stage from 164.6 ms to 138.3 ms per
file (6→8 files/s on one worker) and cost nothing in behavior. Stage 4 shares the
@@ -785,7 +781,7 @@ wrapper and got the same fix for free.
Writing the missing tests for that wrapper turned up a second, worse problem:
**the timeout was never enforceable.** `wait(proc)` returns only once the
captured stdout pipe closes, and any grandchild inherits that pipe — so
captured stdout pipe closes, and any grandchild inherits that pipe, so
signalling the child alone left the worker blocked until the whole process tree
finished on its own (a `sh -c "trap '' TERM; sleep 30"` child ran the full 30 s
against a 1 s timeout, under both the old and new watchdog). The child now runs
@@ -795,7 +791,7 @@ these children are short-lived and timeout-bounded, which is the cheaper side of
it.
**Stage 2 scales to ~8 workers, then flattens**: 8 files/s at 1 worker, 14 at 2,
28 at 4, **49 at 8**, and 49 at 16 — the machine runs out of cores to run Perl
28 at 4, **49 at 8**, and 49 at 16. The machine runs out of cores to run Perl
on, which is exactly what you'd expect of a stage that is ~100% subprocess. Note
that the sweep pulls from a shared counter rather than splitting the corpus into
contiguous slices: per-file exiftool time spans two orders of magnitude on a real
@@ -808,13 +804,13 @@ which is far too fast to be a real disk flush. The durability that
even though the code is correct. That's a correctness question, not a speed one,
and it is not yet resolved.
Reported times are the **minimum** over trials. Flags: `--files`, `--reps`,
Reported times are the minimum over trials. Flags: `--files`, `--reps`,
`--trials`, `--corpus`, `--dir`, `--timeout`, `--threads`, `--no-threads`,
`--no-stay-open`, `--json PATH`.
### Model microbenchmark (`bin/bench_model.jl`)
`bin/bench.jl` reports stage 1 as a single number — the wall time of
`bin/bench.jl` reports stage 1 as a single number: the wall time of
`handle_classify_job`, which is a feature read, an inference, a rename, a
(disabled) debug line, and whatever contention the other three pools create.
That's the right number for capacity planning and the wrong one for "is the
@@ -834,10 +830,10 @@ Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12):
| `read_features` | **4.2–6.0 µs** | flat across a 262,144× size range (1 KiB → 256 MiB) |
| `classify()` | **8.4 µs** | 73% feature read, 27% inference |
So the model is **not** the pipeline's problem, by three orders of magnitude: the
So the model is not the pipeline's problem, by three orders of magnitude: the
same run measured stage 1 at 38.7 ms per file, ~4,500× the 8.4 µs `classify()`
costs. Whatever stage 1 spends its time on, it isn't the network. (That 38.7 ms
predates the `@debug` demotion above and is a whole-pipeline figure — it includes
predates the `@debug` demotion above and is a whole-pipeline figure. It includes
time the stage-1 worker spends *blocked* on a full downstream queue, which is why
it is three orders of magnitude above the 28.3 µs the handler costs in
isolation. For the uncontended split, see
@@ -855,13 +851,13 @@ Two findings worth acting on if stage 1 ever *does* become the constraint:
control kernel through the same sweep to place the blame: the control reaches
14.3× at 16 tasks (90% efficiency) on the same box, so the machine
parallelizes and `Lux.apply` doesn't. GC is only ~1% of it, so allocation
pressure isn't the explanation either — the cause is inside Lux/BLAS and is
pressure isn't the explanation either. The cause is inside Lux/BLAS and is
not diagnosed here. The practical consequence: raising `FS_WORKERS` past ~4
adds no classification throughput.
Reported times are the **minimum** over trials — for a microbenchmark the floor
is the signal and everything above it is scheduler and GC noise — and every timed
loop stores its result in a sink so a pure call can't be hoisted out of the loop.
Reported times are the minimum over trials, because for a microbenchmark the
floor is the signal and everything above it is scheduler and GC noise. Every
timed loop stores its result in a sink so a pure call can't be hoisted out.
Flags: `--model`, `--reps`, `--trials`, `--batches`, `--sizes`, `--no-threads`,
`--json PATH`.