jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add per-stage throughput instrumentation; fix memory-benchmark accuracy

Browse Source 
End-to-end throughput says how fast the pipeline is, not which stage is the
reason. The four stages run concurrently behind their own queues, so the
pipeline's rate *is* the slowest stage's rate and the others are invisible in
it. Nothing outside the server can recover them either: known/, unknown/ and
text/ are transient, and a file can cross one between two directory polls, so an
external sampler misses exactly the stages worth measuring.

So the pipeline counts its own work, and bin/bench.jl turns two scrapes into
rates.

- src/stats.jl: per-stage counters (completed/failed, bytes, busy_ns,
  blocked_ns, in_flight) plus intake counters, monotonic since startup in the
  Prometheus style — rates are the reader's job, so a scrape is stateless and
  two readers can't disturb each other. Recorded in worker_loop, the one place
  every stage's work passes through, so a new stage is instrumented the moment
  it is wired up and never on the read path.
- src/server.jl: GET /stats. An ordinary Oxygen route (no body to stream),
  unlike /upload. Intake counts files at the point they become stage 1's
  problem, so intake totals and stage-1 arrivals refer to the same files.
- src/queue.jl: capacity(q) joins length on the introspection seam — a depth of
  900 means nothing without knowing whether the limit is 1000 or 1_000_000.

utilization = (busy - blocked) / (window * workers) is the number that names the
bottleneck: throughput alone can't tell a saturated stage from one starved by
the stage ahead of it, since both report the same files/s. blocked_ns is what
keeps that true. Stages 1 and 3 apply blocking backpressure — a full downstream
queue means parking, not dropping — and that wait is inside the handler, so
counting it as busy would pin stage 1 at 1.0 whenever stage 2 is the real jam,
making every stage upstream of a jam look like the jam. enqueue_blocking! wraps
the retry loop so the wait is measurable at all, and keeps the three routing
paths from drifting into three different backoff behaviours.

Measured (400 mixed files, 16 KiB, concurrency 16): stage 4 is the constraint at
0.87 utilization and 853 ms/file — github-linguist is a process spawn per file —
while stages 1 and 3 idle under 0.10. Verified the blocked accounting against a
deliberately starved server (FS_TEXT_WORKERS=1, FS_TEXT_QUEUE_CAPACITY=2): stage
3 reported 100% blocked at 0.0 utilization rather than looking saturated too.

bin/bench_model.jl: the classifier alone, no server or queue in the way, because
stage 1's 38.7 ms/file cannot plausibly be a 32-64-16-2 MLP. It isn't:
Lux.apply is 2.3 us, read_features 4.2-6.0 us (flat across 1 KiB - 256 MiB, as
the seek-to-tail design intends), classify() 8.4 us — so ~99.98% of stage 1 is
rename, logging and contention, and the file read costs 3x the inference. Two
findings: batching would buy ~13x (179 ns/file at batch 512 vs 2.34 us at batch
1), and inference does not scale past ~4 threads. A pure-compute control kernel
runs the same sweep to place the blame — it reaches 14.3x at 16 tasks on this
box, so the machine parallelizes and Lux.apply does not. BLAS threads and GC are
both ruled out; the cause is inside Lux and is not diagnosed here.

Three bugs in the memory measurement, all of which produced wrong answers that
looked plausible:

- detect_pid matched any process with the launch command in its argv, including
  the shell that started the server — one run reported 3.64 MiB as the server's
  memory. Candidates are now filtered by /proc/<pid>/comm, what the process is
  rather than what its arguments say; no pattern over argv can do that.
- Baseline RSS was read *before* clear_refs reset the peak counter, so the two
  numbers had different origins. The 2 GiB run reported -1.01 MiB of growth;
  reading the baseline after the reset makes it 31.3 MiB.
- Negative growth is now reported as "none measurable" rather than a negative
  figure, which reads as a memory saving.

Re-measuring with those fixed keeps the claim that matters — growth is flat in
file size (14-31 MiB from 256 MiB to 2 GiB), so nothing is buffering — but the
concurrency coefficient does not survive: a freshly started server settles
anywhere in an ~860-985 MiB band, so baseline variance is comparable to the
growth being measured, and the old table quoted megabyte precision the
measurement never supported. README now states the shape, requires a ~30s settle
before a memory run, and says plainly that linear-in-concurrency is
undemonstrated rather than leaving an authoritative-looking number.

README also gains a single runnable sequence for all three harnesses: the server
prerequisite was never shown inline, so following the benchmarking section
top-to-bottom just produced "cannot reach /health".

Tests: 276 pass (41 new) — the blocked-vs-busy split, worker_loop draining
in_flight through a throwing handler, and the JSON round-trip of the field names
bench.jl reads.
This commit is contained in:
Jeffrey Ward
2026-08-02 23:49:16 -04:00
parent 0d8eba05b8
commit c5d488d9b4
9 changed files with 1312 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

197
README.md
View File

@@ -447,6 +447,9 @@ curl http://127.0.0.1:8080/health
# upload one or more files (multipart/form-data)
curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload
# 202 {"accepted":[{"id":"<uuid>","name":"report.pdf"}, ...]}
# per-stage counters
curl http://127.0.0.1:8080/stats
```
Each file in a request becomes its own job. Responses:
@@ -456,19 +459,93 @@ Each file in a request becomes its own job. Responses:
- `503 Service Unavailable` — queue full, retry later
- `500 Internal Server Error` — failed to write a file to disk
### `GET /stats` — per-stage counters
The pipeline counts its own work (`src/stats.jl`), because nothing outside it
can: `known/`, `unknown/` and `text/` are *transient*, so a file can cross one
between two directory polls and an external sampler will miss exactly the stages
you most want to measure.
```jsonc
{
"now": 1785725300.5, "since": 1785725291.9, "uptime_seconds": 8.5,
"intake": { "requests": 400, "files": 400, "bytes": 6553600, "rejected": 0 },
"stages": [
{ "stage": 4, "name": "language", "workers": 16,
"queue_depth": 184, "queue_capacity": 1000,
"completed": 200, "failed": 0, "bytes": 3276800,
"busy_seconds": 170.5, // summed handler time across the pool
"blocked_seconds": 0.0, // of that, time parked on a full downstream queue
"in_flight": 3 }
]
}
```
Counters are monotonic since startup, Prometheus-style — rates are the reader's
job, so a scrape is stateless and two readers can't disturb each other. Take two
scrapes Δt apart and subtract:
```
throughput = Δcompleted / Δt
utilization = (Δbusy_seconds − Δblocked_seconds) / (Δt × workers)
```
**Utilization is the number that names the bottleneck.** In a pipeline every
stage completes the same files, so at steady state they all report near-identical
files/s no matter which one is the constraint; what separates them is how hard
each pool worked to keep up. The bottleneck sits near 1.0 with its queue backing
up while its neighbours idle.
`blocked_seconds` is what keeps that true. Stages 1 and 3 apply *blocking*
backpressure — a full downstream queue means the handler parks and retries rather
than dropping the file — and that wait happens inside the handler. Counting it as
busy would pin stage 1 at 1.0 whenever stage 2 is the real jam, making every
stage upstream of a jam look like the jam. Subtracted out, utilization means
"doing its own work", and a high blocked share becomes its own signal: a stage
blocked 90% of the time is naming its successor.
The counters are a handful of atomic adds per file, recorded in `worker_loop` —
the one place every stage's work passes through, so a new stage is instrumented
the moment it is wired up, and never on the read path.
## Benchmarking (throughput + memory)
`bin/bench.jl` measures the pipeline against a **running** server. It must run on
the same machine (it reads the sink dirs and `/proc`), and it changes nothing in
`src/` — it only speaks HTTP and counts files.
There are three harnesses. Only the first needs a running server:
| script | measures | server? |
|---|---|---|
| `bin/bench.jl` (below) | intake, end-to-end and per-stage throughput; server RSS | **yes** |
| [`bin/bench_model.jl`](#model-microbenchmark-binbench_modeljl) | the classifier alone: inference, feature reads, thread scaling | no |
| [`bin/cluster_calibrate.jl`](#unknown-format-discovery-stage-5-offline) | stage-5 clustering quality vs. an NCD baseline | no |
Running all of them from a clean checkout:
```bash
julia --project=. -e 'using Pkg; Pkg.instantiate()' # once
# 1. the model, on its own — no server involved
julia --project=. -t auto bin/bench_model.jl
# 2. the pipeline. Start the server in one terminal…
julia --project=. -t auto bin/server.jl
# …and drive it from another. Restart the server between memory runs: Julia's
# GC returns memory to the OS lazily, so a second run starts inflated.
julia --project=. -t auto bin/bench.jl --files 2000 --size 8k --concurrency 32
julia --project=. -t auto bin/bench.jl --files 2 --size 1g --concurrency 1 # memory
julia --project=. -t auto bin/bench.jl --corpus ../training_set --concurrency 16
# 3. stage-5 clustering quality (offline, needs a labelled corpus)
julia --project=. bin/cluster_calibrate.jl ../training_set
```
Two properties of this design dictate how it measures:
The test suite is `julia --project=. -t auto -e 'using Pkg; Pkg.test()'`.
`bin/bench.jl` must run on the same machine as the server (it reads the sink dirs
and `/proc`); otherwise it only speaks HTTP — `/upload`, `/health`, and `/stats`
for the per-stage numbers.
Three properties of this design dictate how it measures:
- **HTTP latency is not throughput.** `/upload` returns `202` once the bytes are
spooled and a reference is enqueued — all four stages run *after* the response,
@@ -476,8 +553,30 @@ Two properties of this design dictate how it measures:
corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
`failed/`) until the count stops moving, and reports both numbers separately:
intake rate *and* end-to-end completion rate. It also samples the intermediate
stage dirs, so the peak depth of `spool/`/`known/`/`unknown/`/`text/` names the
bottleneck stage directly.
stage dirs, so the peak depth of `spool/`/`known/`/`unknown/`/`text/` shows
where work piles up.
- **End-to-end throughput doesn't name the slow stage.** The four stages run
concurrently behind their own queues, so the pipeline's rate *is* the slowest
stage's rate and the others are invisible in it. The bench scrapes
[`/stats`](#get-stats--per-stage-counters) before and after the run and
subtracts, giving each stage its own throughput, mean service time and
utilization:
```
PER-STAGE (server counters, delta over the end-to-end window)
stage files/s MiB/s svc ms util blocked peak queue failed
1 classify 32.52 0.51 38.7 0.08 0.0% 209/1000 0
2 enrich 0.24 0.0 541.0 0.01 0.0% 0/1000 0
3 triage 32.27 0.5 33.9 0.07 0.0% 83/1000 0
4 language 16.26 0.25 852.6 0.87 0.0% 184/1000 0
bottleneck stage 4 (language) at 87.0% utilization of 16 worker(s)
```
(400 mixed files, 16 KiB each, concurrency 16. Stage 4 is the constraint —
`github-linguist` is a process spawn per file at ~850 ms — and the 200 text
files queue up behind it while stages 1 and 3 idle at under 10%.) Read *down*
the util column, not the files/s column: each stage sees a different subset of
the corpus, so a low rate can just mean little work was routed there.
- **Memory should be flat in file size, and the sweep is what proves it.** Both
halves of the pipeline are bounded: the workers read bounded prefixes (16+16
bytes to classify, 8 KB to sniff, 64 KB to language-detect), and intake streams
@@ -486,14 +585,33 @@ Two properties of this design dictate how it measures:
| upload size | concurrency | RSS growth |
|---|---|---|
| 256 MiB × 4 | 1 | 21.8 MiB |
| 1 GiB × 2 | 1 | 20.8 MiB |
| 2 GiB × 1 | 1 | 17.0 MiB |
| 256 MiB × 4 | 4 | 86.9 MiB (21.7 MiB per in-flight upload) |
| 256 MiB × 4 | 1 | 14.0 / 20.4 MiB (two runs) |
| 1 GiB × 2 | 1 | 22.3 MiB |
| 2 GiB × 1 | 1 | 31.3 MiB |
| 256 MiB × 4 | 4 | none measurable (peak stayed under the baseline) |
Flat across a 8× range of file sizes, and linear in concurrency — which is the
shape to expect. (The residual ~20 MiB per in-flight upload is GC churn from the
chunk reads, not retained buffers; it does not grow with the file.) Before intake
**Read the shape, not the digits.** Growth is flat across an 8× range of file
sizes — 14–31 MiB whether the upload is 256 MiB or 2 GiB — which is the claim
that matters: nothing scales with file size, so nothing is buffering. The
absolute figures are *not* precise to the megabyte. A freshly started server
settles anywhere in an ~860–985 MiB band, so run-to-run variance in the
baseline is comparable to the growth being measured; the residual is GC churn
from the chunk reads, not retained buffers.
Two consequences worth knowing before quoting these numbers:
- **Let the server settle ~30s after startup** before a memory run, or the
baseline is sampled mid-fall and the run reports less growth than it caused
(or none at all, as the concurrency-4 row above did).
- **Linear-in-concurrency is not currently demonstrated.** An earlier
measurement put 4 concurrent 256 MiB uploads at 86.9 MiB (21.7 per in-flight
upload), but that run predates a fix to how the baseline was sampled — it was
read *before* the peak counter was reset, against a different origin — and the
re-measurement above cannot reproduce it: the growth sits under the noise
floor. Expect concurrency to cost memory; don't trust a specific coefficient
without a quieter machine or many more runs.
Before intake
was streamed, the same 256 MiB upload grew RSS by ~700 MiB and 4 concurrent ones
pushed a 950 MiB baseline past 2 GiB, so **a `--size` sweep that slopes upward is
the regression signal** — it means something has started buffering bodies again.
@@ -501,7 +619,9 @@ Two properties of this design dictate how it measures:
Flags: `--files`, `--size` (`8k`/`64m`/`1g`), `--concurrency`, `--kind`
(`binary`/`text`/`mixed` — chooses which stages get loaded), `--corpus DIR` (real
files, the only way to exercise stage 2's exiftool path), `--pid`, `--no-mem`,
`--sample-ms`, `--timeout`, `--json PATH`. Full list in the script header.
`--no-stats`, `--sample-ms`, `--timeout`, `--json PATH`. Full list in the script
header. The `--json` output carries the per-stage numbers too, so a sweep can be
compared run to run.
Two caveats the script reports rather than hides: it counts sink *deltas*, so it
warns if the pipeline isn't idle at the start (in-flight leftovers would be
@@ -511,6 +631,53 @@ resets the kernel's peak-RSS counter (`/proc/<pid>/clear_refs`) per run and flag
a drifted baseline, but for a clean growth figure restart the server between
memory runs.
### Model microbenchmark (`bin/bench_model.jl`)
`bin/bench.jl` reports stage 1 as a single number — the wall time of
`handle_classify_job`, which is a feature read, an inference, a rename, a log
line, and whatever contention the other three pools create. That's the right
number for capacity planning and the wrong one for "is the model slow?".
`bin/bench_model.jl` answers that separately, with no server, queue, or HTTP
involved:
```bash
julia --project=. -t auto bin/bench_model.jl
```
Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12):
| what | per file | notes |
|---|---|---|
| `Lux.apply`, batch 1 | **2.3 µs** | 768 B allocated per call |
| `read_features` | **4.2–6.0 µs** | flat across a 262,144× size range (1 KiB → 256 MiB) |
| `classify()` | **8.4 µs** | 73% feature read, 27% inference |
So the model is **not** the pipeline's problem, by three orders of magnitude: the
same run measured stage 1 at 38.7 ms per file, ~4,500× the 8.4 µs `classify()`
costs. Whatever stage 1 spends its time on, it isn't the network.
Two findings worth acting on if stage 1 ever *does* become the constraint:
- **Batching would buy ~13×.** A 32×1 matmul wastes most of a BLAS call:
batch 64 costs 280 ns/file and batch 512 costs 179 ns/file, against 2.34 µs
one at a time. The pipeline classifies strictly one file per job today, so it
pays the worst row in that table.
- **Inference does not scale past ~4 threads.** Concurrent `Lux.apply` on the
shared read-only `Classifier` peaks around 1.2M files/s at 4 tasks and then
*falls back* to single-thread throughput at 16. The script runs a pure-compute
control kernel through the same sweep to place the blame: the control reaches
14.3× at 16 tasks (90% efficiency) on the same box, so the machine
parallelizes and `Lux.apply` doesn't. GC is only ~1% of it, so allocation
pressure isn't the explanation either — the cause is inside Lux/BLAS and is
not diagnosed here. The practical consequence: raising `FS_WORKERS` past ~4
adds no classification throughput.
Reported times are the **minimum** over trials — for a microbenchmark the floor
is the signal and everything above it is scheduler and GC noise — and every timed
loop stores its result in a sink so a pure call can't be hoisted out of the loop.
Flags: `--model`, `--reps`, `--trials`, `--batches`, `--sizes`, `--no-threads`,
`--json PATH`.
## Layout
```
@@ -519,6 +686,7 @@ src/
config.jl Config struct + env parsing
job.jl Job (the queue reference)
queue.jl JobQueue seam + in-process ChannelQueue
stats.jl per-stage counters behind GET /stats (throughput, utilization)
multipart.jl streaming multipart/form-data reader (intake never buffers a file)
spool.jl filename sanitizing, streaming spool/move, startup recovery
model.jl NN architecture + byte→feature mapping (shared with trainer)
@@ -533,6 +701,7 @@ src/
bin/
server.jl entry point
bench.jl throughput + memory harness against a running server
bench_model.jl classifier microbenchmark (inference, feature reads, scaling)
train.jl offline training script → model/classifier.jld2
cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline
cluster_sweep.jl stage-5 phase-B runner: sweep binary/, update catalog, write nominations