Add per-stage throughput instrumentation; fix memory-benchmark accuracy

End-to-end throughput says how fast the pipeline is, not which stage is the
reason. The four stages run concurrently behind their own queues, so the
pipeline's rate *is* the slowest stage's rate and the others are invisible in
it. Nothing outside the server can recover them either: known/, unknown/ and
text/ are transient, and a file can cross one between two directory polls, so an
external sampler misses exactly the stages worth measuring.

So the pipeline counts its own work, and bin/bench.jl turns two scrapes into
rates.

- src/stats.jl: per-stage counters (completed/failed, bytes, busy_ns,
  blocked_ns, in_flight) plus intake counters, monotonic since startup in the
  Prometheus style — rates are the reader's job, so a scrape is stateless and
  two readers can't disturb each other. Recorded in worker_loop, the one place
  every stage's work passes through, so a new stage is instrumented the moment
  it is wired up and never on the read path.
- src/server.jl: GET /stats. An ordinary Oxygen route (no body to stream),
  unlike /upload. Intake counts files at the point they become stage 1's
  problem, so intake totals and stage-1 arrivals refer to the same files.
- src/queue.jl: capacity(q) joins length on the introspection seam — a depth of
  900 means nothing without knowing whether the limit is 1000 or 1_000_000.

utilization = (busy - blocked) / (window * workers) is the number that names the
bottleneck: throughput alone can't tell a saturated stage from one starved by
the stage ahead of it, since both report the same files/s. blocked_ns is what
keeps that true. Stages 1 and 3 apply blocking backpressure — a full downstream
queue means parking, not dropping — and that wait is inside the handler, so
counting it as busy would pin stage 1 at 1.0 whenever stage 2 is the real jam,
making every stage upstream of a jam look like the jam. enqueue_blocking! wraps
the retry loop so the wait is measurable at all, and keeps the three routing
paths from drifting into three different backoff behaviours.

Measured (400 mixed files, 16 KiB, concurrency 16): stage 4 is the constraint at
0.87 utilization and 853 ms/file — github-linguist is a process spawn per file —
while stages 1 and 3 idle under 0.10. Verified the blocked accounting against a
deliberately starved server (FS_TEXT_WORKERS=1, FS_TEXT_QUEUE_CAPACITY=2): stage
3 reported 100% blocked at 0.0 utilization rather than looking saturated too.

bin/bench_model.jl: the classifier alone, no server or queue in the way, because
stage 1's 38.7 ms/file cannot plausibly be a 32-64-16-2 MLP. It isn't:
Lux.apply is 2.3 us, read_features 4.2-6.0 us (flat across 1 KiB - 256 MiB, as
the seek-to-tail design intends), classify() 8.4 us — so ~99.98% of stage 1 is
rename, logging and contention, and the file read costs 3x the inference. Two
findings: batching would buy ~13x (179 ns/file at batch 512 vs 2.34 us at batch
1), and inference does not scale past ~4 threads. A pure-compute control kernel
runs the same sweep to place the blame — it reaches 14.3x at 16 tasks on this
box, so the machine parallelizes and Lux.apply does not. BLAS threads and GC are
both ruled out; the cause is inside Lux and is not diagnosed here.

Three bugs in the memory measurement, all of which produced wrong answers that
looked plausible:

- detect_pid matched any process with the launch command in its argv, including
  the shell that started the server — one run reported 3.64 MiB as the server's
  memory. Candidates are now filtered by /proc/<pid>/comm, what the process is
  rather than what its arguments say; no pattern over argv can do that.
- Baseline RSS was read *before* clear_refs reset the peak counter, so the two
  numbers had different origins. The 2 GiB run reported -1.01 MiB of growth;
  reading the baseline after the reset makes it 31.3 MiB.
- Negative growth is now reported as "none measurable" rather than a negative
  figure, which reads as a memory saving.

Re-measuring with those fixed keeps the claim that matters — growth is flat in
file size (14-31 MiB from 256 MiB to 2 GiB), so nothing is buffering — but the
concurrency coefficient does not survive: a freshly started server settles
anywhere in an ~860-985 MiB band, so baseline variance is comparable to the
growth being measured, and the old table quoted megabyte precision the
measurement never supported. README now states the shape, requires a ~30s settle
before a memory run, and says plainly that linear-in-concurrency is
undemonstrated rather than leaving an authoritative-looking number.

README also gains a single runnable sequence for all three harnesses: the server
prerequisite was never shown inline, so following the benchmarking section
top-to-bottom just produced "cannot reach /health".

Tests: 276 pass (41 new) — the blocked-vs-busy split, worker_loop draining
in_flight through a throwing handler, and the JSON round-trip of the field names
bench.jl reads.
This commit is contained in:
2026-08-02 23:49:16 -04:00
parent 0d8eba05b8
commit c5d488d9b4
9 changed files with 1312 additions and 54 deletions

View File

@@ -14,6 +14,7 @@ include("multipart.jl") # streaming multipart reader (defines UPLOAD_CHUNK_BYTES
include("config.jl")
include("job.jl")
include("queue.jl")
include("stats.jl") # per-stage counters behind GET /stats (needs Config/Job/JobQueue)
include("spool.jl")
include("model.jl") # build_model() + read_features(); shared with bin/train.jl
include("classify.jl") # Classifier + load_classifier/classify (needs model.jl)
@@ -125,16 +126,29 @@ function run(; overrides...)
recovered_text = recover_dir!(cfg.text_dir, text_queue)
@info "starting file-server" host=cfg.host port=cfg.port workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count capacity=cfg.queue_capacity known_capacity=cfg.known_queue_capacity unknown_capacity=cfg.unknown_queue_capacity text_capacity=cfg.text_queue_capacity recovered=recovered recovered_known=recovered_known recovered_unknown=recovered_unknown recovered_text=recovered_text
# Zero the counters here, not at module load: `since` should mean "serving
# since", so a scrape's totals cover the run, not the minutes spent loading
# the classifier. Nothing has been processed yet — recovery only enqueues.
reset_metrics!()
# Each pool gets its stage's counters (src/stats.jl); `worker_loop` records
# into them, `GET /stats` reads them out. Stage 1 and 3 also hand theirs to
# their handler, which charges time parked on a full downstream queue to
# `blocked_ns` so it isn't mistaken for work.
st = METRICS.stages
workers = [Threads.@spawn worker_loop(i, cfg, queue,
(job, c, wid) -> handle_classify_job(job, c, wid, known_queue, unknown_queue))
(job, c, wid) -> handle_classify_job(job, c, wid, known_queue, unknown_queue, st.classify),
st.classify)
for i in 1:cfg.worker_count]
known_workers = [Threads.@spawn worker_loop(i, cfg, known_queue, handle_known_job)
known_workers = [Threads.@spawn worker_loop(i, cfg, known_queue, handle_known_job, st.enrich)
for i in 1:cfg.known_worker_count]
unknown_workers = [Threads.@spawn worker_loop(i, cfg, unknown_queue,
(job, c, wid) -> handle_unknown_job(job, c, wid, text_queue))
(job, c, wid) -> handle_unknown_job(job, c, wid, text_queue, st.triage),
st.triage)
for i in 1:cfg.unknown_worker_count]
text_workers = [Threads.@spawn worker_loop(i, cfg, text_queue,
(job, c, wid) -> handle_text_job(job, c, wid, DETECTOR[]))
(job, c, wid) -> handle_text_job(job, c, wid, DETECTOR[]),
st.language)
for i in 1:cfg.text_worker_count]
register_routes()