Add stage-2 decomposition benchmark; fix run_with_timeout latency and enforceability
bin/bench.jl reports stage 2 as a single throughput number, which can't distinguish slow extraction from a slow spawn — and those have opposite fixes. bin/bench_stage2.jl times each component in isolation, then times the real handle_known_job end to end. It draws its corpus from real files (default data/done) because random bytes make exiftool bail out early and understate the stage by ~10x, and it prices both fork-free alternatives (batched, -stay_open) so the cost of one-fork-per-file is a measurement rather than a guess. The benchmark found stage 2 to be ~98% exiftool, and found two problems in run_with_timeout, which stages 2 and 4 share: 1. The watchdog polled with sleep(0.1) and then joined the polling task, so every call paid the remainder of an in-flight sleep after the child had already exited: ~25 ms per file, and a measured 101 ms on a process that exits instantly. Replaced with a one-shot Timer cancelled when the child exits. Stage 2 goes from 164.6 ms to 138.3 ms per file (6 -> 8 files/s on one worker); the wrapper is now within noise of a bare Base.run. 2. Writing the missing tests showed the timeout was never enforceable, in the old implementation as much as the new. wait(proc) returns only once the captured stdout pipe closes, and grandchildren inherit that pipe, so signalling the child alone left the worker blocked until the whole process tree finished on its own — a `sh -c "trap '' TERM; sleep 30"` child ran the full 30 s against a 1 s timeout. The child now runs in its own process group and the timeout signals the group. The trade is that a hard crash of the server orphans an in-flight child rather than taking it down with it. Three new tests cover the fast path, the timeout, and the SIGTERM-ignoring escalation; the second was previously unexercised, which is why the bug stood. Not addressed here, but measured and documented in the README: Perl interpreter startup is 76.7 ms of the remaining 135.9 ms call, so a persistent exiftool (-stay_open, 40.1 ms/file) would cut the stage by roughly another 70%. And fsync_dir measures 1.75 us, too fast to be a real flush — commit_enriched!'s durability may not hold on this filesystem, which is a correctness question left open. Claude-Session: https://claude.ai/code/session_01Xy9At7HNLHWNmfh1yw71Uy
This commit is contained in:
96
README.md
96
README.md
@@ -510,12 +510,13 @@ the moment it is wired up, and never on the read path.
|
||||
|
||||
## Benchmarking (throughput + memory)
|
||||
|
||||
There are four harnesses. Only the first needs a running server:
|
||||
There are five harnesses. Only the first needs a running server:
|
||||
|
||||
| script | measures | server? |
|
||||
|---|---|---|
|
||||
| `bin/bench.jl` (below) | intake, end-to-end and per-stage throughput; server RSS | **yes** |
|
||||
| [`bin/bench_stage1.jl`](#stage-1-component-benchmark-binbench_stage1jl) | stage 1 taken apart: classify vs. rename vs. enqueue vs. logging | no |
|
||||
| [`bin/bench_stage2.jl`](#stage-2-component-benchmark-binbench_stage2jl) | stage 2 taken apart: exiftool spawn vs. extraction vs. commit | no |
|
||||
| [`bin/bench_model.jl`](#model-microbenchmark-binbench_modeljl) | the classifier alone: inference, feature reads, thread scaling | no |
|
||||
| [`bin/cluster_calibrate.jl`](#unknown-format-discovery-stage-5-offline) | stage-5 clustering quality vs. an NCD baseline | no |
|
||||
|
||||
@@ -530,6 +531,9 @@ julia --project=. -t auto bin/bench_model.jl
|
||||
# 1b. stage 1 taken apart — also no server
|
||||
julia --project=. -t auto bin/bench_stage1.jl
|
||||
|
||||
# 1c. stage 2 taken apart — needs a directory of real files, not generated ones
|
||||
julia --project=. -t auto bin/bench_stage2.jl
|
||||
|
||||
# 2. the pipeline. Start the server in one terminal…
|
||||
julia --project=. -t auto bin/server.jl
|
||||
|
||||
@@ -692,6 +696,96 @@ Reported times are the **minimum** over trials. Flags: `--files`, `--reps`,
|
||||
`--trials`, `--size`, `--dir`, `--model`, `--threads`, `--no-threads`,
|
||||
`--json PATH`.
|
||||
|
||||
### Stage-2 component benchmark (`bin/bench_stage2.jl`)
|
||||
|
||||
Stage 2 is the one stage whose cost is dominated by something outside Julia
|
||||
entirely: it forks `exiftool`, a Perl program, once per file. `bin/bench.jl`
|
||||
reports the stage as a single throughput number, which can't distinguish "the
|
||||
extraction is slow" from "the *spawn* is slow" — and those have opposite fixes.
|
||||
`bin/bench_stage2.jl` times each piece in isolation, then times the real
|
||||
`handle_known_job` end to end:
|
||||
|
||||
```bash
|
||||
julia --project=. -t auto bin/bench_stage2.jl
|
||||
```
|
||||
|
||||
Two things make this benchmark different from the stage-1 one:
|
||||
|
||||
- **The corpus must be real files.** exiftool's cost depends on what it finds; a
|
||||
file of random bytes bails out early and understates the stage by ~10×. The
|
||||
default corpus is `data/done` — files that already went through stage 2 on this
|
||||
machine. `--corpus PATH` points it elsewhere.
|
||||
- **It prices the alternatives to one-fork-per-file**, because if the fork
|
||||
dominates then the only fixes are to stop paying it per file. `exiftool
|
||||
(batched Nx)` runs the whole corpus through one process; `exiftool
|
||||
(-stay_open)` keeps one process alive and feeds it one file at a time over a
|
||||
pipe — the shape a streaming pipeline could actually adopt. Both are measured,
|
||||
not assumed.
|
||||
|
||||
Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12,
|
||||
exiftool 12.40; 150 real files / 102 MiB, minimum of 2 trials):
|
||||
|
||||
| component | per file | share of the handler |
|
||||
|---|---|---|
|
||||
| `run_exiftool()` | 135.9 ms | 98% |
|
||||
| ↳ bare fork + Perl boot (`exiftool -ver`) | 76.7 ms | 55% |
|
||||
| ↳ `JSON3.read` + tag map | 10 µs | 0.0% |
|
||||
| `normalize_metadata` | 1.7 µs | 0.0% |
|
||||
| `commit_enriched!` (sidecar + fsyncs + rename) | 2.0 ms | 1.5% |
|
||||
| per-file logging (`@info`, flush→file) | 47 µs | 0.0% |
|
||||
| **`handle_known_job`** | **138.3 ms** | 100% |
|
||||
| *alt:* `exiftool -stay_open` | 40.1 ms | 29% |
|
||||
| *alt:* `exiftool` batched 150× | 37.3 ms | 27% |
|
||||
|
||||
**Stage 2 is exiftool and nothing else.** Everything the Julia code does —
|
||||
parsing, normalizing, the durable sidecar-first commit, the log line — sums to
|
||||
about 1.5% of the stage. There is no point optimizing any of it.
|
||||
|
||||
**More than half the stage is interpreter startup, not metadata extraction.**
|
||||
The bare `exiftool -ver` (fork, Perl boot, module loads, read no file) costs
|
||||
76.7 ms against a 135.9 ms full call. Both fork-free alternatives agree on what's
|
||||
left: ~37–40 ms of actual work per file. So a persistent exiftool would cut the
|
||||
stage by ~70%, and `-stay_open` gets there without giving up the one-file-in,
|
||||
one-result-out shape the pipeline needs. That remains the single biggest
|
||||
available win in this stage; it is measured here but not yet implemented.
|
||||
|
||||
**This benchmark is also why `run_with_timeout` no longer polls.** The original
|
||||
watchdog polled with `sleep(0.1)` and then joined the polling task, so every call
|
||||
paid the remainder of an in-flight sleep *after* the child had already exited —
|
||||
~25 ms per file here, and a measured 101 ms on a process that exits instantly.
|
||||
Replacing it with a one-shot `Timer` took the stage from 164.6 ms to 138.3 ms per
|
||||
file (6→8 files/s on one worker) and cost nothing in behavior. Stage 4 shares the
|
||||
wrapper and got the same fix for free.
|
||||
|
||||
Writing the missing tests for that wrapper turned up a second, worse problem:
|
||||
**the timeout was never enforceable.** `wait(proc)` returns only once the
|
||||
captured stdout pipe closes, and any grandchild inherits that pipe — so
|
||||
signalling the child alone left the worker blocked until the whole process tree
|
||||
finished on its own (a `sh -c "trap '' TERM; sleep 30"` child ran the full 30 s
|
||||
against a 1 s timeout, under both the old and new watchdog). The child now runs
|
||||
in its own process group and the timeout signals the group. The trade is that a
|
||||
hard crash of the server orphans an in-flight child rather than taking it down;
|
||||
these children are short-lived and timeout-bounded, which is the cheaper side of
|
||||
it.
|
||||
|
||||
**Stage 2 scales to ~8 workers, then flattens**: 8 files/s at 1 worker, 14 at 2,
|
||||
28 at 4, **49 at 8**, and 49 at 16 — the machine runs out of cores to run Perl
|
||||
on, which is exactly what you'd expect of a stage that is ~100% subprocess. Note
|
||||
that the sweep pulls from a shared counter rather than splitting the corpus into
|
||||
contiguous slices: per-file exiftool time spans two orders of magnitude on a real
|
||||
corpus (one 2.1 s archive among 48 files), and a static split reports a scaling
|
||||
ceiling that is really just load imbalance.
|
||||
|
||||
One caveat the numbers raise but don't answer: **`fsync_dir` measures 1.75 µs**,
|
||||
which is far too fast to be a real disk flush. The durability that
|
||||
`commit_enriched!` is written for may not survive power loss on this filesystem,
|
||||
even though the code is correct. That's a correctness question, not a speed one,
|
||||
and it is not yet resolved.
|
||||
|
||||
Reported times are the **minimum** over trials. Flags: `--files`, `--reps`,
|
||||
`--trials`, `--corpus`, `--dir`, `--timeout`, `--threads`, `--no-threads`,
|
||||
`--no-stay-open`, `--json PATH`.
|
||||
|
||||
### Model microbenchmark (`bin/bench_model.jl`)
|
||||
|
||||
`bin/bench.jl` reports stage 1 as a single number — the wall time of
|
||||
|
||||
Reference in New Issue
Block a user