Add per-stage throughput instrumentation; fix memory-benchmark accuracy
End-to-end throughput says how fast the pipeline is, not which stage is the reason. The four stages run concurrently behind their own queues, so the pipeline's rate *is* the slowest stage's rate and the others are invisible in it. Nothing outside the server can recover them either: known/, unknown/ and text/ are transient, and a file can cross one between two directory polls, so an external sampler misses exactly the stages worth measuring. So the pipeline counts its own work, and bin/bench.jl turns two scrapes into rates. - src/stats.jl: per-stage counters (completed/failed, bytes, busy_ns, blocked_ns, in_flight) plus intake counters, monotonic since startup in the Prometheus style — rates are the reader's job, so a scrape is stateless and two readers can't disturb each other. Recorded in worker_loop, the one place every stage's work passes through, so a new stage is instrumented the moment it is wired up and never on the read path. - src/server.jl: GET /stats. An ordinary Oxygen route (no body to stream), unlike /upload. Intake counts files at the point they become stage 1's problem, so intake totals and stage-1 arrivals refer to the same files. - src/queue.jl: capacity(q) joins length on the introspection seam — a depth of 900 means nothing without knowing whether the limit is 1000 or 1_000_000. utilization = (busy - blocked) / (window * workers) is the number that names the bottleneck: throughput alone can't tell a saturated stage from one starved by the stage ahead of it, since both report the same files/s. blocked_ns is what keeps that true. Stages 1 and 3 apply blocking backpressure — a full downstream queue means parking, not dropping — and that wait is inside the handler, so counting it as busy would pin stage 1 at 1.0 whenever stage 2 is the real jam, making every stage upstream of a jam look like the jam. enqueue_blocking! wraps the retry loop so the wait is measurable at all, and keeps the three routing paths from drifting into three different backoff behaviours. Measured (400 mixed files, 16 KiB, concurrency 16): stage 4 is the constraint at 0.87 utilization and 853 ms/file — github-linguist is a process spawn per file — while stages 1 and 3 idle under 0.10. Verified the blocked accounting against a deliberately starved server (FS_TEXT_WORKERS=1, FS_TEXT_QUEUE_CAPACITY=2): stage 3 reported 100% blocked at 0.0 utilization rather than looking saturated too. bin/bench_model.jl: the classifier alone, no server or queue in the way, because stage 1's 38.7 ms/file cannot plausibly be a 32-64-16-2 MLP. It isn't: Lux.apply is 2.3 us, read_features 4.2-6.0 us (flat across 1 KiB - 256 MiB, as the seek-to-tail design intends), classify() 8.4 us — so ~99.98% of stage 1 is rename, logging and contention, and the file read costs 3x the inference. Two findings: batching would buy ~13x (179 ns/file at batch 512 vs 2.34 us at batch 1), and inference does not scale past ~4 threads. A pure-compute control kernel runs the same sweep to place the blame — it reaches 14.3x at 16 tasks on this box, so the machine parallelizes and Lux.apply does not. BLAS threads and GC are both ruled out; the cause is inside Lux and is not diagnosed here. Three bugs in the memory measurement, all of which produced wrong answers that looked plausible: - detect_pid matched any process with the launch command in its argv, including the shell that started the server — one run reported 3.64 MiB as the server's memory. Candidates are now filtered by /proc/<pid>/comm, what the process is rather than what its arguments say; no pattern over argv can do that. - Baseline RSS was read *before* clear_refs reset the peak counter, so the two numbers had different origins. The 2 GiB run reported -1.01 MiB of growth; reading the baseline after the reset makes it 31.3 MiB. - Negative growth is now reported as "none measurable" rather than a negative figure, which reads as a memory saving. Re-measuring with those fixed keeps the claim that matters — growth is flat in file size (14-31 MiB from 256 MiB to 2 GiB), so nothing is buffering — but the concurrency coefficient does not survive: a freshly started server settles anywhere in an ~860-985 MiB band, so baseline variance is comparable to the growth being measured, and the old table quoted megabyte precision the measurement never supported. README now states the shape, requires a ~30s settle before a memory run, and says plainly that linear-in-concurrency is undemonstrated rather than leaving an authoritative-looking number. README also gains a single runnable sequence for all three harnesses: the server prerequisite was never shown inline, so following the benchmarking section top-to-bottom just produced "cannot reach /health". Tests: 276 pass (41 new) — the blocked-vs-busy split, worker_loop draining in_flight through a throwing handler, and the JSON round-trip of the field names bench.jl reads.
This commit is contained in:
256
bin/bench.jl
256
bin/bench.jl
@@ -11,7 +11,16 @@
|
||||
# (done/, text_done/, binary/, failed/), not the response rate. We upload a
|
||||
# corpus, then poll the sinks until the file count stops moving.
|
||||
#
|
||||
# 2. Memory should be flat in file size, and that claim needs checking on two
|
||||
# 2. End-to-end throughput doesn't name the slow stage. The four stages run
|
||||
# concurrently behind their own queues, so the pipeline's rate is the
|
||||
# slowest stage's rate and the others are invisible. Directory polling can't
|
||||
# recover them either — known/, unknown/ and text/ are transient, and a file
|
||||
# can cross one between two samples. So the server keeps per-stage counters
|
||||
# (src/stats.jl) and we scrape GET /stats before and after: the deltas give
|
||||
# each stage's throughput, mean service time, and worker utilization, and
|
||||
# utilization is what actually names the bottleneck (see `stage_report`).
|
||||
#
|
||||
# 3. Memory should be flat in file size, and that claim needs checking on two
|
||||
# axes. The workers read bounded prefixes (16+16 bytes to classify, 8 KB to
|
||||
# sniff, 64 KB to language-detect), and intake streams each upload from the
|
||||
# socket to the spool file a chunk at a time (FS_UPLOAD_CHUNK_BYTES, see
|
||||
@@ -24,9 +33,10 @@
|
||||
# over, and a 256 MiB upload grew RSS by ~700 MiB. If a --size sweep ever
|
||||
# slopes upward again, something has started buffering.)
|
||||
#
|
||||
# This is an external harness: it makes no assumptions about the server beyond
|
||||
# the HTTP contract and the on-disk sink layout, and requires no changes to src/.
|
||||
# It must run on the same machine as the server (it reads sink dirs and /proc).
|
||||
# The harness talks to the server only over HTTP (/upload, /health, /stats) and
|
||||
# reads the on-disk sink layout; it must run on the same machine (sink dirs and
|
||||
# /proc). If /stats is missing — an older build — everything else still works and
|
||||
# the per-stage section is skipped.
|
||||
#
|
||||
# Usage:
|
||||
# julia --project=. -t auto bin/bench.jl [options]
|
||||
@@ -41,6 +51,7 @@
|
||||
# --keep-corpus don't delete the generated corpus on exit
|
||||
# --pid PID server pid for memory sampling (default: autodetect)
|
||||
# --no-mem skip memory sampling entirely
|
||||
# --no-stats skip the per-stage /stats scrape and its table
|
||||
# --sample-ms MS sink/RSS sampling interval (default: 200)
|
||||
# --timeout SEC give up after this long with no drain progress (default: 120)
|
||||
# --json PATH also write the results as JSON
|
||||
@@ -74,13 +85,14 @@ const DEFAULTS = Dict{String,Any}(
|
||||
"keep-corpus" => false,
|
||||
"pid" => nothing,
|
||||
"no-mem" => false,
|
||||
"no-stats" => false,
|
||||
"sample-ms" => 200,
|
||||
"timeout" => 120,
|
||||
"json" => nothing,
|
||||
"force" => false,
|
||||
)
|
||||
|
||||
const FLAGS = ("keep-corpus", "no-mem", "force")
|
||||
const FLAGS = ("keep-corpus", "no-mem", "no-stats", "force")
|
||||
|
||||
# Approximate RSS of a freshly started server (Lux + the loaded classifier + the
|
||||
# language detector, measured on Julia 1.12 / -t auto). Only used to notice that
|
||||
@@ -180,14 +192,35 @@ function read_rss(pid::Int)
|
||||
return (rss === nothing || hwm === nothing) ? nothing : (rss, hwm)
|
||||
end
|
||||
|
||||
"Find the running server process, or `nothing`."
|
||||
"""
|
||||
Find the running server process, or `nothing`.
|
||||
|
||||
`pgrep -f` matches against the whole command line, which catches more than the
|
||||
server: any shell launched with the command in its own argv (`sh -c 'julia …
|
||||
bin/server.jl > log'`, a `setsid`/`nohup` wrapper, even the terminal running the
|
||||
benchmark) matches the same pattern. Sampling one of those reports a few MiB of
|
||||
shell as the server's memory — a wrong answer that looks plausible, which is the
|
||||
worst kind.
|
||||
|
||||
So candidates are filtered by what each process *is* (`/proc/<pid>/comm`, the
|
||||
executable name) rather than by what its arguments say. No pattern over argv can
|
||||
make that distinction.
|
||||
"""
|
||||
function detect_pid()
|
||||
out = try
|
||||
readchomp(`pgrep -f "bin/server.jl"`)
|
||||
catch
|
||||
return nothing
|
||||
end
|
||||
pids = parse.(Int, split(out))
|
||||
candidates = parse.(Int, split(out))
|
||||
pids = filter(candidates) do pid
|
||||
comm = try
|
||||
readchomp("/proc/$pid/comm")
|
||||
catch
|
||||
return false # exited between pgrep and here
|
||||
end
|
||||
startswith(comm, "julia")
|
||||
end
|
||||
isempty(pids) && return nothing
|
||||
length(pids) > 1 && @warn "multiple server processes matched; sampling the first" pids
|
||||
return first(pids)
|
||||
@@ -210,6 +243,130 @@ function reset_peak_rss(pid::Int)::Bool
|
||||
end
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------- per-stage counters
|
||||
#
|
||||
# The server exposes monotonic counters at GET /stats (src/stats.jl). Rates are
|
||||
# ours to compute: scrape once before the run and once after, subtract, divide by
|
||||
# the elapsed *server* clock so a slow scrape doesn't distort the window.
|
||||
|
||||
"Fetch and parse GET /stats, or `nothing` if the server doesn't serve it."
|
||||
function scrape_stats(url::String)
|
||||
try
|
||||
resp = HTTP.get(string(rstrip(url, '/'), "/stats");
|
||||
status_exception = false, retry = false, readtimeout = 5)
|
||||
resp.status == 200 || return nothing
|
||||
return JSON3.read(String(resp.body))
|
||||
catch
|
||||
return nothing
|
||||
end
|
||||
end
|
||||
|
||||
"One stage's activity between two scrapes."
|
||||
struct StageDelta
|
||||
stage::Int
|
||||
name::String
|
||||
workers::Int
|
||||
completed::Int
|
||||
failed::Int
|
||||
bytes::Int
|
||||
busy::Float64 # summed handler seconds across all workers in the pool
|
||||
blocked::Float64 # of `busy`, seconds parked on a full downstream queue
|
||||
peak_depth::Int # deepest its queue got, from the sampler
|
||||
capacity::Int
|
||||
end
|
||||
|
||||
files_per_sec(d::StageDelta, window) = d.completed / max(window, 1e-9)
|
||||
mib_per_sec(d::StageDelta, window) = d.bytes / max(window, 1e-9) / 1024^2
|
||||
"Mean wall time one file spends in one worker of this stage."
|
||||
service_ms(d::StageDelta) = d.completed == 0 ? NaN :
|
||||
(d.busy - d.blocked) / d.completed * 1000
|
||||
"""
|
||||
Fraction of the pool's capacity spent doing this stage's own work.
|
||||
|
||||
Blocked time is subtracted first: a stage parked on a full downstream queue is
|
||||
waiting, not working, and leaving it in would light up every stage upstream of a
|
||||
jam as though each were the jam.
|
||||
"""
|
||||
utilization(d::StageDelta, window) =
|
||||
(d.busy - d.blocked) / max(window * d.workers, 1e-9)
|
||||
blocked_share(d::StageDelta) = d.busy <= 0 ? 0.0 : d.blocked / d.busy
|
||||
|
||||
"Subtract two scrapes into per-stage deltas, folding in sampled peak depths."
|
||||
function stage_deltas(before, after, peak_depth::Dict{Int,Int})
|
||||
out = StageDelta[]
|
||||
for (b, a) in zip(before.stages, after.stages)
|
||||
push!(out, StageDelta(a.stage, String(a.name), a.workers,
|
||||
a.completed - b.completed,
|
||||
a.failed - b.failed,
|
||||
a.bytes - b.bytes,
|
||||
a.busy_seconds - b.busy_seconds,
|
||||
a.blocked_seconds - b.blocked_seconds,
|
||||
get(peak_depth, Int(a.stage), 0),
|
||||
a.queue_capacity))
|
||||
end
|
||||
return out
|
||||
end
|
||||
|
||||
"JSON has no NaN. A stage that completed nothing has no service time, and `null`
|
||||
is the honest way to say that — writing NaN just makes JSON3 throw."
|
||||
json_num(x::Real) = isfinite(x) ? x : nothing
|
||||
|
||||
pad(s, n) = rpad(string(s), n)
|
||||
lpad_(s, n) = lpad(string(s), n)
|
||||
|
||||
"""
|
||||
Print the per-stage table and say which stage is the bottleneck.
|
||||
|
||||
The verdict reads utilization, not throughput: in a pipeline every stage
|
||||
completes the same files, so at steady state they all report nearly the same
|
||||
files/s regardless of which one is the constraint. What separates them is how
|
||||
hard each pool had to work to keep up — the bottleneck is pinned near 1.0 while
|
||||
its neighbours idle.
|
||||
"""
|
||||
function stage_report(deltas::Vector{StageDelta}, window::Float64)
|
||||
println("PER-STAGE (server counters, delta over the end-to-end window)")
|
||||
println(" stage files/s MiB/s svc ms util blocked peak queue failed")
|
||||
for d in deltas
|
||||
svc = service_ms(d)
|
||||
println(" $(d.stage) $(pad(d.name, 10)) " *
|
||||
lpad_(fmt(files_per_sec(d, window)), 8) * " " *
|
||||
lpad_(fmt(mib_per_sec(d, window)), 8) * " " *
|
||||
lpad_(isnan(svc) ? "—" : fmt(svc, 1), 8) * " " *
|
||||
lpad_(fmt(utilization(d, window)), 6) * " " *
|
||||
lpad_(fmt(blocked_share(d) * 100, 0) * "%", 7) * " " *
|
||||
lpad_("$(d.peak_depth)/$(d.capacity)", 11) * " " *
|
||||
lpad_(d.failed, 7))
|
||||
end
|
||||
|
||||
# Per-stage files/s are not comparable across rows and saying so costs one
|
||||
# line: the stages process different subsets (stage 2 only known files,
|
||||
# stage 4 only text), so a low rate can mean "little work arrived here"
|
||||
# rather than "slow". Utilization is the column that compares.
|
||||
println(" (files/s counts only files routed to that stage; svc is per-file wall time in")
|
||||
println(" one worker; util = (busy − blocked) / (window × workers))")
|
||||
|
||||
worked = filter(d -> d.completed > 0, deltas)
|
||||
if isempty(worked)
|
||||
println(" (no stage completed a file in this window)")
|
||||
return nothing
|
||||
end
|
||||
top = argmax(d -> utilization(d, window), worked)
|
||||
println(" bottleneck stage $(top.stage) ($(top.name)) at " *
|
||||
"$(fmt(utilization(top, window) * 100, 0))% utilization of " *
|
||||
"$(top.workers) worker(s)")
|
||||
if utilization(top, window) < 0.5
|
||||
println(" — but no stage is near saturated: the pipeline is " *
|
||||
"waiting on intake,\n not on itself. Raise --concurrency " *
|
||||
"or --files to load it properly.")
|
||||
end
|
||||
for d in worked
|
||||
blocked_share(d) > 0.25 && println(" ! stage $(d.stage) ($(d.name)) spent " *
|
||||
"$(fmt(blocked_share(d) * 100, 0))% of its time parked on a full downstream " *
|
||||
"queue —\n it is being held up by the stage after it, not doing that work itself.")
|
||||
end
|
||||
return nothing
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------------------ corpus
|
||||
|
||||
const WORDS = split("the quick brown fox jumps over a lazy dog while parsing " *
|
||||
@@ -385,10 +542,16 @@ function main(argv)
|
||||
r = read_rss(pid)
|
||||
r === nothing && (@warn "cannot read /proc/$pid/status; skipping memory"; pid = nothing)
|
||||
if pid !== nothing
|
||||
baseline_rss = r[1]
|
||||
peak_reset = reset_peak_rss(pid)
|
||||
peak_reset || @warn "could not reset the peak-RSS counter (/proc/$pid/clear_refs); " *
|
||||
"reporting sampled RSS only"
|
||||
# Baseline is read *after* the reset, not before. VmHWM restarts
|
||||
# from whatever RSS is at the moment of the reset, so a baseline
|
||||
# sampled earlier is measured against a different origin — and if
|
||||
# the GC hands memory back in between, the run reports negative
|
||||
# growth, which is nonsense on its face.
|
||||
r2 = read_rss(pid)
|
||||
baseline_rss = r2 === nothing ? r[1] : r2[1]
|
||||
end
|
||||
end
|
||||
|
||||
@@ -400,10 +563,18 @@ function main(argv)
|
||||
base_sinks = counts(sinks)
|
||||
interval = opts["sample-ms"] / 1000
|
||||
|
||||
# --- sampler: RSS curve + stage depths, for bottleneck attribution.
|
||||
# --- per-stage counters: the "before" half of the delta.
|
||||
stats_before = opts["no-stats"] ? nothing : scrape_stats(url)
|
||||
if stats_before === nothing && !opts["no-stats"]
|
||||
@warn "no /stats endpoint on this server; skipping the per-stage table " *
|
||||
"(the server predates src/stats.jl)"
|
||||
end
|
||||
|
||||
# --- sampler: RSS curve, stage dir depths, and queue depths.
|
||||
stop = Threads.Atomic{Bool}(false)
|
||||
rss_samples = Float64[]
|
||||
depth_max = Dict(k => 0 for k in keys(stages))
|
||||
queue_peak = Dict{Int,Int}()
|
||||
sampler = Threads.@spawn begin
|
||||
while !stop[]
|
||||
if pid !== nothing
|
||||
@@ -414,6 +585,17 @@ function main(argv)
|
||||
for k in keys(d)
|
||||
depth_max[k] = max(depth_max[k], getfield(d, k))
|
||||
end
|
||||
# Queue depth, unlike directory depth, can't be missed by a slow
|
||||
# sample in the same way — a file's *reference* sits in the queue
|
||||
# for the whole time it waits — so this is the depth the stage
|
||||
# table reports.
|
||||
if stats_before !== nothing
|
||||
s = scrape_stats(url)
|
||||
s === nothing || for st in s.stages
|
||||
k = Int(st.stage)
|
||||
queue_peak[k] = max(get(queue_peak, k, 0), Int(st.queue_depth))
|
||||
end
|
||||
end
|
||||
sleep(interval)
|
||||
end
|
||||
end
|
||||
@@ -455,12 +637,25 @@ function main(argv)
|
||||
break
|
||||
end
|
||||
end
|
||||
# Scrape before stopping the sampler, so the window closes as near the
|
||||
# last completion as we can manage.
|
||||
stats_after = stats_before === nothing ? nothing : scrape_stats(url)
|
||||
stop[] = true
|
||||
wait(sampler)
|
||||
|
||||
sink_delta = deltas(counts(sinks), base_sinks)
|
||||
e2e_secs = t_last_completion - t_start
|
||||
|
||||
# The stage window is the server's own clock across the two scrapes, not
|
||||
# e2e_secs: it starts a scrape earlier and ends a scrape later, and using
|
||||
# our wall time against its counters would misattribute the difference.
|
||||
stage_window, stage_delta = if stats_after === nothing
|
||||
(0.0, StageDelta[])
|
||||
else
|
||||
(Float64(stats_after.now - stats_before.now),
|
||||
stage_deltas(stats_before, stats_after, queue_peak))
|
||||
end
|
||||
|
||||
final_rss = pid === nothing ? nothing : read_rss(pid)
|
||||
peak_rss = if final_rss !== nothing && peak_reset
|
||||
final_rss[2] # kernel VmHWM: catches spikes between samples
|
||||
@@ -493,19 +688,33 @@ function main(argv)
|
||||
"$(fmt(corpusbytes / max(e2e_secs, 1e-9) / 1024^2)) MiB/s")
|
||||
println(" sinks done $(sink_delta.done) text_done $(sink_delta.text_done) " *
|
||||
"binary $(sink_delta.binary) failed $(sink_delta.failed)")
|
||||
println(" peak depth spool $(depth_max[:spool]) known $(depth_max[:known]) " *
|
||||
println(" peak dir depth spool $(depth_max[:spool]) known $(depth_max[:known]) " *
|
||||
"unknown $(depth_max[:unknown]) text $(depth_max[:text])")
|
||||
println(" (the stage that backs up is the bottleneck)")
|
||||
println()
|
||||
if !isempty(stage_delta)
|
||||
stage_report(stage_delta, stage_window)
|
||||
println()
|
||||
end
|
||||
if peak_rss !== nothing
|
||||
println("SERVER MEMORY (pid $pid)")
|
||||
println(" baseline RSS $(human(baseline_rss))")
|
||||
println(" peak RSS $(human(peak_rss))" *
|
||||
(peak_reset ? " (kernel VmHWM, reset at start)" : " (sampled — may miss spikes)"))
|
||||
println(" growth $(human(peak_rss - baseline_rss))")
|
||||
println(" per in-flight $(human((peak_rss - baseline_rss) / opts["concurrency"])) " *
|
||||
"at $(human(corpusbytes / length(paths))) avg file size")
|
||||
println(" (should not grow with file size — intake streams to disk)")
|
||||
growth = peak_rss - baseline_rss
|
||||
if growth <= 0
|
||||
# RSS never got back to where it started, so the run's own cost
|
||||
# is below the noise floor of the server settling after startup.
|
||||
# Printing a negative "growth" would invite reading a memory
|
||||
# *saving* into what is really "too small to measure here".
|
||||
println(" growth none measurable (peak never exceeded the baseline)")
|
||||
println(" The baseline was still falling when we sampled it — give the")
|
||||
println(" server ~30s to settle after startup for a comparable figure.")
|
||||
else
|
||||
println(" growth $(human(growth))")
|
||||
println(" per in-flight $(human(growth / opts["concurrency"])) " *
|
||||
"at $(human(corpusbytes / length(paths))) avg file size")
|
||||
println(" (should not grow with file size — intake streams to disk)")
|
||||
end
|
||||
# Julia's GC returns memory to the OS lazily, so a second run on the
|
||||
# same process starts from an inflated baseline and under-reports
|
||||
# growth. Absolute peak is the number to trust across runs.
|
||||
@@ -531,13 +740,24 @@ function main(argv)
|
||||
avg_file_bytes = corpusbytes / length(paths),
|
||||
intake = (; accepted, n_202, n_503, n_err, seconds = intake_secs,
|
||||
files_per_sec = accepted / max(intake_secs, 1e-9),
|
||||
p50_ms = percentile(lat, 0.5) * 1000,
|
||||
p95_ms = percentile(lat, 0.95) * 1000,
|
||||
max_ms = percentile(lat, 1.0) * 1000),
|
||||
p50_ms = json_num(percentile(lat, 0.5) * 1000),
|
||||
p95_ms = json_num(percentile(lat, 0.95) * 1000),
|
||||
max_ms = json_num(percentile(lat, 1.0) * 1000)),
|
||||
end_to_end = (; completed, seconds = e2e_secs, timed_out,
|
||||
files_per_sec = completed / max(e2e_secs, 1e-9),
|
||||
mib_per_sec = corpusbytes / max(e2e_secs, 1e-9) / 1024^2,
|
||||
sinks = sink_delta, peak_stage_depth = depth_max),
|
||||
stages = [(; stage = d.stage, name = d.name, workers = d.workers,
|
||||
completed = d.completed, failed = d.failed, bytes = d.bytes,
|
||||
busy_seconds = d.busy, blocked_seconds = d.blocked,
|
||||
window_seconds = stage_window,
|
||||
files_per_sec = files_per_sec(d, stage_window),
|
||||
mib_per_sec = mib_per_sec(d, stage_window),
|
||||
service_ms = json_num(service_ms(d)),
|
||||
utilization = utilization(d, stage_window),
|
||||
blocked_share = blocked_share(d),
|
||||
peak_queue_depth = d.peak_depth,
|
||||
queue_capacity = d.capacity) for d in stage_delta],
|
||||
memory = (; pid, baseline_rss, peak_rss, peak_is_kernel_hwm = peak_reset,
|
||||
growth = peak_rss === nothing ? nothing : peak_rss - baseline_rss,
|
||||
samples = rss_samples),
|
||||
|
||||
427
bin/bench_model.jl
Executable file
427
bin/bench_model.jl
Executable file
@@ -0,0 +1,427 @@
|
||||
#!/usr/bin/env julia
|
||||
#
|
||||
# bench_model.jl — microbenchmark the classifier in isolation, with no server,
|
||||
# no queue, and no disk in the way.
|
||||
#
|
||||
# bin/bench.jl measures the *pipeline*: it reports stage 1 as one number, the
|
||||
# wall time of `handle_classify_job`, which is feature reads + inference + a
|
||||
# rename + a log line, under whatever thread contention the other three pools are
|
||||
# creating. That number is the right one for capacity planning and the wrong one
|
||||
# for answering "is the model slow?". This script answers that question by taking
|
||||
# the model apart:
|
||||
#
|
||||
# read_features open, read 16 bytes, seek, read 16 bytes, scale
|
||||
# Lux.apply the network itself, on a feature vector already in memory
|
||||
# classify both together — what stage 1 actually calls per file
|
||||
#
|
||||
# Three properties are worth checking beyond the raw per-file cost:
|
||||
#
|
||||
# * Feature reads should be flat in file size. read_features seeks to the tail
|
||||
# rather than slurping, so a 1 GiB file should cost the same as a 1 KiB one.
|
||||
# (This is the same claim bin/bench.jl makes about memory, on the CPU axis.)
|
||||
# * Batching should be much cheaper per file. A 32x1 matmul wastes most of a
|
||||
# BLAS call; if batch-64 inference is many times cheaper per file, that is
|
||||
# the headroom a batching stage-1 would buy — worth knowing before building
|
||||
# one, since today the pipeline classifies strictly one file at a time.
|
||||
# * Inference should scale across threads. `Classifier` is shared read-only by
|
||||
# the whole stage-1 pool on the claim that Lux inference is pure. If per-file
|
||||
# cost degrades as tasks are added, that claim holds but BLAS threading is
|
||||
# fighting the worker pool, and stage-1 workers are contending, not scaling.
|
||||
#
|
||||
# Usage:
|
||||
# julia --project=. -t auto bin/bench_model.jl [options]
|
||||
#
|
||||
# --model PATH classifier artifact (default: $FS_MODEL_PATH or model/classifier.jld2)
|
||||
# --reps N inference calls per timed trial (default: 20000)
|
||||
# --trials N timed trials; the minimum is reported (default: 5)
|
||||
# --batches LIST batch sizes to sweep, comma-separated (default: 1,8,64,512)
|
||||
# --sizes LIST file sizes for the read_features sweep (default: 1k,64k,4m,256m)
|
||||
# --no-threads skip the thread-scaling sweep
|
||||
# --json PATH also write the results as JSON
|
||||
#
|
||||
# Reported times are the *minimum* over trials: for a microbenchmark the floor is
|
||||
# the signal and everything above it is scheduler and GC noise.
|
||||
|
||||
using JSON3
|
||||
using Random
|
||||
using Statistics
|
||||
using Printf
|
||||
|
||||
# The script is run directly, not as part of the package, so pull in exactly the
|
||||
# pieces the classifier needs. `Lux`/`JLD2` first — model.jl and classify.jl both
|
||||
# assume the including scope already has them (see the note at the top of model.jl).
|
||||
using Lux
|
||||
using JLD2
|
||||
using LinearAlgebra
|
||||
|
||||
const SRC = joinpath(dirname(@__DIR__), "src")
|
||||
include(joinpath(SRC, "model.jl"))
|
||||
include(joinpath(SRC, "classify.jl"))
|
||||
|
||||
# ---------------------------------------------------------------- option parsing
|
||||
|
||||
const DEFAULTS = Dict{String,Any}(
|
||||
"model" => get(ENV, "FS_MODEL_PATH", "model/classifier.jld2"),
|
||||
"reps" => 20_000,
|
||||
"trials" => 5,
|
||||
"batches" => "1,8,64,512",
|
||||
"sizes" => "1k,64k,4m,256m",
|
||||
"no-threads" => false,
|
||||
"json" => nothing,
|
||||
)
|
||||
|
||||
const FLAGS = ("no-threads",)
|
||||
|
||||
function parse_size(s::AbstractString)::Int
|
||||
m = match(r"^(\d+(?:\.\d+)?)\s*([kKmMgG]?)[bB]?$", strip(s))
|
||||
m === nothing && error("bad size: $s (expected e.g. 512, 64k, 8m, 1g)")
|
||||
mult = Dict('k' => 1024, 'm' => 1024^2, 'g' => 1024^3)
|
||||
scale = isempty(m[2]) ? 1 : mult[lowercase(m[2])[1]]
|
||||
return round(Int, parse(Float64, m[1]) * scale)
|
||||
end
|
||||
|
||||
function parse_args(argv)
|
||||
opts = copy(DEFAULTS)
|
||||
i = 1
|
||||
while i <= length(argv)
|
||||
a = argv[i]
|
||||
startswith(a, "--") || error("unexpected argument: $a")
|
||||
key = a[3:end]
|
||||
haskey(opts, key) || error("unknown option: $a")
|
||||
if key in FLAGS
|
||||
opts[key] = true; i += 1; continue
|
||||
end
|
||||
i + 1 <= length(argv) || error("option --$key needs a value")
|
||||
opts[key] = key in ("reps", "trials") ? parse(Int, argv[i+1]) : argv[i+1]
|
||||
i += 2
|
||||
end
|
||||
return opts
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------------- measurement
|
||||
|
||||
# Every timed loop stores its result here. Without a visible side effect the
|
||||
# compiler is free to hoist a pure call out of the loop and we would be timing an
|
||||
# empty `for`.
|
||||
const SINK = Ref{Any}(nothing)
|
||||
|
||||
"""
|
||||
measure(f, reps; trials) -> (ns_per_op, bytes_per_op)
|
||||
|
||||
Time `f` over `reps` calls, `trials` times, and report the fastest trial.
|
||||
|
||||
The first call is thrown away: it pays Julia's JIT compilation, which on a
|
||||
function this small is orders of magnitude more than the thing being measured.
|
||||
"""
|
||||
function measure(f, reps::Int; trials::Int = 5)
|
||||
SINK[] = f() # warm up (compile), and keep the result
|
||||
best = Inf
|
||||
for _ in 1:trials
|
||||
GC.gc()
|
||||
t0 = time_ns()
|
||||
for _ in 1:reps
|
||||
SINK[] = f()
|
||||
end
|
||||
best = min(best, (time_ns() - t0) / reps)
|
||||
end
|
||||
bytes = @allocated(SINK[] = f()) # one call, after warmup
|
||||
return (Float64(best), Float64(bytes))
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------------- formatting
|
||||
|
||||
function human_time(ns::Real)
|
||||
ns < 1_000 && return @sprintf("%.0f ns", ns)
|
||||
ns < 1_000_000 && return @sprintf("%.2f µs", ns / 1e3)
|
||||
ns < 1e9 && return @sprintf("%.2f ms", ns / 1e6)
|
||||
return @sprintf("%.2f s", ns / 1e9)
|
||||
end
|
||||
|
||||
human_bytes(b::Real) = b < 1024 ? @sprintf("%.0f B", b) :
|
||||
b < 1024^2 ? @sprintf("%.1f KiB", b / 1024) :
|
||||
@sprintf("%.1f MiB", b / 1024^2)
|
||||
|
||||
rate(ns::Real) = 1e9 / max(ns, 1e-9) # calls per second
|
||||
|
||||
function human_rate(r::Real)
|
||||
r >= 1e6 && return @sprintf("%.2fM/s", r / 1e6)
|
||||
r >= 1e3 && return @sprintf("%.1fk/s", r / 1e3)
|
||||
return @sprintf("%.0f/s", r)
|
||||
end
|
||||
|
||||
fmt2(x::Real) = @sprintf("%.2f", x)
|
||||
|
||||
# ------------------------------------------------------------- scaling control
|
||||
|
||||
"""
|
||||
control_kernel(x) -> Float64
|
||||
|
||||
Pure arithmetic, no allocation, no library call — deliberately dependent
|
||||
(each step needs the last) so the compiler can't vectorize it away, and sized to
|
||||
land in the same microsecond neighbourhood as one `Lux.apply`.
|
||||
"""
|
||||
function control_kernel(x::Float64)
|
||||
a = x
|
||||
@inbounds for i in 1:600
|
||||
a = sqrt(a + i)
|
||||
end
|
||||
return a
|
||||
end
|
||||
|
||||
"""
|
||||
control_scaling(trials) -> Vector
|
||||
|
||||
Run `control_kernel` over the same task counts as the model sweep. This is the
|
||||
machine's own ceiling for perfectly parallel work: if the control scales and the
|
||||
model doesn't, the shortfall is the model's, and no amount of `FS_WORKERS` will
|
||||
recover it.
|
||||
"""
|
||||
function control_scaling(trials::Int)
|
||||
rows = []
|
||||
base = 0.0
|
||||
per_task = 200_000
|
||||
for k in unique([1; 2; 4; 8; Threads.nthreads()])
|
||||
k > Threads.nthreads() && continue
|
||||
best = Inf
|
||||
for _ in 1:trials
|
||||
GC.gc()
|
||||
t0 = time_ns()
|
||||
@sync for _ in 1:k
|
||||
Threads.@spawn begin
|
||||
local acc = 0.0
|
||||
for i in 1:per_task
|
||||
acc += control_kernel(i % 97 + 1.0)
|
||||
end
|
||||
SINK[] = acc
|
||||
end
|
||||
end
|
||||
best = min(best, Float64(time_ns() - t0))
|
||||
end
|
||||
r = k * per_task / (best / 1e9)
|
||||
k == 1 && (base = r)
|
||||
push!(rows, (; tasks = k, ops_per_sec = r, speedup = r / base))
|
||||
end
|
||||
return rows
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------------------- corpus
|
||||
|
||||
"""
|
||||
Write a file of exactly `size` random bytes, in bounded chunks.
|
||||
|
||||
Content is random rather than zeros so the classifier sees a realistic input —
|
||||
and so the filesystem can't cheat with a sparse file, which would make the tail
|
||||
`seek` unrepresentatively fast.
|
||||
"""
|
||||
function write_file(path::AbstractString, size::Int, rng)
|
||||
chunk = 1024 * 1024
|
||||
open(path, "w") do io
|
||||
remaining = size
|
||||
while remaining > 0
|
||||
n = min(chunk, remaining)
|
||||
write(io, rand(rng, UInt8, n))
|
||||
remaining -= n
|
||||
end
|
||||
end
|
||||
return path
|
||||
end
|
||||
|
||||
# ------------------------------------------------------------------------- main
|
||||
|
||||
function main(argv)
|
||||
opts = parse_args(argv)
|
||||
modelpath = String(opts["model"])
|
||||
isfile(modelpath) || (println(stderr, "model artifact not found: $modelpath"); return 1)
|
||||
|
||||
reps, trials = opts["reps"], opts["trials"]
|
||||
batches = [parse(Int, s) for s in split(String(opts["batches"]), ",")]
|
||||
sizes = [parse_size(s) for s in split(String(opts["sizes"]), ",")]
|
||||
|
||||
clf = load_classifier(modelpath)
|
||||
println("model $modelpath")
|
||||
println("architecture $(FEATURE_DIM) → 64 → 16 → 2 (Dense/relu, raw logits)")
|
||||
println("julia threads $(Threads.nthreads()) BLAS threads $(BLAS.get_num_threads())")
|
||||
println("timing min of $trials trials × $reps reps")
|
||||
println("=" ^ 72)
|
||||
|
||||
results = Dict{String,Any}()
|
||||
|
||||
# --- 1. inference alone, one file at a time: the number the pipeline pays.
|
||||
x1 = rand(Float32, FEATURE_DIM, 1)
|
||||
infer_ns, infer_bytes = measure(reps; trials) do
|
||||
Lux.apply(clf.model, x1, clf.ps, clf.st)
|
||||
end
|
||||
println()
|
||||
println("INFERENCE (Lux.apply, batch 1 — features already in memory)")
|
||||
println(" per call $(human_time(infer_ns)) $(human_rate(rate(infer_ns)))")
|
||||
println(" allocations $(human_bytes(infer_bytes)) per call")
|
||||
results["inference_batch1"] = (; ns = infer_ns, bytes = infer_bytes, per_sec = rate(infer_ns))
|
||||
|
||||
# --- 2. batching: how much of that is per-call overhead rather than math?
|
||||
println()
|
||||
println("INFERENCE BATCHED (same net, N files per apply)")
|
||||
println(" batch per batch per file files/s speedup")
|
||||
batch_rows = []
|
||||
for b in batches
|
||||
xb = rand(Float32, FEATURE_DIM, b)
|
||||
ns, _ = measure(max(1, reps ÷ b); trials) do
|
||||
Lux.apply(clf.model, xb, clf.ps, clf.st)
|
||||
end
|
||||
per_file = ns / b
|
||||
@printf(" %8d %13s %12s %13s %7.1fx\n",
|
||||
b, human_time(ns), human_time(per_file),
|
||||
human_rate(rate(per_file)), infer_ns / per_file)
|
||||
push!(batch_rows, (; batch = b, ns_per_batch = ns, ns_per_file = per_file,
|
||||
files_per_sec = rate(per_file), speedup = infer_ns / per_file))
|
||||
end
|
||||
results["batched"] = batch_rows
|
||||
println(" (a large speedup is headroom a batching stage 1 could claim; the pipeline")
|
||||
println(" classifies one file per job today, so it pays the batch-1 row above)")
|
||||
|
||||
# --- 3. feature reads: should be flat in file size (seek, not slurp).
|
||||
println()
|
||||
println("FEATURE READS (read_features: 16 head + 16 tail bytes, scaled)")
|
||||
println(" file size per call calls/s allocations")
|
||||
read_rows = []
|
||||
dir = mktempdir(; prefix = "fsmodel-")
|
||||
try
|
||||
rng = MersenneTwister(1234)
|
||||
for sz in sizes
|
||||
path = write_file(joinpath(dir, "f-$sz.bin"), sz, rng)
|
||||
# Fewer reps for the big files: this touches the page cache, and the
|
||||
# point is the shape of the curve, not another digit of precision.
|
||||
r = max(200, reps ÷ 20)
|
||||
ns, bytes = measure(() -> read_features(path), r; trials)
|
||||
@printf(" %13s %14s %14s %14s\n",
|
||||
human_bytes(sz), human_time(ns), human_rate(rate(ns)), human_bytes(bytes))
|
||||
push!(read_rows, (; size_bytes = sz, ns, bytes, per_sec = rate(ns)))
|
||||
end
|
||||
finally
|
||||
rm(dir; recursive = true, force = true)
|
||||
end
|
||||
results["read_features"] = read_rows
|
||||
flat = length(read_rows) > 1 ?
|
||||
maximum(r.ns for r in read_rows) / minimum(r.ns for r in read_rows) : 1.0
|
||||
@printf(" spread across a %.0fx size range: %.1fx — %s\n",
|
||||
maximum(sizes) / minimum(sizes), flat,
|
||||
flat < 3 ? "flat, as designed (it seeks to the tail)" :
|
||||
"NOT flat: something is reading more than 32 bytes")
|
||||
|
||||
# --- 4. classify(): what stage 1 calls, I/O and inference together.
|
||||
println()
|
||||
println("CLASSIFY (read_features + Lux.apply — one whole stage-1 file)")
|
||||
dir2 = mktempdir(; prefix = "fsmodel-")
|
||||
classify_ns = 0.0
|
||||
try
|
||||
path = write_file(joinpath(dir2, "sample.bin"), 64 * 1024, MersenneTwister(7))
|
||||
classify_ns, classify_bytes = measure(() -> classify(clf, path), max(200, reps ÷ 20); trials)
|
||||
println(" per file $(human_time(classify_ns)) $(human_rate(rate(classify_ns)))")
|
||||
println(" allocations $(human_bytes(classify_bytes)) per file")
|
||||
@printf(" split %.0f%% feature read, %.0f%% inference\n",
|
||||
100 * (classify_ns - infer_ns) / classify_ns, 100 * infer_ns / classify_ns)
|
||||
results["classify"] = (; ns = classify_ns, bytes = classify_bytes, per_sec = rate(classify_ns))
|
||||
finally
|
||||
rm(dir2; recursive = true, force = true)
|
||||
end
|
||||
|
||||
# --- 5. thread scaling: does the shared read-only Classifier actually scale?
|
||||
if !opts["no-threads"] && Threads.nthreads() > 1
|
||||
println()
|
||||
println("THREAD SCALING (concurrent Lux.apply on the one shared Classifier)")
|
||||
println(" tasks files/s per file speedup efficiency GC")
|
||||
thread_rows = []
|
||||
base = 0.0
|
||||
for k in unique([1; 2; 4; 8; Threads.nthreads()])
|
||||
k > Threads.nthreads() && continue
|
||||
# Work per task is held *constant* as tasks are added, so total work
|
||||
# scales with `k`. Splitting a fixed total instead would shrink each
|
||||
# task as the pool grows until `@spawn`/`@sync` overhead dominated,
|
||||
# and the resulting curve would show a collapse that is the
|
||||
# measurement's fault rather than the model's.
|
||||
per_task = max(reps, 20_000)
|
||||
# Each task gets its own input so we measure the model, not cache
|
||||
# line ping-pong on a shared buffer.
|
||||
xs = [rand(Float32, FEATURE_DIM, 1) for _ in 1:k]
|
||||
best, best_gc = Inf, 0.0
|
||||
for _ in 1:trials
|
||||
GC.gc()
|
||||
# Julia's GC stops the world, so it is the one cost that cannot
|
||||
# be parallelized away: measuring its share here is what turns a
|
||||
# bad efficiency number into a diagnosis (see the note below).
|
||||
gc0 = Base.gc_num().total_time
|
||||
t0 = time_ns()
|
||||
@sync for t in 1:k
|
||||
Threads.@spawn begin
|
||||
local acc = 0.0f0
|
||||
for _ in 1:per_task
|
||||
y, _ = Lux.apply(clf.model, xs[t], clf.ps, clf.st)
|
||||
acc += y[1] # consume the result
|
||||
end
|
||||
SINK[] = acc
|
||||
end
|
||||
end
|
||||
elapsed = Float64(time_ns() - t0)
|
||||
if elapsed < best
|
||||
best = elapsed
|
||||
best_gc = Float64(Base.gc_num().total_time - gc0)
|
||||
end
|
||||
end
|
||||
files = k * per_task
|
||||
fps = files / (best / 1e9)
|
||||
k == 1 && (base = fps)
|
||||
@printf(" %8d %13s %11s %7.2fx %10.0f%% %5.0f%%\n",
|
||||
k, human_rate(fps), human_time(best / files), fps / base,
|
||||
100 * fps / base / k, 100 * best_gc / best)
|
||||
push!(thread_rows, (; tasks = k, files_per_sec = fps,
|
||||
ns_per_file = best / files, speedup = fps / base,
|
||||
gc_fraction = best_gc / best))
|
||||
end
|
||||
results["thread_scaling"] = thread_rows
|
||||
|
||||
# A poor scaling curve has two possible authors — the model or the box —
|
||||
# and the table alone can't tell them apart. So run the same sweep on a
|
||||
# kernel that is pure arithmetic with no allocation and no library
|
||||
# underneath: whatever *it* achieves is this machine's ceiling for
|
||||
# embarrassingly parallel work, and the gap between the two curves is
|
||||
# the part that belongs to Lux.apply.
|
||||
ctrl = control_scaling(trials)
|
||||
results["control_scaling"] = ctrl
|
||||
top = last(ctrl)
|
||||
println(" control pure-compute kernel, same sweep: " *
|
||||
"$(fmt2(top.speedup))x at $(top.tasks) tasks " *
|
||||
"($(round(Int, 100 * top.speedup / top.tasks))% efficiency)")
|
||||
model_top = last(thread_rows)
|
||||
if model_top.speedup < 0.6 * top.speedup
|
||||
println(" → the machine parallelizes; Lux.apply does not. Stage-1")
|
||||
println(" workers past ~4 buy little, whatever FS_WORKERS says.")
|
||||
else
|
||||
println(" → inference tracks the machine's own scaling ceiling.")
|
||||
end
|
||||
gc_top = maximum(r.gc_fraction for r in thread_rows)
|
||||
gc_top > 0.15 && println(" ! GC is $(round(Int, 100 * gc_top))% of the " *
|
||||
"worst case: apply allocates per call, and\n" *
|
||||
" collection stops every thread.")
|
||||
end
|
||||
|
||||
println()
|
||||
println("=" ^ 72)
|
||||
println("Stage 1's cost per file in bin/bench.jl is this classify() figure plus a")
|
||||
println("rename, a log line, and whatever contention the other three pools create.")
|
||||
println("A large gap between the two is pipeline overhead, not the model.")
|
||||
|
||||
if opts["json"] !== nothing
|
||||
results["meta"] = (; model = modelpath, feature_dim = FEATURE_DIM,
|
||||
julia_threads = Threads.nthreads(),
|
||||
blas_threads = BLAS.get_num_threads(),
|
||||
reps, trials)
|
||||
open(String(opts["json"]), "w") do io
|
||||
JSON3.write(io, results)
|
||||
end
|
||||
println("\nwrote $(opts["json"])")
|
||||
end
|
||||
return 0
|
||||
end
|
||||
|
||||
if abspath(PROGRAM_FILE) == @__FILE__
|
||||
exit(main(ARGS))
|
||||
end
|
||||
Reference in New Issue
Block a user