jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial text cleanup

Browse Source
This commit is contained in:
Jeffrey Ward
2026-08-19 12:55:19 -04:00
parent 819a4cce7a
commit cc84d70d52
26 changed files with 454 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
bin/bench_stage2.jl
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env julia
#
# bench_stage2.jl — take stage 2 apart and find the slowest component.
# bench_stage2.jl: take stage 2 apart and find the slowest component.
#
# bin/bench.jl reports stage 2 as a single number (its throughput and worker
# utilization under whole-pipeline contention). It doesn't say *which part* of
@@ -27,13 +27,13 @@
#
# * The corpus must be real files. exiftool's cost depends on what it finds;
# random bytes exit early and would understate the stage by a lot. The
# default corpus is `data/done` — files that already went through stage 2 on
# this machine — copied back into a scratch spool/ dir.
# default corpus is `data/done`, files that already went through stage 2 on
# this machine, copied back into a scratch spool/ dir.
# * Two rows price the *alternatives* to one-fork-per-file, because if the
# fork dominates then the only fixes are to stop paying it per file:
# `exiftool (batched Nx)` runs the whole corpus through one process, and
# `exiftool (-stay_open)` keeps a single process alive and feeds it one file
# at a time over a pipe — the shape a streaming pipeline could actually use.
# at a time over a pipe: the shape a streaming pipeline could actually use.
# Both are measured, not assumed.
# * `run_with_timeout` gets its own row *next to* a bare `Base.run` of the same
# command. The difference is what the watchdog costs, and its polling loop
@@ -48,7 +48,7 @@
# julia --project=. -t auto bin/bench_stage2.jl [options]
#
# --files N corpus files per timed pass (default: 48). The concurrency
# sweep wants more than the component rows do — with a single
# sweep wants more than the component rows do. With a single
# 2 s file in the corpus, 48 files can't show more than ~4x no
# matter how many workers run, so pass --files 150 when the
# question is scaling.
@@ -121,7 +121,7 @@ const SINK = Ref{Any}(nothing)
Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
(`pass` returns the number of operations it performed). `prepare()` runs before
each pass and is *not* timed — that is where a consuming benchmark puts the file
each pass and is *not* timed: that is where a consuming benchmark puts the file
back where it started. `pass` comes first so callers can pass it as a `do` block.
The first pass is thrown away: it pays Julia's JIT compilation, which on calls
@@ -174,12 +174,12 @@ end
make_corpus(cfg, corpus_dir, n) -> Vector{Job}
Copy up to `n` real files from `corpus_dir` into `spool/` and build the `Job`
references a stage-2 worker would dequeue for them — the exact input
references a stage-2 worker would dequeue for them: the exact input
`handle_known_job` sees.
Real files, not generated ones: exiftool's cost is a function of what it can
parse, and a file of random bytes bails out early enough to understate the stage
by an order of magnitude. `.meta.json` sidecars are skipped — they are stage-2
by an order of magnitude. `.meta.json` sidecars are skipped, since they are stage-2
*output*, and enriching them would measure the wrong population.
"""
function make_corpus(cfg::FS.Config, corpus_dir::AbstractString, n::Int)
@@ -211,7 +211,7 @@ end
Put every corpus file back in `spool/`, wherever the last pass left it (done/ or
already home), and delete any sidecar it produced. This is the untimed `prepare`
step for benchmarks that consume their input by committing it — stage 2 still
step for benchmarks that consume their input by committing it. Stage 2 still
moves, because its move is the terminal commit, not an inter-stage hop.
"""
function respool!(cfg::FS.Config, jobs::Vector{FS.Job})
@@ -241,13 +241,13 @@ end
Run `f` under one of the loggers the cost of logging is bracketed by:
* `:null` — `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` — `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` — `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written
for. Stage 2's per-file line is `@info`, not `@debug`, so this
row is what the deployed server actually pays.
* `:null` `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written
for. Stage 2's per-file line is `@info`, not `@debug`, so this
row is what the deployed server actually pays.
"""
function with_logger_named(f, which::Symbol, path::AbstractString)
if which === :null
@@ -286,7 +286,7 @@ end
Run the whole corpus through *one* `exiftool` process and divide by the file
count. This is the floor for "what does exiftool cost if you stop paying the
interpreter startup per file" — the fork, the Perl boot and the module loads are
interpreter startup per file": the fork, the Perl boot and the module loads are
paid once for the batch instead of once per file.
"""
function batched_ns(paths::Vector{String}, trials::Int)
@@ -301,8 +301,8 @@ end
Feed files one at a time to a single long-lived `exiftool -stay_open True -@ -`
process over a pipe, reading its `{ready}` sentinel after each. Unlike the
batched row this preserves the pipeline's actual shape — one file in, one result
out, arriving whenever it arrives — so it prices the realistic fix rather than
batched row this preserves the pipeline's actual shape (one file in, one result
out, arriving whenever it arrives) so it prices the realistic fix rather than
an unrealistic one.
"""
function stay_open_ns(paths::Vector{String}, trials::Int)
@@ -439,7 +439,7 @@ function component_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
# One pass over the real sidecar population, not `reps` of them. Two reasons,
# and the first is a correctness trap: thousands of back-to-back fsyncs
# saturate the device's write cache and each one starts waiting on the
# queue, which reported this row at 16 ms/file — eight times the whole
# queue, which reported this row at 16 ms/file: eight times the whole
# `commit_enriched!` that contains it. The real stage fsyncs once per file
# with ~160 ms of exiftool between, and never queues that way. Second, real
# sidecars vary hugely in size (a zip's raw dump dwarfs a jpeg's), so the
@@ -546,7 +546,7 @@ machine's cores, not about Julia.
Workers pull from a shared atomic counter rather than taking a contiguous slice.
That matches the server (its pool pulls from one queue), and it matters here in a
way it doesn't for stage 1: per-file exiftool time spans two orders of magnitude
on a real corpus — a single 2 s archive among 48 files — so a static split leaves
on a real corpus (a single 2 s archive among 48 files), so a static split leaves
whichever worker drew it running alone while the rest idle, and the sweep would
report a scaling ceiling that is really just load imbalance.
"""