jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial text cleanup

Browse Source
This commit is contained in:
Jeffrey Ward
2026-08-19 12:55:19 -04:00
parent 819a4cce7a
commit cc84d70d52
26 changed files with 454 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
bin/bench_stage1.jl
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env julia
#
# bench_stage1.jl — take stage 1 apart and find the slowest component.
# bench_stage1.jl: take stage 1 apart and find the slowest component.
#
# bin/bench.jl reports stage 1 as a single number (the wall time of
# `handle_classify_job` under whole-pipeline contention) and bin/bench_model.jl
@@ -17,7 +17,7 @@
# `handle_classify_job` end to end so the parts can be checked against the whole.
# The two knobs that most change the answer get their own sweeps:
#
# * Logger. The server runs `FlushLogger(ConsoleLogger(stderr))` — it formats
# * Logger. The server runs `FlushLogger(ConsoleLogger(stderr))`, which formats
# and flushes every message. Under redirect (a log file, journald) that is a
# syscall per line, two lines per file, on the hot path. We time the handler
# under a null logger, a formatting-but-discarding logger, and the real
@@ -112,7 +112,7 @@ const SINK = Ref{Any}(nothing)
Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
(`pass` returns the number of operations it performed). `prepare()` runs before
each pass and is *not* timed — that is where a benchmark resets whatever its
each pass and is *not* timed: that is where a benchmark resets whatever its
last pass consumed (today: the queues). `pass` comes first so callers can pass it as a `do` block.
The first pass is thrown away: it pays Julia's JIT compilation, which on calls
@@ -164,7 +164,7 @@ end
"""
Write a file of exactly `size` random bytes, in bounded chunks.
Content is random rather than zeros so the classifier sees a realistic input —
Content is random rather than zeros so the classifier sees a realistic input,
and so the filesystem can't cheat with a sparse file, which would make the tail
`seek` unrepresentatively fast.
"""
@@ -185,7 +185,7 @@ end
make_corpus(cfg, n, size, rng) -> Vector{Job}
Create `n` spooled files and the `Job` references a stage-1 worker would dequeue
for them — the exact input `handle_classify_job` sees.
for them: the exact input `handle_classify_job` sees.
"""
function make_corpus(cfg::FS.Config, n::Int, size::Int, rng)
jobs = FS.Job[]
@@ -218,15 +218,15 @@ end
Run `f` under one of the three loggers the cost of logging is bracketed by:
* `:null` — `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` — `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` — `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written for.
* `:debug` — the same, at `Debug` level: the stage's per-file lines are
`@debug`, so this is the equivalent of running the server with
`JULIA_DEBUG=FileServer` and the only setting under which they
are emitted at all.
* `:null` `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written for.
* `:debug` the same, at `Debug` level: the stage's per-file lines are
`@debug`, so this is the equivalent of running the server with
`JULIA_DEBUG=FileServer` and the only setting under which they
are emitted at all.
"""
function with_logger_named(f, which::Symbol, path::AbstractString)
if which === :null
@@ -380,8 +380,8 @@ end
thread_rows(cfg, jobs, opts) -> Vector
Run the full handler across worker counts, under the server's real logger. A
component that owns a lock — the queue's condition variable, the logger's
stream — stops scaling here even though it looked cheap single-threaded, so this
component that owns a lock (the queue's condition variable, the logger's
stream) stops scaling here even though it looked cheap single-threaded, so this
is where the single-thread ranking gets checked against the deployed one.
"""
function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
@@ -402,7 +402,7 @@ function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
ns = with_logger_named(:flush, logfile) do
best_of(() -> (drain!(known); drain!(unknown)); trials) do
# Static split: each task takes a contiguous slice, so the only
# sharing between workers is the state the server also shares —
# sharing between workers is the state the server also shares:
# the classifier, the queues, the logger, the filesystem.
chunk = cld(nfiles, k)
@sync for t in 1:k
@@ -490,7 +490,7 @@ function main(argv)
# flushing logger at Info level, one worker. Percentages are shares of
# that, so they are directly comparable and the parts can be checked
# against the whole. The JULIA_DEBUG row is deliberately *not* the
# baseline — it is the opt-in configuration, and letting it set the scale
# baseline. It is the opt-in configuration, and letting it set the scale
# would make every other component look free.
total = only(r.ns for r in handlers if r.name == "handle_classify_job (flush→file)")