jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial text cleanup

Browse Source
This commit is contained in:
Jeffrey Ward
2026-08-19 12:55:19 -04:00
parent 819a4cce7a
commit cc84d70d52
26 changed files with 454 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
bin/bench.jl
View File

@@ -1,12 +1,12 @@
#!/usr/bin/env julia
#
# bench.jl — measure end-to-end throughput and server memory for a running FileServer.
# bench.jl: measure end-to-end throughput and server memory for a running FileServer.
#
# Two things make this pipeline awkward to benchmark with off-the-shelf tools
# (ab/hey/wrk), and both shape what this script does:
#
# 1. HTTP latency is not throughput. /upload returns 202 as soon as the bytes
# are spooled and a reference is enqueued — all four stages run afterwards.
# are spooled and a reference is enqueued, and all four stages run afterwards.
# So real throughput is the *arrival rate at the terminal sinks*
# (done/, text_done/, binary/, failed/), not the response rate. We upload a
# corpus, then poll the sinks until the file count stops moving.
@@ -14,7 +14,7 @@
# 2. End-to-end throughput doesn't name the slow stage. The four stages run
# concurrently behind their own queues, so the pipeline's rate is the
# slowest stage's rate and the others are invisible. Directory polling can't
# recover them either — every in-flight file sits in spool/ whatever stage it
# recover them either: every in-flight file sits in spool/ whatever stage it
# is at, since stages route by enqueueing rather than by moving bytes. So the
# server keeps per-stage counters
# (src/stats.jl) and we scrape GET /stats before and after: the deltas give
@@ -36,7 +36,7 @@
#
# The harness talks to the server only over HTTP (/upload, /health, /stats) and
# reads the on-disk sink layout; it must run on the same machine (sink dirs and
# /proc). If /stats is missing — an older build — everything else still works and
# /proc). If /stats is missing (an older build) everything else still works and
# the per-stage section is skipped.
#
# Usage:
@@ -46,7 +46,7 @@
# --files N number of files to upload (default: 200)
# --size S size of each generated file, e.g. 4k, 512k, 8m, 1g (default: 64k)
# --concurrency J uploads in flight at once (default: 8)
# --kind K binary | text | mixed — what to generate (default: binary)
# --kind K binary | text | mixed: what to generate (default: binary)
# --corpus DIR upload an existing directory instead of generating
# (the only way to exercise stage 2: point it at real known files)
# --keep-corpus don't delete the generated corpus on exit
@@ -152,7 +152,7 @@ sinkdirs() = (
failed = get(ENV, "FS_FAILED_DIR", "data/failed"),
)
# One directory, not four. Files no longer move between stages — spool/ holds
# One directory, not four. Files no longer move between stages: spool/ holds
# every in-flight file at every stage, and which stage it has reached lives in
# the queue holding its reference (src/worker.jl header). So this depth is
# "files in flight", full stop; per-stage depth comes from /stats, which is the
@@ -200,7 +200,7 @@ Find the running server process, or `nothing`.
server: any shell launched with the command in its own argv (`sh -c 'julia …
bin/server.jl > log'`, a `setsid`/`nohup` wrapper, even the terminal running the
benchmark) matches the same pattern. Sampling one of those reports a few MiB of
shell as the server's memory — a wrong answer that looks plausible, which is the
shell as the server's memory: a wrong answer that looks plausible, which is the
worst kind.
So candidates are filtered by what each process *is* (`/proc/<pid>/comm`, the
@@ -309,7 +309,7 @@ function stage_deltas(before, after, peak_depth::Dict{Int,Int})
end
"JSON has no NaN. A stage that completed nothing has no service time, and `null`
is the honest way to say that — writing NaN just makes JSON3 throw."
is the honest way to say that; writing NaN just makes JSON3 throw."
json_num(x::Real) = isfinite(x) ? x : nothing
pad(s, n) = rpad(string(s), n)
@@ -321,7 +321,7 @@ Print the per-stage table and say which stage is the bottleneck.
The verdict reads utilization, not throughput: in a pipeline every stage
completes the same files, so at steady state they all report nearly the same
files/s regardless of which one is the constraint. What separates them is how
hard each pool had to work to keep up — the bottleneck is pinned near 1.0 while
hard each pool had to work to keep up: the bottleneck is pinned near 1.0 while
its neighbours idle.
"""
function stage_report(deltas::Vector{StageDelta}, window::Float64)
@@ -356,14 +356,14 @@ function stage_report(deltas::Vector{StageDelta}, window::Float64)
"$(fmt(utilization(top, window) * 100, 0))% utilization of " *
"$(top.workers) worker(s)")
if utilization(top, window) < 0.5
println(" — but no stage is near saturated: the pipeline is " *
println(" ...but no stage is near saturated: the pipeline is " *
"waiting on intake,\n not on itself. Raise --concurrency " *
"or --files to load it properly.")
end
for d in worked
blocked_share(d) > 0.25 && println(" ! stage $(d.stage) ($(d.name)) spent " *
"$(fmt(blocked_share(d) * 100, 0))% of its time parked on a full downstream " *
"queue —\n it is being held up by the stage after it, not doing that work itself.")
"queue.\n It is being held up by the stage after it, not doing that work itself.")
end
return nothing
end
@@ -459,7 +459,7 @@ end
Upload every path, at most `concurrency` in flight.
A bounded set of worker tasks pulling from a shared index keeps exactly
`concurrency` requests in flight for the whole run — unlike batching, where each
`concurrency` requests in flight for the whole run, unlike batching, where each
batch stalls on its slowest (largest) file and the real concurrency sags.
"""
function upload_all(url::String, paths::Vector{String}, concurrency::Int)
@@ -504,7 +504,7 @@ function main(argv)
try
HTTP.get(string(rstrip(url, '/'), "/health"); retry = false, readtimeout = 5)
catch e
println(stderr, "cannot reach $url/health — is the server running?")
println(stderr, "cannot reach $url/health. Is the server running?")
println(stderr, " start it with: julia --project=. -t auto bin/server.jl")
return 1
end
@@ -548,7 +548,7 @@ function main(argv)
"reporting sampled RSS only"
# Baseline is read *after* the reset, not before. VmHWM restarts
# from whatever RSS is at the moment of the reset, so a baseline
# sampled earlier is measured against a different origin — and if
# sampled earlier is measured against a different origin, and if
# the GC hands memory back in between, the run reports negative
# growth, which is nonsense on its face.
r2 = read_rss(pid)
@@ -587,8 +587,8 @@ function main(argv)
depth_max[k] = max(depth_max[k], getfield(d, k))
end
# Queue depth, unlike directory depth, can't be missed by a slow
# sample in the same way — a file's *reference* sits in the queue
# for the whole time it waits — so this is the depth the stage
# sample in the same way, since a file's *reference* sits in the
# queue for the whole time it waits, so this is the depth the stage
# table reports.
if stats_before !== nothing
s = scrape_stats(url)
@@ -673,7 +673,7 @@ function main(argv)
"$(human(corpusbytes / length(paths))) avg")
println("concurrency $(opts["concurrency"])")
println()
println("INTAKE (HTTP 202 — bytes spooled, not processed)")
println("INTAKE (HTTP 202: bytes spooled, not processed)")
println(" accepted $accepted of $(length(paths)) [202: $n_202, 503: $n_503, error: $n_err]")
println(" wall $(fmt(intake_secs))s")
println(" rate $(fmt(accepted / max(intake_secs, 1e-9))) files/s, " *
@@ -700,7 +700,7 @@ function main(argv)
println("SERVER MEMORY (pid $pid)")
println(" baseline RSS $(human(baseline_rss))")
println(" peak RSS $(human(peak_rss))" *
(peak_reset ? " (kernel VmHWM, reset at start)" : " (sampled — may miss spikes)"))
(peak_reset ? " (kernel VmHWM, reset at start)" : " (sampled; may miss spikes)"))
growth = peak_rss - baseline_rss
if growth <= 0
# RSS never got back to where it started, so the run's own cost
@@ -708,13 +708,13 @@ function main(argv)
# Printing a negative "growth" would invite reading a memory
# *saving* into what is really "too small to measure here".
println(" growth none measurable (peak never exceeded the baseline)")
println(" The baseline was still falling when we sampled it — give the")
println(" The baseline was still falling when we sampled it. Give the")
println(" server ~30s to settle after startup for a comparable figure.")
else
println(" growth $(human(growth))")
println(" per in-flight $(human(growth / opts["concurrency"])) " *
"at $(human(corpusbytes / length(paths))) avg file size")
println(" (should not grow with file size — intake streams to disk)")
println(" (should not grow with file size: intake streams to disk)")
end
# Julia's GC returns memory to the OS lazily, so a second run on the
# same process starts from an inflated baseline and under-reports
@@ -729,7 +729,7 @@ function main(argv)
println("=" ^ 68)
sink_delta.failed > 0 &&
println("\nnote: $(sink_delta.failed) file(s) landed in $(sinks.failed) — check the server log.")
println("\nnote: $(sink_delta.failed) file(s) landed in $(sinks.failed); check the server log.")
timed_out &&
println("\nnote: drain stalled with $(accepted - completed) file(s) outstanding. " *
"Check the server log and spool/; raise --timeout if the pipeline is just slow.")

20
bin/bench_model.jl
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env julia
#
# bench_model.jl — microbenchmark the classifier in isolation, with no server,
# bench_model.jl: microbenchmark the classifier in isolation, with no server,
# no queue, and no disk in the way.
#
# bin/bench.jl measures the *pipeline*: it reports stage 1 as one number, the
@@ -12,7 +12,7 @@
#
# read_features open, read 16 bytes, seek, read 16 bytes, scale
# Lux.apply the network itself, on a feature vector already in memory
# classify both together — what stage 1 actually calls per file
# classify both together: what stage 1 actually calls per file
#
# Three properties are worth checking beyond the raw per-file cost:
#
@@ -21,7 +21,7 @@
# (This is the same claim bin/bench.jl makes about memory, on the CPU axis.)
# * Batching should be much cheaper per file. A 32x1 matmul wastes most of a
# BLAS call; if batch-64 inference is many times cheaper per file, that is
# the headroom a batching stage-1 would buy — worth knowing before building
# the headroom a batching stage-1 would buy, worth knowing before building
# one, since today the pipeline classifies strictly one file at a time.
# * Inference should scale across threads. `Classifier` is shared read-only by
# the whole stage-1 pool on the claim that Lux inference is pure. If per-file
@@ -48,7 +48,7 @@ using Statistics
using Printf
# The script is run directly, not as part of the package, so pull in exactly the
# pieces the classifier needs. `Lux`/`JLD2` first — model.jl and classify.jl both
# pieces the classifier needs. `Lux`/`JLD2` first, because model.jl and classify.jl both
# assume the including scope already has them (see the note at the top of model.jl).
using Lux
using JLD2
@@ -156,7 +156,7 @@ fmt2(x::Real) = @sprintf("%.2f", x)
"""
control_kernel(x) -> Float64
Pure arithmetic, no allocation, no library call — deliberately dependent
Pure arithmetic, no allocation, no library call, and deliberately dependent
(each step needs the last) so the compiler can't vectorize it away, and sized to
land in the same microsecond neighbourhood as one `Lux.apply`.
"""
@@ -209,7 +209,7 @@ end
"""
Write a file of exactly `size` random bytes, in bounded chunks.
Content is random rather than zeros so the classifier sees a realistic input —
Content is random rather than zeros so the classifier sees a realistic input,
and so the filesystem can't cheat with a sparse file, which would make the tail
`seek` unrepresentatively fast.
"""
@@ -252,7 +252,7 @@ function main(argv)
Lux.apply(clf.model, x1, clf.ps, clf.st)
end
println()
println("INFERENCE (Lux.apply, batch 1 — features already in memory)")
println("INFERENCE (Lux.apply, batch 1; features already in memory)")
println(" per call $(human_time(infer_ns)) $(human_rate(rate(infer_ns)))")
println(" allocations $(human_bytes(infer_bytes)) per call")
results["inference_batch1"] = (; ns = infer_ns, bytes = infer_bytes, per_sec = rate(infer_ns))
@@ -302,14 +302,14 @@ function main(argv)
results["read_features"] = read_rows
flat = length(read_rows) > 1 ?
maximum(r.ns for r in read_rows) / minimum(r.ns for r in read_rows) : 1.0
@printf(" spread across a %.0fx size range: %.1fx — %s\n",
@printf(" spread across a %.0fx size range: %.1fx, %s\n",
maximum(sizes) / minimum(sizes), flat,
flat < 3 ? "flat, as designed (it seeks to the tail)" :
"NOT flat: something is reading more than 32 bytes")
# --- 4. classify(): what stage 1 calls, I/O and inference together.
println()
println("CLASSIFY (read_features + Lux.apply — one whole stage-1 file)")
println("CLASSIFY (read_features + Lux.apply; one whole stage-1 file)")
dir2 = mktempdir(; prefix = "fsmodel-")
classify_ns = 0.0
try
@@ -378,7 +378,7 @@ function main(argv)
end
results["thread_scaling"] = thread_rows
# A poor scaling curve has two possible authors — the model or the box —
# A poor scaling curve has two possible authors, the model or the box,
# and the table alone can't tell them apart. So run the same sweep on a
# kernel that is pure arithmetic with no allocation and no library
# underneath: whatever *it* achieves is this machine's ceiling for

36
bin/bench_stage1.jl
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env julia
#
# bench_stage1.jl — take stage 1 apart and find the slowest component.
# bench_stage1.jl: take stage 1 apart and find the slowest component.
#
# bin/bench.jl reports stage 1 as a single number (the wall time of
# `handle_classify_job` under whole-pipeline contention) and bin/bench_model.jl
@@ -17,7 +17,7 @@
# `handle_classify_job` end to end so the parts can be checked against the whole.
# The two knobs that most change the answer get their own sweeps:
#
# * Logger. The server runs `FlushLogger(ConsoleLogger(stderr))` — it formats
# * Logger. The server runs `FlushLogger(ConsoleLogger(stderr))`, which formats
# and flushes every message. Under redirect (a log file, journald) that is a
# syscall per line, two lines per file, on the hot path. We time the handler
# under a null logger, a formatting-but-discarding logger, and the real
@@ -112,7 +112,7 @@ const SINK = Ref{Any}(nothing)
Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
(`pass` returns the number of operations it performed). `prepare()` runs before
each pass and is *not* timed — that is where a benchmark resets whatever its
each pass and is *not* timed: that is where a benchmark resets whatever its
last pass consumed (today: the queues). `pass` comes first so callers can pass it as a `do` block.
The first pass is thrown away: it pays Julia's JIT compilation, which on calls
@@ -164,7 +164,7 @@ end
"""
Write a file of exactly `size` random bytes, in bounded chunks.
Content is random rather than zeros so the classifier sees a realistic input —
Content is random rather than zeros so the classifier sees a realistic input,
and so the filesystem can't cheat with a sparse file, which would make the tail
`seek` unrepresentatively fast.
"""
@@ -185,7 +185,7 @@ end
make_corpus(cfg, n, size, rng) -> Vector{Job}
Create `n` spooled files and the `Job` references a stage-1 worker would dequeue
for them — the exact input `handle_classify_job` sees.
for them: the exact input `handle_classify_job` sees.
"""
function make_corpus(cfg::FS.Config, n::Int, size::Int, rng)
jobs = FS.Job[]
@@ -218,15 +218,15 @@ end
Run `f` under one of the three loggers the cost of logging is bracketed by:
* `:null` — `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` — `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` — `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written for.
* `:debug` — the same, at `Debug` level: the stage's per-file lines are
`@debug`, so this is the equivalent of running the server with
`JULIA_DEBUG=FileServer` and the only setting under which they
are emitted at all.
* `:null` `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written for.
* `:debug` the same, at `Debug` level: the stage's per-file lines are
`@debug`, so this is the equivalent of running the server with
`JULIA_DEBUG=FileServer` and the only setting under which they
are emitted at all.
"""
function with_logger_named(f, which::Symbol, path::AbstractString)
if which === :null
@@ -380,8 +380,8 @@ end
thread_rows(cfg, jobs, opts) -> Vector
Run the full handler across worker counts, under the server's real logger. A
component that owns a lock — the queue's condition variable, the logger's
stream — stops scaling here even though it looked cheap single-threaded, so this
component that owns a lock (the queue's condition variable, the logger's
stream) stops scaling here even though it looked cheap single-threaded, so this
is where the single-thread ranking gets checked against the deployed one.
"""
function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
@@ -402,7 +402,7 @@ function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
ns = with_logger_named(:flush, logfile) do
best_of(() -> (drain!(known); drain!(unknown)); trials) do
# Static split: each task takes a contiguous slice, so the only
# sharing between workers is the state the server also shares —
# sharing between workers is the state the server also shares:
# the classifier, the queues, the logger, the filesystem.
chunk = cld(nfiles, k)
@sync for t in 1:k
@@ -490,7 +490,7 @@ function main(argv)
# flushing logger at Info level, one worker. Percentages are shares of
# that, so they are directly comparable and the parts can be checked
# against the whole. The JULIA_DEBUG row is deliberately *not* the
# baseline — it is the opt-in configuration, and letting it set the scale
# baseline. It is the opt-in configuration, and letting it set the scale
# would make every other component look free.
total = only(r.ns for r in handlers if r.name == "handle_classify_job (flush→file)")

42
bin/bench_stage2.jl
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env julia
#
# bench_stage2.jl — take stage 2 apart and find the slowest component.
# bench_stage2.jl: take stage 2 apart and find the slowest component.
#
# bin/bench.jl reports stage 2 as a single number (its throughput and worker
# utilization under whole-pipeline contention). It doesn't say *which part* of
@@ -27,13 +27,13 @@
#
# * The corpus must be real files. exiftool's cost depends on what it finds;
# random bytes exit early and would understate the stage by a lot. The
# default corpus is `data/done` — files that already went through stage 2 on
# this machine — copied back into a scratch spool/ dir.
# default corpus is `data/done`, files that already went through stage 2 on
# this machine, copied back into a scratch spool/ dir.
# * Two rows price the *alternatives* to one-fork-per-file, because if the
# fork dominates then the only fixes are to stop paying it per file:
# `exiftool (batched Nx)` runs the whole corpus through one process, and
# `exiftool (-stay_open)` keeps a single process alive and feeds it one file
# at a time over a pipe — the shape a streaming pipeline could actually use.
# at a time over a pipe: the shape a streaming pipeline could actually use.
# Both are measured, not assumed.
# * `run_with_timeout` gets its own row *next to* a bare `Base.run` of the same
# command. The difference is what the watchdog costs, and its polling loop
@@ -48,7 +48,7 @@
# julia --project=. -t auto bin/bench_stage2.jl [options]
#
# --files N corpus files per timed pass (default: 48). The concurrency
# sweep wants more than the component rows do — with a single
# sweep wants more than the component rows do. With a single
# 2 s file in the corpus, 48 files can't show more than ~4x no
# matter how many workers run, so pass --files 150 when the
# question is scaling.
@@ -121,7 +121,7 @@ const SINK = Ref{Any}(nothing)
Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
(`pass` returns the number of operations it performed). `prepare()` runs before
each pass and is *not* timed — that is where a consuming benchmark puts the file
each pass and is *not* timed: that is where a consuming benchmark puts the file
back where it started. `pass` comes first so callers can pass it as a `do` block.
The first pass is thrown away: it pays Julia's JIT compilation, which on calls
@@ -174,12 +174,12 @@ end
make_corpus(cfg, corpus_dir, n) -> Vector{Job}
Copy up to `n` real files from `corpus_dir` into `spool/` and build the `Job`
references a stage-2 worker would dequeue for them — the exact input
references a stage-2 worker would dequeue for them: the exact input
`handle_known_job` sees.
Real files, not generated ones: exiftool's cost is a function of what it can
parse, and a file of random bytes bails out early enough to understate the stage
by an order of magnitude. `.meta.json` sidecars are skipped — they are stage-2
by an order of magnitude. `.meta.json` sidecars are skipped, since they are stage-2
*output*, and enriching them would measure the wrong population.
"""
function make_corpus(cfg::FS.Config, corpus_dir::AbstractString, n::Int)
@@ -211,7 +211,7 @@ end
Put every corpus file back in `spool/`, wherever the last pass left it (done/ or
already home), and delete any sidecar it produced. This is the untimed `prepare`
step for benchmarks that consume their input by committing it — stage 2 still
step for benchmarks that consume their input by committing it. Stage 2 still
moves, because its move is the terminal commit, not an inter-stage hop.
"""
function respool!(cfg::FS.Config, jobs::Vector{FS.Job})
@@ -241,13 +241,13 @@ end
Run `f` under one of the loggers the cost of logging is bracketed by:
* `:null` — `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` — `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` — `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written
for. Stage 2's per-file line is `@info`, not `@debug`, so this
row is what the deployed server actually pays.
* `:null` `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` `FlushLogger(ConsoleLogger(io))` to a real file: what
`FileServer.run` installs, under the redirect it was written
for. Stage 2's per-file line is `@info`, not `@debug`, so this
row is what the deployed server actually pays.
"""
function with_logger_named(f, which::Symbol, path::AbstractString)
if which === :null
@@ -286,7 +286,7 @@ end
Run the whole corpus through *one* `exiftool` process and divide by the file
count. This is the floor for "what does exiftool cost if you stop paying the
interpreter startup per file" — the fork, the Perl boot and the module loads are
interpreter startup per file": the fork, the Perl boot and the module loads are
paid once for the batch instead of once per file.
"""
function batched_ns(paths::Vector{String}, trials::Int)
@@ -301,8 +301,8 @@ end
Feed files one at a time to a single long-lived `exiftool -stay_open True -@ -`
process over a pipe, reading its `{ready}` sentinel after each. Unlike the
batched row this preserves the pipeline's actual shape — one file in, one result
out, arriving whenever it arrives — so it prices the realistic fix rather than
batched row this preserves the pipeline's actual shape (one file in, one result
out, arriving whenever it arrives) so it prices the realistic fix rather than
an unrealistic one.
"""
function stay_open_ns(paths::Vector{String}, trials::Int)
@@ -439,7 +439,7 @@ function component_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
# One pass over the real sidecar population, not `reps` of them. Two reasons,
# and the first is a correctness trap: thousands of back-to-back fsyncs
# saturate the device's write cache and each one starts waiting on the
# queue, which reported this row at 16 ms/file — eight times the whole
# queue, which reported this row at 16 ms/file: eight times the whole
# `commit_enriched!` that contains it. The real stage fsyncs once per file
# with ~160 ms of exiftool between, and never queues that way. Second, real
# sidecars vary hugely in size (a zip's raw dump dwarfs a jpeg's), so the
@@ -546,7 +546,7 @@ machine's cores, not about Julia.
Workers pull from a shared atomic counter rather than taking a contiguous slice.
That matches the server (its pool pulls from one queue), and it matters here in a
way it doesn't for stage 1: per-file exiftool time spans two orders of magnitude
on a real corpus — a single 2 s archive among 48 files — so a static split leaves
on a real corpus (a single 2 s archive among 48 files), so a static split leaves
whichever worker drew it running alone while the rest idle, and the sweep would
report a scaling ceiling that is really just load imbalance.
"""

14
bin/cluster_calibrate.jl
View File

@@ -5,7 +5,7 @@
# recovered partition against magic-collapsed ground truth with ARI / V-measure,
# grid-tunes (α, β, bg_mass, n), and cross-checks the winning config against a
# model-free NCD (gzip) baseline (§8). The settings printed here are the ones the
# machine rediscovers known formats at — copy the winner into config.jl.
# machine rediscovers known formats at; copy the winner into config.jl.
#
# julia --project=. bin/cluster_calibrate.jl [training_set_dir]
#
@@ -23,7 +23,7 @@ include(joinpath(@__DIR__, "..", "src", "cluster.jl"))
The magic-collapsed format class of a file, read from its actual bytes (so
docx≡zip and the whole ELF family merge, exactly the answer we want the
clustering to reproduce). `tar` is detected by the `ustar` magic at offset 257 —
clustering to reproduce). `tar` is detected by the `ustar` magic at offset 257,
outside the model's front window, so tars are the accepted blind spot that
scatters to background.
"""
@@ -53,7 +53,7 @@ function gz_size(bytes::Vector{UInt8})
return length(take!(out))
end
"NCD(x,y) = (C(xy) - min(C(x),C(y))) / max(C(x),C(y)) — 0 = identical, ~1 = unrelated."
"NCD(x,y) = (C(xy) - min(C(x),C(y))) / max(C(x),C(y)); 0 = identical, ~1 = unrelated."
function ncd(xb, yb, cx, cy)
cxy = gz_size(vcat(xb, yb))
return (cxy - min(cx, cy)) / max(cx, cy)
@@ -62,7 +62,7 @@ end
"""
ncd_1nn_purity(paths, truth; head_bytes) -> Float64
Fraction of files whose NCD-nearest neighbour shares its true label — a cheap,
Fraction of files whose NCD-nearest neighbour shares its true label: a cheap,
O(N²) sanity read on how well raw gzip-similarity alone separates formats on the
same input. The Bayesian clusters should broadly agree; a big gap is a red flag
(DESIGN §10.3). Uses each file's first `head_bytes` so the giant files don't
@@ -173,7 +173,7 @@ function main()
end
# Rank by ARI-excluding-tar (tar is the accepted blind spot; scoring it would
# penalise the correct answer of scattering tars to background — DESIGN §7.2).
# penalise the correct answer of scattering tars to background; DESIGN §7.2).
sort!(results; by=r -> r.e.ari_notar, rev=true)
best = results[1]
println()
@@ -184,7 +184,7 @@ function main()
# Per-cluster composition of the winning partition, and promotion nominations.
pred = best.e.result.assignments
println("\nwinning partition — cluster composition (truth breakdown):")
println("\nwinning partition, cluster composition (truth breakdown):")
for (id, c) in sort(collect(best.e.result.clusters); by=x -> -x[2].members)
members = [truth[i] for i in eachindex(pred) if pred[i] == id]
comp = sort([(l, count(==(l), members)) for l in unique(members)]; by=x -> -x[2])
@@ -211,7 +211,7 @@ function main()
bay_pur = cluster_1nn_purity(rsub.assignments, subtruth)
@printf(" subsample=%d NCD 1-NN label purity=%.3f Bayesian same-cluster purity=%.3f\n",
subn, ncd_pur, bay_pur)
println(" (both high ⇒ header-byte signal agrees with model-free gzip similarity — DESIGN §10.3)")
println(" (both high ⇒ header-byte signal agrees with model-free gzip similarity; DESIGN §10.3)")
end
main()

4
bin/cluster_sweep.jl
View File

@@ -3,14 +3,14 @@
# Stage-5 phase-B runner (model/DESIGN_clustering.md §9): the single-owner,
# periodic/cron process that sweeps `binary/`, folds new files into the durable
# format catalog by sequential CRP-predictive assignment, and (re)writes
# promotion nominations. Run it single-threaded on a schedule — it is the ONLY
# promotion nominations. Run it single-threaded on a schedule: it is the ONLY
# writer of the catalog, so no locking is needed.
#
# julia --project=. bin/cluster_sweep.jl # incremental live sweep
# julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs (seed / recompact)
#
# On a fresh catalog (nothing processed yet) the incremental sweep would send
# every file to background — there are no clusters to match — so the first run
# every file to background (there are no clusters to match), so the first run
# auto-promotes to a compaction pass to seed the catalog. Configure via the
# FS_CLUSTER_* / FS_NOMINATED_DIR env vars (see src/config.jl).