bin/bench.jl reports stage 2 as a single throughput number, which can't
distinguish slow extraction from a slow spawn — and those have opposite fixes.
bin/bench_stage2.jl times each component in isolation, then times the real
handle_known_job end to end. It draws its corpus from real files (default
data/done) because random bytes make exiftool bail out early and understate the
stage by ~10x, and it prices both fork-free alternatives (batched, -stay_open)
so the cost of one-fork-per-file is a measurement rather than a guess.
The benchmark found stage 2 to be ~98% exiftool, and found two problems in
run_with_timeout, which stages 2 and 4 share:
1. The watchdog polled with sleep(0.1) and then joined the polling task, so
every call paid the remainder of an in-flight sleep after the child had
already exited: ~25 ms per file, and a measured 101 ms on a process that
exits instantly. Replaced with a one-shot Timer cancelled when the child
exits. Stage 2 goes from 164.6 ms to 138.3 ms per file (6 -> 8 files/s on one
worker); the wrapper is now within noise of a bare Base.run.
2. Writing the missing tests showed the timeout was never enforceable, in the
old implementation as much as the new. wait(proc) returns only once the
captured stdout pipe closes, and grandchildren inherit that pipe, so
signalling the child alone left the worker blocked until the whole process
tree finished on its own — a `sh -c "trap '' TERM; sleep 30"` child ran the
full 30 s against a 1 s timeout. The child now runs in its own process group
and the timeout signals the group. The trade is that a hard crash of the
server orphans an in-flight child rather than taking it down with it.
Three new tests cover the fast path, the timeout, and the SIGTERM-ignoring
escalation; the second was previously unexercised, which is why the bug stood.
Not addressed here, but measured and documented in the README: Perl interpreter
startup is 76.7 ms of the remaining 135.9 ms call, so a persistent exiftool
(-stay_open, 40.1 ms/file) would cut the stage by roughly another 70%. And
fsync_dir measures 1.75 us, too fast to be a real flush — commit_enriched!'s
durability may not hold on this filesystem, which is a correctness question
left open.
Claude-Session: https://claude.ai/code/session_01Xy9At7HNLHWNmfh1yw71Uy
bin/bench_stage1.jl takes handle_classify_job apart — filesize, read_features,
Lux.apply, classify, move_to, enqueue_blocking!, and the log lines — times each
in isolation, then times the real handler end to end under four loggers so the
parts can be checked against the whole.
It found that logging was stage 1's dominant cost: as @info the two per-file
lines cost ~71 us of the handler's ~118 us, roughly 6x the classifier (10.6 us)
and 6x the rename (11.6 us). Nearly all of it is ConsoleLogger formatting
(~64 us), not the FlushLogger's per-message flush (~8 us).
Demoting them to @debug takes stage 1 from 8.5k files/s to 35.3k files/s on one
worker (4.2x). The messages are still available with JULIA_DEBUG=FileServer,
which the benchmark also prices (133 us/file). What remains splits evenly
between the rename (11.7 us) and classify (10.7 us, itself 74% feature read),
so stage 1 is now filesystem-bound; its thread sweep peaks at ~4 workers.
End-to-end throughput says how fast the pipeline is, not which stage is the
reason. The four stages run concurrently behind their own queues, so the
pipeline's rate *is* the slowest stage's rate and the others are invisible in
it. Nothing outside the server can recover them either: known/, unknown/ and
text/ are transient, and a file can cross one between two directory polls, so an
external sampler misses exactly the stages worth measuring.
So the pipeline counts its own work, and bin/bench.jl turns two scrapes into
rates.
- src/stats.jl: per-stage counters (completed/failed, bytes, busy_ns,
blocked_ns, in_flight) plus intake counters, monotonic since startup in the
Prometheus style — rates are the reader's job, so a scrape is stateless and
two readers can't disturb each other. Recorded in worker_loop, the one place
every stage's work passes through, so a new stage is instrumented the moment
it is wired up and never on the read path.
- src/server.jl: GET /stats. An ordinary Oxygen route (no body to stream),
unlike /upload. Intake counts files at the point they become stage 1's
problem, so intake totals and stage-1 arrivals refer to the same files.
- src/queue.jl: capacity(q) joins length on the introspection seam — a depth of
900 means nothing without knowing whether the limit is 1000 or 1_000_000.
utilization = (busy - blocked) / (window * workers) is the number that names the
bottleneck: throughput alone can't tell a saturated stage from one starved by
the stage ahead of it, since both report the same files/s. blocked_ns is what
keeps that true. Stages 1 and 3 apply blocking backpressure — a full downstream
queue means parking, not dropping — and that wait is inside the handler, so
counting it as busy would pin stage 1 at 1.0 whenever stage 2 is the real jam,
making every stage upstream of a jam look like the jam. enqueue_blocking! wraps
the retry loop so the wait is measurable at all, and keeps the three routing
paths from drifting into three different backoff behaviours.
Measured (400 mixed files, 16 KiB, concurrency 16): stage 4 is the constraint at
0.87 utilization and 853 ms/file — github-linguist is a process spawn per file —
while stages 1 and 3 idle under 0.10. Verified the blocked accounting against a
deliberately starved server (FS_TEXT_WORKERS=1, FS_TEXT_QUEUE_CAPACITY=2): stage
3 reported 100% blocked at 0.0 utilization rather than looking saturated too.
bin/bench_model.jl: the classifier alone, no server or queue in the way, because
stage 1's 38.7 ms/file cannot plausibly be a 32-64-16-2 MLP. It isn't:
Lux.apply is 2.3 us, read_features 4.2-6.0 us (flat across 1 KiB - 256 MiB, as
the seek-to-tail design intends), classify() 8.4 us — so ~99.98% of stage 1 is
rename, logging and contention, and the file read costs 3x the inference. Two
findings: batching would buy ~13x (179 ns/file at batch 512 vs 2.34 us at batch
1), and inference does not scale past ~4 threads. A pure-compute control kernel
runs the same sweep to place the blame — it reaches 14.3x at 16 tasks on this
box, so the machine parallelizes and Lux.apply does not. BLAS threads and GC are
both ruled out; the cause is inside Lux and is not diagnosed here.
Three bugs in the memory measurement, all of which produced wrong answers that
looked plausible:
- detect_pid matched any process with the launch command in its argv, including
the shell that started the server — one run reported 3.64 MiB as the server's
memory. Candidates are now filtered by /proc/<pid>/comm, what the process is
rather than what its arguments say; no pattern over argv can do that.
- Baseline RSS was read *before* clear_refs reset the peak counter, so the two
numbers had different origins. The 2 GiB run reported -1.01 MiB of growth;
reading the baseline after the reset makes it 31.3 MiB.
- Negative growth is now reported as "none measurable" rather than a negative
figure, which reads as a memory saving.
Re-measuring with those fixed keeps the claim that matters — growth is flat in
file size (14-31 MiB from 256 MiB to 2 GiB), so nothing is buffering — but the
concurrency coefficient does not survive: a freshly started server settles
anywhere in an ~860-985 MiB band, so baseline variance is comparable to the
growth being measured, and the old table quoted megabyte precision the
measurement never supported. README now states the shape, requires a ~30s settle
before a memory run, and says plainly that linear-in-concurrency is
undemonstrated rather than leaving an authoritative-looking number.
README also gains a single runnable sequence for all three harnesses: the server
prerequisite was never shown inline, so following the benchmarking section
top-to-bottom just produced "cannot reach /health".
Tests: 276 pass (41 new) — the blocked-vs-busy split, worker_loop draining
in_flight through a throwing handler, and the JSON round-trip of the field names
bench.jl reads.
Adds a benchmark harness, which showed that intake buffered each upload
whole, then makes intake streaming so the service's flat-memory property
holds end to end rather than only for the queue and workers.
The measurement problem first: /upload returns 202 once bytes are spooled
and a reference is enqueued, so HTTP latency measures intake, not the
pipeline. bin/bench.jl instead uploads a corpus and polls the terminal
sinks until the count stops moving, reporting intake rate and end-to-end
rate separately, sampling server RSS (kernel VmHWM, reset per run) and
the intermediate stage depths so the bottleneck stage names itself.
That exposed the buffering: HTTP.jl read the body into req.body,
parse_multipart_form materialized each part, and read(p.data) copied
again before spool_file wrote it — a 256 MiB upload grew RSS ~700 MiB,
and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB.
- src/multipart.jl: incremental multipart/form-data reader. Pulls fixed
chunks off the socket and hands each part's bytes straight to a sink,
so memory is bounded by FS_UPLOAD_CHUNK_BYTES (64 KiB), not file size.
Interface is two calls in a loop (next_part! then write_part_body! /
skip_part_body!) so the handler keeps ordinary control flow. Retains
the last length(delimiter)-1 bytes so a delimiter split across chunks
still parses; part headers are bounded by policy, not by chunking.
- src/server.jl: /upload is served by a stream handler. Oxygen's root
handler wraps HTTP.streamhandler, which does request.body =
read(stream) before dispatching — so no Oxygen route, not even a
@stream route, can stream a body. root_stream_handler intercepts
POST /upload at the stream level and delegates the rest to Oxygen
unchanged; /upload is therefore absent from Oxygen's metrics and docs.
A client hangup is classified as routine (info, not error) and answered
best-effort; every exit path drains the body so keep-alive still works.
- src/spool.jl: spool_file(bytes) -> spool_stream(write_body!, ...),
which removes a partial file on a failed or abandoned write, so restart
recovery can never pick up a truncated upload as if it were complete.
- config.jl: FS_UPLOAD_CHUNK_BYTES, the intake memory dial.
Streaming changes the 503 contract: a buffered handler knew up front how
many files a request held, this one discovers them as they arrive. When
the queue fills mid-request it no longer abandons the connection — it
stops spooling (discarding remaining parts rather than writing files it
cannot queue), drains, and answers 503 with the accepted list. Files
already queued stay queued.
Measured after (fresh server, 64 KiB chunk): 256 MiB +21.8 MiB, 1 GiB
+20.8 MiB, 2 GiB +17.0 MiB at concurrency 1 — flat across a 32x size
range; 4 concurrent 256 MiB uploads +86.9 MiB, linear in concurrency.
A 2 GiB upload sustains 334 MiB/s. Small files did not regress (intake
107 -> 133 files/s, end-to-end 27.6 -> 32.9 files/s, p95 1930 -> 776 ms).
A --size sweep that slopes upward is now the regression signal.
- Tests (235 pass, 55 new): byte-exact round-trip of 10 files in one
request, sizes straddling the chunk boundary (0/1/63/65535/65536/65537/
131072/196615/1e6) plus a payload stuffed with near-boundary sequences;
the same body parsed at chunk sizes 1..10000 to put the delimiter split
at every offset; bounded allocation on a 16 MiB part; malformed and
truncated bodies; spool_stream cleanup on a failed write.
- Known cosmetic caveat, documented: when a body is cut short, HTTP.jl's
own closeread logs an EOFError after the handler returns, because
Content-Length promised more than arrived. Not reachable from a
handler; the old code logged the same thing without replying.
Wraps the assign_file scoring core (cluster.jl) in the live catalog the
design's phase B calls for (DESIGN §5B/§9):
- src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient
stats + processed set + examples); sparse, sidecar-first durable
save/load; incremental catalog_sweep! (deterministic CRP-predictive
assignment of new binary/ files); offline compact! that seeds on first
run and recompacts later; write_nominations! emitting one JSON per
promotable cluster with a hex magic template.
- bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact
forces a recluster; first run auto-compacts to seed).
- config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG,
FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs.
- Tests: durable round-trip, incremental sweep growth + idempotency,
§10.1 nothing-from-noise end-to-end (zero promotions), a recurring
format self-nominating, seed-then-live-assign (165 pass).
- DESIGN_clustering.md: mark phase B built.
Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process
mixture of per-position categoricals over the first 32 header bytes (257-symbol
alphabet) that clusters the binary/ pile by file format, plus signature
extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps
the Dirichlet-multinomial marginal dependency-free).
- src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler
(phase A), sequential CRP-predictive assignment (phase B core), signatures/
promotion, and ARI/V-measure calibration metrics.
- bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed
ground truth and cross-checks a model-free NCD (gzip) baseline.
- FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module.
- Tests for the three DESIGN §10 assertions plus the model primitives.
Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at
ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the
NCD baseline agrees. DESIGN §11 records the results and three assumptions the
data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and
the Bernoulli signature / Occam-penalized restart scoring).
Text files sorted by stage 3 now flow onto a new work queue and worker
pool that enrich them with natural language (Languages.jl LanguageDetector:
name, ISO 639-3 code, confidence) and programming language (github-linguist),
writing a .meta.json sidecar to data/text_done/ like the stage-2 known-file
pipeline.
github-linguist reads the git blob of a path inside a repo, so untracked
data/ files are copied to /tmp (outside any repo, name preserved for
extension heuristics) before detection. Programming-language lookup is
best-effort (startup warning if missing, degraded/null on failure);
natural-language failure yields a degraded sidecar, not a quarantine.
Factored exiftool's timeout-kill into shared run_with_timeout and the
durable sidecar-first commit into commit_enriched!, both reused by stage 4.
Recovery re-drives data/text/; graceful drain closes the text queue after
its stage-3 producers finish.
The NUL-byte heuristic misfiled any non-ASCII UTF-8 text (accents, CJK,
emoji) as binary and let non-NUL control bytes through as text. is_binary
now calls a file text when its 8000-byte sniff window is valid UTF-8 with
no control bytes outside the text-safe set (tab/newline/CR/ESC/etc).
- trim_truncated_utf8 drops a multi-byte char split by the window edge so
it isn't mistaken for malformed bytes.
- NUL still classifies as binary (valid UTF-8 scalar, non-text control).
- Expanded tests: Unicode, ANSI logs, stray control byte, malformed UTF-8,
boundary-split char; updated README stage-3 description.
Unknown files are no longer terminal. Stage 1 now routes :unknown onto a
dedicated queue (with the same blocking backpressure as the known queue),
and a third worker pool sorts each file into data/binary/ or data/text/
using a NUL-byte sniff of the first 8000 bytes.
- content.jl: is_binary content sniff (stage 3)
- worker.jl: handle_unknown_job; stage-1 routes unknown with backpressure;
KNOWN_ENQUEUE_RETRY_SECONDS -> ROUTE_ENQUEUE_RETRY_SECONDS (serves both)
- config.jl: unknown_worker_count/queue_capacity, binary_dir, text_dir + env
- FileServer.jl: unknown queue, pool, stage-aware recovery, drain ordering
- tests for is_binary and handle_unknown_job; tmp_config isolates new dirs
- README: three-stage pipeline
Address code-review findings on the metadata pipeline:
- finalize_known! now fsyncs the sidecar bytes before the rename and
fsyncs done/ after, so the "file in done/ implies sidecar present"
invariant holds across power loss, not just process crashes. The
docstring previously claimed an fsync the code never performed.
- run_exiftool's timeout escalates SIGTERM -> (2s grace) -> SIGKILL, so
an exiftool that ignores SIGTERM can't pin a worker forever on
wait(proc). Previously the timeout sent only SIGTERM.
- Add test/ (48 tests) covering the correctness-critical paths:
sanitize_filename, normalize_metadata, degraded build_metadata,
real exiftool extraction, finalize_known! end-to-end, recover_dir!.
Known-classified files now flow to a second queue with its own worker pool
that extracts metadata via exiftool and writes a normalized JSON sidecar
next to the file in done/, leaving the original bytes untouched.
- Two-stage pipeline: spool/ → classify → known/ → enrich → done/;
unknowns park in unknown/ as a seam for a future pool
- src/metadata.jl: exiftool -json -G -n with timeout, normalized schema
(file_type, mime_type, author, created_by, dimensions, ...) + raw dump;
degraded sidecar on extraction failure rather than quarantine
- Sidecar-first commit so a file in done/ always has its sidecar
- Parametrized worker_loop with classify/enrich handlers; blocking
backpressure on a full known queue (never drop a classified file)
- Stage-aware recovery: spool/ and known/ resume at their correct stage
- Ordered drain: close stage-1 and wait its workers (the known queue's
only producer) before closing the known queue
- exiftool required at startup (fail-fast); new FS_KNOWN_*/FS_UNKNOWN_DIR/
FS_EXIFTOOL_TIMEOUT config knobs; combined-pool thread warning
Each uploaded file is scored by a fixed-structure neural net that labels it
known (resembling the training set) or unknown — novelty detection over the
first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax.
- src/model.jl: shared architecture + byte->feature mapping (trainer + server)
- src/classify.jl: load committed artifact, classify a file at inference
- bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag),
seeded 80/20 split, writes model/classifier.jld2
- worker: classify (annotate-only) and log classification=known|unknown
- config: FS_MODEL_PATH; server fails fast if the artifact is missing
- deps: Lux, JLD2, Optimisers, Zygote
Julia block-buffers stderr when it isn't a TTY, so a long-running
server's logs stayed trapped in the buffer until exit whenever output
was redirected to a file/pipe (log file, tee, journald, container log
driver). This made it look like workers never ran, when in fact the
"received file" lines were only being flushed at shutdown.
Install a FlushLogger wrapper as the global logger in run(), flushing
after every message so app logs, Oxygen request logs, and startup lines
all appear in real time regardless of where stderr points.
Track the resolved dependency versions so deployments build byte-for-byte
identically. Un-ignore Manifest.toml (kept only data/ ignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
REST endpoint (Oxygen.jl POST /upload, multipart) that spools uploaded
files to disk, enqueues lightweight references onto a bounded thread-safe
work queue, and hands off immediately (202 + job IDs; 503 when full). A
configurable pool of worker threads pulls jobs off the queue, logs the
received filename (placeholder for real processing), and moves files to
done/ on success or failed/ on error.
- Queue behind an enqueue!/dequeue!/close! seam for a future RabbitMQ swap
- Startup recovery: re-enqueues leftover files in spool/
- Graceful drain on SIGINT and SIGTERM (via atexit)
- Env-var config; filenames sanitized + UUID-prefixed on disk
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
