Adds a benchmark harness, which showed that intake buffered each upload
whole, then makes intake streaming so the service's flat-memory property
holds end to end rather than only for the queue and workers.
The measurement problem first: /upload returns 202 once bytes are spooled
and a reference is enqueued, so HTTP latency measures intake, not the
pipeline. bin/bench.jl instead uploads a corpus and polls the terminal
sinks until the count stops moving, reporting intake rate and end-to-end
rate separately, sampling server RSS (kernel VmHWM, reset per run) and
the intermediate stage depths so the bottleneck stage names itself.
That exposed the buffering: HTTP.jl read the body into req.body,
parse_multipart_form materialized each part, and read(p.data) copied
again before spool_file wrote it — a 256 MiB upload grew RSS ~700 MiB,
and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB.
- src/multipart.jl: incremental multipart/form-data reader. Pulls fixed
chunks off the socket and hands each part's bytes straight to a sink,
so memory is bounded by FS_UPLOAD_CHUNK_BYTES (64 KiB), not file size.
Interface is two calls in a loop (next_part! then write_part_body! /
skip_part_body!) so the handler keeps ordinary control flow. Retains
the last length(delimiter)-1 bytes so a delimiter split across chunks
still parses; part headers are bounded by policy, not by chunking.
- src/server.jl: /upload is served by a stream handler. Oxygen's root
handler wraps HTTP.streamhandler, which does request.body =
read(stream) before dispatching — so no Oxygen route, not even a
@stream route, can stream a body. root_stream_handler intercepts
POST /upload at the stream level and delegates the rest to Oxygen
unchanged; /upload is therefore absent from Oxygen's metrics and docs.
A client hangup is classified as routine (info, not error) and answered
best-effort; every exit path drains the body so keep-alive still works.
- src/spool.jl: spool_file(bytes) -> spool_stream(write_body!, ...),
which removes a partial file on a failed or abandoned write, so restart
recovery can never pick up a truncated upload as if it were complete.
- config.jl: FS_UPLOAD_CHUNK_BYTES, the intake memory dial.
Streaming changes the 503 contract: a buffered handler knew up front how
many files a request held, this one discovers them as they arrive. When
the queue fills mid-request it no longer abandons the connection — it
stops spooling (discarding remaining parts rather than writing files it
cannot queue), drains, and answers 503 with the accepted list. Files
already queued stay queued.
Measured after (fresh server, 64 KiB chunk): 256 MiB +21.8 MiB, 1 GiB
+20.8 MiB, 2 GiB +17.0 MiB at concurrency 1 — flat across a 32x size
range; 4 concurrent 256 MiB uploads +86.9 MiB, linear in concurrency.
A 2 GiB upload sustains 334 MiB/s. Small files did not regress (intake
107 -> 133 files/s, end-to-end 27.6 -> 32.9 files/s, p95 1930 -> 776 ms).
A --size sweep that slopes upward is now the regression signal.
- Tests (235 pass, 55 new): byte-exact round-trip of 10 files in one
request, sizes straddling the chunk boundary (0/1/63/65535/65536/65537/
131072/196615/1e6) plus a payload stuffed with near-boundary sequences;
the same body parsed at chunk sizes 1..10000 to put the delimiter split
at every offset; bounded allocation on a 16 MiB part; malformed and
truncated bodies; spool_stream cleanup on a failed write.
- Known cosmetic caveat, documented: when a body is cut short, HTTP.jl's
own closeread logs an EOFError after the handler returns, because
Content-Length promised more than arrived. Not reachable from a
handler; the old code logged the same thing without replying.
Wraps the assign_file scoring core (cluster.jl) in the live catalog the
design's phase B calls for (DESIGN §5B/§9):
- src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient
stats + processed set + examples); sparse, sidecar-first durable
save/load; incremental catalog_sweep! (deterministic CRP-predictive
assignment of new binary/ files); offline compact! that seeds on first
run and recompacts later; write_nominations! emitting one JSON per
promotable cluster with a hex magic template.
- bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact
forces a recluster; first run auto-compacts to seed).
- config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG,
FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs.
- Tests: durable round-trip, incremental sweep growth + idempotency,
§10.1 nothing-from-noise end-to-end (zero promotions), a recurring
format self-nominating, seed-then-live-assign (165 pass).
- DESIGN_clustering.md: mark phase B built.
Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process
mixture of per-position categoricals over the first 32 header bytes (257-symbol
alphabet) that clusters the binary/ pile by file format, plus signature
extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps
the Dirichlet-multinomial marginal dependency-free).
- src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler
(phase A), sequential CRP-predictive assignment (phase B core), signatures/
promotion, and ARI/V-measure calibration metrics.
- bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed
ground truth and cross-checks a model-free NCD (gzip) baseline.
- FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module.
- Tests for the three DESIGN §10 assertions plus the model primitives.
Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at
ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the
NCD baseline agrees. DESIGN §11 records the results and three assumptions the
data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and
the Bernoulli signature / Occam-penalized restart scoring).
Each uploaded file is scored by a fixed-structure neural net that labels it
known (resembling the training set) or unknown — novelty detection over the
first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax.
- src/model.jl: shared architecture + byte->feature mapping (trainer + server)
- src/classify.jl: load committed artifact, classify a file at inference
- bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag),
seeded 80/20 split, writes model/classifier.jld2
- worker: classify (annotate-only) and log classification=known|unknown
- config: FS_MODEL_PATH; server fails fast if the artifact is missing
- deps: Lux, JLD2, Optimisers, Zygote
REST endpoint (Oxygen.jl POST /upload, multipart) that spools uploaded
files to disk, enqueues lightweight references onto a bounded thread-safe
work queue, and hands off immediately (202 + job IDs; 503 when full). A
configurable pool of worker threads pulls jobs off the queue, logs the
received filename (placeholder for real processing), and moves files to
done/ on success or failed/ on error.
- Queue behind an enqueue!/dequeue!/close! seam for a future RabbitMQ swap
- Startup recovery: re-enqueues leftover files in spool/
- Graceful drain on SIGINT and SIGTERM (via atexit)
- Env-var config; filenames sanitized + UUID-prefixed on disk
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
