Adds a benchmark harness, which showed that intake buffered each upload
whole, then makes intake streaming so the service's flat-memory property
holds end to end rather than only for the queue and workers.
The measurement problem first: /upload returns 202 once bytes are spooled
and a reference is enqueued, so HTTP latency measures intake, not the
pipeline. bin/bench.jl instead uploads a corpus and polls the terminal
sinks until the count stops moving, reporting intake rate and end-to-end
rate separately, sampling server RSS (kernel VmHWM, reset per run) and
the intermediate stage depths so the bottleneck stage names itself.
That exposed the buffering: HTTP.jl read the body into req.body,
parse_multipart_form materialized each part, and read(p.data) copied
again before spool_file wrote it — a 256 MiB upload grew RSS ~700 MiB,
and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB.
- src/multipart.jl: incremental multipart/form-data reader. Pulls fixed
chunks off the socket and hands each part's bytes straight to a sink,
so memory is bounded by FS_UPLOAD_CHUNK_BYTES (64 KiB), not file size.
Interface is two calls in a loop (next_part! then write_part_body! /
skip_part_body!) so the handler keeps ordinary control flow. Retains
the last length(delimiter)-1 bytes so a delimiter split across chunks
still parses; part headers are bounded by policy, not by chunking.
- src/server.jl: /upload is served by a stream handler. Oxygen's root
handler wraps HTTP.streamhandler, which does request.body =
read(stream) before dispatching — so no Oxygen route, not even a
@stream route, can stream a body. root_stream_handler intercepts
POST /upload at the stream level and delegates the rest to Oxygen
unchanged; /upload is therefore absent from Oxygen's metrics and docs.
A client hangup is classified as routine (info, not error) and answered
best-effort; every exit path drains the body so keep-alive still works.
- src/spool.jl: spool_file(bytes) -> spool_stream(write_body!, ...),
which removes a partial file on a failed or abandoned write, so restart
recovery can never pick up a truncated upload as if it were complete.
- config.jl: FS_UPLOAD_CHUNK_BYTES, the intake memory dial.
Streaming changes the 503 contract: a buffered handler knew up front how
many files a request held, this one discovers them as they arrive. When
the queue fills mid-request it no longer abandons the connection — it
stops spooling (discarding remaining parts rather than writing files it
cannot queue), drains, and answers 503 with the accepted list. Files
already queued stay queued.
Measured after (fresh server, 64 KiB chunk): 256 MiB +21.8 MiB, 1 GiB
+20.8 MiB, 2 GiB +17.0 MiB at concurrency 1 — flat across a 32x size
range; 4 concurrent 256 MiB uploads +86.9 MiB, linear in concurrency.
A 2 GiB upload sustains 334 MiB/s. Small files did not regress (intake
107 -> 133 files/s, end-to-end 27.6 -> 32.9 files/s, p95 1930 -> 776 ms).
A --size sweep that slopes upward is now the regression signal.
- Tests (235 pass, 55 new): byte-exact round-trip of 10 files in one
request, sizes straddling the chunk boundary (0/1/63/65535/65536/65537/
131072/196615/1e6) plus a payload stuffed with near-boundary sequences;
the same body parsed at chunk sizes 1..10000 to put the delimiter split
at every offset; bounded allocation on a 16 MiB part; malformed and
truncated bodies; spool_stream cleanup on a failed write.
- Known cosmetic caveat, documented: when a body is cut short, HTTP.jl's
own closeread logs an EOFError after the handler returns, because
Content-Length promised more than arrived. Not reachable from a
handler; the old code logged the same thing without replying.
Wraps the assign_file scoring core (cluster.jl) in the live catalog the
design's phase B calls for (DESIGN §5B/§9):
- src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient
stats + processed set + examples); sparse, sidecar-first durable
save/load; incremental catalog_sweep! (deterministic CRP-predictive
assignment of new binary/ files); offline compact! that seeds on first
run and recompacts later; write_nominations! emitting one JSON per
promotable cluster with a hex magic template.
- bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact
forces a recluster; first run auto-compacts to seed).
- config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG,
FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs.
- Tests: durable round-trip, incremental sweep growth + idempotency,
§10.1 nothing-from-noise end-to-end (zero promotions), a recurring
format self-nominating, seed-then-live-assign (165 pass).
- DESIGN_clustering.md: mark phase B built.
Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process
mixture of per-position categoricals over the first 32 header bytes (257-symbol
alphabet) that clusters the binary/ pile by file format, plus signature
extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps
the Dirichlet-multinomial marginal dependency-free).
- src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler
(phase A), sequential CRP-predictive assignment (phase B core), signatures/
promotion, and ARI/V-measure calibration metrics.
- bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed
ground truth and cross-checks a model-free NCD (gzip) baseline.
- FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module.
- Tests for the three DESIGN §10 assertions plus the model primitives.
Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at
ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the
NCD baseline agrees. DESIGN §11 records the results and three assumptions the
data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and
the Bernoulli signature / Occam-penalized restart scoring).
Text files sorted by stage 3 now flow onto a new work queue and worker
pool that enrich them with natural language (Languages.jl LanguageDetector:
name, ISO 639-3 code, confidence) and programming language (github-linguist),
writing a .meta.json sidecar to data/text_done/ like the stage-2 known-file
pipeline.
github-linguist reads the git blob of a path inside a repo, so untracked
data/ files are copied to /tmp (outside any repo, name preserved for
extension heuristics) before detection. Programming-language lookup is
best-effort (startup warning if missing, degraded/null on failure);
natural-language failure yields a degraded sidecar, not a quarantine.
Factored exiftool's timeout-kill into shared run_with_timeout and the
durable sidecar-first commit into commit_enriched!, both reused by stage 4.
Recovery re-drives data/text/; graceful drain closes the text queue after
its stage-3 producers finish.
The NUL-byte heuristic misfiled any non-ASCII UTF-8 text (accents, CJK,
emoji) as binary and let non-NUL control bytes through as text. is_binary
now calls a file text when its 8000-byte sniff window is valid UTF-8 with
no control bytes outside the text-safe set (tab/newline/CR/ESC/etc).
- trim_truncated_utf8 drops a multi-byte char split by the window edge so
it isn't mistaken for malformed bytes.
- NUL still classifies as binary (valid UTF-8 scalar, non-text control).
- Expanded tests: Unicode, ANSI logs, stray control byte, malformed UTF-8,
boundary-split char; updated README stage-3 description.
Unknown files are no longer terminal. Stage 1 now routes :unknown onto a
dedicated queue (with the same blocking backpressure as the known queue),
and a third worker pool sorts each file into data/binary/ or data/text/
using a NUL-byte sniff of the first 8000 bytes.
- content.jl: is_binary content sniff (stage 3)
- worker.jl: handle_unknown_job; stage-1 routes unknown with backpressure;
KNOWN_ENQUEUE_RETRY_SECONDS -> ROUTE_ENQUEUE_RETRY_SECONDS (serves both)
- config.jl: unknown_worker_count/queue_capacity, binary_dir, text_dir + env
- FileServer.jl: unknown queue, pool, stage-aware recovery, drain ordering
- tests for is_binary and handle_unknown_job; tmp_config isolates new dirs
- README: three-stage pipeline
Address code-review findings on the metadata pipeline:
- finalize_known! now fsyncs the sidecar bytes before the rename and
fsyncs done/ after, so the "file in done/ implies sidecar present"
invariant holds across power loss, not just process crashes. The
docstring previously claimed an fsync the code never performed.
- run_exiftool's timeout escalates SIGTERM -> (2s grace) -> SIGKILL, so
an exiftool that ignores SIGTERM can't pin a worker forever on
wait(proc). Previously the timeout sent only SIGTERM.
- Add test/ (48 tests) covering the correctness-critical paths:
sanitize_filename, normalize_metadata, degraded build_metadata,
real exiftool extraction, finalize_known! end-to-end, recover_dir!.
Known-classified files now flow to a second queue with its own worker pool
that extracts metadata via exiftool and writes a normalized JSON sidecar
next to the file in done/, leaving the original bytes untouched.
- Two-stage pipeline: spool/ → classify → known/ → enrich → done/;
unknowns park in unknown/ as a seam for a future pool
- src/metadata.jl: exiftool -json -G -n with timeout, normalized schema
(file_type, mime_type, author, created_by, dimensions, ...) + raw dump;
degraded sidecar on extraction failure rather than quarantine
- Sidecar-first commit so a file in done/ always has its sidecar
- Parametrized worker_loop with classify/enrich handlers; blocking
backpressure on a full known queue (never drop a classified file)
- Stage-aware recovery: spool/ and known/ resume at their correct stage
- Ordered drain: close stage-1 and wait its workers (the known queue's
only producer) before closing the known queue
- exiftool required at startup (fail-fast); new FS_KNOWN_*/FS_UNKNOWN_DIR/
FS_EXIFTOOL_TIMEOUT config knobs; combined-pool thread warning
Each uploaded file is scored by a fixed-structure neural net that labels it
known (resembling the training set) or unknown — novelty detection over the
first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax.
- src/model.jl: shared architecture + byte->feature mapping (trainer + server)
- src/classify.jl: load committed artifact, classify a file at inference
- bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag),
seeded 80/20 split, writes model/classifier.jld2
- worker: classify (annotate-only) and log classification=known|unknown
- config: FS_MODEL_PATH; server fails fast if the artifact is missing
- deps: Lux, JLD2, Optimisers, Zygote
Julia block-buffers stderr when it isn't a TTY, so a long-running
server's logs stayed trapped in the buffer until exit whenever output
was redirected to a file/pipe (log file, tee, journald, container log
driver). This made it look like workers never ran, when in fact the
"received file" lines were only being flushed at shutdown.
Install a FlushLogger wrapper as the global logger in run(), flushing
after every message so app logs, Oxygen request logs, and startup lines
all appear in real time regardless of where stderr points.
Track the resolved dependency versions so deployments build byte-for-byte
identically. Un-ignore Manifest.toml (kept only data/ ignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
REST endpoint (Oxygen.jl POST /upload, multipart) that spools uploaded
files to disk, enqueues lightweight references onto a bounded thread-safe
work queue, and hands off immediately (202 + job IDs; 503 when full). A
configurable pool of worker threads pulls jobs off the queue, logs the
received filename (placeholder for real processing), and moves files to
done/ on success or failed/ on error.
- Queue behind an enqueue!/dequeue!/close! seam for a future RabbitMQ swap
- Startup recovery: re-enqueues leftover files in spool/
- Graceful drain on SIGINT and SIGTERM (via atexit)
- Env-var config; filenames sanitized + UUID-prefixed on disk
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
