jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stream multipart intake; add throughput/memory benchmark harness

Browse Source 
Adds a benchmark harness, which showed that intake buffered each upload
whole, then makes intake streaming so the service's flat-memory property
holds end to end rather than only for the queue and workers.

The measurement problem first: /upload returns 202 once bytes are spooled
and a reference is enqueued, so HTTP latency measures intake, not the
pipeline. bin/bench.jl instead uploads a corpus and polls the terminal
sinks until the count stops moving, reporting intake rate and end-to-end
rate separately, sampling server RSS (kernel VmHWM, reset per run) and
the intermediate stage depths so the bottleneck stage names itself.

That exposed the buffering: HTTP.jl read the body into req.body,
parse_multipart_form materialized each part, and read(p.data) copied
again before spool_file wrote it — a 256 MiB upload grew RSS ~700 MiB,
and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB.

- src/multipart.jl: incremental multipart/form-data reader. Pulls fixed
  chunks off the socket and hands each part's bytes straight to a sink,
  so memory is bounded by FS_UPLOAD_CHUNK_BYTES (64 KiB), not file size.
  Interface is two calls in a loop (next_part! then write_part_body! /
  skip_part_body!) so the handler keeps ordinary control flow. Retains
  the last length(delimiter)-1 bytes so a delimiter split across chunks
  still parses; part headers are bounded by policy, not by chunking.
- src/server.jl: /upload is served by a stream handler. Oxygen's root
  handler wraps HTTP.streamhandler, which does request.body =
  read(stream) before dispatching — so no Oxygen route, not even a
  @stream route, can stream a body. root_stream_handler intercepts
  POST /upload at the stream level and delegates the rest to Oxygen
  unchanged; /upload is therefore absent from Oxygen's metrics and docs.
  A client hangup is classified as routine (info, not error) and answered
  best-effort; every exit path drains the body so keep-alive still works.
- src/spool.jl: spool_file(bytes) -> spool_stream(write_body!, ...),
  which removes a partial file on a failed or abandoned write, so restart
  recovery can never pick up a truncated upload as if it were complete.
- config.jl: FS_UPLOAD_CHUNK_BYTES, the intake memory dial.

Streaming changes the 503 contract: a buffered handler knew up front how
many files a request held, this one discovers them as they arrive. When
the queue fills mid-request it no longer abandons the connection — it
stops spooling (discarding remaining parts rather than writing files it
cannot queue), drains, and answers 503 with the accepted list. Files
already queued stay queued.

Measured after (fresh server, 64 KiB chunk): 256 MiB +21.8 MiB, 1 GiB
+20.8 MiB, 2 GiB +17.0 MiB at concurrency 1 — flat across a 32x size
range; 4 concurrent 256 MiB uploads +86.9 MiB, linear in concurrency.
A 2 GiB upload sustains 334 MiB/s. Small files did not regress (intake
107 -> 133 files/s, end-to-end 27.6 -> 32.9 files/s, p95 1930 -> 776 ms).
A --size sweep that slopes upward is now the regression signal.

- Tests (235 pass, 55 new): byte-exact round-trip of 10 files in one
  request, sizes straddling the chunk boundary (0/1/63/65535/65536/65537/
  131072/196615/1e6) plus a payload stuffed with near-boundary sequences;
  the same body parsed at chunk sizes 1..10000 to put the delimiter split
  at every offset; bounded allocation on a 16 MiB part; malformed and
  truncated bodies; spool_stream cleanup on a failed write.
- Known cosmetic caveat, documented: when a body is cut short, HTTP.jl's
  own closeread logs an EOFError after the handler returns, because
  Content-Length promised more than arrived. Not reachable from a
  handler; the old code logged the same thing without replying.
This commit is contained in:
Jeffrey Ward
2026-08-02 22:22:35 -04:00
parent e18ac45d70
commit 0d8eba05b8
8 changed files with 1371 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

113
README.md
View File

@@ -20,9 +20,9 @@ enrichment mixes CPU with a subprocess):
POST /upload (multipart)
│
▼
┌─────────────────┐ spool bytes to disk
┌─────────────────┐ stream bytes to disk (never buffered)
│ HTTP handler │────────────────────────► data/spool/<uuid>-<name>
│ (Oxygen.jl) │
│ (streaming) │
└────────┬─────────┘ enqueue reference (non-blocking)
│ │
▼ ▼
@@ -70,7 +70,12 @@ enrichment) for every file it sorts as text.
Key properties:
- **Fast intake:** the queue only ever carries small references; file bytes live
on disk, so memory stays flat regardless of file size.
on disk, so memory stays flat regardless of file size. This holds end to end:
intake **streams** each upload from the socket to the spool file a chunk at a
time (`FS_UPLOAD_CHUNK_BYTES`, default 64 KiB) rather than buffering the body,
and every worker reads only a bounded prefix. Measured: uploads of 256 MiB,
1 GiB and 2 GiB each grow resident memory by ~20 MiB — a flat line in file
size. See "Streaming intake" and "Benchmarking" below.
- **Backpressure:** each queue is bounded (default 1000). When the *intake* queue
is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
the stage-1 worker blocks and retries (a classified file is never dropped).
@@ -90,6 +95,46 @@ Key properties:
- **Safe filenames:** client-supplied names are sanitized and prefixed with a
server-minted UUID before touching the filesystem (no path traversal).
### Streaming intake
The upload endpoint never holds a file in memory. Bytes go socket → spool file in
`FS_UPLOAD_CHUNK_BYTES` chunks, so resident memory per in-flight upload is set by
the chunk size, not the file size — a 2 GiB upload costs about what a 2 KiB one
does. Two pieces make that work, and both are deliberate:
- **`src/multipart.jl` — an incremental multipart parser.** HTTP.jl's
`parse_multipart_form` takes the *complete* body as a byte vector, so using it
means every file in the request is in memory at once (and copied again per
part). The reader here pulls fixed-size chunks and hands each part's bytes
straight to its spool file. Its interface is two calls in a loop —
`next_part!` then `write_part_body!` (or `skip_part_body!`) — so the handler
keeps ordinary control flow instead of inverting into callbacks. The subtle
part is that a boundary delimiter can straddle two chunks, so the buffer always
retains the last `length(delimiter)-1` bytes; the test suite parses the same
body at chunk sizes from 1 byte upward to put that split at every offset.
- **`/upload` bypasses Oxygen's router.** Oxygen's root handler wraps
`HTTP.streamhandler`, which does `request.body = read(stream)` *before*
dispatching — even for an Oxygen `@stream` route, so no route can stream an
upload. `run` therefore passes its own `handler` to `serve`
(`root_stream_handler`), which intercepts `POST /upload` at the stream level and
delegates everything else to Oxygen unchanged. The trade-off: `/upload` is
absent from Oxygen's built-in metrics and docs.
Streaming also changes what the endpoint can promise. A buffered handler knows up
front how many files a request holds; this one discovers them as they arrive. So
when the intake queue fills mid-request it does not abandon the connection: it
stops spooling (discarding the remaining parts rather than writing files it can't
queue), drains the body, and answers `503` with the `accepted` list of whatever
got in first. Files already queued stay queued, and the client can retry the rest.
A client that hangs up mid-upload is treated as routine: the partial spool file is
removed (so restart recovery can never pick up a truncated upload as if it were
complete) and the event is logged `upload aborted by client`. One cosmetic caveat,
like the SIGTERM one below: when a request body is cut short, HTTP.jl's own
`closeread` logs an `EOFError` after the handler returns, because the connection
promised more bytes via `Content-Length` than arrived. It's harmless noise from
inside HTTP.jl — the partial file is already cleaned up and the connection closed.
### Metadata enrichment (stage 2)
Files the classifier labels **known** are handed to a second pool that extracts
@@ -374,6 +419,7 @@ init, so the artifact is exactly regenerable from the same inputs.
| `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) |
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
| `FS_UPLOAD_CHUNK_BYTES` | `65536` | Socket read size at intake; bounds intake memory per in-flight upload |
| `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed |
| `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed |
| `FS_CLUSTER_DIR` | `data/binary` | Stage-5 input: the unknown/binary pile to sweep |
@@ -410,6 +456,61 @@ Each file in a request becomes its own job. Responses:
- `503 Service Unavailable` — queue full, retry later
- `500 Internal Server Error` — failed to write a file to disk
## Benchmarking (throughput + memory)
`bin/bench.jl` measures the pipeline against a **running** server. It must run on
the same machine (it reads the sink dirs and `/proc`), and it changes nothing in
`src/` — it only speaks HTTP and counts files.
```bash
julia --project=. -t auto bin/bench.jl --files 2000 --size 8k --concurrency 32
julia --project=. -t auto bin/bench.jl --files 2 --size 1g --concurrency 1 # memory
julia --project=. -t auto bin/bench.jl --corpus ../training_set --concurrency 16
```
Two properties of this design dictate how it measures:
- **HTTP latency is not throughput.** `/upload` returns `202` once the bytes are
spooled and a reference is enqueued — all four stages run *after* the response,
so `ab`/`hey`/`wrk` would only ever measure intake. The bench instead uploads a
corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
`failed/`) until the count stops moving, and reports both numbers separately:
intake rate *and* end-to-end completion rate. It also samples the intermediate
stage dirs, so the peak depth of `spool/`/`known/`/`unknown/`/`text/` names the
bottleneck stage directly.
- **Memory should be flat in file size, and the sweep is what proves it.** Both
halves of the pipeline are bounded: the workers read bounded prefixes (16+16
bytes to classify, 8 KB to sniff, 64 KB to language-detect), and intake streams
each upload to disk a chunk at a time. So peak RSS should track *concurrency*,
not size. Measured on this machine (16 threads, 64 KiB chunk):
| upload size | concurrency | RSS growth |
|---|---|---|
| 256 MiB × 4 | 1 | 21.8 MiB |
| 1 GiB × 2 | 1 | 20.8 MiB |
| 2 GiB × 1 | 1 | 17.0 MiB |
| 256 MiB × 4 | 4 | 86.9 MiB (21.7 MiB per in-flight upload) |
Flat across a 8× range of file sizes, and linear in concurrency — which is the
shape to expect. (The residual ~20 MiB per in-flight upload is GC churn from the
chunk reads, not retained buffers; it does not grow with the file.) Before intake
was streamed, the same 256 MiB upload grew RSS by ~700 MiB and 4 concurrent ones
pushed a 950 MiB baseline past 2 GiB, so **a `--size` sweep that slopes upward is
the regression signal** — it means something has started buffering bodies again.
Flags: `--files`, `--size` (`8k`/`64m`/`1g`), `--concurrency`, `--kind`
(`binary`/`text`/`mixed` — chooses which stages get loaded), `--corpus DIR` (real
files, the only way to exercise stage 2's exiftool path), `--pid`, `--no-mem`,
`--sample-ms`, `--timeout`, `--json PATH`. Full list in the script header.
Two caveats the script reports rather than hides: it counts sink *deltas*, so it
warns if the pipeline isn't idle at the start (in-flight leftovers would be
counted as its own throughput); and because Julia's GC returns memory to the OS
lazily, a second run in the same process starts from an inflated baseline — it
resets the kernel's peak-RSS counter (`/proc/<pid>/clear_refs`) per run and flags
a drifted baseline, but for a clean growth figure restart the server between
memory runs.
## Layout
```
@@ -418,7 +519,8 @@ src/
config.jl Config struct + env parsing
job.jl Job (the queue reference)
queue.jl JobQueue seam + in-process ChannelQueue
spool.jl filename sanitizing, spool/move, startup recovery
multipart.jl streaming multipart/form-data reader (intake never buffers a file)
spool.jl filename sanitizing, streaming spool/move, startup recovery
model.jl NN architecture + byte→feature mapping (shared with trainer)
classify.jl load artifact + classify a file at inference time
metadata.jl exiftool extraction + normalized sidecar (stage 2)
@@ -427,9 +529,10 @@ src/
cluster.jl header-byte clustering model + Gibbs + scoring core (stage 5, science)
catalog.jl durable single-owner format catalog + sweep + nominations (stage 5, phase B)
worker.jl parametrized worker loop + classify/enrich/triage/language handlers
server.jl HTTP routes/handlers
server.jl HTTP routes + the streaming /upload handler
bin/
server.jl entry point
bench.jl throughput + memory harness against a running server
train.jl offline training script → model/classifier.jld2
cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline
cluster_sweep.jl stage-5 phase-B runner: sweep binary/, update catalog, write nominations