jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stream multipart intake; add throughput/memory benchmark harness

Browse Source 
Adds a benchmark harness, which showed that intake buffered each upload
whole, then makes intake streaming so the service's flat-memory property
holds end to end rather than only for the queue and workers.

The measurement problem first: /upload returns 202 once bytes are spooled
and a reference is enqueued, so HTTP latency measures intake, not the
pipeline. bin/bench.jl instead uploads a corpus and polls the terminal
sinks until the count stops moving, reporting intake rate and end-to-end
rate separately, sampling server RSS (kernel VmHWM, reset per run) and
the intermediate stage depths so the bottleneck stage names itself.

That exposed the buffering: HTTP.jl read the body into req.body,
parse_multipart_form materialized each part, and read(p.data) copied
again before spool_file wrote it — a 256 MiB upload grew RSS ~700 MiB,
and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB.

- src/multipart.jl: incremental multipart/form-data reader. Pulls fixed
  chunks off the socket and hands each part's bytes straight to a sink,
  so memory is bounded by FS_UPLOAD_CHUNK_BYTES (64 KiB), not file size.
  Interface is two calls in a loop (next_part! then write_part_body! /
  skip_part_body!) so the handler keeps ordinary control flow. Retains
  the last length(delimiter)-1 bytes so a delimiter split across chunks
  still parses; part headers are bounded by policy, not by chunking.
- src/server.jl: /upload is served by a stream handler. Oxygen's root
  handler wraps HTTP.streamhandler, which does request.body =
  read(stream) before dispatching — so no Oxygen route, not even a
  @stream route, can stream a body. root_stream_handler intercepts
  POST /upload at the stream level and delegates the rest to Oxygen
  unchanged; /upload is therefore absent from Oxygen's metrics and docs.
  A client hangup is classified as routine (info, not error) and answered
  best-effort; every exit path drains the body so keep-alive still works.
- src/spool.jl: spool_file(bytes) -> spool_stream(write_body!, ...),
  which removes a partial file on a failed or abandoned write, so restart
  recovery can never pick up a truncated upload as if it were complete.
- config.jl: FS_UPLOAD_CHUNK_BYTES, the intake memory dial.

Streaming changes the 503 contract: a buffered handler knew up front how
many files a request held, this one discovers them as they arrive. When
the queue fills mid-request it no longer abandons the connection — it
stops spooling (discarding remaining parts rather than writing files it
cannot queue), drains, and answers 503 with the accepted list. Files
already queued stay queued.

Measured after (fresh server, 64 KiB chunk): 256 MiB +21.8 MiB, 1 GiB
+20.8 MiB, 2 GiB +17.0 MiB at concurrency 1 — flat across a 32x size
range; 4 concurrent 256 MiB uploads +86.9 MiB, linear in concurrency.
A 2 GiB upload sustains 334 MiB/s. Small files did not regress (intake
107 -> 133 files/s, end-to-end 27.6 -> 32.9 files/s, p95 1930 -> 776 ms).
A --size sweep that slopes upward is now the regression signal.

- Tests (235 pass, 55 new): byte-exact round-trip of 10 files in one
  request, sizes straddling the chunk boundary (0/1/63/65535/65536/65537/
  131072/196615/1e6) plus a payload stuffed with near-boundary sequences;
  the same body parsed at chunk sizes 1..10000 to put the delimiter split
  at every offset; bounded allocation on a 16 MiB part; malformed and
  truncated bodies; spool_stream cleanup on a failed write.
- Known cosmetic caveat, documented: when a body is cut short, HTTP.jl's
  own closeread logs an EOFError after the handler returns, because
  Content-Length promised more than arrived. Not reachable from a
  handler; the old code logged the same thing without replying.
This commit is contained in:
Jeffrey Ward
2026-08-02 22:22:35 -04:00
parent e18ac45d70
commit 0d8eba05b8
8 changed files with 1371 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

171
test/runtests.jl
View File

@@ -18,7 +18,10 @@ using FileServer: Job, Config, ChannelQueue, enqueue!, dequeue!, length,
adjusted_rand_index, v_measure, HEADER_N, ALPHABET, PAST_EOF,
Catalog, load_catalog, save_catalog!, catalog_sweep!, compact!,
write_nominations!, run_cluster_sweep, binary_files, record_example!,
signature_hex, ensure_dirs
signature_hex, ensure_dirs,
MultipartReader, MultipartError, MultipartPart, next_part!,
write_part_body!, skip_part_body!, multipart_boundary,
parse_part_headers, spool_stream, UPLOAD_CHUNK_BYTES
using Random: MersenneTwister
using Languages: LanguageDetector
@@ -28,6 +31,39 @@ const PNG_1x1 = UInt8[137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82,0,0,0,1,0,
0,0,1,8,6,0,0,0,31,21,196,137,0,0,0,11,73,68,65,84,120,218,99,100,248,255,
191,30,0,5,132,2,127,194,91,30,42,0,0,0,0,73,69,78,68,174,66,96,130]
"""
Assemble a multipart/form-data body. `parts` are `(name, filename, content_type,
data)` tuples; a `nothing` filename makes a plain form field rather than a file.
"""
function multipart_body(boundary, parts; preamble = "", terminate = true)
io = IOBuffer()
write(io, preamble)
for (name, filename, content_type, data) in parts
write(io, "--$boundary\r\n")
write(io, "Content-Disposition: form-data; name=\"$name\"")
filename === nothing || write(io, "; filename=\"$filename\"")
write(io, "\r\n")
content_type === nothing || write(io, "Content-Type: $content_type\r\n")
write(io, "\r\n")
write(io, data)
write(io, "\r\n")
end
write(io, terminate ? "--$boundary--\r\n" : "--$boundary\r\n")
return take!(io)
end
"Read every part out of `bytes`, returning `(part, body, nbytes)` triples."
function read_all_parts(bytes, boundary; chunk_bytes = UPLOAD_CHUNK_BYTES)
r = MultipartReader(IOBuffer(bytes), boundary; chunk_bytes = chunk_bytes)
out = Tuple{MultipartPart,String,Int}[]
while (part = next_part!(r)) !== nothing
sink = IOBuffer()
n = write_part_body!(sink, r)
push!(out, (part, String(take!(sink)), n))
end
return out
end
"Build a Config whose data dirs all live under a fresh temp directory."
function tmp_config(root; kwargs...)
cfg = Config(;
@@ -65,6 +101,139 @@ end
@test Base.length(sanitize_filename("a"^500)) == FileServer.MAX_NAME_LEN
end
@testset "multipart_boundary: extraction from Content-Type" begin
@test multipart_boundary("multipart/form-data; boundary=abc") == "abc"
@test multipart_boundary("multipart/form-data; boundary=\"a b;c\"") == "a b;c"
@test multipart_boundary("MULTIPART/FORM-DATA; BOUNDARY=xyz") == "xyz"
@test multipart_boundary("multipart/form-data; charset=utf-8; boundary=q1") == "q1"
# Anything that isn't a usable multipart header is the same 400 to a caller.
@test multipart_boundary("multipart/form-data") === nothing
@test multipart_boundary("application/json") === nothing
@test multipart_boundary(nothing) === nothing
end
@testset "parse_part_headers" begin
p = parse_part_headers("Content-Disposition: form-data; name=\"f\"; filename=\"a b.txt\"\r\n" *
"Content-Type: text/plain")
@test p.name == "f"
@test p.filename == "a b.txt"
@test p.content_type == "text/plain"
# `name=` must not match inside `filename=` — that would label every
# file part with a bogus name and (worse) hide a missing real name.
p = parse_part_headers("Content-Disposition: form-data; filename=\"only.txt\"")
@test p.name === nothing
@test p.filename == "only.txt"
# No filename means a plain form field, which intake must not spool.
p = parse_part_headers("Content-Disposition: form-data; name=\"note\"")
@test p.filename === nothing
end
@testset "MultipartReader: parts, fields, and bodies" begin
B = "----testboundary"
body = multipart_body(B, [("f0", "a.txt", "text/plain", "hello world"),
("note", nothing, nothing, "just-a-field"),
("f1", "b.bin", nothing, "\x00\x01\x02")])
got = read_all_parts(body, B)
@test length(got) == 3
@test got[1][1].filename == "a.txt"
@test got[1][2] == "hello world"
@test got[1][3] == 11 # reported byte count
@test got[1][1].content_type == "text/plain"
@test got[2][1].filename === nothing # the form field
@test got[2][2] == "just-a-field"
@test codeunits(got[3][2]) == UInt8[0x00, 0x01, 0x02]
# A zero-byte file is legal and must survive as zero bytes.
empty_got = read_all_parts(multipart_body(B, [("f", "empty.bin", nothing, "")]), B)
@test empty_got[1][3] == 0
@test empty_got[1][2] == ""
end
@testset "MultipartReader: delimiter straddling every chunk offset" begin
# The one thing a chunked parser can get catastrophically wrong is a
# delimiter split across two reads. Parsing the same body at many chunk
# sizes puts the split at every offset. The payload deliberately contains
# CR, LF and '-' bytes, so a sloppy scan finds false delimiters.
B = "----testboundary"
rng = MersenneTwister(7)
payload = String(rand(rng, UInt8[0x41:0x5a; 0x0d; 0x0a; 0x2d], 5000))
body = multipart_body(B, [("f", "big.bin", nothing, payload)])
for chunk in (1, 2, 3, 5, 7, 13, 16, 17, 64, 255, 4096, 10_000)
got = read_all_parts(body, B; chunk_bytes = chunk)
@test length(got) == 1
@test got[1][2] == payload
end
# A payload containing a *prefix* of the real delimiter must not end the part.
tricky = "aaa\r\n--" * "----testboundar" * "bbb\r\n--x\r\nccc"
body2 = multipart_body(B, [("f", "t.bin", nothing, tricky)])
for chunk in (1, 4, 9, 64, 4096)
got = read_all_parts(body2, B; chunk_bytes = chunk)
@test length(got) == 1
@test got[1][2] == tricky
end
end
@testset "MultipartReader: memory stays bounded, not proportional to the part" begin
# The whole point of the streaming reader. A 16 MiB part read with a
# 64 KiB chunk must allocate on the order of the chunk, not the part.
B = "----testboundary"
payload = String(rand(MersenneTwister(11), UInt8, 16 * 1024 * 1024))
body = multipart_body(B, [("f", "huge.bin", nothing, payload)])
r = MultipartReader(IOBuffer(body), B; chunk_bytes = 64 * 1024)
next_part!(r)
GC.gc()
allocated = @allocated write_part_body!(devnull, r)
@test allocated < 4 * 1024 * 1024
end
@testset "MultipartReader: malformed bodies raise MultipartError" begin
B = "----testboundary"
valid = multipart_body(B, [("f", "a.bin", nothing, "hello")])
@test_throws MultipartError read_all_parts(Vector{UInt8}("no delimiter here"), B)
@test_throws MultipartError read_all_parts(valid[1:end-20], B) # truncated mid-part
@test_throws MultipartError read_all_parts(
multipart_body(B, [("f", "a.bin", nothing, "x")]; terminate = false), B)
# Part headers must be bounded regardless of how the body was chunked,
# since they are the one thing that has to be buffered whole to parse.
oversized = Vector{UInt8}("--$B\r\nContent-Disposition: form-data; name=\"" *
"x"^30_000 * "\"\r\n\r\ndata\r\n--$B--\r\n")
@test_throws MultipartError read_all_parts(oversized, B)
# A part's body must be consumed before advancing: the reader cannot skip
# a body on its own, because a body only ends at the next delimiter.
r = MultipartReader(IOBuffer(valid), B)
next_part!(r)
@test_throws MultipartError next_part!(r)
end
@testset "spool_stream: streams to disk, cleans up a failed write" begin
mktempdir() do root
cfg = tmp_config(root)
job = spool_stream(cfg, "report v2.pdf") do io
write(io, "abc") + write(io, "de")
end
@test isfile(job.path)
@test read(job.path, String) == "abcde"
@test job.size == 5 # from the bytes actually written
@test job.original_name == "report v2.pdf"
@test basename(job.path) == "$(job.id)-report_v2.pdf" # sanitized, uuid-prefixed
# A write that throws must leave nothing behind: recovery on restart
# re-enqueues whatever is in spool/, and a truncated upload there
# would be silently processed as if it were complete.
before = length(readdir(cfg.spool_dir))
@test_throws ErrorException spool_stream(cfg, "bad.bin") do io
write(io, "partial")
error("disk went away")
end
@test length(readdir(cfg.spool_dir)) == before
end
end
@testset "normalize_metadata" begin
job = Job("id-1", "photo.jpg", "/data/known/id-1-photo.jpg", 4242, 0.0)
# Group-prefixed tags as exiftool -G emits them are already group-stripped