# FileServer A minimal Julia service that receives files over HTTP and hands them off to a pool of worker threads for processing. The HTTP endpoint does no real work: it spools each uploaded file to disk, pushes a lightweight reference onto a work queue, and responds immediately — staying free to accept the next upload. The per-file "processing" runs each file through a small neural-network classifier that labels it **known** (a file type resembling the training set) or **unknown**, and logs the result. See "File classifier" below. ## Architecture The pipeline is four stages, each with its own bounded queue and its own worker pool (tuned independently, since classification is CPU-bound, known-file enrichment is process-/IO-bound, content triage is cheap IO, and language enrichment mixes CPU with a subprocess): ``` POST /upload (multipart) │ ▼ ┌─────────────────┐ stream bytes to disk (never buffered) │ HTTP handler │────────────────────────► data/spool/- │ (streaming) │ └────────┬─────────┘ enqueue reference (non-blocking) │ │ ▼ ▼ 202 + job IDs ┌────────────────────┐ (503 if full) │ stage-1 queue │ classification └─────────┬──────────┘ │ dequeue ┌───────────────────┼───────────────────┐ ▼ ▼ ▼ classify wkr 1 classify wkr 2 … classify wkr N │ ┌────────────┴────────────┐ :unknown :known │ move to data/unknown/, │ move to data/known/, then ▼ then enqueue (blocking) ▼ enqueue (blocking backpressure) ┌────────────────────┐ ┌────────────────────┐ │ unknown queue │ │ known queue │ enrichment └─────────┬──────────┘ └─────────┬──────────┘ │ dequeue │ dequeue ┌────────┼────────┐ ┌───────────┼───────────┐ ▼ ▼ ▼ ▼ ▼ ▼ unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M │ binary-vs-text sniff │ exiftool → normalized sidecar ├─► data/binary/- success ──┴──► data/done/- │ (terminal) data/done/-.meta.json │ (sidecar-first commit) │ :text move to data/text/, failure ───────► data/failed/- ▼ then enqueue (blocking backpressure) ┌────────────────────┐ │ text queue │ language enrichment └─────────┬──────────┘ │ dequeue ┌────────┼────────┐ ▼ ▼ ▼ txt 1 txt 2 … txt P │ Languages.jl (natural language) + github-linguist (programming language) └─► data/text_done/- + data/text_done/-.meta.json (sidecar-first commit) ``` Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1 feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language enrichment) for every file it sorts as text. Key properties: - **Fast intake:** the queue only ever carries small references; file bytes live on disk, so memory stays flat regardless of file size. This holds end to end: intake **streams** each upload from the socket to the spool file a chunk at a time (`FS_UPLOAD_CHUNK_BYTES`, default 64 KiB) rather than buffering the body, and every worker reads only a bounded prefix. Measured: uploads of 256 MiB, 1 GiB and 2 GiB each grow resident memory by ~20 MiB — a flat line in file size. See "Streaming intake" and "Benchmarking" below. - **Backpressure:** each queue is bounded (default 1000). When the *intake* queue is full, uploads get `503 Service Unavailable`. When the *known* queue is full, the stage-1 worker blocks and retries (a classified file is never dropped). - **Crash-resilient:** files survive on disk. On startup, recovery is stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/` re-enter enrichment, `data/unknown/` re-enter content triage, and `data/text/` re-enter language enrichment (`recovered` / `recovered_known` / `recovered_unknown` / `recovered_text` in the log), so a file resumes at its correct stage instead of restarting from scratch. - **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`) both stop accepting uploads, then drain the stages *in order* — close the stage-1 queue and wait out the classify workers (the only producer of the known *and* unknown queues), then close those queues and wait out the enrich and content-triage workers (content triage being the only producer of the text queue), then close the text queue and wait out the language-enrichment workers. (See "Shutdown" below for one cosmetic caveat on SIGTERM.) - **Safe filenames:** client-supplied names are sanitized and prefixed with a server-minted UUID before touching the filesystem (no path traversal). ### Streaming intake The upload endpoint never holds a file in memory. Bytes go socket → spool file in `FS_UPLOAD_CHUNK_BYTES` chunks, so resident memory per in-flight upload is set by the chunk size, not the file size — a 2 GiB upload costs about what a 2 KiB one does. Two pieces make that work, and both are deliberate: - **`src/multipart.jl` — an incremental multipart parser.** HTTP.jl's `parse_multipart_form` takes the *complete* body as a byte vector, so using it means every file in the request is in memory at once (and copied again per part). The reader here pulls fixed-size chunks and hands each part's bytes straight to its spool file. Its interface is two calls in a loop — `next_part!` then `write_part_body!` (or `skip_part_body!`) — so the handler keeps ordinary control flow instead of inverting into callbacks. The subtle part is that a boundary delimiter can straddle two chunks, so the buffer always retains the last `length(delimiter)-1` bytes; the test suite parses the same body at chunk sizes from 1 byte upward to put that split at every offset. - **`/upload` bypasses Oxygen's router.** Oxygen's root handler wraps `HTTP.streamhandler`, which does `request.body = read(stream)` *before* dispatching — even for an Oxygen `@stream` route, so no route can stream an upload. `run` therefore passes its own `handler` to `serve` (`root_stream_handler`), which intercepts `POST /upload` at the stream level and delegates everything else to Oxygen unchanged. The trade-off: `/upload` is absent from Oxygen's built-in metrics and docs. Streaming also changes what the endpoint can promise. A buffered handler knows up front how many files a request holds; this one discovers them as they arrive. So when the intake queue fills mid-request it does not abandon the connection: it stops spooling (discarding the remaining parts rather than writing files it can't queue), drains the body, and answers `503` with the `accepted` list of whatever got in first. Files already queued stay queued, and the client can retry the rest. A client that hangs up mid-upload is treated as routine: the partial spool file is removed (so restart recovery can never pick up a truncated upload as if it were complete) and the event is logged `upload aborted by client`. One cosmetic caveat, like the SIGTERM one below: when a request body is cut short, HTTP.jl's own `closeread` logs an `EOFError` after the handler returns, because the connection promised more bytes via `Content-Length` than arrived. It's harmless noise from inside HTTP.jl — the partial file is already cleaned up and the connection closed. ### Metadata enrichment (stage 2) Files the classifier labels **known** are handed to a second pool that extracts metadata with [`exiftool`](https://exiftool.org/) (`exiftool -json -G -n`) — chosen because no native Julia library comes close to its multi-format coverage. The output is normalized into a small, stable, documented schema and written as a JSON **sidecar** next to the file in `data/done/`, e.g. `data/done/-.meta.json`. The original bytes are never modified. > **Prerequisite:** `exiftool` must be on `PATH` (e.g. `apt install > libimage-exiftool-perl`). The server **fails fast at startup** if it's missing. Sidecar top-level fields (all nullable — present only when available), plus the complete raw `exiftool` object under `raw`: | Field | Meaning | |---|---| | `id`, `original_name` | job id and client-supplied name | | `file_type`, `mime_type` | e.g. `PDF` / `application/pdf` | | `file_size` | bytes (authoritative, from intake — not exiftool) | | `created_date`, `modified_date` | content timestamps | | `author` | person (`Author`/`Artist`/`By-line`) | | `created_by` | authoring app/tool (`Producer`/`CreatorTool`/`Creator`/`Software`/…) | | `dimensions` | `{width, height}` for media | | `duration` | seconds, for audio/video | | `page_count` | for documents | | `error` | set on a *degraded* sidecar (see below) | | `raw` | full `exiftool` output | Each normalized field is a coalesce over a priority list of exiftool tags (`src/metadata.jl`); extend a field by appending tag names. If extraction fails or `exiftool` times out (`FS_EXIFTOOL_TIMEOUT`, default 30s), the file still completes to `data/done/` with a **degraded sidecar** — `file_size`/`file_type` plus an `error` note — rather than being quarantined, because it's still a wanted known file. Only genuine I/O errors (can't write the sidecar or move the file) send it to `data/failed/`. The sidecar is committed **before** the file is moved into `data/done/`, so a file's presence there always implies its sidecar is already present; a crash in between leaves only a harmless orphan sidecar, and recovery re-enriches idempotently. ### Content triage (stage 3) Files the classifier labels **unknown** are handed to a third pool that sorts them into two coarse buckets so downstream tooling can treat them differently: - **`data/binary/`** — the file looks like binary data. - **`data/text/`** — the file looks like text. The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text when that window is valid UTF-8 and holds no control bytes outside the text-safe set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise binary. It's cheap (no full read) and Unicode-aware — unlike the older NUL-byte or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in `text/` instead of misfiling it, while binary formats — which rarely form valid UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but not a text control byte, so it still reads as binary. A multi-byte character split by the 8000-byte boundary is trimmed before the check so it isn't mistaken for malformed bytes. An empty file is treated as text. `binary/` is terminal on the live path (but is the input the offline **stage-5 discovery** sweeps — see below); `text/` is handed to stage 4 (`src/content.jl`). ### Language enrichment (stage 4) Files that stage 3 sorts as **text** are handed to a fourth pool that identifies their language and writes a `.meta.json` sidecar, mirroring the stage-2 known-file enrichment. Two detectors run per file: - **natural language** — [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s `LanguageDetector` (a Julia port of the `whatlang` n-gram model) reads a bounded prefix (up to `LANG_SAMPLE_BYTES`, 64 KiB) and reports the language's English name, ISO 639-3 code, and a confidence in `[0,1]`. Pure Julia, no subprocess. The detector is built once at startup and shared read-only across the pool. - **programming language** — the [`github-linguist`](https://github.com/github-linguist/linguist) CLI recognizes source and markup by extension + content heuristics (e.g. `Python`, `Markdown`). Plain prose reports as `Text` and unrecognized content as `null`; both collapse to *no programming language*. The sidecar schema: | field | meaning | |---|---| | `id`, `original_name` | from intake | | `file_size` | bytes (authoritative, from intake) | | `content_type` | always `"text"` | | `language` | natural-language English name (e.g. `English`), or `null` | | `language_code` | ISO 639-3 code (e.g. `eng`), or `null` | | `language_confidence` | detector confidence in `[0,1]`, or `null` | | `programming_language` | e.g. `Python`, `Markdown`, or `null` | | `error` | set if natural-language detection produced nothing | > **`github-linguist` and the git-repo quirk:** run against a path *inside* a git > repository, linguist reads the file's committed git blob, not the on-disk bytes > — and an untracked file (which everything under `data/` is) has no blob, so it > crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under > `/tmp` (outside any repo, preserving the name so extension heuristics still > fire) and pointing linguist there. > > Programming-language detection is **best-effort**: if `github-linguist` is > missing (a startup warning, not a fatal error, unlike `exiftool`), fails, or > times out (`FS_LINGUIST_TIMEOUT`, default 30s), `programming_language` is simply > `null` and the file still completes. Natural-language detection failing produces > a **degraded sidecar** (with an `error` note) rather than a quarantine, because > the file is still wanted text. Like stage 2, the sidecar is committed **before** the file is moved into `data/text_done/`, so the file's presence there always implies its sidecar is present; recovery re-enriches idempotently (`src/language.jl`). ### Unknown-format discovery (stage 5, offline) The `binary/` sink from stage 3 is the pile of genuinely *unrecognized* files. Stage 5 mines it for **recurring new file formats** by clustering files on their header bytes — a growing catalog of discovered formats, each with a magic-byte signature that can eventually be promoted into the classifier's fast path. Unlike stages 1–4 it is **not on the request hot path**: it is a single-owner *batch* process (the catalog is mutable shared state, the opposite of the stateless classifier), and because promotion is human-gated nothing here is latency-sensitive. The full rationale — and the assumptions we deliberately rejected — live in [`model/DESIGN_clustering.md`](model/DESIGN_clustering.md). The model (`src/cluster.jl`, base-Julia, no extra deps) is a Dirichlet-process mixture of **per-position categoricals** over the first 32 header bytes, on a 257-symbol alphabet (byte `0–255` plus a `past-EOF` symbol so short fixed-length formats are modeled honestly). Bytes are treated as **categorical, not numeric** — `0x89` and `0x88` are not "close" — so this deliberately does *not* reuse the classifier's `[0,1]` byte scaling. A fixed uniform **background** component absorbs structureless (compressed/encrypted) blobs so they don't mint spurious clusters. A cluster's spiked positions become a libmagic-style signature; clusters with enough members and enough fixed positions self-**nominate** for promotion (a human does the one irreversible step, redefining "known"). **Status:** both phases are implemented and calibrated. Phase A (offline Gibbs) is the science; phase B (`src/catalog.jl`) is the live catalog: a durable single-owner state that sweeps `binary/`, folds each new file into a cluster with the deterministic CRP-predictive rule, and writes promotion nominations. The catalog process is run periodically (cron), single-threaded — it is the **only** writer of the catalog, so it needs no locking: ```bash julia --project=. bin/cluster_sweep.jl # incremental live sweep of new binary/ files julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs re-cluster (seed / recompact) ``` The catalog is a single durable file (`FS_CLUSTER_CATALOG`, default `data/catalog.json`) committed with the same sidecar-first temp→fsync→rename→fsync-dir discipline as the stage-2 sidecars, so a crash can neither corrupt it nor lose a write. On the **first** run (empty catalog) the sweep auto-promotes to a `--compact` pass to seed clusters; later runs assign incrementally, touching only files they have not seen. A cluster that clears the member/magic thresholds writes a nomination — a hex magic template, member count, and example filenames — into `FS_NOMINATED_DIR` (default `data/nominated/`) for a human to glance at and promote. Under the calibrated `bg_mass > α`, the live sweep never mints single-file clusters; genuinely new formats surface from the periodic `--compact` re-clustering of the background residue, not the live path. Calibration is its own offline script (like training — never in the request path), scored against magic-collapsed ground truth (so `docx`≡`zip` and the whole ELF family count as one format each, which is the *correct* answer, not an error): ```bash julia --project=. bin/cluster_calibrate.jl [training_set_dir] # defaults to ../training_set ``` It grid-tunes the hyperparameters to maximize Adjusted Rand Index against known formats and cross-checks against a model-free NCD (gzip) baseline. On the 700-file training corpus the calibrated defaults (`n=32`, `α=1.0`, `β=0.1`) recover the known formats at **ARI 0.77** (0.885 excluding tar), with `gzip`, `pkzip` (`docx`+`zip` merged), and `jpeg` forming clean, promotable clusters; the NCD baseline agrees. See `DESIGN_clustering.md` §11 for the full results, including the one known limitation (ELF and these tarballs share a long run of header zero- padding and merge — the v2 fix is inverse-entropy position weighting). ## The queue seam (→ RabbitMQ later) The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and `close!` on a `JobQueue` (see `src/queue.jl`). Today that's an in-process `ChannelQueue`. To move to RabbitMQ (or any broker), implement a new `JobQueue` subtype with those three methods and swap the construction in `run` — no handler or worker code changes. ## Running ```bash # install deps (first time) julia --project=. -e 'using Pkg; Pkg.instantiate()' # external tools: exiftool (stage 2, required) and github-linguist (stage 4, # optional — programming-language detection). e.g. on Debian/Ubuntu: # apt install libimage-exiftool-perl # gem install github-linguist # start the server; -t sets the number of OS threads available to workers julia --project=. -t auto bin/server.jl ``` ## Shutdown Both SIGINT and SIGTERM trigger the same idempotent graceful drain (stop serving → close queue → wait for workers → exit): - **SIGINT** is caught as an `InterruptException` (we call `Base.exit_on_sigint(false)`), so shutdown is clean and quiet. - **SIGTERM** can't be intercepted directly — Julia blocks it on worker threads and handles it in its own runtime, so a user `signal()` handler never fires. Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path does run. Caveat: Julia prints its own `signal 15: Terminated` backtrace *before* `atexit` runs. It's harmless noise — the drain still completes right after it — but if you want a fully quiet stop under a process manager, configure it to send SIGINT instead (systemd: `KillSignal=SIGINT`; Docker: `STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain in-flight work (systemd: `TimeoutStopSec`). ## File classifier Each file is scored by a fixed-structure neural network (Lux.jl) that answers a single binary question: is this file **known** (like the types in the training set) or **unknown**? It's novelty detection, not exact file-typing — it won't tell you "PDF", just "this looks like something I was trained on, or not". - **Features:** the first 16 bytes + last 16 bytes of the file, each scaled 0–255 → `[0,1]`, giving a 32-dim input. Files under 32 bytes can't form that window and are classified `unknown` without touching the model. - **Architecture:** `Dense(32→64,relu) → Dense(64→16,relu) → Dense(16→2)`, raw logits; decision is `argmax` (class 1 = known, class 2 = unknown). - **Artifact:** trained weights live in `model/classifier.jld2` (committed), so the server just loads them at startup. Missing/unreadable ⇒ the server fails fast rather than run without classification. - **Effect today:** *active routing*. The class is logged (`classification=known|unknown`) and drives the pipeline split: `:known` files go to `known/` for metadata enrichment (stage 2), `:unknown` files go to `unknown/` for content triage (stage 3). The class chooses the downstream stage; what's still unproven is the model's *accuracy*, not whether the routing path runs. The architecture and byte→feature mapping are defined once in `src/model.jl` and shared by the trainer and the server, so they can't drift apart. ### Training Training is a separate, offline script — it never runs in the request path: ```bash julia --project=. bin/train.jl [negatives_dir] ``` - **positives_dir** — every file in it (≥32 bytes) is a "known" example. - **negatives_dir** *(optional)* — a grab-bag of *other* real file types used as "unknown" examples. Negatives are generated ~1:1 with positives, split 50/50 between uniform-random byte vectors and grab-bag files. With no grab-bag dir, negatives are all random (weaker: the net may just learn "high entropy = unknown" rather than your actual types, so a grab-bag of real off-distribution files is recommended). The script uses an 80/20 seeded split, reports validation accuracy, and writes `model/classifier.jld2` (path overridable via `FS_MODEL_PATH`). A fixed seed (`FS_TRAIN_SEED`, default 42) drives negative generation, the split, and weight init, so the artifact is exactly regenerable from the same inputs. ## Configuration (environment variables) | Variable | Default | Meaning | |---------------------|----------------|------------------------------------------| | `FS_HOST` | `127.0.0.1` | Bind address | | `FS_PORT` | `8080` | Port | | `FS_WORKERS` | `nthreads()` | Stage-1 (classification) worker tasks | | `FS_QUEUE_CAPACITY` | `1000` | Max pending intake jobs before `503` | | `FS_KNOWN_WORKERS` | `nthreads()` | Stage-2 (enrichment) worker tasks | | `FS_KNOWN_QUEUE_CAPACITY` | `1000` | Max pending enrichment jobs (then backpressure) | | `FS_UNKNOWN_WORKERS` | `nthreads()` | Stage-3 (content triage) worker tasks | | `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) | | `FS_TEXT_WORKERS` | `nthreads()` | Stage-4 (language enrichment) worker tasks | | `FS_TEXT_QUEUE_CAPACITY` | `1000` | Max pending language jobs (then backpressure) | | `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending classification) | | `FS_KNOWN_DIR` | `data/known` | Classified-known, awaiting enrichment | | `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, awaiting content triage | | `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary | | `FS_TEXT_DIR` | `data/text` | Classified-text, awaiting language enrichment | | `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) | | `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) | | `FS_FAILED_DIR` | `data/failed` | Files whose processing threw | | `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup | | `FS_UPLOAD_CHUNK_BYTES` | `65536` | Socket read size at intake; bounds intake memory per in-flight upload | | `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed | | `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed | | `FS_CLUSTER_DIR` | `data/binary` | Stage-5 input: the unknown/binary pile to sweep | | `FS_CLUSTER_N` | `32` | Header bytes modeled per file | | `FS_CLUSTER_ALPHA` | `1.0` | CRP concentration (propensity to spawn new formats) | | `FS_CLUSTER_PSEUDOCOUNT` | `0.1` | Dirichlet pseudocount β (calibrated) | | `FS_CLUSTER_BG_MASS` | `5.0` | Mass of the uniform background component | | `FS_PROMOTE_MIN_MEMBERS` | `20` | Cluster size threshold for promotion nomination | | `FS_PROMOTE_MIN_MAGIC` | `3` | Required fixed signature positions to nominate | | `FS_CLUSTER_CATALOG` | `data/catalog.json` | Durable stage-5 catalog file (single-owner) | | `FS_NOMINATED_DIR` | `data/nominated` | One JSON per self-nominated cluster (human promote gate) | > To get real parallelism, start Julia with enough threads (`-t N`) to cover all > pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS` > exceeds available threads you'll get a warning (non-fatal) and workers will > share threads. ## Usage ```bash # health check curl http://127.0.0.1:8080/health # {"status":"ok"} # upload one or more files (multipart/form-data) curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload # 202 {"accepted":[{"id":"","name":"report.pdf"}, ...]} # per-stage counters curl http://127.0.0.1:8080/stats ``` Each file in a request becomes its own job. Responses: - `202 Accepted` — all files spooled and queued (with per-file job IDs) - `400 Bad Request` — not multipart, or no files present - `503 Service Unavailable` — queue full, retry later - `500 Internal Server Error` — failed to write a file to disk ### `GET /stats` — per-stage counters The pipeline counts its own work (`src/stats.jl`), because nothing outside it can: `known/`, `unknown/` and `text/` are *transient*, so a file can cross one between two directory polls and an external sampler will miss exactly the stages you most want to measure. ```jsonc { "now": 1785725300.5, "since": 1785725291.9, "uptime_seconds": 8.5, "intake": { "requests": 400, "files": 400, "bytes": 6553600, "rejected": 0 }, "stages": [ { "stage": 4, "name": "language", "workers": 16, "queue_depth": 184, "queue_capacity": 1000, "completed": 200, "failed": 0, "bytes": 3276800, "busy_seconds": 170.5, // summed handler time across the pool "blocked_seconds": 0.0, // of that, time parked on a full downstream queue "in_flight": 3 } ] } ``` Counters are monotonic since startup, Prometheus-style — rates are the reader's job, so a scrape is stateless and two readers can't disturb each other. Take two scrapes Δt apart and subtract: ``` throughput = Δcompleted / Δt utilization = (Δbusy_seconds − Δblocked_seconds) / (Δt × workers) ``` **Utilization is the number that names the bottleneck.** In a pipeline every stage completes the same files, so at steady state they all report near-identical files/s no matter which one is the constraint; what separates them is how hard each pool worked to keep up. The bottleneck sits near 1.0 with its queue backing up while its neighbours idle. `blocked_seconds` is what keeps that true. Stages 1 and 3 apply *blocking* backpressure — a full downstream queue means the handler parks and retries rather than dropping the file — and that wait happens inside the handler. Counting it as busy would pin stage 1 at 1.0 whenever stage 2 is the real jam, making every stage upstream of a jam look like the jam. Subtracted out, utilization means "doing its own work", and a high blocked share becomes its own signal: a stage blocked 90% of the time is naming its successor. The counters are a handful of atomic adds per file, recorded in `worker_loop` — the one place every stage's work passes through, so a new stage is instrumented the moment it is wired up, and never on the read path. ## Benchmarking (throughput + memory) There are three harnesses. Only the first needs a running server: | script | measures | server? | |---|---|---| | `bin/bench.jl` (below) | intake, end-to-end and per-stage throughput; server RSS | **yes** | | [`bin/bench_model.jl`](#model-microbenchmark-binbench_modeljl) | the classifier alone: inference, feature reads, thread scaling | no | | [`bin/cluster_calibrate.jl`](#unknown-format-discovery-stage-5-offline) | stage-5 clustering quality vs. an NCD baseline | no | Running all of them from a clean checkout: ```bash julia --project=. -e 'using Pkg; Pkg.instantiate()' # once # 1. the model, on its own — no server involved julia --project=. -t auto bin/bench_model.jl # 2. the pipeline. Start the server in one terminal… julia --project=. -t auto bin/server.jl # …and drive it from another. Restart the server between memory runs: Julia's # GC returns memory to the OS lazily, so a second run starts inflated. julia --project=. -t auto bin/bench.jl --files 2000 --size 8k --concurrency 32 julia --project=. -t auto bin/bench.jl --files 2 --size 1g --concurrency 1 # memory julia --project=. -t auto bin/bench.jl --corpus ../training_set --concurrency 16 # 3. stage-5 clustering quality (offline, needs a labelled corpus) julia --project=. bin/cluster_calibrate.jl ../training_set ``` The test suite is `julia --project=. -t auto -e 'using Pkg; Pkg.test()'`. `bin/bench.jl` must run on the same machine as the server (it reads the sink dirs and `/proc`); otherwise it only speaks HTTP — `/upload`, `/health`, and `/stats` for the per-stage numbers. Three properties of this design dictate how it measures: - **HTTP latency is not throughput.** `/upload` returns `202` once the bytes are spooled and a reference is enqueued — all four stages run *after* the response, so `ab`/`hey`/`wrk` would only ever measure intake. The bench instead uploads a corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`, `failed/`) until the count stops moving, and reports both numbers separately: intake rate *and* end-to-end completion rate. It also samples the intermediate stage dirs, so the peak depth of `spool/`/`known/`/`unknown/`/`text/` shows where work piles up. - **End-to-end throughput doesn't name the slow stage.** The four stages run concurrently behind their own queues, so the pipeline's rate *is* the slowest stage's rate and the others are invisible in it. The bench scrapes [`/stats`](#get-stats--per-stage-counters) before and after the run and subtracts, giving each stage its own throughput, mean service time and utilization: ``` PER-STAGE (server counters, delta over the end-to-end window) stage files/s MiB/s svc ms util blocked peak queue failed 1 classify 32.52 0.51 38.7 0.08 0.0% 209/1000 0 2 enrich 0.24 0.0 541.0 0.01 0.0% 0/1000 0 3 triage 32.27 0.5 33.9 0.07 0.0% 83/1000 0 4 language 16.26 0.25 852.6 0.87 0.0% 184/1000 0 bottleneck stage 4 (language) at 87.0% utilization of 16 worker(s) ``` (400 mixed files, 16 KiB each, concurrency 16. Stage 4 is the constraint — `github-linguist` is a process spawn per file at ~850 ms — and the 200 text files queue up behind it while stages 1 and 3 idle at under 10%.) Read *down* the util column, not the files/s column: each stage sees a different subset of the corpus, so a low rate can just mean little work was routed there. - **Memory should be flat in file size, and the sweep is what proves it.** Both halves of the pipeline are bounded: the workers read bounded prefixes (16+16 bytes to classify, 8 KB to sniff, 64 KB to language-detect), and intake streams each upload to disk a chunk at a time. So peak RSS should track *concurrency*, not size. Measured on this machine (16 threads, 64 KiB chunk): | upload size | concurrency | RSS growth | |---|---|---| | 256 MiB × 4 | 1 | 14.0 / 20.4 MiB (two runs) | | 1 GiB × 2 | 1 | 22.3 MiB | | 2 GiB × 1 | 1 | 31.3 MiB | | 256 MiB × 4 | 4 | none measurable (peak stayed under the baseline) | **Read the shape, not the digits.** Growth is flat across an 8× range of file sizes — 14–31 MiB whether the upload is 256 MiB or 2 GiB — which is the claim that matters: nothing scales with file size, so nothing is buffering. The absolute figures are *not* precise to the megabyte. A freshly started server settles anywhere in an ~860–985 MiB band, so run-to-run variance in the baseline is comparable to the growth being measured; the residual is GC churn from the chunk reads, not retained buffers. Two consequences worth knowing before quoting these numbers: - **Let the server settle ~30s after startup** before a memory run, or the baseline is sampled mid-fall and the run reports less growth than it caused (or none at all, as the concurrency-4 row above did). - **Linear-in-concurrency is not currently demonstrated.** An earlier measurement put 4 concurrent 256 MiB uploads at 86.9 MiB (21.7 per in-flight upload), but that run predates a fix to how the baseline was sampled — it was read *before* the peak counter was reset, against a different origin — and the re-measurement above cannot reproduce it: the growth sits under the noise floor. Expect concurrency to cost memory; don't trust a specific coefficient without a quieter machine or many more runs. Before intake was streamed, the same 256 MiB upload grew RSS by ~700 MiB and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB, so **a `--size` sweep that slopes upward is the regression signal** — it means something has started buffering bodies again. Flags: `--files`, `--size` (`8k`/`64m`/`1g`), `--concurrency`, `--kind` (`binary`/`text`/`mixed` — chooses which stages get loaded), `--corpus DIR` (real files, the only way to exercise stage 2's exiftool path), `--pid`, `--no-mem`, `--no-stats`, `--sample-ms`, `--timeout`, `--json PATH`. Full list in the script header. The `--json` output carries the per-stage numbers too, so a sweep can be compared run to run. Two caveats the script reports rather than hides: it counts sink *deltas*, so it warns if the pipeline isn't idle at the start (in-flight leftovers would be counted as its own throughput); and because Julia's GC returns memory to the OS lazily, a second run in the same process starts from an inflated baseline — it resets the kernel's peak-RSS counter (`/proc//clear_refs`) per run and flags a drifted baseline, but for a clean growth figure restart the server between memory runs. ### Model microbenchmark (`bin/bench_model.jl`) `bin/bench.jl` reports stage 1 as a single number — the wall time of `handle_classify_job`, which is a feature read, an inference, a rename, a log line, and whatever contention the other three pools create. That's the right number for capacity planning and the wrong one for "is the model slow?". `bin/bench_model.jl` answers that separately, with no server, queue, or HTTP involved: ```bash julia --project=. -t auto bin/bench_model.jl ``` Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12): | what | per file | notes | |---|---|---| | `Lux.apply`, batch 1 | **2.3 µs** | 768 B allocated per call | | `read_features` | **4.2–6.0 µs** | flat across a 262,144× size range (1 KiB → 256 MiB) | | `classify()` | **8.4 µs** | 73% feature read, 27% inference | So the model is **not** the pipeline's problem, by three orders of magnitude: the same run measured stage 1 at 38.7 ms per file, ~4,500× the 8.4 µs `classify()` costs. Whatever stage 1 spends its time on, it isn't the network. Two findings worth acting on if stage 1 ever *does* become the constraint: - **Batching would buy ~13×.** A 32×1 matmul wastes most of a BLAS call: batch 64 costs 280 ns/file and batch 512 costs 179 ns/file, against 2.34 µs one at a time. The pipeline classifies strictly one file per job today, so it pays the worst row in that table. - **Inference does not scale past ~4 threads.** Concurrent `Lux.apply` on the shared read-only `Classifier` peaks around 1.2M files/s at 4 tasks and then *falls back* to single-thread throughput at 16. The script runs a pure-compute control kernel through the same sweep to place the blame: the control reaches 14.3× at 16 tasks (90% efficiency) on the same box, so the machine parallelizes and `Lux.apply` doesn't. GC is only ~1% of it, so allocation pressure isn't the explanation either — the cause is inside Lux/BLAS and is not diagnosed here. The practical consequence: raising `FS_WORKERS` past ~4 adds no classification throughput. Reported times are the **minimum** over trials — for a microbenchmark the floor is the signal and everything above it is scheduler and GC noise — and every timed loop stores its result in a sink so a pure call can't be hoisted out of the loop. Flags: `--model`, `--reps`, `--trials`, `--batches`, `--sizes`, `--no-threads`, `--json PATH`. ## Layout ``` src/ FileServer.jl module + run() (startup, recovery, workers, serve, shutdown) config.jl Config struct + env parsing job.jl Job (the queue reference) queue.jl JobQueue seam + in-process ChannelQueue stats.jl per-stage counters behind GET /stats (throughput, utilization) multipart.jl streaming multipart/form-data reader (intake never buffers a file) spool.jl filename sanitizing, streaming spool/move, startup recovery model.jl NN architecture + byte→feature mapping (shared with trainer) classify.jl load artifact + classify a file at inference time metadata.jl exiftool extraction + normalized sidecar (stage 2) content.jl binary-vs-text sniff for unknown files (stage 3) language.jl natural + programming language enrichment for text (stage 4) cluster.jl header-byte clustering model + Gibbs + scoring core (stage 5, science) catalog.jl durable single-owner format catalog + sweep + nominations (stage 5, phase B) worker.jl parametrized worker loop + classify/enrich/triage/language handlers server.jl HTTP routes + the streaming /upload handler bin/ server.jl entry point bench.jl throughput + memory harness against a running server bench_model.jl classifier microbenchmark (inference, feature reads, scaling) train.jl offline training script → model/classifier.jld2 cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline cluster_sweep.jl stage-5 phase-B runner: sweep binary/, update catalog, write nominations model/ classifier.jld2 committed trained weights (loaded at startup) DESIGN_clustering.md stage-5 design rationale + calibration results ```