Jeffrey Ward
fac3adbaf6
Text files sorted by stage 3 now flow onto a new work queue and worker pool that enrich them with natural language (Languages.jl LanguageDetector: name, ISO 639-3 code, confidence) and programming language (github-linguist), writing a .meta.json sidecar to data/text_done/ like the stage-2 known-file pipeline. github-linguist reads the git blob of a path inside a repo, so untracked data/ files are copied to /tmp (outside any repo, name preserved for extension heuristics) before detection. Programming-language lookup is best-effort (startup warning if missing, degraded/null on failure); natural-language failure yields a degraded sidecar, not a quarantine. Factored exiftool's timeout-kill into shared run_with_timeout and the durable sidecar-first commit into commit_enriched!, both reused by stage 4. Recovery re-drives data/text/; graceful drain closes the text queue after its stage-3 producers finish.
357 lines
19 KiB
Markdown
357 lines
19 KiB
Markdown
# FileServer
|
||
|
||
A minimal Julia service that receives files over HTTP and hands them off to a
|
||
pool of worker threads for processing. The HTTP endpoint does no real work: it
|
||
spools each uploaded file to disk, pushes a lightweight reference onto a work
|
||
queue, and responds immediately — staying free to accept the next upload.
|
||
|
||
The per-file "processing" runs each file through a small neural-network
|
||
classifier that labels it **known** (a file type resembling the training set) or
|
||
**unknown**, and logs the result. See "File classifier" below.
|
||
|
||
## Architecture
|
||
|
||
The pipeline is four stages, each with its own bounded queue and its own worker
|
||
pool (tuned independently, since classification is CPU-bound, known-file
|
||
enrichment is process-/IO-bound, content triage is cheap IO, and language
|
||
enrichment mixes CPU with a subprocess):
|
||
|
||
```
|
||
POST /upload (multipart)
|
||
│
|
||
▼
|
||
┌─────────────────┐ spool bytes to disk
|
||
│ HTTP handler │────────────────────────► data/spool/<uuid>-<name>
|
||
│ (Oxygen.jl) │
|
||
└────────┬─────────┘ enqueue reference (non-blocking)
|
||
│ │
|
||
▼ ▼
|
||
202 + job IDs ┌────────────────────┐
|
||
(503 if full) │ stage-1 queue │ classification
|
||
└─────────┬──────────┘
|
||
│ dequeue
|
||
┌───────────────────┼───────────────────┐
|
||
▼ ▼ ▼
|
||
classify wkr 1 classify wkr 2 … classify wkr N
|
||
│
|
||
┌────────────┴────────────┐
|
||
:unknown :known
|
||
│ move to data/unknown/, │ move to data/known/, then
|
||
▼ then enqueue (blocking) ▼ enqueue (blocking backpressure)
|
||
┌────────────────────┐ ┌────────────────────┐
|
||
│ unknown queue │ │ known queue │ enrichment
|
||
└─────────┬──────────┘ └─────────┬──────────┘
|
||
│ dequeue │ dequeue
|
||
┌────────┼────────┐ ┌───────────┼───────────┐
|
||
▼ ▼ ▼ ▼ ▼ ▼
|
||
unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M
|
||
│ binary-vs-text sniff │ exiftool → normalized sidecar
|
||
├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name>
|
||
│ (terminal) data/done/<uuid>-<name>.meta.json
|
||
│ (sidecar-first commit)
|
||
│ :text move to data/text/, failure ───────► data/failed/<uuid>-<name>
|
||
▼ then enqueue (blocking backpressure)
|
||
┌────────────────────┐
|
||
│ text queue │ language enrichment
|
||
└─────────┬──────────┘
|
||
│ dequeue
|
||
┌────────┼────────┐
|
||
▼ ▼ ▼
|
||
txt 1 txt 2 … txt P
|
||
│ Languages.jl (natural language) + github-linguist (programming language)
|
||
└─► data/text_done/<uuid>-<name> + data/text_done/<uuid>-<name>.meta.json
|
||
(sidecar-first commit)
|
||
```
|
||
|
||
Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1
|
||
feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language
|
||
enrichment) for every file it sorts as text.
|
||
|
||
Key properties:
|
||
|
||
- **Fast intake:** the queue only ever carries small references; file bytes live
|
||
on disk, so memory stays flat regardless of file size.
|
||
- **Backpressure:** each queue is bounded (default 1000). When the *intake* queue
|
||
is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
|
||
the stage-1 worker blocks and retries (a classified file is never dropped).
|
||
- **Crash-resilient:** files survive on disk. On startup, recovery is
|
||
stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/`
|
||
re-enter enrichment, `data/unknown/` re-enter content triage, and `data/text/`
|
||
re-enter language enrichment (`recovered` / `recovered_known` /
|
||
`recovered_unknown` / `recovered_text` in the log), so a file resumes at its
|
||
correct stage instead of restarting from scratch.
|
||
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
|
||
both stop accepting uploads, then drain the stages *in order* — close the
|
||
stage-1 queue and wait out the classify workers (the only producer of the known
|
||
*and* unknown queues), then close those queues and wait out the enrich and
|
||
content-triage workers (content triage being the only producer of the text
|
||
queue), then close the text queue and wait out the language-enrichment workers.
|
||
(See "Shutdown" below for one cosmetic caveat on SIGTERM.)
|
||
- **Safe filenames:** client-supplied names are sanitized and prefixed with a
|
||
server-minted UUID before touching the filesystem (no path traversal).
|
||
|
||
### Metadata enrichment (stage 2)
|
||
|
||
Files the classifier labels **known** are handed to a second pool that extracts
|
||
metadata with [`exiftool`](https://exiftool.org/) (`exiftool -json -G -n`) —
|
||
chosen because no native Julia library comes close to its multi-format coverage.
|
||
The output is normalized into a small, stable, documented schema and written as a
|
||
JSON **sidecar** next to the file in `data/done/`, e.g.
|
||
`data/done/<uuid>-<name>.meta.json`. The original bytes are never modified.
|
||
|
||
> **Prerequisite:** `exiftool` must be on `PATH` (e.g. `apt install
|
||
> libimage-exiftool-perl`). The server **fails fast at startup** if it's missing.
|
||
|
||
Sidecar top-level fields (all nullable — present only when available), plus the
|
||
complete raw `exiftool` object under `raw`:
|
||
|
||
| Field | Meaning |
|
||
|---|---|
|
||
| `id`, `original_name` | job id and client-supplied name |
|
||
| `file_type`, `mime_type` | e.g. `PDF` / `application/pdf` |
|
||
| `file_size` | bytes (authoritative, from intake — not exiftool) |
|
||
| `created_date`, `modified_date` | content timestamps |
|
||
| `author` | person (`Author`/`Artist`/`By-line`) |
|
||
| `created_by` | authoring app/tool (`Producer`/`CreatorTool`/`Creator`/`Software`/…) |
|
||
| `dimensions` | `{width, height}` for media |
|
||
| `duration` | seconds, for audio/video |
|
||
| `page_count` | for documents |
|
||
| `error` | set on a *degraded* sidecar (see below) |
|
||
| `raw` | full `exiftool` output |
|
||
|
||
Each normalized field is a coalesce over a priority list of exiftool tags
|
||
(`src/metadata.jl`); extend a field by appending tag names. If extraction fails
|
||
or `exiftool` times out (`FS_EXIFTOOL_TIMEOUT`, default 30s), the file still
|
||
completes to `data/done/` with a **degraded sidecar** — `file_size`/`file_type`
|
||
plus an `error` note — rather than being quarantined, because it's still a wanted
|
||
known file. Only genuine I/O errors (can't write the sidecar or move the file)
|
||
send it to `data/failed/`.
|
||
|
||
The sidecar is committed **before** the file is moved into `data/done/`, so a
|
||
file's presence there always implies its sidecar is already present; a crash in
|
||
between leaves only a harmless orphan sidecar, and recovery re-enriches
|
||
idempotently.
|
||
|
||
### Content triage (stage 3)
|
||
|
||
Files the classifier labels **unknown** are handed to a third pool that sorts
|
||
them into two coarse buckets so downstream tooling can treat them differently:
|
||
|
||
- **`data/binary/`** — the file looks like binary data.
|
||
- **`data/text/`** — the file looks like text.
|
||
|
||
The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text
|
||
when that window is valid UTF-8 and holds no control bytes outside the text-safe
|
||
set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise
|
||
binary. It's cheap (no full read) and Unicode-aware — unlike the older NUL-byte
|
||
or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in
|
||
`text/` instead of misfiling it, while binary formats — which rarely form valid
|
||
UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but
|
||
not a text control byte, so it still reads as binary. A multi-byte character
|
||
split by the 8000-byte boundary is trimmed before the check so it isn't mistaken
|
||
for malformed bytes. An empty file is treated as text. `binary/` is terminal;
|
||
`text/` is handed to stage 4 (`src/content.jl`).
|
||
|
||
### Language enrichment (stage 4)
|
||
|
||
Files that stage 3 sorts as **text** are handed to a fourth pool that identifies
|
||
their language and writes a `.meta.json` sidecar, mirroring the stage-2
|
||
known-file enrichment. Two detectors run per file:
|
||
|
||
- **natural language** — [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s
|
||
`LanguageDetector` (a Julia port of the `whatlang` n-gram model) reads a bounded
|
||
prefix (up to `LANG_SAMPLE_BYTES`, 64 KiB) and reports the language's English
|
||
name, ISO 639-3 code, and a confidence in `[0,1]`. Pure Julia, no subprocess.
|
||
The detector is built once at startup and shared read-only across the pool.
|
||
- **programming language** — the [`github-linguist`](https://github.com/github-linguist/linguist)
|
||
CLI recognizes source and markup by extension + content heuristics (e.g.
|
||
`Python`, `Markdown`). Plain prose reports as `Text` and unrecognized content as
|
||
`null`; both collapse to *no programming language*.
|
||
|
||
The sidecar schema:
|
||
|
||
| field | meaning |
|
||
|---|---|
|
||
| `id`, `original_name` | from intake |
|
||
| `file_size` | bytes (authoritative, from intake) |
|
||
| `content_type` | always `"text"` |
|
||
| `language` | natural-language English name (e.g. `English`), or `null` |
|
||
| `language_code` | ISO 639-3 code (e.g. `eng`), or `null` |
|
||
| `language_confidence` | detector confidence in `[0,1]`, or `null` |
|
||
| `programming_language` | e.g. `Python`, `Markdown`, or `null` |
|
||
| `error` | set if natural-language detection produced nothing |
|
||
|
||
> **`github-linguist` and the git-repo quirk:** run against a path *inside* a git
|
||
> repository, linguist reads the file's committed git blob, not the on-disk bytes
|
||
> — and an untracked file (which everything under `data/` is) has no blob, so it
|
||
> crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under
|
||
> `/tmp` (outside any repo, preserving the name so extension heuristics still
|
||
> fire) and pointing linguist there.
|
||
>
|
||
> Programming-language detection is **best-effort**: if `github-linguist` is
|
||
> missing (a startup warning, not a fatal error, unlike `exiftool`), fails, or
|
||
> times out (`FS_LINGUIST_TIMEOUT`, default 30s), `programming_language` is simply
|
||
> `null` and the file still completes. Natural-language detection failing produces
|
||
> a **degraded sidecar** (with an `error` note) rather than a quarantine, because
|
||
> the file is still wanted text.
|
||
|
||
Like stage 2, the sidecar is committed **before** the file is moved into
|
||
`data/text_done/`, so the file's presence there always implies its sidecar is
|
||
present; recovery re-enriches idempotently (`src/language.jl`).
|
||
|
||
## The queue seam (→ RabbitMQ later)
|
||
|
||
The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and
|
||
`close!` on a `JobQueue` (see `src/queue.jl`). Today that's an in-process
|
||
`ChannelQueue`. To move to RabbitMQ (or any broker), implement a new `JobQueue`
|
||
subtype with those three methods and swap the construction in `run` — no handler
|
||
or worker code changes.
|
||
|
||
## Running
|
||
|
||
```bash
|
||
# install deps (first time)
|
||
julia --project=. -e 'using Pkg; Pkg.instantiate()'
|
||
|
||
# external tools: exiftool (stage 2, required) and github-linguist (stage 4,
|
||
# optional — programming-language detection). e.g. on Debian/Ubuntu:
|
||
# apt install libimage-exiftool-perl
|
||
# gem install github-linguist
|
||
|
||
# start the server; -t sets the number of OS threads available to workers
|
||
julia --project=. -t auto bin/server.jl
|
||
```
|
||
|
||
## Shutdown
|
||
|
||
Both SIGINT and SIGTERM trigger the same idempotent graceful drain
|
||
(stop serving → close queue → wait for workers → exit):
|
||
|
||
- **SIGINT** is caught as an `InterruptException` (we call
|
||
`Base.exit_on_sigint(false)`), so shutdown is clean and quiet.
|
||
- **SIGTERM** can't be intercepted directly — Julia blocks it on worker threads
|
||
and handles it in its own runtime, so a user `signal()` handler never fires.
|
||
Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path
|
||
does run. Caveat: Julia prints its own `signal 15: Terminated` backtrace
|
||
*before* `atexit` runs. It's harmless noise — the drain still completes right
|
||
after it — but if you want a fully quiet stop under a process manager,
|
||
configure it to send SIGINT instead (systemd: `KillSignal=SIGINT`; Docker:
|
||
`STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain
|
||
in-flight work (systemd: `TimeoutStopSec`).
|
||
|
||
## File classifier
|
||
|
||
Each file is scored by a fixed-structure neural network (Lux.jl) that answers a
|
||
single binary question: is this file **known** (like the types in the training
|
||
set) or **unknown**? It's novelty detection, not exact file-typing — it won't
|
||
tell you "PDF", just "this looks like something I was trained on, or not".
|
||
|
||
- **Features:** the first 16 bytes + last 16 bytes of the file, each scaled
|
||
0–255 → `[0,1]`, giving a 32-dim input. Files under 32 bytes can't form that
|
||
window and are classified `unknown` without touching the model.
|
||
- **Architecture:** `Dense(32→64,relu) → Dense(64→16,relu) → Dense(16→2)`,
|
||
raw logits; decision is `argmax` (class 1 = known, class 2 = unknown).
|
||
- **Artifact:** trained weights live in `model/classifier.jld2` (committed), so
|
||
the server just loads them at startup. Missing/unreadable ⇒ the server fails
|
||
fast rather than run without classification.
|
||
- **Effect today:** *annotate-only*. The class is logged
|
||
(`classification=known|unknown`) but every file still moves to `done/`; the
|
||
classifier can't misroute real files while it's unproven.
|
||
|
||
The architecture and byte→feature mapping are defined once in `src/model.jl` and
|
||
shared by the trainer and the server, so they can't drift apart.
|
||
|
||
### Training
|
||
|
||
Training is a separate, offline script — it never runs in the request path:
|
||
|
||
```bash
|
||
julia --project=. bin/train.jl <positives_dir> [negatives_dir]
|
||
```
|
||
|
||
- **positives_dir** — every file in it (≥32 bytes) is a "known" example.
|
||
- **negatives_dir** *(optional)* — a grab-bag of *other* real file types used as
|
||
"unknown" examples. Negatives are generated ~1:1 with positives, split 50/50
|
||
between uniform-random byte vectors and grab-bag files. With no grab-bag dir,
|
||
negatives are all random (weaker: the net may just learn "high entropy =
|
||
unknown" rather than your actual types, so a grab-bag of real off-distribution
|
||
files is recommended).
|
||
|
||
The script uses an 80/20 seeded split, reports validation accuracy, and writes
|
||
`model/classifier.jld2` (path overridable via `FS_MODEL_PATH`). A fixed seed
|
||
(`FS_TRAIN_SEED`, default 42) drives negative generation, the split, and weight
|
||
init, so the artifact is exactly regenerable from the same inputs.
|
||
|
||
## Configuration (environment variables)
|
||
|
||
| Variable | Default | Meaning |
|
||
|---------------------|----------------|------------------------------------------|
|
||
| `FS_HOST` | `127.0.0.1` | Bind address |
|
||
| `FS_PORT` | `8080` | Port |
|
||
| `FS_WORKERS` | `nthreads()` | Stage-1 (classification) worker tasks |
|
||
| `FS_QUEUE_CAPACITY` | `1000` | Max pending intake jobs before `503` |
|
||
| `FS_KNOWN_WORKERS` | `nthreads()` | Stage-2 (enrichment) worker tasks |
|
||
| `FS_KNOWN_QUEUE_CAPACITY` | `1000` | Max pending enrichment jobs (then backpressure) |
|
||
| `FS_UNKNOWN_WORKERS` | `nthreads()` | Stage-3 (content triage) worker tasks |
|
||
| `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) |
|
||
| `FS_TEXT_WORKERS` | `nthreads()` | Stage-4 (language enrichment) worker tasks |
|
||
| `FS_TEXT_QUEUE_CAPACITY` | `1000` | Max pending language jobs (then backpressure) |
|
||
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending classification) |
|
||
| `FS_KNOWN_DIR` | `data/known` | Classified-known, awaiting enrichment |
|
||
| `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, awaiting content triage |
|
||
| `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary |
|
||
| `FS_TEXT_DIR` | `data/text` | Classified-text, awaiting language enrichment |
|
||
| `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) |
|
||
| `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) |
|
||
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
|
||
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
|
||
| `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed |
|
||
| `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed |
|
||
|
||
> To get real parallelism, start Julia with enough threads (`-t N`) to cover all
|
||
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS`
|
||
> exceeds available threads you'll get a warning (non-fatal) and workers will
|
||
> share threads.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# health check
|
||
curl http://127.0.0.1:8080/health
|
||
# {"status":"ok"}
|
||
|
||
# upload one or more files (multipart/form-data)
|
||
curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload
|
||
# 202 {"accepted":[{"id":"<uuid>","name":"report.pdf"}, ...]}
|
||
```
|
||
|
||
Each file in a request becomes its own job. Responses:
|
||
|
||
- `202 Accepted` — all files spooled and queued (with per-file job IDs)
|
||
- `400 Bad Request` — not multipart, or no files present
|
||
- `503 Service Unavailable` — queue full, retry later
|
||
- `500 Internal Server Error` — failed to write a file to disk
|
||
|
||
## Layout
|
||
|
||
```
|
||
src/
|
||
FileServer.jl module + run() (startup, recovery, workers, serve, shutdown)
|
||
config.jl Config struct + env parsing
|
||
job.jl Job (the queue reference)
|
||
queue.jl JobQueue seam + in-process ChannelQueue
|
||
spool.jl filename sanitizing, spool/move, startup recovery
|
||
model.jl NN architecture + byte→feature mapping (shared with trainer)
|
||
classify.jl load artifact + classify a file at inference time
|
||
metadata.jl exiftool extraction + normalized sidecar (stage 2)
|
||
content.jl binary-vs-text sniff for unknown files (stage 3)
|
||
language.jl natural + programming language enrichment for text (stage 4)
|
||
worker.jl parametrized worker loop + classify/enrich/triage/language handlers
|
||
server.jl HTTP routes/handlers
|
||
bin/
|
||
server.jl entry point
|
||
train.jl offline training script → model/classifier.jld2
|
||
model/
|
||
classifier.jld2 committed trained weights (loaded at startup)
|
||
```
|