Add Lux.jl file classifier (known/unknown) with offline trainer

Each uploaded file is scored by a fixed-structure neural net that labels it
known (resembling the training set) or unknown — novelty detection over the
first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax.

- src/model.jl: shared architecture + byte->feature mapping (trainer + server)
- src/classify.jl: load committed artifact, classify a file at inference
- bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag),
  seeded 80/20 split, writes model/classifier.jld2
- worker: classify (annotate-only) and log classification=known|unknown
- config: FS_MODEL_PATH; server fails fast if the artifact is missing
- deps: Lux, JLD2, Optimisers, Zygote
This commit is contained in:
2026-07-02 14:13:57 -04:00
parent 6d685cfcbb
commit e55129e3a4
10 changed files with 1145 additions and 12 deletions

View File

@@ -5,8 +5,9 @@ pool of worker threads for processing. The HTTP endpoint does no real work: it
spools each uploaded file to disk, pushes a lightweight reference onto a work
queue, and responds immediately — staying free to accept the next upload.
Right now the "processing" is just logging the received filename, to prove the
flow. That's the seam where real heavy-lifting goes later.
The per-file "processing" runs each file through a small neural-network
classifier that labels it **known** (a file type resembling the training set) or
**unknown**, and logs the result. See "File classifier" below.
## Architecture
@@ -82,6 +83,49 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
`STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain
in-flight work (systemd: `TimeoutStopSec`).
## File classifier
Each file is scored by a fixed-structure neural network (Lux.jl) that answers a
single binary question: is this file **known** (like the types in the training
set) or **unknown**? It's novelty detection, not exact file-typing — it won't
tell you "PDF", just "this looks like something I was trained on, or not".
- **Features:** the first 16 bytes + last 16 bytes of the file, each scaled
0255 → `[0,1]`, giving a 32-dim input. Files under 32 bytes can't form that
window and are classified `unknown` without touching the model.
- **Architecture:** `Dense(32→64,relu) → Dense(64→16,relu) → Dense(16→2)`,
raw logits; decision is `argmax` (class 1 = known, class 2 = unknown).
- **Artifact:** trained weights live in `model/classifier.jld2` (committed), so
the server just loads them at startup. Missing/unreadable ⇒ the server fails
fast rather than run without classification.
- **Effect today:** *annotate-only*. The class is logged
(`classification=known|unknown`) but every file still moves to `done/`; the
classifier can't misroute real files while it's unproven.
The architecture and byte→feature mapping are defined once in `src/model.jl` and
shared by the trainer and the server, so they can't drift apart.
### Training
Training is a separate, offline script — it never runs in the request path:
```bash
julia --project=. bin/train.jl <positives_dir> [negatives_dir]
```
- **positives_dir** — every file in it (≥32 bytes) is a "known" example.
- **negatives_dir** *(optional)* — a grab-bag of *other* real file types used as
"unknown" examples. Negatives are generated ~1:1 with positives, split 50/50
between uniform-random byte vectors and grab-bag files. With no grab-bag dir,
negatives are all random (weaker: the net may just learn "high entropy =
unknown" rather than your actual types, so a grab-bag of real off-distribution
files is recommended).
The script uses an 80/20 seeded split, reports validation accuracy, and writes
`model/classifier.jld2` (path overridable via `FS_MODEL_PATH`). A fixed seed
(`FS_TRAIN_SEED`, default 42) drives negative generation, the split, and weight
init, so the artifact is exactly regenerable from the same inputs.
## Configuration (environment variables)
| Variable | Default | Meaning |
@@ -93,6 +137,7 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending) |
| `FS_DONE_DIR` | `data/done` | Files after successful processing |
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
> To get real parallelism, start Julia with enough threads (`-t N`) to match
> `FS_WORKERS`. If `FS_WORKERS` exceeds available threads you'll get a warning
@@ -126,8 +171,13 @@ src/
job.jl Job (the queue reference)
queue.jl JobQueue seam + in-process ChannelQueue
spool.jl filename sanitizing, spool/move, startup recovery
worker.jl worker loop + per-job processing (placeholder)
model.jl NN architecture + byte→feature mapping (shared with trainer)
classify.jl load artifact + classify a file at inference time
worker.jl worker loop + per-job processing (classify + move)
server.jl HTTP routes/handlers
bin/
server.jl entry point
train.jl offline training script → model/classifier.jld2
model/
classifier.jld2 committed trained weights (loaded at startup)
```