Add Lux.jl file classifier (known/unknown) with offline trainer
Each uploaded file is scored by a fixed-structure neural net that labels it known (resembling the training set) or unknown — novelty detection over the first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax. - src/model.jl: shared architecture + byte->feature mapping (trainer + server) - src/classify.jl: load committed artifact, classify a file at inference - bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag), seeded 80/20 split, writes model/classifier.jld2 - worker: classify (annotate-only) and log classification=known|unknown - config: FS_MODEL_PATH; server fails fast if the artifact is missing - deps: Lux, JLD2, Optimisers, Zygote
This commit is contained in:
56
README.md
56
README.md
@@ -5,8 +5,9 @@ pool of worker threads for processing. The HTTP endpoint does no real work: it
|
||||
spools each uploaded file to disk, pushes a lightweight reference onto a work
|
||||
queue, and responds immediately — staying free to accept the next upload.
|
||||
|
||||
Right now the "processing" is just logging the received filename, to prove the
|
||||
flow. That's the seam where real heavy-lifting goes later.
|
||||
The per-file "processing" runs each file through a small neural-network
|
||||
classifier that labels it **known** (a file type resembling the training set) or
|
||||
**unknown**, and logs the result. See "File classifier" below.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -82,6 +83,49 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
|
||||
`STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain
|
||||
in-flight work (systemd: `TimeoutStopSec`).
|
||||
|
||||
## File classifier
|
||||
|
||||
Each file is scored by a fixed-structure neural network (Lux.jl) that answers a
|
||||
single binary question: is this file **known** (like the types in the training
|
||||
set) or **unknown**? It's novelty detection, not exact file-typing — it won't
|
||||
tell you "PDF", just "this looks like something I was trained on, or not".
|
||||
|
||||
- **Features:** the first 16 bytes + last 16 bytes of the file, each scaled
|
||||
0–255 → `[0,1]`, giving a 32-dim input. Files under 32 bytes can't form that
|
||||
window and are classified `unknown` without touching the model.
|
||||
- **Architecture:** `Dense(32→64,relu) → Dense(64→16,relu) → Dense(16→2)`,
|
||||
raw logits; decision is `argmax` (class 1 = known, class 2 = unknown).
|
||||
- **Artifact:** trained weights live in `model/classifier.jld2` (committed), so
|
||||
the server just loads them at startup. Missing/unreadable ⇒ the server fails
|
||||
fast rather than run without classification.
|
||||
- **Effect today:** *annotate-only*. The class is logged
|
||||
(`classification=known|unknown`) but every file still moves to `done/`; the
|
||||
classifier can't misroute real files while it's unproven.
|
||||
|
||||
The architecture and byte→feature mapping are defined once in `src/model.jl` and
|
||||
shared by the trainer and the server, so they can't drift apart.
|
||||
|
||||
### Training
|
||||
|
||||
Training is a separate, offline script — it never runs in the request path:
|
||||
|
||||
```bash
|
||||
julia --project=. bin/train.jl <positives_dir> [negatives_dir]
|
||||
```
|
||||
|
||||
- **positives_dir** — every file in it (≥32 bytes) is a "known" example.
|
||||
- **negatives_dir** *(optional)* — a grab-bag of *other* real file types used as
|
||||
"unknown" examples. Negatives are generated ~1:1 with positives, split 50/50
|
||||
between uniform-random byte vectors and grab-bag files. With no grab-bag dir,
|
||||
negatives are all random (weaker: the net may just learn "high entropy =
|
||||
unknown" rather than your actual types, so a grab-bag of real off-distribution
|
||||
files is recommended).
|
||||
|
||||
The script uses an 80/20 seeded split, reports validation accuracy, and writes
|
||||
`model/classifier.jld2` (path overridable via `FS_MODEL_PATH`). A fixed seed
|
||||
(`FS_TRAIN_SEED`, default 42) drives negative generation, the split, and weight
|
||||
init, so the artifact is exactly regenerable from the same inputs.
|
||||
|
||||
## Configuration (environment variables)
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
@@ -93,6 +137,7 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
|
||||
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending) |
|
||||
| `FS_DONE_DIR` | `data/done` | Files after successful processing |
|
||||
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
|
||||
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
|
||||
|
||||
> To get real parallelism, start Julia with enough threads (`-t N`) to match
|
||||
> `FS_WORKERS`. If `FS_WORKERS` exceeds available threads you'll get a warning
|
||||
@@ -126,8 +171,13 @@ src/
|
||||
job.jl Job (the queue reference)
|
||||
queue.jl JobQueue seam + in-process ChannelQueue
|
||||
spool.jl filename sanitizing, spool/move, startup recovery
|
||||
worker.jl worker loop + per-job processing (placeholder)
|
||||
model.jl NN architecture + byte→feature mapping (shared with trainer)
|
||||
classify.jl load artifact + classify a file at inference time
|
||||
worker.jl worker loop + per-job processing (classify + move)
|
||||
server.jl HTTP routes/handlers
|
||||
bin/
|
||||
server.jl entry point
|
||||
train.jl offline training script → model/classifier.jld2
|
||||
model/
|
||||
classifier.jld2 committed trained weights (loaded at startup)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user