Add Lux.jl file classifier (known/unknown) with offline trainer

Each uploaded file is scored by a fixed-structure neural net that labels it
known (resembling the training set) or unknown — novelty detection over the
first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax.

- src/model.jl: shared architecture + byte->feature mapping (trainer + server)
- src/classify.jl: load committed artifact, classify a file at inference
- bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag),
  seeded 80/20 split, writes model/classifier.jld2
- worker: classify (annotate-only) and log classification=known|unknown
- config: FS_MODEL_PATH; server fails fast if the artifact is missing
- deps: Lux, JLD2, Optimisers, Zygote
This commit is contained in:
2026-07-02 14:13:57 -04:00
parent 6d685cfcbb
commit e55129e3a4
10 changed files with 1145 additions and 12 deletions

View File

@@ -5,11 +5,15 @@ using UUIDs
using HTTP
using JSON3
using Oxygen
using Lux
using JLD2
include("config.jl")
include("job.jl")
include("queue.jl")
include("spool.jl")
include("model.jl") # build_model() + read_features(); shared with bin/train.jl
include("classify.jl") # Classifier + load_classifier/classify (needs model.jl)
include("worker.jl")
# Globals the HTTP handlers read at request time. Set once in `run`, before the
@@ -17,6 +21,7 @@ include("worker.jl")
# `Config`/`ChannelQueue` types exist.
const CONFIG = Ref{Config}()
const QUEUE = Ref{ChannelQueue}()
const CLASSIFIER = Ref{Classifier}() # loaded once at startup, shared read-only across workers
include("server.jl") # registers routes (references CONFIG/QUEUE at call time)
@@ -71,6 +76,11 @@ function run(; overrides...)
CONFIG[] = cfg
QUEUE[] = queue
# Load the classifier before serving. Fail fast: a server that silently
# doesn't classify is a worse surprise than a clear startup error.
CLASSIFIER[] = load_classifier(cfg.model_path)
@info "loaded classifier" path=cfg.model_path
recovered = recover_spool!(cfg, queue)
@info "starting file-server" host=cfg.host port=cfg.port workers=cfg.worker_count capacity=cfg.queue_capacity recovered=recovered