Add Lux.jl file classifier (known/unknown) with offline trainer
Each uploaded file is scored by a fixed-structure neural net that labels it known (resembling the training set) or unknown — novelty detection over the first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax. - src/model.jl: shared architecture + byte->feature mapping (trainer + server) - src/classify.jl: load committed artifact, classify a file at inference - bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag), seeded 80/20 split, writes model/classifier.jld2 - worker: classify (annotate-only) and log classification=known|unknown - config: FS_MODEL_PATH; server fails fast if the artifact is missing - deps: Lux, JLD2, Optimisers, Zygote
This commit is contained in:
@@ -5,11 +5,15 @@ using UUIDs
|
||||
using HTTP
|
||||
using JSON3
|
||||
using Oxygen
|
||||
using Lux
|
||||
using JLD2
|
||||
|
||||
include("config.jl")
|
||||
include("job.jl")
|
||||
include("queue.jl")
|
||||
include("spool.jl")
|
||||
include("model.jl") # build_model() + read_features(); shared with bin/train.jl
|
||||
include("classify.jl") # Classifier + load_classifier/classify (needs model.jl)
|
||||
include("worker.jl")
|
||||
|
||||
# Globals the HTTP handlers read at request time. Set once in `run`, before the
|
||||
@@ -17,6 +21,7 @@ include("worker.jl")
|
||||
# `Config`/`ChannelQueue` types exist.
|
||||
const CONFIG = Ref{Config}()
|
||||
const QUEUE = Ref{ChannelQueue}()
|
||||
const CLASSIFIER = Ref{Classifier}() # loaded once at startup, shared read-only across workers
|
||||
|
||||
include("server.jl") # registers routes (references CONFIG/QUEUE at call time)
|
||||
|
||||
@@ -71,6 +76,11 @@ function run(; overrides...)
|
||||
CONFIG[] = cfg
|
||||
QUEUE[] = queue
|
||||
|
||||
# Load the classifier before serving. Fail fast: a server that silently
|
||||
# doesn't classify is a worse surprise than a clear startup error.
|
||||
CLASSIFIER[] = load_classifier(cfg.model_path)
|
||||
@info "loaded classifier" path=cfg.model_path
|
||||
|
||||
recovered = recover_spool!(cfg, queue)
|
||||
@info "starting file-server" host=cfg.host port=cfg.port workers=cfg.worker_count capacity=cfg.queue_capacity recovered=recovered
|
||||
|
||||
|
||||
Reference in New Issue
Block a user