Add Lux.jl file classifier (known/unknown) with offline trainer
Each uploaded file is scored by a fixed-structure neural net that labels it known (resembling the training set) or unknown — novelty detection over the first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax. - src/model.jl: shared architecture + byte->feature mapping (trainer + server) - src/classify.jl: load committed artifact, classify a file at inference - bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag), seeded 80/20 split, writes model/classifier.jld2 - worker: classify (annotate-only) and log classification=known|unknown - config: FS_MODEL_PATH; server fails fast if the artifact is missing - deps: Lux, JLD2, Optimisers, Zygote
This commit is contained in:
51
src/model.jl
Normal file
51
src/model.jl
Normal file
@@ -0,0 +1,51 @@
|
||||
# Shared model definition + feature extraction, used by BOTH the training script
|
||||
# (bin/train.jl) and the server (worker inference). Keeping the architecture and
|
||||
# the byte->feature mapping in one place guarantees train and inference can never
|
||||
# drift out of sync (same input layout, same scaling, same net shape).
|
||||
#
|
||||
# The including scope is expected to have already done `using Lux` and
|
||||
# `using JLD2` (FileServer.jl and bin/train.jl both do), so this file adds no
|
||||
# top-level `using` of its own and works whether it's pulled into the FileServer
|
||||
# module or a bare script.
|
||||
|
||||
"Number of input features: first 16 bytes + last 16 bytes of a file."
|
||||
const FEATURE_DIM = 32
|
||||
|
||||
"Minimum file size (bytes) the model can accept: needs a disjoint 16+16 window."
|
||||
const MIN_FILE_BYTES = 32
|
||||
|
||||
"""
|
||||
build_model()
|
||||
|
||||
The fixed-structure classifier: 32-dim byte features → two logits
|
||||
(class 1 = known, class 2 = unknown). Raw logits out (no softmax layer); the
|
||||
loss applies log-softmax and inference just takes an argmax.
|
||||
"""
|
||||
build_model() = Chain(
|
||||
Dense(FEATURE_DIM => 64, relu),
|
||||
Dense(64 => 16, relu),
|
||||
Dense(16 => 2),
|
||||
)
|
||||
|
||||
"""
|
||||
read_features(path) -> Union{Vector{Float32}, Nothing}
|
||||
|
||||
Read the first 16 and last 16 bytes of the file at `path`, concatenate to a
|
||||
32-element vector, and scale each byte from 0–255 into [0,1]. Returns `nothing`
|
||||
for files shorter than `MIN_FILE_BYTES` (the caller decides what that means:
|
||||
training drops them, the server classifies them as unknown without the model).
|
||||
|
||||
Memory stays flat: we read 16 bytes from the front and `seek` to end-16 for the
|
||||
tail rather than slurping the whole file.
|
||||
"""
|
||||
function read_features(path::AbstractString)
|
||||
sz = filesize(path)
|
||||
sz < MIN_FILE_BYTES && return nothing
|
||||
bytes = open(path) do io
|
||||
head = read(io, 16)
|
||||
seek(io, sz - 16)
|
||||
tail = read(io, 16)
|
||||
vcat(head, tail)
|
||||
end
|
||||
return Float32.(bytes) ./ 255f0
|
||||
end
|
||||
Reference in New Issue
Block a user