Jeffrey Ward
e55129e3a4
Each uploaded file is scored by a fixed-structure neural net that labels it known (resembling the training set) or unknown — novelty detection over the first 16 + last 16 bytes (scaled to [0,1]), Dense(32->64->16->2), argmax. - src/model.jl: shared architecture + byte->feature mapping (trainer + server) - src/classify.jl: load committed artifact, classify a file at inference - bin/train.jl: offline trainer, 1:1 blended negatives (random + grab-bag), seeded 80/20 split, writes model/classifier.jld2 - worker: classify (annotate-only) and log classification=known|unknown - config: FS_MODEL_PATH; server fails fast if the artifact is missing - deps: Lux, JLD2, Optimisers, Zygote
52 lines
1.9 KiB
Julia
52 lines
1.9 KiB
Julia
# Shared model definition + feature extraction, used by BOTH the training script
|
||
# (bin/train.jl) and the server (worker inference). Keeping the architecture and
|
||
# the byte->feature mapping in one place guarantees train and inference can never
|
||
# drift out of sync (same input layout, same scaling, same net shape).
|
||
#
|
||
# The including scope is expected to have already done `using Lux` and
|
||
# `using JLD2` (FileServer.jl and bin/train.jl both do), so this file adds no
|
||
# top-level `using` of its own and works whether it's pulled into the FileServer
|
||
# module or a bare script.
|
||
|
||
"Number of input features: first 16 bytes + last 16 bytes of a file."
|
||
const FEATURE_DIM = 32
|
||
|
||
"Minimum file size (bytes) the model can accept: needs a disjoint 16+16 window."
|
||
const MIN_FILE_BYTES = 32
|
||
|
||
"""
|
||
build_model()
|
||
|
||
The fixed-structure classifier: 32-dim byte features → two logits
|
||
(class 1 = known, class 2 = unknown). Raw logits out (no softmax layer); the
|
||
loss applies log-softmax and inference just takes an argmax.
|
||
"""
|
||
build_model() = Chain(
|
||
Dense(FEATURE_DIM => 64, relu),
|
||
Dense(64 => 16, relu),
|
||
Dense(16 => 2),
|
||
)
|
||
|
||
"""
|
||
read_features(path) -> Union{Vector{Float32}, Nothing}
|
||
|
||
Read the first 16 and last 16 bytes of the file at `path`, concatenate to a
|
||
32-element vector, and scale each byte from 0–255 into [0,1]. Returns `nothing`
|
||
for files shorter than `MIN_FILE_BYTES` (the caller decides what that means:
|
||
training drops them, the server classifies them as unknown without the model).
|
||
|
||
Memory stays flat: we read 16 bytes from the front and `seek` to end-16 for the
|
||
tail rather than slurping the whole file.
|
||
"""
|
||
function read_features(path::AbstractString)
|
||
sz = filesize(path)
|
||
sz < MIN_FILE_BYTES && return nothing
|
||
bytes = open(path) do io
|
||
head = read(io, 16)
|
||
seek(io, sz - 16)
|
||
tail = read(io, 16)
|
||
vcat(head, tail)
|
||
end
|
||
return Float32.(bytes) ./ 255f0
|
||
end
|