Add stage-3 content triage: sort unknown files into binary/ and text/
Unknown files are no longer terminal. Stage 1 now routes :unknown onto a dedicated queue (with the same blocking backpressure as the known queue), and a third worker pool sorts each file into data/binary/ or data/text/ using a NUL-byte sniff of the first 8000 bytes. - content.jl: is_binary content sniff (stage 3) - worker.jl: handle_unknown_job; stage-1 routes unknown with backpressure; KNOWN_ENQUEUE_RETRY_SECONDS -> ROUTE_ENQUEUE_RETRY_SECONDS (serves both) - config.jl: unknown_worker_count/queue_capacity, binary_dir, text_dir + env - FileServer.jl: unknown queue, pool, stage-aware recovery, drain ordering - tests for is_binary and handle_unknown_job; tmp_config isolates new dirs - README: three-stage pipeline
This commit is contained in:
24
src/content.jl
Normal file
24
src/content.jl
Normal file
@@ -0,0 +1,24 @@
|
||||
# Stage-3 content triage for unknown files.
|
||||
#
|
||||
# A file that stage-1 couldn't recognize is still sorted into one of two coarse
|
||||
# buckets so downstream tooling can treat them differently: `text/` for
|
||||
# human-readable content, `binary/` for everything else. The test is the classic
|
||||
# "NUL byte in the first sniff window" heuristic that git and file(1) use — cheap
|
||||
# (no full read), and reliable in practice: text encodings don't embed NUL bytes,
|
||||
# while binary formats almost always do near the start.
|
||||
|
||||
const CONTENT_SNIFF_BYTES = 8000
|
||||
|
||||
"""
|
||||
is_binary(path) -> Bool
|
||||
|
||||
Classify a file as binary (`true`) or text (`false`) by sniffing its first
|
||||
`CONTENT_SNIFF_BYTES` bytes for a NUL byte. An empty file has no NUL, so it is
|
||||
treated as text.
|
||||
"""
|
||||
function is_binary(path::AbstractString)::Bool
|
||||
open(path, "r") do io
|
||||
chunk = read(io, CONTENT_SNIFF_BYTES)
|
||||
return any(==(0x00), chunk)
|
||||
end
|
||||
end
|
||||
Reference in New Issue
Block a user