Add stage-3 content triage: sort unknown files into binary/ and text/

Unknown files are no longer terminal. Stage 1 now routes :unknown onto a
dedicated queue (with the same blocking backpressure as the known queue),
and a third worker pool sorts each file into data/binary/ or data/text/
using a NUL-byte sniff of the first 8000 bytes.

- content.jl: is_binary content sniff (stage 3)
- worker.jl: handle_unknown_job; stage-1 routes unknown with backpressure;
  KNOWN_ENQUEUE_RETRY_SECONDS -> ROUTE_ENQUEUE_RETRY_SECONDS (serves both)
- config.jl: unknown_worker_count/queue_capacity, binary_dir, text_dir + env
- FileServer.jl: unknown queue, pool, stage-aware recovery, drain ordering
- tests for is_binary and handle_unknown_job; tmp_config isolates new dirs
- README: three-stage pipeline
This commit is contained in:
2026-07-02 16:54:17 -04:00
parent 2a46f5021a
commit e42e8ef8af
6 changed files with 227 additions and 83 deletions

24
src/content.jl Normal file
View File

@@ -0,0 +1,24 @@
# Stage-3 content triage for unknown files.
#
# A file that stage-1 couldn't recognize is still sorted into one of two coarse
# buckets so downstream tooling can treat them differently: `text/` for
# human-readable content, `binary/` for everything else. The test is the classic
# "NUL byte in the first sniff window" heuristic that git and file(1) use — cheap
# (no full read), and reliable in practice: text encodings don't embed NUL bytes,
# while binary formats almost always do near the start.
const CONTENT_SNIFF_BYTES = 8000
"""
is_binary(path) -> Bool
Classify a file as binary (`true`) or text (`false`) by sniffing its first
`CONTENT_SNIFF_BYTES` bytes for a NUL byte. An empty file has no NUL, so it is
treated as text.
"""
function is_binary(path::AbstractString)::Bool
open(path, "r") do io
chunk = read(io, CONTENT_SNIFF_BYTES)
return any(==(0x00), chunk)
end
end