Jeffrey Ward
4a22123001
A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.
Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.
Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.
The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.
Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.
Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.
68 lines
3.2 KiB
Julia
68 lines
3.2 KiB
Julia
# Stage-3 content triage for unknown files.
|
||
#
|
||
# A file that stage-1 couldn't recognize is still sorted into one of two coarse
|
||
# buckets so downstream tooling can treat them differently: text (human-readable)
|
||
# and binary (everything else). Only binary is a directory — it is terminal, so
|
||
# the file is committed to `binary/`; a text file has stage 4 still to come, so it
|
||
# stays in `spool/` and only its queue reference moves on. We sniff only the first
|
||
# `CONTENT_SNIFF_BYTES` (no full read) and ask two questions: does the window
|
||
# decode as valid UTF-8, and are any of its control bytes ones that don't belong
|
||
# in text? This is the Unicode-aware successor to the classic "NUL byte" test —
|
||
# it accepts non-ASCII text (accents, CJK, emoji) instead of misfiling it as
|
||
# binary, while still rejecting binary formats, which almost never form valid
|
||
# UTF-8 near their start (and a NUL is never a valid UTF-8 scalar, so it still
|
||
# reads as binary for free).
|
||
|
||
const CONTENT_SNIFF_BYTES = 8000
|
||
|
||
# Control bytes (< 0x20) that appear legitimately in text: BS, TAB, LF, VT, FF,
|
||
# CR, and ESC (ANSI-colored logs). Any *other* control byte is a binary signal.
|
||
const TEXT_CONTROL_BYTES = (0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x1b)
|
||
|
||
# Drop a trailing UTF-8 sequence that the sniff window cut in half, so a
|
||
# multi-byte character straddling the boundary isn't mistaken for invalid bytes.
|
||
# Continuation bytes are 0x80–0xBF; a lead byte encodes its own sequence length
|
||
# in its high bits. We walk back over the trailing continuation bytes, and if
|
||
# the lead byte we land on expects more bytes than the window actually holds,
|
||
# trim the whole incomplete sequence.
|
||
function trim_truncated_utf8(chunk::AbstractVector{UInt8})
|
||
n = length(chunk)
|
||
n == 0 && return chunk
|
||
# Find the start of the final byte sequence: skip back over continuations.
|
||
i = n
|
||
while i > 0 && (chunk[i] & 0xc0) == 0x80
|
||
i -= 1
|
||
end
|
||
i == 0 && return chunk # all continuations; leave as-is
|
||
lead = chunk[i]
|
||
# How many bytes does this lead byte announce?
|
||
expected = lead < 0x80 ? 1 : # ASCII
|
||
lead < 0xe0 ? 2 : # 110xxxxx
|
||
lead < 0xf0 ? 3 : # 1110xxxx
|
||
4 # 11110xxx
|
||
have = n - i + 1
|
||
return have < expected ? view(chunk, 1:i-1) : chunk
|
||
end
|
||
|
||
"""
|
||
is_binary(path) -> Bool
|
||
|
||
Classify a file as binary (`true`) or text (`false`) by sniffing its first
|
||
`CONTENT_SNIFF_BYTES` bytes. A file is text when that window (minus any
|
||
multi-byte character truncated by the window edge) is valid UTF-8 and contains
|
||
no control bytes outside the text-safe set (`TEXT_CONTROL_BYTES`). An empty file
|
||
is treated as text.
|
||
"""
|
||
function is_binary(path::AbstractString)::Bool
|
||
open(path, "r") do io
|
||
chunk = read(io, CONTENT_SNIFF_BYTES)
|
||
isempty(chunk) && return false
|
||
window = trim_truncated_utf8(chunk)
|
||
# Malformed UTF-8 → binary.
|
||
isvalid(String(copy(window))) || return true
|
||
# Valid UTF-8, but a stray non-text control byte still means binary.
|
||
# (NUL is a valid UTF-8 scalar, so it's rejected here, not above.)
|
||
return any(b -> b < 0x20 && !(b in TEXT_CONTROL_BYTES), window)
|
||
end
|
||
end
|