Switch stage-3 triage from NUL sniff to UTF-8 validity

The NUL-byte heuristic misfiled any non-ASCII UTF-8 text (accents, CJK,
emoji) as binary and let non-NUL control bytes through as text. is_binary
now calls a file text when its 8000-byte sniff window is valid UTF-8 with
no control bytes outside the text-safe set (tab/newline/CR/ESC/etc).

- trim_truncated_utf8 drops a multi-byte char split by the window edge so
  it isn't mistaken for malformed bytes.
- NUL still classifies as binary (valid UTF-8 scalar, non-text control).
- Expanded tests: Unicode, ANSI logs, stray control byte, malformed UTF-8,
  boundary-split char; updated README stage-3 description.

src/content.jl

@@ -2,23 +2,64 @@
 #
 # A file that stage-1 couldn't recognize is still sorted into one of two coarse
 # buckets so downstream tooling can treat them differently: `text/` for
-# human-readable content, `binary/` for everything else. The test is the classic
-# "NUL byte in the first sniff window" heuristic that git and file(1) use — cheap
-# (no full read), and reliable in practice: text encodings don't embed NUL bytes,
-# while binary formats almost always do near the start.
+# human-readable content, `binary/` for everything else. We sniff only the first
+# `CONTENT_SNIFF_BYTES` (no full read) and ask two questions: does the window
+# decode as valid UTF-8, and are any of its control bytes ones that don't belong
+# in text? This is the Unicode-aware successor to the classic "NUL byte" test —
+# it accepts non-ASCII text (accents, CJK, emoji) instead of misfiling it as
+# binary, while still rejecting binary formats, which almost never form valid
+# UTF-8 near their start (and a NUL is never a valid UTF-8 scalar, so it still
+# reads as binary for free).
 
 const CONTENT_SNIFF_BYTES = 8000
 
+# Control bytes (< 0x20) that appear legitimately in text: BS, TAB, LF, VT, FF,
+# CR, and ESC (ANSI-colored logs). Any *other* control byte is a binary signal.
+const TEXT_CONTROL_BYTES = (0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x1b)
+
+# Drop a trailing UTF-8 sequence that the sniff window cut in half, so a
+# multi-byte character straddling the boundary isn't mistaken for invalid bytes.
+# Continuation bytes are 0x80–0xBF; a lead byte encodes its own sequence length
+# in its high bits. We walk back over the trailing continuation bytes, and if
+# the lead byte we land on expects more bytes than the window actually holds,
+# trim the whole incomplete sequence.
+function trim_truncated_utf8(chunk::AbstractVector{UInt8})
+    n = length(chunk)
+    n == 0 && return chunk
+    # Find the start of the final byte sequence: skip back over continuations.
+    i = n
+    while i > 0 && (chunk[i] & 0xc0) == 0x80
+        i -= 1
+    end
+    i == 0 && return chunk                 # all continuations; leave as-is
+    lead = chunk[i]
+    # How many bytes does this lead byte announce?
+    expected = lead < 0x80 ? 1 :           # ASCII
+               lead < 0xe0 ? 2 :           # 110xxxxx
+               lead < 0xf0 ? 3 :           # 1110xxxx
+                             4             # 11110xxx
+    have = n - i + 1
+    return have < expected ? view(chunk, 1:i-1) : chunk
+end
+
 """
     is_binary(path) -> Bool
 
 Classify a file as binary (`true`) or text (`false`) by sniffing its first
-`CONTENT_SNIFF_BYTES` bytes for a NUL byte. An empty file has no NUL, so it is
-treated as text.
+`CONTENT_SNIFF_BYTES` bytes. A file is text when that window (minus any
+multi-byte character truncated by the window edge) is valid UTF-8 and contains
+no control bytes outside the text-safe set (`TEXT_CONTROL_BYTES`). An empty file
+is treated as text.
 """
 function is_binary(path::AbstractString)::Bool
     open(path, "r") do io
         chunk = read(io, CONTENT_SNIFF_BYTES)
-        return any(==(0x00), chunk)
+        isempty(chunk) && return false
+        window = trim_truncated_utf8(chunk)
+        # Malformed UTF-8 → binary.
+        isvalid(String(copy(window))) || return true
+        # Valid UTF-8, but a stray non-text control byte still means binary.
+        # (NUL is a valid UTF-8 scalar, so it's rejected here, not above.)
+        return any(b -> b < 0x20 && !(b in TEXT_CONTROL_BYTES), window)
     end
 end