Add stage-4 language enrichment for text files
Text files sorted by stage 3 now flow onto a new work queue and worker pool that enrich them with natural language (Languages.jl LanguageDetector: name, ISO 639-3 code, confidence) and programming language (github-linguist), writing a .meta.json sidecar to data/text_done/ like the stage-2 known-file pipeline. github-linguist reads the git blob of a path inside a repo, so untracked data/ files are copied to /tmp (outside any repo, name preserved for extension heuristics) before detection. Programming-language lookup is best-effort (startup warning if missing, degraded/null on failure); natural-language failure yields a degraded sidecar, not a quarantine. Factored exiftool's timeout-kill into shared run_with_timeout and the durable sidecar-first commit into commit_enriched!, both reused by stage 4. Recovery re-drives data/text/; graceful drain closes the text queue after its stage-3 producers finish.
This commit is contained in:
109
README.md
109
README.md
@@ -11,9 +11,10 @@ classifier that labels it **known** (a file type resembling the training set) or
|
||||
|
||||
## Architecture
|
||||
|
||||
The pipeline is three stages, each with its own bounded queue and its own worker
|
||||
pool (tuned independently, since classification is CPU-bound, enrichment is
|
||||
process-/IO-bound, and content triage is cheap IO):
|
||||
The pipeline is four stages, each with its own bounded queue and its own worker
|
||||
pool (tuned independently, since classification is CPU-bound, known-file
|
||||
enrichment is process-/IO-bound, content triage is cheap IO, and language
|
||||
enrichment mixes CPU with a subprocess):
|
||||
|
||||
```
|
||||
POST /upload (multipart)
|
||||
@@ -46,13 +47,25 @@ process-/IO-bound, and content triage is cheap IO):
|
||||
unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M
|
||||
│ binary-vs-text sniff │ exiftool → normalized sidecar
|
||||
├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name>
|
||||
└─► data/text/<uuid>-<name> data/done/<uuid>-<name>.meta.json
|
||||
(sidecar-first commit)
|
||||
failure ───────► data/failed/<uuid>-<name>
|
||||
│ (terminal) data/done/<uuid>-<name>.meta.json
|
||||
│ (sidecar-first commit)
|
||||
│ :text move to data/text/, failure ───────► data/failed/<uuid>-<name>
|
||||
▼ then enqueue (blocking backpressure)
|
||||
┌────────────────────┐
|
||||
│ text queue │ language enrichment
|
||||
└─────────┬──────────┘
|
||||
│ dequeue
|
||||
┌────────┼────────┐
|
||||
▼ ▼ ▼
|
||||
txt 1 txt 2 … txt P
|
||||
│ Languages.jl (natural language) + github-linguist (programming language)
|
||||
└─► data/text_done/<uuid>-<name> + data/text_done/<uuid>-<name>.meta.json
|
||||
(sidecar-first commit)
|
||||
```
|
||||
|
||||
Stages 2 (enrichment) and 3 (content triage) run in parallel: stage 1 feeds both
|
||||
the known and unknown queues.
|
||||
Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1
|
||||
feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language
|
||||
enrichment) for every file it sorts as text.
|
||||
|
||||
Key properties:
|
||||
|
||||
@@ -63,14 +76,16 @@ Key properties:
|
||||
the stage-1 worker blocks and retries (a classified file is never dropped).
|
||||
- **Crash-resilient:** files survive on disk. On startup, recovery is
|
||||
stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/`
|
||||
re-enter enrichment, and `data/unknown/` re-enter content triage (`recovered` /
|
||||
`recovered_known` / `recovered_unknown` in the log), so a file resumes at its
|
||||
re-enter enrichment, `data/unknown/` re-enter content triage, and `data/text/`
|
||||
re-enter language enrichment (`recovered` / `recovered_known` /
|
||||
`recovered_unknown` / `recovered_text` in the log), so a file resumes at its
|
||||
correct stage instead of restarting from scratch.
|
||||
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
|
||||
both stop accepting uploads, then drain the stages *in order* — close the
|
||||
stage-1 queue and wait out the classify workers (the only producer of the known
|
||||
*and* unknown queues) before closing those two queues and waiting out the
|
||||
enrich and content-triage workers.
|
||||
*and* unknown queues), then close those queues and wait out the enrich and
|
||||
content-triage workers (content triage being the only producer of the text
|
||||
queue), then close the text queue and wait out the language-enrichment workers.
|
||||
(See "Shutdown" below for one cosmetic caveat on SIGTERM.)
|
||||
- **Safe filenames:** client-supplied names are sanitized and prefixed with a
|
||||
server-minted UUID before touching the filesystem (no path traversal).
|
||||
@@ -134,8 +149,55 @@ or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in
|
||||
UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but
|
||||
not a text control byte, so it still reads as binary. A multi-byte character
|
||||
split by the 8000-byte boundary is trimmed before the check so it isn't mistaken
|
||||
for malformed bytes. An empty file is treated as text. Richer handling can hang
|
||||
off either bucket later (`src/content.jl`).
|
||||
for malformed bytes. An empty file is treated as text. `binary/` is terminal;
|
||||
`text/` is handed to stage 4 (`src/content.jl`).
|
||||
|
||||
### Language enrichment (stage 4)
|
||||
|
||||
Files that stage 3 sorts as **text** are handed to a fourth pool that identifies
|
||||
their language and writes a `.meta.json` sidecar, mirroring the stage-2
|
||||
known-file enrichment. Two detectors run per file:
|
||||
|
||||
- **natural language** — [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s
|
||||
`LanguageDetector` (a Julia port of the `whatlang` n-gram model) reads a bounded
|
||||
prefix (up to `LANG_SAMPLE_BYTES`, 64 KiB) and reports the language's English
|
||||
name, ISO 639-3 code, and a confidence in `[0,1]`. Pure Julia, no subprocess.
|
||||
The detector is built once at startup and shared read-only across the pool.
|
||||
- **programming language** — the [`github-linguist`](https://github.com/github-linguist/linguist)
|
||||
CLI recognizes source and markup by extension + content heuristics (e.g.
|
||||
`Python`, `Markdown`). Plain prose reports as `Text` and unrecognized content as
|
||||
`null`; both collapse to *no programming language*.
|
||||
|
||||
The sidecar schema:
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `id`, `original_name` | from intake |
|
||||
| `file_size` | bytes (authoritative, from intake) |
|
||||
| `content_type` | always `"text"` |
|
||||
| `language` | natural-language English name (e.g. `English`), or `null` |
|
||||
| `language_code` | ISO 639-3 code (e.g. `eng`), or `null` |
|
||||
| `language_confidence` | detector confidence in `[0,1]`, or `null` |
|
||||
| `programming_language` | e.g. `Python`, `Markdown`, or `null` |
|
||||
| `error` | set if natural-language detection produced nothing |
|
||||
|
||||
> **`github-linguist` and the git-repo quirk:** run against a path *inside* a git
|
||||
> repository, linguist reads the file's committed git blob, not the on-disk bytes
|
||||
> — and an untracked file (which everything under `data/` is) has no blob, so it
|
||||
> crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under
|
||||
> `/tmp` (outside any repo, preserving the name so extension heuristics still
|
||||
> fire) and pointing linguist there.
|
||||
>
|
||||
> Programming-language detection is **best-effort**: if `github-linguist` is
|
||||
> missing (a startup warning, not a fatal error, unlike `exiftool`), fails, or
|
||||
> times out (`FS_LINGUIST_TIMEOUT`, default 30s), `programming_language` is simply
|
||||
> `null` and the file still completes. Natural-language detection failing produces
|
||||
> a **degraded sidecar** (with an `error` note) rather than a quarantine, because
|
||||
> the file is still wanted text.
|
||||
|
||||
Like stage 2, the sidecar is committed **before** the file is moved into
|
||||
`data/text_done/`, so the file's presence there always implies its sidecar is
|
||||
present; recovery re-enriches idempotently (`src/language.jl`).
|
||||
|
||||
## The queue seam (→ RabbitMQ later)
|
||||
|
||||
@@ -151,6 +213,11 @@ or worker code changes.
|
||||
# install deps (first time)
|
||||
julia --project=. -e 'using Pkg; Pkg.instantiate()'
|
||||
|
||||
# external tools: exiftool (stage 2, required) and github-linguist (stage 4,
|
||||
# optional — programming-language detection). e.g. on Debian/Ubuntu:
|
||||
# apt install libimage-exiftool-perl
|
||||
# gem install github-linguist
|
||||
|
||||
# start the server; -t sets the number of OS threads available to workers
|
||||
julia --project=. -t auto bin/server.jl
|
||||
```
|
||||
@@ -227,19 +294,24 @@ init, so the artifact is exactly regenerable from the same inputs.
|
||||
| `FS_KNOWN_QUEUE_CAPACITY` | `1000` | Max pending enrichment jobs (then backpressure) |
|
||||
| `FS_UNKNOWN_WORKERS` | `nthreads()` | Stage-3 (content triage) worker tasks |
|
||||
| `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) |
|
||||
| `FS_TEXT_WORKERS` | `nthreads()` | Stage-4 (language enrichment) worker tasks |
|
||||
| `FS_TEXT_QUEUE_CAPACITY` | `1000` | Max pending language jobs (then backpressure) |
|
||||
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending classification) |
|
||||
| `FS_KNOWN_DIR` | `data/known` | Classified-known, awaiting enrichment |
|
||||
| `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, awaiting content triage |
|
||||
| `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary |
|
||||
| `FS_TEXT_DIR` | `data/text` | Stage-3 sink: unknown files that look like text |
|
||||
| `FS_TEXT_DIR` | `data/text` | Classified-text, awaiting language enrichment |
|
||||
| `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) |
|
||||
| `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) |
|
||||
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
|
||||
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
|
||||
| `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed |
|
||||
| `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed |
|
||||
|
||||
> To get real parallelism, start Julia with enough threads (`-t N`) to cover all
|
||||
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS` exceeds available
|
||||
> threads you'll get a warning (non-fatal) and workers will share threads.
|
||||
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS`
|
||||
> exceeds available threads you'll get a warning (non-fatal) and workers will
|
||||
> share threads.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -273,7 +345,8 @@ src/
|
||||
classify.jl load artifact + classify a file at inference time
|
||||
metadata.jl exiftool extraction + normalized sidecar (stage 2)
|
||||
content.jl binary-vs-text sniff for unknown files (stage 3)
|
||||
worker.jl parametrized worker loop + classify/enrich/triage handlers
|
||||
language.jl natural + programming language enrichment for text (stage 4)
|
||||
worker.jl parametrized worker loop + classify/enrich/triage/language handlers
|
||||
server.jl HTTP routes/handlers
|
||||
bin/
|
||||
server.jl entry point
|
||||
|
||||
Reference in New Issue
Block a user