Add stage-3 content triage: sort unknown files into binary/ and text/
Unknown files are no longer terminal. Stage 1 now routes :unknown onto a dedicated queue (with the same blocking backpressure as the known queue), and a third worker pool sorts each file into data/binary/ or data/text/ using a NUL-byte sniff of the first 8000 bytes. - content.jl: is_binary content sniff (stage 3) - worker.jl: handle_unknown_job; stage-1 routes unknown with backpressure; KNOWN_ENQUEUE_RETRY_SECONDS -> ROUTE_ENQUEUE_RETRY_SECONDS (serves both) - config.jl: unknown_worker_count/queue_capacity, binary_dir, text_dir + env - FileServer.jl: unknown queue, pool, stage-aware recovery, drain ordering - tests for is_binary and handle_unknown_job; tmp_config isolates new dirs - README: three-stage pipeline
This commit is contained in:
76
README.md
76
README.md
@@ -11,9 +11,9 @@ classifier that labels it **known** (a file type resembling the training set) or
|
||||
|
||||
## Architecture
|
||||
|
||||
The pipeline is two stages, each with its own bounded queue and its own worker
|
||||
pool (tuned independently, since classification is CPU-bound and enrichment is
|
||||
process-/IO-bound):
|
||||
The pipeline is three stages, each with its own bounded queue and its own worker
|
||||
pool (tuned independently, since classification is CPU-bound, enrichment is
|
||||
process-/IO-bound, and content triage is cheap IO):
|
||||
|
||||
```
|
||||
POST /upload (multipart)
|
||||
@@ -35,21 +35,25 @@ process-/IO-bound):
|
||||
│
|
||||
┌────────────┴────────────┐
|
||||
:unknown :known
|
||||
│ │ move to data/known/, then
|
||||
▼ ▼ enqueue (blocking backpressure)
|
||||
data/unknown/<uuid>-<name> ┌────────────────────┐
|
||||
(parked; future pipeline) │ known queue │ enrichment
|
||||
└─────────┬──────────┘
|
||||
│ dequeue
|
||||
┌───────────────────┼───────────────────┐
|
||||
▼ ▼ ▼
|
||||
known wkr 1 known wkr 2 … known wkr M
|
||||
│ exiftool → normalized sidecar
|
||||
success ────┴──► data/done/<uuid>-<name>
|
||||
data/done/<uuid>-<name>.meta.json (sidecar-first commit)
|
||||
failure ───────► data/failed/<uuid>-<name>
|
||||
│ move to data/unknown/, │ move to data/known/, then
|
||||
▼ then enqueue (blocking) ▼ enqueue (blocking backpressure)
|
||||
┌────────────────────┐ ┌────────────────────┐
|
||||
│ unknown queue │ │ known queue │ enrichment
|
||||
└─────────┬──────────┘ └─────────┬──────────┘
|
||||
│ dequeue │ dequeue
|
||||
┌────────┼────────┐ ┌───────────┼───────────┐
|
||||
▼ ▼ ▼ ▼ ▼ ▼
|
||||
unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M
|
||||
│ binary-vs-text sniff │ exiftool → normalized sidecar
|
||||
├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name>
|
||||
└─► data/text/<uuid>-<name> data/done/<uuid>-<name>.meta.json
|
||||
(sidecar-first commit)
|
||||
failure ───────► data/failed/<uuid>-<name>
|
||||
```
|
||||
|
||||
Stages 2 (enrichment) and 3 (content triage) run in parallel: stage 1 feeds both
|
||||
the known and unknown queues.
|
||||
|
||||
Key properties:
|
||||
|
||||
- **Fast intake:** the queue only ever carries small references; file bytes live
|
||||
@@ -58,13 +62,15 @@ Key properties:
|
||||
is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
|
||||
the stage-1 worker blocks and retries (a classified file is never dropped).
|
||||
- **Crash-resilient:** files survive on disk. On startup, recovery is
|
||||
stage-aware: leftovers in `data/spool/` re-enter classification and leftovers
|
||||
in `data/known/` re-enter enrichment (`recovered` / `recovered_known` in the
|
||||
log), so a file resumes at its correct stage instead of restarting from scratch.
|
||||
stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/`
|
||||
re-enter enrichment, and `data/unknown/` re-enter content triage (`recovered` /
|
||||
`recovered_known` / `recovered_unknown` in the log), so a file resumes at its
|
||||
correct stage instead of restarting from scratch.
|
||||
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
|
||||
both stop accepting uploads, then drain the stages *in order* — close the
|
||||
stage-1 queue and wait out the classify workers (the only producer of the known
|
||||
queue) before closing the known queue and waiting out the enrich workers.
|
||||
*and* unknown queues) before closing those two queues and waiting out the
|
||||
enrich and content-triage workers.
|
||||
(See "Shutdown" below for one cosmetic caveat on SIGTERM.)
|
||||
- **Safe filenames:** client-supplied names are sanitized and prefixed with a
|
||||
server-minted UUID before touching the filesystem (no path traversal).
|
||||
@@ -111,6 +117,21 @@ file's presence there always implies its sidecar is already present; a crash in
|
||||
between leaves only a harmless orphan sidecar, and recovery re-enriches
|
||||
idempotently.
|
||||
|
||||
### Content triage (stage 3)
|
||||
|
||||
Files the classifier labels **unknown** are handed to a third pool that sorts
|
||||
them into two coarse buckets so downstream tooling can treat them differently:
|
||||
|
||||
- **`data/binary/`** — the file looks like binary data.
|
||||
- **`data/text/`** — the file looks like text.
|
||||
|
||||
The test is the classic **NUL-byte sniff** (the same heuristic `git` and
|
||||
`file(1)` use): read the first 8000 bytes and, if any is NUL, call it binary,
|
||||
else text. It's cheap (no full read) and reliable in practice — text encodings
|
||||
don't embed NUL bytes, while binary formats almost always do near the start. An
|
||||
empty file has no NUL, so it's treated as text. This is deliberately simple for
|
||||
now; richer handling can hang off either bucket later (`src/content.jl`).
|
||||
|
||||
## The queue seam (→ RabbitMQ later)
|
||||
|
||||
The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and
|
||||
@@ -199,17 +220,21 @@ init, so the artifact is exactly regenerable from the same inputs.
|
||||
| `FS_QUEUE_CAPACITY` | `1000` | Max pending intake jobs before `503` |
|
||||
| `FS_KNOWN_WORKERS` | `nthreads()` | Stage-2 (enrichment) worker tasks |
|
||||
| `FS_KNOWN_QUEUE_CAPACITY` | `1000` | Max pending enrichment jobs (then backpressure) |
|
||||
| `FS_UNKNOWN_WORKERS` | `nthreads()` | Stage-3 (content triage) worker tasks |
|
||||
| `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) |
|
||||
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending classification) |
|
||||
| `FS_KNOWN_DIR` | `data/known` | Classified-known, awaiting enrichment |
|
||||
| `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, parked for a future pool |
|
||||
| `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, awaiting content triage |
|
||||
| `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary |
|
||||
| `FS_TEXT_DIR` | `data/text` | Stage-3 sink: unknown files that look like text |
|
||||
| `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) |
|
||||
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
|
||||
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
|
||||
| `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed |
|
||||
|
||||
> To get real parallelism, start Julia with enough threads (`-t N`) to cover both
|
||||
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS` exceeds available threads you'll get a
|
||||
> warning (non-fatal) and workers will share threads.
|
||||
> To get real parallelism, start Julia with enough threads (`-t N`) to cover all
|
||||
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS` exceeds available
|
||||
> threads you'll get a warning (non-fatal) and workers will share threads.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -242,7 +267,8 @@ src/
|
||||
model.jl NN architecture + byte→feature mapping (shared with trainer)
|
||||
classify.jl load artifact + classify a file at inference time
|
||||
metadata.jl exiftool extraction + normalized sidecar (stage 2)
|
||||
worker.jl parametrized worker loop + classify/enrich handlers
|
||||
content.jl binary-vs-text sniff for unknown files (stage 3)
|
||||
worker.jl parametrized worker loop + classify/enrich/triage handlers
|
||||
server.jl HTTP routes/handlers
|
||||
bin/
|
||||
server.jl entry point
|
||||
|
||||
Reference in New Issue
Block a user