Added RabbitMQ

README.md

@@ -102,13 +102,14 @@ Key properties:
   it is abandoned.
 
   The cost of replay is redoing stages a file had already cleared. That is a
-  property of the *queue*, not of the directory layout: the queues are
+  property of the *queue*, not of the directory layout: the default queues are
   in-process (`src/queue.jl`), so a crash destroys the only record of how far
   each file got. Per-stage directories used to stand in for that record, at the
   price of a rename per file per stage on the hot path: a permanent cost on
-  every file, to buy a cheaper restart. `src/queue.jl` already defines the seam
-  for the real fix: swap `ChannelQueue` for a broker-backed `JobQueue` and exact
-  resume comes back durably, rather than being inferred from a pathname.
+  every file, to buy a cheaper restart. The seam in `src/queue.jl` is where that
+  is actually fixed: run with `FS_QUEUE_BACKEND=rabbitmq` and a job stays unacked
+  until its handler commits, so a restart resumes each file at the stage it had
+  reached instead of re-driving `spool/` from stage 1. See "Queue backends".
 - **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
   both stop accepting uploads, then drain the stages *in order*: close the
   stage-1 queue and wait out the classify workers (the only producer of the known
@@ -336,13 +337,94 @@ baseline agrees. See `DESIGN_clustering.md` §11 for the full results, including
 one known limitation: ELF and these tarballs share a long run of header
 zero-padding and merge. The v2 fix is inverse-entropy position weighting.
 
-## The queue seam (→ RabbitMQ later)
+## Queue backends
 
-The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and
-`close!` on a `JobQueue` (see `src/queue.jl`). Today that's an in-process
-`ChannelQueue`. To move to RabbitMQ (or any broker), implement a new `JobQueue`
-subtype with those three methods and swap the construction in `run`. No handler
-or worker code changes.
+The HTTP handler and the workers only ever call `enqueue!`, `dequeue!`, `ack!`,
+`nack!` and `close!` on a `JobQueue` (`src/queue.jl`). Two implementations sit
+behind those five methods, chosen at startup by `FS_QUEUE_BACKEND`:
+
+| | `channel` (default) | `rabbitmq` |
+|---|---|---|
+| Where jobs live | in-process, bounded buffer | durable broker queues, persistent messages |
+| Crash recovery | everything in `spool/` replays from stage 1 | each file resumes at the stage it had reached |
+| External dependency | none | a RabbitMQ broker |
+| Capacity | hard limit, enforced per enqueue | advisory, checked against a polled depth |
+| Delivery | exactly once (nothing to redeliver) | at least once |
+
+`ack!` is the whole difference. A job is settled only after its handler commits,
+so a crash mid-enrichment leaves that job on the enrich queue and the restart
+picks it up there — not at stage 1, and not lost. `ChannelQueue` implements
+`ack!`/`nack!` as no-ops, which is honest rather than lazy: an in-process queue
+has no delivery to settle, and its recovery story is `recover_dir!`.
+
+### Running it
+
+```bash
+# broker + server, both in compose
+docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up
+
+# just the broker, with the server on the host (5672 and the management UI on
+# 15672 are published for exactly this)
+docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up -d rabbitmq
+FS_QUEUE_BACKEND=rabbitmq \
+FS_AMQP_URL=amqp://fileserver:fileserver@localhost:5672/ \
+  julia --project=. -t auto bin/server.jl
+```
+
+Queues are named `<FS_AMQP_PREFIX>.<stage>` — `fileserver.classify`,
+`.enrich`, `.triage`, `.language` — so the broker's queue list reads like the
+pipeline, and `/stats` reports each one's depth from a once-a-second poll.
+
+### What it guarantees, and what it deliberately doesn't
+
+- **At least once, not exactly once.** Stages 1 and 3 publish downstream *then*
+  ack upstream, so a crash in that window redelivers a job that was already
+  routed. Safe for the same reason replay is: every stage is a pure function of
+  the file's bytes and every commit is idempotent. The one new case is that a
+  duplicate can run *concurrently* with the original and find the file already
+  committed; `worker_loop` recognises a vanished `job.path` and settles it
+  quietly, rather than quarantining a file that in fact succeeded.
+- **Publisher confirms on intake only.** A `202` means the broker has the file,
+  because a client may delete its copy on the strength of it. That costs a round
+  trip per file and serializes intake publishes; `FS_AMQP_CONFIRMS=false` turns
+  it off. Inter-stage publishes are fire-and-forget by design — a lost one leaves
+  its source job unacked, and redelivery repairs it for free, on a handoff that
+  otherwise costs 0.12 µs.
+- **Advisory capacity.** `enqueue!` compares `FS_QUEUE_CAPACITY` against a depth
+  polled once a second (and adjusted locally in between), so the existing `503`
+  and `blocked_ns` behaviour still works, but a burst can overshoot by up to a
+  poll interval. Making the limit real would mean `x-max-length` with
+  `overflow: reject-publish`, which needs a confirm per message to detect.
+- **No reconnect.** A dropped connection invalidates every in-flight delivery
+  tag, so reconnecting would silently reprocess whatever the workers were
+  holding. Instead the server drains and exits non-zero, and the restart policy
+  brings it back to redelivered messages and an intact `spool/`.
+- **No dead-letter queue.** A failed job is quarantined to `failed/` and then
+  acked: the file's story and the message's story end in the same place. A DLQ
+  would hold messages pointing at files that had already moved.
+- **One consumer process.** `Job` is a claim check carrying a *local* path, so a
+  second server on another host would be handed jobs whose files it cannot see.
+  The broker buys durable resume across restarts, not horizontal scale; scaling
+  out would additionally need `spool/` on shared storage and an aggregated
+  `/stats`.
+- **Restart recovery skips `spool/`.** The broker is the record of what is in
+  flight, so re-driving `spool/` would duplicate the whole backlog on every
+  restart. `FS_RECOVER_SPOOL=true` forces it, for the one case the broker cannot
+  cover: it was purged or recreated and the spooled files are all that is left.
+
+### Tests
+
+The pure parts (URL parsing, the wire format, backend selection, the duplicate
+branch) run in the normal suite. The round trip against a real broker is gated:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up -d rabbitmq
+FS_TEST_AMQP_URL=amqp://fileserver:fileserver@localhost:5672/ \
+  julia --project=. -e 'using Pkg; Pkg.test()'
+```
+
+Without `FS_TEST_AMQP_URL` those tests are skipped with a notice, so the suite
+still passes on a machine with no Docker.
 
 ## Running
 
@@ -359,6 +441,10 @@ julia --project=. -e 'using Pkg; Pkg.instantiate()'
 julia --project=. -t auto bin/server.jl
 ```
 
+By default the pipeline's queues are in-process, and nothing external is needed.
+For durable queues that survive a crash, run against RabbitMQ instead — see
+"Queue backends" above.
+
 ## Shutdown
 
 Both SIGINT and SIGTERM trigger the same idempotent graceful drain
@@ -366,6 +452,19 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
 
 - **SIGINT** is caught as an `InterruptException` (we call
   `Base.exit_on_sigint(false)`), so shutdown is clean and quiet.
+
+  One caveat specific to the RabbitMQ backend: Julia delivers SIGINT to whichever
+  task happens to be running on thread 1, which is usually the main loop but is
+  not guaranteed to be — AMQPClient runs receiver tasks of its own, and an
+  interrupt that lands in one of those kills the broker connection instead of
+  reaching the main loop. That is not a stuck server: the depth poller notices
+  the closed connection within a second and asks for the same drain, so the
+  process still stops (within ~4s, exiting non-zero, and logging it as a lost
+  connection rather than an interrupt). **Under a process manager, prefer SIGTERM
+  for the broker backend** — it goes through `atexit`, which has no such lottery.
+  Shutdown on either signal also prints a few `Consumer ... task exiting` warnings
+  from AMQPClient, which are cosmetic: they are its consumer tasks noticing that
+  we cancelled them.
 - **SIGTERM** can't be intercepted directly: Julia blocks it on worker threads
   and handles it in its own runtime, so a user `signal()` handler never fires.
   Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path
@@ -454,6 +553,12 @@ init, so the artifact is exactly regenerable from the same inputs.
 | `FS_PROMOTE_MIN_MAGIC`    | `3`            | Required fixed signature positions to nominate |
 | `FS_CLUSTER_CATALOG`      | `data/catalog.json` | Durable stage-5 catalog file (single-owner) |
 | `FS_NOMINATED_DIR`        | `data/nominated` | One JSON per self-nominated cluster (human promote gate) |
+| `FS_QUEUE_BACKEND`        | `channel`      | `channel` (in-process) or `rabbitmq` (durable); see "Queue backends" |
+| `FS_AMQP_URL`             | `amqp://guest:guest@localhost:5672/` | Broker connection, credentials included |
+| `FS_AMQP_PREFIX`          | `fileserver`   | Queue names are `<prefix>.<stage>`           |
+| `FS_AMQP_PREFETCH`        | worker count   | Unacked messages the broker hands one stage at a time |
+| `FS_AMQP_CONFIRMS`        | `true`         | Wait for a publisher confirm before a `202` (intake only) |
+| `FS_RECOVER_SPOOL`        | `false`        | Re-drive `spool/` at startup even on the broker backend (use after a purged broker) |
 
 > To get real parallelism, start Julia with enough threads (`-t N`) to cover all
 > pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS`
@@ -869,6 +974,7 @@ src/
   config.jl       Config struct + env parsing
   job.jl          Job (the queue reference)
   queue.jl        JobQueue seam + in-process ChannelQueue
+  rabbit.jl       RabbitMQ-backed JobQueue: durable queues, acks, confirms
   stats.jl        per-stage counters behind GET /stats (throughput, utilization)
   multipart.jl    streaming multipart/form-data reader (intake never buffers a file)
   spool.jl        filename sanitizing, streaming spool/move, startup recovery
@@ -881,6 +987,8 @@ src/
   catalog.jl      durable single-owner format catalog + sweep + nominations (stage 5, phase B)
   worker.jl       parametrized worker loop + classify/enrich/triage/language handlers
   server.jl       HTTP routes + the streaming /upload handler
+docker-compose.yml           the server, in-process queues
+docker-compose.rabbitmq.yml  overlay: adds the broker and switches the backend
 bin/
   server.jl           entry point
   bench.jl            throughput + memory harness against a running server