Initial file-ingestion service
REST endpoint (Oxygen.jl POST /upload, multipart) that spools uploaded files to disk, enqueues lightweight references onto a bounded thread-safe work queue, and hands off immediately (202 + job IDs; 503 when full). A configurable pool of worker threads pulls jobs off the queue, logs the received filename (placeholder for real processing), and moves files to done/ on success or failed/ on error. - Queue behind an enqueue!/dequeue!/close! seam for a future RabbitMQ swap - Startup recovery: re-enqueues leftover files in spool/ - Graceful drain on SIGINT and SIGTERM (via atexit) - Env-var config; filenames sanitized + UUID-prefixed on disk Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
133
README.md
Normal file
133
README.md
Normal file
@@ -0,0 +1,133 @@
|
||||
# FileServer
|
||||
|
||||
A minimal Julia service that receives files over HTTP and hands them off to a
|
||||
pool of worker threads for processing. The HTTP endpoint does no real work: it
|
||||
spools each uploaded file to disk, pushes a lightweight reference onto a work
|
||||
queue, and responds immediately — staying free to accept the next upload.
|
||||
|
||||
Right now the "processing" is just logging the received filename, to prove the
|
||||
flow. That's the seam where real heavy-lifting goes later.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
POST /upload (multipart)
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐ spool bytes to disk
|
||||
│ HTTP handler │────────────────────────► data/spool/<uuid>-<name>
|
||||
│ (Oxygen.jl) │
|
||||
└────────┬─────────┘ enqueue reference (non-blocking)
|
||||
│ │
|
||||
▼ ▼
|
||||
202 + job IDs ┌───────────────┐
|
||||
(503 if full) │ work queue │ bounded, thread-safe
|
||||
│ (Channel-ish)│
|
||||
└───────┬───────┘
|
||||
│ dequeue
|
||||
┌───────────────┼───────────────┐
|
||||
▼ ▼ ▼
|
||||
worker 1 worker 2 … worker N (Threads.@spawn)
|
||||
│
|
||||
success ────┴──► data/done/<uuid>-<name>
|
||||
failure ───────► data/failed/<uuid>-<name>
|
||||
```
|
||||
|
||||
Key properties:
|
||||
|
||||
- **Fast intake:** the queue only ever carries small references; file bytes live
|
||||
on disk, so memory stays flat regardless of file size.
|
||||
- **Backpressure:** the queue is bounded (default 1000). When full, uploads get
|
||||
`503 Service Unavailable` instead of silently piling up.
|
||||
- **Crash-resilient:** files survive on disk. On startup, anything left in
|
||||
`data/spool/` is re-enqueued (`recovered = N` in the log).
|
||||
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
|
||||
both stop accepting uploads, drain the queue, wait for in-flight files to
|
||||
finish, then exit. (See "Shutdown" below for one cosmetic caveat on SIGTERM.)
|
||||
- **Safe filenames:** client-supplied names are sanitized and prefixed with a
|
||||
server-minted UUID before touching the filesystem (no path traversal).
|
||||
|
||||
## The queue seam (→ RabbitMQ later)
|
||||
|
||||
The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and
|
||||
`close!` on a `JobQueue` (see `src/queue.jl`). Today that's an in-process
|
||||
`ChannelQueue`. To move to RabbitMQ (or any broker), implement a new `JobQueue`
|
||||
subtype with those three methods and swap the construction in `run` — no handler
|
||||
or worker code changes.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
# install deps (first time)
|
||||
julia --project=. -e 'using Pkg; Pkg.instantiate()'
|
||||
|
||||
# start the server; -t sets the number of OS threads available to workers
|
||||
julia --project=. -t auto bin/server.jl
|
||||
```
|
||||
|
||||
## Shutdown
|
||||
|
||||
Both SIGINT and SIGTERM trigger the same idempotent graceful drain
|
||||
(stop serving → close queue → wait for workers → exit):
|
||||
|
||||
- **SIGINT** is caught as an `InterruptException` (we call
|
||||
`Base.exit_on_sigint(false)`), so shutdown is clean and quiet.
|
||||
- **SIGTERM** can't be intercepted directly — Julia blocks it on worker threads
|
||||
and handles it in its own runtime, so a user `signal()` handler never fires.
|
||||
Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path
|
||||
does run. Caveat: Julia prints its own `signal 15: Terminated` backtrace
|
||||
*before* `atexit` runs. It's harmless noise — the drain still completes right
|
||||
after it — but if you want a fully quiet stop under a process manager,
|
||||
configure it to send SIGINT instead (systemd: `KillSignal=SIGINT`; Docker:
|
||||
`STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain
|
||||
in-flight work (systemd: `TimeoutStopSec`).
|
||||
|
||||
## Configuration (environment variables)
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---------------------|----------------|------------------------------------------|
|
||||
| `FS_HOST` | `127.0.0.1` | Bind address |
|
||||
| `FS_PORT` | `8080` | Port |
|
||||
| `FS_WORKERS` | `nthreads()` | Number of worker tasks |
|
||||
| `FS_QUEUE_CAPACITY` | `1000` | Max pending jobs before `503` |
|
||||
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending) |
|
||||
| `FS_DONE_DIR` | `data/done` | Files after successful processing |
|
||||
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
|
||||
|
||||
> To get real parallelism, start Julia with enough threads (`-t N`) to match
|
||||
> `FS_WORKERS`. If `FS_WORKERS` exceeds available threads you'll get a warning
|
||||
> and workers will share threads.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# health check
|
||||
curl http://127.0.0.1:8080/health
|
||||
# {"status":"ok"}
|
||||
|
||||
# upload one or more files (multipart/form-data)
|
||||
curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload
|
||||
# 202 {"accepted":[{"id":"<uuid>","name":"report.pdf"}, ...]}
|
||||
```
|
||||
|
||||
Each file in a request becomes its own job. Responses:
|
||||
|
||||
- `202 Accepted` — all files spooled and queued (with per-file job IDs)
|
||||
- `400 Bad Request` — not multipart, or no files present
|
||||
- `503 Service Unavailable` — queue full, retry later
|
||||
- `500 Internal Server Error` — failed to write a file to disk
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
src/
|
||||
FileServer.jl module + run() (startup, recovery, workers, serve, shutdown)
|
||||
config.jl Config struct + env parsing
|
||||
job.jl Job (the queue reference)
|
||||
queue.jl JobQueue seam + in-process ChannelQueue
|
||||
spool.jl filename sanitizing, spool/move, startup recovery
|
||||
worker.jl worker loop + per-job processing (placeholder)
|
||||
server.jl HTTP routes/handlers
|
||||
bin/
|
||||
server.jl entry point
|
||||
```
|
||||
Reference in New Issue
Block a user