file-server/README.md

FileServer

A minimal Julia service that receives files over HTTP and hands them off to a pool of worker threads for processing. The HTTP endpoint does no real work: it spools each uploaded file to disk, pushes a lightweight reference onto a work queue, and responds immediately — staying free to accept the next upload.

Right now the "processing" is just logging the received filename, to prove the flow. That's the seam where real heavy-lifting goes later.

Architecture

        POST /upload (multipart)
                 │
                 ▼
        ┌─────────────────┐     spool bytes to disk
        │  HTTP handler    │────────────────────────► data/spool/<uuid>-<name>
        │  (Oxygen.jl)     │
        └────────┬─────────┘     enqueue reference (non-blocking)
                 │                        │
                 ▼                        ▼
          202 + job IDs           ┌───────────────┐
          (503 if full)           │  work queue   │  bounded, thread-safe
                                  │  (Channel-ish)│
                                  └───────┬───────┘
                                          │ dequeue
                          ┌───────────────┼───────────────┐
                          ▼               ▼               ▼
                      worker 1        worker 2   …    worker N   (Threads.@spawn)
                          │
              success ────┴──► data/done/<uuid>-<name>
              failure  ───────► data/failed/<uuid>-<name>

Key properties:

• Fast intake: the queue only ever carries small references; file bytes live on disk, so memory stays flat regardless of file size.
• Backpressure: the queue is bounded (default 1000). When full, uploads get 503 Service Unavailable instead of silently piling up.
• Crash-resilient: files survive on disk. On startup, anything left in data/spool/ is re-enqueued (recovered = N in the log).
• Graceful shutdown: SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s stop) both stop accepting uploads, drain the queue, wait for in-flight files to finish, then exit. (See "Shutdown" below for one cosmetic caveat on SIGTERM.)
• Safe filenames: client-supplied names are sanitized and prefixed with a server-minted UUID before touching the filesystem (no path traversal).

The queue seam (→ RabbitMQ later)

The HTTP handler and workers only ever call enqueue!, dequeue!, and close! on a JobQueue (see src/queue.jl). Today that's an in-process ChannelQueue. To move to RabbitMQ (or any broker), implement a new JobQueue subtype with those three methods and swap the construction in run — no handler or worker code changes.

Running

# install deps (first time)
julia --project=. -e 'using Pkg; Pkg.instantiate()'

# start the server; -t sets the number of OS threads available to workers
julia --project=. -t auto bin/server.jl

Shutdown

Both SIGINT and SIGTERM trigger the same idempotent graceful drain (stop serving → close queue → wait for workers → exit):

• SIGINT is caught as an InterruptException (we call Base.exit_on_sigint(false)), so shutdown is clean and quiet.
• SIGTERM can't be intercepted directly — Julia blocks it on worker threads and handles it in its own runtime, so a user signal() handler never fires. Instead we hook the drain into an atexit handler, which Julia's SIGTERM path does run. Caveat: Julia prints its own signal 15: Terminated backtrace before atexit runs. It's harmless noise — the drain still completes right after it — but if you want a fully quiet stop under a process manager, configure it to send SIGINT instead (systemd: KillSignal=SIGINT; Docker: STOPSIGNAL SIGINT). Give the stop timeout enough headroom to drain in-flight work (systemd: TimeoutStopSec).

Configuration (environment variables)

Variable	Default	Meaning
FS_HOST	127.0.0.1	Bind address
FS_PORT	8080	Port
FS_WORKERS	nthreads()	Number of worker tasks
FS_QUEUE_CAPACITY	1000	Max pending jobs before 503
FS_SPOOL_DIR	data/spool	Incoming files (pending)
FS_DONE_DIR	data/done	Files after successful processing
FS_FAILED_DIR	data/failed	Files whose processing threw

To get real parallelism, start Julia with enough threads (-t N) to match FS_WORKERS. If FS_WORKERS exceeds available threads you'll get a warning and workers will share threads.

Usage

# health check
curl http://127.0.0.1:8080/health
# {"status":"ok"}

# upload one or more files (multipart/form-data)
curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload
# 202 {"accepted":[{"id":"<uuid>","name":"report.pdf"}, ...]}

Each file in a request becomes its own job. Responses:

• 202 Accepted — all files spooled and queued (with per-file job IDs)
• 400 Bad Request — not multipart, or no files present
• 503 Service Unavailable — queue full, retry later
• 500 Internal Server Error — failed to write a file to disk

Layout

src/
  FileServer.jl   module + run() (startup, recovery, workers, serve, shutdown)
  config.jl       Config struct + env parsing
  job.jl          Job (the queue reference)
  queue.jl        JobQueue seam + in-process ChannelQueue
  spool.jl        filename sanitizing, spool/move, startup recovery
  worker.jl       worker loop + per-job processing (placeholder)
  server.jl       HTTP routes/handlers
bin/
  server.jl       entry point