Store

Every task is one JSON file on disc, and its state is the directory it sits in. There is no database. The filesystem is the queue. That is what makes a task durable, and it is why the lifecycle can be reasoned about so plainly. A separate, durable event log records what each task did, out of band of the queue's short retention.

On-disc layout

Under the server's items directory (/srv/taskhoster-items), one folder per host, then per queue, then three state directories:

/srv/taskhoster-items/
  task.example.com/               host
    resize/                        queue
      pending/    9f2c1a0b7d4e6f83.json   waiting to be claimed
      processing/ 3a71bd0c5e2f9481.json   leased to a worker
      done/       18c4e05a9f2b7d63.json   finished or failed (terminal)

The directory a task's file is in is its status.

A task's id is 16 hex characters (8 random bytes). The filename is <id>.json, and which of the three directories it is in is its status: pending, processing, or done (where done holds both finished and failed tasks).

The item object

The full record on disc. The status view a producer reads is a subset of these fields.

{
  "id": "9f2c1a0b7d4e6f83",
  "host": "task.example.com",
  "queue": "resize",
  "status": "processing",
  "item": { "url": "https://ex.com/a.png", "format": "jpg", "width": 800 },
  "postedBy": "alice",
  "postedAt": 1751500000000,
  "updatedAt": 1751500012000,
  "attempts": 1,
  "lease": { "worker": "box-1", "until": 1751500072000 },
  "progress": { "percent": 40 },
  "result": null,
  "error": null,
  "finishedAt": null,
  "fetchedAt": null
}

One task on disc, mid-flight.

Field	Meaning
`id`, `host`, `queue`	Identity: the task id and the queue it belongs to.
`status`	`pending` / `processing` / `finished` / `failed`. Mirrors the directory (`finished` and `failed` both live in `done`).
`item`	The validated task the producer posted.
`postedBy`	The name of the user who posted it, or `null` when posted publicly.
`postedAt`, `updatedAt`, `finishedAt`	Millisecond timestamps. `finishedAt` is set when the task goes terminal.
`attempts`	Number of times the task has been claimed; rises on each re-claim after a re-queue.
`lease`	`{ worker, until }` while processing; `null` otherwise.
`progress`	The last progress record, or `null`.
`result`	The worker's result once finished, or `null`.
`error`	`"timeout"` on a queue timeout; otherwise `null` (application errors travel inside `result`).
`fetchedAt`	When a producer first read the terminal result; drives retention under `retainFrom: "fetched"`. `null` until then.

The claim is an atomic rename

Claiming a task renames its file from pending/<id>.json to processing/<id>.json. A rename is atomic and the source vanishes, so if two workers race for the same task exactly one rename succeeds; the other gets ENOENT and moves on to the next candidate. That single primitive is the entire lock: no lock files, no coordination service. Candidates are read and ordered oldest-first (FIFO by postedAt) before the rename, so the oldest matching task is the one claimed.

Every content write is temp-file-then-rename, so a reader never sees a half-written task. Moves between states are content-first (write the destination, then unlink the source), so a crash mid-move at worst leaves the task briefly in two directories; the reader prefers pending then processing then done, and the sweep reconciles. All filesystem races are treated as “someone else changed it” and skipped, which is correct for this single-box, file-per-task design.

The sweep

A periodic sweep runs across every live queue and applies the three time-based transitions using each queue's settings:

Pass	Looks at	Action
Queue timeout	`pending` + `processing`	Task older than `queueSeconds` → move to `done` as `failed` with `error: "timeout"`.
Lease timeout	`processing`	Lapsed `lease.until` → move back to `pending` (re-queue).
Retention	`done`	Past `retainSeconds` from the `finished` (or `fetched`) clock → delete the file.

The queue-timeout pass runs before the lease pass, so a task that has outlived the whole queue fails rather than being pointlessly re-queued. Each status change the sweep makes also notifies any monitor subscribers, and appends an event to the history below.

Because state lives entirely in the filesystem, a server restart loses nothing: pending tasks are still pending, processing tasks resume their leases (and time out normally if their worker is gone), and done tasks keep their results until retention clears them.

The durable task-event log

The item store is a working queue, not an archive: every task is deleted a short while after it finishes (retainSeconds), and an unclaimed task is timed out and deleted too, so nothing accumulates. To keep a memory of what the queue actually did, the server records each task's lifecycle to a separate, durable log — out of band of that retention. It is the source the Task Logs browser mirrors down and keeps.

On every transition the store appends one JSON line to a dated day-file under /srv/taskhoster-logs/tasks/. Dated files are the rotation; files past a few weeks are pruned on the server, but the local browser keeps every line it ever pulled.

{
  "t": 1751500012,
  "at": "2026-07-26T09:06:52.004Z",
  "host": "task.example.com",
  "queue": "resize",
  "id": "9f2c1a0b7d4e6f83",
  "event": "result",
  "status": "finished",
  "attempts": 1,
  "worker": "box-1",
  "bytes": { "item": 74, "progress": 14, "result": 2048, "error": 0 },
  "archived": ["result"]
}

One event line; a task leaves a trail of these from posted to cleaned.

Field	Meaning
`event`	The transition: `posted`, `claimed`, `progress`, `result`, `failed`, `timeout`, `requeue`, `fetched`, `cleaned`.
`t`, `at`	Unix seconds (the day-file bucket) and the ISO time.
`status`, `attempts`, `worker`	The task's status after the transition, its attempt count, and the worker that held it (when there is one).
`bytes`	The size in bytes of each payload (item, progress, result, error) as known at that moment. Always recorded, whether or not the bodies are archived.
`archived`	Which payload bodies this event wrote to the archive (only when the queue enables it).

Reading the log is out of band. The fetched and cleaned events are recorded by the store as they happen (when a producer first reads a result, and when retention deletes the file); the Task Logs tool only reads the log, so browsing your history never stamps a task's fetchedAt and never shortens retention.

taskhoster-archivePayloads

By default the history records only payload sizes. Set taskhoster-archivePayloads on a queue and the server also keeps the actual bodies — the posted item, each progress, the result or error — as files, so Task Logs can show them and keep them as an archive. They are stored separately from the queue's own item storage, which still self-cleans on the retention schedule, so archiving never changes how long a task occupies the queue.

/srv/taskhoster-logs/tasks-payloads/
  task.example.com/resize/
    9f2c1a0b7d4e6f83/
      item.json       the posted task body
      result.json     the worker's result body

One folder per task, a file per archived payload kind.

Task Logs links to each archived payload so it opens in a new browser tab, never embedded inline in a list (a payload can be large). The archive ages out on the same rolling window as the event log.