Lifecycle

The three states

An item's state is where its file lives on the server: one directory per state. A claim is an atomic move from pending to processing; finishing moves it to done. The producer-visible status is pending, processing, finished or failed; the last two both live in done and are terminal.

                post
                  │
                  ▼
             ┌──────────┐    claim     ┌────────────┐   result    ┌───────────────────┐
   ┌────────▶│ pending  │─────────────▶│ processing │────────────▶│ done · finished   │
   │         └──────────┘              └────────────┘             └───────────────────┘
   │              │                          │
   │              │                          │  lease lapses  /  worker socket drops
   │  re-queue    │                          │
   └──────────────┼──────────────────────────┘
                  │
                  │  age > queueSeconds (from either state)
                  ▼
          ┌───────────────────┐
          │ done · failed      │   error: "timeout"
          └───────────────────┘

  done items are deleted after retainSeconds  (from finished, or from first fetch)

Every transition

Leases and re-queue

A claim leases the item to the worker for leaseSeconds. If the worker finishes in time, it posts a result and the item is done. If it doesn't (the process is slow, or it crashed), one of two things re-queues the item:

• Lease timeout. A periodic sweep finds any processing item whose lease has lapsed and moves it back to pending, where another worker can claim it. This is the general safety net; it catches a worker that hung, lost power, or fell off the network.
• Socket drop. A worker holding items over the /work socket has them re-queued the instant the socket closes, no waiting for the lease. This is the fast path for the common case of a worker process exiting.

Each re-queue increments attempts the next time the item is claimed, so you can see how many times an item has been through a worker.

Late but useful results

Re-queuing raises an obvious worry: what if the "stalled" worker wasn't dead, just slow, and it hands in its result after the item was already given to someone else? Taskhoster keeps that work.

A result is accepted whenever the item is not already terminal. So the original worker can still finish an item that was re-queued; its late result is used, as long as no other worker finished it first and the queue didn't time it out. The first result to arrive wins; the loser is told accepted: false and its result is discarded. Nobody's effort is wasted when everyone is slow, and the item is never finished twice.

Queue timeout: the terminal backstop

Some tasks simply cannot be finished: no worker can handle them, or every worker keeps failing. Left alone they would be re-queued forever. queueSeconds bounds the whole life of an item: measured from when it was posted, once it is older than that it is moved to done with status: "failed" and error: "timeout", from either pending or processing.

After that the producer has been told the task timed out, so a late result is refused (accepted: false); the verdict is final. This is the one failure the server itself declares; application failures ride back in-band in the result instead.

Retention

A finished or failed item (and its result) is kept for retainSeconds, then deleted by the sweep. When that clock starts depends on retainFrom:

Under "fetched", an item whose result nobody has read yet is held indefinitely; the countdown only begins once you have seen it. Under "finished", the countdown begins at completion regardless.

All the timers at a glance

A single periodic sweep runs across every live queue applying all three: re-queue lapsed leases, fail aged items, delete retained ones. Configure the timings per queue in its settings.