Use dead letters and controlled replay

Some messages will never succeed, no matter how many retries you give them. A payload from a buggy deploy, a reference to a deleted record, an unexpected schema version. A poison message can consume retries forever or block useful work, so the queue needs somewhere to put it.

That somewhere is a dead-letter queue: a normal queue that collects messages after the configured delivery attempts are exhausted:

{
  "queues": {
    "consumers": [
      {
        "queue": "order-jobs",
        "max_retries": 5,
        "dead_letter_queue": "order-jobs-dlq"
      }
    ]
  }
}

After the fifth failed attempt, the message moves to order-jobs-dlq instead of being dropped. Without this setting, exhausted messages are deleted — the failure and its evidence vanish together.

A dead letter is evidence, so treat it that way. Alert on dead-letter growth and include safe failure context in your logs: which handler failed, which error class, which jobId. Check the backlog directly:

npx wrangler queues info order-jobs-dlq
# ...
# backlog size: 3

A backlog that was zero yesterday and is three hundred today is telling you a deploy broke something, hours before a customer does. And since payloads sit in the DLQ for days, sensitive payloads need the same retention and access controls as source data — the dead-letter queue is not exempt from your data rules.

Replay is a deliberate act

The wrong response to a full DLQ is pointing the consumer back at it and hoping. The messages failed for a reason; unfixed, they’ll fail again. Do not blindly replay everything: fix code or data first, select the messages that the fix actually addresses, preserve their operation IDs so idempotency checks still recognize them, and watch the result.

A small replay consumer on the DLQ does this well:

async queue(batch, env) {
  for (const message of batch.messages) {
    await env.ORDER_JOBS.send(message.body)  // original body, original IDs
    message.ack()
  }
}

Because the original jobId travels with the replayed message, work that partially completed before dying won’t run twice.

Rehearse the loop before you need it: force a schema-version failure, inspect its dead letter, deploy compatible handling, and replay that single message safely. One rehearsed message teaches you the whole procedure — selection, replay, verification — at a moment when nothing is on fire.