Queue producers and consumers

Send and consume messages

Bind a producer and consumer, enqueue only after authoritative state exists, and process each batch explicitly.

A queue has two sides. The producer is the Worker that sends messages. The consumer is the Worker that receives them. Both are bindings in wrangler.jsonc, and they can live in the same Worker or in two different ones:

{
  "queues": {
    "producers": [{ "binding": "EMAIL_QUEUE", "queue": "email-jobs" }],
    "consumers": [
      { "queue": "email-jobs", "max_retries": 3, "dead_letter_queue": "email-jobs-dlq" }
    ]
  }
}

Create the queue itself once with npx wrangler queues create email-jobs.

Send after you store

The producer validates the request, stores the authoritative record, and only then sends. If the message goes out first and the database write fails, the consumer works on something that doesn’t exist.

Send a small versioned message:

await env.EMAIL_QUEUE.send({
  version: 1,
  messageId,
  userId,
  template: 'welcome'
})

The consumer should validate the version and required fields before doing work. Keep secrets and full user records out of the message. Test a valid message, an unknown version, malformed data, and a provider failure before increasing batch size.

When the result will only be ready later, respond with 202 Accepted and the job ID. Don’t return 200 as if the email already went out.

Decide per message

The consumer exports a queue handler. It receives a batch, and each message carries its body plus ack() and retry():

export default {
  async queue(batch, env) {
    for (const message of batch.messages) {
      const { version, messageId, userId, template } = message.body

      if (version !== 1) {
        console.error('unknown message version', message.id)
        message.retry()   // lands in the dead-letter queue after max_retries
        continue
      }

      try {
        await sendEmail(env, userId, template)
        message.ack()
      } catch (e) {
        message.retry()
      }
    }
  },
}

The per-message decision is the point. If the handler returns without throwing, the whole batch counts as delivered. If it throws, every message you didn’t explicitly ack() comes back, including the emails that already went out. So be explicit: ack() means done, retry() means try again later.

Try this with three export jobs. Make the middle one fail on purpose. After the batch runs, the first and third should be acknowledged and only the second should come back on the retry path. If all three come back, you’re throwing out of the handler instead of calling retry().

Lesson completed