Design for at-least-once delivery

Cloudflare Queues gives you at-least-once delivery. Every message will reach a consumer. Some messages will reach it twice.

This is not a bug. It’s the price of reliability. A consumer processes a message, and then the acknowledgement gets lost on the way back. The queue can’t tell the difference between “processed, ack lost” and “never processed”. So it delivers again. A rare duplicate after a normal success is part of the contract.

So the question is never “how do I stop duplicates?”. It’s “how do I make a duplicate harmless?”.

Give every operation a stable key

Each message needs an idempotency key: an identifier that stays the same across deliveries. Use the messageId your producer put in the payload, or the natural key of the operation, like order-8812-charge.

Then enforce uniqueness somewhere durable. D1, a Durable Object, or the external provider’s own idempotency support all work.

Use a stable message ID as the idempotency boundary:

insert into processed_messages (id, processed_at)
values (?, current_timestamp)
on conflict (id) do nothing;

Only perform the side effect when the insert claims the ID. Then deliberately retry the same message twice. A queue retry is normal delivery behavior, so duplicate handling belongs in the design rather than in an emergency cleanup script.

In a D1 consumer that looks like this:

const claim = await env.DB.prepare(
  `insert into processed_messages (id, processed_at)
   values (?, current_timestamp) on conflict (id) do nothing`
).bind(message.body.messageId).run()

if (claim.meta.changes === 0) {
  message.ack()   // already done by an earlier delivery
  continue
}

await sendWelcomeEmail(env, message.body.userId)
message.ack()

meta.changes is 1 when this delivery claimed the ID and 0 when a previous one did. The second delivery acks without sending anything.

Two ways to get it wrong

An in-memory Set of seen IDs feels like the same thing. It isn’t. It lives in one isolate, and the duplicate can land on a different one, or arrive after a deploy wiped it. Only durable storage counts.

Acknowledging before the side effect is the other trap. If you ack() and then the email call fails, the message is gone and the email never went out. Ack last.

Try this: deliver the same payment-like message twice and prove the durable result is created once. One row in processed_messages, one charge, two acks.