Test, observe, and recover async work

The happy path of an async system is easy. Owning the failures is the real product. A background system that works when everything works has delivered almost nothing. The whole point was surviving the days when something breaks.

So your test list is a list of failures. Duplicate messages, partial steps, old payloads, dependency 429s, dead letters, missing events, cancellation, deployment overlap:

duplicate delivery     -> durable result created once
step 3 of 5 fails      -> steps 1-2 not re-executed, job recovers
version 1 payload      -> new consumer still handles it
provider returns 429   -> backoff, no dead-letter flood
event never arrives    -> timeout path runs, someone is notified
cancel mid-run         -> partial side effects accounted for

Each line is one cheap test in a practice environment. Skip it, and production runs the test for you, loudly.

Observability is part of the contract

When a job misbehaves, you need to see it from both directions. From the platform side:

npx wrangler queues info order-jobs        # backlog, consumer status
npx wrangler workflows instances describe order-workflow job_01J2AB3CD4
# status: errored
# step "reserve inventory": failed after 3 attempts

From the application side, expose job status without leaking private payloads. I like a status table keyed by job ID: state, timestamps, error class, attempt count. It answers “where is my export” without copying the customer data into it. Support reads it, dashboards alert on it, and nobody greps raw payloads.

Correlate the request, the message, the Workflow instance, and the external calls with the one stable job ID from the previous lesson. When a trace ID in a log line leads you to the queue message, the workflow instance, and the provider’s request log, an incident takes minutes instead of an afternoon.

Decide recovery before the incident

Define who may retry, cancel, or repair a job, and what those actions mean after partial success. “Retry the order job” is ambiguous when the charge succeeded and the inventory step failed. Does retry mean resume from the failed step, or re-run everything? Write the answer down while you are calm. The person on call at 2 AM should not have to invent policy.

Then rehearse it. Run a tabletop exercise: the email provider fails for an hour while order processing must continue. Walk through which queues back up, which alerts fire, who gets paged, and what gets replayed when the provider recovers. The gaps you find on paper are the ones production would have found for you, with interest.