Test, observe, and recover async work

Async happy paths are easy; failure ownership is the real product. A background system that works when everything works has delivered roughly nothing — the entire point was surviving the days when something doesn’t.

So the test list is a list of failures. Test duplicate messages, partial steps, old payloads, dependency 429s, dead letters, missing events, cancellation, and deployment overlap:

duplicate delivery     -> durable result created once
step 3 of 5 fails      -> steps 1-2 not re-executed, job recovers
version 1 payload      -> new consumer still handles it
provider returns 429   -> backoff, no dead-letter flood
event never arrives    -> timeout path runs, someone is notified
cancel mid-run         -> partial side effects accounted for

Each line is one cheap test in a practice environment, and each one fails loudly in production if you skip it.

Observability is part of the contract

When a job misbehaves, you need to see it from both directions. From the platform side:

npx wrangler queues info order-jobs        # backlog, consumer status
npx wrangler workflows instances describe order-workflow job_01J2AB3CD4
# status: errored
# step "reserve inventory": failed after 3 attempts

From the application side, expose job status without leaking private payloads. A status table keyed by job ID — state, timestamps, error class, attempt count — answers “where is my export” without reproducing the customer data inside it. Support reads it, dashboards alert on it, and nobody greps raw payloads.

Correlate request, message, Workflow instance, and external calls with the one stable job ID from the previous lesson. When the trace ID in a log line leads to the queue message, the workflow instance, and the provider’s request log, an incident takes minutes instead of an afternoon.

Decide recovery before the incident

Define who may retry, cancel, or repair a job and what those actions mean after partial success. “Retry the order job” is ambiguous when the charge succeeded and the inventory step failed — does retry mean resume from the failed step, or re-run everything? Write the answer down while calm; the on-call person at 2 AM shouldn’t be inventing policy.

Then rehearse it. Run a tabletop exercise where the email provider fails for an hour while order processing must continue. Walk through what queues back up, what alerts fire, who gets paged, and what gets replayed when the provider recovers. The gaps you find on paper are the ones production would have found for you, with interest.