Bound retries and side effects

Retries look like free reliability. An upstream fails, the proxy quietly tries another one, the client never notices. Sometimes that’s exactly what happens. Sometimes the retry charges a customer twice.

The line between those outcomes is where the failure happened. A proxy may safely retry a failed connection before sending a request — nothing reached the backend, so nothing happened. But retrying after an upstream received a state-changing request can duplicate work. If a POST creates an order and the connection drops before the response arrives, the proxy can’t know whether the order exists. Retrying it might create a second one.

This is why HTTP method semantics matter here. GET, HEAD, PUT, and DELETE are defined as idempotent — repeating them shouldn’t change the outcome. POST is not.

Configure a bounded retry window

Configure a short selection window:

reverse_proxy 127.0.0.1:4001 127.0.0.1:4002 {
  lb_try_duration 2s
  lb_try_interval 250ms
  lb_retry_match GET
}

lb_try_duration 2s gives Caddy up to two seconds to find a working upstream for a request. lb_try_interval 250ms spaces the attempts. lb_retry_match GET restricts which requests may be retried after reaching an upstream: only GETs, the safest class.

Verify both sides of the boundary

Stop one backend and test GET recovery:

curl --silent http://127.0.0.1:8080/
# {"port":4001,"path":"/"}

Even when the balancer first picks the dead upstream, the request lands on the live one within the try window. No visible failure.

Now the other side. Send a POST while one upstream is down and watch some fail rather than retry — that’s lb_retry_match doing its job. For a POST, use an application idempotency key instead of assuming the proxy can infer safety: the client sends a unique key per logical operation, the backend stores it, and a duplicate arrival returns the original result instead of repeating the work. That moves retry safety into the layer that actually knows what the request does.

Respect the caller’s deadline

Keep retry budgets below the caller deadline. If your client gives up at 2 seconds and your lb_try_duration is also 2 seconds, the proxy’s heroics finish after the audience has left — the client saw a timeout anyway, and may retry on its own.

That stacking is the real danger. Layered retries can multiply load during an outage: client retries three times, proxy tries multiple upstreams per attempt, and a struggling backend receives several times its normal traffic at the worst possible moment. Budget retries at one layer, keep the window short, and let the layers above fail fast.