Balance and check upstreams

One backend is a single point of failure. Run two, and Caddy can spread traffic between them and route around the one that dies.

List multiple upstreams and add health checking:

:8080 {
  reverse_proxy 127.0.0.1:4001 127.0.0.1:4002 {
    lb_policy round_robin
    health_uri /health
    lb_try_duration 3s
  }
}

Three subdirectives, three jobs. lb_policy round_robin alternates requests between the upstreams instead of the default random pick. health_uri enables active health checks: Caddy requests /health on each upstream on a regular interval, and an upstream that fails the check is marked unhealthy and removed from rotation. lb_try_duration 3s gives each incoming request a three-second budget to find a working upstream before Caddy gives up and returns an error.

Start two throwaway backends to test with:

caddy respond --listen 127.0.0.1:4001 "backend one" &
caddy respond --listen 127.0.0.1:4002 "backend two" &

Send a few requests and watch them alternate:

for i in 1 2 3 4; do curl -s http://127.0.0.1:8080/; echo; done

Now run the failure drill. Kill backend one and keep sending requests. At first, requests that land on the dead upstream get retried onto the survivor within the try duration — clients see slow responses, not errors. Once the active health check fails, the dead upstream leaves rotation entirely and responses are fast again. Restart it and watch it return to rotation after its health check passes. That full cycle — degrade, eject, recover — is what you’re buying with three lines of config.

Retries carry a trap worth understanding. Retrying a failed GET is safe. Retrying a POST that charges a credit card is not: the upstream may have processed the request before dying, and a retry runs it twice. Only allow retries of state-changing requests when the application handles them idempotently. A missing response never proves the upstream did nothing.

One failure mode to recognize: if the /health endpoint itself breaks — say a deploy removed it — Caddy marks every upstream down and serves 502s while the apps are actually fine. When the whole fleet goes unhealthy at the same moment, suspect the check before the backends.