Add health checks

Two lessons ago, killing one backend made half your requests fail. The balancer kept selecting a dead upstream because nothing told it to stop. Health checks close that gap: the proxy probes each upstream and removes failing ones from selection.

A health check should answer whether the instance can safely receive new work. That’s a narrower question than “is the process running”. A backend can be alive but useless — database connection lost, disk full, still warming up. It should be cheap and distinguish process life from readiness.

Give the backend a health endpoint

Add a /health route to the lab backend that you can flip on demand:

let healthy = true

// inside the request handler:
if (request.url === '/health') {
  response.statusCode = healthy ? 200 : 503
  return response.end()
}
if (request.url === '/break') {
  healthy = false
  return response.end('broken\n')
}

The /break route lets you simulate the interesting failure: a process that’s alive but not ready.

Add an active check

reverse_proxy 127.0.0.1:4001 127.0.0.1:4002 {
  health_uri /health
  health_interval 5s
  health_timeout 2s
}

health_uri is the path Caddy probes on each upstream. health_interval 5s sets the probe frequency, and health_timeout 2s counts a slow answer as a failure — an upstream too overloaded to answer its health check in two seconds shouldn’t get new work either.

Watch removal and recovery

Start the counting loop from the earlier lessons, then make one backend’s health endpoint fail while its process remains alive:

curl http://127.0.0.1:4002/break

Within one probe interval, port 4002 disappears from the response counts. No failed client requests this time — Caddy checked before selecting. Confirm Caddy stops selecting it and later restores it: restart the backend (fresh state, healthy is true again) and 4002 rejoins within a probe cycle. Compare this with the raw failure rate you recorded before health checks existed.

Design the contract

The endpoint’s meaning deserves a decision, not an accident. Do not make health checks perform expensive writes or depend on every remote service. A check that queries five dependencies turns any dependency blip into full fleet removal — every upstream fails the check at once, and the balancer has nowhere left to send traffic. A check that does real writes generates load precisely when the system is weakest.

Define the readiness contract intentionally: verify the few things this instance needs to serve correctly, keep it read-only, keep it fast.