Run the final failover lab

Every lesson so far isolated one behavior. Real proxies run all of them at once, and the interesting failures happen at the seams. The final lab joins every boundary. Start two private backends, one HTTPS proxy, structured logs, health checks, and bounded retries.

Assemble the full configuration

This Caddyfile is the course in one file:

app.lab.test {
  tls internal
  log {
    output file access.json
    format json
  }
  reverse_proxy 127.0.0.1:4001 127.0.0.1:4002 {
    lb_policy round_robin
    health_uri /health
    health_interval 5s
    health_timeout 2s
    lb_try_duration 2s
    lb_retry_match GET
  }
}

Start both backends bound to loopback, validate, run. Confirm the baseline before injecting any fault: both ports appear in responses, the certificate is the local one, access.json grows.

Run the observation loop

Run a repeatable request loop:

for number in {1..20}; do
  curl --silent --show-error --fail https://app.lab.test/ || echo failed
  sleep 0.5
done

Keep it running in one terminal. It’s your client’s-eye view for the whole exercise — every fault you inject either shows up here or it doesn’t, and both results are findings.

Inject faults, one at a time

Work through the sequence: stop one backend, restore it, reload one routing change, and inspect distribution and failures.

Stop backend 4002 mid-loop. You may see a failed line or two before the next health probe, then clean responses from 4001 only. Restore it and watch 4002 rejoin the rotation within a probe interval. Then edit one route, caddy validate, caddy reload, and confirm the loop never misses.

After each fault, write the first failed boundary for every intentional fault. Which layer noticed first — the health check, the retry window, the client loop? If the answer surprises you, that’s the lab working. A fault the loop never saw is a resilience win worth understanding; a fault it saw for ten seconds reveals a probe interval worth tuning.

Finish with the privacy check from the trust module: the backend ports must be unreachable from outside while https://app.lab.test answers.

Clean up deliberately

Keep the whole lab local or on infrastructure you own. And when finished, remove what you installed: the /etc/hosts entries, and the local CA root — caddy untrust removes it from the trust store. Remove local CA trust and stop listeners when finished. A forgotten trusted root on a development machine is a real credential, and cleanup is part of operating a proxy, not an afterthought.