Monitor, update, rotate, and recover

A tunnel that works today is not done. Somebody has to watch it, update it, and know what to do when it breaks. That somebody is you, so let’s make the job small.

What to watch

Monitor five things: connector health, connection count, origin errors, latency, and route changes. The dashboard shows connector status and the number of active connections per replica. Your own logs show whether the origin answered. A connector that is HEALTHY with four connections but returning 502s means the tunnel is fine and your app is not. Knowing which side broke is half the incident.

Route changes deserve an alert of their own. A new public hostname or a widened CIDR is exactly what an attacker with a stolen token would add.

Keep it updated, on your schedule

cloudflared gets regular releases. Update it through the same process you use for any other package on the host, not by hand at 11 PM:

cloudflared --version
sudo apt update && sudo apt install --only-upgrade cloudflared

Update one replica, confirm it reconnects, then update the next. That’s the payoff of the previous lesson: you can update without an outage.

Inventory and rotation

Write down every hostname, CIDR, connector host, token owner, policy, and fallback path. When a token leaks, this list is how you know which hosts to touch.

Rotating a compromised token means revoking it in the dashboard and installing the new credential on the replicas you trust. Any connector still running the old token drops off. If a connector you don’t recognize was running, that is your answer about the leak.

Test what users see

Stop the connector and load the site. Stop the origin but leave the connector running and load it again. The two failures look different, and you should know both screens before a user describes them to you.

Then write the runbook. Five scenarios: lost connector host, leaked token, bad route, expired origin certificate, and rollback of a config change. For each one, how you detect it, the first command you run, and who decides. One page each.

Prove that a second connector can serve the same route before maintenance:

cloudflared tunnel info practice-app
systemctl status cloudflared
journalctl -u cloudflared --since '15 minutes ago'

Stop one connector and request the application through the public hostname or private route. Rotate credentials in a planned window, then remove the retired connector. Keep an origin-local recovery path so a Cloudflare or identity failure does not lock operators out.