Observe services and host

Monitoring starts with the failures users care about, then connects them to host and service evidence. Nobody in your house cares about CPU load; they care that file sync stopped. So the first check is always the user-visible one, and everything else exists to explain it.

The local checklist

Run a compact local check:

systemctl --failed
docker compose ps
df -h
free -h
ip route

Each line answers one question. systemctl --failed — did any host service die? You want 0 loaded units listed. docker compose ps (run in your service directory) — are the containers Up, and not restart-looping? df -h — is any filesystem creeping past 80%? free -h — is memory exhausted, with swap climbing? ip route — does the host still have its default route, or did the network config quietly break?

When something looks wrong, logs are the next layer down:

journalctl -p err --since -1h
docker compose logs --since 1h web

The first shows host-level errors from the last hour; the second, the service’s own output.

Check from the outside

Every check so far runs on the server, and a down server can’t report itself. Add one external check from another authorized device:

# from your laptop:
curl -I --max-time 5 https://homeserver.lab.test
# HTTP/2 200

That single request exercises DNS, the network path, the proxy, TLS, and the container. It’s the closest thing to what a real user experiences. Run it from your laptop when you think of it, or schedule it on any always-on device you own — even a cron job on another machine that pings you when the check fails.

Record thresholds and where alerts go when you are away. “Disk over 80% → email me” written in the runbook beats a perfect dashboard nobody looks at.

The boundary

Do not expose monitoring dashboards without authentication. They reveal names, versions, and internal state — a map of your setup, drawn for whoever finds it. Keep dashboards on the tailnet or behind the LAN-only proxy, never port-forwarded. And prefer the boring failure mode: if the checklist takes more than a minute to run, you’ll stop running it. Five commands you actually type beat thirty you don’t.