Observe the connection lifecycle

Request counts from your HTTP router will not explain a live outage. You need metrics on connection opens, closes, close codes, replay gaps, queue depth, and how old the newest delivered event is.

Without those signals, “the board feels stuck” becomes a multi-hour guess.

Metrics worth exporting

live_connections (gauge)
connection_open_total (counter)
connection_close_total{code} (counter)
last_event_age_seconds (gauge)
replay_gap_total (counter)
slow_consumer_disconnect_total (counter)

Keep labels low cardinality. Track code, not user id, in Prometheus-style counters.

Structured logs for traces

logger.info('connection closed', {
  connectionId,
  code: ws.closeCode,
  lastEventId: ws.context.lastEventId,
  bufferedAmount: ws.bufferedAmount,
})

When an operator reports a stale board, search by connectionId and compare lastEventId to the server’s latest retained id. That tells you whether the gap is server silence or client reconnect failure.

Alerts that match user pain

Alert when last_event_age_seconds exceeds your latency budget for five minutes. Pair it with replay_gap_total so on-call knows whether to restart fanout or tell clients to resync.

Create one dashboard panel per state: connecting, live, reconnecting, stale. Firefox and Chrome also expose WebSocket frames in the Network tab when you need a single-session trace.

Redacted samples beat payload dumps

Logging every incident body gets expensive and risky. Sample one redacted frame per minute for debugging:

logger.debug('sample frame', {
  type: msg.type,
  id: msg.id,
  incidentId: msg.payload?.incidentId,
})

Keep full payloads behind a feature flag for local dev only.

When replay_gap_total rises but live_connections stays flat, the server dropped history while clients still think they are caught up. That pattern points to retention policy, not network blips.

Export the same metrics from a local docker compose stack before you wire production Grafana. If you cannot reproduce the graphs locally, on-call will not trust them at 3 a.m.

Try this on your own project: add a stale-delivery alert, force a disconnect, and use metrics to separate “server stopped sending” from “client failed to reconnect.”