Back up, monitor, and restore

Replicas are not backups because a bad delete or mutation can propagate. Replication faithfully copies your mistake to every server within seconds — a dropped partition, a wrong ALTER TABLE ... DELETE, a truncate on the wrong environment. Only a backup holds the state from before the mistake.

Back up the tables, metadata, users, and configuration required by the recovery goal. ClickHouse has native commands for the data and schema:

BACKUP TABLE analytics.events
TO Disk('backups', 'events-2026-08-03.zip');

The backups destination is a disk you configure in the server’s storage settings; S3-compatible object storage is the other common target. The backup contains both schema and data, and later backups to the same destination can be incremental. Don’t forget the parts a table backup misses: access entities (users, roles, grants) and server configuration files need their own copy, or your restored cluster has data nobody can query.

Keep the raw event source long enough to rebuild when that is part of the design — a pipeline whose queue or object store retains 30 days of events is itself a recovery path for recent data.

Monitor the things that fail quietly

Monitor query failures, ingestion lag, disk, parts, merges, replication queues, memory, and backup jobs. Most of these come straight from system tables — failed queries from system.query_log, part counts from system.parts, replication debt from system.replication_queue, and every backup’s outcome:

SELECT name, status, error
FROM system.backups
ORDER BY start_time DESC
LIMIT 5;

A backup job that has been failing for three weeks is indistinguishable from no backup at all, and it’s always discovered at the worst moment. Alert on status != 'BACKUP_CREATED', not just on the job running.

A restore you haven’t run is a hypothesis

Restore a backup into an isolated database or cluster — never onto the production table first:

RESTORE TABLE analytics.events AS analytics.events_restored
FROM Disk('backups', 'events-2026-08-03.zip');

Then compare row counts, one aggregate, permissions, and the dashboard result before accepting the recovery:

SELECT count() FROM analytics.events_restored;
SELECT service, count() FROM analytics.events_restored
GROUP BY service ORDER BY service;

The counts tell you the data arrived; the aggregate tells you it’s the right data; a login with the dashboard user tells you access survived; and loading one real dashboard against the restored copy tells you the whole chain works.

Run this drill on a schedule, not during an incident. The point of rehearsal is finding the missing grant or the misconfigured backup disk on a calm Tuesday, when it’s a ticket instead of an outage.