Schema and MergeTree

Partition and model deliberately

Use coarse partitions for lifecycle operations, denormalize common dimensions, and avoid creating thousands of small partitions or runtime joins.

9 minute lesson

~~~

Partitions group parts for operations such as dropping old data. They are not a replacement for the ordering key.

This distinction trips up almost everyone coming from other databases. The ordering key is what makes queries fast. A partition is a lifecycle unit: a chunk of the table you can drop, move, or detach as one cheap operation.

Monthly partitions are common for time data:

CREATE TABLE events (
  ts      DateTime,
  service LowCardinality(String),
  user_id UInt64
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(ts)
ORDER BY (service, ts);

Now a retention policy is one statement:

ALTER TABLE events DROP PARTITION '202507';

Dropping a partition deletes its parts instantly, with none of the cost of deleting rows one by one.

The too-many-partitions failure

High-cardinality partition keys create too many parts. Partition by day and keep three years of data: 1,095 partitions. Partition by user_id: potentially millions. Every partition holds its own parts that can never merge across partition boundaries, so part counts explode, inserts touching many partitions slow down, and eventually inserts fail with a Too many parts error.

The symptom shows up in system.parts:

SELECT partition, count() AS parts
FROM system.parts
WHERE table = 'events' AND active
GROUP BY partition
ORDER BY parts DESC;

Hundreds of partitions with a handful of tiny parts each means the partition key is too fine. My advice: partition by month, or don’t partition at all. You need a reason — usually retention — to partition, not a reason to skip it.

Denormalize the hot path

Analytical tables often denormalize dimensions used in every query. In a normalized OLTP schema you’d store service_id and join to a services table. Here, if every dashboard query filters or groups by service name, copy the name into the events table.

Dictionaries or joins can still help — a dimension that changes often, or one used rarely, can stay external. But copying stable labels into events may make the hot path simpler: no join to plan, no second table to keep available, and LowCardinality(String) makes the repeated values nearly free to store.

Try it

Choose a partition, ordering key, and retention operation for one year of application events. Estimate the number of partitions before creating the table.

Monthly gives you 12; daily gives you 365. Say the retention rule out loud — “each month, drop the partition from 13 months ago” — and check the operation matches the partition unit. If your retention is expressed in days but your partitions are months, one of the two needs to change.

Lesson completed

Take this course offline

Get every free book and course as PDF and EPUB files.

Get the download library →