Monitoring System Design Interview: Metrics, Alerting, and the 45-Minute Walkthrough

At some point, every on-call engineer owes their sanity to a latency graph that spiked before the users noticed. Monitoring is the infrastructure behind reliability. The monitoring system design interview tests how well you think about write-heavy workloads, time-series storage, and the tension between data resolution and cost.

If you've ever been woken up by an alert sound, this one's for you.

A meme showing a person in a coffin at their own funeral. After a PagerDuty alert fires, they burst out of the coffin.

r/ProgrammerHumor: "thatSoundCanRaiseTheDead", the most accurate depiction of on-call life ever committed to image format

Start With the Scope, Not the Architecture

Most candidates jump straight to boxes and arrows. The interviewer wants to see you ask the right questions first.

The four questions that drive every subsequent decision:

What kinds of metrics? Infrastructure (CPU, memory, disk), application (request rate, error rate, p99 latency), or business metrics (orders per second, revenue)?
Who queries? Internal engineers watching dashboards and responding to alerts, not external customers.
What is the ingestion scale? How many hosts, how many metrics per host, at what scrape interval?
How long do we keep data? A week of raw data at 10-second resolution is very different from two years of hourly rollups.

For this walkthrough, assume: 1,000 hosts, each emitting 100 unique metrics every 10 seconds, with one year of retention and sub-second dashboard queries. That puts ingestion at roughly 10,000 data points per second at baseline. Not Datadog-scale (they handle trillions of data points per day), but large enough to force real architectural decisions.

Non-functional requirements worth naming: data should be queryable with at most a few seconds of lag, the system should tolerate a collector node going down without data loss, and alert delivery should happen within 30 seconds of a threshold breach.

Draw Five Boxes Before Anything Else

Before any detail, draw these five boxes. Everything else is a refinement.

Five-box monitoring architecture showing Data Sources, Collectors, Kafka message queue, TSDB, Query Layer with alerting pipeline branching off

The full pipeline. Kafka is the unsung hero here. Without it, a storage hiccup cascades straight into data loss.

Kafka sits between collectors and the TSDB so a storage hiccup does not cascade into data loss. Data sources are your servers, services, and containers. Collectors scrape or receive metrics. The TSDB is the core. The query layer and alerting engine sit on top.

The Data Model Is the Most Consequential Decision

A metric has four fields, not one number. Each metric is uniquely identified by its name and a set of key-value labels.

http_requests_total{method="POST", status="200", region="us-east-1"} 42831 1717000000000

Name, labels (a dict), timestamp (milliseconds), float64 value. This is the Prometheus exposition format, and it became the industry default because it is simple and composable. The label set is how you slice and dice: filter by region, aggregate by status code, group by service.

Metric data model diagram showing the four fields: name in amber, labels in blue, value in yellow, and timestamp in purple, with a cardinality trap warning box below

The four fields. The cardinality trap at the bottom is the part candidates forget to mention.

The trap is cardinality. Cardinality is the total number of unique label combinations across all metrics. Three labels with 10 methods, 50 status codes, and 20 regions gives you 10,000 unique time series for a single metric. That is fine. Put a user ID or a trace ID in a label and your cardinality becomes unbounded. Every unique series gets its own chunk, its own index entry, and its own memory allocation. InfluxDB's own docs flag high series cardinality as the primary driver of memory pressure, and operators commonly report read and write latency degrading once series counts climb into the millions. VictoriaMetrics handles this better, but no system handles it gracefully at infinite cardinality.

The rule: labels must be low-cardinality. High-cardinality dimensions belong in logs or traces, not metrics.

Why a TSDB, Not PostgreSQL

Metrics are pathologically write-heavy. 10,000 inserts per second is routine. At Datadog scale, the indexing service alone handles trillions of events per day, with a single ingestion host taking hundreds of thousands per second. A conventional relational database uses a B-tree index that rebalances on every insert. At metric write rates, that index thrashes the disk. It's like trying to keep a library in alphabetical order while someone dumps a new box of books on you every ten milliseconds.

Time-series databases use a Log-Structured Merge tree (LSM tree), which converts random writes into sequential writes. Incoming data goes first into an in-memory buffer, then flushes to immutable sorted files on disk. Background compaction merges those files. Sequential writes are an order of magnitude faster than random writes on spinning disk, and competitive on SSDs.

LSM tree write path diagram showing incoming writes flowing through WAL and Head Block into compacted immutable blocks, with tiering to object storage

The write path. Recent data never touches disk. Old blocks get tiered to S3. The WAL is your crash-recovery safety net.

Prometheus organizes storage into a Head block (recent ~2 hours in memory, backed by a Write-Ahead Log for crash recovery) and compacted immutable blocks on disk. Each block covers a fixed time range. Queries that touch only the most recent data never touch disk at all.

Beyond the write structure, TSDBs compress aggressively. The Facebook Gorilla paper (VLDB 2015) showed two key tricks:

Delta-of-delta for timestamps. If your scrape interval is 10 seconds, consecutive timestamps differ by 10,000ms. The delta is constant. The delta-of-delta is zero. Zero encodes to a single bit. About 96% of all timestamps compress to one bit.
XOR for float values. Adjacent metric values are usually similar. XOR two similar floats and the result has many leading and trailing zero bits. You only store the meaningful bits in the middle.

Gorilla compression diagram showing delta-of-delta timestamp encoding (DoD = 0, one bit) and XOR float encoding with leading/trailing zeros stripped

Gorilla achieves roughly 1.37 bytes per sample on average, down from 16 bytes (8-byte float + 8-byte timestamp). That is a 12x reduction.

Prometheus implements Gorilla encoding in its chunk layer.

Pull vs Push: There Is a Real Tradeoff

Prometheus pulls. Every monitored target exposes a /metrics HTTP endpoint. The Prometheus server scrapes it on a configured interval. Datadog pushes. An agent on every host collects metrics and forwards them to Datadog's intake.

Pull's underrated property is failure-mode clarity. When Prometheus cannot scrape a target, up == 0 is a metric. The monitoring system knows the target is unreachable. With push, if an agent stops sending, you get silence. Silence and "everything is fine" look identical without extra machinery. This is not a subtle distinction. Silence is the most dangerous failure mode in monitoring.

Push wins when targets are ephemeral. Short-lived batch jobs and serverless functions finish before the monitoring server scrapes them. They have to push.

Real monitoring stacks are almost always hybrid. Pull-based inside production clusters, push-based for edge and ephemeral workloads. Accept both.

See The Trade-off Maze for a deeper look at push vs pull in distributed systems.

Retention Tiers: Store Less, Keep More

A year of raw data at 10-second resolution for 1,000 hosts is enormous. 10,000 points/sec for one year is about 315 billion data points. Even at 1.37 bytes each, that is 430 GB. And most of it goes unread.

Downsampling solves this by replacing raw data with aggregated rollups as data ages.

A typical three-tier policy:

Tier	Resolution	Retention
Raw	10 seconds	7 days
Medium	1 minute	30 days
Long	1 hour	1 year

Retention tiers diagram showing raw tier at 430GB, medium rollup at 70GB, long rollup at 1.2GB, with arrows showing rollup computation and tiering to S3

A background job reads the raw tier, computes min/max/average/p99 per window, writes to the lower-resolution tier, and deletes the raw data. 90% storage reduction. The right tradeoff for most incident patterns.

A background job runs on a schedule, reads the raw tier, computes min/max/average/p99 per window, and writes them to the lower-resolution tier. The raw data is then deleted. What you lose: the ability to investigate a 15-second spike that happened three weeks ago. What you gain: a 90% reduction in storage. This is the right tradeoff for most incident patterns (recent incidents get raw data, trend analysis uses rollups).

InfluxDB has built-in downsampling via continuous queries. Prometheus delegates it to recording rules and external tools like Thanos Compact or Cortex.

The Alerting Pipeline Has Three Stages

Alerting is not "send an email when CPU > 90%." It has to handle de-duplication, grouping, silencing, and routing. Conflating rule evaluation, alert management, and notification routing into one description is the most common design gap in this interview.

Three-stage alerting pipeline: Stage 1 rule evaluation with 'for' duration, Stage 2 Alertmanager with grouping/de-duplication/inhibition, Stage 3 notification routing to PagerDuty/Slack

Stage 1: Rule evaluation. The Alert Engine queries the TSDB on a fixed interval (typically 15 to 60 seconds). Each rule is a query with a threshold. If the condition holds for longer than a configured for duration (to avoid noise from single-point spikes), the rule fires an alert.

Stage 2: Alert management. The Alertmanager (or equivalent) receives firing alerts and applies grouping (bundle 50 "host down" alerts from the same cluster into one notification), de-duplication (suppress repeated notifications for the same firing alert), and inhibition (suppress "disk full" alerts for a host that is already known to be down).

Stage 3: Notification routing. Route by severity and team. Sev-1 goes to PagerDuty and wakes someone up. Sev-3 goes to a Slack channel and waits for business hours. Routing rules are defined in config, not code.

The de-duplication step alone is what separates a usable alerting system from one that drives engineers into burnout. Without it, a single bad deploy can fire five hundred notifications before anyone has time to look at the dashboard.

Where the System Breaks (and What to Do)

The write bottleneck. A single TSDB node can handle roughly 500K to several million data points per second depending on the system. VictoriaMetrics scaled from 800K samples/sec on a 2-vCPU box to 19M samples/sec on a 64-vCPU box in its own vertical-scalability benchmark, and the ceiling moves with cardinality and cores. For most companies, a single node is enough. When it is not, partition by metric name hash (each shard owns a subset of metrics) or use a purpose-built clustered TSDB like Cortex, Thanos, or VictoriaMetrics Cluster.

The query bottleneck. Time-range queries over millions of series are expensive. Two mitigations: recording rules (pre-compute expensive aggregations and store results as new time series) and a caching layer (Redis or Memcached) in front of the query engine for repeated dashboard queries. Grafana refreshes dashboards every 30 seconds; the underlying query is almost always the same.

The storage bottleneck. Long-term storage gets expensive. The standard pattern is to tier storage to object storage (S3 or GCS) for data older than a few days. Thanos does this transparently: recent data stays on local SSD, older compacted blocks upload to S3. Queries that need old data read it from S3. Cold queries are slower but acceptable for historical analysis.

The cardinality bottleneck. Worth repeating because it kills clusters. An inverted index maps each label-value pair to the set of series that match it. At 10 million unique series, this index alone can consume tens of GB of RAM. The fix is upstream: reject or normalize high-cardinality labels at the collector before they reach the TSDB. Bloom filters at the collector can catch label keys on a deny-list at ingestion time. See Bloom Filter for how the probabilistic check works.

What the Interviewer Is Actually Watching

Raise cardinality unprompted. Candidates who bring up the label cardinality problem before being asked demonstrate real operational experience. It signals you've seen a TSDB fall over in production, or at least read enough to know it happens.

Justify the TSDB, not just name it. "Write patterns destroy B-tree indexes" is worth more than "I'd use InfluxDB." Show the reasoning.

Separate alerting into three stages. Bundling rule evaluation, de-duplication, and notification routing into one description is the most common oversight. Interviewers who've been on call know exactly why this matters. They've seen the ungrouped 500-alert flood. They are not impressed when you skip it.

How to Run a Monitoring System Design Interview in 45 Minutes

Block	Time	Topic
Clarify	0-5 min	Metric types, scale, retention, alerting requirements
High-level	5-15 min	Five-box architecture, data model, pull vs push choice
Storage deep-dive	15-25 min	TSDB vs SQL, LSM tree, Gorilla compression, WAL + blocks
Alerting deep-dive	25-33 min	Rule evaluation, Alertmanager, grouping + de-duplication
Scale + tradeoffs	33-43 min	Sharding, rollups, cardinality mitigation, object storage tiering
Wrap-up	43-45 min	Summarize key decisions, open tradeoffs

Six Decisions That Define the Design

Data model: name + labels + timestamp + float64. Labels must be low-cardinality.
Storage: LSM-based TSDB with Gorilla compression. WAL for durability, compacted blocks for query efficiency.
Collection: hybrid pull (scraping) and push (agent intake), with Kafka as a durability buffer between collectors and the TSDB.
Retention: three tiers (raw, 1-minute rollup, 1-hour rollup), offloading old compacted blocks to object storage.
Alerting: rule evaluation engine → Alertmanager (grouping + de-duplication + inhibition) → notification routing.
Scaling: single-node TSDB is usually enough; partition by metric name when it is not; recording rules and Redis caching for the read path.

If you want to practice explaining this live, with follow-up questions on the cardinality problem or what happens when the Kafka consumer falls behind, SpaceComplexity runs voice-based system design mock interviews with rubric-based feedback on exactly this kind of problem.