📡The Metrics Pipeline

The metrics pipeline

Follow a single number — say, the duration of one checkout request — from the line of code that records it to the page that wakes someone up. It passes through six stages, and almost every tool in this space is really a bundle of two or three of them.

• Instrument. Your code creates a metric (a counter, a gauge, a histogram) using a client library or an OpenTelemetry SDK and updates it as the program runs.
• Collect. An agent gathers the numbers — either by scraping an HTTP endpoint your app exposes (pull) or by receiving what your app sends it (push) — then batches, enriches, and forwards them. This is Prometheus's server, the OpenTelemetry Collector, or Telegraf.
• Store. A time-series database persists the samples on disk, compressing aggressively and indexing by label.
• Query. A query engine reads it back and aggregates on demand — PromQL for Prometheus-shaped data, SQL for InfluxDB 3 and TimescaleDB.
• Visualize. A dashboard layer — almost always Grafana — turns queries into graphs.
• Alert. A rules engine evaluates queries on a schedule and fires when a condition holds, routing the notification to a human.

Instrumenting code

Everything starts with a measurement in your program. There are only four shapes a metric takes, and picking the right one is most of the craft.

A working example, using the Prometheus Python client. The shape barely changes across languages or with an OpenTelemetry SDK — what matters is the discipline around labels.

from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter("http_requests_total", "Total HTTP requests",
                    ["method", "route", "status"])   # bounded labels only
LATENCY  = Histogram("http_request_seconds", "Request latency", ["route"])

@LATENCY.labels(route="/checkout").time()        # observe the duration
def handle_checkout(req):
    REQUESTS.labels("POST", "/checkout", "200").inc()   # count this request
    ...

start_http_server(9090)   # exposes GET /metrics for a scraper to pull

How to choose what to measure

Two methodologies dominate, and they're complementary:

• RED — Rate, Errors, Duration, measured per request or endpoint. It tells you how happy your users are, so it's the right thing to alert on (symptoms, not causes).
• USE — Utilization, Saturation, Errors, measured per resource (CPU, memory, disk). It tells you how happy your machines are, which is where you look to diagnose a symptom RED surfaced.

One modern refinement worth knowing: classic histograms force you to pick bucket boundaries up front, and each bucket is a series. Native (exponential) histograms, a stable Prometheus feature, replace fixed buckets with an automatically-scaled exponential layout — far better resolution at a fraction of the series cost.

Pull vs push

The oldest argument in metrics: does the collector reach out and scrape your service, or does your service send its numbers somewhere? Prometheus is the famous pull system; InfluxDB/Telegraf, OpenTelemetry, and StatsD are push. Each genuinely wins in different places.

Pull (Prometheus) wins on operability: every scrape produces an up metric, so the server knows whether each target is healthy, and when service discovery says an instance is gone, its series simply stop — no cleanup. Push wins on reach: a function that lives for 200 ms, a nightly batch job, or anything behind a firewall can't be scraped, so it sends instead.

The cardinality problem

If you remember one thing about operating a metrics system, make it this. A time series is one unique combination of a metric name and its label values. Every distinct combination is stored and indexed separately. So the cost of a metric isn't how often you update it — it's how many label combinations it can produce.

This is exactly where the three pillars divide the labor. Metrics are aggregations into bounded series, so unbounded identifiers are toxic to them. Logs and traces are per-event records — high-cardinality IDs are precisely the dimension you want to search them on. The discipline: keep the unbounded ID on the log line, the trace span, or a Prometheus exemplar — and let the metric tell you, cheaply, only that something is wrong.

Storage engines compared

The middle of the pipeline — store and query — is where the well-known databases actually compete. Four worth knowing: Prometheus's built-in TSDB, InfluxDB 3, TimescaleDB, and VictoriaMetrics. Use the controls to filter rows to one dimension or highlight a single engine's column. (Both persist across reloads; Copy as Markdown always exports the full matrix.)

Read the columns as philosophies. Prometheus assumes a single node is enough for one team's recent data and pushes everything else to a separate long-term layer. InfluxDB rebuilt itself around columnar files in object storage and SQL. TimescaleDB makes the radical-but-pragmatic bet that you shouldn't need a separate database at all — time-series is just Postgres with good partitioning. VictoriaMetrics keeps Prometheus's interface but rewrites the engine underneath for efficiency at scale.

Scaling Prometheus

Prometheus is deliberately a single binary on a single node: no clustering, no built-in long-term storage (the local disk defaults to roughly 15 days of retention), and no multi-tenancy. Its own scaling features — functional sharding and federation — only move aggregated series upward, not a complete global view. That gap created an entire ecosystem of long-term, horizontally-scalable backends that speak Prometheus's protocols.

Note the family split: Thanos, Cortex, and Mimir all reuse Prometheus's own code and storage format, while VictoriaMetrics is a clean-room reimplementation that merely speaks the same protocols. Mimir is the Cortex fork that Grafana hardened for extreme scale and runs behind Grafana Cloud.

OpenTelemetry & OTLP

For years, picking a metrics backend meant picking an instrumentation library — switching vendors meant re-instrumenting every service. OpenTelemetry (a CNCF project, and now the de-facto standard) breaks that coupling. You instrument once against the OTel SDK, emit OTLP over the wire, and decide later and separately where it lands.

The workhorse is the OpenTelemetry Collector, a small pipeline of its own: receivers take telemetry in (by push or by scraping Prometheus targets), processors batch / filter / transform it, and exporters fan it out to one or more backends. It runs either as a per-host agent or as a central gateway.

Interop runs both ways. The Collector's prometheus receiver scrapes existing targets; its exporters write to Prometheus, Mimir, or InfluxDB. And Prometheus 3.x now ingests OTLP natively (enable the OTLP receiver and POST to /api/v1/otlp/v1/metrics).

Query, visualize, alert

Querying

Two query philosophies. PromQL (Prometheus, and VictoriaMetrics' compatible MetricsQL) is a functional language built around time-series math — rate(), histogram_quantile(), vector matching. SQL (InfluxDB 3 and TimescaleDB) trades that purpose-built terseness for the familiarity, joins, and tooling of the relational world. A taste of PromQL, the dialect you'll meet most:

# per-second rate of 5xx responses over the last 5 minutes
rate(http_requests_total{status=~"5.."}[5m])

# p99 latency, reconstructed from histogram buckets
histogram_quantile(0.99, rate(http_request_seconds_bucket[5m]))

Visualizing

Grafana is the near-universal dashboard layer, and its superpower is being multi-datasource: a single dashboard can pull one panel from Prometheus, another from InfluxDB, another from a SQL database. It's why the storage decision and the visualization decision are largely independent — almost everything graphs in Grafana.

Alerting

Two rule types do the work. Recording rules pre-compute an expensive query on a schedule and save the result as a new series, so dashboards and alerts read something cheap. Alerting rules evaluate a condition and, when it holds, fire to Alertmanager, which deduplicates, groups, silences, and routes the notification to a human. The best-practice pairing with RED: alert on symptoms (a rising error rate users feel), then use your dashboards to find the cause.

Deployment & decision guide

The honest mapping from situation to stack:

• Cloud-native infra & app monitoring (Kubernetes). Reach for Prometheus + Grafana + Alertmanager — the default, with first-class service discovery and a vast exporter catalog. Add Mimir, Thanos, or VictoriaMetrics when one node isn't enough.
• Metrics next to relational / business data, with a SQL team. Reach for TimescaleDB — keep metrics and the rest of your data in one Postgres, join freely, and skip operating a second database.
• High-volume push, IoT, or sensor telemetry. Reach for InfluxDB — line-protocol ingest, the Telegraf agent ecosystem, and a columnar engine built for write-heavy event streams.
• Huge scale or tight cost on a Prometheus-shaped stack. Reach for VictoriaMetrics or Grafana Mimir — same PromQL and Grafana, far more series per dollar.
• Future-proof instrumentation, undecided backend. Instrument with OpenTelemetry and route through the Collector — you can change your mind about storage later without touching code.

Common misconceptions

Further reading

Prometheus & the scaling tier

• prometheus.io — the primary source for the data model, local storage, PromQL, pull model, and Alertmanager; see also the Prometheus 3.0 announcement and the naming & cardinality guidance.
• Thanos, Cortex, Grafana Mimir, and VictoriaMetrics — the long-term / horizontally-scalable backends.

InfluxDB & TimescaleDB

• InfluxDB 3 docs — Core / Enterprise / Cloud editions, line protocol, SQL, and the Arrow/DataFusion/Parquet engine; Telegraf for collection.
• TimescaleDB (by Tiger Data) — hypertables, continuous aggregates, and the Hypercore columnar engine; the docs.

OpenTelemetry & methodology

• OpenTelemetry — metrics, the OTLP spec, the Collector architecture, and the observability primer (metrics vs logs vs traces); plus Prometheus's OTLP guide.
• The RED method (and USE) — what to measure and what to alert on.

On this site

• ClickHouse: An Engineering Doc — the columnar, vectorized OLAP engine that shares this world's DNA (and is itself a popular store for very high-volume metrics and observability data).