Prometheus alerting that doesn't wake you up for nothing

The fastest way to make on-call useless is to page people for things that do not matter. After a few nights of "CPU above 80%" at 3 a.m. with nothing broken, engineers mute the channel, and the alert that does matter gets missed. Most noisy setups I see have the same causes, and they can be fixed without buying anything. This is how I design Prometheus alerting so that a page means someone has to act now.

Rule one: page on symptoms, not causes

A page should mean users are affected, or will be very soon. That means alerting on what users experience:

• requests failing (error rate);
• requests too slow (latency);
• the service unreachable;
• work not getting done (queue age, jobs not completing).

High CPU, a pod restart, a full-ish disk or a replica down are causes. They might lead to user impact, or they might not: autoscaling, retries and redundancy exist precisely so that they don't. Causes belong on dashboards and in tickets, not in someone's phone at night.

The exception is a cause that will certainly become an outage with no automatic recovery, such as a disk that will be full within hours, or a certificate that expires in days. Those are predictable, so alert on the prediction:

- alert: DiskWillFillIn4Hours
  expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0
  for: 30m
  labels:
    severity: page
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} will be full within 4 hours"
    runbook: "https://runbooks.example.com/disk-full"

Two severities, two destinations

I use two levels and resist adding more:

• page: a person must act now, at any hour. Goes to the on-call phone.
• ticket: someone must look within working hours. Goes to a channel or the issue tracker.

Everything else is not an alert. If no one would do anything about it, it is a dashboard panel. A "warning" level that goes to a channel nobody reads is just noise with a nicer name.

Alert on error budgets, not on single spikes

A five-minute error spike that fixes itself does not need a human. A small but steady error rate that will eat a month's reliability target in two days does. Static thresholds handle both badly. SLO burn-rate alerts handle both well.

Start with a target, for example: 99.9% of requests succeed over 30 days. That leaves an error budget of 0.1%. The burn rate is how fast you are spending it: a burn rate of 1 uses the budget in exactly 30 days, and 14.4 uses it in about two days.

First, a recording rule for the error ratio over several windows:

groups:
- name: api-slo
  rules:
  - record: job:request_errors:ratio_rate5m
    expr: |
      sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
      /
      sum by (job) (rate(http_requests_total[5m]))
  - record: job:request_errors:ratio_rate1h
    expr: |
      sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
      /
      sum by (job) (rate(http_requests_total[1h]))

Then alert when both a long and a short window burn fast. The long window proves it is sustained; the short one makes the alert stop soon after the problem is fixed:

  - alert: ApiErrorBudgetBurn
    expr: |
      job:request_errors:ratio_rate1h{job="api"} > (14.4 * 0.001)
      and
      job:request_errors:ratio_rate5m{job="api"} > (14.4 * 0.001)
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "API is burning its error budget 14x too fast"
      runbook: "https://runbooks.example.com/api-errors"

A second rule with a lower burn rate over longer windows (for example 6 hours and 30 minutes at a burn rate of 6) catches slower problems. For slow, steady leaks, a ticket-level alert at a burn rate of 1 over three days is enough. This multi-window approach is described in detail in Google's SRE workbook, and it is the single change that removes the most noise in the setups I review.

Make every alert actionable

Each alert rule should answer three questions for the person who receives it:

• What is broken, in the summary, in plain words and with the affected service or instance.
• How bad, through the severity and, where useful, the current value.
• What to do, through a runbook link: how to confirm the problem, the usual causes, and how to mitigate.

An alert without a runbook is a sign the alert is not understood yet. Writing the runbook often shows that the alert should be a dashboard panel instead.

Use for: so that a single bad scrape does not page anyone, but keep it short on paging alerts. The burn-rate windows already smooth out spikes.

Let Alertmanager do the grouping

Many noisy setups are really routing problems. Alertmanager can turn fifty notifications into one:

route:
  receiver: tickets
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: on-call
      repeat_interval: 1h

inhibit_rules:
  - source_matchers: [alertname="ClusterUnreachable"]
    target_matchers: [severity=~"page|ticket"]
    equal: [cluster]

• Grouping sends one notification for all pods of a failing service, not one per pod.
• Inhibition suppresses the symptoms of a bigger failure. If the whole cluster is unreachable, nobody needs forty more alerts about the services in it.
• Silences are for planned maintenance. Create them with an expiry and a comment, never as a permanent mute.

Test alerts like code

Alert rules are code, so they live in Git and go through review. Two tools make that real:

• promtool check rules in CI catches syntax errors before they reach Prometheus.
• promtool test rules runs unit tests: given this series of metric values, this alert must fire after this many minutes, with these labels. It is the only way to be sure an alert fires when it should and stays quiet when it should not.

A dead man's switch completes the picture: an alert that always fires, routed to an external service that pages you if it stops arriving. Otherwise a broken Prometheus or Alertmanager is silent, which looks exactly like a quiet night.

Review the pages every week

The rules above get you a good start. What keeps alerting healthy is a short weekly review of every page from the past week:

• Was it real? Did someone have to act?
• Was it actionable from the alert and the runbook alone?
• Could it have waited until morning?

Every page that fails one of these questions gets a change: delete it, downgrade it to a ticket, raise the threshold, fix the runbook, or fix the underlying problem so it stops happening. A useful target is no more than one or two pages per on-call shift; beyond that, people stop trusting the pager.

Summary

• Page on user-facing symptoms. Causes go on dashboards, except predictable failures.
• Two severities: page now, or ticket for working hours.
• Use SLO burn-rate alerts with a long and a short window instead of static thresholds.
• Every alert has a clear summary and a runbook.
• Group and inhibit in Alertmanager; silences expire.
• Test rules with promtool, and add a dead man's switch.
• Review every page weekly and remove the ones that did not need a human.

If your team is tired of its pager, or you are setting up Prometheus and Grafana for the first time, this is the kind of work I do: dashboards, alerts and SLOs that the team trusts. The contact form is the quickest way to reach me.