How to Write Alerts That Wake a Human Only When a Human Is Needed

An on-call rota does not fail when someone quits. It fails earlier, on the quiet morning when a page arrives and the person holding the phone glances at it and goes back to sleep, because the last nine were nothing. Everything below exists to keep that morning from arriving.

Page on what a customer can feel

Google's SRE material draws the line between what is broken and why it is broken, and puts the pager on the first. A disk filling up, a pod restarting or a cache miss rate climbing are all causes. They may or may not have reached anybody outside the building.

The same chapter offers a set of questions to run any candidate alert through. Does this detect an urgent, actionable condition that is affecting users and would otherwise go unnoticed. Will there be occasions when I know the alert is benign and ignore it. Does it definitely mean users are having a worse time. Can I act on it, and could that action be automated. Is somebody else already being paged for the same thing.

An alert that fails any of those is not a page. It is a dashboard panel, or a ticket for working hours, and the same material is blunt about email alerts, which drown in their own noise.

The budget is smaller than most teams assume. A human can respond to a handful of genuine emergencies in a day before fatigue sets in. Every page you send that did not need a human spends part of that.

Alert on the budget, not on the threshold

The threshold alert is the one everybody writes first. Error rate above one percent for five minutes, page. It fires during a thirty second blip that nobody noticed, and it stays silent through a week of quiet degradation that burns your entire month of allowance.

Burn rate fixes both ends. If your objective is 99.9 percent, then a sustained error rate of 0.1 percent consumes exactly the whole error budget over the window, which is a burn rate of 1. Ten times that rate exhausts it ten times faster.

The SRE workbook pairs each rate with two windows, a long one and a short one at about a twelfth of its length, and requires both to be over the threshold. The long window gives the alert meaning and the short one lets it stop firing once the problem is over. For a 99.9 percent objective over thirty days it recommends these.

The percentages are worth checking rather than trusting. Thirty days is 720 hours. A burn rate of 14.4 empties the budget in 720 divided by 14.4, which is 50 hours, so one hour at that rate spends one fiftieth of it, or 2 percent. A rate of 6 empties it in 120 hours, so 6 hours is 5 percent. A rate of 1 empties it in the full 30 days, so 3 days is 10 percent.

In Prometheus, record the ratio first so the alert stays readable.

# /etc/prometheus/rules/checkout-sli.yml
groups:
  - name: checkout-sli
    interval: 30s
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[5m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{job="checkout"}[1h]))

Substitute your own counter and label names. Then the alert, with the 0.001 standing for one minus the 99.9 percent objective.

# /etc/prometheus/rules/checkout-slo.yml
groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutErrorBudgetBurningFast
        expr: |
          job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
        for: 2m
        keep_firing_for: 5m
        labels:
          severity: page
        annotations:
          summary: Checkout is spending its error budget 14.4 times too fast
          runbook: https://example.com/runbooks/checkout-errors

for holds the alert in a pending state until the condition has been true throughout that period, which kills the blips. keep_firing_for, available since Prometheus 2.42, keeps it firing for a while after the condition clears, which kills the flapping and the false all-clear. Without it an alerting rule deactivates on the first evaluation where the condition is not met.

Check the file before it goes anywhere near a reload.

promtool check rules /etc/prometheus/rules/checkout-slo.yml

One incident, one notification

Prometheus decides what is wrong and Alertmanager decides who hears about it. Grouping is the part that turns forty simultaneous alerts into one message. group_by collects alerts sharing the labels you name, group_wait holds the first notification for 30 seconds by default so that stragglers join the same group, group_interval waits 5 minutes before sending an update about a group that has changed, and repeat_interval waits 4 hours before saying the same thing again.

Inhibition is the other half. An inhibit rule mutes a target alert while a source alert is firing, with both required to share the values of the labels listed under equal. A cluster wide outage suppressing the fifty service alerts underneath it is the standard use, and it is the difference between one page and a phone that will not stop.

Retire the alerts nobody acts on

Alerts accumulate. Nobody deletes them, because deleting one feels like accepting risk. So measure them instead. Prometheus writes a synthetic ALERTS series for every active alert, labelled with the alert name and whether it is pending or firing.

sort_desc(
  sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[30d]))
)

That ranks alerts by how much of the last thirty days each spent firing, counted in evaluation samples rather than in distinct incidents. The names at the top are the ones your team has learned to ignore.

Take each one and ask what a human did the last three times it fired. If the answer is nothing, the alert has already been retired in practice and the only honest move is to make that official. Send it to a ticket queue, turn it into a dashboard panel, or delete it. If the answer is that somebody always runs the same command, that is not an alert either, it is a script waiting to be written.

Doing this once a quarter costs an hour and is the only thing that keeps a rota trustworthy. If you would rather have the alerting rebuilt around objectives than pruned in place, that is our monitoring and observability work, and the practice around it, incident response and postmortems included, sits in our SRE service.