Build a production monitoring stack with Prometheus, Grafana, and Alertmanager for infrastructure visibility, custom dashboards, and incident alerting.
Why Prometheus and Grafana
Monitoring is the foundation of reliable infrastructure. Without visibility into CPU usage, memory pressure, disk I/O, network throughput, and application-level metrics, operational teams are flying blind. Prometheus and Grafana have become the industry-standard open-source monitoring stack, used by organizations from startups to Fortune 500 companies.
This guide covers a production-grade deployment of the monitoring stack, from installation through alerting, based on patterns we implement across our server management engagements.
Architecture Overview
Targets (exporters)
├── node_exporter (system metrics)
├── mysqld_exporter (database metrics)
├── nginx_exporter (web server metrics)
└── application /metrics endpoint
│
▼
Prometheus Server
(scrape, store, query)
│
├─────┼──────┐
▼ ▼ ▼
Grafana Alert Recording
(viz) Manager Rules
│
├─────┼──────┐
▼ ▼ ▼
Slack PagerDuty Email
Prometheus pulls metrics from targets at configured intervals, stores them in a time-series database, and evaluates alerting rules. Grafana queries Prometheus to render dashboards. Alertmanager handles alert routing, grouping, and notification delivery.
Installing Prometheus
Docker Compose Deployment
# docker-compose.yml
version: "3.8"
services:
prometheus:
image: prom/prometheus:v2.51.0
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus/rules:/etc/prometheus/rules
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=30d"
- "--storage.tsdb.retention.size=50GB"
- "--web.enable-lifecycle"
restart: unless-stopped
grafana:
image: grafana/grafana:10.4.0
container_name: grafana
ports:
- "3000:3000"
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
GF_INSTALL_PLUGINS: grafana-piechart-panel
restart: unless-stopped
alertmanager:
image: prom/alertmanager:v0.27.0
container_name: alertmanager
ports:
- "9093:9093"
volumes:
- ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
restart: unless-stopped
volumes:
prometheus_data:
grafana_data:
Prometheus Configuration
# prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
rule_files:
- "rules/*.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node"
static_configs:
- targets:
- "web-server-1:9100"
- "web-server-2:9100"
- "db-server-1:9100"
- job_name: "mysql"
static_configs:
- targets: ["db-server-1:9104"]
- job_name: "nginx"
static_configs:
- targets:
- "web-server-1:9113"
- "web-server-2:9113"
- job_name: "application"
metrics_path: "/metrics"
static_configs:
- targets:
- "app-server-1:8080"
- "app-server-2:8080"
Node Exporter Setup
Install node_exporter on every server to collect system metrics:
# Download and install
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service <<'SERVICE'
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter \
--collector.systemd \
--collector.processes \
--collector.conntrack \
--no-collector.infiniband \
--no-collector.nfs
Restart=always
[Install]
WantedBy=multi-user.target
SERVICE
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
Alerting Rules
Define alert rules for critical infrastructure conditions:
# prometheus/rules/infrastructure.yml
groups:
- name: infrastructure
interval: 30s
rules:
- alert: HighCPUUsage
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 10m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage is {{ $value | printf "%.1f" }}% for over 10 minutes."
- alert: DiskSpaceLow
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
for: 5m
labels:
severity: critical
annotations:
summary: "Disk space critically low on {{ $labels.instance }}"
description: "Only {{ $value | printf "%.1f" }}% disk space remaining."
- alert: HighMemoryUsage
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 10m
labels:
severity: warning
annotations:
summary: "High memory usage on {{ $labels.instance }}"
description: "Memory usage is {{ $value | printf "%.1f" }}%."
- alert: InstanceDown
expr: up == 0
for: 3m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} is down"
description: "{{ $labels.job }} exporter on {{ $labels.instance }} has been unreachable for 3 minutes."
Alertmanager Configuration
Route alerts to the right channels based on severity:
# alertmanager/alertmanager.yml
global:
resolve_timeout: 5m
slack_api_url: "https://hooks.slack.com/services/T00/B00/xxxx"
route:
receiver: "slack-default"
group_by: ["alertname", "instance"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: "pagerduty-critical"
repeat_interval: 1h
- match:
severity: warning
receiver: "slack-warnings"
repeat_interval: 4h
receivers:
- name: "slack-default"
slack_configs:
- channel: "#monitoring"
title: '{{ template "slack.default.title" . }}'
text: '{{ template "slack.default.text" . }}'
- name: "slack-warnings"
slack_configs:
- channel: "#monitoring-warnings"
- name: "pagerduty-critical"
pagerduty_configs:
- service_key: "your-pagerduty-integration-key"
severity: critical
inhibit_rules:
- source_match:
severity: critical
target_match:
severity: warning
equal: ["instance"]
The inhibit rule suppresses warning alerts when a critical alert is already firing for the same instance, reducing alert noise during incidents.
Grafana Dashboard Design
Data Source Provisioning
# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
Key Dashboard Panels
Build dashboards with these essential panels:
System Overview: CPU, memory, disk, and network for all servers in a single view using node_exporter metrics.
Application Performance: Request rate, error rate, and latency percentiles (the RED method).
Database Health: Query throughput, slow queries, replication lag, connection pool utilization.
Example PromQL Queries
# Request rate per second (5-minute average)
rate(http_requests_total[5m])
# 99th percentile latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
# Disk I/O utilization
rate(node_disk_io_time_seconds_total[5m]) * 100
Recording Rules for Performance
Pre-compute expensive queries as recording rules to speed up dashboards:
# prometheus/rules/recording.yml
groups:
- name: recording
rules:
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))
- record: job:http_errors:rate5m
expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
- record: instance:node_cpu:utilization
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
Best Practices
- Set retention based on your needs - 30 days for operational metrics, use Thanos or Cortex for long-term storage
- Use recording rules for dashboard queries to reduce Prometheus load
- Label targets consistently with
environment,team, andservicelabels - Create runbooks linked from alert annotations for faster incident response
- Monitor the monitoring stack itself - alert on Prometheus scrape failures and high memory usage
- Secure Grafana with SSO and role-based access control
- Integrate with your infrastructure management workflows for automated remediation
A well-designed monitoring stack turns reactive firefighting into proactive operations, catching issues before they impact users.
Frequently asked questions
How do I run Prometheus, Grafana and Alertmanager with Docker Compose?
Define three services in docker-compose.yml using prom/prometheus:v2.51.0 on port 9090, grafana/grafana:10.4.0 on port 3000 and prom/alertmanager:v0.27.0 on port 9093, each with a restart policy of unless-stopped. Mount prometheus.yml and a rules directory into the Prometheus container, and give Prometheus and Grafana named volumes for their data. Pass the Grafana admin password through GF_SECURITY_ADMIN_PASSWORD from an environment variable so it stays out of the file.
How do I set data retention in Prometheus?
Prometheus retention is set with startup flags, for example --storage.tsdb.retention.time=30d and --storage.tsdb.retention.size=50GB next to --storage.tsdb.path=/prometheus. Thirty days suits operational metrics. For long-term storage, add Thanos or Cortex. In a Docker Compose setup the /prometheus path sits on the prometheus_data named volume.
How do I install node_exporter as a systemd service?
Download node_exporter-1.7.0.linux-amd64.tar.gz from the node_exporter GitHub releases, extract it with tar xzf and copy the node_exporter binary to /usr/local/bin. Create /etc/systemd/system/node_exporter.service to run it as the node_exporter user with --collector.systemd, --collector.processes and --collector.conntrack, plus --no-collector.infiniband and --no-collector.nfs. Then run sudo systemctl daemon-reload and sudo systemctl enable --now node_exporter. Prometheus scrapes it on port 9100.
What Prometheus alert rules should I set up for Linux servers?
A basic set covers four conditions. HighCPUUsage fires when CPU stays above 85 percent for 10 minutes and HighMemoryUsage when memory use passes 90 percent for 10 minutes, both as warnings. DiskSpaceLow is critical when the root filesystem has under 15 percent free for 5 minutes, and InstanceDown is critical when up == 0 for 3 minutes. Runbooks linked from the alert annotations speed up incident response.
How do I route Prometheus alerts to PagerDuty and Slack by severity?
Alertmanager routes on the severity label. Alerts with severity critical go to a pagerduty-critical receiver with a repeat_interval of 1h, warnings go to a slack-warnings receiver with 4h, and everything else falls to a default Slack receiver posting in #monitoring. Grouping by alertname and instance, with group_wait at 30s and group_interval at 5m, controls how related alerts are batched before they are sent.
How do I suppress warning alerts while a critical alert is firing in Alertmanager?
Use an inhibit rule with source_match on severity critical, target_match on severity warning and equal set to instance. Alertmanager then suppresses warning alerts for an instance whenever a critical alert is already firing for that same instance, which cuts alert noise during incidents.
How can I make slow Grafana dashboards faster with Prometheus?
Pre-compute expensive queries as Prometheus recording rules and point dashboard panels at the recorded series. A rule named job:http_requests:rate5m can store sum by (job) (rate(http_requests_total[5m])), and instance:node_cpu:utilization can hold per-instance CPU utilization. Recording rules also reduce the query load on Prometheus itself.
What panels should a Grafana server monitoring dashboard include?
A Grafana dashboard for servers can be built from three panels: a system overview with CPU, memory, disk and network for all servers from node_exporter metrics, application performance using the RED method of request rate, error rate and latency percentiles, and database health covering query throughput, slow queries, replication lag and connection pool utilization. For 99th percentile latency, a panel can query histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).
Read how we handle it in Cloud & Server Management.
Related Articles
How to Change a Schema on a Busy MySQL Table Without Locking It
A plain ALTER on a large InnoDB table can hold up every writer until it finishes, which on a busy table means an outage nobody scheduled. Modern MySQL does far more instantly than most teams realise, so the first job is checking whether you need a tool at all. When you do, the copy-and-swap approach builds a shadow table, keeps it in step from the binary log, and swaps the two at the end. This guide covers what the table has to look like for that to work, and how to stop a migration safely once it is running.
Server & DevOpsHow to Set Up GlusterFS on Ubuntu
A complete guide to setting up a distributed, replicated GlusterFS filesystem across multiple Ubuntu 22.04 nodes, including installation, volume creation, client mounting, maintenance, and troubleshooting.
Server & DevOpsBuilding CI/CD Pipelines with GitHub Actions
Design production-grade CI/CD pipelines using GitHub Actions with matrix builds, environment protection rules, and automated rollback strategies.