Monitoring and Alerting Setup
A robust monitoring and alerting system is the backbone of reliable production infrastructure. Without it, you are flying blind -- discovering outages only when users complain. This guide covers setting up a complete monitoring stack and designing effective alert rules.
The Four Golden Signals
Google's SRE book defines four key metrics for user-facing systems:
* **Latency** -- Time to service a request. Measure both average and high percentiles (p95, p99).
2\. **Traffic** -- Request rate (RPS, QPS) or throughput.
3\. **Errors** -- Rate of failed requests (5xx, timeouts, explicit error responses).
4\. **Saturation** -- How full the service is (CPU, memory, queue depth).
Every monitoring system should capture these four signals for each service.
Metrics Collection Stack
The Prometheus ecosystem has become the standard for metrics collection:
Application → Metrics Export → Prometheus → Grafana
↑ ↓
Node Exporter Alertmanager
↑ ↓
System Notification
Metrics Channels
Install Prometheus and configure it to scrape targets:
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['localhost:9100']
- job_name: 'app'
static_configs:
- targets: ['localhost:3000']
Use `scrape_interval` of 15 seconds for most metrics. For high-cardinality metrics (e.g., per-request tracing), use a longer interval or sample.
Application Instrumentation
Export application metrics in Prometheus format:
// Node.js with prom-client
const client = require('prom-client');
const httpRequestDuration = new client.Histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration in seconds',
labelNames: ['method', 'route', 'status'],
buckets: [0.01, 0.05, 0.1, 0.5, 1, 5]
});
// Record metrics in middleware
app.use((req, res, next) => {
const end = httpRequestDuration.startTimer();
res.on('finish', () => {
end({ method: req.method, route: req.route?.path || 'unknown', status: res.statusCode });
});
next();
});
Use Histogram metrics for latency, Counter for request counts, and Gauge for current resource usage. Avoid unbounded label cardinality.
Centralized Logging
The ELK stack (Elasticsearch, Logstash, Kibana) remains popular, but the Grafana Loki stack is simpler and cheaper for log aggregation:
# docker-compose for Loki + Promtail
services:
loki:
image: grafana/loki:3.0
ports: ["3100:3100"]
promtail:
image: grafana/promtail:3.0
volumes:
- /var/log:/var/log
- ./promtail.yml:/etc/promtail/promtail.yml
grafana:
image: grafana/grafana:latest
ports: ["3000:3000"]
Promtail tails log files, adds labels, and ships them to Loki. Grafana queries both Prometheus (metrics) and Loki (logs) in a unified dashboard.
Effective Alerting Rules
Design alerts that are actionable and meaningful:
# prometheus-alerts.yml
groups:
- name: application
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate ({{ $value | humanizePercentage }})"
- alert: HighLatency
expr: |
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])
) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "p99 latency is {{ $value }}s"
- alert: InstanceDown
expr: up == 0
for: 1m
labels:
severity: critical
Alert Design Principles
* **Alert on symptoms, not causes.** Alert on error rate, not on "CPU is high." CPU spikes may be normal; error rate spikes always require investigation.
* **Use the `for` clause.** Require the condition to persist for several minutes before firing to avoid flapping.
* **Set appropriate severity.** Critical alerts page someone immediately. Warning alerts create a ticket for next-day investigation.
* **Include runbooks.** Every alert annotation should reference a runbook URL.
Notification Channels
Route alerts through Alertmanager:
# alertmanager.yml
route:
receiver: 'team-page'
routes:
- match:
severity: warning
receiver: 'team-slack'
receivers:
- name: 'team-page'
pagerduty_configs:
- routing_key: '...'
- name: 'team-slack'
slack_configs:
- api_url: 'https://hooks.slack.com/services/...'
channel: '#alerts'
Critical alerts go to PagerDuty or Opsgenie for immediate attention. Warnings go to Slack for team awareness.
Dashboard Best Practices
Effective Grafana dashboards follow these principles:
* **One dashboard per service**, not one dashboard per engineer.
* **Show what matters**, not everything. Start with RED metrics (Rate, Errors, Duration).
* **Use templates** for environment and service selection.
* **Link to logs** -- a metric spike should have a one-click path to relevant logs.
Synthetic Monitoring
Complement real-user monitoring with synthetic checks:
# blackbox-exporter targets
modules:
http_2xx:
prober: http
http:
preferred_ip_protocol: "ip4"
valid_status_codes: [200, 201, 204]
Run synthetic checks from multiple geographic locations to detect regional outages.
Summary
A complete monitoring stack requires metrics (Prometheus), logs (Loki), dashboards (Grafana), and alerting (Alertmanager). Instrument your applications with the four golden signals, design alerts that fire on symptoms not causes, and ensure every alert has a clear path to resolution. Start simple with Prometheus and Grafana, then add log aggregation and synthetic monitoring as your infrastructure grows.