Alerting Rules

Configure these alerts in Amazon Managed Grafana (or self-hosted Grafana) to detect infrastructure issues before they cause user-visible impact.

Critical Alerts (PagerDuty/OpsGenie)

Alert	PromQL / Condition	For	Action
Consumer lag spike	surfacesec_consumer_lag > 50000	5m	Ingestion falling behind. Check ingestion pod logs, verify ClickHouse connectivity, manually scale pods if HPA is too slow.
ClickHouse insert errors	rate(clickhouse_insert_errors_total[5m]) / rate(clickhouse_inserts_total[5m]) > 0.01	3m	Check system.parts for merge storms, verify async/batch inserts are enabled, check ClickHouse Cloud status.
API 5xx rate	sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05	3m	Check API pod logs, verify database connectivity, check for upstream dependency failures.
MSK/Redpanda unreachable	surfacesec_broker_kafka_api_responsive == 0	2m	Check MSK console for cluster health, verify network connectivity from EKS nodes, check security group rules.

Warning Alerts (Slack)

Alert	PromQL / Condition	For	Action
PG connection pool saturation	pg_pool_used / pg_pool_max > 0.9	5m	Approaching max connections. Check pg_stat_activity for idle connections, verify PgBouncer is routing correctly.
ALB/Envoy 429 rate	sum(rate(envoy_http_downstream_rq{envoy_response_code="429"}[5m])) > 100	5m	Rate limiting is active. Check if legitimate traffic spike or abuse. Consider increasing rate limits.
RDS CPU utilization	AWS CloudWatch CPUUtilization > 80	10m	Database overload. Check pg_stat_activity for long-running queries, review slow query log.
EKS nodes at max	kube_node_spec_unschedulable == 0 AND kube_node_status_condition{condition="Ready",status="true"} == kube_deployment_spec_replicas{deployment="cluster-autoscaler"}	10m	Cluster autoscaler ceiling hit. Increase maxNodes in EKS node group config.
Disk usage warning	surfacesec_broker_disk_used_percent > 80	5m	Redpanda/Kafka disk filling up. Check retention settings, consider expanding storage.
Consumer lag elevated	surfacesec_consumer_lag > 10000	5m	Severe ingestion backlog. May indicate ClickHouse outage or ingestion pod crash loop.

Notification Channels

• Critical: SNS topic -> PagerDuty/OpsGenie integration
• Warning: Slack webhook to #surfacesec-ops channel
• Info: Grafana dashboard annotations only (no notification)

Setup

Amazon Managed Grafana

1. Create alert contact points in AMG for SNS and Slack
2. Import alert rules via the Grafana API or provisioning
3. Configure notification policies: route severity=critical to PagerDuty, severity=warning to Slack

Self-hosted Grafana

1. Enable alerting in grafana.ini: [unified_alerting] enabled = true
2. Configure contact points via the Grafana UI
3. Create alert rules matching the PromQL expressions above