Skip to main content

Alerting Rules

Configure these alerts in Amazon Managed Grafana (or self-hosted Grafana) to detect infrastructure issues before they cause user-visible impact.

Critical Alerts (PagerDuty/OpsGenie)​

AlertPromQL / ConditionForAction
Consumer lag spikesurfacesec_consumer_lag > 500005mIngestion falling behind. Check ingestion pod logs, verify ClickHouse connectivity, manually scale pods if HPA is too slow.
ClickHouse insert errorsrate(clickhouse_insert_errors_total[5m]) / rate(clickhouse_inserts_total[5m]) > 0.013mCheck system.parts for merge storms, verify async/batch inserts are enabled, check ClickHouse Cloud status.
API 5xx ratesum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.053mCheck API pod logs, verify database connectivity, check for upstream dependency failures.
MSK/Redpanda unreachablesurfacesec_broker_kafka_api_responsive == 02mCheck MSK console for cluster health, verify network connectivity from EKS nodes, check security group rules.

Warning Alerts (Slack)​

AlertPromQL / ConditionForAction
PG connection pool saturationpg_pool_used / pg_pool_max > 0.95mApproaching max connections. Check pg_stat_activity for idle connections, verify PgBouncer is routing correctly.
ALB/Envoy 429 ratesum(rate(envoy_http_downstream_rq{envoy_response_code="429"}[5m])) > 1005mRate limiting is active. Check if legitimate traffic spike or abuse. Consider increasing rate limits.
RDS CPU utilizationAWS CloudWatch CPUUtilization > 8010mDatabase overload. Check pg_stat_activity for long-running queries, review slow query log.
EKS nodes at maxkube_node_spec_unschedulable == 0 AND kube_node_status_condition{condition="Ready",status="true"} == kube_deployment_spec_replicas{deployment="cluster-autoscaler"}10mCluster autoscaler ceiling hit. Increase maxNodes in EKS node group config.
Disk usage warningsurfacesec_broker_disk_used_percent > 805mRedpanda/Kafka disk filling up. Check retention settings, consider expanding storage.
Consumer lag elevatedsurfacesec_consumer_lag > 100005mSevere ingestion backlog. May indicate ClickHouse outage or ingestion pod crash loop.

Notification Channels​

  • Critical: SNS topic -> PagerDuty/OpsGenie integration
  • Warning: Slack webhook to #surfacesec-ops channel
  • Info: Grafana dashboard annotations only (no notification)

Setup​

Amazon Managed Grafana​

  1. Create alert contact points in AMG for SNS and Slack
  2. Import alert rules via the Grafana API or provisioning
  3. Configure notification policies: route severity=critical to PagerDuty, severity=warning to Slack

Self-hosted Grafana​

  1. Enable alerting in grafana.ini: [unified_alerting] enabled = true
  2. Configure contact points via the Grafana UI
  3. Create alert rules matching the PromQL expressions above

Surface Security

This documentation is protected. Enter the password to continue.