Skip to main content

Operational Runbooks

1. Consumer Lag Spike​

Symptom: surfacesec_consumer_lag metric exceeds 10,000 and rising.

Impact: Dashboard data becomes stale. New events are queued in MSK/Redpanda but not yet stored in ClickHouse. No data loss (events are retained in the broker).

Diagnosis:

# Check ingestion pod health
kubectl get pods -n surfacesec -l component=ingestion

# Check ingestion logs for errors
kubectl logs -n surfacesec -l component=ingestion --tail=50 | grep -i error

# Check ClickHouse connectivity from an ingestion pod
kubectl exec -n surfacesec deploy/surfacesec-ingestion -- wget -qO- http://localhost:9090/ready

# Check for ClickHouse merge storms (if self-hosted)
# system.parts active count > 300 indicates merge backlog
kubectl exec -n surfacesec deploy/surfacesec-clickhouse -- \
  clickhouse-client -q "SELECT table, count() as parts FROM system.parts WHERE active GROUP BY table ORDER BY parts DESC"

Resolution:

  1. If ingestion pods are crash-looping: check logs for the root cause (ClickHouse connection refused, OOM, etc.)
  2. If ClickHouse is slow: check system.merges for active merges. If parts > 300, the batch inserter may need a larger batch size.
  3. If HPA is scaling too slowly: manually scale ingestion pods: 
    kubectl scale deploy/surfacesec-ingestion -n surfacesec --replicas=20
  4. Monitor lag -- it should decrease steadily. Once caught up, HPA will scale down.

2. ClickHouse "Too Many Parts"​

Symptom: ClickHouse insert errors with "too many parts" in ingestion logs.

Impact: New events are rejected by ClickHouse. Consumer lag rises.

Diagnosis:

# Check active parts count per table
kubectl exec -n surfacesec deploy/surfacesec-clickhouse -- clickhouse-client -q "
  SELECT database, table, count() as parts, sum(rows) as total_rows
  FROM system.parts
  WHERE active
  GROUP BY database, table
  ORDER BY parts DESC
"

# Check active merges
kubectl exec -n surfacesec deploy/surfacesec-clickhouse -- clickhouse-client -q "SELECT * FROM system.merges"

Resolution:

  1. Verify async inserts are enabled: async_insert=1 in connection settings
  2. Verify batch inserter is running: check ingestion logs for "flushed ... event batch" messages
  3. If parts > 300 on a single table: wait for merges to complete (can take minutes)
  4. For ClickHouse Cloud: check the ClickHouse Cloud console for auto-scaling status
  5. As a last resort: OPTIMIZE TABLE <table> FINAL (blocks until merge completes, use sparingly)

3. PostgreSQL Connection Exhaustion​

Symptom: API pods returning 500 errors. pg_stat_activity shows connections at max.

Impact: API requests fail. Dashboard returns errors.

Diagnosis:

# Check connection count vs max
psql -c "SELECT count(*) FROM pg_stat_activity;"
psql -c "SHOW max_connections;"

# Check for idle connections
psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"

# Check PgBouncer stats (if enabled)
psql -h pgbouncer -p 6432 -U pgbouncer -c "SHOW POOLS;"

Resolution:

  1. If PgBouncer is enabled: check PgBouncer stats for connection pool saturation
  2. Kill idle connections: SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND query_start < NOW() - INTERVAL '10 minutes';
  3. Scale down non-essential pods temporarily to free connections
  4. Long-term: increase RDS instance class for more max_connections

4. RDS Failover​

Symptom: Elevated API error rate for 60-120 seconds. Connections reset.

Impact: Brief service disruption during Multi-AZ failover. Automatic recovery.

What happens:

  • RDS Multi-AZ failover is automatic (hardware failure, OS patching, AZ outage)
  • The pgx connection pool detects broken connections and reconnects
  • Failover typically completes in 60-120 seconds
  • No manual intervention required

Monitoring:

# Check RDS events in AWS console
aws rds describe-events --source-type db-instance --source-identifier surfacesec-prod-pg --duration 60

# Check API error rate during failover window
# Expect a spike in 5xx responses, then recovery

Post-failover verification:

  1. Confirm API readiness probes are passing: kubectl get pods -n surfacesec -l component=api
  2. Confirm dashboard loads data correctly
  3. Check pg_stat_activity shows normal connection count

5. EKS Node Failure​

Symptom: Pods rescheduled to other nodes. Brief availability dip.

Impact: PodDisruptionBudgets maintain minimum availability. Recovery is automatic.

What happens:

  • Cluster autoscaler or Karpenter detects the failed node
  • A replacement node is provisioned (2-5 minutes for EC2, faster for Karpenter)
  • Kubernetes reschedules pods from the failed node to healthy nodes
  • PDBs ensure minimum pod counts are maintained during rescheduling

Monitoring:

# Check node status
kubectl get nodes

# Check pod rescheduling
kubectl get pods -n surfacesec -o wide --sort-by=.status.startTime | tail -20

# Check events for scheduling issues
kubectl get events -n surfacesec --sort-by=.lastTimestamp | tail -20

If pods are stuck in Pending:

  1. Check if cluster autoscaler is at max nodes: kubectl logs -n kube-system -l app=cluster-autoscaler --tail=20
  2. Increase maxNodes in EKS node group configuration
  3. Check for resource constraints: kubectl describe pod <pending-pod> -n surfacesec | grep -A5 Events