Monitoring & Observability

Surface Security includes a full observability stack for monitoring system health, tracing requests, and collecting metrics.

Observability Stack

Component	Purpose	Docker Compose Port	K8s Service Port
Grafana	Dashboards and visualization	3001	3000
Tempo	Distributed tracing storage	3200	3200
OTel Collector	Receives traces and metrics	4317 (gRPC), 4318 (HTTP)	Same
MinIO	Object storage (screenshots)	9500 (API), 9501 (Console)	9000 (API), 9001 (Console)

Health Checks

Surface Security provides comprehensive health check endpoints with dependency verification.

Liveness Check

Confirms the process is alive. Used by Kubernetes liveness probes.

curl http://localhost:8080/health
# Response: {"status": "healthy"}

Readiness Check

Confirms the service is ready to accept traffic. Verifies connectivity to PostgreSQL, ClickHouse, and Redpanda.

curl http://localhost:8080/ready
# Response: {"status": "healthy", "ready": true}

Full Health Response

{
  "status": "healthy",
  "version": "1.2.0",
  "uptime_seconds": 86400,
  "timestamp": "2024-01-15T10:30:00Z",
  "dependencies": {
    "postgresql": {
      "name": "postgresql",
      "status": "healthy",
      "latency_ms": 2,
      "last_check": "2024-01-15T10:30:00Z"
    },
    "clickhouse": {
      "name": "clickhouse",
      "status": "healthy",
      "latency_ms": 5,
      "last_check": "2024-01-15T10:30:00Z"
    },
    "kafka": {
      "name": "kafka",
      "status": "healthy",
      "latency_ms": 3,
      "last_check": "2024-01-15T10:30:00Z"
    }
  }
}

Status Values:

Status	Meaning
healthy	All systems operational
degraded	Some non-critical systems impaired
unhealthy	Critical systems failing

Kubernetes Probes

The API deployment includes properly configured probes:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 15  # Allow 150s for startup

Prometheus Metrics

Prometheus metrics are available at the following endpoints:

Service	Endpoint
API	http://localhost:8080/metrics
Ingestion	http://localhost:9090/metrics
OTel Collector	http://localhost:8889/metrics

Grafana Dashboards

Grafana is pre-configured with datasources for Tempo (tracing) and Prometheus (metrics).

Docker Compose: Access Grafana at http://localhost:3001

Kubernetes: Access Grafana via the cluster service on port 3000, or through your ingress configuration.

Grafana provisioning configuration is located at infra/docker/config/grafana/provisioning/.

Distributed Tracing

Tempo provides distributed tracing for backend API requests. Traces are collected by the OpenTelemetry Collector and forwarded to Tempo for storage and querying.

• View traces in Grafana by selecting the Tempo datasource
• Trace context propagates through the API, ingestion, and event processing pipeline
• Configuration: infra/docker/config/tempo/tempo.yaml

Logs

Docker Compose

# All services
docker compose logs -f

# Specific service
docker compose logs -f api

# Filter errors
docker compose logs api 2>&1 | grep -i error

Kubernetes

# API logs
kubectl logs deployment/surfacesec-api -n surfacesec --tail=100

# Ingestion logs
kubectl logs deployment/surfacesec-ingestion -n surfacesec --tail=100

# Follow logs in real time
kubectl logs -f deployment/surfacesec-api -n surfacesec

Alerting

Configure alerting in the admin dashboard:

1. Go to Settings > Alerts
2. Enable Email Notifications or Webhook
3. Set alert thresholds for severity levels and event types