Skip to main content

Monitoring & Observability

Surface Security includes a full observability stack for monitoring system health, tracing requests, and collecting metrics.

Observability Stack​

ComponentPurposeDocker Compose PortK8s Service Port
GrafanaDashboards and visualization30013000
TempoDistributed tracing storage32003200
OTel CollectorReceives traces and metrics4317 (gRPC), 4318 (HTTP)Same
MinIOObject storage (screenshots)9500 (API), 9501 (Console)9000 (API), 9001 (Console)

Health Checks​

Surface Security provides comprehensive health check endpoints with dependency verification.

Liveness Check​

Confirms the process is alive. Used by Kubernetes liveness probes.

curl http://localhost:8080/health
# Response: {"status": "healthy"}

Readiness Check​

Confirms the service is ready to accept traffic. Verifies connectivity to PostgreSQL, ClickHouse, and Redpanda.

curl http://localhost:8080/ready
# Response: {"status": "healthy", "ready": true}

Full Health Response​

{
"status": "healthy",
"version": "1.2.0",
"uptime_seconds": 86400,
"timestamp": "2024-01-15T10:30:00Z",
"dependencies": {
"postgresql": {
"name": "postgresql",
"status": "healthy",
"latency_ms": 2,
"last_check": "2024-01-15T10:30:00Z"
},
"clickhouse": {
"name": "clickhouse",
"status": "healthy",
"latency_ms": 5,
"last_check": "2024-01-15T10:30:00Z"
},
"kafka": {
"name": "kafka",
"status": "healthy",
"latency_ms": 3,
"last_check": "2024-01-15T10:30:00Z"
}
}
}

Status Values:

StatusMeaning
healthyAll systems operational
degradedSome non-critical systems impaired
unhealthyCritical systems failing

Kubernetes Probes​

The API deployment includes properly configured probes:

livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 15
failureThreshold: 3

readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3

startupProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 15 # Allow 150s for startup

Prometheus Metrics​

Prometheus metrics are available at the following endpoints:

ServiceEndpoint
APIhttp://localhost:8080/metrics
Ingestionhttp://localhost:9090/metrics
OTel Collectorhttp://localhost:8889/metrics

Grafana Dashboards​

Grafana is pre-configured with datasources for Tempo (tracing) and Prometheus (metrics).

Docker Compose: Access Grafana at http://localhost:3001

Kubernetes: Access Grafana via the cluster service on port 3000, or through your ingress configuration.

Grafana provisioning configuration is located at infra/docker/config/grafana/provisioning/.

Distributed Tracing​

Tempo provides distributed tracing for backend API requests. Traces are collected by the OpenTelemetry Collector and forwarded to Tempo for storage and querying.

  • View traces in Grafana by selecting the Tempo datasource
  • Trace context propagates through the API, ingestion, and event processing pipeline
  • Configuration: infra/docker/config/tempo/tempo.yaml

Logs​

Docker Compose​

# All services
docker compose logs -f

# Specific service
docker compose logs -f api

# Filter errors
docker compose logs api 2>&1 | grep -i error

Kubernetes​

# API logs
kubectl logs deployment/surfacesec-api -n surfacesec --tail=100

# Ingestion logs
kubectl logs deployment/surfacesec-ingestion -n surfacesec --tail=100

# Follow logs in real time
kubectl logs -f deployment/surfacesec-api -n surfacesec

Alerting​

Configure alerting in the admin dashboard:

  1. Go to Settings > Alerts
  2. Enable Email Notifications or Webhook
  3. Set alert thresholds for severity levels and event types

Surface Security

This documentation is protected. Enter the password to continue.