Upgrade & Rollback

Operational procedures for upgrading and rolling back Surface Security deployments.

Pre-Upgrade Checklist

Before upgrading, verify each item:

Current version identified: kubectl get deployment surfacesec-api -n surfacesec -o jsonpath='{.spec.template.spec.containers[0].image}'
Target version confirmed: Review release notes for breaking changes
License valid: Verify license is not expired and covers the target version tier
Backup recent: Confirm database backup completed within the last 24 hours
Disk space sufficient: At least 20% free on all nodes
All pods healthy: kubectl get pods -n surfacesec shows all pods Running/Ready
No dirty migrations: Check schema_migrations table — dirty=true means the previous startup failed mid-migration
Maintenance window communicated: Notify stakeholders of the upgrade window

Standard Upgrade (Connected)

Via Admin Dashboard

Navigate to Settings > System > Updates
The dashboard shows available updates fetched from the Portal
Click Update on the target version
The update controller handles:
- Preflight checks (database connectivity, disk space, schema compatibility)
- Rolling update of API and ingestion deployments
- Postflight verification (health checks, consumer lag)
Monitor progress in the update history panel

Via Helm

# Update Helm repo (if using a chart repository)
helm repo update

# Review changes
helm diff upgrade surfacesec ./surfacesec \
  --set image.tag=X.Y.Z \
  -n surfacesec

# Apply upgrade
helm upgrade surfacesec ./surfacesec \
  --set image.tag=X.Y.Z \
  -n surfacesec \
  --wait --timeout 10m

Via Update Bundle

# Download update bundle from Portal
curl -o update-bundle-X.Y.Z.tar.gz \
  "https://portal.surface-security.com/api/v1/updates/X.Y.Z/bundle"

# Upload via admin dashboard: Settings > System > Updates > Upload Bundle
# Or via API:
curl -X POST \
  -H "Authorization: Bearer <admin-token>" \
  -F "bundle=@update-bundle-X.Y.Z.tar.gz" \
  https://surfacesec.example.com/api/v1/admin/updates/upload

Air-Gapped Upgrade

Step 1: Prepare on Connected Machine

VERSION=X.Y.Z

# Pull new images
for img in api ingestion preflight frontend; do
  docker pull "ghcr.io/<owner>/surfacesec-${img}:${VERSION}"
done

# Verify image signatures (see Image Verification guide)
for img in api ingestion preflight frontend; do
  cosign verify \
    --certificate-identity-regexp=".*" \
    --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
    "ghcr.io/<owner>/surfacesec-${img}:${VERSION}"
done

# Save images
docker save \
  ghcr.io/<owner>/surfacesec-api:${VERSION} \
  ghcr.io/<owner>/surfacesec-ingestion:${VERSION} \
  ghcr.io/<owner>/surfacesec-preflight:${VERSION} \
  ghcr.io/<owner>/surfacesec-frontend:${VERSION} \
  | gzip > surfacesec-images-${VERSION}.tar.gz

# Download Helm chart
helm package ./infra/helm/surfacesec

Step 2: Transfer to Air-Gapped Environment

Transfer the following files via secure media:

surfacesec-images-X.Y.Z.tar.gz
surfacesec-X.Y.Z.tgz (Helm chart)
Release notes / changelog

Step 3: Load and Deploy

# Load images
docker load < surfacesec-images-${VERSION}.tar.gz

# Tag and push to local registry
for img in api ingestion preflight frontend; do
  docker tag "ghcr.io/<owner>/surfacesec-${img}:${VERSION}" \
    "registry.internal/surfacesec/${img}:${VERSION}"
  docker push "registry.internal/surfacesec/${img}:${VERSION}"
done

# Upgrade via Helm
helm upgrade surfacesec ./surfacesec-${VERSION}.tgz \
  -f values-airgapped.yaml \
  --set image.tag=${VERSION} \
  -n surfacesec \
  --wait --timeout 10m

Database Migration Handling

Surface Security uses an Expand-Migrate-Contract pattern for zero-downtime schema changes.

Phase	Description	Rollback Support
Expand	Add new columns/tables (backward compatible)	No (forward-only — revert the app image, leave the columns)
Migrate	Idempotent data fixups on startup (`RunStartupDataMigrations`)	Idempotent (re-runnable)
Contract	Remove old columns/tables (separate release)	No (forward-only)

Migrations are forward-only: there are no .down.sql files. To roll back a release, revert the application image (Helm/kubectl rollout below). The expand columns/tables added by the newer version are backward-compatible and safe to leave in place under the older image.

Checking Migration Status

golang-migrate's schema_migrations table has two columns: version (bigint) and dirty (boolean).

-- PostgreSQL: check the highest applied migration and whether it succeeded
SELECT version, dirty FROM schema_migrations ORDER BY version DESC LIMIT 5;

A row with dirty=true means the API crashed or was killed while applying that migration. The database may be in a partially-applied state.

Manual Migration Recovery

If dirty=true after a failed startup:

# Check the migration logs for the error
kubectl logs deployment/surfacesec-api -n surfacesec | grep -i migration

The surfacesec-api image is distroless — it contains no psql and defines no DATABASE_URL. Run a one-off postgres client pod instead, pulling the same DB credentials the API uses from the DB credentials secret. schema_migrations lives on the direct Postgres host (the secret's host key — the migrator connects there via PG_MIGRATION_HOST, not the PgBouncer pooler).

SECRET=surfacesec-db-credentials   # your externalSecrets.dbCredentials secret name

kubectl run pg-toolbox --rm -it --restart=Never -n surfacesec --image=postgres:18-alpine \
  --env="PGHOST=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.host}' | base64 -d)" \
  --env="PGDATABASE=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.database}' | base64 -d)" \
  --env="PGUSER=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.username}' | base64 -d)" \
  --env="PGPASSWORD=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.password}' | base64 -d)" \
  -- psql

Then, inside the psql session:

-- Identify the failed version
SELECT version, dirty FROM schema_migrations ORDER BY version DESC LIMIT 1;

-- Rewind to the previous version so the next startup RE-RUNS the failed migration
-- from the top, rather than skipping it.
--
-- Do NOT just set dirty=false at the same version: golang-migrate would treat that
-- version as cleanly applied and advance to the next one, permanently leaving any
-- statements after the failure point unapplied. Every migration in this repo is
-- idempotent (IF NOT EXISTS / ADD COLUMN IF NOT EXISTS), so re-running the whole file
-- is safe — already-applied statements no-op.
UPDATE schema_migrations SET version=<version-minus-1>, dirty=false;

schema_migrations holds a single row (the current version). Setting it to the last good version (one below the failed one) makes Up() apply everything above it on the next boot, which re-runs the failed migration cleanly. If the migration failed on a genuine SQL error rather than a pod kill, fix the underlying cause (e.g. bad data) before restarting, or it will fail again.

Rollback Procedures

Automatic Rollback

The update controller triggers automatic rollback when postflight checks fail:

API health endpoint returns non-200
Readiness probes fail after timeout
Ingestion consumer lag exceeds threshold

The controller reverts to the previous image tag automatically.

Manual Rollback via Admin Dashboard

Navigate to Settings > System > Updates > Update History
Find the failed update entry
Click Rollback to revert to the previous version

Manual Rollback via Helm

# List Helm release history
helm history surfacesec -n surfacesec

# Rollback to previous revision
helm rollback surfacesec <previous-revision> -n surfacesec --wait --timeout 10m

Manual Rollback via kubectl

# Rollback API deployment
kubectl rollout undo deployment/surfacesec-api -n surfacesec

# Rollback ingestion deployment
kubectl rollout undo deployment/surfacesec-ingestion -n surfacesec

# Rollback frontend deployment
kubectl rollout undo deployment/surfacesec-frontend -n surfacesec

# Verify rollback
kubectl rollout status deployment/surfacesec-api -n surfacesec

Database Rollback

Migrations are forward-only — there is no automated down path. Roll back by reverting the application image (Helm or kubectl rollout above); the newer schema is backward-compatible, so the older image runs against it unchanged.

If a specific expand column or table genuinely must be removed, do it manually and deliberately as a follow-up — there is no down command:

SECRET=surfacesec-db-credentials

kubectl run pg-toolbox --rm -it --restart=Never -n surfacesec --image=postgres:18-alpine \
  --env="PGHOST=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.host}' | base64 -d)" \
  --env="PGDATABASE=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.database}' | base64 -d)" \
  --env="PGUSER=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.username}' | base64 -d)" \
  --env="PGPASSWORD=$(kubectl get secret $SECRET -n surfacesec -o jsonpath='{.data.password}' | base64 -d)" \
  -- psql -c "ALTER TABLE <table> DROP COLUMN IF EXISTS <column>;"

Prefer leaving expand changes in place. Only the application image needs to roll back to restore previous behavior.

Break-Glass Recovery

When the admin dashboard is inaccessible and pods are crash-looping:

Step 1: Set Recovery Key

# Set the RECOVERY_KEY environment variable on the API deployment
kubectl set env deployment/surfacesec-api -n surfacesec \
  RECOVERY_KEY="$(openssl rand -hex 32)"

# Note the key for step 2
kubectl get deployment surfacesec-api -n surfacesec \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="RECOVERY_KEY")].value}'

Step 2: Call Recovery Endpoint

# Reset admin access
curl -X POST \
  -H "X-Recovery-Key: <key-from-step-1>" \
  -H "Content-Type: application/json" \
  -d '{"email": "admin@surfacesecurity.dev", "password": "new-password"}' \
  https://surfacesec.example.com/api/v1/recovery/reset

Step 3: Remove Recovery Key

kubectl set env deployment/surfacesec-api -n surfacesec RECOVERY_KEY-

Postflight Verification

After any upgrade or rollback, verify:

# 1. All pods are ready
kubectl get pods -n surfacesec -o wide

# 2. API health
kubectl exec -it deployment/surfacesec-api -n surfacesec -- \
  wget -qO- http://localhost:8080/health

# 3. API readiness (checks DB connectivity)
kubectl exec -it deployment/surfacesec-api -n surfacesec -- \
  wget -qO- http://localhost:8080/ready

# 4. Ingestion health
kubectl exec -it deployment/surfacesec-ingestion -n surfacesec -- \
  wget -qO- http://localhost:9090/health

# 5. Check Redpanda consumer lag
kubectl exec -it surfacesec-redpanda-0 -n surfacesec -- \
  rpk group describe surfacesec-ingestion

# 6. Verify frontend is accessible
curl -sf https://surfacesec.example.com/login | head -1

# 7. Check for error logs in the last 5 minutes
kubectl logs deployment/surfacesec-api -n surfacesec --since=5m | grep -i error

Troubleshooting Upgrades

Pod CrashLoopBackOff After Upgrade

# Get the actual error
kubectl logs <pod-name> -n surfacesec --tail=50

# Common causes:
# - Database migration failure: check migration logs
# - Missing secret: kubectl describe pod <pod-name> -n surfacesec
# - Image pull error (air-gapped): verify images loaded in local registry

Migrations Hang on Startup

Migrations run automatically inside the API binary on every boot — there is no separate command and no timeout knob. A hang almost always means the migrator's session-level advisory lock is being broken by PgBouncer transaction pooling.

# Confirm migrations connect DIRECTLY to Postgres, not through the pooler.
# PG_MIGRATION_HOST must resolve to the direct Postgres host, never the
# surfacesec-pgbouncer ClusterIP.
kubectl get deployment surfacesec-api -n surfacesec \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="PG_MIGRATION_HOST")]}'

# If it points at the pooler, set the direct host (Helm value pgbouncer.migrationHost)
# and roll the deployment.

If a migration is genuinely slow (large table rewrite), it holds the advisory lock until it finishes; other replicas wait, then see no-change. Let it complete — do not kill the pod mid-migration, or you create the dirty state handled in Manual Migration Recovery above.

Image Pull Failure (Air-Gapped)

# Verify images exist in local registry
docker manifest inspect registry.internal/surfacesec/api:X.Y.Z

# Check imagePullSecrets
kubectl get secret surfacesec-registry -n surfacesec -o yaml

# Verify image pull policy
kubectl get deployment surfacesec-api -n surfacesec \
  -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}'
# Should be "IfNotPresent" for air-gapped

License Validation Failure

# Check license secret exists
kubectl get secret surfacesec-surface-license -n surfacesec

# Verify license key format
kubectl get secret surfacesec-surface-license -n surfacesec \
  -o jsonpath='{.data.license-key}' | base64 -d | head -c 20
# Should start with a valid license prefix

# Check API logs for license errors
kubectl logs deployment/surfacesec-api -n surfacesec | grep -i license

Breaking Changes by Version

Pulumi Stack Output Removal (`adminEmail`, `adminPassword`)

The adminEmail and adminPassword Pulumi stack outputs were removed. They previously exported a randomly generated credential for an admin user that the schema-init Job would insert into admin_users. That Job has been removed — schema is now applied by the API binary on startup, and the first admin user is created through the setup wizard on first visit to the frontend.

If your automation reads these outputs:

# Before: credential was a usable login
pulumi stack output adminPassword --show-secrets

# After: output no longer exists; use the setup wizard instead
# Navigate to the frontendURL stack output and complete the setup wizard.
pulumi stack output frontendURL

Any scripts that call pulumi stack output adminPassword will receive an error after upgrading to this version. Remove those references and provision the initial admin through the setup wizard.

Pre-Upgrade Checklist​

Standard Upgrade (Connected)​

Via Admin Dashboard​

Via Helm​

Via Update Bundle​

Air-Gapped Upgrade​

Step 1: Prepare on Connected Machine​

Step 2: Transfer to Air-Gapped Environment​

Step 3: Load and Deploy​

Database Migration Handling​

Checking Migration Status​

Manual Migration Recovery​

Rollback Procedures​

Automatic Rollback​

Manual Rollback via Admin Dashboard​

Manual Rollback via Helm​

Manual Rollback via kubectl​

Database Rollback​

Break-Glass Recovery​

Step 1: Set Recovery Key​

Step 2: Call Recovery Endpoint​

Step 3: Remove Recovery Key​

Postflight Verification​

Troubleshooting Upgrades​

Pod CrashLoopBackOff After Upgrade​

Migrations Hang on Startup​

Image Pull Failure (Air-Gapped)​

License Validation Failure​

Breaking Changes by Version​

Pulumi Stack Output Removal (adminEmail, adminPassword)​