Data Retention and TTL Configuration
ClickHouse uses TTL (Time-To-Live) rules to automatically expire old data. Without TTLs, tables grow without bound and will eventually exhaust storage. The following retention tiers align with SOC 2, ISO 27001, PCI DSS 10.7, and NIST 800-53 AU-11 requirements.
Retention Tiers
| Tier | Retention | Rationale | Data Categories |
|---|---|---|---|
| Audit / Security Incident | 730 days (2 years) | SOC 2, ISO 27001, PCI DSS 10.7, NIST 800-53 AU-11, GDPR Article 30 | Credential attempts, authentication flows, alerts, credential reuse, phishing detection, AITM detection |
| Operational Telemetry | 365 days (1 year) | Sufficient for trend analysis and asset inventory; not compliance-critical | Technology fingerprints, extension events, extension inventory snapshots, extension version changes |
| High-Volume Ephemeral | 180 days (6 months) | Navigation noise and clipboard events -- analytics only, no compliance mandate | Navigation events, clipboard threat events |
| Verification Audit | 730 days (2 years) | Compliance audit trail for identity verification | Verification events, daily code verification attempts, OTP verification attempts, OTP lifecycle events |
| Cache | Self-expiring | TTL on expires_at column | Threat intelligence cache |
Storage Projections
Maximum ClickHouse data at steady state with all TTLs active:
| Extensions | Navigation (180d) | Audit (730d) | Operational (365d) | Total Max |
|---|---|---|---|---|
| 10,000 | ~50 GB | ~200 GB | ~50 GB | ~300 GB |
| 25,000 | ~125 GB | ~500 GB | ~125 GB | ~750 GB |
| 60,000 | ~300 GB | ~1.2 TB | ~300 GB | ~1.8 TB |
These projections assume typical browsing patterns and default event generation rates. Actual storage usage depends on browsing activity, policy configuration, and the number of monitored authentication endpoints.
Compliance Alignment
| Standard | Requirement | Surface Security Coverage |
|---|---|---|
| SOC 2 | Audit log retention | 2-year retention for all security events |
| ISO 27001 | Event logging and monitoring | Immutable audit trails in ClickHouse |
| PCI DSS 10.7 | Audit trail history | 2-year retention exceeds the 1-year requirement |
| NIST 800-53 AU-11 | Audit record retention | Configurable per compliance needs |
| GDPR Article 30 | Records of processing activities | Credential and authentication event records |
Adjusting Retention
Retention periods can be adjusted based on your organization's compliance requirements and storage capacity. Contact your Surface Security account team for guidance on modifying TTL configurations for your deployment.