Skip to content

Data Retention Safety — Design Spec

Date: 2026-09-20 Status: Draft Context: UAT customer hit disk-full crashes twice — Kafka (10Gi PVC, no retention defaults) then Elasticsearch. Multiple support emails to diagnose and fix. The system ships with no storage safety net: no Kafka retention defaults, ES disk watermarks disabled, no storage visibility in the dashboard.

Goal

Make data retention foolproof: the system should never crash from out-of-space, regardless of what PVC size the customer provisions. Retention must be sized relative to available storage, not hardcoded to an assumed disk size.

Non-Goals

  • Changing default PVC sizes (customers may already have running installs)
  • Implementing ILM/rollover in ES (retention.py's delete_by_query approach is simpler and sufficient)
  • External alerting integrations (webhook, email, PagerDuty) — dashboard warnings are sufficient for now

1. Kafka — Safe Retention Defaults

Problem

values.yaml ships zero log.retention.* entries. Every install inherits Kafka's built-in defaults: unlimited size retention (log.retention.bytes=-1), 7-day time retention (log.retention.hours=168), and 1 GiB segments (log.segment.bytes=1073741824). On a 10Gi PVC, this fills the disk within days and corrupts KRaft metadata on crash.

Design

Add three retention settings to kafka.config in values.yaml:

Setting Value Rationale
log.retention.hours 72 3-day time window. Primary retention mechanism for low-volume deployments.
log.retention.bytes 157286400 (~150 MiB) Per-partition size cap. 48 partitions (8 topics x 6) x 150 MiB ≈ 7 Gi = 70% of 10Gi. Leaves 30% headroom for in-flight segments + KRaft metadata. Safety net for high-volume bursts.
log.segment.bytes 52428800 (50 MiB) Must be smaller than log.retention.bytes. Kafka only deletes closed segments — if a single segment exceeds retention.bytes, nothing is ever deleted. 50 MiB gives ~3 segments per partition before retention kicks in.

Files changed:

  • deploy/openshift/logster/values.yaml — add the three settings to kafka.config, with comments documenting the formula: "48 partitions x this value ≈ 70% of storageSize; adjust proportionally if you change storageSize or num.partitions"
  • deploy/openshift/logster/templates/NOTES.txt — add two warnings:
  • If kafka.storageSize < 20Gi: "Kafka storage is {size} — retention defaults are sized for 10Gi. Verify retention settings are proportional to your disk."
  • Reminder that log.retention.bytes and log.segment.bytes must be passed with --set-string on helm upgrade — Helm converts large integers to scientific notation, which Strimzi silently rejects.
  • deploy/docker-compose.yml — add matching KAFKA_LOG_RETENTION_HOURS=72, KAFKA_LOG_RETENTION_BYTES=157286400, KAFKA_LOG_SEGMENT_BYTES=52428800 to the Kafka environment block.

Upgrade path

Existing installs using helm upgrade --reuse-values will NOT pick up the new defaults (--reuse-values preserves the old values.yaml, which had no retention entries). The NOTES.txt warning should call this out: "If upgrading from a version without Kafka retention defaults, add the retention settings explicitly via --set-string."

2. Elasticsearch — Watermarks + Emergency Retention Sweep

Problem

init.py explicitly disables ES disk watermarks (cluster.routing.allocation.disk.threshold_enabled: false). retention.py runs every 24h with time-based-only deletion. A burst can fill 15Gi before the next cleanup cycle. When ES fills its disk, data files corrupt — total loss, not graceful degradation.

Design — Three Layers

2a. Re-enable disk watermarks in init.py

Replace disk.threshold_enabled: false with explicit watermark settings:

Setting Value Effect on single-node cluster
cluster.routing.allocation.disk.watermark.low 85% Blocks new shard allocation — harmless since our indices are created once at init, not dynamically.
cluster.routing.allocation.disk.watermark.high 90% Tries to relocate shards — no-op on single node, harmless.
cluster.routing.allocation.disk.watermark.flood_stage 95% Sets indices read-only. Circuit breaker — prevents writes that would corrupt the disk. Vector and inference buffer and retry when ES recovers.

On a single-node cluster only the flood stage actively fires. That is exactly the protection needed: ES goes read-only instead of filling the disk and corrupting.

2b. Emergency sweep in retention.py

Currently retention.py sleeps for RETENTION_INTERVAL_HOURS (24h) between cycles. Change it to poll disk usage on a shorter interval and react when storage is under pressure.

Normal mode (disk < 80%): Check disk usage every 30 minutes (one _cluster/stats call — cheap). Run the scheduled retention cycle at the configured interval (default 24h) as today.

Emergency mode (disk >= 80%): 1. Log a WARNING with disk percentage. 2. Halve the configured retention windows (e.g., 30d events -> 15d, 90d inferences -> 45d, 14d graphs -> 7d). 3. Run delete_by_query with the halved windows. 4. Re-check disk after each index cleanup — stop once usage drops below 75% (the target). 5. If still above 80% after halving, halve again and repeat.

Critical mode (disk >= 90%): 1. Log a CRITICAL with disk percentage. 2. Delete oldest data in 1-day chunks starting from the oldest timestamp, regardless of configured retention windows. 3. Continue until usage drops below 75%. 4. Log exactly what was deleted: index name, date range, document count.

Auto-recovery from flood stage: If ES hit the 95% flood stage and indices went read-only (index.blocks.read_only_allow_delete set), retention.py should detect this after freeing space and automatically clear the block with PUT /<index>/_settings {"index.blocks.read_only_allow_delete": null}. This avoids the common ES pitfall where disk frees up but indices stay locked until someone manually clears it.

Customer responsibility note: The emergency sweep prevents crashes, not data loss. If storage is undersized for the configured retention window and ingest rate, the sweep will shorten effective retention to keep the system alive. Sizing the PVC to match holding requirements is the customer's responsibility.

2c. Configuration in values.yaml

elasticsearch:
  retention:
    events: 30                      # days
    inferences: 90                  # days
    graphs: 14                      # days
    stripRawEventAfter: 7           # days
    intervalHours: 24               # normal retention cycle
    checkIntervalMinutes: 30        # disk usage poll interval
    emergencyThresholdPercent: 80   # start emergency sweep
    criticalThresholdPercent: 90    # start aggressive deletion
    targetPercent: 75               # sweep until disk drops below this

Files changed:

  • deploy/elasticsearch/init.py — replace disk.threshold_enabled: false with the three watermark settings.
  • deploy/elasticsearch/retention.py — add disk usage polling, emergency sweep logic, flood-stage auto-recovery.
  • deploy/openshift/logster/values.yaml — add the retention threshold configuration under elasticsearch.retention.
  • deploy/openshift/logster/templates/elasticsearch/retention-deployment.yaml — pass the new env vars from values.
  • deploy/docker-compose.yml — add matching env vars to the elasticsearch-retention service.

3. Dashboard Storage Warning Banner

Problem

The System Health tab already shows storage insights (ES index sizes, disk usage). But users only see it when they navigate to that tab — by then it's often too late.

Design

Add a persistent warning banner at the top of the dashboard, visible on every page (not just System Health), that appears when storage thresholds are crossed. Similar to how AWS/Grafana/GitHub surface account-level warnings as a dismissible (but recurring) banner above the main navigation.

  • < 70% disk usage: No banner. Everything is normal.
  • >= 70% (warning): Amber banner appears at the top of every page:

    "Storage warning — Kafka / Elasticsearch disk usage at {n}%. Consider expanding storage or reviewing retention settings."

  • >= 85% (critical): Red banner, non-dismissible:

    "Storage critical — Kafka / Elasticsearch disk at {n}%. Data retention may be shortened automatically to prevent data loss."

The banner is dismissible at warning level (user can close it, reappears on next poll if still above threshold). At critical level it stays pinned — you can't dismiss a fire alarm.

Data source

The dashboard server already queries ES and Prometheus for System Health. The storage route polls on a short interval (every 60s) and the client fetches the status alongside existing health data. When any storage component crosses a threshold, the banner state is included in the response.

  • ES disk: _cluster/stats → nodes.fs.total_in_bytes / nodes.fs.available_in_bytes (dashboard server already has ES connection)
  • Kafka disk: Prometheus kubelet_volume_stats_* metrics (dashboard server already queries Prometheus for health metrics)

Thresholds

Configurable via dashboard env vars (following the existing pattern like ATTACK_PROB_THRESHOLD):

  • STORAGE_WARNING_THRESHOLD=70
  • STORAGE_CRITICAL_THRESHOLD=85

Files changed

  • services/dashboard/server/src/routes/storage.js — storage route (file already exists as untracked); add threshold comparison logic that returns a banner state (ok / warning / critical + message + component name + percentage).
  • services/dashboard/server/src/routes/index.js — wire the route.
  • services/dashboard/server/src/config.js — add threshold env vars.
  • services/dashboard/client/src/components/common/StorageBanner.jsx — new component: a top-of-page banner (amber/red, dismissible/pinned) rendered in the app shell above the main content area.
  • services/dashboard/client/src/App.jsx — render StorageBanner at the top level so it appears on every view.
  • services/dashboard/client/src/components/health/StorageSection.jsx — already exists as untracked; continues to show detailed storage info in the System Health tab.

Summary

Layer Component What it does When it acts
Prevention Kafka retention defaults Size-based + time-based retention in values.yaml Always — from first install
Circuit breaker ES disk watermarks Indices go read-only at 95% Before disk corruption
Self-healing ES emergency sweep Accelerated retention at 80%, aggressive delete at 90% Under storage pressure
Auto-recovery ES flood-stage clear Removes read-only block after space is freed After emergency sweep
Visibility Dashboard warning banner Amber/red banner at top of every page when thresholds crossed On every dashboard view
Deployment-time NOTES.txt warnings PVC size + --set-string reminders Every helm install/upgrade