Elasticsearch Retention on ILM — Design Spec

Status: Approved (revision 2; independent review findings addressed, see §16; decisions resolved, see §15) Supersedes: §2 of 2026-09-20-data-retention-safety-design.md (emergency / critical / flood-stage ladder) Targets: Docker Compose (dev + OVA appliance) and the OpenShift Helm chart (customer, air-gapped)

1. Goal

Replace the escalating retention ladder with two simple rules:

Age rule. Data older than its retention period is deleted. Elasticsearch enforces this itself through Index Lifecycle Management (ILM), so Logster has no code for it.
Disk rule. While disk usage is above the target (default 75%), delete the oldest whole index until usage is back under target. Two limits apply: never delete data newer than a minimum age (default 24 h), and never delete an index that is currently being written to.

The system must stay up under every failure mode. Elasticsearch must never be pushed into corruption or a crash loop, and the retention worker must never exit on bad input. All of this runs on the stock basic-license Elasticsearch 9 with no new images and no network access beyond the cluster, so it works air-gapped.

2. Non-goals

Changing Kafka retention, the dashboard storage banners (STORAGE_WARNING_THRESHOLD/STORAGE_CRITICAL_THRESHOLD) or the watermark values (85/90/95%).
Multi-node Elasticsearch, hot/warm/cold tiers, snapshots or searchable snapshots.
Reindexing or moving existing data. Legacy data stays where it is and ages out (§8).
Changing the verdict path, schemas (NormalizedEvent, InferenceResult, Alert) or anything GNN/LLM related.

3. Why the current design has to go

These were observed on the dev box on 2026-09-25 and in the code audit:

Problem	Evidence
Halving retention is an indirect control. It jumps 30→15→7→3→1 days no matter how much space is actually needed.	`retention.py:308-338`. On the dev box the emergency sweep ran 3× in 2.5 h, each time cutting events/graphs to 1 day, and disk stayed at 88%.
Deleting documents doesn't free disk until segments merge, so decisions are made on stale numbers and the worker over-deletes.	`logster-graphs` held 3,571 live docs next to 7,722 deleted-but-unmerged docs (3.8 GB).
Disk % includes non-Elasticsearch usage. Elasticsearch was 20.5 GB of 383 GB used, so deleting all of it could not have reached 75%.	`_cluster/stats` `nodes.fs` measures the whole filesystem. On the OVA it is the root disk (single LVM disk, `packer/files/logster.ovf.template:14-19`).
Four code paths (nightly, emergency, critical, flood), each with its own guards.	573 lines. The OpenShift copy has drifted and lacks the critical-sweep guards (`deploy/openshift/logster/files/elasticsearch/retention.py`).
A bulk `_update_by_query` rewrite (raw_event strip) runs on the live index.	`retention.py:425-452`

Deleting a whole index is the one operation Elasticsearch guarantees will free space immediately, including at flood stage, where the read_only_allow_delete block still permits it. Every item in the table goes away once the unit of deletion changes from "documents" to "whole indices".

4. Decisions

#	Decision	Rationale
D1	Index layout: ILM rollover behind write aliases for `events`, `inferences`, `graphs`.	This is Elastic's intended pattern for time-series data. It gives native age-based deletion and instant space reclaim.
D2	Aliases, not data streams.	Data streams keep the first write for an `_id` and reject later writes with 409. `InferenceResult` is re-published with the same `inference_id` after TTP enrichment (`service.py:806-810`), and Kafka redelivery re-indexes the same ids. Elastic's docs say to use an alias with a write index when last-write-wins is needed.
D3	ILM, not data stream lifecycle (DSL).	DSL only works with data streams (ruled out by D2).
D4	Disk rule measures whole-disk %, with a minimum-age floor (chosen by the user).	Elasticsearch fails at flood stage based on whole-disk %. The floor stops a non-Elasticsearch consumer (Kafka, images, logs) from wiping recent evidence.
D5	Oldest data first, across all three families.	This is the literal "delete oldest data" rule. Under pressure every family converges to the same history horizon. It is easy to explain and predictable.
D6	Readers use wildcard patterns (`logster-events`); writers use write aliases* (`logster-events-write`).	Upgraded installs keep the old concrete index named `logster-events`, and an alias can't share a name with an index. A wildcard covers the legacy index and all rollover indices with no data movement.
D7	Legacy concrete indices are adopted, never moved. They get a delete-only ILM policy whose age is measured from their newest document (`index.lifecycle.origination_date`), plus `index.lifecycle.indexing_complete: true`.	No reindexing, no write blocks, no downtime. They age out naturally and are also the first candidates for the disk rule. The flag releases a bare name auto-created while rolled back (§8.3): it picked up the family template's policy, and ES keeps running the cached hot/rollover step after a policy switch, so without the flag it never reaches the legacy delete phase (verified on ES 9.5.4). On other legacy indices it is a no-op.
D8	Remove the `raw_event` strip, and add `index.codec: best_compression` to the three templates.	The strip is a bulk rewrite of live data, the kind of heavy background mutation this redesign removes. It saved about 28% of event JSON on a 300-doc dev sample. `best_compression` is Elastic's native way to shrink stored `_source` with no rewrites. The net saving is measured in the integration test (§12.2).
D9	Auto-create guard. Elasticsearch refuses to auto-create the write-alias names or the bare legacy names.	If the alias is missing, a write to its name would otherwise auto-create a concrete index that permanently squats on the alias name. With the guard, that write fails as a per-item 404, and self-heal (D10) restores the alias within one check. Vector does not retry per-item 404s (verified in Vector 0.52 `src/sinks/elasticsearch/retry.rs`: only 429 and 5xx items are retried). Events in flight while the alias is missing are therefore dropped. This state cannot occur at startup (D11), and nothing in Logster deletes an alias or a write index, so it can only follow a manual deletion.
D10	One declarative `converge()` function, used by both init and the worker, owns the layout: policies, templates, legacy adoption, aliases.	Idempotent and self-healing. It removes the OpenShift ordering race: the init Job is a post-install hook that runs after the writers start (`init-job.yaml:9-12`).
D11	Vector waits for the write aliases before starting (OpenShift initContainer; Compose already gates on init). Compose `inference` stops depending on `elasticsearch-init`, which matches OpenShift, where inference has never been gated.	Vector is the main writer. It must not write before the layout exists. Inference is the verdict path and must not depend on Elasticsearch to start. Under D9, a graph snapshot written before the alias exists fails instead of auto-creating a mis-mapped index, and is counted and dropped. This is the reason the Compose gate existed, and it no longer applies.
D14	`request_retry_partial: true` on both Vector ES sinks.	Per-item 429s (the flood-stage `read_only_allow_delete` block) and 5xxs (e.g. a write index whose shard can't be allocated above the high watermark) are then retried instead of silently dropped. Retries only resend the failed items. `id_key` makes a retry an idempotent overwrite, so no duplicates. Retries are unbounded by default, so Vector's buffer fills and applies backpressure to the Kafka source, which stops consuming. Kafka retention (`KAFKA_LOG_RETENTION_HOURS=72`, `RETENTION_BYTES=146800640` per partition) is then the limit on how long a disk emergency can last without loss.
D12	Rollover size cap is computed from disk size: `clamp(1% of filesystem, 1 GB, 50 GB)` per primary shard, plus `max_age: 1d` for all three families.	Each index stays at or below about 1% of disk (disks under 200 GB get 1 GB), so the disk rule deletes in small steps. The index count stays bounded regardless of disk size: about 134 daily single-shard indices at the default 30/90/14 days, plus about 75 size-capped ones at most. That is well under the 1,000-shards-per-node default. This count is accepted deliberately: the same rollover age for every family keeps retention semantics uniform. There is no knob to misconfigure.
D13	Worker runs as a single instance: `strategy: RollingUpdate` with `maxSurge: 0` / `maxUnavailable: 1`, and the worker exits on SIGTERM.	Two workers could both plan deletions and delete twice as much. `type: Recreate` was rejected after testing on CRC: switching an existing Deployment to it fails `helm upgrade` under Helm 4 (server-side apply), with or without `rollingUpdate: null`. The RollingUpdate form upgrades cleanly with Helm 3.8.2, 3.22.0 and 4.2.4.

5. Architecture

                          ┌──────────────────────────────────────────┐
  Vector ──bulk index──▶  │ logster-events-write ─▶ logster-events-0000NN (write) │
  Vector ──bulk index──▶  │ logster-inferences-write ─▶ ...-0000NN (write)        │
  graph_sink ──bulk────▶  │ logster-graphs-write ─▶ ...-0000NN (write)            │
                          │                                                        │
  dashboard / api /       │ logster-events*  = legacy logster-events (if any)     │
  hydration  ──search──▶  │                   + logster-events-000001..0000NN     │
                          └──────────────────────────────────────────┘
        ▲ ILM (in Elasticsearch): rollover at 1d or size cap; delete at N days
        ▲ retention worker (every 5 min): converge() → ILM error report → disk rule → capacity forecast
        ▲ init (one-shot at deploy): watermarks → converge() → wait green

5.1 Components and responsibilities

Component	Owns	Does not own
`deploy/elasticsearch/layout.py` (new, stdlib only)	`converge()`: ILM policies, index templates, legacy adoption, write aliases and bootstrap indices. `wait_for_aliases()` for the Vector initContainer: waits until every write alias has a writer; a squatted alias name counts as ready (writes land there) and is logged as `ERROR` once.	Deleting data
`init.py`	Watermark cluster settings, then `converge()`, then a bounded wait (60 s) for the write indices to be allocated. Exits non-zero only when Elasticsearch is unreachable or rejects credentials, or when a PUT of cluster settings, a policy, a template or a bootstrap index is rejected. Those are misconfigurations that break the pipeline anyway. Before failing, init makes up to 5 `converge()` attempts, 10 s apart (~50 s); a rejected `/_cluster/reroute` is only a warning; and a failed GET of a policy, template or alias, a failed rollover-index listing (`_cat/indices`) while bootstrapping, a missing template file, or a crash in the policy, template or write-alias step counts as the "unreachable / misconfiguration" class, because exiting 0 with a missing write alias would let Compose Vector write into a 404 that it never retries. An allocation timeout (e.g. disk above the 90% high watermark), a squatting index, or a legacy-adoption failure logs WARN/ERROR and exits 0: on Compose/OVA, init gates Vector and never re-runs until the next boot, so failing there would block ingestion even after the worker has freed space.	Retention decisions
`retention.py` (rewritten)	The loop: `converge()` (repair) → ILM error report → disk rule → capacity forecast. Never exits.	Age-based deletion (ILM does it)
Elasticsearch ILM	Rollover, age-based deletion of rollover indices and adopted legacy indices	Disk-based deletion
`capacity.py` (simplified)	Forecast of whether the configured retention fits under target	Deleting anything

5.2 Names

Family	Write alias	Rollover indices	Read pattern	Legacy index (upgrades only)	ILM policies
events	`logster-events-write`	`logster-events-000001`…	`logster-events*`	`logster-events`	`logster-events`, `logster-events-legacy`
inferences	`logster-inferences-write`	`logster-inferences-000001`…	`logster-inferences*`	`logster-inferences`	`logster-inferences`, `logster-inferences-legacy`
graphs	`logster-graphs-write`	`logster-graphs-000001`…	`logster-graphs*`	`logster-graphs`	`logster-graphs`, `logster-graphs-legacy`

Managed-index regex (the only names the worker may ever delete): ^logster-(events|inferences|graphs)(-\d{6,})?$.

logster-graphs* does not match logster-graph-whitelists (verified: graphs vs graph-). logster-capacity, logster-dashboard-auth and logster-graph-whitelists are unmanaged and unchanged.

6. Index layout

6.1 ILM policies

Family policy (N = RETENTION_<FAMILY>_DAYS, S = the computed size cap from D12):

PUT _ilm/policy/logster-events
{ "policy": {
    "_meta": { "managed_by": "logster", "logster_hash": "<sha256 of the canonical desired body>" },
    "phases": {
      "hot":    { "min_age": "0ms", "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "<S>" } } },
      "delete": { "min_age": "<N>d", "actions": { "delete": {} } }
} } }

Legacy policy: a delete phase only, no rollover.

PUT _ilm/policy/logster-events-legacy
{ "policy": { "_meta": { "managed_by": "logster", "logster_hash": "<sha256>" },
  "phases": { "delete": { "min_age": "<N>d", "actions": { "delete": {} } } } } }

N = 0 means "keep forever" (the current contract): the delete phase is omitted, and the disk rule still caps usage.
S is computed as clamp(floor(total_fs_bytes × 1%), 1 GB, 50 GB) and rounded down to whole GB, so small filesystem fluctuations never cause the policy to be rewritten. If _cluster/stats is unavailable: an existing policy is left as is, and a first bootstrap uses 5gb.
Change detection uses _meta.logster_hash, the SHA-256 of the canonical JSON (sorted keys) of the body Logster would send. Elasticsearch stores _meta verbatim but normalizes the rest of the policy (it adds hot.min_age: "0ms" and delete.delete_searchable_snapshot: true, verified on the dev cluster), so comparing whole bodies would find a difference on every run. converge() PUTs only when the stored hash differs from the desired hash, so a converged cluster sees zero PUTs and no policy-version churn.
ILM measures min_age from rollover time (Elastic docs). An index therefore holds data for between N and N+1 days plus the ILM poll interval (10 min default, verified on the live cluster). The docs must say "at least N days, at most N+1".
Empty indices are never rolled over (indices.lifecycle.rollover.only_if_has_documents: true, verified default).

6.2 Index templates (additions)

Each of the three *.template.json files gets (composable-template structure: _meta is top-level, beside template):

{
  "index_patterns": ["logster-<family>*"],
  "priority": 200,
  "_meta": { "managed_by": "logster", "logster_hash": "<sha256 of the canonical desired body>" },
  "template": {
    "settings": {
      "index.lifecycle.name": "logster-<family>",
      "index.lifecycle.rollover_alias": "logster-<family>-write",
      "index.codec": "best_compression"
    },
    "mappings": { "properties": { "@timestamp": { "type": "date" } } }
  }
}

The existing keys of each template (number_of_shards, number_of_replicas, refresh_interval, the graphs mappings) are kept inside template. The same hash-based change detection is used for all six templates, including the unmanaged capacity, dashboard-auth and graph-whitelists ones.

logster-events also maps "timestamp": {"type": "float"}.

Why explicit mappings: after a rollover the new write index is empty until its first document arrives. Sorting across logster-events* on a field that one index hasn't mapped fails for that shard. The sort sites are listed in §9: @timestamp in the dashboard/api, and timestamp in hydration (hydration.py:308). The types match what dynamic mapping produces today (verified on the live cluster: @timestamp date, events timestamp float), so there are no cross-index type conflicts.

Verified on 9.5.4: a sort across two indices where one hasn't mapped the field returns HTTP 200 with _shards.failed: 1. That is harmless for results but noisy. The only free-form sort is the dashboard /inferences ?sort= parameter (routes/inferences.js:15,26), which the client never sends. It gets an allowlist with known types and matching unmapped_type: @timestamp→date, attack_prob/confidence/lc_attack_prob/lc_confidence→float. Any other value → 400. The worker's own queries only aggregate on @timestamp, which is always mapped.

Existing settings (shards 1, replicas 0, refresh, graphs mappings) are unchanged.

6.3 Auto-create guard

This is an Elasticsearch startup argument, kept identical in deploy/docker-compose.yml and templates/elasticsearch/statefulset.yaml (CLAUDE.md invariant):

-Eaction.auto_create_index=-logster-events,-logster-inferences,-logster-graphs,-logster-*-write,+*

Patterns are evaluated in order and the first match wins.
Rollover and bootstrap indices are created explicitly, so they are unaffected.
logster-capacity, logster-dashboard-auth, logster-graph-whitelists and system indices still auto-create through +*.
To verify in the integration test: that ES 9.5.4 accepts this dynamic setting as a node startup argument. If it does not, init sets it as a persistent cluster setting before converge(). The OpenShift window before init then relies on D11 alone.

7. Behavior

7.1 `converge()` — declarative, idempotent, safe to run concurrently

Order matters: policies before templates before indices.

Policies. For each of the 6 policies: GET it. PUT only if it is missing or its _meta.logster_hash differs from the desired hash (§6.1).
Templates. For each of the 6 template files: GET it. PUT only if it is missing or its _meta.logster_hash differs.
Legacy adoption. For each family: if a concrete index named exactly logster-<family> exists:
compute max(@timestamp) (if there are no docs, use the index creation_date);
if its index.lifecycle.name isn't the legacy policy, its origination_date differs from that max, or index.lifecycle.indexing_complete isn't true, PUT all three: index.lifecycle.name=logster-<family>-legacy, index.lifecycle.origination_date=<max>, index.lifecycle.indexing_complete=true.

The origination date is re-checked on every run. During a rolling upgrade the old Vector pod may keep writing to the legacy index for a few seconds, so the date has to follow its newest document. indexing_complete is for a bare name auto-created while rolled back (§8.3), which the new family template put in the family policy's rollover step; after the policy switch ES keeps running that cached step, which errors forever without the flag (D7; verified on ES 9.5.4 and in the integration suite). 4. Write alias. For each family: GET /_alias/logster-<family>-write. - No alias: create logster-<family>-<next> with aliases: {logster-<family>-write: {is_write_index: true}}. The create request also sets the two lifecycle settings explicitly, so the index is correctly managed even if a stale template was used. <next> is 1 + the highest existing 6-digit suffix (000001 on a fresh install). - Alias but no write index: same as above. This is the "someone deleted the write index" repair (verified: writes fail with no write index is defined). - Write index present: make sure its index.lifecycle.name and rollover_alias are set, and PUT them if missing. - A concrete index squats on the alias name: this can't be repaired automatically without moving data. Writes still succeed (they land in the squatter), so ingestion keeps working; only lifecycle management is lost. Log ERROR with the exact manual fix. init exits 0 (§5.1), and the worker repeats the error each check. The Vector initContainer (layout.py wait) treats a squatted name as ready, since writes succeed there, and logs the same ERROR once; waiting for an alias converge() will never create would only block ingestion. With D9 in place this can only happen if the guard was removed. - Race (init and worker bootstrapping at the same moment): two processes may pick the same or different <next>. The loser gets either resource_already_exists_exception or illegal_argument_exception ("more than one write index"). After any 4xx on the bootstrap create, re-read the alias. If it now has a write index, that is success. Only a 4xx with the alias still missing a write index counts as a failure. 5. Orphaned former write indices. If an alias was deleted by hand, the index that was the write index at that moment never completes ILM's rollover step. It sits in ERROR ("rollover alias does not point to index") and never reaches the delete phase. For every managed rollover index that is not the current write index and whose ILM step is still in hot/rollover: PUT index.lifecycle.indexing_complete: true (Elastic's documented switch to skip rollover), and POST /<index>/_ilm/retry if it is in ERROR. A 400 from the retry counts as success: ES auto-retries check-rollover-ready, so the index can leave ERROR between the explain and the retry, and with the flag set ILM moves on by itself. The integration test must prove the orphan then reaches delete (§12.2 scenario 9).

converge() never deletes anything. Empty non-write orphan indices (docs.count 0) are accepted: they use no meaningful space, and ILM deletes them once they reach the delete-phase age.

7.2 Worker loop

startup: parse config (§10), log the effective config, WARN for each removed env var that is still set
loop forever (first iteration immediately, then every RETENTION_CHECK_INTERVAL_MINUTES):
    if Elasticsearch is unreachable: log, retry in 30 s (ES restarts on every install/upgrade;
        Vector's initContainer waits for the aliases this loop creates), continue
    each step runs in its own try/except, so one failure never skips the others:
      1. converge()
      2. ILM error report: GET logster-*/_ilm/explain?only_errors=true → WARN per index with the failed step
      3. disk rule (§7.3)
      4. capacity forecast (§11), best effort

A catch-all around each iteration means the process never exits on a runtime error.

7.3 Disk rule

usage = _cluster/stats nodes.fs                 # unknown/partial/failed nodes → skip this check
used  = total_in_bytes - available_in_bytes     # same definition as today, the dashboard
                                                # (storage.js:92) and ES's watermark math
                                                # ("available", not "free": they differ by reserved blocks)
if used/total*100 <= target: return
excess = used - target/100 * total

indices = _cat/indices for the 3 read patterns  # name, store.size (bytes), docs.count
writers = _alias/logster-*-write                # names with is_write_index=true
newest  = one search over the 3 patterns, size 0:
            terms(_index, size: 10000) → max(@timestamp)
            filter(@timestamp >= now-<floor>h) → terms(_index, size: 10000)   # "recent" set, ES's clock
          # fail closed (skip this check) if: shards.failed > 0, timed_out, or
          # sum_other_doc_count > 0 on either terms agg (a truncated "recent" set
          # would silently make recent indices look eligible)

candidates = indices where
                 name matches the managed regex
             and name not in writers
             and docs.count > 0
             and name in newest                 # an index with no max(@timestamp) is never eligible
             and name not in recent
             sorted by (newest asc, name asc)

plan = shortest prefix of candidates whose store.size sum >= excess
       (if the sum of all candidates is still < excess: plan = all candidates, then WARN "floor reached")
if plan is empty: WARN "disk <pct>% > target <t>%, nothing older than <floor>h to delete"; return

re-read writers; drop any plan entry that became a write index
for idx in plan: DELETE /<idx>                   # exact name; 404 = already gone (ILM) = success
    log "retention: deleted <idx> (<size>, newest <ts>) — disk <pct>% > target <t>%"

Properties: - The plan is computed up front from exact index sizes. It never reads disk usage in a loop, so there is no stale-reading over-deletion. The next check re-evaluates and corrects any estimation error. - Deletion is instant and works at flood stage. Elasticsearch releases read_only_allow_delete by itself once disk drops below the high watermark (90%). - The floor uses Elasticsearch's clock (now-<floor>h in the query), so worker/ES clock skew can't make it delete recent data. - Only names matching the managed regex can ever be deleted, and only by exact name (action.destructive_requires_name=true is the verified default, and the worker never sends wildcards to DELETE).

Known consequence (as chosen): on a shared disk where something other than Elasticsearch fills the disk, the rule deletes every Logster index older than the floor, and the disk may still stay above target. The floor bounds the damage, and every such check logs a WARN. On OpenShift the Elasticsearch PVC is dedicated (statefulset.yaml:80-87), so this case can only arise on Compose and OVA hosts.

8. Upgrade, migration and rollback

8.1 Fresh install (Compose, OVA, OpenShift)

converge() creates policies, templates and logster-<family>-000001 with write aliases. There is no legacy index. Readers' wildcards match only the rollover indices.

8.2 Upgrade from the current release (legacy concrete indices exist)

The new ES args (auto-create guard) apply when Elasticsearch restarts. That restart already happens today on any StatefulSet or Compose change.
converge() adopts logster-events/-inferences/-graphs under the legacy policies and bootstraps the write aliases. No data is moved, blocked or deleted.
New writers write to the aliases, and readers see legacy plus new data through the wildcard.
ILM deletes each legacy index once its newest document is older than the retention period. The disk rule may delete it earlier (it is the oldest data).

Switch-over duplicates: events buffered while Elasticsearch restarts can be written both to the legacy index and to the new rollover index (id_key deduplicates only within one index; CRC measured 468 at upgrade), so delivery is at-least-once: nothing is lost, and counts over that window are slightly inflated.

Legacy granularity: one legacy index holds up to N days. If the disk is over target at upgrade time, the disk rule drops the whole legacy index in one step, once its newest document is older than the floor. This is a one-time coarse step and must be stated in the release notes.

OpenShift ordering: - Helm updates all resources, and then the post-upgrade init hook runs. - The new Vector pod waits in its initContainer until the write aliases exist (a squatted alias name counts as existing, §7.1 step 4). - The new retention worker runs converge() at startup, so the aliases appear even before the hook runs. This means helm upgrade --wait (which waits for readiness before running post hooks) cannot deadlock. - Under the default RollingUpdate, the old Vector pod keeps writing to the legacy index until the new one is ready. Step 3 of §7.1 moves the legacy origination date along accordingly. - The switch-over loses no events. The old pod writes successfully to the existing legacy index, and the new pod does not start until the aliases exist, so no write ever hits a missing alias. Both pods share the Kafka consumer group, with end-to-end acknowledgements enabled (vector.yaml). The ES restart during the upgrade produces request-level 5xx/connection errors, which Vector retries already, plus per-item 429/5xx, which D14 now covers. The §12.4 upgrade test proves this by comparing counts before and after.

8.3 Rollback (`helm rollback` / previous OVA image)

No init hook runs on rollback (the chart has no post-rollback hook).
The previous release's writers write to concrete logster-events etc.:
On an upgraded cluster, the legacy index still exists, so writes land there.
On a cluster installed fresh on the new release, the auto-create guard is gone after rollback (the StatefulSet args revert), so the concrete index is auto-created.
The previous readers see only the concrete indices, so data written by the new release in rollover indices becomes invisible. It is still ILM-managed and ages out.
Nothing crashes, and there is no data corruption. History visibility is degraded until you roll forward again. Rolling forward again re-adopts the concrete index as legacy, including a bare name auto-created during the rollback under the family policy (indexing_complete releases it from the rollover step, §7.1 step 3).
Disk protection is reduced while rolled back. The previous retention.py only knows the three concrete names (MANAGED_INDICES), so its sweeps can't touch logster-*-0000NN. Only ILM, whose policies persist in Elasticsearch, ages those out. On a small disk this can reach flood stage. The release notes must say: roll forward promptly, and don't stay rolled back on a full disk.

9. Reader and writer changes

The inventory was verified file-by-file. Items marked ◆ are latent bugs today that this change fixes along the way, because they sit on the changed path.

Area	File	Change
Vector writes	`deploy/vector/vector.yaml:70,85`, `deploy/openshift/logster/files/vector/vector.yaml:70,85`	Defaults `logster-events-write`, `logster-inferences-write`
Graph writes	`libs/logster-common/logster_common/config.py:126` (`GraphPersistConfig.es_index`)	Default `logster-graphs-write`
◆ Graph writes	`services/inference/src/logster_inference/sinks/graph_sink.py:152-160`	Inspect `_bulk` `errors`/per-item status. Today item failures are counted as indexed, which would hide a missing alias.
Hydration reads	`deploy/service-config.yaml:71`, `config.py:150`	`es_index: "logster-events*"`
Dashboard reads	`services/dashboard/server/src/config.js:6-8`	`logster-events`, `logster-inferences`, `GRAPHS_INDEX` default `logster-graphs*`. All `es.search`/`count`/`msearch` sites work as they are.
◆ Dashboard graph by-id	`routes/graph.js:96-109`	Replace `es.get` + fallback with one `ids` search on `logster-graphs*`, sorted `@timestamp` desc, size 1. The current fallback queries `inference_id.keyword`, which can't match because the graphs template maps `inference_id` as keyword with `dynamic:false`.
Dashboard size-1 lookup	`routes/inferenceDetail.js:206-212`	Add `sort @timestamp desc`, so the newest copy wins if a rare post-rollover duplicate exists.
Dashboard storage	`routes/storage.js:19-36,94-101,247-250`, `client/.../StorageSection.jsx:105-109,198`	Group `_cat/indices` rows by family (legacy + rollover), summing docs and store. The UI keeps showing one row per family.
API graph by-id	`services/api/src/logster_api/whitelist_store.py:77-120`, `evidence.py:366,457`	`get_graph_doc`: `ids` search on `logster-graphs`, returning `_source`, `_id` and `_index`*. `update_graph_doc(index, id, partial)`: `_update` on the concrete `_index`. Evidence caching passes `_index` through.
API inference lookup	`whitelist_store.py:109-112`	Pattern default and `sort @timestamp desc`
API defaults	`whitelist_store.py:31-33`	`logster-graphs`, `logster-inferences`
Retention/capacity	`deploy/elasticsearch/retention.py`, `capacity.py`, `init.py`	Rewrite / simplify / converge (§5.1)
Templates	`deploy/elasticsearch/logster-{events,inferences,graphs}.template.json`	§6.2
e2e	`tests/e2e-openshift/lib.sh:30-31`	`logster-events`, `logster-inferences` (`_count` on a wildcard with no match returns 0, not 404)
Chart copies	`deploy/openshift/logster/files/elasticsearch/*`	Byte-identical copies, enforced by a test (§12.1)
Vector retries	both `vector.yaml` ES sinks	`request_retry_partial: true` (D14)
Dashboard sort	`routes/inferences.js:15,26`	`?sort=` allowlist with `unmapped_type` (§6.2)
Compose gating	`deploy/docker-compose.yml` `inference.depends_on`	Drop `elasticsearch-init` (D11)
ES args parity	`deploy/docker-compose.yml:83-89`	Add the D9 guard. Also add `-Expack.ml.enabled=false`, which the chart already has (`statefulset.yaml:42`) and Compose is missing, so the CLAUDE.md "identical arg list" invariant holds again. The drift test (§12.1) compares the two arg lists.
Hydration default	`libs/logster-ml/logster_ml/graph/hydration.py:116`	Default `logster-events*`. The engine always passes config; this is a default only.
API construction	`services/api/src/logster_api/main.py:59-63`	No change: it passes no index names, so the `whitelist_store.py` defaults apply.
Forecast contract tests	`services/dashboard/server/test/storage.test.js:71-77`, `capacity.test.js:26-37`	Drop `emergency_percent`/`strip_ratio`/`strip_raw_event_days`, add `target_percent` (§11)
Chart docs	`deploy/openshift/logster/README.md:286-288`, `templates/NOTES.txt:23-24`	Patterns, new knobs, no emergency sweep
Script docstring	`scripts/stress_test/producer.py:17`	Cleanup hint: `logster-events` → `logster-events*`
Unmanaged	`routes/storage.js:23` `logster-alerts`	Stays listed and unmanaged; nothing writes it

Known limitation: an _id re-written through an alias after that family's rollover creates a second copy in the new index. The two sources are the TTP re-publish (off in the shipped config: ttp.service_url: "") and Kafka redelivery. The window is the time between the first and second write compared with one rollover per day. The size-1 lookups prefer the newest copy, and list views may show the duplicate. Accepted, and documented.

10. Configuration

The Compose env and values.yaml retention: keys map 1:1, as today.

Env var	values.yaml	Default	Status	Validation (invalid → WARN + default)
`RETENTION_EVENTS_DAYS`	`eventsDays`	30	kept (now ILM)	int 0–3650; 0 = keep forever
`RETENTION_INFERENCES_DAYS`	`inferencesDays`	90	kept (now ILM)	int 0–3650
`RETENTION_GRAPHS_DAYS`	`graphsDays`	14	kept (now ILM)	int 0–3650
`RETENTION_TARGET_PERCENT`	`targetPercent`	75	kept	int 1–89 (must stay below the 90% high watermark)
`RETENTION_MIN_AGE_HOURS`	`minAgeHours`	24	new	int 1–720
`RETENTION_CHECK_INTERVAL_MINUTES`	`checkIntervalMinutes`	5 (was 30)	kept	int 1–1440
`RETENTION_STRIP_RAW_EVENT_DAYS`, `RETENTION_INTERVAL_HOURS`, `RETENTION_EMERGENCY_THRESHOLD_PERCENT`, `RETENTION_CRITICAL_THRESHOLD_PERCENT`	same keys	—	removed	WARN at startup if set: " `is no longer used (ILM retention); ignoring"`

Invalid values never crash the worker. This matches the direction of the current uncommitted _env_int change.
Both components get the days: the init Job and the Compose elasticsearch-init service receive the three *_DAYS variables from the same values/env source, because converge() writes the policies.
5-minute default interval: a check is a handful of GETs and deletes are cheap, so reacting faster to bursts costs almost nothing.

11. Observability

Worker logs (stdout, one line per event, as today): the effective config at startup; every converge repair; every deletion (index, size, newest timestamp, disk %); every "floor reached" warning; ILM errors.
Capacity forecast (capacity.py → logster-capacity/_doc/current → dashboard). Changes:
reads per family through the patterns;
drops the strip model (strip_ratio removed, and lib/capacity.js updated to match);
the provisioning warning compares against target_pct instead of emergency_pct.

The document contract is shared by capacity.py and lib/capacity.js only, and both change together. - Dashboard storage banners (70/85% of disk) are unchanged and remain the operator-facing alarm. - No new Prometheus metrics. The worker is a stdlib sidecar with no metrics endpoint today, and adding one is out of scope. It is noted as a follow-up.

12. Testing strategy (everything runs on the current stack before release)

12.1 Unit (mocked `request`, stdlib `unittest`, as today)

test_layout.py (new): each converge() branch:
fresh install; already converged → zero PUTs;
stale template/policy version → PUT;
legacy adoption, including the origination-date update;
missing alias; alias without write index; squatter; bootstrap race;
N = 0 omits the delete phase;
size-cap computation and clamps;
hash-based change detection: an ES-normalized GET body with a matching _meta.logster_hash → no PUT;
bootstrap race: illegal_argument_exception then a re-read showing a write index → success;
orphaned former write index → indexing_complete + _ilm/retry;
init exit codes: fatal only for the §5.1 cases; allocation timeout, squatter and adoption failure → exit 0.
test_retention.py (rewritten): the disk rule:
under target → no-op;
plan picks the oldest first and stops at the excess;
write index never planned; floor respected; empty indices skipped; non-managed names never planned;
more than 10 managed indices → every index appears in newest/recent (terms size); sum_other_doc_count > 0 → skip; an index missing from newest is never planned;
used = total − available (not free);
404 on DELETE = success; partial search → skip;
all candidates insufficient → delete all + WARN;
config validation and removed-var warnings;
step isolation (a converge failure doesn't stop the disk rule);
the loop never exits.
test_capacity.py: updated for patterns and no strip.
Drift test: deploy/openshift/logster/files/elasticsearch/* is byte-identical to deploy/elasticsearch/* for every file the chart ships. Same for files/vector/vector.yaml vs deploy/vector/vector.yaml, apart from the ES_PASSWORD line. The Elasticsearch -E arg list in deploy/docker-compose.yml must equal the one in templates/elasticsearch/statefulset.yaml.
API (services/api/tests): the whitelist_store graph lookup returns _index, and the update targets the concrete index (fake HTTP).
Inference: graph_sink counts per-item bulk failures.
Dashboard (node --test): storage grouping (legacy + rollover rows → one family row), and the graph by-id ids search.

12.2 Integration against the real image (new)

deploy/elasticsearch/integration/: a throwaway container from the same logster-elasticsearch image shipped air-gapped. - Unique docker run --name, never Compose (avoids the project-name collision). - Data directory on a size-limited tmpfs (e.g. 1 GiB of host RAM), mounted as --tmpfs /usr/share/elasticsearch/data:rw,size=1g,uid=1000,gid=0. A root-owned tmpfs reproduces the node-lock failure the Dockerfile warns about. _cluster/stats nodes.fs measures the data path's filesystem, so disk % is real and dominated by Elasticsearch. - Scenarios: 1. fresh converge(); the second run is a no-op; 2. auto-create guard: a write to a missing alias → error, and no concrete index is created; 3. legacy adoption on seeded concrete indices; 4. rollover through the policy (small max_docs test override) keeps new writes on the new index and reads across both; 5. empty write index + wildcard sort on @timestamp/timestamp → no shard failures; 6. disk rule with seeded indices and a low target → oldest non-write indices are deleted, the floor is honored, the write index survives; 7. real flood stage (watermarks lowered in the test cluster) → block appears, worker deletions succeed, block auto-releases; 8. ILM age deletion with a short poll interval and origination_date in the past; a re-PUT of origination_date on a live index takes effect (the §7.1 step 3 re-check depends on it); 9. self-heal after the write index/alias is deleted, and the orphaned former write index reaches the delete phase after indexing_complete + retry; 10. init ∥ worker concurrent converge() (both exit cleanly, exactly one write index); 11. best_compression size saving on a fixed corpus (recorded in the PR); 12. Vector (timberio/vector:0.52.0-distroless-static, the shipped image) with request_retry_partial: true writing through a real flood-stage block: a known number of events is produced to a scratch Kafka topic, or through Vector's demo_logs/stdin source to keep the harness small. Assert that every event is present after the block releases, with no duplicates (count by _id); 13. auto-create guard as an -E startup arg is accepted by ES 9.5.4 (if not, the §6.3 fallback is used and this test asserts the persistent setting).

12.3 Compose stack on the dev box (upgrade path)

The current stack has legacy concrete indices. 1. Rebuild the images and redeploy. 2. Verify adoption and aliases. 3. Run the stress producer for ingestion. 4. Exercise the dashboard pages: inferences, detail, graph view, evidence, storage, whitelist create/preview. 5. Check that hydration lookups work. 6. Temporarily lower RETENTION_TARGET_PERCENT below current usage and confirm that only eligible indices are deleted.

12.4 OpenShift on CRC (air-gapped posture)

Images come from the internal registry via global.imageRegistry, as in the existing e2e. - Fresh install: helm install --wait completes (no deadlock), the Vector initContainer gates correctly, verify-ingest.sh passes with patterns. - Upgrade: install the current release v1.2.1, ingest, then helm upgrade to the new chart. Check adoption, the rolling Vector switch-over with no lost events (counts before and after), and retention logs. - Rollback: helm rollback, then confirm nothing crash-loops and the §8.3 behavior holds.

12.5 CI (new, following existing workflow conventions)

On PRs touching deploy/elasticsearch/**, deploy/openshift/** or deploy/vector/**, run: - python -m unittest discover -s deploy/elasticsearch, including the drift test; - helm lint and helm template for the chart.

Today no workflow runs any of these tests.

13. Failure modes

Failure	Behavior
ES down or unreachable	Worker logs and skips the check. init retries (Job `backoffLimit`) and fails the deploy if ES never comes up.
Search partial / timed out	Disk rule skips this check (fails closed)
`_cluster/stats` missing or failed nodes	Disk rule skips this check
Write alias or write index deleted by hand	Writes fail per-item with 404, and Vector drops those events (it doesn't retry 404s; D9). The next check's `converge()` re-bootstraps (≤ 5 min) and fixes the orphan (§7.1 step 5). Loss is bounded by the check interval, and it can only follow a manual deletion.
Concrete index squatting on the alias name	Writes land in the squatter (no loss). Loud ERROR each check with the manual fix. init logs ERROR, exit 0. The Vector initContainer does not block on it and logs ERROR once. Prevented by D9.
ILM stuck (e.g. rollover error)	Reported as WARN each check. The disk rule still protects the disk. A bare legacy name stuck in the family policy's rollover step after a rollback is released on roll-forward by adoption (§7.1 step 3).
Disk at flood stage (≥95%)	ES blocks writes (per-item 429). Vector retries them (D14) and backpressures Kafka consumption. The disk rule deletes (allowed under the block), and ES auto-releases below 90%, after which the retried events land. If the floor prevents deletion, ingestion stays paused, bounded by Kafka retention (72 h / 140 MiB per partition; beyond that Kafka drops the oldest segments), and the red dashboard banner shows. No corruption.
Disk above the high watermark at rollover time	The new index's primary can't be allocated until disk < 90%. Writes to it fail per-item with 5xx, which Vector retries (D14). The disk rule is already acting by then (target 75%). init's allocation wait times out with a WARN and exit 0 (§5.1), so ingestion is not blocked on Compose/OVA.
Floor reached (non-ES consumer)	WARN each check. Nothing newer than the floor is ever deleted.
Two workers briefly (misconfig)	Deletions are idempotent (404 = success). The chart's `maxSurge: 0` strategy plus the SIGTERM handler prevent overlap.
Bad env values	WARN + default. Never exits.
Clock skew worker ↔ ES	Floor evaluated with ES's clock

14. Documentation to update

docs/technical-guides/openshift-deployment/data-retention.md (currently says "does not use ILM").
deploy/openshift/logster/README.md retention section, templates/NOTES.txt:23.
docs/technical-guides/admin-guide/{daily-operations,important-considerations}.md, and packer/files/env.example (document the knobs).
CLAUDE.md: the retention invariant (no emergency 80%, no "clears blocks"), config quick reference, and the new invariants:
readers use patterns, writers use aliases;
never GET/_update by id on a pattern;
never delete by wildcard.
Release notes: new layout, removed knobs, the one-time legacy granularity (§8.2), rollback behavior and reduced disk protection while rolled back (§8.3), "at least N, at most N+1 days", and that a disk emergency pauses ingestion for up to Kafka's retention (D14).

15. Resolved decisions (user, 2026-09-25)

Floor: default 24 h, configurable via RETENTION_MIN_AGE_HOURS / retention.minAgeHours (1–720). The floor is not a target. It is the outer limit of how far the disk rule may go in extreme cases while trying to meet the threshold. In normal operation the disk rule meets the target long before it reaches the floor.
raw_event strip removed, and best_compression added (D8).
Check interval 5 min.
CI workflow is part of this change (§12.5).

16. Review log

Revision 2, independent review (Fable 5.1), verdict "not ready". All findings addressed:

#	Severity	Finding	Resolution
1	BLOCKER	Vector drops per-item failures, so "Kafka buffers" was false	D14 `request_retry_partial: true` (429/5xx retried, verified in Vector 0.52 source). 404 remains non-retriable: D9 states this, and §13 row rewritten. §12.2 scenario 12 added.
2	MAJOR	`terms(_index)` default size 10 silently breaks the floor	§7.3: `size: 10000`, fail closed on `sum_other_doc_count > 0`, index missing from `newest` never eligible. Unit tests added.
3	MAJOR	Compose gates inference on init; new fatal init paths would block the verdict path	D11: Compose inference ungated. §5.1: init fatal only on misconfiguration; allocation timeout, squatter and adoption failure exit 0.
4	MAJOR	Bootstrap race returns `illegal_argument_exception`, not `already_exists`	§7.1: any 4xx → re-read alias; a write index present = success
5	MAJOR	Orphaned former write index stuck in ILM `ERROR`, never deleted	§7.1 step 5: `indexing_complete` + `_ilm/retry`. §12.2 scenario 9 asserts it reaches delete.
6	MAJOR	ES normalizes policies, so a naive compare means a PUT every run	`_meta.logster_hash` change detection. Template `_meta` placement fixed.
7	MINOR	Free-form `?sort=` on an empty rollover index → shard failures	`?sort=` allowlist with `unmapped_type`
8	MINOR	Inventory gaps	Added to §9 (hydration default, API construction, forecast contract tests, chart docs, producer docstring, `logster-alerts`, ES-args parity incl. `xpack.ml`)
9	MINOR	Rollback leaves rollover indices without disk protection	§8.3 + release notes
10	MINOR	`used` undefined; free ≠ available	§7.3: `used = total − available`
11	MINOR	Shard count with 1d rollover	D12: accepted deliberately and documented (about 134 + about 75)
12	MINOR	tmpfs ownership	§12.2: `uid=1000,gid=0`
13	NIT	1 GB floor for small disks; empty orphans	D12 note; §7.1 accepts empty orphans

Revision 3 — plan review (Fable 5.1, verification-focused), findings folded into the plan: curl-free operator commands (eshttp.py CLI), Helm-3.8-safe upgrade (helm get values + -f; minAgeHours | default 24), cluster.info.update.interval=10s in the integration harness, rollingUpdate: null for the Recreate switch (helm/helm#5144), and F5 — phantom-hydration latency over logster-events*, measured at the customer's scale (licence cap 10k events/day; 100k events tested, chart ES sizing 2 CPU / 2 GiB heap, exact hydration query):

Layout	Callers	p50	p99	Max	> 0.5 s timeout
1 index (today)	4	10 ms	40 ms	51 ms	0/400
36 daily indices	1	15 ms	40 ms	104 ms	0/400
36 daily indices	4	37 ms	160 ms	174 ms	0/400

Hits identical across layouts. Decision: no timeout change (graph-building behaviour untouched); the live tests (plan Tasks 19/20) compare logster_phantom_hydration_* error share and latency before and after the upgrade and stop on any regression.

CRC verification (OCP 4.22.1), strategy switch on an existing Deployment: Helm 3.8.2 / 3.22.0 → Recreate works with or without rollingUpdate: null; Helm 4.2.4 → UPGRADE FAILED: server-side apply … spec.strategy.rollingUpdate: Forbidden in both forms; RollingUpdate{maxSurge: 0, maxUnavailable: 1} → works on all three. D13 changed accordingly.

CRC verification (OCP 4.22.1, 2026-09-25) — upgrade path from the customer's install. Baseline: chart 1.2.1+splunk-extip.1 with the customer's install flags (LoadBalancer receiver, Kafka retention overrides), v1.2.1 images; seeded legacy data (300 events / 30 inferences / 10 graphs). Upgrade target: the same chart with only this spec's retention changes (scripts, templates, Vector config, chart templates), so the result isolates this work.

Step	Result
Helm minimum	Helm 3.8.2 and 3.9.4 cannot install or render the chart (`secrets.yaml:43 … index of untyped nil`, pre-existing); 3.10.0 → 4.2.4 work. Install guide prerequisite must read v3.10+.
Upgrade, Helm 3.10.0, `helm get values` → `upgrade -f --wait`	`deployed`. Worker strategy `maxSurge 0 / maxUnavailable 1`; ≤ 1 worker pod at any second (1 s sampler). Vector initContainer admitted under `restricted-v2`, `Completed` exit 0 after 130 s (ES restart). Worker adopted the three legacy indices (origination = newest doc) and bootstrapped `logster-*-000001` write aliases; post-upgrade init hook `done.`; ES args carry the auto-create guard; ILM errors none; all pods Running.
Vector end-to-end	An event produced to `normalized-endpoint-events` landed in `logster-events-000001` through the write alias in ~15 s.
Install guide B7 (`--reuse-values --set splunkReceiver.service.type=LoadBalancer`)	`deployed`; every `RETENTION_*` env renders a value (template defaults), strategy and Vector initContainer preserved.
`helm rollback` to v1.2.1	Success; no crash loops; ES args back to v1.2.1; old worker cycles cleanly; old Vector writes to the concrete `logster-events` (300 → 301); the post-upgrade event is preserved in `logster-events-000001`.
`--reuse-values` to a new chart (Helm 3.10 / 3.22 / 4.2.4)	Keeps the old chart's default image tag and drops new default keys; `helm get values` + `-f` takes the new defaults and keeps user values. Customer docs use the latter.
`helm upgrade --wait` with a LoadBalancer receiver and no LB provider	Hangs until timeout waiting for an external IP (pre-existing; customer commands do not use `--wait`; doc note added).
Roll forward after rollback, Helm 4.2.4 (server-side apply on a Helm-3-managed release), `get values` → `-f --wait`	`deployed` (rev 6), no SSA conflicts; strategy applied; ≤ 1 worker; Vector initContainer `Completed` under `restricted-v2`; legacy `origination_date` moved to the newest document (the event written while rolled back); policies/templates unchanged (hash match); ILM errors none.
Fresh install with `--wait` (Helm 3.10.0, default NodePort receiver)	`deployed` in 125 s, no deadlock: Vector initContainer waited ~100 s for ES's first start, the worker created 6 policies + 6 templates + 3 write aliases, the post-install init hook `done.`; writing to the bare name `logster-events` → `404 … [action.auto_create_index] contains [-logster-events] which forbids automatic creation`, no index created; all pods Running.

Revision 4, Task 4 execution against a real ES 9.5.4 (probe, then integration scenarios test_14 / test_15):

#	Severity	Finding	Resolution
1	MAJOR	After upgrade → legacy aged out → `helm rollback` → roll-forward, the bare name auto-created while rolled back sits in the family policy's cached hot/rollover step and is never deleted; relatedly, a squatted alias name made the Vector initContainer wait forever, and `_ilm/retry` on an orphan could answer 400 when ES's auto-retry got there first	Adoption also sets `index.lifecycle.indexing_complete: true` (D7, §7.1 step 3); `layout.py wait` treats a squatted name as ready and logs ERROR once (§5.1, §7.1 step 4, §13); a retry 400 counts as success (§7.1 step 5). Proven on the shipped image by the plan's integration scenarios test_14 (rollback-created index released and deleted) and test_15 (`GET /_alias/<concrete index>` → 404, squatter reported, `wait` ready).