Data Retention & Storage Management

Once Logster is running on OpenShift, two stateful components accumulate data on disk: Kafka (the message broker) and Elasticsearch (the query store). Each has its own retention mechanism and storage PVC.

The chart sizes Kafka's retention for its default 10 Gi PVC: each partition keeps 72 hours or 140 MiB, whichever limit is reached first (51 partitions × 140 MiB ≈ 7 Gi, about 70 % of the PVC). If you enlarge the PVC, raise log.retention.bytes with it.

Component	Default PVC	Default retention	Prod recommendation
Kafka (`logster-dual-0`)	10 Gi	72 h or 140 MiB per partition, whichever comes first	50 Gi + retention scaled to it (see below)
Elasticsearch	15 Gi	Events 30 d, inferences 90 d, graphs 14 d (ILM); oldest whole index deleted above 75 % disk	100 Gi

Kafka log rotation

The Helm chart passes Kafka's config: block straight through to the Strimzi Kafka CR — any Kafka broker config key can be added as a value under kafka.config without modifying the chart's templates.

The chart sets these retention values (kafka.config in values.yaml), which is approach C below:

Setting	Chart default	Meaning
`log.retention.hours`	72 (3 days)	Delete closed log segments older than this
`log.retention.bytes`	146800640 (140 MiB)	Maximum bytes to keep per partition before deleting the oldest segments
`log.segment.bytes`	52428800 (50 MiB)	Size at which a log segment file is closed and a new one started

Why log.segment.bytes matters

Kafka can only delete a closed segment — it never truncates the active one. If a segment hasn't reached log.segment.bytes yet, the data inside it stays even when it's older than log.retention.hours. Small segments (the chart ships 50 MiB, against Kafka's built-in 1 GiB) close sooner, making old data eligible for cleanup faster — important on a small disk. Do not raise it on the default 10 Gi PVC: each of the 51 partitions can hold one open segment of this size on top of log.retention.bytes.

Topics and partition counts

The chart creates 8 topics with a total of 51 partitions:

Topic	Partitions
`sysmon-logs`	6
`linux-auditd-logs`	6
`linux-ebpf-process-logs`	6
`linux-ebpf-file-logs`	6
`linux-ebpf-network-logs`	6
`normalized-endpoint-events`	12
`logster-inference-results`	6
`logster-alerts`	3

log.retention.bytes is a per-partition limit. To estimate total disk use, multiply the per-partition cap by the partition count (51).

Three approaches to rotation

A — Time-basedB — Size-basedC — Combined (recommended)

Delete log segments older than a fixed duration. The simplest and most common approach. Disk usage is proportional to throughput × retention window.

helm upgrade logster ./logster-<ver>.tgz -n logster --reuse-values \
  --set 'kafka.config.log\.retention\.hours=72' \
  --timeout 20m

Under light-to-moderate load, 3 days on a 10 Gi disk is usually safe. Increase the PVC (below) for longer windows.

Cap the on-disk size per partition. When a partition exceeds this limit, the oldest segments are deleted regardless of age.

helm upgrade logster ./logster-<ver>.tgz -n logster --reuse-values \
  --set-string 'kafka.config.log\.retention\.bytes=146800640' \
  --timeout 20m

Use --set-string for byte sizes: --set stores an integer, which a later helm get values -o yaml → -f upgrade renders as 1.4680064e+08, and Strimzi ignores that.

140 MiB × 51 partitions ≈ 7 GiB total, leaving headroom on the default 10 Gi PVC for the open 50 MiB segments. Scale proportionally for larger disks.

Use time-based as the primary policy and size-based as a safety net. Data is deleted when either condition is met — whichever triggers first.

helm upgrade logster ./logster-<ver>.tgz -n logster --reuse-values \
  --set 'kafka.config.log\.retention\.hours=72' \
  --set-string 'kafka.config.log\.retention\.bytes=146800640' \
  --timeout 20m

Under normal throughput, time-based does the work. Under a burst (many endpoints onboarded at once), the size cap prevents the disk from filling before the time window expires. The byte size uses --set-string for the reason given under B.

Escape the dots in --set

Helm's --set flag uses dots as path separators. Kafka config keys like log.retention.hours must be escaped with backslashes:

--set 'kafka.config.log\.retention\.hours=72'

In a values file no escaping is needed.

Using a values file

A values file is cleaner when setting multiple keys:

# retention-overrides.yaml
kafka:
  storageSize: 50Gi
  config:
    log.retention.hours: "72"
    log.retention.bytes: "734003200"    # 700 MiB per partition ≈ 35 GiB total (70 % of 50 Gi)

helm upgrade logster ./logster-<ver>.tgz -n logster \
  -f retention-overrides.yaml --reuse-values \
  --timeout 20m

Values must be strings

Helm renders large YAML integers in scientific notation (e.g. 1.048576e+07), which Strimzi rejects. Always quote numeric Kafka config values.

Increasing Kafka storage

The default PVC is 10 Gi (kafka.storageSize). To resize, first check whether your StorageClass supports expansion:

oc get storageclass \
  -o custom-columns=NAME:.metadata.name,EXPAND:.allowVolumeExpansion

If allowVolumeExpansion is true, Strimzi can resize the broker PVC online:

helm upgrade logster ./logster-<ver>.tgz -n logster --reuse-values \
  --set kafka.storageSize=50Gi \
  --timeout 20m

If expansion is not supported, you need to recreate the PVC (back up data, delete the PVC, reinstall with the larger size).

Elasticsearch retention

Logster stores events, inferences and graph snapshots in daily Elasticsearch indices managed by Index Lifecycle Management (ILM). Two rules keep storage in check:

Rule	Who enforces it	What it does
Age	Elasticsearch ILM	Deletes whole indices (each rolls over daily or at the size cap) older than `retention.eventsDays` / `inferencesDays` / `graphsDays`. Data is kept at least N days and at most N+1. `0` keeps data forever.
Disk	`elasticsearch-retention` worker, every `retention.checkIntervalMinutes`	While the Elasticsearch volume is above `retention.targetPercent`, deletes the oldest eligible whole Logster indices, as many as their sizes say are needed to get back under the target. The next check deletes more if that estimate fell short.

The disk rule never deletes: - the index currently being written; - an index holding data newer than retention.minAgeHours (default 24 h). This is the limit of how far the worker will go when trying to meet the target; - anything that is not a Logster data index. Only an index named exactly logster-events, logster-inferences or logster-graphs, optionally followed by - and six or more digits (for example logster-events-000042), can be deleted. Alerts, whitelists, dashboard accounts and the capacity forecast are never touched; - an empty index, or one whose size or age Elasticsearch cannot report during that check (for example a red index). That index is skipped for this check only; the other eligible indices are still deleted. If Elasticsearch stops answering while the worker reads index ages, nothing is deleted that check.

If the volume stays above the target because nothing old enough is left, the worker logs a warning every check and the dashboard's storage banner stays up. Elasticsearch's own flood-stage protection (95 %) blocks writes before the disk can fill. Ingestion then pauses and Kafka buffers the data, for as long as Kafka's own retention keeps it (by default 72 hours or 140 MiB per partition, whichever comes first; see Kafka log rotation). No data is corrupted.

Each check, the worker also restores any missing or changed ILM policy, index template or write alias, and logs a WARNING for every index ILM is stuck on. A failing step never skips the others, and the worker never exits on an error: while Elasticsearch is unreachable it retries every 30 seconds, and a log line that cannot be printed is dropped. Only one worker runs at a time; on upgrade the old pod stops before the new one starts.

Settings

values.yaml	Default	Meaning
`retention.eventsDays`	30	Age rule for events (0 = keep forever; 0–3650)
`retention.inferencesDays`	90	Age rule for inferences (0 = keep forever; 0–3650)
`retention.graphsDays`	14	Age rule for graph snapshots (0 = keep forever; 0–3650)
`retention.targetPercent`	75	Disk rule target, % of the Elasticsearch volume (1–89)
`retention.minAgeHours`	24	Disk rule never deletes data newer than this (1–720)
`retention.checkIntervalMinutes`	5	How often the worker checks (1–1440)

Invalid values are logged as warnings and replaced by the defaults. The worker never stops because of them. retention.stripRawEventDays, intervalHours, emergencyThresholdPercent and criticalThresholdPercent are no longer used; if your values still set them, they are ignored.

To change a setting on the chart version you already run:

helm upgrade logster ./logster-<ver>.tgz -n logster --reuse-values \
  --set retention.eventsDays=14 \
  --set retention.targetPercent=70 \
  --timeout 20m

The worker restarts with the new values and updates the ILM policies on its first check. To move to a new release, follow Upgrading instead.

You can also edit the RETENTION_* variables on the elasticsearch-retention Deployment in the OpenShift console (Workloads → Deployments → elasticsearch-retention → Environment). The pod rolls out with the new values, but the next helm upgrade puts back the Helm values, so pass the same values to Helm as well.

Upgrading from a release without ILM retention

Nothing is moved or reindexed. Your existing logster-events, logster-inferences and logster-graphs indices stay readable and are deleted by ILM once their newest document is older than the retention period. New data goes to new indices, which roll over daily or at the size cap.

Upgrade by re-applying the values you installed with. This works on every supported Helm version (3.10+) and picks up the new release's image tags and defaults:

helm get values logster -n logster -o yaml > logster-values.yaml
helm upgrade logster ./logster-<ver>.tgz -n logster -f logster-values.yaml --timeout 20m && rm logster-values.yaml

The rm deletes the saved values file, because it contains the Elasticsearch password and the license. It runs only if the upgrade succeeds, so a failed upgrade leaves the file in place for a retry; delete it yourself once you are done.

Values saved from an earlier release have no retention.targetPercent, minAgeHours or checkIntervalMinutes; those take their defaults (75, 24, 5). Any retention keys the new release no longer uses (listed under Settings) are ignored, so you can leave them in the file.

Do not upgrade to a new release with --reuse-values alone: Helm then keeps the previous release's chart defaults, including its image tags.

Do not add --wait if the Splunk receiver is a LoadBalancer Service on a cluster without a load-balancer provider: Helm then waits for an external IP that never arrives, and the upgrade times out (verified on CRC with v1.2.1's own values; this predates the retention change).

During the upgrade the vector pod shows Init:0/1 for a few minutes while Elasticsearch restarts and the retention worker creates the new indices (measured on CRC: ~2 minutes). Vector's previous pod keeps shipping until the new one is ready.

Each existing index holds up to N days of data. If the volume is above the target at upgrade time, the disk rule deletes the whole oldest existing index in one step, once its newest document is older than retention.minAgeHours.

Upgrading from v1.2.1

Release notes for the move from v1.2.1 to ILM retention:

A few events can appear twice. While Elasticsearch restarts during the upgrade, events Vector has buffered but not yet written can be written both to the old logster-events index and to the new write index (measured on CRC: 468 duplicate events for one upgrade). Delivery is at-least-once, so nothing is lost, but event counts that cover the few minutes of the restart are slightly inflated. The same can happen on a rollback and on the roll-forward after it.
The storage forecast is approximate until the old indices age out. Its bytes-per-document figure averages the old indices with the new ones, which are stored with best_compression and are smaller per document. The forecast settles once ILM deletes the old index.
The dashboard's attack threshold drops from 0.80 to 0.70. Upgrading with the documented helm get values … -o yaml / -f path picks up the new chart default for the dashboard's ATTACK_PROB_THRESHOLD, 0.70, which matches logster_classifier.attack_min_attack_prob (0.70). Probability bars between 0.70 and 0.80 now paint red, so more rows look like attacks than before; the verdicts themselves do not change.
A fresh install can log a few graph-snapshot 404 warnings. On a new install, the inference service can try to write per-window graph snapshots before the retention worker has created the logster-graphs-write alias. Those writes fail with a per-item 404, logged as a WARN and counted in the graph sink's failure metric. Snapshots are debug-only data, and this lasts at most until the worker's first check.
Do not use --reuse-values from v1.2.1. It keeps v1.2.1's retention.checkIntervalMinutes of 30 instead of the new default of 5, along with the old image tags. Use the helm get values … -f path above.
v1.2.1 retention keys are ignored. If your saved values still set retention.stripRawEventDays, intervalHours, emergencyThresholdPercent or criticalThresholdPercent, they have no effect. Use the keys under Settings.
Compose (development): recreate Vector. Vector reads its config only when its container starts, and Compose does not recreate a container just because a mounted file changed. After upgrading, run docker compose up -d --force-recreate vector from deploy/, or Vector can keep writing to the old index names. The Compose file carries a label that changes with vector.yaml, so a plain docker compose up -d also recreates Vector. If the command fails with "dependency failed to start", see Troubleshooting.
Appliance: restart the Logster service. Do not run docker compose by hand on the appliance: its Compose file takes the Elasticsearch password only from the environment that logster.service loads, so a container recreated from a shell starts with an empty password and every write it makes is rejected with 401. Run sudo systemctl restart logster.service instead. The service stops the stack and starts it again, which recreates Vector with the new config and the right password.
OpenShift is not affected: the chart rolls the Vector pod when its config changes.

Rolling back

helm rollback is safe: nothing crashes and no data is corrupted. The previous release reads only the pre-upgrade indices, so data written after the upgrade is hidden until you roll forward again. Its retention worker cannot free space in the new indices, so roll forward promptly and do not stay rolled back on a nearly full volume. ILM still ages the new indices out, because its policies stay in Elasticsearch.

While rolled back, the previous release writes to the pre-upgrade index names again (Elasticsearch creates them if they do not exist). When you roll forward, the retention worker adopts those indices like any pre-upgrade index: it marks them complete for writing (index.lifecycle.indexing_complete) so that ILM deletes each one once its newest document is older than the retention period.

Checking status

The Elasticsearch image has no curl, so these commands run through the retention worker pod, using the same stdlib client the worker itself uses (/templates/eshttp.py METHOD PATH [JSON]).

oc logs deploy/elasticsearch-retention -n logster --tail=50
oc exec -n logster deploy/elasticsearch-retention -- python3 /templates/eshttp.py GET '/_cat/aliases/logster-*-write?v'
oc exec -n logster deploy/elasticsearch-retention -- python3 /templates/eshttp.py GET '/logster-*/_ilm/explain?only_errors=true'

The aliases query lists logster-events-write, logster-inferences-write and logster-graphs-write; the ILM query returns {"indices":{}} when ILM has no errors.

The elasticsearch-init Job runs after every install and upgrade. It sets the disk watermarks and creates the ILM policies, index templates and write aliases, retrying up to 5 times, 10 seconds apart, while Elasticsearch settles. It fails only when Elasticsearch is unreachable or rejects the password; when Elasticsearch rejects a setting, policy, template or new index; when a template file is missing from the image; or when the policy, template or write-alias step crashes. Its log names the cause. Anything else is logged as a WARNING and the Job still completes:

a rejected shard reroute: Elasticsearch retries allocation by itself;
write indices still unallocated after 60 seconds (usually a disk above the 90 % high watermark): they allocate once space is freed, and Vector retries its writes meanwhile;
a write-alias squatter: the retention worker reports it every check until it is fixed (see Write-alias squatter);
a pre-upgrade index it could not adopt: the retention worker retries the adoption every check.

oc logs job/elasticsearch-init -n logster

While a vector pod is stuck in Init:0/1, this shows what it is waiting for:

oc logs -n logster "$(oc get pod -n logster -l app=vector --field-selector=status.phase=Pending -o name | head -n 1)" -c wait-es-layout

It picks the pod that is still initializing. During an upgrade, oc logs deploy/vector would pick the previous release's running pod instead, which has no wait-es-layout container. On a fresh install the waiting pod is the only one, so oc logs deploy/vector -n logster -c wait-es-layout also works.

Write-alias squatter

What the message means. Logster writes each family through an alias: logster-events-write, logster-inferences-write and logster-graphs-write. If one of those names is held by a plain (concrete) index instead, the retention worker cannot create the alias and logs, every check:

WARNING: layout: SQUATTER: a concrete index named logster-<family>-write holds the write-alias name, so writes land in an unmanaged index. See 'Write-alias squatter' in docs/technical-guides/openshift-deployment/data-retention.md and contact support.

elasticsearch-init logs the same message as a WARNING, and on OpenShift the vector pod's wait-es-layout init container logs it once as an ERROR. On Compose and the appliance the worker logs the same message. This only happens if the auto-create guard was removed from Elasticsearch's settings or someone created the index by hand.

Confirm it (read-only). The first command lists the write aliases that exist; a squatted family is missing from it. The second lists every Logster index with its document count and size; the squatter is the row whose name ends in -write. Only concrete indices appear in that list, so such a row is never an alias.

On OpenShift:

oc exec -n logster deploy/elasticsearch-retention -- python3 /templates/eshttp.py GET '/_cat/aliases/logster-*-write?v&h=alias,index'
oc exec -n logster deploy/elasticsearch-retention -- python3 /templates/eshttp.py GET '/_cat/indices/logster-*?v&h=index,health,docs.count,store.size&s=index'

On Compose and the appliance, from the deploy directory. On the appliance, load the Elasticsearch password first; elsewhere, use the ELASTIC_PASSWORD you exported for Compose.

ELASTIC_PASSWORD=$(sudo sed -n 's/^ELASTIC_PASSWORD=//p' /etc/logster/logster-secrets.env)

curl -s -u "elastic:$ELASTIC_PASSWORD" 'http://localhost:9200/_cat/aliases/logster-*-write?v&h=alias,index'
curl -s -u "elastic:$ELASTIC_PASSWORD" 'http://localhost:9200/_cat/indices/logster-*?v&h=index,health,docs.count,store.size&s=index'

Impact. Writes to that family still land in the squatting index, so ingestion keeps working and nothing is lost. But neither ILM nor the disk rule ever deletes that index, so it grows until it is fixed (the worker may also log ILM is stuck on logster-<family>-write). Its size still counts toward the disk target, so the disk rule reaches the other, older indices sooner. On OpenShift the vector pod still starts: its init container does not wait for a squatted alias.

What to do. Contact Logster support. Do not delete, block, clone or rename the index yourself: it holds the data written since it appeared.

Increasing Elasticsearch storage

The default PVC is 15 Gi (elasticsearch.storageSize). The same StorageClass expansion check applies:

oc get storageclass \
  -o custom-columns=NAME:.metadata.name,EXPAND:.allowVolumeExpansion

helm upgrade logster ./logster-<ver>.tgz -n logster --reuse-values \
  --set elasticsearch.storageSize=100Gi \
  --timeout 20m