Install Logster on OpenShift with Helm

This is the install guide for OpenShift, for both internet-connected and air-gapped clusters..

Logster is delivered for OpenShift as two Helm charts, packaged as .tgz archives in your delivery:

Chart archive	What it deploys	Where it runs
`logster-classifier-<ver>.tgz`	The LogsterClassifier LLM model server — the OpenAI-compatible endpoint that produces Logster's authoritative verdict.	A GPU node (needs `nvidia.com/gpu`).
`logster-<ver>.tgz`	The Logster application stack — normalizer, inference, alerts, API, dashboard, plus Kafka (operator-managed), Elasticsearch, Vector and Redis.	Any worker nodes, entirely under the default `restricted-v2` SCC (no privileged access).

Install order matters. Deploy the classifier chart first: it publishes the model endpoint URL that the application chart's inference service must be given at install time. Then deploy the application chart and point it at that endpoint.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Roboto, Helvetica, Arial, sans-serif","fontSize":"15px","lineColor":"#8B6DB8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"curve":"basis","nodeSpacing":45,"rankSpacing":80,"padding":16}}}%%
flowchart LR
    subgraph EP["&nbsp;Windows / Linux endpoints&nbsp;"]
        SH["<b>Log shipper</b><br><small>Winlogbeat / rsyslog</small>"]
    end
    subgraph OCP["&nbsp;OpenShift cluster&nbsp;"]
        subgraph APP["&nbsp;logster &mdash; application chart&nbsp;"]
            KR["<b>Kafka routes</b><br><small>bootstrap + per-broker</small>"]
            N["<b>normalizer → inference → alerts</b>"]
            DB["<b>dashboard route</b>"]
            KR --> N
            N --> DB
        end
        subgraph GPU["&nbsp;logster-classifier &mdash; model chart&nbsp;"]
            CR["<b>classifier route</b><br><small>HTTPS :8000 /v1/chat/completions</small>"]
        end
        N -- "verdict requests" --> CR
    end
    U["<b>Analyst browser</b>"]
    SH -- "raw logs · TLS" --> KR
    U -- "HTTPS" --> DB

    classDef external fill:#E8F1FC,stroke:#3B82C4,stroke-width:2px,color:#12324F
    classDef app fill:#EFE6FB,stroke:#7A3FBF,stroke-width:2px,color:#2D0A6B
    classDef model fill:#FFF6E5,stroke:#D9922B,stroke-width:2px,color:#4A3208
    classDef entry fill:#E4DAF6,stroke:#5B1FA8,stroke-width:3px,color:#2D0A6B
    class SH,U external
    class N app
    class KR,DB entry
    class CR model

Prerequisites

Item	Notes
OpenShift 4.12+	a default StorageClass is required (the application chart requests ~27 GB of persistent storage)
`oc` CLI, logged in	project-admin rights for the installs; cluster-admin only for the one-time operator installs (GPU Operator, Kafka operator)
`helm` CLI v3.8+	single static binary — see helm.sh/docs/intro/install
GPU node + NVIDIA GPU Operator	provides the `nvidia.com/gpu` resource the classifier requests; install from OperatorHub together with Node Feature Discovery. See the GPU Node guide for hardware/driver requirements. The classifier image is ~80–90 GB — the GPU node needs ≥150 GB free container storage for the pull (compressed + unpacked layers coexist briefly).
Registry credentials	username/token with pull access to the `eunomatix/logster-*` image repositories (provided with your delivery)
License file	`license.lic`, provided with your delivery
Both chart archives	`logster-classifier-<ver>.tgz` and `logster-<ver>.tgz` from your delivery

Air-gapped clusters additionally need:

Item	Notes
A bastion / transfer host	Can reach both the delivered files and your internal registry. Needs any one of `podman`, `docker`, or `skopeo` — the archives are standard `docker save` format, loadable by all three. No extra tools required.
Bastion scratch disk	~90 GB for the classifier zip, plus ~90 GB to unzip its `.docker.tar.gz`, plus ~90 GB more while it is pushed (`podman`/`docker` stage the image in local storage; `skopeo` needs the archive decompressed first). Delete the zip after unpacking to save one copy — budget ~250–300 GB
Red Hat Quay (or equivalent)	An organization to hold the images (this guide uses `logster`) and a robot account with write permission on it
Mirrored OperatorHub catalog	See the note in B1 — the Kafka and GPU operator images come from the operators, not from the Logster charts

Before installing, confirm the GPU is actually schedulable — after the GPU Operator install (and after any GPU node reboot) it takes a few minutes to build the driver and advertise the resource:

oc get nodes -o custom-columns=NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu
# the GPU node must report GPU: 1 (not 0 or <none>) before you install

Part 0 — Air-gapped only: load the images into your registry

Connected cluster? Skip to Part 1.

This whole part is only for clusters that cannot pull from the internet.

0.1 — What's in the delivery

The release is delivered as two self-contained zips, each with a sidecar .zip.sha256, plus the signed license.lic:

File	Contents
`logster-<tag>.zip` + `logster-<tag>.zip.sha256`	The application bundle (charts, sensors, service images) — unpacked below.
`logster-classifier-<ver>.zip` + `logster-classifier-<ver>.zip.sha256`	The classifier bundle — the model image, its manifest and checksum — unpacked in §0.1.1.
`license.lic`	The signed license required by B2, delivered as a separate file (it is customer-specific).

The application zip, logster-<tag>.zip, unpacks to a logster-<tag>/ folder:

logster-<tag>.zip
└── logster-<tag>/
    ├── logster-<ver>.tgz                       # application Helm chart
    ├── logster-classifier-<ver>.tgz            # classifier Helm chart
    ├── sensors/                                # Linux endpoint sensor packages
    │   ├── logster-sensor_<ver>_amd64.deb
    │   ├── logster-sensor-<ver>.x86_64.rpm
    │   ├── logster-sensor-<ver>-x86_64.tar.gz
    │   └── sha256sums.txt                      # covers this folder only
    └── images/                                 # air-gapped image archives
        ├── logster-normalizer-<tag>.docker.tar.gz
        ├── logster-inference-<tag>.docker.tar.gz
        ├── logster-alerts-<tag>.docker.tar.gz
        ├── logster-api-<tag>.docker.tar.gz
        ├── logster-dashboard-<tag>.docker.tar.gz
        ├── logster-splunk-receiver-<tag>.docker.tar.gz
        ├── logster-elasticsearch-<tag>.docker.tar.gz
        ├── logster-elasticsearch-init-<tag>.docker.tar.gz
        ├── vector-<ver>-distroless-static.docker.tar.gz
        ├── redis-<ver>-alpine.docker.tar.gz
        ├── prometheus-<ver>.docker.tar.gz
        ├── images-manifest.txt                 # "image-ref  archive-filename", one per line
        └── sha256sums.txt                      # covers this folder only

Path (inside the zip)	Contents
`logster-<tag>/logster-<ver>.tgz`	Application Helm chart
`logster-<tag>/logster-classifier-<ver>.tgz`	Classifier Helm chart
`logster-<tag>/sensors/`	Linux endpoint sensor packages (`.deb`, `.rpm`, tarball, `sha256sums.txt`). Windows endpoints do not use these — they ship via Winlogbeat or a Splunk Universal Forwarder (B6 / B7)
`logster-<tag>/images/*.docker.tar.gz`	One offline archive per image — the 8 Logster service images (normalizer, inference, alerts, API, dashboard, Splunk receiver, Elasticsearch, Elasticsearch-init) plus Vector, Redis and Prometheus
`logster-<tag>/images/images-manifest.txt`	`original-image-ref archive-filename` mapping, one per line
`logster-<tag>/images/sha256sums.txt`	Checksums for every archive in `images/`

The two sha256sums.txt files each cover only their own folder; the charts have no checksum inside the zip, which is what the sidecar logster-<tag>.zip.sha256 is for.

0.1.1 — Inside the classifier zip

logster-classifier-<ver>.zip unpacks to a logster-classifier-<ver>/ folder holding the model image (~80–90 GB) as a single archive, its manifest, and a checksum:

logster-classifier-<ver>.zip
└── logster-classifier-<ver>/
    ├── logster-classifier-<ver>.docker.tar.gz     # the model image archive
    ├── images-manifest.txt                        # "image-ref  archive-filename"
    └── sha256sums.txt                             # checksum for the archive above

Path (inside the zip)	Contents
`logster-classifier-<ver>/logster-classifier-<ver>.docker.tar.gz`	The classifier model image
`logster-classifier-<ver>/images-manifest.txt`	`original-image-ref archive-filename` mapping (single line)
`logster-classifier-<ver>/sha256sums.txt`	Checksum for the `.docker.tar.gz` in this folder

As with the application zip, sha256sums.txt covers only the folder's contents; the sidecar logster-classifier-<ver>.zip.sha256 is what verifies the zip itself.

All images are linux/amd64.

0.2 — Download and verify

Copy both zips (and their .zip.sha256 sidecars) to the bastion, then verify and unpack each. First the application zip:

sha256sum -c logster-<tag>.zip.sha256
unzip logster-<tag>.zip
cd logster-<tag>/images/
sha256sum -c sha256sums.txt          # every line must say OK
cd -                                 # back to the download directory

Then the classifier zip — same pattern (verify the zip, unpack, then verify the archive inside it):

sha256sum -c logster-classifier-<ver>.zip.sha256
                                     # ~80–90 GB zip; this takes a few minutes
unzip logster-classifier-<ver>.zip
cd logster-classifier-<ver>/
sha256sum -c sha256sums.txt          # must say OK
cd -

0.3 — Prepare the registry

In the Quay UI (or API):

Create an organization, e.g. logster.
Create a robot account (e.g. logster+deploy) and grant it write permission on the organization.

Repository creation

Quay creates repositories automatically on first push when "Automatically create repositories on push" is enabled (CREATE_PRIVATE_REPO_ON_PUSH / FEATURE_AUTO_CREATE_REPO_ON_PUSH). If your Quay disallows this, pre-create one repository per archive name listed in images-manifest.txt (e.g. logster-api, logster-elasticsearch, logster-elasticsearch-init, redis, vector, …) plus logster-classifier. Auto-created repositories are private by default — that is fine; Part 1 configures the pull secret the cluster needs.

0.4 — Load the images

Each line of images-manifest.txt gives the original image reference and its archive file, for example:

eunomatix/logster-api:v1.0.2  logster-api-v1.0.2.docker.tar.gz
eunomatix/logster-elasticsearch:v1.0.2  logster-elasticsearch-v1.0.2.docker.tar.gz
timberio/vector:0.52.0-distroless-static  vector-0.52.0-distroless-static.docker.tar.gz

Every image is pushed to quay.example.com/logster/<name>:<tag> where <name>:<tag> is the final component of the original reference (so timberio/vector:0.52.0-distroless-static becomes quay.example.com/logster/vector:0.52.0-distroless-static). Keeping these names is what lets the charts remap everything with the single global.imageRegistry value at install time.

Use whichever tool your bastion already has — the result is identical. Replace quay.example.com with your registry host throughout.

podmandockerskopeo

podman login quay.example.com     # robot account name + token

while read -r ref file; do
  target="quay.example.com/logster/${ref##*/}"
  podman load -i "$file"
  podman tag "$ref" "$target"
  podman push "$target"
  podman rmi "$ref" "$target"     # keep local storage small
done < images-manifest.txt

# classifier (from the unzipped folder — needs the ~90 GB in local storage)
podman load -i logster-classifier-<ver>/logster-classifier-<ver>.docker.tar.gz
podman tag eunomatix/logster-classifier:<ver> \
  quay.example.com/logster/logster-classifier:<ver>
podman push quay.example.com/logster/logster-classifier:<ver>

docker login quay.example.com     # robot account name + token

while read -r ref file; do
  target="quay.example.com/logster/${ref##*/}"
  docker load -i "$file"
  docker tag "$ref" "$target"
  docker push "$target"
  docker rmi "$ref" "$target"
done < images-manifest.txt

# classifier (from the unzipped folder — needs the ~90 GB in local storage)
docker load -i logster-classifier-<ver>/logster-classifier-<ver>.docker.tar.gz
docker tag eunomatix/logster-classifier:<ver> \
  quay.example.com/logster/logster-classifier:<ver>
docker push quay.example.com/logster/logster-classifier:<ver>

skopeo pushes straight from the archive without staging the image in local container storage. Its docker-archive transport reads an uncompressed tar, so decompress each archive first (gunzip replaces the .gz in place, so this needs no extra headroom beyond the uncompressed size):

while read -r ref file; do
  gunzip "$file"                       # foo.docker.tar.gz -> foo.docker.tar
  skopeo copy "docker-archive:${file%.gz}" \
    "docker://quay.example.com/logster/${ref##*/}" \
    --dest-creds 'logster+deploy:<robot-token>'
  rm -f "${file%.gz}"
done < images-manifest.txt

# classifier (from the unzipped folder)
gunzip logster-classifier-<ver>/logster-classifier-<ver>.docker.tar.gz
skopeo copy docker-archive:logster-classifier-<ver>/logster-classifier-<ver>.docker.tar \
  docker://quay.example.com/logster/logster-classifier:<ver> \
  --dest-creds 'logster+deploy:<robot-token>'

Note

If your skopeo build accepts the compressed archive directly, you can skip the gunzip and pass the .tar.gz path — the decompression step above simply always works.

Registry with a private / self-signed TLS certificate

If the bastion does not trust your Quay certificate, the push fails with a TLS error. Install the CA on the bastion:

podman / skopeo: place the CA at /etc/containers/certs.d/quay.example.com/ca.crt (skopeo alternatively accepts --dest-cert-dir <dir>)
docker: place it at /etc/docker/certs.d/quay.example.com/ca.crt

Disabling verification (--tls-verify=false on podman/skopeo) works but should be a last resort.

Spot-check the result in the Quay UI: the logster organization should list one repository per manifest line plus logster-classifier, each with the expected tag.

Part 1 — Create the project and pull secret

Everything installs into a single project named logster. Create it once and reuse it for both charts (a dedicated project is required — OpenShift's default namespace has no SCC UID range, so pods there fail admission with container has runAsNonRoot and image will run as root):

oc new-project logster

The Logster images are in private repositories. Create a pull secret named regcred and link it to the project's default service account. The application chart references regcred by name (its imagePullSecrets value, default regcred); linking the same secret to the default service account additionally covers the classifier pods, which pull via the service account. One secret then serves both charts:

ConnectedAir-gapped

oc create secret docker-registry regcred \
  --docker-server=docker.io \
  --docker-username=<your-registry-user> \
  --docker-password=<your-registry-token> \
  -n logster

oc secrets link default regcred --for=pull -n logster

Same secret, holding the robot account credentials and targeting your registry (the repositories created in Part 0 are private):

oc create secret docker-registry regcred \
  --docker-server=quay.example.com \
  --docker-username='logster+deploy' \
  --docker-password='<robot-token>' \
  -n logster

oc secrets link default regcred --for=pull -n logster

Cluster nodes must trust the registry's certificate

The nodes pull the images, so they must trust the Quay CA too. If your Quay uses a private CA and the cluster doesn't already trust it, your cluster administrator adds it once via an additional trusted CA for image registries (image.config.openshift.io/cluster + a ConfigMap in openshift-config). Clusters that already mirror images from this Quay will have this configured.

Note

If your delivery expects a different secret name, pass it to the application install with --set imagePullSecrets[0].name=<name>.

Part A — Install the LogsterClassifier (model)

The classifier serves the bundled model over HTTPS on port 8000. Install it first so you can capture its endpoint URL for Part B.

It is a self-contained model application — a single image (eunomatix/logster-classifier) with the model weights baked in and an OpenAI-compatible server on port 8000; nothing else from Logster is needed to run it. Deploy it with its Helm chart (A1). Outcome: a reachable https://<host>/v1/chat/completions endpoint.

A1 — Install the classifier chart

Install directly from the .tgz archive. Enable a Route so the endpoint is reachable by name, and add GPU tolerations if your GPU nodes are tainted:

ConnectedAir-gapped

helm install logster-classifier ./logster-classifier-<ver>.tgz -n logster \
  --set route.enabled=true \
  --timeout 40m
  # GPU nodes are commonly tainted — usually also needed:
  #   --set tolerations[0].key=nvidia.com/gpu \
  #   --set tolerations[0].operator=Exists \
  #   --set tolerations[0].effect=NoSchedule \
  #   --set nodeSelector."nvidia\.com/gpu\.present"=true

Identical, plus global.imageRegistry pointing at the registry org you loaded in Part 0:

helm install logster-classifier ./logster-classifier-<ver>.tgz -n logster \
  --set global.imageRegistry=quay.example.com/logster \
  --set 'imagePullSecrets[0].name=regcred' \
  --set route.enabled=true \
  --timeout 40m
  # plus the same GPU tolerations / nodeSelector as the connected case

Note

The route.enabled=true flag exposes the model outside the cluster so the application's inference service can reach it by hostname. The route is TLS passthrough — end-to-end HTTPS is preserved to the model's self-signed certificate on port 8000.

A2 — Wait for the model to be ready

The image bakes the model weights in (~80–90 GB), so the first pull is long and the server then loads the weights into VRAM before it answers. The chart's startup probe allows up to ~30 minutes for this. That budget covers only the in-container weight load — the ~80–90 GB image pull happens first and is not counted against it, so on a cold GPU node the rollout can take longer than the 40-minute wait below. A timeout there is not a failure: re-run oc rollout status to resume watching.

oc rollout status deploy/logster-classifier -n logster --timeout=40m
oc get pods -l app=logster-classifier -n logster       # READY 1/1 when the model is up

The pod is READY 1/1 only once the model server passes its /health check — i.e. the model is loaded and serving.

A3 — Capture the model endpoint

Read the classifier Route host and form the OpenAI-compatible chat endpoint URL. Note this URL down — you pass it to the application chart in Part B:

oc get route logster-classifier -n logster \
  -o jsonpath='https://{.spec.host}/v1/chat/completions{"\n"}'

The result looks like:

https://logster-classifier-logster.apps.<cluster-domain>/v1/chat/completions

Important

The application's inference service must be able to resolve and reach this hostname. When both charts share the cluster this is automatic. If you run the classifier on a separate GPU cluster, confirm the application cluster can resolve the route host (DNS) and reach it over TLS (443) before continuing.

Part B — Install the Logster application

B1 — Install the Kafka operator (one-time, cluster-admin)

Logster's Kafka cluster is not created by the application chart — it is created by Red Hat AMQ Streams (or the upstream Strimzi operator — both provide the same CRDs) from a Kafka custom resource. Install the operator once per cluster, either from the console (Operators → OperatorHub → "Streams for Apache Kafka" → Install) or via CLI:

oc apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-streams
  namespace: openshift-operators
spec:
  channel: stable
  name: amq-streams            # community alternative: strimzi-kafka-operator
  source: redhat-operators     #                        community-operators
  sourceNamespace: openshift-marketplace
EOF

Wait until the operator reports Succeeded:

oc get csv -n openshift-operators | grep -Ei 'amq-streams|strimzi'

Air-gapped: operator images do not come from your registry

The subscription above is unchanged on a disconnected cluster — it simply resolves from your mirrored OperatorHub catalog. But the operator injects its own image references into the broker and entity-operator pods (registry.redhat.io/amq-streams/... for AMQ Streams, quay.io/strimzi/... for community Strimzi). Those are not in your delivery archives, and global.imageRegistry does not remap them — that value only rewrites images rendered by the Helm charts.

They come across with the operator itself when your administrator mirrors the catalog with oc-mirror, provided the operator's related images are mirrored too (they are, by default). If only the operator bundle is mirrored, every chart image pulls correctly while the Kafka pods sit in ImagePullBackOff — see Troubleshooting image pulls.

The same applies to the NVIDIA GPU Operator. Pick one of AMQ Streams or community Strimzi and make sure it matches what was mirrored — the two pull from different registries.

B2 — Install the application chart

Install from the .tgz archive, passing the classifier endpoint you captured in A3:

ConnectedAir-gapped

helm install logster ./logster-<ver>.tgz -n logster \
  --set elasticsearch.password="$(openssl rand -hex 16)" \
  --set-file license.file=./license.lic \
  --set inference.llmEndpoint="https://logster-classifier-logster.apps.<cluster-domain>/v1/chat/completions" \
  --set inference.llmUseHttps=true \
  --set inference.modelName="eunomatix/logster-26b-a4b-it" \
  --timeout 20m

helm install logster ./logster-<ver>.tgz -n logster \
  --set global.imageRegistry=quay.example.com/logster \
  --set elasticsearch.password="$(openssl rand -hex 16)" \
  --set-file license.file=./license.lic \
  --set inference.llmEndpoint="https://logster-classifier-logster.apps.<cluster-domain>/v1/chat/completions" \
  --set inference.llmUseHttps=true \
  --set inference.modelName="eunomatix/logster-26b-a4b-it" \
  --timeout 20m

If your registry layout can't follow the default naming

Every image also has an individual override (vector.image, redis.image, observability.prometheus.image, images.registry for the service and Elasticsearch images, and image.repository for the classifier image) — set those instead of global.imageRegistry.

The Prometheus image ships in the bundle, and observability is enabled by default (observability.enabled: true) — disable it with --set observability.enabled=false if you don't want it.

inference.llmEndpoint — the classifier URL from Part A. This is the only wiring between the two charts.
inference.modelName — the model the inference service requests from that endpoint. Set it to eunomatix/logster-26b-a4b-it — the model the LogsterClassifier serves. Leaving it unset falls back to the built-in default, which is not the shipped production model.
elasticsearch.password — generated once here and stored in the logster-elasticsearch Secret. Retrieve it later with: oc get secret logster-elasticsearch -n logster -o jsonpath='{.data.ELASTIC_PASSWORD}' | base64 -d
license.file — the signed license from your delivery. The inference service will not start without it.

B3 — Watch it converge

oc get pods -n logster -w

Within ~5–10 minutes (image pulls dominate the first install) every pod reaches Running / READY 1/1, the elasticsearch-init Job shows Completed, and Kafka reports Ready:

oc get kafka logster -n logster        # READY: True
oc get kafkatopic -n logster           # 8 topics, READY: True

Note

It is normal for the inference pod to restart a few times while Kafka and Redis are still coming up — the stack converges on its own.

If a pod sticks in ImagePullBackOff, see Troubleshooting image pulls.

B4 — The routes the application creates

The application chart publishes three routes. List them:

oc get routes -n logster

Route	Purpose	Consumed by
`dashboard`	The Logster web console (edge TLS).	Analysts' browsers.
`logster-kafka-bootstrap`	Kafka bootstrap listener (TLS passthrough).	Endpoint log shippers — the initial connect address.
`logster-kafka-0`	Kafka per-broker listener (TLS passthrough, one per broker).	Endpoint log shippers — brokers the client is redirected to.

The two logster-kafka-* routes are the Kafka routes your endpoints ship logs to. They are covered in B6. (Confirm the exact route names on your cluster with oc get routes -n logster — Strimzi names them <cluster>-kafka-bootstrap and <cluster>-kafka-<broker-id>.)

B5 — Open the dashboard

oc get route dashboard -n logster -o jsonpath='https://{.spec.host}{"\n"}'

Open that URL in a browser. First login is admin / admin; you are required to set a new password immediately.

B6 — Ship endpoint logs to Logster

Endpoints publish raw logs to Logster's Kafka external listener, exposed through the OpenShift router as TLS-passthrough routes. A Kafka client first connects to the bootstrap route, then is redirected to the per-broker route(s) advertised back to it — so your endpoints must be able to reach both Kafka routes.

1. Get the bootstrap address. This is the authoritative connect address (host + port 443):

oc get kafka logster -n logster \
  -o jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'

2. Point the endpoint shipper at the bootstrap route (over TLS). Update the log shipper's Kafka output on every endpoint. The routes are TLS-passthrough with a self-signed (Kafka cluster CA) certificate, so the shipper must connect over TLS with ssl.verification_mode: none — that keeps the transport encrypted without distributing the CA to every endpoint.

Both Kafka routes must resolve from the endpoint

After connecting to the bootstrap route, the Kafka client is redirected to the advertised per-broker route host (logster-kafka-0-...). Every endpoint must therefore be able to resolve both the bootstrap and per-broker route hostnames in DNS — both point to the OpenShift ingress/router IP — and reach them on port 443. If only the bootstrap name resolves, the initial handshake succeeds but no data flows. List the exact hostnames with oc get routes -n logster and add them to endpoint DNS (or hosts files) as needed.

To connect windows endpoints refer to Connect Windows Endpoints

To connect linux endpoints refer to Connect Linux Endpoints

B7 — Ship Windows endpoints via a Splunk Universal Forwarder

If your Windows fleet ships Sysmon with a Splunk Universal Forwarder (UF) instead of Winlogbeat, Logster's Splunk receiver accepts the UF's raw-TCP stream (sendCookedData=false) on port 9997.

Why not a Route?

OpenShift Routes are HTTP / TLS-SNI only and cannot carry the UF's raw TCP stream. So — unlike the dashboard and API — the Splunk receiver is not exposed through a Route. In-cluster shippers use the ClusterIP Service splunk-receiver:9997; out-of-cluster endpoints reach it through a LoadBalancer or NodePort Service.

1. Choose how out-of-cluster endpoints reach it. By default the receiver is exposed with a NodePort Service — reachable from outside the cluster out of the box on every cluster, no load-balancer infrastructure required. If your cluster has a load-balancer provider (a cloud LB, or MetalLB on bare metal) and you prefer a stable IP on port 9997, switch to LoadBalancer (either with --set or in a values file you pass with -f):

splunkReceiver:
  service:
    # NodePort     — <node-ip>:<nodePort> — DEFAULT (works everywhere)
    # LoadBalancer — external IP on 9997 (needs a cloud LB or MetalLB)
    # ClusterIP    — in-cluster shippers only (no external access)
    type: NodePort
    # NodePort only (optional): pin the external port (30000-32767)
    nodePort: ""
    # LoadBalancer only (optional): request a specific address
    loadBalancerIP: ""
    # LoadBalancer/NodePort (recommended): restrict who may connect
    loadBalancerSourceRanges: []

To switch to a LoadBalancer on an existing install:

helm upgrade logster ./logster -n logster --reuse-values \
  --set splunkReceiver.service.type=LoadBalancer

2. Find the address endpoints should connect to.

NodePort (default)LoadBalancer

oc get svc splunk-receiver-ingest -n logster \
  -o jsonpath='{.spec.ports[0].nodePort}{"\n"}'

Endpoints connect to <any-worker-node-ip>:<nodePort>.

oc get svc splunk-receiver-ingest -n logster \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'

Endpoints connect to <that-ip>:9997.

3. Configure the endpoints. Follow Connect Windows Endpoints, setting the UF's outputs.conf server to the address from step 2 (instead of <appliance-host>:9997).

Encrypt untrusted paths

A plain L4 LoadBalancer/NodePort carries no TLS — the UF stream is unencrypted on the wire. For endpoints reaching the cluster over an untrusted network, terminate TLS at the receiver (see below) and scope loadBalancerSourceRanges to the CIDRs your endpoints ship from.

4. (Optional) Terminate TLS at the receiver. This works with the embedded config — no image rebuild. Issue the receiver a server certificate whose SAN matches the address the UF connects to (the hostname or the node / LB IP from step 2), put it in a kubernetes.io/tls Secret, then enable TLS pointing at it:

# The SAN must cover exactly what the UF puts in outputs.conf `server = ...`
openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr \
  -subj "/CN=logster-cluster"
printf 'subjectAltName=DNS:logster-cluster,IP:10.0.0.10\nextendedKeyUsage=serverAuth\n' > ext.cnf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt -days 365 -extfile ext.cnf

oc create secret tls splunk-receiver-tls \
  --cert=server.crt --key=server.key -n logster

helm upgrade logster ./logster -n logster --reuse-values \
  --set splunkReceiver.tls.enabled=true \
  --set splunkReceiver.tls.existingSecret=splunk-receiver-tls

The chart mounts the Secret and points the receiver at it (via SPLUNK_TLS_ENABLED / SPLUNK_TLS_CERT_FILE / SPLUNK_TLS_KEY_FILE). Confirm the listener came up in TLS mode:

oc logs -n logster deploy/splunk-receiver | grep "listening on"
# ... "Splunk receiver listening on 0.0.0.0:9997 (tls=True) ..."

Then configure the UF — see Connect Windows Endpoints → TLS to the receiver. The receiver does not validate client certificates, but the UF still requires a clientCert of its own before it will verify the server's.

The UF falls back to plaintext silently

If the UF's TLS settings are wrong (SSL not switched on, or an unreadable sslRootCAPath), it keeps forwarding cleartext into the TLS listener and reports no error on the tcpout path — every event is dropped. The receiver is where you see it:

TLS handshake failed from <ip>: [SSL: WRONG_VERSION_NUMBER] wrong version number

After enabling TLS, always confirm logster_splunk_receiver_events_total {outcome="produced"} is still climbing.

Expected log noise

With TLS on, the pod's readiness probe opens a plain TCP socket to 9997 every 15s, so the receiver logs a recurring TLS handshake failed ... UNEXPECTED_EOF_WHILE_READING from the in-cluster kubelet address. This is the probe, not an endpoint — harmless.

Troubleshooting image pulls

If a pod sticks in ImagePullBackOff:

oc get events -n logster --field-selector reason=Failed | grep -i pull

unauthorized → the regcred secret has wrong credentials, or the repository is private and the pod's service account isn't linked to the secret (oc secrets link default regcred --for=pull).
x509: certificate signed by unknown authority → the nodes don't trust the registry CA (see Part 1).
manifest unknown → air-gapped: the image/tag wasn't pushed — re-check 0.4 against images-manifest.txt.

Only the Kafka pods fail to pull

If every chart pod runs but <cluster>-kafka-* / <cluster>-entity-operator-* stay in ImagePullBackOff, the problem is the operator catalog mirror, not your registry org — those images are referenced by the operator, not the charts (see B1). Confirm what the pod is actually asking for:

oc get pod -n logster -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}' \
  | grep -Ei 'kafka|entity-operator'

A registry.redhat.io/... or quay.io/strimzi/... reference here means the operator's related images were not mirrored — ask your cluster administrator to re-run oc-mirror for that operator.

Uninstall

helm uninstall logster -n logster              # application workloads
helm uninstall logster-classifier -n logster   # model server
oc delete pvc --all -n logster                 # additionally removes stored data