Skip to content

Admin Guide: Installation

This page walks you from a fresh clone of Logster to a running stack you can query and explore. By the end, you will have all infrastructure (Kafka, Elasticsearch, Redis, Prometheus, Grafana, Kibana, Logstash) and all five Logster services (normalizer, inference, alerts, api, dashboard) running in Docker, with a verified end-to-end pipeline.

For configuration details after installation, see Installation Parameters. For routine operations after the stack is up, see Daily Operations.

Prerequisites

Requirement Minimum Notes
Docker Engine 24.x With the docker compose v2 subcommand.
RAM available to Docker ~8 GB Elasticsearch requests ~1 GB heap; Kafka and the GNN inference workers add on top.
Free disk ~10 GB Images, Kafka log segments, Elasticsearch indices, and pre-trained models.
Free TCP ports 9092, 9200, 6379, 3000, 5601, 9090, 5001, 8080, 4317, 4318, 3200 Remap in deploy/docker-compose.yml if any are taken.
Operating system Linux, macOS, or Windows with WSL2 Any Docker-supported platform.

You do not need Python or Node.js on the host — every service runs inside a container.

Step 1 — Clone the repository

git clone <your-logster-remote> logster
cd logster

The important top-level directories:

  • deploy/ — Docker Compose stack, shared config, Logstash pipelines, Prometheus config.
  • services/ — The five Python/Node services.
  • libs/ — Shared Python libraries (logster-common, logster-ml).
  • models/models/ — Pre-trained GNN model weights (mounted into the inference container).

Step 2 — Start the infrastructure

Everything runs from the deploy/ directory.

cd deploy/
docker compose up -d

This brings up the infrastructure profile defined in deploy/docker-compose.yml:

  • Kafka (KRaft mode, single broker) on 9092.
  • kafka-init — a one-shot container that creates the eight Logster topics (sysmon-logs, linux-auditd-logs, linux-ebpf-*-logs, normalized-endpoint-events, logster-inference-results, logster-alerts).
  • Elasticsearch 8.13 (single-node, security disabled) on 9200.
  • Redis 7 on 6379 — per-endpoint inference state.
  • Logstash — two pipelines shipping Kafka topics to Elasticsearch indices.
  • Kibana on 5601 — raw event exploration.
  • Prometheus on 9090 and Grafana on 3000 — metrics.
  • Tempo on 3200 and OpenTelemetry Collector on 4317/4318 — distributed traces.

[!NOTE] Give Kafka and Elasticsearch approximately 60 seconds to become healthy before proceeding to Step 3. Starting the services profile before the infrastructure is healthy will produce "connection refused" errors.

Verify the infrastructure

# Elasticsearch should report "status":"green" or "yellow"
curl http://localhost:9200/_cluster/health

# All eight Kafka topics should be listed
docker compose exec kafka \
    kafka-topics --bootstrap-server localhost:9092 --list

Step 3 — Start the Logster services

The services live behind the services Compose profile so they only start on demand.

docker compose --profile services up -d

This builds and starts:

Service Container port Host port Purpose
normalizer — — Raw → NormalizedEvent
inference — — GNN prediction per endpoint window
alerts — — Dedup, correlation, TTP enrichment
api 8080 8080 REST API
dashboard 5000 5001 React UI + Express backend

[!IMPORTANT] The dashboard container listens on 5000 internally but is published to host port 5001. Always use http://localhost:5001 from your browser.

Verify the services

# API health
curl http://localhost:8080/health

# Dashboard backend summary (returns zeros until events start flowing)
curl http://localhost:5001/api/summary

The API should return {"status":"ok","service":"logster-api",...}. The dashboard summary endpoint returns aggregate counts from Elasticsearch — if no events have been ingested yet, most fields will be zero, which is expected.

Step 4 — Open the user interfaces

UI URL Credentials
Logster Dashboard http://localhost:5001 None (auth disabled via DISABLE_AUTH=true)
REST API http://localhost:8080 (/docs for Swagger) None
Grafana http://localhost:3000 admin / logster
Kibana http://localhost:5601 None
Prometheus http://localhost:9090 None

For a full tour of what lives at each URL, see Accessing Logster.

Step 5 — Feed the pipeline with test events

A fresh stack has no data. To see the pipeline work end-to-end, you need to publish events into one of the raw Kafka topics. The canonical load generator lives in stress-tests/.

In production, endpoints ship events through Winlogbeat, the Splunk Universal Forwarder, rsyslog, or syslog-ng into Kafka. The dev stack skips all of that and lets you publish straight to Kafka — see Splunk Integration Guide and Key Features: Data Ingestion for the full production ingestion surface.

Confirm events are arriving

curl 'http://localhost:9200/logster-events/_count'
curl 'http://localhost:9200/logster-inferences/_count'

Both counts should climb as events flow through the pipeline.

Step 6 — Tear down

cd deploy/
docker compose --profile services down     # stop services only
docker compose down                          # stop everything
docker compose down -v                       # stop everything AND delete volumes

[!WARNING] docker compose down -v is a "factory reset" — it drops Kafka logs, Elasticsearch indices, Redis state, Prometheus/Grafana volumes, and Tempo traces. Only use it when you want a fully clean slate.

Common installation issues

Services fail with "connection refused to kafka:9092"

Cause: Kafka was not ready when the service started.

Fix: Wait approximately 60 seconds after docker compose up -d before starting the services profile, or simply restart the services:

docker compose --profile services restart

Elasticsearch reports status: red

Cause: Usually disk pressure or low memory.

Fix: Check docker compose logs elasticsearch. The dev stack uses ES_JAVA_OPTS=-Xms512m -Xmx512m — increase it in deploy/docker-compose.yml if you give Docker more RAM.

Inference service fails with "model file not found"

Cause: The inference container mounts models/models/ read-only at /app/models, and the path referenced in model.path or model.linux_model_path does not exist.

Fix: See Model Deployment for the procedure to verify model paths and swap in retrained weights.

A required port is already in use

Cause: Another process on the host is bound to one of the ports listed in deploy/docker-compose.yml.

Fix: Either stop the conflicting process, or change the host-side port mapping. Only edit the left side of the "HOST:CONTAINER" string.

For the full troubleshooting matrix, see the Troubleshooting Guide.

Next steps