LLM Evidence Highlighting — Design Spec

Problem

When the LLM flags an inference window as an attack, the analyst sees the full graph — 30-80+ nodes of mixed operational noise and malicious activity. The LLM's rationale describes the attack chain in prose ("powershell.exe spawned cmd.exe which wrote to the startup folder"), but there's no structured mapping from those steps back to specific nodes in the graph visualizer or events in the investigation panel. The analyst has to mentally cross-reference prose descriptions against a dense graph to find the nodes that matter.

Solution

On-demand evidence extraction: when an analyst opens the Investigate panel on an attack prediction, a second LLM call extracts structured per-step node references from the attack chain. The results highlight malicious events in the event list and evidence nodes on the graph canvas, with per-step interactive linkage from the rationale text.

Design Decisions

• On-demand, not at inference time. The second LLM call happens when the analyst clicks Investigate or Graph — not during the inference pipeline. This keeps the critical verdict path untouched and only incurs LLM cost when an analyst actually drills in.
• Grounded annotation, not re-analysis. The second call receives the original verdict, rationale, and techniques alongside the graph. It maps existing rationale steps to node IDs — it doesn't decide what's malicious. This prevents the second call from diverging from the first.
• Same LLM. Uses the same model and endpoint as the verdict call.
• Attack chain only. Highlights the tight kill chain (3-8 nodes typically), not the broader neighborhood.
• Per-step linkage. Each rationale step maps to specific node IDs, enabling step-by-step walk-through.
• Forward-looking only. No backfill of historical inferences.
• Graceful degradation. Evidence extraction failure doesn't break existing functionality.

Architecture & Data Flow

User clicks Investigate on an attack inference
  → InvestigationModal opens, events load from ES as normal
  → POST /api/evidence/{inference_id} fires immediately
  → Express proxy → Python API service:
      1. Check logster-graphs doc for cached `evidence` field → return if found
      2. Fetch graph snapshot from logster-graphs ES index
      3. Fetch inference result from logster-inferences ES index
         (need: lc_verdict, lc_rationale, lc_techniques, lc_prompt)
      4. Build graph text for the LLM:
         a. Prefer lc_prompt from inference result (exact text from first call)
         b. Fall back: re-serialize from graph doc's nodes[]/edges[] arrays
      5. Call LLM: graph text + original verdict + rationale + techniques
         + instruction: "annotate each rationale step with node IDs"
      6. Parse response, validate node_ids against graph doc's nodes[]
      7. Enrich: map valid node_ids → originating_event_ids
      8. Cache: ES partial update on graph doc with `evidence` field
      9. Return evidence + event_id mapping
  → Frontend receives evidence:
      → InvestigationModal: evidence-linked events float to top + highlighted
      → RationaleText steps become clickable
  → User clicks Graph button:
      → GraphModal opens, evidence already in React state
      → Evidence nodes highlighted on Cytoscape canvas
      → Per-step click in rationale → spotlights specific nodes

Component Diagram

┌─────────────────────────────────────────────────────────┐
│ Dashboard Client (React)                                │
│                                                         │
│  ┌──────────────────┐    ┌──────────────────────────┐   │
│  │ EvidenceProvider  │◄───│ InvestigationModal       │   │
│  │ (React Context)   │    │  - InferenceSummary      │   │
│  │                   │    │  - RationaleText (click) │   │
│  │ state per         │    │  - EventList (reorder)   │   │
│  │ inference_id:     │    │  - EvidenceProgressBar   │   │
│  │  idle/loading/    │◄───│                          │   │
│  │  success/error    │    └──────────────────────────┘   │
│  │                   │                                   │
│  │                   │◄───┌──────────────────────────┐   │
│  │                   │    │ GraphModal                │   │
│  │                   │    │  - CytoscapeGraph         │   │
│  │                   │    │    (evidence highlight)   │   │
│  │                   │    │  - DebugPanel             │   │
│  │                   │    │    RationaleText (click)  │   │
│  └────────┬─────────┘    └──────────────────────────┘   │
│           │ POST /api/evidence/:id                       │
└───────────┼─────────────────────────────────────────────┘
            │
┌───────────▼─────────────────────────────────────────────┐
│ Dashboard Server (Express)                              │
│  routes/evidence.js — proxy to API service              │
└───────────┬─────────────────────────────────────────────┘
            │ HTTP forward
┌───────────▼─────────────────────────────────────────────┐
│ API Service (FastAPI)                                   │
│  POST /api/evidence/{inference_id}                      │
│  └─ evidence.py                                         │
│     - fetch graph doc from ES (logster-graphs)          │
│     - fetch inference result from ES (logster-inferences)│
│     - build LLM prompt (graph + verdict + rationale)    │
│     - call LLM (OpenAI-compatible, same model)          │
│     - parse & validate response                         │
│     - cache evidence in graph doc (ES partial update)   │
└─────────────────────────────────────────────────────────┘

Backend: Evidence Extraction Endpoint

Endpoint

POST /api/evidence/{inference_id} on the Python API service (services/api/src/logster_api/).

New Module

services/api/src/logster_api/evidence.py containing: - Evidence extraction prompt template - Graph re-serialization from doc arrays (adjacency + cypher formats) - LLM call logic (OpenAI-compatible chat completion) - Response parsing and node ID validation - ES caching logic

Configuration

The API service reads from service-config.yaml (same file all services use): - logster_classifier.model — LLM model name - logster_classifier.service_url — LLM endpoint URL - logster_classifier.graph_format — adjacency or cypher - logster_classifier.timeout — HTTP timeout for LLM call

OPENROUTER_API_KEY env var added to the API service container in docker-compose.yml.

Express Proxy

New route in services/dashboard/server/src/routes/evidence.js: - POST /api/evidence/:inferenceId → forwards to ${API_SERVICE_URL}/api/evidence/${inferenceId} - API_SERVICE_URL env var on the dashboard container (default: http://api:8000) - Uses Node.js built-in fetch (Node 18+) — no new dependency needed

Request Flow Detail

1. Check cache: GET logster-graphs/_doc/{inference_id}
   → If doc has `evidence` field with `status: "success"` → return cached

2. Fetch graph doc: already fetched in step 1
   → 404 if not found → return {status: "unavailable", reason: "graph_not_persisted"}

3. Fetch inference result: GET logster-inferences/_search
   → Filter by inference_id
   → Extract: lc_verdict, lc_rationale, lc_techniques, lc_prompt

4. Guard: if lc_verdict != "attack" → return {status: "not_applicable"}

5. Build graph text:
   → If lc_prompt is non-null and non-empty: use it directly
   → Else: re-serialize from graph doc nodes[]/edges[] using configured graph_format

6. Build LLM messages:
   → System: evidence extraction system prompt
   → User: graph text + verdict block + rationale + techniques + instruction

7. Call LLM:
   → POST to service_url with model, messages, temperature=0
   → Parse JSON response

8. Validate:
   → For each step in response.evidence:
     → For each node_id: check existence in graph doc nodes[]
     → Drop invalid IDs, log warning
   → Map valid node_ids → originating_event_ids from graph doc nodes[]

9. Cache: POST logster-graphs/_update/{doc_id}
   → Add evidence field to the document

10. Return response

Response Schema

Success:

{
  "status": "success",
  "evidence": [
    {
      "step": 1,
      "description": "explorer.exe spawned powershell.exe via command line",
      "node_ids": ["{ABC-DEF-123}", "{GHI-JKL-456}"],
      "event_ids": ["evt-001", "evt-002"]
    },
    {
      "step": 2,
      "description": "powershell.exe wrote encoded script to startup folder",
      "node_ids": ["{GHI-JKL-456}", "{MNO-PQR-789}"],
      "event_ids": ["evt-002", "evt-003"]
    }
  ],
  "all_evidence_node_ids": ["{ABC-DEF-123}", "{GHI-JKL-456}", "{MNO-PQR-789}"],
  "all_evidence_event_ids": ["evt-001", "evt-002", "evt-003"],
  "validation": {
    "total_node_ids_returned": 5,
    "valid_node_ids": 5,
    "dropped_node_ids": 0
  }
}

Error/degradation:

{"status": "unavailable", "reason": "graph_not_persisted"}
{"status": "not_applicable", "reason": "benign_verdict"}
{"status": "error", "reason": "llm_call_failed", "error": "timeout after 30s"}
{"status": "error", "reason": "parse_failed", "error": "invalid JSON in LLM response"}

Concurrency

No distributed lock. Evidence extraction is idempotent — if two analysts trigger it simultaneously for the same inference, both calls proceed, last cache-write wins (results should be near-identical). The cost of an occasional duplicate LLM call is negligible vs. locking complexity.

LLM Prompt & Response Schema

Evidence Extraction System Prompt

You are an evidence annotator for a threat detection system. You will receive:
1. A graph of system activity from an endpoint (processes, files, network connections, etc.)
2. A previous analysis verdict and rationale for this graph

Your task: for each numbered step in the provided rationale, identify the specific
node IDs from the graph that are involved in that step. You are NOT re-analyzing
the graph — the verdict and rationale are authoritative. You are mapping the prose
steps to concrete graph node IDs.

Return a JSON object:
{
  "evidence": [
    {
      "step": <step number from the rationale>,
      "description": "<brief description of what this step shows>",
      "node_ids": ["<node_id_1>", "<node_id_2>"]
    }
  ]
}

Rules:
- Use ONLY node IDs that appear in the graph. Do not invent IDs.
- Each step should reference the nodes directly involved in that step's activity.
- Include both the actor node (e.g., the process) and the target node (e.g., the file written to).
- If a rationale step mentions activity you cannot find specific nodes for, include the step
  with an empty node_ids array.
- Return valid JSON only, no markdown formatting.

Evidence Extraction User Message

## Graph

<graph text — either the original lc_prompt or re-serialized from the graph doc>

## Previous Analysis

Verdict: <lc_verdict>
Attack probability: <lc_attack_prob>
Confidence: <lc_confidence>

Rationale:
<lc_rationale>

Techniques: <lc_techniques as comma-separated list>

## Task

For each numbered step in the rationale above, identify the specific node IDs
from the graph that are involved. Return the JSON evidence structure.

Expected LLM Response

{
  "evidence": [
    {
      "step": 1,
      "description": "explorer.exe spawned powershell.exe",
      "node_ids": ["{GUID-of-explorer}", "{GUID-of-powershell}"]
    },
    {
      "step": 2,
      "description": "powershell.exe executed encoded command writing to startup",
      "node_ids": ["{GUID-of-powershell}", "{startup-file-node-id}"]
    }
  ]
}

Response Parsing

• Parse as JSON; on failure → status: "error", reason: "parse_failed"
• Extract evidence array; on missing → status: "error", reason: "parse_failed"
• For each step:
• Validate step is an integer
• Validate node_ids is an array of strings
• For each node_id: check it exists in the graph doc's nodes[].id
• Drop invalid IDs; if all IDs in a step are invalid, keep the step with empty node_ids
• Enrich each step with event_ids by looking up originating_event_id for each valid node

Graph Re-serialization (Fallback)

When lc_prompt is unavailable, re-serialize from the graph doc's nodes[] and edges[]:

Adjacency format:

Nodes:
[{ABC-DEF}] type=process executable_name=powershell.exe command_line="powershell -enc ..."
[{GHI-JKL}] type=file file_path=C:\Users\...\Startup\malware.vbs

Edges:
[{ABC-DEF}] --WROTE_TO--> [{GHI-JKL}] timestamp=2026-09-20T10:30:00Z

Cypher format:

(n0:Process {id: "{ABC-DEF}", executable_name: "powershell.exe", command_line: "powershell -enc ..."})
(n1:File {id: "{GHI-JKL}", file_path: "C:\\Users\\...\\Startup\\malware.vbs"})
(n0)-[:WROTE_TO {timestamp: "2026-09-20T10:30:00Z"}]->(n1)

The serializer iterates the graph doc arrays directly — no NetworkX reconstruction. Node attributes come from nodes[].attributes; node type from nodes[].type; edge type from edges[].type.

Evidence Caching & Persistence

Evidence is cached as a new field on the logster-graphs ES document:

{
  "evidence": {
    "status": "success",
    "extracted_at": "2026-09-20T10:35:00Z",
    "model": "qwen/qwen3-32b",
    "evidence": [
      {
        "step": 1,
        "description": "...",
        "node_ids": ["...", "..."],
        "event_ids": ["...", "..."]
      }
    ],
    "all_evidence_node_ids": ["..."],
    "all_evidence_event_ids": ["..."],
    "validation": { ... }
  }
}

Written via ES partial update (POST logster-graphs/_update/{doc_id} with doc merge). No TTL — evidence for a given inference is immutable once the verdict is final.

The ES index template (deploy/elasticsearch/logster-graphs.template.json) needs updating to include the evidence mapping. Dynamic mapping would also work but explicit is preferable.

Frontend: State Management

EvidenceContext (React Context)

New file: services/dashboard/client/src/contexts/EvidenceContext.jsx

EvidenceProvider wraps App (or the modal layer)

State shape:
{
  [inference_id]: {
    status: 'idle' | 'loading' | 'success' | 'error',
    evidence: null | EvidenceResponse,
    error: null | string
  }
}

Actions:
- fetchEvidence(inference_id) → sets loading, calls API, sets success/error
- getEvidence(inference_id) → returns current state for that inference

Behavior:
- Deduplicates: if status is 'loading' for an inference_id, don't start another fetch
- Persists across modal open/close (context lives above the modals)
- On success: evidence is available to both InvestigationModal and GraphModal
- Only triggers for attack predictions (benign inferences skip the fetch)

Fetch Trigger

• InvestigationModal mount (attack prediction): if no cached evidence for this inference_id → fetchEvidence(inference_id)
• GraphModal mount (attack prediction, no evidence in context): same trigger as fallback if user goes to Graph directly

Frontend: InvestigationModal Changes

Progress Bar

A slim, non-intrusive progress bar at the top of the modal, below the header. Visible only during loading state. Indeterminate (animated stripe) since we don't know the LLM call duration. Disappears on success/error.

Placement: below the InferenceSummary header area, above the event list. Does not push content down — it's positioned absolutely/overlaid.

Event List Reordering

When evidence arrives (status: 'success'):

1. Compute the set of evidence event IDs: all_evidence_event_ids from the response
2. Partition the event list: evidence events first, then remaining events
3. Within each partition, preserve timestamp order
4. Render with a visual divider between evidence events and the rest

The reorder happens once on evidence arrival. No animation needed — the progress bar's disappearance signals the change.

Event Highlighting

Evidence-linked events receive: - A colored left border (using the theme's attack/danger color) - A subtle background tint - A step badge: small pill showing "Step 1", "Step 2" etc. (an event may appear in multiple steps — show the first) - Non-evidence events remain fully readable (no dimming in the event list — dimming makes sense in the graph but the event list is a reading surface)

Interactive Rationale

RationaleText component changes (used in both InferenceSummary and DebugPanel):

When evidence is available: - Each numbered step becomes a clickable element (hover shows pointer cursor, subtle highlight) - Clicking a step: - In InvestigationModal: scrolls to and pulses the first event in that step's event_ids - In GraphModal: spotlights the step's node_ids on the Cytoscape canvas (dims everything else, similar to existing subgraph spotlight) - Active step gets a visual indicator (left border highlight, background) - Clicking the same step again deselects it (returns to default view)

New file: services/dashboard/client/src/components/common/InteractiveRationale.jsx

This replaces the simple RationaleText component when evidence is available. It receives the evidence data and an onStepClick(stepIndex, nodeIds, eventIds) callback.

Frontend: GraphModal Changes

Evidence Highlight Mode

New highlight mode in CytoscapeGraph, alongside existing highlightSubgraphIndex and highlightTopAttention:

highlightEvidenceNodes prop — an object:

{
  allNodeIds: Set<string>,        // all evidence nodes (for the "show all" mode)
  steps: [                        // per-step data (for step-click spotlight)
    { step: 1, nodeIds: Set<string>, description: "..." },
    ...
  ],
  activeStep: null | number       // which step is currently spotlighted
}

Visual treatment of evidence nodes: - Distinct colored border (different from subgraph borders — use a dedicated evidence color, e.g., amber/gold) - Slight glow effect (box-shadow equivalent in Cytoscape) - Non-evidence nodes dimmed to 30% opacity (same approach as existing subgraph spotlight) - Edges between evidence nodes within the active step are emphasized (thicker, colored) - Edges not part of the evidence path are dimmed

Evidence Toggle

New toolbar button: "Evidence" toggle (similar to existing "Top attention" button): - Off by default when no evidence is available - Auto-enabled when evidence arrives - Toggles the evidence highlight mode on/off - When on: shows evidence nodes highlighted, everything else dimmed - When off: standard graph view (no dimming)

Per-Step Spotlight from DebugPanel

The DebugPanel renders InteractiveRationale (same component as InvestigationModal). Clicking a step: 1. Sets activeStep in the evidence highlight data 2. CytoscapeGraph spotlights only that step's nodes (other evidence nodes return to normal opacity) 3. Canvas pans/zooms to fit the spotlighted nodes 4. Clicking the same step deselects → returns to "all evidence highlighted" mode

Graph Button State

The Graph button in InferenceSummary should indicate when evidence is available: - If evidence is loaded: Graph button gets a subtle indicator (small dot or "Enhanced" label) - If evidence is loading: Graph button is still clickable (evidence will appear when ready)

Error Handling & Graceful Degradation

Scenario	Behavior
Graph not persisted for this inference	No evidence fetch attempted. InvestigationModal and GraphModal work exactly as today.
LLM call fails (timeout, network, 5xx)	Small dismissible banner: "Evidence extraction unavailable." Everything works as before. Retry button in the banner.
LLM response is unparseable	Same as LLM failure — banner, no highlighting, everything else works.
LLM returns node IDs not in the graph	Invalid IDs silently dropped. If ALL IDs in a step are invalid, step shows with "(nodes not found)" note. If ALL steps have no valid IDs, treated as extraction failure.
Benign inference	No evidence fetch. No visual changes.
User closes modal during fetch	Fetch continues in background (context lives above modals). If user reopens, result is there.
Two analysts trigger same inference	Both calls proceed; last cache-write wins. Results should be near-identical.
API service unreachable	Express proxy returns 502. Frontend shows error banner with retry.
persist_llm_io is off AND graph doc lacks sufficient node attributes for re-serialization	Re-serialization uses whatever attributes are in the graph doc. Worst case the LLM sees a sparser graph and produces lower-quality evidence. Not a hard failure.

Dependencies & Prerequisites

1. persist_graphs.enabled: true — required. Evidence extraction needs the graph snapshot in ES. This is already required for the Graph view to work.
2. OPENROUTER_API_KEY on the API service container — new. Added to docker-compose.yml.
3. API_SERVICE_URL on the dashboard container — new. Added to docker-compose.yml (default: http://api:8000).
4. ES index template update — add evidence mapping to logster-graphs.template.json.

Not required: - persist_graphs.debug — not required, but evidence quality is better when the original prompt is available - persist_llm_io — not required, but avoids re-serialization when available

Files to Create / Modify

New Files

File	Purpose
services/api/src/logster_api/evidence.py	Evidence extraction logic: prompt template, LLM call, graph re-serializer, response parser, validator, caching
services/dashboard/server/src/routes/evidence.js	Express proxy route to API service
services/dashboard/client/src/contexts/EvidenceContext.jsx	React context for evidence state management
services/dashboard/client/src/components/common/InteractiveRationale.jsx	Clickable rationale steps component (replaces RationaleText when evidence available)
services/dashboard/client/src/components/investigation/EvidenceProgressBar.jsx	Slim progress bar for evidence loading

Modified Files

File	Change
services/api/src/logster_api/app.py	Add evidence route
services/dashboard/server/src/server.js (or routes index)	Register evidence proxy route
services/dashboard/client/src/App.jsx	Wrap modals with EvidenceProvider
services/dashboard/client/src/components/InvestigationModal.jsx	Trigger evidence fetch on mount, pass evidence to event list and rationale
services/dashboard/client/src/components/InferenceSummary.jsx	Use InteractiveRationale when evidence available
services/dashboard/client/src/components/GraphModal.jsx	Pass evidence highlight data to CytoscapeGraph
services/dashboard/client/src/components/graph/CytoscapeGraph.jsx	Add evidence highlight mode (new prop, styling, toggle)
services/dashboard/client/src/components/graph/DebugPanel.jsx	Use InteractiveRationale for LLM rationale section
services/dashboard/client/src/components/EventCard.jsx	Accept highlight/step props, render evidence styling
services/dashboard/client/src/lib/api.js	Add fetchEvidence(inferenceId) helper
deploy/docker-compose.yml	Add OPENROUTER_API_KEY to api service, add API_SERVICE_URL to dashboard service
deploy/elasticsearch/logster-graphs.template.json	Add evidence field mapping
deploy/service-config.yaml	No changes needed (API service reads existing logster_classifier config)

Out of Scope

• Backfill of historical inferences
• Evidence extraction for benign inferences
• Evidence in the alerts pipeline or API response (future work)
• Batch evidence extraction (e.g., "extract evidence for all attacks in the last 24h")
• Evidence-based alert severity adjustment
• Custom evidence extraction prompts (uses built-in prompt only)