Observability guide

Observability with Grafana

Accessing Telemetry Data in Grafana

Accessing Grafana Dashboards

1. Navigate to the Grafana web interface at:

   https://your-domain:5116
   

2. Log in using the administrator credentials configured in your grafana.ini configuration file.

3. From the left navigation sidebar, select "Dashboards" to view available dashboard options.

4. Locate and open the "LLM-Inspect" dashboard, which provides comprehensive observability and traceability metrics for the system.

Analyzing OpenTelemetry Traces

The LLM-Inspect dashboard includes a dedicated Telemetry section that enables detailed trace analysis:

1. Within the Telemetry row, you'll find:
2. A time-series graph showing trace metrics over time
3. A detailed table listing all captured traces with associated metadata

1. Each trace entry in the table contains a clickable trace ID. These IDs serve as unique identifiers for individual request traces.

2. Clicking a trace ID reveals a detailed view of the request journey, including:

   • Complete request path through system services
   • Individual span information and timing
   • Associated span attributes and metadata
   • Service dependencies and interactions

1. You can also view the span attributes for detail debugging and observability.

   For examle this span shows the request body and headers as they land on the LLM-GW server

   

This trace visualization allows administrators to: - Track request flow across distributed services - Identify performance bottlenecks - Debug service interactions - Monitor system behavior and health

These telemetry insights are crucial for maintaining optimal system performance and quickly resolving any issues that may arise.

Monitoring User Interactions

Access user activity dashboard: - Message counts - Token usage - Failed validations - Response times