Troubleshooting Guide

Troubleshooting

Common Issues and Solutions

1. Service Failed to Start

   # Check logs
   docker compose logs [service-name]
   
   # Verify configurations
   docker compose config
   
   # Check resource usage
   docker stats
   

2. Authentication Issues

   # Check Keycloak logs
   docker compose logs keycloak
   
   # Verify network connectivity (find the Keycloak network, then inspect it)
   docker network ls --filter name=keycloak_net
   docker network inspect <network-name>
   

Viewing Logs

Access and analyze logs through InspectBoard and Docker:

1. InspectBoard - API Gateway Audit Logs

Administrators can review recent API Gateway activity in InspectBoard, the LLMInspect observability dashboard.

• Open the Audit Logs:

  • Log in to InspectBoard at https://your-domain:5119 using your LLMInspect credentials
  • Open Chat Analytics or API Analytics from the left navigation
  • Scroll to the Detail section and find the API Gateway audit logs panel

• Adjust Time Range: If you don't see the entries you expect:

  • Open the time range selector in the top bar
  • Select a longer preset range (e.g., Last 24 hours or Last 7 days)
  • The panel only shows log entries from the selected time range

• Filter Logs:

  • Type the text you are looking for, such as an error message, in the filter lines field
  • Press Enter to show only the log lines that contain that text
  • To clear the filter, delete the text and press Enter again

• Additional Features:

  • Click on a log line to expand it and see the full entry
  • The dashboard refreshes automatically every 30 seconds; click the refresh button in the top bar to update immediately

For more details, see Viewing Audit Logs in the Observability Guide. To view logs from individual LLMInspect services, use Docker Container Logs.

2. Local Deployment Logs

• Accessing Logs:
  • Navigate to the logs folder in your LLMInspect installation directory:

    cd /full/path/to/llminspect/logs
    

  • This directory contains raw log files organized by service component and date, allowing for manual inspection when necessary.
• Log Format and Content: Understand the format of the logs (e.g., JSON, plain text) and the information they contain, such as timestamps, log level, and service identifiers.

3. Docker Container Logs

• Viewing Real-Time Container Logs:

  docker logs container-name
  

• Monitoring Logs with Timestamps:

  docker logs -f --timestamps container-name
  

• Inspecting Recent Log Entries:

  docker logs --tail 100 container-name
  

• Tips for Effective Log Management:
  • Use docker compose logs for services managed via Docker Compose:

    docker compose logs --follow service-name
    

  • Combine logs from all containers in a Docker Compose setup:

    docker compose logs --tail=50 --follow
    

  • Filter logs by time or keywords using grep or other command-line tools for more specific log analysis.

Additional Tips

• Setting Up Log Rotation: Ensure that log rotation is configured to manage disk space effectively. Docker and system-level log rotation settings should be verified to prevent service disruption due to filled disk spaces.