Skip to content

LLMInspect Basic Deployment Guide

This guide will walk you through the process of deploying LLMInspect using Docker. By following the instructions, you'll be able to set up a basic environment to run the LLMInspect system smoothly.

Overview

LLMInspect is a comprehensive system for managing large language models and their related services. The eunomatix/llminspect repository contains submodules for various microservices required for this product, alongside a docker-compose.yml file and an example override file.

Prerequisites

  • Docker installed
  • Docker Compose installed
  • Basic knowledge of using the terminal and Docker commands

1. Clone the LLMInspect Repository

Start by cloning the main repository, which includes all necessary submodules.

git clone --recurse-submodules https://github.com/eunomatix/llminspect.git cd llminspect

2. Configure the Override File

To customize your deployment, copy the provided example override file and edit it to suit your needs: highlight the things that are important

cp docker-compose.override.yml.example docker-compose.override.yml

Update Your Configuration

Open docker-compose.override.yml in your favorite editor and set the appropriate API keys, credentials, and other local configurations. For instance, you can add sensitive information like:

services:
  api:
    environment:
      - OPENAI_API_KEY=<your-openai-api-key>
  mongo:
    environment:
      - MONGO_INITDB_ROOT_PASSWORD=<your-password>
      - MONGO_INITDB_ROOT_USERNAME=admin

3. Ensure Directory Permissions

Create the required directories on your host system and set the correct permissions to allow Docker containers to write into them.

Directory Creation and Permission Setting

# Create directories
mkdir -p logs/inspectHexagon logs/libreChat meili_data_v1.7
# Set appropriate permissions
chmod -R 775 logs/inspectHexagon logs/libreChat meili_data_v1.7

Ensure the directories are created with appropriate ownership if required:

sudo chmod -R 777 logs/ librechat/images/  meili_data_v1.7/

4. Start the Docker Containers

Now that the configuration files and directory permissions are set up, start all services with Docker Compose:

docker-compose up -d

This command will start all the micro-services defined in the Docker Compose files in detached mode.

5. Configure TLS/SSL

Enable SSL for Key Microservices

Grafana

Ensure that the SSL certificates are correctly mounted in the docker-compose.yml file and configured in the Grafana config file (/etc/grafana/grafana.ini):

services:
  grafana:
    volumes:
      - ./ssl/certificate.crt:/etc/ssl/certificate.crt
      - ./ssl/private.key:/etc/ssl/private.key
    environment:
      - GF_SERVER_PROTOCOL=https
      - GF_SERVER_CERT_FILE=/etc/ssl/certificate.crt
      - GF_SERVER_CERT_KEY=/etc/ssl/private.key

/etc/grafana/grafana.ini

#################################### Server ####################################
[server]
# Protocol (http, https, h2, socket)
protocol = https

domain = https://<your-domain-name>/

#path to cert files same as the volume mounted
cert_file = /etc/ssl/certificate.crt
cert_key = /etc/ssl/private.key

Keycloak

To enable SSL for Keycloak, mount your certificate files and provide the necessary environment variables:

For older versions of keycloak < 18.0.0

services:
  keycloak:
    volumes:
      - ./ssl/certificate.crt:/etc/x509/https/tls.crt
      - ./ssl/private.key:/etc/x509/https/tls.key
    environment:
      - KC_HTTPS_CERTIFICATE_FILE=/etc/x509/https/tls.crt
      - KC_HTTPS_CERTIFICATE_KEY_FILE=/etc/x509/https/tls.key

LLM Gateway

Ensure SSL is enabled by setting ENABLE_HTTPS=true and placing the required certificates in llmgw/Cert directory

Ensure the nginx service or any other reverse proxy handling the traffic is also configured with SSL.

ApiSix

To enable SSL listen on ApiSix service, make the following changes in ./apisix_conf/config.yaml

apisix:
    ssl:
        enable: true
        listen:
            - port: 8116
...

Next you will need to upload your certificate and private key to the ApiSix server.

After you have started the services by running docker-compose up -d you can open a browser and type localhost:9000 which will open the ApiSix dashboard.

Head towards the SSL section in the Navigation bar on the left and upload the certificate.crt and private.key, This will enable SSL on ApiSix

Generating Self-Signed Certificates (Optional)

If you don't have SSL certificates, you can generate self-signed certificates for local development:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout private.key -out certificate.crt

Place the generated certificates in the ssl directory and update the service definitions accordingly.

6. Accessing the Services

Once the services are up and running, you can access them using the configured ports:

  • Grafana: https://localhost:5116
  • Keycloak: https://localhost:4116
  • LLM Gateway (llmgw): https://localhost:8082
  • LibreChat: https://localhost:6116
  • API Gateway: https://localhost:8116

7. Managing the Deployment

Stopping the Services

To stop all services, run:

docker-compose down

Restarting the Services

To restart specific services, use the following command:

docker-compose restart <service-name>

Replace <service-name> with the appropriate service you want to restart.

Deleting the volumes

To stop services and delete the volumes, use the following command

docker-compose down -v

Use this command with caution all save data might be lost.

8. Data Management

Volumes and Directories

Several services use Docker volumes to persist data. Here is a summary of the key data locations:

  • Images:
  • ./librechat/images: Contains user-uploaded images for the LibreChat service.
  • Logs:
  • logs/inspectHexagon: Stores logs for the Inspect Hexagon service.
  • logs/libreChat: Stores logs for the LibreChat service.
  • llmgw/LLM.log: Stores logs for LLMGW service.
  • Databases:
  • meili_data_v1.7: Stores MeiliSearch index data.
  • mongo_data: MongoDB data volume.
  • pgvector: Stores context for RAG in librechat