Enterprise Architecture

Figure: End-to-end flow of a GenAI request through the LLMInspect platform.

The architecture of LLMInspect is designed to offer a robust, secure, and scalable framework that governs every GenAI interaction between your users and Large Language Model providers — whether those models are hosted locally or accessed through public APIs. As shown in the diagram above, all traffic converges on the LLMInspect Gateway, which validates, inspects, and logs each request so the organization retains complete control over data flow and compliance with security protocols.

Requests enter the platform through two channels. API Clients send queries programmatically and are verified by the authentication service for API users, while Chat Clients interact conversationally through InspectChat. Before a prompt reaches a model, it can be enriched by the RAG service, which incorporates a vector database and feature store with its own LLM to provide relevant context to prompts. All queries — from either channel — are then routed to LLMInspect.

At the core, the LLMInspect Gateway applies a layer of inline safeguards to every prompt and response. A PII Masking Service protects against PII leakage, an NLP / Machine Learning module performs negative sentiment detection, and the gateway inspects all outbound queries to prevent data-loss-prevention (DLP) incidents. Approved requests are then dispatched through LLM Routing (on-demand or automatic) to the appropriate provider, including image models such as DALL-E and text models such as ChatGPT and Gemini.

Every GenAI transaction is recorded in InspectDB, the platform's security database, giving the organization a complete and auditable record of AI usage. For observability, SOC Analysts analyze chat and query activity through SIEM dashboards — integrating with tools such as Splunk and Elastic — to perform data-leakage analysis and maintain full visibility over GenAI queries.

Together, these components allow enterprises to harness the power of AI responsibly: enforcing strict control over data security and compliance while preserving the flexibility to access a wide range of public and private LLM providers.