AI Proxy Advanced

Looking for the plugin's configuration parameters? You can find them in the AI Proxy Advanced configuration reference doc.

The AI Proxy Advanced plugin lets you transform and proxy requests to multiple AI providers and models at the same time. This lets you set up load balancing between targets.

The plugin accepts requests in one of a few defined and standardised formats, translates them to the configured target format, and then transforms the response back into a standard format.

The following table describes which providers and requests the AI Proxy Advanced plugin supports:

How it works

The AI Proxy Advanced plugin will mediate the following for you:

• Request and response formats appropriate for the configured provider and route_type
• The following service request coordinates (unless the model is self-hosted):
  • Protocol
  • Host name
  • Port
  • Path
  • HTTP method
• Authentication on behalf of the Kong API consumer
• Decorating the request with parameters from the config.options block, appropriate for the chosen provider
• Recording of usage statistics of the configured LLM provider and model into your selected Kong log plugin output
• Optionally, additionally recording all post-transformation request and response messages from users, to and from the configured LLM
• Fulfillment of requests to self-hosted models, based on select supported format transformations

Flattening all of the provider formats allows you to standardize the manipulation of the data before and after transmission. It also allows your to provide a choice of LLMs to the Kong consumers, using consistent request and response formats, regardless of the backend provider or model.

This plugin currently only supports REST-based full text responses.

Load balancing

This plugin supports several load-balancing algorithms, similar to those used for Kong upstreams, allowing efficient distribution of requests across different AI models. The supported algorithms include:

• Lowest-usage: The lowest-usage algorithm in AI Proxy Advanced is based on the volume of usage for each model. It balances the load by distributing requests to models with the lowest usage, measured by factors such as prompt token counts, response token counts, or other resource metrics.
• Round-robin (weighted)
• Consistent-hashing (sticky-session on given header value)

Additionally, semantic routing works similarly to load-balancing algorithms like lowest-usage or least-connections, but instead of volume or connection metrics, it uses the similarity score between the incoming prompt and the descriptions of each model. This allows Kong to automatically choose the model best suited for handling the request, based on performance in similar contexts.

Semantic routing

The AI Proxy Advanced plugin supports semantic routing, which enables distribution of requests based on the similarity between the prompt and the description of each model. This allows Kong to automatically select the model that is best suited for the given domain or use case.

By analyzing the content of the request, the plugin can match it to the most appropriate model that is known to perform better in similar contexts. This feature enhances the flexibility and efficiency of model selection, especially when dealing with a diverse range of AI providers and models.

Request and response formats

The plugin’s config.route_type should be set based on the target upstream endpoint and model, based on this capability matrix:

The following upstream URL patterns are used:

While only the Llama2 and Mistral models are classed as self-hosted, the target URL can be overridden for any of the supported providers. For example, a self-hosted or otherwise OpenAI-compatible endpoint can be called by setting the same config.model.options.upstream_url plugin option.

Input formats

Kong will mediate the request and response format based on the selected config.provider and config.route_type, as outlined in the table above.

The Kong AI Proxy accepts the following inputs formats, standardized across all providers; the config.route_type must be configured respective to the required request and response format examples:

{
    "messages": [
        {
            "role": "system",
            "content": "You are a scientist."
        },
        {
            "role": "user",
            "content": "What is the theory of relativity?"
        }
    ]
}

{
    "prompt": "You are a scientist. What is the theory of relativity?"
}

Response formats

Conversely, the response formats are also transformed to a standard format across all providers:

{
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "The theory of relativity is a...",
                "role": "assistant"
            }
        }
    ],
    "created": 1707769597,
    "id": "chatcmpl-ID",
    "model": "gpt-4-0613",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 5,
        "prompt_tokens": 26,
        "total_tokens": 31
    }
}

{
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "text": "The theory of relativity is a..."
        }
    ],
    "created": 1707769597,
    "id": "cmpl-ID",
    "model": "gpt-3.5-turbo-instruct",
    "object": "text_completion",
    "usage": {
        "completion_tokens": 10,
        "prompt_tokens": 7,
        "total_tokens": 17
    }
}

The request and response formats are loosely based on OpenAI. See the sample OpenAPI specification for more detail on the supported formats.

Get started with the AI Proxy plugin

• Configuration reference
• Basic configuration example
• Learn how to use the plugin with different providers:
  • OpenAI
  • Cohere
  • Azure
  • Anthropic
  • Mistral
  • Llama2

All AI Gateway plugins

• AI Proxy
• AI Proxy Advanced
• AI Request Transformer
• AI Response Transformer
• AI Semantic Cache
• AI Semantic Prompt Guard
• AI Rate Limiting Advanced
• AI Azure Content Safety
• AI Prompt Template
• AI Prompt Guard
• AI Prompt Decorator