Set up AI Proxy with Azure OpenAI Service - Plugin

This guide walks you through setting up the AI Proxy plugin with Azure OpenAI Service.

For all providers, the Kong AI Proxy plugin attaches to route entities.

It can be installed into one route per operation, for example:

OpenAI chat route
Cohere chat route
Cohere completions route

Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the AI Proxy plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.

Prerequisites

Azure OpenAI Service account and subscription
You need a service to contain the route for the LLM provider. Create a service first:

curl -X POST http://localhost:8001/services \
  --data "name=ai-proxy" \
  --data "url=http://localhost:32000"

Remember that the upstream URL can point anywhere empty, as it won’t be used by the plugin.

Provider configuration

Create or locate OpenAI instance

Name: azure_instance
Access key as the header_value

Create or locate model deployment

Once it has instantiated, create (if necessary) a model deployment in this instance. Record its name as your azure_deployment_id:

Set up route and plugin

Now you can create an AI Proxy route and plugin configuration:

Kong Admin API

YAML

Create the route:

curl -X POST http://localhost:8001/services/ai-proxy/routes \
  --data "name=azure-chat" \
  --data "paths[]=~/azure-chat$"

Enable and configure the AI Proxy plugin for Azure, replacing the <azure_ai_access_key> with your own API key:

curl -X POST http://localhost:8001/routes/azure-chat/plugins \
  --data "name=ai-proxy" \
  --data "config.route_type=llm/v1/chat" \
  --data "config.auth.header_name=api-key" \
  --data "config.auth.header_value=<azure_ai_access_key>" \
  --data "config.model.provider=azure" \
  --data "config.model.name=gpt-35-turbo" \
  --data "config.model.options.azure_instance=ai-proxy-regression" \
  --data "config.model.options.azure_deployment_id=kong-gpt-3-5"

name: azure-chat
paths:
  - "~/azure-chat$"
methods:
  - POST
plugins:
  - name: ai-proxy
    config:
      route_type: "llm/v1/chat"
      auth:
        header_name: "api-key"
        header_value: "<azure_ai_access_key>"  # add your own 'Azure OpenAI' access key
      model:
        provider: "azure"
        name: "gpt-35-turbo"
        options:
          azure_instance: "ai-proxy-regression"
          azure_deployment_id: "kong-gpt-3-5"

Test the configuration

Make an llm/v1/chat type request to test your new endpoint:

curl -X POST http://localhost:8000/azure-chat \
  -H 'Content-Type: application/json' \
  --data-raw '{ "messages": [ { "role": "system", "content": "You are a mathematician" }, { "role": "user", "content": "What is 1+1?"} ] }'

Previous Set up AI Proxy with Anthropic

Next Set up AI Proxy with Bedrock