Get started with AI Gateway

We’ll be using decK for this tutorial, so let’s check that Kong Gateway is running and that decK can access it:

deck gateway ping

If everything is running, then you should get the following response:

Successfully connected to Kong!
Kong version: 3.9.0.0

Successfully Konnected to the Kong organization!

Create a Service to contain the Route for the LLM provider:

echo '
_format_version: "3.0"
services:
  - name: llm-service
    url: http://localhost:32000
' | deck gateway apply -

The URL can point to any empty host, as it won’t be used by the plugin.

Create a Route for the LLM provider. In this example we’re creating a chat route, so we’ll use /chat as the path:

echo '
_format_version: "3.0"
routes:
  - name: openai-chat
    service:
      name: llm-service
    paths:
    - "/chat"
' | deck gateway apply -

Enable the AI Proxy plugin to create a chat route:

echo '
_format_version: "3.0"
plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      model:
        provider: openai
' | deck gateway apply -

In this example, we’re setting up the plugin with minimal configuration, which means:

• The client is allowed to use any model in the openai provider and must provide the model name in the request body.
• The client must provide an Authorization header with an OpenAI API key.

If needed, you can restrict the models that can be consumed by specifying the model name explicitly using the config.model.name parameter.

You can also provide the OpenAI API key directly in the configuration with the config.auth.header_name and config.auth.header_value parameters so that the client doesn’t have to send them.

To validate, you can send a POST request to the /chat endpoint, using the correct input format. Since we didn’t add the model name and API key in the plugin configuration, make sure to include them in the request:

 curl -X POST "$KONNECT_PROXY_URL/chat" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"\
     -H "Authorization: Bearer $OPENAI_KEY" \
     --json '{
       "model": "gpt-4",
       "messages": [
         {
           "role": "user",
           "content": "Say this is a test!"
         }
       ]
     }'

 curl -X POST "http://localhost:8000/chat" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"\
     -H "Authorization: Bearer $OPENAI_KEY" \
     --json '{
       "model": "gpt-4",
       "messages": [
         {
           "role": "user",
           "content": "Say this is a test!"
         }
       ]
     }'

You should get a 200 OK response, and the response body should contain This is a test.