Set up AI Proxy Advanced with Anthropic in Kong Gateway

Uses: Kong Gateway AI Gateway decK
Related Resources
Minimum Version
Kong Gateway - 3.8
TL;DR

Create a Gateway Service and a Route, then enable the AI Proxy Advanced plugin, configure it with the Anthropic provider, then add the model and your API key.

Prerequisites

This is a Konnect tutorial and requires a Konnect personal access token.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Export your token to an environment variable:

     export KONNECT_TOKEN='YOUR_KONNECT_PAT'
    
    Copied to clipboard!
  3. Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:

     curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-output
    
    Copied to clipboard!

    This sets up a Konnect Control Plane named quickstart, provisions a local Data Plane, and prints out the following environment variable exports:

     export DECK_KONNECT_TOKEN=$KONNECT_TOKEN
     export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart
     export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
     export KONNECT_PROXY_URL='http://localhost:8000'
    
    Copied to clipboard!

    Copy and paste these into your terminal to configure your session.

Configure the plugin

To set up AI Proxy Advanced with Anthropic, we need to specify the model and Anthropic API version to use.

In this example, we’ll use the Claude 2.1 model and version 2023-06-01 of the API:

echo '
_format_version: "3.0"
plugins:
  - name: ai-proxy-advanced
    config:
      targets:
      - route_type: llm/v1/chat
        auth:
          header_name: x-api-key
          header_value: "${{ env "DECK_ANTHROPIC_API_KEY" }}"
        model:
          provider: anthropic
          name: claude-2.1
          options:
            anthropic_version: '2023-06-01'
            max_tokens: 1024
' | deck gateway apply -
Copied to clipboard!

Validate

Send a request to the Route to validate.

 curl -X POST "$KONNECT_PROXY_URL/anything" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json" \
     --json '{
       "messages": [
         {
           "role": "system",
           "content": "You are a mathematician"
         },
         {
           "role": "user",
           "content": "What is 1+1?"
         }
       ]
     }'
Copied to clipboard!

Cleanup

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

Did this doc help?

Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!