Set up AI Proxy Advanced with Anthropic in Kong Gateway
Create a Gateway Service and a Route, then enable the AI Proxy Advanced plugin, configure it with the Anthropic provider, then add the model and your API key.
Prerequisites
Kong Konnect
This is a Konnect tutorial and requires a Konnect personal access token.
-
Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
-
Export your token to an environment variable:
export KONNECT_TOKEN='YOUR_KONNECT_PAT'
Copied to clipboard! -
Run the quickstart script to automatically provision a Control Plane and Data Plane, and configure your environment:
curl -Ls https://get.konghq.com/quickstart | bash -s -- -k $KONNECT_TOKEN --deck-output
Copied to clipboard!This sets up a Konnect Control Plane named
quickstart
, provisions a local Data Plane, and prints out the following environment variable exports:export DECK_KONNECT_TOKEN=$KONNECT_TOKEN export DECK_KONNECT_CONTROL_PLANE_NAME=quickstart export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com export KONNECT_PROXY_URL='http://localhost:8000'
Copied to clipboard!Copy and paste these into your terminal to configure your session.
Configure the plugin
To set up AI Proxy Advanced with Anthropic, we need to specify the model and Anthropic API version to use.
In this example, we’ll use the Claude 2.1 model and version 2023-06-01 of the API:
echo '
_format_version: "3.0"
plugins:
- name: ai-proxy-advanced
config:
targets:
- route_type: llm/v1/chat
auth:
header_name: x-api-key
header_value: "${{ env "DECK_ANTHROPIC_API_KEY" }}"
model:
provider: anthropic
name: claude-2.1
options:
anthropic_version: '2023-06-01'
max_tokens: 1024
' | deck gateway apply -
Validate
Send a request to the Route to validate.
curl -X POST "$KONNECT_PROXY_URL/anything" \
-H "Accept: application/json"\
-H "Content-Type: application/json" \
--json '{
"messages": [
{
"role": "system",
"content": "You are a mathematician"
},
{
"role": "user",
"content": "What is 1+1?"
}
]
}'
curl -X POST "http://localhost:8000/anything" \
-H "Accept: application/json"\
-H "Content-Type: application/json" \
--json '{
"messages": [
{
"role": "system",
"content": "You are a mathematician"
},
{
"role": "user",
"content": "What is 1+1?"
}
]
}'
Cleanup
Clean up Konnect environment
If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.