Skip to content
Kong Docs are moving soon! Our docs are migrating to a new home. You'll be automatically redirected to the new site in the future. In the meantime, view this page on the new site!
Kong Logo | Kong Docs Logo
  • Docs
    • Explore the API Specs
      View all API Specs View all API Specs View all API Specs arrow image
    • Documentation
      API Specs
      Kong Gateway
      Lightweight, fast, and flexible cloud-native API gateway
      Kong Konnect
      Single platform for SaaS end-to-end connectivity
      Kong AI Gateway
      Multi-LLM AI Gateway for GenAI infrastructure
      Kong Mesh
      Enterprise service mesh based on Kuma and Envoy
      decK
      Helps manage Kong’s configuration in a declarative fashion
      Kong Ingress Controller
      Works inside a Kubernetes cluster and configures Kong to proxy traffic
      Kong Gateway Operator
      Manage your Kong deployments on Kubernetes using YAML Manifests
      Insomnia
      Collaborative API development platform
  • Plugin Hub
    • Explore the Plugin Hub
      View all plugins View all plugins View all plugins arrow image
    • Functionality View all View all arrow image
      View all plugins
      AI's icon
      AI
      Govern, secure, and control AI traffic with multi-LLM AI Gateway plugins
      Authentication's icon
      Authentication
      Protect your services with an authentication layer
      Security's icon
      Security
      Protect your services with additional security layer
      Traffic Control's icon
      Traffic Control
      Manage, throttle and restrict inbound and outbound API traffic
      Serverless's icon
      Serverless
      Invoke serverless functions in combination with other plugins
      Analytics & Monitoring's icon
      Analytics & Monitoring
      Visualize, inspect and monitor APIs and microservices traffic
      Transformations's icon
      Transformations
      Transform request and responses on the fly on Kong
      Logging's icon
      Logging
      Log request and response data using the best transport for your infrastructure
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
1.6.x (latest)
  • Home icon
  • Kong Gateway Operator
  • Guides
  • AI Gateway
github-edit-pageEdit this page
report-issueReport an issue
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Kong AI Gateway
  • Plugin Hub
  • decK
  • Kong Ingress Controller
  • Kong Gateway Operator
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • unreleased
  • 1.6.x (latest)
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
  • Introduction
    • Overview
    • Deployment Topologies
      • Hybrid Mode
      • DB-less Mode
    • Key Concepts
      • Gateway API
      • Gateway Configuration
      • Managed Gateways
    • Changelog
    • Version Support Policy
    • FAQ
  • Get Started
    • Konnect
      • Install Gateway Operator
      • Create a KonnectExtension
      • Deploy a Data Plane
      • Create a Route
    • Kong Ingress Controller
      • Install Gateway Operator
      • Create a Gateway
      • Create a Route
  • Production Deployment
    • Overview
    • Install
    • Enterprise License
    • Monitoring
      • Metrics
      • Status fields
        • Overview
        • DataPlane
        • ControlPlane
        • Gateway
    • Upgrade Gateway Operator
    • Certificates
      • Using custom CA for signing operator certificates
  • Guides
    • AI Gateway
    • Customization
      • Set data plane image
      • Deploying Sidecars
      • Customizing PodTemplateSpec
      • Defining PodDisruptionBudget for DataPlane
    • Autoscaling Kong Gateway
    • Autoscaling Workloads
      • Overview
      • Prometheus
      • Datadog
    • Hardening
      • Limiting namespaces watched by ControlPlane
    • Upgrading Data Planes
      • Rolling Deployment
      • Blue / Green Deployment
    • Kong Custom Plugin Distribution
    • Managing Konnect entities
      • Architecture overview
      • Gateway Control Plane
      • Service and Route
      • Consumer, Credentials and Consumer Groups
      • Key and Key Set
      • Upstream and Targets
      • Certificate and CA Certificate
      • Vault
      • Data Plane Client Certificate
      • Tagging and Labeling
      • Managing Plugin Bindings by CRD
      • Cloud Gateways - Networks
      • Cloud Gateways - Data Plane Group Configuration
      • Cloud Gateways - Transit Gateways
      • FAQ
    • Migration
      • Migrate Konnect DataPlanes from KGO v1.4.x to v1.5.x
  • Reference
    • Custom Resources
      • Overview
      • GatewayConfiguration
      • ControlPlane
      • DataPlane
      • KongPluginInstallation
    • Understanding KonnectExtension
    • Configuration Options
    • License
    • Version Compatibility
enterprise-switcher-icon Switch to OSS
On this pageOn this page
  • Prerequisites
    • Install Kong Gateway Operator
  • Get Started
  • Call the API

AI Gateway

The AIGateway CRD is an opinionated CRD to simplify getting started with Kong’s AI capabilities.

AIGateway allows you to configure largeLanguageModels and will translate the configuration in to Gateway, HTTPRoute and KongPlugin resources automatically.

Prerequisites: Install the Kong Gateway Operator in your Kubernetes cluster with AI Gateway support enabled.

Prerequisites

Install Kong Gateway Operator

Update the Helm repository:

helm repo add kong https://charts.konghq.com
helm repo update kong

Install Kong Gateway Operator with Helm:

helm upgrade --install kgo kong/gateway-operator -n kong-system --create-namespace  \
  --set image.tag=1.6 \
  --set env.ENABLE_CONTROLLER_AIGATEWAY=true

You can wait for the operator to be ready using kubectl wait:

kubectl -n kong-system wait --for=condition=Available=true --timeout=120s deployment/kgo-gateway-operator-controller-manager

Get Started

Before using AIGateway, you need to provide API credentials for your AI providers. AIGateway supports the following providers:

  • openai
  • azure
  • cohere
  • mistral
echo '
---
apiVersion: v1
kind: Secret
metadata:
  name: acme-ai-cloud-providers
type: Opaque
stringData:
  openai: "<INSERT TOKEN HERE>"
' | kubectl apply -f -

After providing authentication credentials, create a GatewayClass and AIGateway resource. The AIGateway resource contains a list of largeLanguageModels. Each of these will be exposed as individual HTTPRoutes based on the identifier field.

echo '
---
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: kong-ai-gateways
spec:
  controllerName: konghq.com/gateway-operator
---
apiVersion: gateway-operator.konghq.com/v1alpha1
kind: AIGateway
metadata:
  name: kong-aigateway
spec:
  gatewayClassName: kong-ai-gateways
  largeLanguageModels:
    cloudHosted:
    - identifier: marketing-team-classic-chatgpt
      model: gpt-3.5-turbo-instruct
      promptType: completions
      aiCloudProvider:
        name: openai
    - identifier: devteam-chatgpt
      model: gpt-4
      promptType: chat
      defaultPrompts:
      - role: system
        content: "You are a helpful assistant who responds in the style of Sherlock Holmes."
      defaultPromptParams:
        maxTokens: 50 # shorter responses
      aiCloudProvider:
        name: openai
  cloudProviderCredentials:
    name: acme-ai-cloud-providers
' | kubectl apply -f -

Kong Gateway Operator converts the AIGateway definition in to a Gateway and multiple HTTPRoute definitions. The creation of a Gateway results in a ControlPlane and a DataPlane being deployed to handle traffic.

Call the API

Once the ControlPlane and DataPlane pods are running, you can call the API.

❯ kubectl get pods
NAME                                                       READY   STATUS    RESTARTS   AGE
dataplane-kong-aigateway-8w9v2-hb7dn-7c4bdf74d4-lsqsv      1/1     Running   0          12m
controlplane-kong-aigateway-4mtd8-dzlmz-589bfb8fbd-8lrgc   1/1     Running   0          12m

To call the API, fetch the PROXY_IP for the Gateway:

export PROXY_IP=$(kubectl get gateway kong-aigateway -o jsonpath='{.status.addresses[0].value}')

Finally, make a curl request to one of the identifier paths that you defined. If you used the above example, try devteam-chatgpt:

curl $PROXY_IP/devteam-chatgpt -H 'Content-Type: application/json' -X POST -d '{
    "messages": [
        {
            "role": "user",
            "content": "What is the theory of relativity?"
        }
    ]
}'

For more information about how to use the AI plugins, see the plugin hub.

Thank you for your feedback.
Was this page useful?
Too much on your plate? close cta icon
More features, less infrastructure with Kong Konnect. 1M requests per month for free.
Try it for Free
  • Kong
    Powering the API world

    Increase developer productivity, security, and performance at scale with the unified platform for API management, service mesh, and ingress controller.

    • Products
      • Kong Konnect
      • Kong Gateway Enterprise
      • Kong Gateway
      • Kong Mesh
      • Kong Ingress Controller
      • Kong Insomnia
      • Product Updates
      • Get Started
    • Documentation
      • Kong Konnect Docs
      • Kong Gateway Docs
      • Kong Mesh Docs
      • Kong Insomnia Docs
      • Kong Konnect Plugin Hub
    • Open Source
      • Kong Gateway
      • Kuma
      • Insomnia
      • Kong Community
    • Company
      • About Kong
      • Customers
      • Careers
      • Press
      • Events
      • Contact
  • Terms• Privacy• Trust and Compliance
© Kong Inc. 2025