Configuration
This plugin is partially compatible with DB-less mode.
The cluster strategy is not supported in DB-less and hybrid modes. For Kong
Gateway in DB-less or hybrid mode, use the redis
strategy.
Compatible protocols
The AI Rate Limiting Advanced plugin is compatible with the following protocols:
grpc
, grpcs
, http
, https
Parameters
Here's a list of all the parameters which can be used in this plugin's configuration:
-
string required
The name of the plugin, in this case
ai-rate-limiting-advanced
.- If using the Kong Admin API, Konnect API, declarative configuration, or decK files, the field is
name
. - If using the KongPlugin object in Kubernetes, the field is
plugin
.
- If using the Kong Admin API, Konnect API, declarative configuration, or decK files, the field is
-
string
An optional custom name to identify an instance of the plugin, for example
ai-rate-limiting-advanced_my-service
.The instance name shows up in Kong Manager and in Konnect, so it's useful when running the same plugin in multiple contexts, for example, on multiple services. You can also use it to access a specific plugin instance via the Kong Admin API.
An instance name must be unique within the following context:
- Within a workspace for Kong Gateway Enterprise
- Within a control plane or control plane group for Konnect
- Globally for Kong Gateway (OSS)
-
string
The name or ID of the service the plugin targets. Set one of these parameters if adding the plugin to a service through the top-level
/plugins
endpoint. Not required if using/services/{serviceName|Id}/plugins
. -
string
The name or ID of the route the plugin targets. Set one of these parameters if adding the plugin to a route through the top-level
/plugins
endpoint. Not required if using/routes/{routeName|Id}/plugins
. -
string
The name or ID of the consumer the plugin targets. Set one of these parameters if adding the plugin to a consumer through the top-level
/plugins
endpoint. Not required if using/consumers/{consumerName|Id}/plugins
. -
string
The name or ID of the consumer group the plugin targets. If set, the plugin will activate only for requests where the specified group has been authenticated
/plugins
endpoint. Not required if using/consumer_groups/{consumerGroupName|Id}/plugins
. -
boolean default:
true
Whether this plugin will be applied.
-
record required
-
string required default:
consumer
Must be one of:ip
,credential
,consumer
,service
,header
,path
,consumer-group
The type of identifier used to generate the rate limit key. Defines the scope used to increment the rate limiting counters. Can be
ip
,credential
,consumer
,service
,header
,path
orconsumer-group
.
-
string default:
sliding
Must be one of:fixed
,sliding
Sets the time window type to either
sliding
(default) orfixed
. Sliding windows apply the rate limiting logic while taking into account previous hit rates (from the window that immediately precedes the current) using a dynamic weight. Fixed windows consist of buckets that are statically assigned to a definitive time range, each request is mapped to only one fixed window based on its timestamp and will affect only that window’s counters.
-
number
How often to sync counter data to the central data store. A value of 0 results in synchronous behavior; a value of -1 ignores sync behavior entirely and only stores counters in node memory. A value greater than 0 will sync the counters in the specified number of seconds. The minimum allowed interval is 0.02 seconds (20ms).
-
array of type
record
requiredThe provider config. Takes an array of
name
,limit
andwindow size
values.-
number required
The window size to apply a limit (defined in seconds).
-
string required Must be one of:
openai
,azure
,anthropic
,cohere
,mistral
,llama2
,bedrock
,gemini
,huggingface
,requestPrompt
The LLM provider to which the rate limit applies.
-
number required
The limit applies to the LLM provider within the defined window size. It used the query cost from the tokens to increment the counter.
-
-
string required default:
local
Must be one of:cluster
,redis
,local
The rate-limiting strategy to use for retrieving and incrementing the limits. Available values are:
local
andcluster
.
-
string required default:
kong_rate_limiting_counters
The shared dictionary where counters are stored. When the plugin is configured to synchronize counter data externally (that is
config.strategy
iscluster
orredis
andconfig.sync_rate
isn’t-1
), this dictionary serves as a buffer to populate counters in the data store on each synchronization cycle.
-
boolean default:
false
Optionally hide informative response headers that would otherwise provide information about the current status of limits and counters.
-
number default:
0
The upper bound of a jitter (random delay) in seconds to be added to the
Retry-After
header of denied requests (status =429
) in order to prevent all the clients from coming back at the same time. The lower bound of the jitter is0
; in this case, theRetry-After
header is equal to theRateLimit-Reset
header.
-
string
A string representing an HTTP header name.
-
string starts_with:
/
A string representing a URL path, such as /path/to/resource. Must start with a forward slash (/) and must not contain empty segments (i.e., two consecutive forward slashes).
-
record required
-
string default:
127.0.0.1
A string representing a host name, such as example.com.
-
integer default:
6379
between:0
65535
An integer representing a port number between 0 and 65535, inclusive.
-
integer default:
2000
between:0
2147483646
An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2.
-
integer default:
2000
between:0
2147483646
An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2.
-
integer default:
2000
between:0
2147483646
An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2.
-
string referenceable
Username to use for Redis connections. If undefined, ACL authentication won’t be performed. This requires Redis v6.0.0+. To be compatible with Redis v5.x.y, you can set it to
default
.
-
string referenceable encrypted
Password to use for Redis connections. If undefined, no AUTH commands are sent to Redis.
-
string referenceable
Sentinel username to authenticate with a Redis Sentinel instance. If undefined, ACL authentication won’t be performed. This requires Redis v6.2.0+.
-
string referenceable encrypted
Sentinel password to authenticate with a Redis Sentinel instance. If undefined, no AUTH commands are sent to Redis Sentinels.
-
integer default:
0
Database to use for the Redis connection when using the
redis
strategy
-
integer default:
256
between:1
2147483646
The size limit for every cosocket connection pool associated with every remote server, per worker process. If neither
keepalive_pool_size
norkeepalive_backlog
is specified, no pool is created. Ifkeepalive_pool_size
isn’t specified butkeepalive_backlog
is specified, then the pool uses the default value. Try to increase (e.g. 512) this value if latency is high or throughput is low.
-
integer between:
0
2147483646
Limits the total number of opened connections for a pool. If the connection pool is full, connection queues above the limit go into the backlog queue. If the backlog queue is full, subsequent connect operations fail and return
nil
. Queued operations (subject to set timeouts) resume once the number of connections in the pool is less thankeepalive_pool_size
. If latency is high or throughput is low, try increasing this value. Empirically, this value is larger thankeepalive_pool_size
.
-
string
Sentinel master to use for Redis connections. Defining this value implies using Redis Sentinel.
-
string Must be one of:
master
,slave
,any
Sentinel role to use for Redis connections when the
redis
strategy is defined. Defining this value implies using Redis Sentinel.
-
array of type
record
len_min:1
Sentinel node addresses to use for Redis connections when the
redis
strategy is defined. Defining this field implies using a Redis Sentinel. The minimum length of the array is 1 element.
-
array of type
record
len_min:1
Cluster addresses to use for Redis connections when the
redis
strategy is defined. Defining this field implies using a Redis Cluster. The minimum length of the array is 1 element.
-
boolean default:
false
If set to true, uses SSL to connect to Redis.
-
boolean default:
false
If set to true, verifies the validity of the server SSL certificate. If setting this parameter, also configure
lua_ssl_trusted_certificate
inkong.conf
to specify the CA (or server) certificate used by your Redis server. You may also need to configurelua_ssl_verify_depth
accordingly.
-
string
A string representing an SNI (server name indication) value for TLS.
-
integer default:
5
Maximum retry attempts for redirection.
-
boolean default:
false
If the connection to Redis is proxied (e.g. Envoy), set it
true
. Set thehost
andport
to point to the proxy address.
-
-
boolean default:
false
If set to
true
, this doesn’t count denied requests (status =429
). If set tofalse
, all requests, including denied ones, are counted. This parameter only affects thesliding
window_type and the request prompt provider.
-
string
If defined, it use custom function to count requests for the request prompt provider
-
number default:
429
Set a custom error code to return when the rate limit is exceeded.
-
string default:
AI token rate limit exceeded for provider(s):
Set a custom error message to return when the rate limit is exceeded.
-
boolean default:
false
Optionally hide informative response that would otherwise provide information about the provider in the error message.
-
string required default:
total_tokens
Must be one of:total_tokens
,prompt_tokens
,completion_tokens
,cost
What tokens to use for cost calculation. Available values are:
total_tokens
prompt_tokens
,completion_tokens
orcost
.
-