AI Rate Limiting Advanced Configuration

You are browsing unreleased documentation.

Configuration

This plugin is partially compatible with DB-less mode.

The cluster strategy is not supported in DB-less and hybrid modes. For Kong Gateway in DB-less or hybrid mode, use the redis strategy.

Compatible protocols

The AI Rate Limiting Advanced plugin is compatible with the following protocols:

grpc, grpcs, http, https

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

• name or plugin

  string required

  The name of the plugin, in this case ai-rate-limiting-advanced.

  • If using the Kong Admin API, Konnect API, declarative configuration, or decK files, the field is name.
  • If using the KongPlugin object in Kubernetes, the field is plugin.

• instance_name

  string

  An optional custom name to identify an instance of the plugin, for example ai-rate-limiting-advanced_my-service.

  The instance name shows up in Kong Manager and in Konnect, so it's useful when running the same plugin in multiple contexts, for example, on multiple services. You can also use it to access a specific plugin instance via the Kong Admin API.

  An instance name must be unique within the following context:

  • Within a workspace for Kong Gateway Enterprise
  • Within a control plane or control plane group for Konnect
  • Globally for Kong Gateway (OSS)

• service.name or service.id

  string

  The name or ID of the service the plugin targets. Set one of these parameters if adding the plugin to a service through the top-level /plugins endpoint. Not required if using /services/{serviceName|Id}/plugins.

• route.name or route.id

  string

  The name or ID of the route the plugin targets. Set one of these parameters if adding the plugin to a route through the top-level /plugins endpoint. Not required if using /routes/{routeName|Id}/plugins.

• consumer.name or consumer.id

  string

  The name or ID of the consumer the plugin targets. Set one of these parameters if adding the plugin to a consumer through the top-level /plugins endpoint. Not required if using /consumers/{consumerName|Id}/plugins.

• consumer_group.name or consumer_group.id

  string

  The name or ID of the consumer group the plugin targets. If set, the plugin will activate only for requests where the specified group has been authenticated /plugins endpoint. Not required if using /consumer_groups/{consumerGroupName|Id}/plugins.

• enabled

  boolean default: true

  Whether this plugin will be applied.

• config

  record required
  • identifier

    string required default: consumer Must be one of: ip, credential, consumer, service, header, path, consumer-group

    The type of identifier used to generate the rate limit key. Defines the scope used to increment the rate limiting counters. Can be ip, credential, consumer, service, header, path or consumer-group.

  • window_type

    string default: sliding Must be one of: fixed, sliding

    Sets the time window type to either sliding (default) or fixed. Sliding windows apply the rate limiting logic while taking into account previous hit rates (from the window that immediately precedes the current) using a dynamic weight. Fixed windows consist of buckets that are statically assigned to a definitive time range, each request is mapped to only one fixed window based on its timestamp and will affect only that window’s counters.

  • sync_rate

    number

    How often to sync counter data to the central data store. A value of 0 results in synchronous behavior; a value of -1 ignores sync behavior entirely and only stores counters in node memory. A value greater than 0 will sync the counters in the specified number of seconds. The minimum allowed interval is 0.02 seconds (20ms).

  • llm_providers

    array of type record required

    The provider config. Takes an array of name, limit and window size values.

    • window_size

      number required

      The window size to apply a limit (defined in seconds).

    • name

      string required Must be one of: openai, azure, anthropic, cohere, mistral, llama2, bedrock, gemini, requestPrompt

      The LLM provider to which the rate limit applies.

    • limit

      number required

      The limit applies to the LLM provider within the defined window size. It used the query cost from the tokens to increment the counter.

  • strategy

    string required default: local Must be one of: cluster, redis, local

    The rate-limiting strategy to use for retrieving and incrementing the limits. Available values are: local and cluster.

  • dictionary_name

    string required default: kong_rate_limiting_counters

    The shared dictionary where counters are stored. When the plugin is configured to synchronize counter data externally (that is config.strategy is cluster or redis and config.sync_rate isn’t -1), this dictionary serves as a buffer to populate counters in the data store on each synchronization cycle.

  • hide_client_headers

    boolean default: false

    Optionally hide informative response headers that would otherwise provide information about the current status of limits and counters.

  • retry_after_jitter_max

    number default: 0

    The upper bound of a jitter (random delay) in seconds to be added to the Retry-After header of denied requests (status = 429) in order to prevent all the clients from coming back at the same time. The lower bound of the jitter is 0; in this case, the Retry-After header is equal to the RateLimit-Reset header.

  • header_name

    string

    A string representing an HTTP header name.

  • path

    string starts_with: /

    A string representing a URL path, such as /path/to/resource. Must start with a forward slash (/) and must not contain empty segments (i.e., two consecutive forward slashes).

  • redis

    record required
    • host

      string

      A string representing a host name, such as example.com.

    • port

      integer between: 0 65535

      An integer representing a port number between 0 and 65535, inclusive.

    • connect_timeout

      integer default: 2000 between: 0 2147483646

      An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2.

    • send_timeout

      integer default: 2000 between: 0 2147483646

      An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2.

    • read_timeout

      integer default: 2000 between: 0 2147483646

      An integer representing a timeout in milliseconds. Must be between 0 and 2^31-2.

    • username

      string referenceable

      Username to use for Redis connections. If undefined, ACL authentication won’t be performed. This requires Redis v6.0.0+. To be compatible with Redis v5.x.y, you can set it to default.

    • password

      string referenceable encrypted

      Password to use for Redis connections. If undefined, no AUTH commands are sent to Redis.

    • sentinel_username

      string referenceable

      Sentinel username to authenticate with a Redis Sentinel instance. If undefined, ACL authentication won’t be performed. This requires Redis v6.2.0+.

    • sentinel_password

      string referenceable encrypted

      Sentinel password to authenticate with a Redis Sentinel instance. If undefined, no AUTH commands are sent to Redis Sentinels.

    • database

      integer default: 0

      Database to use for the Redis connection when using the redis strategy

    • keepalive_pool_size

      integer default: 256 between: 1 2147483646

      The size limit for every cosocket connection pool associated with every remote server, per worker process. If neither keepalive_pool_size nor keepalive_backlog is specified, no pool is created. If keepalive_pool_size isn’t specified but keepalive_backlog is specified, then the pool uses the default value. Try to increase (e.g. 512) this value if latency is high or throughput is low.

    • keepalive_backlog

      integer between: 0 2147483646

      Limits the total number of opened connections for a pool. If the connection pool is full, connection queues above the limit go into the backlog queue. If the backlog queue is full, subsequent connect operations fail and return nil. Queued operations (subject to set timeouts) resume once the number of connections in the pool is less than keepalive_pool_size. If latency is high or throughput is low, try increasing this value. Empirically, this value is larger than keepalive_pool_size.

    • sentinel_master

      string

      Sentinel master to use for Redis connections. Defining this value implies using Redis Sentinel.

    • sentinel_role

      string Must be one of: master, slave, any

      Sentinel role to use for Redis connections when the redis strategy is defined. Defining this value implies using Redis Sentinel.

    • sentinel_nodes

      array of type record len_min: 1

      Sentinel node addresses to use for Redis connections when the redis strategy is defined. Defining this field implies using a Redis Sentinel. The minimum length of the array is 1 element.

      • host

        string required default: 127.0.0.1

        A string representing a host name, such as example.com.

      • port

        integer default: 6379 between: 0 65535

        An integer representing a port number between 0 and 65535, inclusive.

    • cluster_nodes

      array of type record len_min: 1

      Cluster addresses to use for Redis connections when the redis strategy is defined. Defining this field implies using a Redis Cluster. The minimum length of the array is 1 element.

      • ip

        string required default: 127.0.0.1

        A string representing a host name, such as example.com.

      • port

        integer default: 6379 between: 0 65535

        An integer representing a port number between 0 and 65535, inclusive.

    • ssl

      boolean default: false

      If set to true, uses SSL to connect to Redis.

    • ssl_verify

      boolean default: false

      If set to true, verifies the validity of the server SSL certificate. If setting this parameter, also configure lua_ssl_trusted_certificate in kong.conf to specify the CA (or server) certificate used by your Redis server. You may also need to configure lua_ssl_verify_depth accordingly.

    • server_name

      string

      A string representing an SNI (server name indication) value for TLS.

    • cluster_max_redirections

      integer default: 5

      Maximum retry attempts for redirection.

    • connection_is_proxied

      boolean default: false

      If the connection to Redis is proxied (e.g. Envoy), set it true. Set the host and port to point to the proxy address.

  • disable_penalty

    boolean default: false

    If set to true, this doesn’t count denied requests (status = 429). If set to false, all requests, including denied ones, are counted. This parameter only affects the sliding window_type and the request prompt provider.

  • request_prompt_count_function

    string

    If defined, it use custom function to count requests for the request prompt provider

  • error_code

    number default: 429

    Set a custom error code to return when the rate limit is exceeded.

  • error_message

    string default: API rate limit exceeded for provider(s):

    Set a custom error message to return when the rate limit is exceeded.

  • error_hide_providers

    boolean default: false

    Optionally hide informative response that would otherwise provide information about the provider in the error message.

  • tokens_count_strategy

    string required default: total_tokens Must be one of: total_tokens, prompt_tokens, completion_tokens, cost

    What tokens to use for cost calculation. Available values are: total_tokens prompt_tokens, completion_tokens or cost.