AI Response Transformer Configuration

You are browsing documentation for an outdated plugin version.

Configuration

This plugin is compatible with DB-less mode.

Compatible protocols

The AI Response Transformer plugin is compatible with the following protocols:

grpc, grpcs, http, https

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

• name or plugin

  string required

  The name of the plugin, in this case ai-response-transformer.

  • If using the Kong Admin API, Konnect API, declarative configuration, or decK files, the field is name.
  • If using the KongPlugin object in Kubernetes, the field is plugin.

• instance_name

  string

  An optional custom name to identify an instance of the plugin, for example ai-response-transformer_my-service.

  The instance name shows up in Kong Manager and in Konnect, so it's useful when running the same plugin in multiple contexts, for example, on multiple services. You can also use it to access a specific plugin instance via the Kong Admin API.

  An instance name must be unique within the following context:

  • Within a workspace for Kong Gateway Enterprise
  • Within a control plane or control plane group for Konnect
  • Globally for Kong Gateway (OSS)

• service.name or service.id

  string

  The name or ID of the service the plugin targets. Set one of these parameters if adding the plugin to a service through the top-level /plugins endpoint. Not required if using /services/{serviceName|Id}/plugins.

• route.name or route.id

  string

  The name or ID of the route the plugin targets. Set one of these parameters if adding the plugin to a route through the top-level /plugins endpoint. Not required if using /routes/{routeName|Id}/plugins.

• consumer_group.name or consumer_group.id

  string

  The name or ID of the consumer group the plugin targets. If set, the plugin will activate only for requests where the specified group has been authenticated /plugins endpoint. Not required if using /consumer_groups/{consumerGroupName|Id}/plugins.

• enabled

  boolean default: true

  Whether this plugin will be applied.

• config

  record required
  • prompt

    string required

    Use this prompt to tune the LLM system/assistant message for the returning proxy response (from the upstream), adn what response format you are expecting.

  • transformation_extract_pattern

    string

    Defines the regular expression that must match to indicate a successful AI transformation at the response phase. The first match will be set as the returning body. If the AI service’s response doesn’t match this pattern, a failure is returned to the client.

  • parse_llm_response_json_instructions

    boolean required default: false

    Set true to read specific response format from the LLM, and accordingly set the status code / body / headers that proxy back to the client. You need to engineer your LLM prompt to return the correct format, see plugin docs ‘Overview’ page for usage instructions.

  • http_timeout

    integer required default: 60000

    Timeout in milliseconds for the AI upstream service.

  • https_verify

    boolean required default: true

    Verify the TLS certificate of the AI upstream service.

  • http_proxy_host

    string

    A string representing a host name, such as example.com.

  • http_proxy_port

    integer between: 0 65535

    An integer representing a port number between 0 and 65535, inclusive.

  • https_proxy_host

    string

    A string representing a host name, such as example.com.

  • https_proxy_port

    integer between: 0 65535

    An integer representing a port number between 0 and 65535, inclusive.

  • llm

    record required
    • route_type

      string required Must be one of: llm/v1/chat, llm/v1/completions

      The model’s operation implementation, for this provider.

    • auth

      record
      • header_name

        string referenceable

        If AI model requires authentication via Authorization or API key header, specify its name here.

      • header_value

        string referenceable encrypted

        Specify the full auth header value for ‘header_name’, for example ‘Bearer key’ or just ‘key’.

      • param_name

        string referenceable

        If AI model requires authentication via query parameter, specify its name here.

      • param_value

        string referenceable encrypted

        Specify the full parameter value for ‘param_name’.

      • param_location

        string Must be one of: query, body

        Specify whether the ‘param_name’ and ‘param_value’ options go in a query string, or the POST form/JSON body.

    • model

      record required
      • provider

        string required Must be one of: openai, azure, anthropic, cohere, mistral, llama2

        AI provider request format - Kong translates requests to and from the specified backend compatible formats.

      • name

        string

        Model name to execute.

      • options

        record

        Key/value settings for the model

        • max_tokens

          integer default: 256

          Defines the max_tokens, if using chat or completion models.

        • temperature

          number default: 1 between: 0 5

          Defines the matching temperature, if using chat or completion models.

        • top_p

          number default: 1 between: 0 1

          Defines the top-p probability mass, if supported.

        • top_k

          integer default: 0 between: 0 500

          Defines the top-k most likely tokens, if supported.

        • anthropic_version

          string

          Defines the schema/API version, if using Anthropic provider.

        • azure_instance

          string

          Instance name for Azure OpenAI hosted models.

        • azure_api_version

          string default: 2023-05-15

          ‘api-version’ for Azure OpenAI instances.

        • azure_deployment_id

          string

          Deployment ID for Azure OpenAI instances.

        • llama2_format

          string Must be one of: raw, openai, ollama

          If using llama2 provider, select the upstream message format.

        • mistral_format

          string Must be one of: openai, ollama

          If using mistral provider, select the upstream message format.

        • upstream_url

          string

          Manually specify or override the full URL to the AI operation endpoints, when calling (self-)hosted models, or for running via a private endpoint.

    • logging

      record required
      • log_statistics

        boolean required default: true

        If enabled and supported by the driver, will add model usage and token metrics into the Kong log plugin(s) output.

      • log_payloads

        boolean required default: false

        If enabled, will log the request and response body into the Kong log plugin(s) output.