Looking for the plugin's configuration parameters? You can find them in the AI Response Transformer configuration reference doc.
The AI Response Transformer plugin uses a configured LLM service to introspect and transform the upstream’s HTTP(S) response before returning it back to the client. It can also be configured to terminate or otherwise nullify the response, should it fail (for example) a compliance or formatting check from the configured LLM service.
This plugin supports llm/v1/chat
style requests for all of the same providers as the AI Proxy plugin.
It also uses all of the same configuration and tuning parameters as the AI Proxy plugin, under the config.llm
block.
The AI Response Transformer plugin runs after the AI Proxy plugin, and after proxying to the upstream (backend) service, allowing it to also introspect LLM responses against the same, or a different, LLM service.
How it works
Figure 1: The diagram shows the journey of a client’s request through Kong Gateway to the backend service, where it is transformed by both an AI LLM service and Kong’s AI Request Transformer and the AI Response Transformer plugins.
- The Kong Gateway admin sets up an
llm:
configuration block, following the same configuration format as the AI Proxy plugin, and the samedriver
capabilities. - The Kong Gateway admin sets up a
prompt
for the request introspection. The prompt becomes thesystem
message in the LLM chat request, and prepares the LLM with transformation instructions for the returning upstream response body. - The user makes an HTTP(S) call.
-
After proxying the user’s request to the backend, Kong Gateway sets the entire response body as the
user
message in the LLM chat request, then sends it to the configured LLM service. - The LLM service returns a response
assistant
message, which is subsequently set as the upstream response body. - The plugin returns early (
kong.response.exit
) and can handle gzip or chunked requests, similarly to the Forward Proxy plugin.
Adjusting response headers, status codes, and body
You can additionally instruct the LLM to respond in the following format, which lets you adjust the response headers, response status code, and response body:
{
"headers":
{
"new-header": "new-value"
},
"status": 201,
"body": "new response body"
}
Kong Gateway will parse these instructions and set all given response headers, set the response status code,
and set replacement response body.
This lets you change specific headers such as Content-Type
, or throw errors from the LLM.
Get started with the AI Response Transformer plugin
All AI Gateway plugins
- AI Proxy
-
AI Proxy Advanced Available with Kong Gateway Enterprise subscription - Contact Sales
- AI Request Transformer
- AI Response Transformer
-
AI Semantic Cache Available with Kong Gateway Enterprise subscription - Contact Sales
-
AI Semantic Prompt Guard Available with Kong Gateway Enterprise subscription - Contact Sales
-
AI Rate Limiting Advanced Available with Kong Gateway Enterprise subscription - Contact Sales
-
AI Azure Content Safety Available with Kong Gateway Enterprise subscription - Contact Sales
- AI Prompt Template
- AI Prompt Guard
- AI Prompt Decorator