The Datakit plugin provides the following node types:
-
branch: Execute different nodes based on matching input conditions.
-
cache: Store and fetch cached data.
-
call: Send third-party HTTP calls.
-
jq: Transform data and cast variables with jq to be shared with other nodes.
-
exit: Return directly to the client.
-
property: Get and set Kong Gateway-specific data.
-
static: Configure static input values ahead of time.
|
Node type
|
Inputs
|
Outputs
|
Attributes
|
branch
|
user-defined
|
none
|
then, else
|
cache
|
key, ttl, data
|
hit, miss, stored, data
|
bypass_on_error, ttl
|
call
|
body, headers, query
|
body, headers, status
|
url, method, timeout, ssl_server_name
|
jq
|
user-defined
|
user-defined
|
jq
|
exit
|
body, headers
|
none
|
status
|
property
|
$self
|
$self
|
property, content_type
|
static
|
none
|
user-defined
|
values
|
Datakit also defines a number of implicit nodes that can’t be declared directly under the nodes configuration section.
These nodes can either be used without being explicitly declared, or declared under the global resources object.
These reserved node names can’t be used for user-defined
nodes. They include:
|
Node
|
Inputs
|
Outputs
|
Description
|
declaration
|
request
|
none
|
body, headers, query
|
Reads incoming client request properties
|
none
|
service_request
|
body, headers, query
|
none
|
Updates properties of the request sent to the service being proxied to
|
none
|
service_response
|
none
|
body, headers
|
Reads response properties from the service being proxied to
|
none
|
response
|
body, headers
|
none
|
Updates properties of the outgoing client response
|
none
|
vault
|
none
|
$self
|
Vault reference to hold secret values
|
resources.vault
|
The headers type produces and consumes maps from header names to their values:
- Keys are header names. Original header name casing is preserved for maximum
compatibility.
- Values are strings if there is a single instance of a header or arrays of
strings if there are multiple instances of the same header.
The query type produces and consumes maps with key-value pairs representing
decoded URL query strings.
The service_request.body and response.body inputs both accept any data type.
If the data is an object, it will automatically be JSON-encoded, and the
Content-Type header set to application/json (if not already set in the
headers input).
The request.body and service_response.body outputs have a similar behavior.
If the corresponding Content-Type header matches the JSON mime-type, the
body output is automatically JSON-decoded.
The vault node is an implicit node that allows you to declare secret references
and can be used in other nodes as a source of secret values. Vault references are declared
under the resources.vault configuration.
Declare two vault references and use them in a jq node:
resources:
vault:
secret1: "{vault://env/my-secret1}"
secret2: "{vault://aws/my-secret2}"
nodes:
- name: JQ
type: jq
inputs:
secret1: vault.secret1
secret2: vault.secret2
jq: "."
Execute different nodes based on matching input conditions, such as a cache hit or miss.
The input to a branch node represents a boolean condition to test and branch on:
- If the input is
true, the nodes named by the then array are executed.
- If the input is
false, the nodes named by the else array are executed.
- If the input is a non-boolean value, an error is raised.
-
then: Array of nodes to execute if the input condition is true.
-
else: Array of nodes to execute if the input condition is false.
Note: When using the branch node, the then and else parameters must list all nodes for both branches.
If a node isn’t listed in the branch, the node will run depending on its location in the flow configuration.
The following example configuration takes the input of a cache node named GET:
- If it sees a
miss, it executes the nodes DATA, SET, and EXIT_MISS.
- If it doesn’t see a
miss, it executes the node EXIT_HIT.
Cache input:
- type: static
values:
key: cache key
name: CACHE_KEY_GET
- type: cache
input: CACHE_KEY_GET
ttl: 200
name: GET
Branch node based on cache hit or miss:
- type: branch
then:
- DATA
- SET
- EXIT_MISS
else:
- EXIT_HIT
input: GET.miss
name: branch
See Conditionally fetching or storing cache data for a full example.
Store data into cache and fetch cached data from cache.
Inputs:
-
key (required): the cache key string
-
ttl: The TTL (Time to Live) in seconds
-
data: The data to be cached. If not null, the cache node works in set mode,
storing data into cache; if null, the cache node fetches data
Outputs:
-
hit: true if a cache hit occured
-
miss: true if a cache miss occurred
-
stored: true if data was successfuly stored into cache
-
data: The data that was stored into cache
Configuration attributes:
-
bypass_on_error: if true, cache backend errors are treated as a cache
miss
-
ttl: The TTL (Time to Live) in seconds
Send an HTTP request and retrieve the response.
Inputs:
-
body: Request body
-
headers: Request headers
-
query: Key-value pairs to encode as the request query string
The cache node requires a resources.cache resource definition containing
cache configuration.
This is defined via the resources.cache.strategy parameter, which can be one of:
-
redis: A Redis database. Define properties for this strategy using resources.cache.redis or using a Redis Partial.
-
memory: A shared dictionary. Define the dictionary name for this strategy using resources.cache.memory.dictionary_name.
The default dictionary, kong_db_cache, is also used by other plugins and elements of Kong Gateway to store unrelated database cache entities.
Using the kong_db_cache dictionary is an easy way to bootstrap and test the plugin, but we don’t recommend using it for large-scale installations as significant usage will put pressure on other facets of Kong Gateway’s database caching operations.
In production, we recommend defining a custom lua_shared_dict via a custom Nginx template.
Cache entities are stored for a configurable period of time (see cache TTL), after which subsequent requests to the same resource will fetch and store the resource again.
In serverless gateways, only the memory strategy is supported.
Outputs:
-
body: The response body
-
headers: The response headers
-
status: The HTTP status code of the response
Configuration attributes:
-
url (required): The URL
-
method: The HTTP method (default is GET)
-
timeout: The dispatch timeout, in milliseconds
Make an external API call:
- name: CALL
type: call
url: https://example.com/foo
Send a POST request with a JSON body:
- name: POST
type: call
url: https://example.com/create-entity
method: POST
inputs:
body: ENTITY
- name: ENTITY
type: static
values:
id: 123
name: Datakit
If the data connected to the body input is an object, it will automatically be
encoded as JSON, and the request Content-Type header will be set to
application/json unless already present in the headers input.
Similarly, if the response Content-Type header matches the JSON mime-type, the
body output will be JSON-decoded automatically.
This is an async node. This means that the request will be sent in the
background while Datakit executes any other nodes (save for any nodes which
depend on it). Multiple call nodes are executed concurrently when no dependency
order enforces it.
In this example, both CALL_FOO and CALL_BAR will be started as soon as
possible, and then Datakit will block until both have finished to run
JOIN:
- name: CALL_FOO
type: call
url: https://example.com/foo
- name: CALL_BAR
type: call
url: https://example.com/bar
- name: JOIN
type: jq
jq: "."
inputs:
foo: CALL_FOO.body
bar: CALL_BAR.body
The call node fails execution if a network-level error is encountered or if
the endpoint returns a non-2xx status code. It will also fail if the endpoint
returns a JSON mime-type in the Content-Type header if the response body is
not valid JSON.
Due to platform limitations, the call node can’t be executed after proxying a
request, so attempting to configure the node using outputs from the upstream service
response will yield an error:
- name: CALL
type: call
url: https://example.com/
method: POST
inputs:
# dependency error!
body: service_response.body
Error:
invalid dependency (node #1 (CALL) -> node service_response): circular dependency
The jq node executes a jq script for processing JSON. See the official
jq docs for more details.
User-defined. For node-wise ($self) connections, jq can handle input of
any type:
- name: SERVICE
type: property
property: kong.router.service
- name: IP
type: property
property: kong.client.ip
- name: FILTER_SERVICE
type: jq
input: SERVICE
# yields: "object"
jq: ". | type"
- name: FILTER_IP
type: jq
input: IP
# yields: "string"
jq: ". | type"
By defining individual inputs, jq’s input will be coerced to an object with
string keys and values of any type. Referencing input fields from within the
filter is done by using dot (.) notation:
- name: SERVICE
type: property
property: kong.router.service
- name: IP
type: property
property: kong.client.ip
- name: FILTER_SERVICE_AND_IP
type: jq
inputs:
service: SERVICE
ip: IP
# yields: { "$self": "object", "service": "object", "ip": "string" }
jq: |
{
"$self": (. | type),
"service": (.service | type),
"ip": (.ip | type)
}
User-defined. A jq filter script can produce any type of data:
- name: STRING
type: jq
jq: |
"my string"
- name: NUMBER
type: jq
jq: |
54321
- name: BOOLEAN
type: jq
jq: |
true
- name: OBJECT
type: jq
jq: |
{
a: 1,
b: 2
}
It’s impossible for Datakit to know ahead of time what kind of data jq will
emit, so Datakit uses runtime checks when the output of jq is connected to
another node’s input. It’s important to carefully test and validate your Datakit
configurations to avoid this case:
- name: HEADERS
type: jq
jq: |
"oops, not an object/map"
- name: EXIT
type: exit
inputs:
# this will cause Datakit to return a 500 error to the client when
# encountered
headers: HEADERS
This is also why the jq node doesn’t allow explicitly referencing individual
fields with outputs at config-time:
- name: HEADERS
type: jq
jq: |
"this is completely opaque to Datakit"
# Datakit will reject this configuration because it can't confirm that the
# output of HEADERS is an object or has a `body` field
outputs:
body: EXIT.body
- name: EXIT
type: exit
jq: the jq script to execute when the node is triggered.
To enable a high level of transparency and compatibility when
communicating with external services, headers outputs in Datakit always
preserve the original case of header names. While HTTP-centric nodes within
Datakit are careful to account for this and perform header lookups and
transformations in a case-insensitive manner, jq at its core is a library for
operating upon JSON data, and JSON object keys are strictly case-sensitive.
Be mindful of this when handling headers in a jq filter to avoid buggy, error-prone behavior.
For example:
# adds the `X-Extra` header to the upstream service request if not set by the client
- name: ADD_HEADERS
type: jq
input: request.headers
output: service_request.headers
jq: |
{
"X-Extra": ( .["X-Extra"] // "default value" ),
}
This filter will function correctly if the client sets the X-Extra header or
omits it entirely, but it won’t have the intended effect if the client sets
the header X-EXTRA or x-extra.
jq lets you write a robust filter that handles this condition.
The following implementation normalizes header names to lowercase before looking up values from the input:
# adds the `X-Extra` header to the upstream service request if not set by the client
- name: ADD_HEADERS
type: jq
input: request.headers
output: service_request.headers
jq: |
with_entries(.key |= ascii_downcase)
| {
"X-Extra": ( .["x-extra"] // "default value" ),
}
These examples take in client request headers and update them from a set of
pre-defined values.
The HTTP specification RFC
defines header names to be case-insensitive, so in most cases it’s
enough to normalize header names to lowercase for ease of merging the
two objects:
- name: header_updates
type: static
values:
X-Foo: "123"
X-Custom: "my header"
X-Multi:
- "first"
- "second"
- name: merged_headers
type: jq
inputs:
original: request.headers
updates: header_updates
jq: |
(.original | with_entries(.key |= ascii_downcase))
*
(.updates | with_entries(.key |= ascii_downcase))
- name: api
type: call
url: "https://example.com/"
inputs:
headers: merged_headers
However, when dealing with a upstream service or API that isn’t fully compliant
with the HTTP spec, it might be necessary to preserve original header name
casing. For example:
- name: header_updates
type: static
values:
X-Foo: "123"
X-Custom: "my header"
X-Multi:
- "first"
- "second"
- name: merged_headers
type: jq
inputs:
original: request.headers
updates: header_updates
jq: |
. as $input
# store .original key names for lookup
| $input.original
| with_entries({ key: .key | ascii_downcase, value: .key })
as $keys
# rewrite .updates with .original key names
| $input.updates
| with_entries(.key = ($keys[.key | ascii_downcase] // .key))
as $updates
| $input.original * $updates
- name: api
type: call
url: "https://example.com/"
inputs:
headers: merged_headers
Coerce the client request body to an object:
- name: BODY
type: jq
input: request.body
jq: |
if type == "object" then
.
else
{ data: . }
end
Join the output of two API calls:
- name: FOO
type: call
url: https://example.com/foo
- name: BAR
type: call
url: https://example.com/bar
- name: JOIN
type: jq
inputs:
foo: FOO.body
bar: BAR.body
jq: "."
Trigger an early exit that produces a direct response, rather than forwarding
a proxied response.
Inputs:
-
body: Body to use in the early-exit response.
-
headers: Headers to use in the early-exit response.
Outputs: None
Configuration attributes:
-
status: The HTTP status code to use in the early-exit response (default is
200).
Make an HTTP request and send the response directly to the client:
- name: CALL
type: call
url: https://example.com/
- name: EXIT
type: exit
input: CALL
Get and set Kong Gateway host and request properties.
Whether a get or set operation is performed depends upon the node inputs:
- If an input is connected,
set the property
- If no input is connected,
get the property and map it to the output
This node accepts the $self input:
- name: STORE_REQUEST
type: property
property: kong.ctx.shared.my_request
input: request
No individual field-level inputs are permitted:
- name: STORE_REQUEST_BY_FIELD
type: property
property: kong.ctx.shared.my_request
# error! property input doesn't allow field access
inputs:
body: request.body
This node produces the $self output.
- name: GET_ROUTE
type: property
property: kong.router.route
output: response.body
Field-level output connections are not supported, even if the output data has known fields:
- name: GET_ROUTE_ID
type: property
property: kong.router.route
# error! property output doesn't allow field access
outputs:
id: response.body
-
property (required): The name of the property
-
content_type: The expected mime type of the property value. When set to
application/json, set operations will JSON-encode input data before
writing it, and get operations will JSON-decode output data after
reading it. Otherwise, this setting has no effect.
The following properties support get operations:
|
Property
|
Description
|
Data type
|
kong.client.consumer
|
kong.client.get_consumer()
|
object
|
kong.client.consumer_groups
|
kong.client.get_consumer_groups()
|
array
|
kong.client.credential
|
kong.client.get_credential()
|
object
|
kong.client.get_identity_realm_source
|
kong.client.get_identity_realm_source()
|
object
|
kong.client.forwarded_ip
|
kong.client.get_forwarded_ip()
|
string
|
kong.client.forwarded_port
|
kong.client.get_forwarded_port()
|
number
|
kong.client.ip
|
kong.client.get_ip()
|
string
|
kong.client.port
|
kong.client.get_port()
|
number
|
kong.client.protocol
|
kong.client.get_protocol()
|
string
|
kong.request.forwarded_host
|
kong.request.get_forwarded_host()
|
string
|
kong.request.forwarded_port
|
kong.request.get_forwarded_port()
|
number
|
kong.request.forwarded_scheme
|
kong.request.get_forwarded_scheme()
|
string
|
kong.request.port
|
kong.request.get_port()
|
number
|
kong.response.source
|
kong.response.get_source()
|
string
|
kong.router.route
|
kong.router.get_route()
|
object
|
kong.route_id
|
Gets the current Route’s ID
|
string
|
kong.route_name
|
Gets the current Route’s name
|
string
|
kong.router.service
|
kong.router.get_service()
|
object
|
kong.service_name
|
Gets the current Service’s name
|
string
|
kong.service_id
|
Gets the current Service’s ID
|
string
|
kong.service.response.status
|
kong.service.response.status
|
number
|
kong.version
|
Gets the Kong version
|
string
|
kong.node.id
|
kong.node.get_id()
|
string
|
kong.configuration.{key}
|
Reads {key} from the node configuration
|
any
|
The following properties support set operations:
|
Property
|
Description
|
Data type
|
kong.service.target
|
kong.service.set_target({host}, {port})
|
string ({host}:{port})
|
kong.service.request_scheme
|
kong.service.set_service_request_scheme({scheme})
|
string ({scheme})
|
The following properties support get and set operations:
|
Property
|
Description
|
Data type
|
kong.ctx.plugin.{key}
|
Gets or sets kong.ctx.plugin.{key}
|
any
|
kong.ctx.shared.{key}
|
Gets or sets kong.ctx.shared.{key}
|
any
|
Emits static values to be used as inputs for other nodes. The static node can help you with hardcoding some known value for an input.
None.
This node produces outputs for each item in its values attribute:
- name: CALL_INPUTS
type: static
values:
headers:
X-Foo: "123"
X-Multi:
- first
- second
query:
a: true
b: 10
body:
data: "my request body data"
- name: CALL
type: call
url: https://example.com/
method: POST
input: CALL_INPUTS
The static nature of these values comes in handy, because Datakit can
validate them when creating or updating the plugin configuration. Attempting to
create a plugin with the following configuration will yield an Admin API
validation error instead of bubbling up at runtime:
- name: CALL_INPUTS
type: static
values:
headers: "oops not valid headers"
- name: CALL
type: call
url: https://example.com/
method: POST
input: CALL_INPUTS
-
values (required): A mapping of string keys to arbitrary values
Set a property from a static value:
- name: VALUE
type: static
values:
data:
a: 1
b: 2
- name: PROPERTY
type: property
property: kong.ctx.shared.my_property
input: VALUE.data
Set a default value for a jq filter:
- name: VALUE
type: static
values:
default: "my default value"
- name: FILTER
type: jq
inputs:
query: request.query
default: VALUE.default
jq: ".query.foo // .default"
Set common request headers for different API requests:
- name: HEADERS
type: static
values:
X-Common: "we always need this header"
- name: CALL_FOO
type: call
url: https://example.com/foo
inputs:
headers: HEADERS
- name: CALL_BAR
type: call
url: https://example.com/bar
inputs:
headers: HEADERS
