Edit this page
Report an issueActive Tracing in Konnect
This feature is released as a tech preview (alpha-quality) and should not be depended upon in a production environment.
Active tracing enables control plane administrators to initiate targeted “deep tracing” sessions in specific data plane nodes. During an active tracing session, the selected data plane generates detailed, OpenTelemetry-compatible traces for all requests matching the sampling criteria. The detailed spans are captured for the entire request/response lifecycle. These traces can be visualized with Konnect’s built-in span viewer with no additional instrumentation or telemetry tools.
Kong Konnect’s active tracing capability offers exclusive, in-depth insights that cannot be replicated by third-party telemetry tools. The detailed traces generated during live active tracing session are unique to Kong and provide unparalleled visibility into system performance.
Active Tracing adheres to OpenTelemetry naming conventions for spans and attributes, wherever possible, ensuring consistency and interoperability.
Key highlights
- Traces can be generated for a service or per route
- Refined traces can be generated for all requests matching a sampling criteria
- Sampling criteria can be defined with simple expressions language, for example:
http.method == GET - Trace sessions are retained for up to 7 days
- Traces can be visualized in Konnect’s built in trace viewer
Although active tracing is designed as a debug and troubleshooting tool, it can unlock in-depth insights into the API traffic and serve as a monitoring and observability tool. Under normal conditions, active tracing adds negligible latency. However, under heavy loads, it may affect the throughput.
Reading traces in Konnect trace viewer
Traces captured in an active tracing session can be visualized in Konnect’s built-in trace viewer. The trace viewer displays a Summary view and a Trace view. You can gain instant insights with the summary view while the trace view will help you dive deeper.
Summary view
Summary view helps you visualize the entire API request-response flow in a single glance. This view provides a concise overview of critical metrics and a transaction map. The transaction map includes the plugins executed by Kong Gateway on both the request and the response along with the times spent in each phase. Use the summary view to quickly understand the end-to-end API flow, identify performance bottlenecks, and optimize your API strategy.
Trace view
The trace view gives you unparalleled visibility into Kong Gateway’s internal workings. This detailed view breaks down into individual spans, providing a comprehensive understanding of:
- Kong Gateway’s internal processes and phases
- Plugin execution and performance
- Request and response handling
Use the trace view to troubleshoot issues, optimize performance, and refine your configuration.
Get started with tracing
Active Tracing requires the following data plane version and environment variables in kong.conf:
- Version: 3.9.1 or above
-
Environment variables:
KONG_CLUSTER_RPC=onKONG_ACTIVE_TRACING=on
Start a trace session
- Navigate to Gateway Manager.
- Select a Control Plane which has the data plane to be traced.
- Click on Active Tracing in left navigation menu.
- Click New tracing session, define the criteria and, click Start Session.
Once started, traces will begin to be captured. Click on a trace to visualize it in the trace viewer.
The default session duration is 5 minutes or 200 traces per session. Note the sessions are retained for up to 7 days.
Sampling rules
To capture only the relevant API traffic, use sampling rules. Sampling rules filter and refine the requests to be matched. The matching requests are then traced and captured in the session. There are two options.
- Basic sampling rules: Allow filtering on route and services.
-
Advanced sampling rules: Specify the desired criteria using expressions. For example, to capture traces for all requests matching 503 response code, specify the following rule
http.response.status_code==503
Known issues in tech preview
Here is a list of known issues in the tech preview:
-
Incorrect span orders
- When spans have very short duration few spans may be displayed in wrong order.
-
Incorrect handling of certain error conditions
- Traces may be broken when there are certain error conditions. For example, when DNS name resolution fails.
-
Missing spans during high traffic volumes
- When no sampling rule is enabled during a high traffic volume scenario, some traces could be missing spans.
Sample trace
A sample trace is shown below. By inspecting the spans, it is clear that the bulk of the latency occurs in the pre-function plugin during the access phase.
Spans
kong
The root span.
This span has the following attributes:
| Name | Description |
|---|---|
| url.full | The full url, but without query params |
| client.address | Remote address of the client making the request. This considers forwarded addresses in cases when a load balancer is in front of Kong. Note: this requires configuring the real_ip_header and trusted_ips global configuration options. |
| client.port | Remote port of the client making the request. This considers forwarded ports in cases when a load balancer is in front of Kong. Note: this requires configuring the real_ip_header and trusted_ips global configuration options. |
| network.peer.address | IP of the component that is connecting to Kong |
| network.peer.port | Port of the component that is connecting to Kong |
| server.address | Kong's dns name or ip used in client connection |
| server.port | Kong's public port |
| network.protocol.name | http, grpc, ws, kafka etc |
| http.request.method | the http request method |
| http.request.body.size | request content-length or equivalent in bytes |
| http.request.size | request.body.size + request.headers.size in bytes |
| http.response.body.size | response content-length or equivalent in bytes |
| http.response.size | response.body.size + response.headers.size in bytes |
| kong.request.id | proxy.kong.request_id - unique id for each request |
| url.scheme | Protocol identifier |
| network.protocol.version | version of the http protocol used in establishing connection [1.2, 2.0] |
| tls.client.server_name | SNI |
| http.request.header.host | Host header if present. This can be different from SNI |
| proxy.kong.consumer_id | authenticated consumer_id if present |
| proxy.kong.upstream_id | resolved upstream_id |
| proxy.kong.upstream_status_code | status code returned by upstream |
| http.response.status_code | status code sent back by Kong |
| proxy.kong.latency.upstream | time between connect() to upstream and last byte of response ($upstream_response_time) |
| proxy.kong.latency.total | first byte into kong, last byte out of kong ($request_time) |
| proxy.kong.latency.internal | Time taken by Kong to process the request. Excludes client and upstream read/write times, and i/o times |
| proxy.kong.latency.net_io_timings | array of (ip, connect_time, rw_time) - i/o outside of request context is not considered |
| proxy.kong.client_KA | did downstream use a KeepAlive connection |
| tls.resumed | was the TLS session reused |
| tls.client.subject | x509 client DN (if mtls) |
| tls.server.subject | x509 DN for cert Kong presented |
| tls.cipher | the negotiated cipher |
kong.phase.certificate
A span capturing the execution of the certificate phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.
kong.certificate.plugin.{plugin_name}
A span capturing the execution of a plugin configured to run in the certificate phase. Multiple such spans can occur in a trace.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.plugin.instance_id | The instance id of the plugin configuration that ran |
kong.read_client_http_headers
A span capturing the time taken to read HTTP headers from the client. This span is useful for detecting clients that are coming over a slow network or a buggy CDN, or simply take too long to send in the HTTP headers.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.http_headers_count | The number of headers sent by the client |
| proxy.kong.http_headers_size | The size (in bytes) of headers sent by the client |
kong.read_client_http_body
A span capturing the total time taken to read the full body sent by the client. This span can identify slow clients, a buggy CDN and very large body submissions.
kong.phase.rewrite
A span capturing the execution of the rewrite phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.
kong.rewrite.plugin.{plugin_name}
A span capturing the execution of a plugin configured to run in the rewrite phase. Multiple such spans can occur in a trace.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.plugin.instance_id | The instance id of the plugin configuration that ran |
kong.io.
A span capturing network i/o timing that occurs during plugin execution or other request processing.
Can be one of:
- kong.io.http.request (requests done by the internal http client during the flow)
- kong.io.http.connect (connections done by the internal http client during the flow)
- kong.io.redis.
(redis functions) - kong.io.socket.
(functions called on the internal nginx socket)
Examples:
- OIDC plugin making calls to IdP
- Rate Limiting Advanced plugin making calls to Redis
- Custom plugins calling HTTP URLs
Multiple instances of this span can occur anywhere in the trace when i/o happens.
This span has the following attributes:
| Name | Description |
|---|---|
| network.peer.address | The address of the peer Kong connected with |
| network.protocol.name | The protocol that was used (redis, tcp, http, grpc etc) |
kong.router
A span capturing the execution of the Kong router.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.router.matched | yes/no - did the router find a match for the request |
| proxy.kong.router.route_id | The ID of the route that was matched |
| proxy.kong.router.service_id | The ID of the service that was matched |
| proxy.kong.router.upstream_path | The path of the upstream url returned by the match |
| proxy.kong.router.cache_hit | yes/no - was the match returned from cache |
kong.phase.access
A span capturing the execution of the access phase of request processing.
Any plugins configured for running in this phase will show up as individual child spans.
kong.access.plugin.{plugin_name}
A span capturing the execution of a plugin configured to run in the access phase. Multiple such spans can occur in a trace.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.plugin.instance_id | The instance id of the plugin configuration that ran |
kong.dns
A span capturing the time spent in looking up DNS.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.dns.entry | A list of DNS attempts, responses and errors if any |
kong.upstream.selection
A span capturing the total time spent in finding a healthy upstream. Depending on configuration, Kong will try to find a healthy upstream by trying various targets in order determined by the load balancing algorithm. Child spans of this span capture the individual attempts.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.upstream.lb_algorithm | the load balancing algorithm used for finding the upstream |
kong.upstream.find_upstream
A span capturing the attempt to verify a specific upstream. Kong attempts to open a TCP connection (if not KeepAlive cache is found), do a TLS handshake and send down the HTTP headers. If all of this succeeds, the upstream is healthy and Kong will finish sending the full request and wait for a response. If any of the step fails, Kong will switch to the next target and try again.
This span has the following attributes:
| Name | Description |
|---|---|
| network.peer.address | the IP address of the target upstream |
| network.peer.name | the DNS name of the target upstream |
| network.peer.port | the port number of the target |
| try_count | The number of attempts Kong has made to find a healthy upstream |
| keepalive | Is this a KeepAlive connection? |
kong.send_request_to_upstream
A span capturing the time taken to finish writing the http request to upstream. This span can be used to identify network delays between Kong and an upstream.
kong.read_headers_from_upstream
A span capturing the time taken for the upstream to generate the response headers. This span can be used to identify slowness in response generation from upstreams.
kong.read_body_from_upstream
A span capturing the time taken for the upstream to generate the response body. This span can be used to identify slowness in response generation from upstreams.
kong.phase.response
A span capturing the execution of the response phase.
Any plugins configured for running in this phase will show up as individual child spans. This phase will not run if response streaming is enabled.
kong.response.plugin.{plugin_name}
A span capturing the execution of a plugin configured to run in the response phase. Multiple such spans can occur in a trace.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.plugin.instance_id | The instance id of the plugin configuration that ran |
kong.phase.header_filter
A span capturing the execution of the header filter phase of response processing. Any plugins configured for running in this phase will show up as individual child spans.
kong.header_filter.plugin.{plugin_name}
A span capturing the execution of a plugin configured to run in the header_filter phase. Multiple such spans can occur in a trace.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.plugin.instance_id | The instance id of the plugin configuration that ran |
kong.phase.body_filter
A span capturing the execution of the body filter phase of response processing. Any plugins configured for running in this phase will show up as individual child spans.
kong.body_filter.plugin.{plugin_name}
A span capturing the execution of a plugin configured to run in the body_filter phase. Multiple such spans can occur in a trace.
This span has the following attributes:
| Name | Description |
|---|---|
| proxy.kong.plugin.instance_id | The instance id of the plugin configuration that ran |
