Skip to content
|
search
Plugin Hub
header icon

Zipkin

Propagate Zipkin distributed tracing spans, and report spans to a Zipkin server.

Configuration Reference

This plugin is compatible with DB-less mode.

In DB-less mode, you configure Kong Gateway declaratively. Therefore, the Admin API is mostly read-only. The only tasks it can perform are all related to handling the declarative config, including:

  • Setting a target's health status in the load balancer
  • Validating configurations against schemas
  • Uploading the declarative configuration using the /config endpoint

Example plugin configuration

The following examples provide some typical configurations for enabling the zipkin plugin on a service.

Make the following request:

curl -X POST http://localhost:8001/services/SERVICE_NAME|SERVICE_ID/plugins \
    --data "name=zipkin"  \
    --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
    --data "config.sample_ratio=0.001" \
    --data "config.include_credential=true"

Replace SERVICE_NAME|SERVICE_ID with the id or name of the service that this plugin configuration will target.

First, create a KongPlugin resource:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: zipkin-example
config: 
  http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  sample_ratio: 0.001
  include_credential: true
plugin: zipkin

Next, apply the KongPlugin resource to a service by annotating the service as follows:

apiVersion: v1
kind: Service
metadata:
  name: SERVICE_NAME|SERVICE_ID
  labels:
    app: SERVICE_NAME|SERVICE_ID
  annotations:
    konghq.com/plugins: zipkin-example
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: SERVICE_NAME|SERVICE_ID
  selector:
    app: SERVICE_NAME|SERVICE_ID

Replace SERVICE_NAME|SERVICE_ID with the id or name of the service that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any service, consumer, or route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

Add this section to your declarative configuration file:

plugins:
- name: zipkin
  service: SERVICE_NAME|SERVICE_ID
  config: 
    http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
    sample_ratio: 0.001
    include_credential: true

Replace SERVICE_NAME|SERVICE_ID with the id or name of the service that this plugin configuration will target.

You can configure this plugin through the Kong Manager UI.

  1. In Kong Manager, select the workspace.
  2. From the Services section, click View for the service row.
  3. From the plugin section, click Add Plugin.
  4. Find and select the Zipkin plugin.
  5. If the option is available, select Scoped.
  6. Add the service name and ID to the Service field if it is not already pre-filled.

  7. Configure the plugin’s parameters.

    You can test out the plugin with the following sample configuration:

    • Config.Include Credential: select checkbox
  8. Click Create.

The following examples provide some typical configurations for enabling the zipkin plugin on a route.

Make the following request:

curl -X POST http://localhost:8001/routes/ROUTE_NAME|ROUTE_ID/plugins \
  --data "name=zipkin"  \
    --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
    --data "config.sample_ratio=0.001" \
    --data "config.include_credential=true"

Replace ROUTE_NAME|ROUTE_ID with the id or name of the route that this plugin configuration will target.

First, create a KongPlugin resource:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: zipkin-example
config: 
  http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  sample_ratio: 0.001
  include_credential: true
plugin: zipkin

Then, apply it to an ingress (route or routes) by annotating the ingress as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ROUTE_NAME|ROUTE_ID
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/plugins: zipkin-example
spec:
  rules:
  - host: examplehostname.com
    http:
      paths:
      - path: /bar
        backend:
          service:
            name: echo
            port:
              number: 80

Replace ROUTE_NAME|ROUTE_ID with the id or name of the route that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any service, consumer, or route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

Add this section to your declarative configuration file:

plugins:
- name: zipkin
  route: ROUTE_NAME
  config: 
    http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
    sample_ratio: 0.001
    include_credential: true

Replace ROUTE_NAME|ROUTE_ID with the id or name of the route that this plugin configuration will target.

You can configure this plugin through the Kong Manager UI.

  1. In Kong Manager, select the workspace.
  2. Open Routes from the menu, then click View for the route row.
  3. From the plugin section, click Add Plugin.
  4. Find and select the Zipkin plugin.
  5. If the option is available, select Scoped.
  6. Add the route ID if it is not already prefilled.

  7. Configure the plugin’s parameters.

    You can test out the plugin with the following sample configuration:

    • Config.Include Credential: select checkbox
  8. Click Create.

The following examples provide some typical configurations for enabling the zipkin plugin on a consumer.

Make the following request:

curl -X POST http://localhost:8001/consumers/CONSUMER_NAME|CONSUMER_ID/plugins \
  --data "name=zipkin"  \
    --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
    --data "config.sample_ratio=0.001" \
    --data "config.include_credential=true"

Replace CONSUMER_NAME|CONSUMER_ID with the id or name of the consumer that this plugin configuration will target.

You can combine consumer.id, service.id, or route.id in the same request, to further narrow the scope of the plugin.

First, create a KongPlugin resource:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: zipkin-example
config: 
  http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  sample_ratio: 0.001
  include_credential: true
plugin: zipkin

Then, apply it to a consumer by annotating the KongConsumer resource as follows:

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: CONSUMER_NAME|CONSUMER_ID
  annotations:
    konghq.com/plugins: zipkin-example
    kubernetes.io/ingress.class: kong

Replace CONSUMER_NAME|CONSUMER_ID with the id or name of the consumer that this plugin configuration will target.

Note: The KongPlugin resource only needs to be defined once and can be applied to any service, consumer, or route in the namespace. If you want the plugin to be available cluster-wide, create the resource as a KongClusterPlugin instead of KongPlugin.

Add this section to your declarative configuration file:

plugins:
- name: zipkin
  consumer: CONSUMER_NAME|CONSUMER_ID
  config: 
    http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
    sample_ratio: 0.001
    include_credential: true

Replace CONSUMER_NAME|CONSUMER_ID with the id or name of the consumer that this plugin configuration will target.

You can configure this plugin through the Kong Manager UI.

  1. In Kong Manager, select the workspace.
  2. From the Consumers section, click View for the consumer row.
  3. Select the Plugins tab, then click Add Plugin.
  4. Find and select the Zipkin plugin.
  5. If the option is available, select Scoped.
  6. Add the consumer ID if it is not already prefilled.

  7. Configure the plugin’s parameters.

    You can test out the plugin with the following sample configuration:

    • Config.Include Credential: select checkbox
  8. Click Create.

A plugin which is not associated to any service, route, or consumer is considered global, and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

The following examples provide some typical configurations for enabling the zipkin plugin globally.

Make the following request:

curl -X POST http://localhost:8001/plugins/ \
  --data "name=zipkin"  \
    --data "config.http_endpoint=http://your.zipkin.collector:9411/api/v2/spans" \
    --data "config.sample_ratio=0.001" \
    --data "config.include_credential=true"

Create a KongClusterPlugin resource and label it as global:

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-zipkin>
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: \"true\"
config: 
  http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
  sample_ratio: 0.001
  include_credential: true
plugin: zipkin

Add a plugins entry in the declarative configuration file:

plugins:
- name: zipkin
  config: 
    http_endpoint: http://your.zipkin.collector:9411/api/v2/spans
    sample_ratio: 0.001
    include_credential: true

You can configure this plugin through the Kong Manager UI.

  1. In Kong Manager, select the workspace.
  2. Open Plugins from the menu, then click New Plugin.
  3. Find and select the Zipkin plugin.
  4. If the option is available, set the plugin scope to Global.

  5. Configure the plugin’s parameters.

    You can test out the plugin with the following sample configuration:

    • Config.Include Credential: select checkbox
  6. Click Create.

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Form Parameter Description
name
required

Type: string		 The name of the plugin, in this case zipkin.
service.name or service.id

Type: string		 The name or ID of the service the plugin targets.

Set one of these parameters if adding the plugin to a service through the top-level /plugins endpoint.

Not required if using /services/SERVICE_NAME|SERVICE_ID/plugins.
route.name or route.id

Type: string		 The name or ID of the route the plugin targets.

Set one of these parameters if adding the plugin to a route through the top-level /plugins endpoint.

Not required if using /routes/ROUTE_NAME|ROUTE_ID/plugins.
consumer.name or consumer.id

Type: string		 The name or ID of the consumer the plugin targets.

Set one of these parameters if adding the plugin to a consumer through the top-level /plugins endpoint.

Not required if using /consumers/CONSUMER_NAME|CONSUMER_ID/plugins.
enabled

Type: boolean

Default value: true		 Whether this plugin will be applied.
config.local_service_name
required

Type: string

Default value: kong

The name of the service as displayed in Zipkin. Customize this name to tell your Kong Gateway services apart in Zipkin request traces.
config.http_endpoint
optional

Type: string

Default value:

The full HTTP(S) endpoint to which Zipkin spans should be sent by Kong. If not specified, the Zipkin plugin will only act as a tracing header generator/transmitter.
config.sample_ratio
optional

Type: number

Default value: 0.001

How often to sample requests that do not contain trace IDs. Set to 0 to turn sampling off, or to 1 to sample all requests. The value must be between zero (0) and one (1), inclusive.
config.default_service_name
optional

Type: string

Set a default service name to override unknown-service-name in the Zipkin spans.
config.include_credential
required

Type: boolean

Default value: true

Specify whether the credential of the currently authenticated consumer should be included in metadata sent to the Zipkin server.
config.traceid_byte_count
required

Type: integer

Default value: 16

The length in bytes of each request’s Trace ID. The value can be either 8 or 16.
config.header_type
required

Type: string

Default value: preserve

All HTTP requests going through the plugin are tagged with a tracing HTTP request. This property codifies what kind of tracing header the plugin expects on incoming requests.

Possible values: b3, b3-single, w3c, preserve, jaeger, ot, or ignore.

  • b3: Expects Zipkin’s B3 multiple headers on incoming requests, and will add them to the transmitted requests if the headers are missing from those requests.
  • b3-single: Expects or adds Zipkin’s B3 single-header tracing headers.
  • w3c: Expects or adds W3C’s traceparent tracing header.
  • preserve: Does not expect any format, and will transmit whatever header is recognized or present, with a default of b3 if none is found. In case of a mismatch between the expected and incoming tracing headers (for example, when header_type is set to b3 but a w3c-style tracing header is found in the incoming request), then the plugin will add both kinds of tracing headers to the request and generate a mismatch warning in the logs.
  • jaeger: Expects or adds Jaeger-style tracing headers (uber-trace-id).
  • ot: Expects or adds OpenTelemetry tracing headers of the form ot-tracer-*.
  • ignore: Does not read any tracing headers from the incoming request. Starts a new request using the default_header_type value, or falls back to b3 if there is no default_header_type value set.
config.default_header_type
required

Type: string

Default value: b3

Allows specifying the type of header to be added to requests with no pre-existing tracing headers and when config.header_type is set to "preserve". When header_type is set to any other value, default_header_type is ignored.

Possible values are b3, b3-single, w3c, jaeger, or ot. See the entry for header_type for value definitions.
config.tags_header
required

Type: string

Default value: Zipkin-Tags

The Zipkin plugin will add extra headers to the tags associated with any HTTP requests that come with a header named as configured by this property. The format is name_of_tag=value_of_tag, separated by semicolons (;).

For example: with the default value, a request with the header Zipkin-Tags: fg=blue; bg=red will generate a trace with the tag fg with value blue, and another tag called bg with value red.
config.static_tags
optional

Type: array of string tags

Default value: []

The tags specified on this property will be added to the generated request traces. For example: [ { "name": "color", "value": "red" } ].
config.http_span_name
required

Type: string

Default value: method

Specify whether to include the HTTP path in the span name.

Options:

  • method: Do not include the HTTP path. This is the default.
  • method_path: Include the HTTP path.
config.connect_timeout
optional

Type: number

Default value: 2000

The timeout, in milliseconds, for establishing a connection to the Zipkin server.
config.send_timeout
optional

Type: number

Default value: 5000

The timeout, in milliseconds, between two successive write operations when sending a request to the Zipkin server.
config.read_timeout
optional

Type: number

Default value: 5000

The timeout, in milliseconds, between two successive read operations when receiving a response from the Zipkin server.
config.response_header_for_traceid
optional

Type: string

Set a header name to append to responses. This header name is sent to the client, making it possible to trace the ID of the correlated request.

How it Works

When enabled, this plugin traces requests in a way compatible with zipkin.

The code is structured around an OpenTracing core using the opentracing-lua library to collect timing data of a request in each of Kong’s phases. The plugin uses opentracing-lua compatible extractor, injector, and reporters to implement Zipkin’s protocols.

Reporter

An OpenTracing “reporter” is how tracing data is reported to another system. This plugin records tracing data for a given request, and sends it as a batch to a Zipkin server using the Zipkin v2 API. Note that zipkin version 1.31 or higher is required.

The http_endpoint configuration variable must contain the full uri including scheme, host, port and path sections (i.e. your uri likely ends in /api/v2/spans).

Spans

The plugin does request sampling. For each request which triggers the plugin, a random number between 0 and 1 is chosen.

If the number is greater than the configured sample_ratio, then a trace with several spans will be generated. If sample_ratio is set to 1, then all requests will generate a trace (this might be very noisy).

For each request that gets traced, the following spans are produced:

  • Request span: 1 per request. Encompasses the whole request in kong (kind: SERVER). The Proxy and Balancer spans are children of this span. It contains the following logs/annotations for the rewrite phase:

    • krs - kong.rewrite.start
    • krf - kong.rewrite.finish

    The Request span has the following tags:

    • lc: Hardcoded to kong.
    • kong.service: The uuid of the service matched when processing the request, if any.
    • kong.service_name: The name of the service matched when processing the request, if service exists and has a name attribute.
    • kong.route: The uuid of the route matched when processing the request, if any (it can be nil on non-matched requests).

    • kong.route_name: The name of the route matched when processing the request, if route exists and has a name attribute.

    • http.method: The HTTP method used on the original request (only for HTTP requests).
    • http.path: The path of the request (only for HTTP requests).
    • If the plugin tags_header config option is set, and the request contains headers with the appropriate name and correct encoding tags, then the trace will include the tags.
    • If the plugin static_tags config option is set, then the tags in the config option will be included in the trace.
  • Proxy span: 1 per request, encompassing most of Kong’s internal processing of a request (kind: CLIENT). Contains the following logs/annotations for the start/finish of the of the Kong plugin phases:
    • kas - kong.access.start
    • kaf - kong.access.finish
    • kbs - kong.body_filter.start
    • kbf - kong.body_filter.finish
    • khs - kong.header_filter.start
    • khf - kong.header_filter.finish
    • kps - kong.preread.start (only for stream requests)
    • kpf - kong.preread.finish (only for stream requests)
  • Balancer span(s): 0 or more per request, each encompassing one balancer attempt (kind: CLIENT). Contains the following tags specific to load balancing:
    • kong.balancer.try: A number indicating the attempt (1 for the first load-balancing attempt, 2 for the second, and so on).
    • peer.ipv4 or peer.ipv6 for the balancer IP.
    • peer.port for the balanced port.
    • error: Set to true if the balancing attempt was unsuccessful, otherwise unset.
    • http.status_code: The HTTP status code received, in case of error.
    • kong.balancer.state: An NGINX-specific description of the error, next/failed for HTTP failures, or 0 for stream failures. Equivalent to state_name in OpenResty’s balancer’s get_last_failure function.

See also

For more information, read the Kong blog post.

Changelog

Kong Gateway 3.1.x

  • Added the parameter response_header_for_traceid.

Kong Gateway 3.0.x

  • Added support for including the HTTP path in the span name with the http_span_name configuration parameter. #8150
  • Added support for socket connect and send/read timeouts through the connect_timeout, send_timeout, and read_timeout configuration parameters. This can help mitigate ngx.timer saturation when upstream collectors are unavailable or slow. #8735

Kong Gateway 2.7.x

  • Added a new parameter: local_service_name.
  • Added a new ignore option for the header_type parameter.

Kong Gateway 2.5.x

  • The plugin now includes the following tags: kong.route, kong.service_name, and kong.route_name.

Kong Gateway 2.4.x

  • Added support for OT and Jaeger style uber-trace-id headers.
  • The plugin now allows insertion of custom tags on the Zipkin request trace.
  • The plugin now allows the creation of baggage items on child spans.

Kong Gateway 2.3.x

  • Added the default_header_type and static_tags configuration parameters.