You are browsing unreleased documentation. See the latest documentation here.

MeshTrace

This policy uses new policy matching algorithm. Do not combine with TrafficTrace.

This policy enables publishing traces to a third party tracing solution.

Tracing is supported over HTTP, HTTP2, and gRPC protocols. You must explicitly specify the protocol for each service and data plane proxy you want to enable tracing for.

Kong Mesh currently supports the following trace exposition formats:

Zipkin traces in this format can be sent to many different tracing backends
Datadog

Services still need to be instrumented to preserve the trace chain across requests made across different services.

You can instrument with a language library of your choice (for Zipkin and for Datadog). For HTTP you can also manually forward the following headers:

x-request-id

x-b3-traceid

x-b3-parentspanid

x-b3-spanid

x-b3-sampled

x-b3-flags

TargetRef support matrix

Sidecar

Builtin Gateway

Delegated Gateway

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `Dataplane`, `MeshSubset(deprecated)`

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `MeshGateway`

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `MeshSubset`

To learn more about the information in this table, see the matching docs.

Configuration

Sampling

Most of the time setting only overall is sufficient. random and client are for advanced use cases.

You can configure sampling settings equivalent to Envoy’s:

The value is always a percentage and is between 0 and 100.

Example:

sampling:
  overall: 80
  random: 60
  client: 40

Backends

Datadog

You can configure a Datadog backend with a url and splitService.

Example:

datadog:
  url: http://my-agent:8080 # Required. The url to reach a running datadog agent
  splitService: true # Default to false. If true, it will split inbound and outbound requests in different services in Datadog

The splitService property determines if Datadog service names should be split based on traffic direction and destination. For example, with splitService: true and a backend service that communicates with a couple of databases, you would get service names like backend_INBOUND, backend_OUTBOUND_db1, and backend_OUTBOUND_db2 in Datadog.

Zipkin

In most cases the only field you’ll want to set is url.

Example:

zipkin:
  url: http://jaeger-collector:9411/api/v2/spans # Required. The url to a zipkin collector to send traces to 
  traceId128bit: false # Default to false which will expose a 64bits traceId. If true, the id of the trace is 128bits
  apiVersion: httpJson # Default to httpJson. It can be httpJson, httpProto and is the version of the zipkin API
  sharedSpanContext: false # Default to true. If true, the inbound and outbound traffic will share the same span. 

OpenTelemetry

The only field you can set is endpoint.

Example:

openTelemetry:
  endpoint: otel-collector:4317 # Required. Address of OpenTelemetry collector

Examples

Zipkin

Simple example:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    backends:
    - type: Zipkin
      zipkin:
        url: http://jaeger-collector:9411/api/v2/spans
        apiVersion: httpJson

    
      type: MeshTrace
name: default
mesh: default
spec:
  default:
    backends:
    - type: Zipkin
      zipkin:
        url: http://jaeger-collector:9411/api/v2/spans
        apiVersion: httpJson

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "default" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "default"
  spec = {
    default = {
      backends = [
        {
          type = "Zipkin"
          zipkin = {
            url = "http://jaeger-collector:9411/api/v2/spans"
            api_version = "httpJson"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Full example:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: Zipkin
      zipkin:
        url: http://jaeger-collector:9411/api/v2/spans
        apiVersion: httpJson

    
      type: MeshTrace
name: default
mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: Zipkin
      zipkin:
        url: http://jaeger-collector:9411/api/v2/spans
        apiVersion: httpJson

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "default" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "default"
  spec = {
    default = {
      tags = [
        {
          name = "team"
          literal = "core"
        },
        {
          name = "env"
          header = {
            name = "x-env"
            default = "prod"
          }
        },
        {
          name = "version"
          header = {
            name = "x-version"
          }
        }
      ]
      sampling = {
        overall = "80"
        random = "60"
        client = "40"
      }
      backends = [
        {
          type = "Zipkin"
          zipkin = {
            url = "http://jaeger-collector:9411/api/v2/spans"
            api_version = "httpJson"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Datadog

This assumes a Datadog agent is configured and running. If you haven’t already check the Datadog observability page.

Simple example:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    backends:
    - type: Datadog
      datadog:
        url: http://127.0.0.1:8126

    
      type: MeshTrace
name: default
mesh: default
spec:
  default:
    backends:
    - type: Datadog
      datadog:
        url: http://127.0.0.1:8126

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "default" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "default"
  spec = {
    default = {
      backends = [
        {
          type = "Datadog"
          datadog = {
            url = "http://127.0.0.1:8126"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Full example:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: Datadog
      datadog:
        url: http://127.0.0.1:8126
        splitService: true

    
      type: MeshTrace
name: default
mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: Datadog
      datadog:
        url: http://127.0.0.1:8126
        splitService: true

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "default" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "default"
  spec = {
    default = {
      tags = [
        {
          name = "team"
          literal = "core"
        },
        {
          name = "env"
          header = {
            name = "x-env"
            default = "prod"
          }
        },
        {
          name = "version"
          header = {
            name = "x-version"
          }
        }
      ]
      sampling = {
        overall = "80"
        random = "60"
        client = "40"
      }
      backends = [
        {
          type = "Datadog"
          datadog = {
            url = "http://127.0.0.1:8126"
            split_service = "true"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

OpenTelemetry

This assumes a OpenTelemetry collector is configured and running. If you haven’t already check the OpenTelementry operator.

Simple example:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    backends:
    - type: OpenTelemetry
      openTelemetry:
        endpoint: otel-collector.com:4317

    
      type: MeshTrace
name: default
mesh: default
spec:
  default:
    backends:
    - type: OpenTelemetry
      openTelemetry:
        endpoint: otel-collector.com:4317

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "default" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "default"
  spec = {
    default = {
      backends = [
        {
          type = "OpenTelemetry"
          open_telemetry = {
            endpoint = "otel-collector.com:4317"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Full example:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: OpenTelemetry
      openTelemetry:
        endpoint: otel-collector.com:4317

    
      type: MeshTrace
name: default
mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: OpenTelemetry
      openTelemetry:
        endpoint: otel-collector.com:4317

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "default" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "default"
  spec = {
    default = {
      tags = [
        {
          name = "team"
          literal = "core"
        },
        {
          name = "env"
          header = {
            name = "x-env"
            default = "prod"
          }
        },
        {
          name = "version"
          header = {
            name = "x-version"
          }
        }
      ]
      sampling = {
        overall = "80"
        random = "60"
        client = "40"
      }
      backends = [
        {
          type = "OpenTelemetry"
          open_telemetry = {
            endpoint = "otel-collector.com:4317"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Targeting parts of the infrastructure

While usually you want all the traces to be sent to the same tracing backend, you can target parts of a Mesh by using a finer-grained targetRef and a designated backend to trace different paths of our service traffic. This is especially useful when you want traces to never leave a world region, or a cloud, for example.

In this example, we have two zones east and west, each of these with their own Zipkin collector: east.zipkincollector:9411/api/v2/spans and west.zipkincollector:9411/api/v2/spans. We want data plane proxies in each zone to only send traces to their local collector.

To do this, we use a TargetRef kind value of MeshSubset to filter which data plane proxy a policy applies to.

West only policy:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: trace-west
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: west
  default:
    backends:
    - type: Zipkin
      zipkin:
        url: http://west.zipkincollector:9411/api/v2/spans
        apiVersion: httpJson

    
      type: MeshTrace
name: trace-west
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: west
  default:
    backends:
    - type: Zipkin
      zipkin:
        url: http://west.zipkincollector:9411/api/v2/spans
        apiVersion: httpJson

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "trace_west" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "trace-west"
  spec = {
    target_ref = {
      kind = "Dataplane"
      labels = {
        kuma.io/zone = "west"
      }
    }
    default = {
      backends = [
        {
          type = "Zipkin"
          zipkin = {
            url = "http://west.zipkincollector:9411/api/v2/spans"
            api_version = "httpJson"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

East only policy:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: trace-east
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: east
  default:
    backends:
    - zipkin:
        url: http://east.zipkincollector:9411/api/v2/spans
        apiVersion: httpJson

    
      type: MeshTrace
name: trace-east
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: east
  default:
    backends:
    - zipkin:
        url: http://east.zipkincollector:9411/api/v2/spans
        apiVersion: httpJson

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_trace" "trace_east" {
  provider = konnect-beta
  type = "MeshTrace"
  name = "trace-east"
  spec = {
    target_ref = {
      kind = "Dataplane"
      labels = {
        kuma.io/zone = "east"
      }
    }
    default = {
      backends = [
        {
          zipkin = {
            url = "http://east.zipkincollector:9411/api/v2/spans"
            api_version = "httpJson"
          }
        }
      ]
    }
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

All policy options

Spec is the specification of the Kuma MeshTrace resource.

Type: object

Properties

default
- MeshTrace configuration.
- Type: object
- Properties
  - backends
    - A one element array of backend definition.Envoy allows configuring only 1 backend, so the natural way ofrepresenting that would be just one object. Unfortunately due to thereasons explained in MADR 009-tracing-policy this has to be a one elementarray for now.
    - Type: array
    - Item Count: ≤ 1
      - Items
      - Only one of zipkin, datadog or openTelemetry can be used.
      - Type: object
      - Properties
        
        datadog
        
        Datadog backend configuration.
        
        Type: object
        
        Properties
        
        splitService
        
        Determines if datadog service name should be split based on trafficdirection and destination. For example, with splitService: true and abackend service that communicates with a couple of databases, you wouldget service names like backend_INBOUND, backend_OUTBOUND_db1, andbackend_OUTBOUND_db2 in Datadog.
        
        Type: boolean
        
        Default: false
        
        url required
        
        Address of Datadog collector, only host and port are allowed (no paths,fragments etc.)
        
        Type: string
        
        openTelemetry
        
        OpenTelemetry backend configuration.
        
        Type: object
        
        Properties
        
        endpoint required
        
        Address of OpenTelemetry collector.
        
        Type: string
        
        Length: ≥ 1
        
        type required
        
        Type: string
        
        The value is restricted to the following:
        
        "Zipkin"
        
        "Datadog"
        
        "OpenTelemetry"
        
        zipkin
        
        Zipkin backend configuration.
        
        Type: object
        
        Properties
        
        apiVersion
        
        Version of the API.https://github.com/envoyproxy/envoy/blob/v1.22.0/api/envoy/config/trace/v3/zipkin.proto#L66
        
        Type: string
        
        The value is restricted to the following:
        
        "httpJson"
        
        "httpProto"
        
        Default: "httpJson"
        
        sharedSpanContext
        
        Determines whether client and server spans will share the same spancontext.https://github.com/envoyproxy/envoy/blob/v1.22.0/api/envoy/config/trace/v3/zipkin.proto#L63
        
        Type: boolean
        
        Default: true
        
        traceId128bit
        
        Generate 128bit traces.
        
        Type: boolean
        
        Default: false
        
        url required
        
        Address of Zipkin collector.
        
        Type: string
  - sampling
    - Sampling configuration.Sampling is the process by which a decision is made on whether toprocess/export a span or not.
    - Type: object
    - Properties
      - client
        
        Target percentage of requests that will be force traced if the'x-client-trace-id' header is set. Mirror of clientsampling in Envoyhttps://github.com/envoyproxy/envoy/blob/v1.22.0/api/envoy/config/filter/network/httpconnectionmanager/v2/httpconnection_manager.proto#L127-L133Either int or decimal represented as string.If not specified then the default value is 100.
      - overall
        
        Target percentage of requests will be tracedafter all other sampling checks have been applied (client, force tracing,random sampling). This field functions as an upper limit on the totalconfigured sampling rate. For instance, setting client to 100but overall to 1 will result in only 1% of client requests withthe appropriate headers to be force traced. Mirror ofoverallsampling in Envoyhttps://github.com/envoyproxy/envoy/blob/v1.22.0/api/envoy/config/filter/network/httpconnectionmanager/v2/httpconnection_manager.proto#L142-L150Either int or decimal represented as string.If not specified then the default value is 100.
      - random
        
        Target percentage of requests that will be randomly selected for tracegeneration, if not requested by the client or not forced.Mirror of randomsampling in Envoyhttps://github.com/envoyproxy/envoy/blob/v1.22.0/api/envoy/config/filter/network/httpconnectionmanager/v2/httpconnection_manager.proto#L135-L140Either int or decimal represented as string.If not specified then the default value is 100.
  - tags
    - Custom tags configuration. You can add custom tags to traces based onheaders or literal values.
    - Type: array
      - Items
      - Custom tags configuration.Only one of literal or header can be used.
      - Type: object
      - Properties
        
        header
        
        Tag taken from a header.
        
        Type: object
        
        Properties
        
        default
        
        Default value to use if header is missing.If the default is missing and there is no value the tag will not beincluded.
        
        Type: string
        
        name required
        
        Name of the header.
        
        Type: string
        
        literal
        
        Tag taken from literal value.
        
        Type: string
        
        name required
        
        Name of the tag.
        
        Type: string
targetRef
- TargetRef is a reference to the resource the policy takes an effect on.The resource could be either a real store object or virtual resourcedefined inplace.
- Type: object
- Properties
  - kind required
    - Kind of the referenced resource
    - Type: string
    - The value is restricted to the following:
      1. "Mesh"
      2. "MeshSubset"
      3. "MeshGateway"
      4. "MeshService"
      5. "MeshExternalService"
      6. "MeshMultiZoneService"
      7. "MeshServiceSubset"
      8. "MeshHTTPRoute"
      9. "Dataplane"
  - labels
    - Labels are used to select group of MeshServices that match labels. Either Labels orName and Namespace can be used.
    - Type: object
    - This schema accepts additional properties.
    - Properties
  - mesh
    - Mesh is reserved for future use to identify cross mesh resources.
    - Type: string
  - name
    - Name of the referenced resource. Can only be used with kinds: MeshService,MeshServiceSubset and MeshGatewayRoute
    - Type: string
  - namespace
    - Namespace specifies the namespace of target resource. If empty only resources in policy namespacewill be targeted.
    - Type: string
  - proxyTypes
    - ProxyTypes specifies the data plane types that are subject to the policy. When not specified,all data plane types are targeted by the policy.
    - Type: array
      - Items
      - Type: string
      - The value is restricted to the following:
        
        "Sidecar"
        
        "Gateway"
  - sectionName
    - SectionName is used to target specific section of resource.For example, you can target port from MeshService.ports[] by its name. Only traffic to this port will be affected.
    - Type: string
  - tags
    - Tags used to select a subset of proxies by tags. Can only be used with kindsMeshSubset and MeshServiceSubset
    - Type: object
    - This schema accepts additional properties.
    - Properties

Generated with json-schema-md-doc Fri May 23 2025 04:57:08 GMT+0000 (Coordinated Universal Time)