Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.1.x (latest)
  • 2.0.x
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Deploy a standalone control plane
    • Multi-zone deployment
    • Deploy a multi-zone global control plane
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Non-mesh traffic
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Understanding TargetRef policies
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshAccessLog (Beta)
    • MeshCircuitBreaker (Beta)
    • MeshFaultInjection (Beta)
    • MeshHealthCheck (Beta)
    • MeshHTTPRoute (Beta)
    • MeshProxyPatch (Beta)
    • MeshRateLimit (Beta)
    • MeshRetry (Beta)
    • MeshTimeout (Beta)
    • MeshTrace (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • MeshOPA (beta)
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference
    • Open source License
    • Contribute to Mesh

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • TargetRef support matrix
  • Configuration
    • Sampling
    • Tags
    • Backends
  • Examples
    • Zipkin
    • Datadog
    • Targeting parts of the infrastructure
Kong Mesh
2.1.x (latest)
  • Home
  • Kong Mesh
  • Policies
  • MeshTrace (beta)

MeshTrace (beta)

This policy uses new policy matching algorithm and is in beta state, it should not be mixed with TrafficTrace.

This policy enables publishing traces to a third party tracing solution.

Tracing is supported over HTTP, HTTP2, and gRPC protocols. You must explicitly specify the protocol for each service and data plane proxy you want to enable tracing for.

Kong Mesh currently supports the following trace exposition formats:

  • zipkin traces in this format can be sent to many different tracing backends
  • datadog

Services still need to be instrumented to preserve the trace chain across requests made across different services.

You can instrument with a language library of your choice (for zipkin and for datadog). For HTTP you can also manually forward the following headers:

  • x-request-id
  • x-b3-traceid
  • x-b3-parentspanid
  • x-b3-spanid
  • x-b3-sampled
  • x-b3-flags

TargetRef support matrix

TargetRef type top level to from
Mesh ✅ ❌ ❌
MeshSubset ✅ ❌ ❌
MeshService ✅ ❌ ❌
MeshServiceSubset ✅ ❌ ❌
MeshGatewayRoute ✅ ❌ ❌

To learn more about the information in this table, see the matching docs.

Configuration

Sampling

Most of the time setting only overall is sufficient. random and client are for advanced use cases.

You can configure sampling settings equivalent to Envoy’s:

  • overall
  • random
  • client

The value is always a percentage and is between 0 and 100.

Example:

sampling:
  overall: 80
  random: 60
  client: 40

Tags

You can add tags to trace metadata by directly supplying the value (literal) or by taking it from a header (header).

Example:

tags:
  - name: team
    literal: core
  - name: env
    header:
      name: x-env
      default: prod
  - name: version
    header:
      name: x-version

If a value is missing for header, default is used. If default isn’t provided, then the tag won’t be added.

Backends

Datadog

You can configure a Datadog backend with a url and splitService.

Example:

datadog:
  url: http://my-agent:8080 # Required. The url to reach a running datadog agent
  splitService: true # Default to false. If true, it will split inbound and outbound requests in different services in Datadog

The splitService property determines if Datadog service names should be split based on traffic direction and destination. For example, with splitService: true and a backend service that communicates with a couple of databases, you would get service names like backend_INBOUND, backend_OUTBOUND_db1, and backend_OUTBOUND_db2 in Datadog.

Zipkin

In most cases the only field you’ll want to set in url.

Example:

zipkin:
  url: http://jaeger-collector:9411/api/v2/spans # Required. The url to a zipkin collector to send traces to 
  traceId128bit: false # Default to false which will expose a 64bits traceId. If true, the id of the trace is 128bits
  apiVersion: httpJson # Default to httpJson. It can be httpJson, httpProto and is the version of the zipkin API
  sharedSpanContext: false # Default to true. If true, the inbound and outbound traffic will share the same span. 

Examples

Zipkin

Kubernetes
Universal

Simple example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - zipkin:
          url: http://jaeger-collector.mesh-observability:9411/api/v2/spans

Full example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - zipkin:
          url: http://jaeger-collector.mesh-observability:9411/api/v2/spans
          apiVersion: httpJson
    tags:
      - name: team
        literal: core
      - name: env
        header:
          name: x-env
          default: prod
      - name: version
        header:
          name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40

Apply the configuration with kubectl apply -f [..].

Simple example:

type: MeshTrace
name: default
mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - zipkin:
          url: http://jaeger-collector:9411/api/v2/spans

Full example:

type: MeshTrace
name: default
mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - zipkin:
          url: http://jaeger-collector:9411/api/v2/spans
          apiVersion: httpJson
    tags:
      - name: team
        literal: core
      - name: env
        header:
          name: x-env
          default: prod
      - name: version
        header:
          name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Datadog

This assumes a Datadog agent is configured and running. If you haven’t already check the Datadog observability page.

Kubernetes
Universal

Simple Example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - datadog:
          url: http://trace-svc.default.svc.cluster.local:8126

Full Example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - datadog:
          url: http://trace-svc.default.svc.cluster.local:8126
          splitService: true
    tags:
      - name: team
        literal: core
      - name: env
        header:
          name: x-env
          default: prod
      - name: version
        header:
          name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40

where trace-svc is the name of the Kubernetes Service you specified when you configured the Datadog APM agent.

Apply the configuration with kubectl apply -f [..].

Simple example:

type: MeshTrace
name: default
mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - datadog:
          url: http://127.0.0.1:8126

Full example:

type: MeshTrace
name: default
mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - datadog:
          url: http://127.0.0.1:8126
          splitService: true
    tags:
      - name: team
        literal: core
      - name: env
        header:
          name: x-env
          default: prod
      - name: version
        header:
          name: x-version
    sampling:
      overall: 80
      random: 60
      client: 40

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Targeting parts of the infrastructure

While usually you want all the traces to be sent to the same tracing backend, you can target parts of a Mesh by using a finer-grained targetRef and a designated backend to trace different paths of our service traffic. This is especially useful when you want traces to never leave a world region, or a cloud, for example.

In this example, we have two zones east and west, each of these with their own Zipkin collector: east.zipkincollector:9411/api/v2/spans and west.zipkincollector:9411/api/v2/spans. We want dataplane proxies in each zone to only send traces to their local collector.

To do this, we use a TargetRef kind value of MeshSubset to filter which dataplane proxy a policy applies to.

Universal
Kubernetes

West only policy:

type: MeshTrace
name: trace-west
mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      kuma.io/zome: west
  default:
    backends:
      - zipkin:
          url: http://west.zipkincollector:9411/api/v2/spans

East only policy:

type: MeshTrace
name: trace-east
mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      kuma.io/zome: east
  default:
    backends:
      - zipkin:
          url: http://east.zipkincollector:9411/api/v2/spans

West only policy:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
    name: trace-west
    namespace: kong-mesh-system
spec:
  targetRef:
    kind: MeshSubset
    tags:
      kuma.io/zome: west
  default:
    backends:
      - zipkin:
          url: http://west.zipkincollector:9411/api/v2/spans

East only policy:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: trace-east
  namespace: kong-mesh-system
spec:
  targetRef:
    kind: MeshSubset
    tags:
      kuma.io/zome: east
  default:
    backends:
      - zipkin:
          url: http://east.zipkincollector:9411/api/v2/spans
Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023