Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Early Access
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.0.x (latest)
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Multi-zone deployment
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Policy matching
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshTrace (Beta)
    • MeshAccessLog (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • Add a tracing backend to the mesh
    • Zipkin
    • Datadog
  • Add TrafficTrace resource
Kong Mesh
2.0.x (latest)
  • Home
  • Kong Mesh
  • Policies
  • Traffic Trace

Traffic Trace

This policy enables tracing logging to a third party tracing solution.

Tracing is supported over HTTP, HTTP2, and gRPC protocols. You must explicitly specify the protocol for each service and data plane proxy you want to enable tracing for.

You must also:

  1. Add a tracing backend. You specify a tracing backend as a Mesh resource property.
  2. Add a TrafficTrace resource. You pass the backend to the TrafficTrace resource.

Kong Mesh currently supports the following trace exposition formats:

  • zipkin traces in this format can be sent to many different tracing backends.
  • datadog

Services still need to be instrumented to preserve the trace chain across requests made across different services.

You can instrument with a language library of your choice (for zipkin and for datadog). For HTTP you can also manually forward the following headers:

  • x-request-id
  • x-b3-traceid
  • x-b3-parentspanid
  • x-b3-spanid
  • x-b3-sampled
  • x-b3-flags

Add a tracing backend to the mesh

Zipkin

This assumes you already have a zipkin compatible collector running. If you haven’t, read the observability docs.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  tracing:
    defaultBackend: jaeger-collector
    backends:
    - name: jaeger-collector
      type: zipkin
      sampling: 100.0
      conf:
        url: http://jaeger-collector.mesh-observability:9411/api/v2/spans # If not using `kuma install observability` replace by any zipkin compatible collector address.

Apply the configuration with kubectl apply -f [..].

type: Mesh
name: default
tracing:
  defaultBackend: jaeger-collector
  backends:
  - name: jaeger-collector
    type: zipkin
    sampling: 100.0
    conf:
      url: http://my-jaeger-collector:9411/api/v2/spans # Replace by any zipkin compatible collector address.

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Datadog

This assumes a Datadog agent is configured and running. If you haven’t already check the Datadog observability page.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  tracing:
    defaultBackend: datadog-collector
    backends:
    - name: datadog-collector
      type: datadog
      sampling: 100.0
      conf:
        address: trace-svc.datadog.svc.cluster.local
        port: 8126

where trace-svc is the name of the Kubernetes Service you specified when you configured the Datadog APM agent.

Apply the configuration with kubectl apply -f [..].

type: Mesh
name: default
tracing:
  defaultBackend: datadog-collector
  backends:
  - name: datadog-collector
    type: datadog
    sampling: 100.0
    conf:
      address: 127.0.0.1
      port: 8126

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

The defaultBackend property specifies the tracing backend to use if it’s not explicitly specified in the TrafficTrace resource.

Add TrafficTrace resource

Next, create TrafficTrace resources that specify how to collect traces, and which backend to send them to.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: TrafficTrace
mesh: default
metadata:
  name: trace-all-traffic
spec:
  selectors:
  - match:
      kuma.io/service: '*'
  conf:
    backend: jaeger-collector # or the name of any backend defined for the mesh 

Apply the configuration with kubectl apply -f [..].

type: TrafficTrace
name: trace-all-traffic
mesh: default
selectors:
- match:
    kuma.io/service: '*'
conf:
  backend: jaeger-collector # or the name of any backend defined for the mesh

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

When backend field is omitted, the logs will be forwarded into the defaultBackend of that Mesh.

You can also add tags to apply the TrafficTrace resource only a subset of data plane proxies. TrafficTrace is a Dataplane policy, so you can specify any of the selectors tags.

While most commonly we want all the traces to be sent to the same tracing backend, we can optionally create multiple tracing backends in a Mesh resource and store traces for different paths of our service traffic in different backends by leveraging Kong Mesh tags. This is especially useful when we want traces to never leave a world region, or a cloud, for example.

Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023