Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Early Access
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.0.x (latest)
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Multi-zone deployment
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Policy matching
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshTrace (Beta)
    • MeshAccessLog (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • Demo setup
  • Control plane metrics
  • Configuring Prometheus
    • Using an already existing prometheus setup
  • Configuring Grafana
    • Grafana extensions
  • Configuring Datadog
    • Metrics
    • Tracing
    • Logs
  • Observability in multi-zone
    • Prometheus
    • Jaeger, Loki, Datadog and others
Kong Mesh
2.0.x (latest)
  • Home
  • Kong Mesh
  • Explore
  • Observability

Observability

This page will describe how to configure different observability tools to work with Kong Mesh.

Demo setup

kumactl ships with a builtin observability stack which consists of:

  • prometheus for metrics
  • jaeger for ingesting and storing traces
  • loki for ingesting and storing logs
  • grafana for querying and displaying metrics, traces and logs

First, remember to configure Kong Mesh appropriately for the tools in the observability stack:

  • Traffic metrics for telemetry
  • TrafficTrace for tracing
  • TrafficLog for logging

On Kubernetes, the stack can be installed with:

kumactl install observability | kubectl apply -f -

This will create a namespace mesh-observability with prometheus, jaeger, loki and grafana installed and setup to work with Kong Mesh.

This setup is meant to be used for trying out Kong Mesh. It is in no way fit for use in production. For production setups we recommend referring to each project’s website or to use a hosted solution like Grafana cloud or Datadog.

Control plane metrics

Control plane metrics are exposed on port :5680 and available under the standard path /metrics.

Configuring Prometheus

The Kong Mesh community has contributed a builtin service discovery to Prometheus, it is documented in the Prometheus docs. This service discovery will connect to the control plane and retrieve all data planes with enabled metrics which Prometheus will scrape and retrieve metrics according to your traffic metrics setup.

There are 2 ways you can run prometheus:

  1. Inside the mesh (default for kumactl install observability). In this case you can use mTLS to retrieve the metrics. This provides high security but will require one prometheus per mesh and might not be accessible if your mesh becomes unavailable. It will also require one Prometheus deployment per Kong Mesh mesh.
  2. Outside the mesh. In this case you’ll need to specify skipMTLS: true in the traffic metrics configuration. This is less secured but will ensure Prometheus is as available as possible. It is also easier to add to an existing setup with services in and outside the mesh.

In production, we recommend the second option as it provides better visibility when things go wrong, and it’s usually acceptable for metrics to be less secure.

Using an already existing prometheus setup

In Prometheus version 2.29 and later, you can add Kong Mesh metrics to your prometheus.yml:

scrape_configs:
    - job_name: 'kuma-dataplanes'
      scrape_interval: "5s"
      relabel_configs:
      - source_labels:
        - __meta_kuma_mesh
        regex: "(.*)"
        target_label: mesh
      - source_labels:
        - __meta_kuma_dataplane
        regex: "(.*)"
        target_label: dataplane
      - source_labels:
        - __meta_kuma_service
        regex: "(.*)"
        target_label: service
      - action: labelmap
        regex: __meta_kuma_label_(.+)
      kuma_sd_configs:
      - server: "http://kong-mesh-control-plane.kong-mesh-system.svc:5676" # replace with the url of your control plane

For more information, see the Prometheus documentation.

If you have traffic metrics enabled for your mesh, check the Targets page in the Prometheus dashboard. You should see a list of data plane proxies from your mesh. For example:

A screenshot of Targets page on Prometheus UI

Configuring Grafana

Visualizing traces

To visualise your traces you need to have Grafana up and running.

kumactl install observability sets this up out of the box.

With Grafana installed you can configure a new datasource with url:http://jaeger-query.mesh-observability/ (or whatever url jaeger can be queried at). Grafana will then be able to retrieve the traces from Jaeger.

Jaeger Grafana configuration

You can then add a TrafficTrace policy to your mesh to start emitting traces. At this point you can visualize your traces in Grafana by choosing the jaeger datasource in the explore section.

Visualizing logs

To visualise your containers’ logs and your access logs you need to have a Grafana up and running.

kumactl install observability sets this up out of the box.

Loki Grafana configuration

You can then add a TrafficLog policy to your mesh to start emitting access logs. Loki will pick up logs that are sent to stdout. To send logs to stdout you can configure the logging backend as shown below:

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  logging:
    defaultBackend: stdout
    backends:
      - name: stdout
        type: file
        conf:
          path: /dev/stdout
type: Mesh
name: default
logging:
  defaultBackend: stdout
  backends:
    - name: stdout
      type: file
      conf:
        path: /dev/stdout

At this point you can visualize your containers’ logs and your access logs in Grafana by choosing the loki datasource in the explore section.

For example, running: {container="kuma-sidecar"} |= "GET" will show all GET requests on your cluster. To learn more about the search syntax check the Loki docs.

Nice to have

Having your Logs and Traces in the same visualisation tool can come really handy. By adding the traceId in your app logs you can visualize your logs and the related Jaeger traces. To learn more about it go read this article.

Grafana extensions

The Kong Mesh community has built a datasource and a set of dashboards to provide great interactions between Kong Mesh and Grafana.

Datasource and service map

The Grafana Datasource is a datasource specifically built to relate information from the control plane with Prometheus metrics.

Current features include:

  • Display the graph of your services with the MeshGraph using Grafana nodeGraph panel.
  • List meshes.
  • List zones.
  • List services.

To use the plugin you’ll need to add the binary to your Grafana instance by following the installation instructions.

To make things simpler the datasource is installed and configured when using kumactl install observability.

Dashboards

Kong Mesh ships with default dashboards that are available to import from the Grafana Labs repository.

Kong Mesh Dataplane

This dashboard lets you investigate the status of a single dataplane in the mesh. In order to see those metrics, you need to create Traffic Metrics policy first.

Kuma Dataplane dashboard Kuma Dataplane dashboard Kuma Dataplane dashboard Kuma Dataplane dashboard
Kong Mesh Mesh

This dashboard lets you investigate the aggregated statistics of a single mesh. It provides a topology view of your service traffic dependencies (Service Map) and includes information such as number of requests and error rates.

Kuma Mesh dashboard
Kong Mesh Service to Service

This dashboard lets you investigate aggregated statistics from dataplanes of specified source services to dataplanes of specified destination service.

Kuma Service to Service dashboard Kuma Service to Service HTTP
Kong Mesh CP

This dashboard lets you investigate control plane statistics.

Kuma CP dashboard Kuma CP dashboard Kuma CP dashboard
Kong Mesh Service

This dashboard lets you investigate aggregated statistics for each service.

Kuma Service dashboard
Kong Mesh MeshGateway

This dashboard lets you investigate aggregated statistics for each builtin gateway.

Kuma Gateway dashboard

Configuring Datadog

The recommended way to use Datadog is with its agent.

Kubernetes
Universal

The Datadog agent docs have in-depth installation methods.

Checkout the Datadog agent docs.

Metrics

Kong Mesh exposes metrics with traffic metrics in Prometheus format.

You can add annotations to your pods to enable the Datadog agent to scrape metrics.

Kubernetes
Universal

Please refer to the dedicated documentation.

You need to setup your agent with an openmetrics.d/conf.yaml.

Tracing

Checkout the

  1. Set up the Datadog agent.
  2. Set up APM.
Kubernetes
Universal

Configure the Datadog agent for APM.

If Datadog is not running on each node you can expose the APM agent port to Kong Mesh via Kubernetes service.

apiVersion: v1
kind: Service
metadata:
  name: trace-svc
spec:
  selector:
    app.kubernetes.io/name: datadog-agent-deployment
  ports:
    - protocol: TCP
      port: 8126
      targetPort: 8126

Apply the configuration with kubectl apply -f [..].

Check if the label of the datadog pod installed has not changed (app.kubernetes.io/name: datadog-agent-deployment), if it did adjust accordingly.

Checkout the Datadog agent docs

Once the agent is configured to ingest traces you’ll need to configure a TrafficTrace.

Logs

The best way to have Kong Mesh and Datadog work together is with TCP ingest.

Once your agent is configured with TCP ingest you can configure a TrafficLog for data plane proxies to send logs.

Observability in multi-zone

Kong Mesh is multi-zone at heart. We explain here how to architect your telemetry stack to accommodate multi-zone.

Prometheus

When Kong Mesh is used in multi-zone the recommended approach is to use 1 Prometheus instance in each zone and to send the metrics of each zone to a global Prometheus instance.

Prometheus offers different ways to do this:

  • Federation the global Prometheus will scrape each zone Prometheuses.
  • Remote Write each zone Prometheuses will directly write their metrics to the global, this is meant to be more efficient the API the federation.
  • Remote Read like remote write but the other way around.

Jaeger, Loki, Datadog and others

Most telemetry components don’t have a hierarchical setup like Prometheus. If you want to have a central view of everything you can set up the system in global and have each zone send their data to it. Because zone is present in data plane tags you shouldn’t be worried about metrics, logs and traces overlapping between zones.

Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023