Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.1.x (latest)
  • 2.0.x
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Deploy a standalone control plane
    • Multi-zone deployment
    • Deploy a multi-zone global control plane
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Non-mesh traffic
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Understanding TargetRef policies
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshAccessLog (Beta)
    • MeshCircuitBreaker (Beta)
    • MeshFaultInjection (Beta)
    • MeshHealthCheck (Beta)
    • MeshHTTPRoute (Beta)
    • MeshProxyPatch (Beta)
    • MeshRateLimit (Beta)
    • MeshRetry (Beta)
    • MeshTimeout (Beta)
    • MeshTrace (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • MeshOPA (beta)
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference
    • Open source License
    • Contribute to Mesh

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • Expose metrics from data plane proxies
  • Expose metrics from applications
  • Override Prometheus settings per data plane proxy
  • Filter Envoy metrics
  • Secure data plane proxy metrics
Kong Mesh
2.1.x (latest)
  • Home
  • Kong Mesh
  • Policies
  • Traffic Metrics

Traffic Metrics

Kong Mesh facilitates consistent traffic metrics across all data plane proxies in your mesh.

You can add metrics to a mesh configuration, or to an individual data plane proxy configuration. For example, you might need metrics for individual data plane proxies to override the default metrics port if it’s already in use on the specified machine.

Kong Mesh provides full integration with Prometheus:

  • Each proxy can expose its metrics in Prometheus format.
  • Because metrics are part of the mesh configuration, Kuma exposes an API called the monitoring assignment service (MADS) which exposes every proxy in the mesh.

To collect metrics from Kong Mesh, you need to expose metrics from proxies and applications.

In the rest of this page we assume you have already configured your observability tools to work with Kong Mesh. If you haven’t already read the observability docs.

Expose metrics from data plane proxies

To expose metrics from every proxy in the mesh, configure the Mesh resource:

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus

which is a shortcut for:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: false
        port: 5670
        path: /metrics
        tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
          kuma.io/service: dataplane-metrics
type: Mesh
name: default
metrics:
  enabledBackend: prometheus-1
  backends:
  - name: prometheus-1
    type: prometheus
    conf:
      skipMTLS: true # by default mTLS metrics are also protected by mTLS. Scraping metrics with mTLS without transparent proxy is not supported at the moment.

which is a shortcut for:

type: Mesh
name: default
metrics:
  enabledBackend: prometheus-1
  backends:
  - name: prometheus-1
    type: prometheus
    conf:
      skipMTLS: true
      port: 5670
      path: /metrics
      tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
        kuma.io/service: dataplane-metrics

This tells Kong Mesh to configure every proxy in the default mesh to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

The metrics endpoint is forwarded to the standard Envoy Prometheus metrics endpoint and supports the same query parameters. You can pass the filter query parameter to limit the results to metrics whose names match a given regular expression. By default all available metrics are returned.

Expose metrics from applications

In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies. Kong Mesh allows scraping Prometheus metrics from the applications endpoint running in the same Pod or VM. Later those metrics are aggregated and exposed at the same port/path as data plane proxy metrics. It is possible to configure it at the Mesh level, for all the applications in the Mesh, or just for specific applications.

Here are reasons where you’d want to use this feature:

  • Application metrics are labelled with your mesh parameters (tags, mesh, data plane name…), this means that in mixed Universal and Kubernetes mode metrics are reported with the same types of labels.
  • Both application and sidecar metrics are scraped at the same time. This makes sure they are coherent (with 2 different scrapers they can end up scraping at different intervals and make metrics harder to correlate).
  • If you disable passthrough and your mesh uses mTLS but Prometheus is outside the mesh (skipMTLS: true) this will be the only way to retrieve these metrics as the application is completely hidden behind the sidecar.

Any configuration change requires redeployment of the data plane.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: false
        port: 5670
        path: /metrics
        tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS 
          kuma.io/service: dataplane-metrics
        aggregate:
          - name: my-service # name of the metric, required to later disable/override with pod annotations 
            path: "/metrics/prometheus"
            port: 8888
          - name: other-sidecar
            # default path is going to be used, default: /metrics
            port: 8000
type: Mesh
name: default
metrics:
  enabledBackend: prometheus-1
  backends:
  - name: prometheus-1
    type: prometheus
    conf:
      port: 5670
      path: /metrics
      skipMTLS: true # by default mTLS metrics are also protected by mTLS. Scraping metrics with mTLS without transparent proxy is not supported at the moment.
      aggregate:
      - name: my-service # name of the metric, required to later disable/override in the Dataplane resource
        path: "/metrics/prometheus"
        port: 8888
      - name: other-sidecar
        # default path is going to be used, default: /metrics
        port: 8000

This configuration will cause every application in the mesh to be scrapped for metrics by the data plane proxy. If you need to expose metrics only for the specific application it is possible through annotation for Kubernetes or Dataplane resource for Universal deployment.

Kubernetes
Universal

Kubernetes allows to configure it through annotations. In case to configure you can use prometheus.metrics.kuma.io/aggregate-<name>-(path/port/enabled), where name is used to match the Mesh configuration and override or disable it.

apiVersion: apps/v1
kind: Deployment
metadata:
 namespace: kuma-example
 name: kuma-tcp-echo
spec:
 ...
 template:
   metadata:
     ...
     annotations:
       prometheus.metrics.kuma.io/aggregate-my-service-enabled: "false"  # causes that configuration from Mesh to be disabled and result in this endpoint's metrics to not be exposed
       prometheus.metrics.kuma.io/aggregate-other-sidecar-port: "1234" # override port from Mesh
       prometheus.metrics.kuma.io/aggregate-application-port: "80"
       prometheus.metrics.kuma.io/aggregate-application-path: "/stats"
   spec:
     containers:
     ...
type: Dataplane
mesh: default
name: example
metrics:
  type: prometheus
  conf:
    path: /metrics/overridden
    aggregate:
    - name: my-service # causes that configuration from Mesh to be disabled and result in this endpoint's metrics to not be exposed
      enabled: false
    - name: other-sidecar
      port: 1234 # override port from Mesh
      address: "127.0.0.1"
    - name: application
      path: "/stats"
      port: 80`

By default, sidecar scrapes metrics on dataplane.networking.address. If you set dataplane.networking.inbound[].address.serviceAddress, and you want to scrape metrics from the application, you also need to set dataplane.metrics.conf.aggregate[].address.

Override Prometheus settings per data plane proxy

Kubernetes
Universal

To override mesh-wide defaults for a particular Pod, use the following annotations:

  • prometheus.metrics.kuma.io/port - to override mesh-wide default port
  • prometheus.metrics.kuma.io/path - to override mesh-wide default path

For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kuma-example
  name: kuma-tcp-echo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        prometheus.metrics.kuma.io/port: "1234"               # override Mesh-wide default port
        prometheus.metrics.kuma.io/path: "/non-standard-path" # override Mesh-wide default path
    spec:
      containers:
      ...

Proxies for this Pod expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

To override mesh-wide defaults on a particular machine, configure the Dataplane resource:

type: Dataplane
mesh: default
name: example
metrics:
  type: prometheus
  conf:
    skipMTLS: false
    port: 1234
    path: /non-standard-path

This proxy exposes an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

Filter Envoy metrics

In case you don’t want to retrieve all Envoy’s metrics, it’s possible to filter them. Configuration is dynamic and doesn’t require a restart of a sidecar. You are able to specify regex which causes that metric’s endpoint returns only matching metrics. Also, you can set flag usedOnly that returns only metrics updated by Envoy.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: false
        port: 5670
        path: /metrics
        envoy:
          filterRegex: http2_act.*
          usedOnly: true
type: Mesh
name: default
metrics:
  enabledBackend: prometheus-1
  backends:
  - name: prometheus-1
    type: prometheus
    conf:
      port: 5670
      path: /metrics
      envoy:
        filterRegex: http2_act.*
        usedOnly: true

Secure data plane proxy metrics

Kong Mesh lets you expose proxy metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the mesh for this feature to work, which is the default deployment mode on Kubernetes when using kumactl install observability.

Kubernetes
Universal

Make sure that mTLS is enabled in the mesh.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        port: 5670
        path: /metrics
        skipMTLS: false
        tags: # tags that can be referred in a TrafficPermission resource 
          kuma.io/service: dataplane-metrics

If you have strict traffic permissions you will want to allow the traffic from Grafana to Prometheus and from Prometheus to data plane proxy metrics:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
mesh: default
metadata:
  name: metrics-permissions
spec:
  sources:
    - match:
       kuma.io/service: prometheus-server_mesh-observability_svc_80
  destinations:
    - match:
       kuma.io/service: dataplane-metrics
---
apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
mesh: default
metadata:
  name: grafana-to-prometheus
spec:
   sources:
   - match:
      kuma.io/service: "grafana_mesh-observability_svc_80"
   destinations:
   - match:
      kuma.io/service: "prometheus-server_mesh-observability_svc_80"

Make sure that mTLS is enabled in the mesh.

type: Mesh
name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        port: 5670
        path: /metrics
        skipMTLS: false
        tags: # tags that can be referred in a TrafficPermission resource 
          kuma.io/service: dataplane-metrics

If you have strict traffic permissions you will want to allow the traffic from Grafana to Prometheus and from Prometheus to data plane proxy metrics:

type: TrafficPermission
mesh: default
name: metrics-permissions
spec:
  sources:
    - match:
       kuma.io/service: prometheus-server
  destinations:
    - match:
       kuma.io/service: dataplane-metrics
---
type: TrafficPermission
mesh: default
name: grafana-to-prometheus
spec:
   sources:
   - match:
      kuma.io/service: "grafana"
   destinations:
   - match:
      kuma.io/service: "prometheus-server"
Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023