Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.1.x (latest)
  • 2.0.x
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Multi-zone deployment
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Non-mesh traffic
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Understanding TargetRef policies
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshAccessLog (Beta)
    • MeshCircuitBreaker (Beta)
    • MeshFaultInjection (Beta)
    • MeshHealthCheck (Beta)
    • MeshHTTPRoute (Beta)
    • MeshProxyPatch (Beta)
    • MeshRateLimit (Beta)
    • MeshRetry (Beta)
    • MeshTimeout (Beta)
    • MeshTrace (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • MeshOPA (beta)
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference
    • Open source License
    • Contribute to Mesh

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • Add a logging backend
  • Add a TrafficLog resource
    • Matching
  • Logging external services
  • Builtin Gateway support
  • Access Log Format
    • Access Logs for TCP and HTTP traffic
Kong Mesh
2.1.x (latest)
  • Home
  • Kong Mesh
  • Policies
  • Traffic Log

Traffic Log

With the Traffic Log policy you can easily set up access logs on every data plane in a mesh.

This policy only records outbound traffic. It doesn’t record inbound traffic.

To configure access logs in Kong Mesh you need to:

  • 1. Add a logging backend
  • 2. Add a TrafficLog resource

In the rest of this page we assume you have already configured your observability tools to work with Kong Mesh. If you haven’t already read the observability docs.

Add a logging backend

A logging backend is essentially a sink for access logs.

Currently, it can be either a file or a TCP log collector, such as Logstash, Splunk or other.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  logging:
    # TrafficLog policies may leave the `backend` field undefined.
    # In that case the logs will be forwarded into the `defaultBackend` of that Mesh.
    defaultBackend: file
    # List of logging backends that can be referred to by name
    # from TrafficLog policies of that Mesh.
    backends:
      - name: logstash
        # Use `format` field to adjust the access log format to your use case.
        format: '{"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%", "source_address": "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%", "destination_address": "%UPSTREAM_HOST%", "duration_millis": "%DURATION%", "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%"}'
        type: tcp
        # Use `config` field to co configure a TCP logging backend.
        conf:
          # Address of a log collector (like logstash, splunk or other).
          address: 127.0.0.1:5000
      - name: file
        type: file
        # Use `file` field to configure a file-based logging backend.
        conf:
          path: /tmp/access.log
        # When `format` field is omitted, the default access log format will be used.
type: Mesh
name: default
logging:
  # TrafficLog policies may leave the `backend` field undefined.
  # In that case the logs will be forwarded into the `defaultBackend` of that Mesh.
  defaultBackend: file
  # List of logging backends that can be referred to by name
  # from TrafficLog policies of that Mesh.
  backends:
    - name: logstash
      # Use `format` field to adjust the access log format to your use case.
      format: '{"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%", "source_address": "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%", "destination_address": "%UPSTREAM_HOST%", "duration_millis": "%DURATION%", "bytes_received": "%BYTES_RECEIVED%", "bytes_sent": "%BYTES_SENT%"}'
      type: tcp
      conf: # Use `config` field to configure a TCP logging backend.
        # Address of a log collector.
        address: 127.0.0.1:5000
    - name: file
      type: file
      # Use `config` field to configure a file-based logging backend.
      conf:
        path: /tmp/access.log
      # When `format` field is omitted, the default access log format will be used.

Add a TrafficLog resource

You need to create a TrafficLog policy to select a subset of traffic and write its access logs into one of the backends configured for that mesh.

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: TrafficLog
metadata:
  name: all-traffic
mesh: default
spec:
  # This TrafficLog policy applies all traffic in that Mesh.
  sources:
    - match:
        kuma.io/service: "*"
  destinations:
    - match:
        kuma.io/service: "*"
apiVersion: kuma.io/v1alpha1
kind: TrafficLog
metadata:
  name: backend-to-database-traffic
mesh: default
spec:
  # This TrafficLog policy applies only to traffic from service `backend` to service `database`.
  sources:
    - match:
        kuma.io/service: backend_kuma-example_svc_8080
  destinations:
    - match:
        kuma.io/service: database_kuma-example_svc_5432
  conf:
    # Forward the logs into the logging backend named `logstash`.
    backend: logstash
type: TrafficLog
name: all-traffic
mesh: default
# This TrafficLog policy applies to all traffic in the Mesh.
sources:
  - match:
      kuma.io/service: "*"
destinations:
  - match:
      kuma.io/service: "*"
type: TrafficLog
name: backend-to-database-traffic
mesh: default
# this TrafficLog policy applies only to traffic from service `backend` to service `database`.
sources:
  - match:
      kuma.io/service: backend
destinations:
  - match:
      kuma.io/service: database
conf:
  # Forward the logs into the logging backend named `logstash`.
  backend: logstash

When backend field is omitted, the logs will be forwarded into the defaultBackend of that Mesh.

Matching

TrafficLog is an Outbound Connection Policy. For this reason the only supported value for destinations.match is kuma.io/service.

Logging external services

When running Kong Mesh on Kubernetes you can also log the traffic to external services. To do it, the matched destination section has to have wildcard * value. In such case %KUMA_DESTINATION_SERVICE% will have value external and %UPSTREAM_HOST% will have an IP of the service.

Builtin Gateway support

Traffic Log is a Kong Mesh outbound connection policy, so Kong Mesh chooses a Traffic Log policy by matching the service tag of the data plane’s outbounds. Since a builtin gateway data plane does not have outbounds, Kong Mesh always uses the builtin service name pass_through to match the Traffic Log policy for Gateways.

Access Log Format

Kong Mesh gives you full control over the format of the access logs.

The shape of a single log record is defined by a template string that uses command operators to extract and format data about a TCP connection or an HTTP request.

E.g.,

%START_TIME% %KUMA_SOURCE_SERVICE% => %KUMA_DESTINATION_SERVICE% %DURATION%

where %START_TIME% and %KUMA_SOURCE_SERVICE% are examples of available command operators.

All command operators defined by Envoy are supported, along with additional command operators defined by Kong Mesh:

Command Operator Description
%KUMA_MESH% name of the mesh in which traffic is flowing
%KUMA_SOURCE_SERVICE% name of a service that is the source of traffic
%KUMA_DESTINATION_SERVICE% name of a service that is the destination of traffic
%KUMA_SOURCE_ADDRESS_WITHOUT_PORT% address of a Dataplane that is the source of traffic
%KUMA_TRAFFIC_DIRECTION% direction of the traffic, INBOUND, OUTBOUND or UNSPECIFIED

Access Logs for TCP and HTTP traffic

All access log command operators are valid to use with both TCP and HTTP traffic.

If a command operator is specific to HTTP traffic, such as %REQ(X?Y):Z% or %RESP(X?Y):Z%, it will be replaced by a symbol “-” in case of TCP traffic.

Internally, Kong Mesh determines traffic protocol based on the value of kuma.io/protocol tag on the inbound interface of a destination Dataplane.

The default format string for TCP traffic is:

[%START_TIME%] %RESPONSE_FLAGS% %KUMA_MESH% %KUMA_SOURCE_ADDRESS_WITHOUT_PORT%(%KUMA_SOURCE_SERVICE%)->%UPSTREAM_HOST%(%KUMA_DESTINATION_SERVICE%) took %DURATION%ms, sent %BYTES_SENT% bytes, received: %BYTES_RECEIVED% bytes

The default format string for HTTP traffic is:

[%START_TIME%] %KUMA_MESH% "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%KUMA_SOURCE_SERVICE%" "%KUMA_DESTINATION_SERVICE%" "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%" "%UPSTREAM_HOST%"

To provide different format for TCP and HTTP logging you can define two separate logging backends with the same address and different format. Then define two TrafficLog entity, one for TCP and one for HTTP with matching kuma.io/protocol selector.

JSON format

If you need an access log with entries in JSON format, you have to provide a template string that is a valid JSON object, e.g.

{
  "start_time":          "%START_TIME%",
  "source":              "%KUMA_SOURCE_SERVICE%",
  "destination":         "%KUMA_DESTINATION_SERVICE%",
  "source_address":      "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%",
  "destination_address": "%UPSTREAM_HOST%",
  "duration_millis":     "%DURATION%",
  "bytes_received":      "%BYTES_RECEIVED%",
  "bytes_sent":          "%BYTES_SENT%"
}
Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023