Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.1.x (latest)
  • 2.0.x
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Multi-zone deployment
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Understanding TargetRef policies
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshTrace (Beta)
    • MeshAccessLog (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • About
    • How it works
    • Components of a multi-zone deployment
  • Usage
    • Set up the global control plane
    • Set up the zone control planes
    • Verify control plane connectivity
    • Ensure mTLS is enabled on the multi-zone meshes
    • Cross-zone communication details
  • Failure modes
    • Global control plane offline
    • Zone control plane offline
    • Communication between Global and Zone control plane failing
    • Communication between 2 zones failing
  • Delete a zone
  • Disable a zone
Kong Mesh
2.0.x
  • Home
  • Kong Mesh
  • Deployments
  • Multi-zone deployment
You are browsing documentation for an outdated version. See the latest documentation here.

Multi-zone deployment

About

Kong Mesh supports running your service mesh in multiple zones. It is even possible to run with a mix of Kubernetes and Universal zones. Your mesh environment can include multiple isolated service meshes (multi-tenancy), and workloads running in different regions, on different clouds, or in different datacenters. A zone can be a Kubernetes cluster, a VPC, or any other deployment you need to include in the same distributed mesh environment. The only condition is that all the data planes running within the zone must be able to connect to the other data planes in this same zone.

Kuma service mesh multi zone deployment without zone egress

Or without the optional zone egress:

Kuma service mesh multi zone deployment with zone egress

How it works

In Kong Mesh, zones are abstracted away, meaning that your data plane proxies will find services wherever they run. This way you can make a service multi-zone by having data planes using the same kuma.io/service in different zones. This gives you automatic fail-over of services in case a specific zone fails.

Let’s look at how a service backend in zone-b is advertised to zone-a and a request from the local zone zone-a is routed to the remote service in zone-b.

Destination service zone

When the new service backend joins the mesh in zone-b, the zone-b zone control plane adds this service to the availableServices on the zone-b ZoneIngress resource. The kuma-dp proxy running as a zone ingress is configured with this list of services so that it can route incoming requests. This ZoneIngress resource is then also synchronized to the global control plane.

The global control-plane will propagate the zone ingress resources and all policies to all other zones over Kong Mesh Discovery Service (KDS), which is a protocol based on xDS.

Source service zone

The zone-b ZoneIngress resource is synchronized from the global control plane to the zone-a zone control plane. Requests to the availableServices from zone-a are load balanced between local instances and remote instances of this service. Requests send to zone-b are routed to the zone ingress proxy of zone-b.

For load-balancing, the zone ingress endpoints are weighted with the number of instances running behind them. So a zone with 2 instances will receive twice as much traffic than a zone with 1 instance. You can also favor local service instances with locality-aware load balancing.

In the presence of a zone egress, the traffic is routed through the local zone egress before being sent to the remote zone ingress.

When using transparent proxy (default in Kubernetes), Kong Mesh generates a VIP, a DNS entry with the format <kuma.io/service>.mesh, and will listen for traffic on port 80. The <kuma.io/service>.mesh:80 format is just a convention. VirtualOutboundss enable you to customize the listening port and how the DNS name for these services looks.

A zone ingress is not an API gateway. It is only used for cross-zone communication within a mesh. API gateways are supported in Kong Mesh gateway mode and can be deployed in addition to zone ingresses.

Components of a multi-zone deployment

A multi-zone deployment includes:

  • The global control plane:
    • Accept connections only from zone control planes.
    • Accept creation and changes to policies that will be applied to the data plane proxies.
    • Send policies down to zone control planes.
    • Send zone ingresses down to zone control plane.
    • Keep an inventory of all data plane proxies running in all zones (this is only done for observability but is not required for operations).
    • Reject connections from data plane proxies.
  • The zone control planes:
    • Accept connections from data plane proxies started within this zone.
    • Receive policy updates from the global control plane.
    • Send data plane proxies and zone ingress changes to the global control plane.
    • Compute and send configurations using XDS to the local data plane proxies.
    • Update list of services which exist in the zone in the zone ingress.
    • Reject policy changes that do not come from global.
  • The data plane proxies:
    • Connect to the local zone control plane.
    • Receive configurations using XDS from the local zone control plane.
    • Connect to other local data plane proxies.
    • Connect to zone ingresses for sending cross zone traffic.
    • Receive traffic from local data plane proxies and local zone ingresses.
  • The zone ingress:
    • Receive XDS configuration from the local zone control plane.
    • Proxy traffic from other zone data plane proxies to local data plane proxies.
  • (optional) The zone egress:
    • Receive XDS configuration from the local zone control plane.
    • Proxy traffic from local data plane proxies:
      • to zone ingress proxies from other zones;
      • to external services from local zone;

Usage

To set up a multi-zone deployment we will need to:

  • Set up the global control plane
  • Set up the zone control planes
  • Verify control plane connectivity
  • Ensure mTLS is enabled for the multi-zone meshes

Set up the global control plane

The global control plane must run on a dedicated cluster, and cannot be assigned to a zone.

Kubernetes
Helm
Universal

The global control plane on Kubernetes must reside on its own Kubernetes cluster, to keep its resources separate from the resources the zone control planes create during synchronization.

  1. Run:

    kumactl install control-plane --mode=global | kubectl apply -f -
    
  2. Find the external IP and port of the global-remote-sync service in the kong-mesh-system namespace:

    kubectl get services -n kong-mesh-system
    NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
    kong-mesh-system   global-remote-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
    kong-mesh-system   kong-mesh-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s
    

    In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

  1. Set the controlPlane.mode value to global in the chart (values.yaml), then install. On the command line, run:

    helm install kong-mesh --create-namespace --namespace kong-mesh-system --set controlPlane.mode=global kong-mesh/kong-mesh
    

    Or you can edit the chart and pass the file to the helm install kong-mesh command. To get the default values, run:

    helm show values kong-mesh/kong-mesh
    
  2. Find the external IP and port of the global-remote-sync service in the kong-mesh-system namespace:

    kubectl get services -n kong-mesh-system
    NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
    kong-mesh-system   global-remote-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
    kong-mesh-system   kong-mesh-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s
    

    By default, it’s exposed on port 5685. In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

  1. Set up the global control plane, and add the global environment variable:

    KUMA_MODE=global kuma-cp run
    

Set up the zone control planes

You need the following values to pass to each zone control plane setup:

  • zone – the zone name. An arbitrary string. This value registers the zone control plane with the global control plane.
  • kds-global-address – the external IP and port of the global control plane.
Kubernetes
Helm
Universal

Without zone egress:

  1. On each zone control plane, run:

    kumactl install control-plane \
    --mode=zone \
    --zone=<zone name> \
    --ingress-enabled \
    --kds-global-address grpcs://<global-kds-address>:5685 | kubectl apply -f -
    

    where zone is the same value for all zone control planes in the same zone.

With zone egress:

  1. On each zone control plane, run:

    kumactl install control-plane \
    --mode=zone \
    --zone=<zone-name> \
    --ingress-enabled \
    --egress-enabled \
    --kds-global-address grpcs://<global-kds-address>:5685 | kubectl apply -f -
    

    where zone is the same value for all zone control planes in the same zone.

Without zone egress:

  1. On each zone control plane, run:

    helm install kong-mesh \
    --create-namespace \
    --namespace kong-mesh-system \
    --set controlPlane.mode=zone \
    --set controlPlane.zone=<zone-name> \
    --set ingress.enabled=true \
    --set controlPlane.kdsGlobalAddress=grpcs://<global-kds-address>:5685 kong-mesh/kong-mesh
    

    where controlPlane.zone is the same value for all zone control planes in the same zone.

With zone egress:

  1. On each zone control plane, run:

    helm install kong-mesh \
    --create-namespace \
    --namespace kong-mesh-system \
    --set controlPlane.mode=zone \
    --set controlPlane.zone=<zone-name> \
    --set ingress.enabled=true \
    --set egress.enabled=true \
    --set controlPlane.kdsGlobalAddress=grpcs://<global-kds-address>:5685 kong-mesh/kong-mesh
    

    where controlPlane.zone is the same value for all zone control planes in the same zone.

  1. On each zone control plane, run:

    KUMA_MODE=zone \
    KUMA_MULTIZONE_ZONE_NAME=<zone-name> \
    KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS=grpcs://<global-kds-address>:5685 \
    ./kuma-cp run
    

    where KUMA_MULTIZONE_ZONE_NAME is the same value for all zone control planes in the same zone.

  2. Generate the zone proxy token:

    To register the zone ingress and zone egress with the zone control plane, we need to generate a token first

    kumactl generate zone-token --zone=<zone-name> --scope egress --scope ingress > /tmp/zone-token
    

    You can also generate the token with the REST API. Alternatively, you could generate separate tokens for ingress and egress.

  3. Create an ingress data plane proxy configuration to allow kuma-cp services to be exposed for cross-zone communication:

    echo "type: ZoneIngress
    name: ingress-01
    networking:
      address: 127.0.0.1 # address that is routable within the zone
      port: 10000
      advertisedAddress: 10.0.0.1 # an address which other zones can use to consume this zone-ingress
      advertisedPort: 10000 # a port which other zones can use to consume this zone-ingress" > ingress-dp.yaml
    
  4. Apply the ingress config, passing the IP address of the zone control plane to cp-address:

    kuma-dp run \
    --proxy-type=ingress \
    --cp-address=https://<kuma-cp-address>:5678 \
    --dataplane-token-file=/tmp/zone-token \
    --dataplane-file=ingress-dp.yaml
    

    If zone-ingress is running on a different machine than zone-cp you need to copy CA cert file from zone-cp (located in ~/.kuma/kuma-cp.crt) to somewhere accessible by zone-ingress (e.g. /tmp/kuma-cp.crt). Modify the above command and provide the certificate path in --ca-cert-file argument.

    kuma-dp run \
    --proxy-type=ingress \
    --cp-address=https://<kuma-cp-address>:5678 \
    --dataplane-token-file=/tmp/zone-token \
    --ca-cert-file=/tmp/kuma-cp.crt \
    --dataplane-file=ingress-dp.yaml
    
  5. Optional: if you want to deploy zone egress

    Create a ZoneEgress data plane proxy configuration to allow kuma-cp services to be configured to proxy traffic to other zones or external services through zone egress:

    echo "type: ZoneEgress
    name: zoneegress-01
    networking:
      address: 127.0.0.1 # address that is routable within the zone
      port: 10002" > zoneegress-dataplane.yaml
    
  6. Apply the egress config, passing the IP address of the zone control plane to cp-address:

     ```sh
     kuma-dp run \
     --proxy-type=egress \
     --cp-address=https://<kuma-cp-address>:5678 \
     --dataplane-token-file=/tmp/zone-token \
     --dataplane-file=zoneegress-dataplane.yaml
     ```
    

Verify control plane connectivity

You can run kumactl get zones, or check the list of zones in the web UI for the global control plane, to verify zone control plane connections.

When a zone control plane connects to the global control plane, the Zone resource is created automatically in the global control plane.

The Zone Ingress tab of the web UI also lists zone control planes that you deployed with zone ingress.

Ensure mTLS is enabled on the multi-zone meshes

MTLS is mandatory to enable cross-zone service communication. mTLS can be configured in your mesh configuration as indicated in the mTLS section. This is required because Kong Mesh uses the Server Name Indication field, part of the TLS protocol, as a way to pass routing information cross zones.

Cross-zone communication details

For this example we will assume we have a service running in a Kubernetes zone exposing a kuma.io/service with value echo-server_echo-example_svc_1010. The following examples are running in the remote zone trying to access the previously mentioned service.

Kubernetes
Universal

To view the list of service names available, run:

kubectl get serviceinsight all-services-default -oyaml
apiVersion: kuma.io/v1alpha1
kind: ServiceInsight
mesh: default
metadata:
  name: all-services-default
spec:
  services:
    echo-server_echo-example_svc_1010:
      dataplanes:
        online: 1
        total: 1
      issuedBackends:
        ca-1: 1
      status: online

The following are some examples of different ways to address echo-server in the echo-example Namespace in a multi-zone mesh.

To send a request in the same zone, you can rely on Kubernetes DNS and use the usual Kubernetes hostnames and ports:

curl http://echo-server:1010

Requests are distributed round robin between zones. You can use locality-aware load balancing to keep requests in the same zone.

To send a request to any zone, you can use the generated kuma.io/service and Kong Mesh DNS:

curl http://echo-server_echo-example_svc_1010.mesh:80

Kong Mesh DNS also supports RFC 1123 compatible names, where underscores are replaced with dots:

curl http://echo-server.echo-example.svc.1010.mesh:80
kumactl inspect services
SERVICE                                  STATUS               DATAPLANES
echo-service_echo-example_svc_1010       Online               1/1

To consume the service in a Universal deployment without transparent proxy add the following outbound to your dataplane configuration:

outbound:
  - port: 20012
    tags:
      kuma.io/service: echo-server_echo-example_svc_1010

From the data plane running you will now be able to reach the service using localhost:20012.

Alternatively, if you configure transparent proxy you can just call echo-server_echo-example_svc_1010.mesh without defining an outbound section.

For security reasons it’s not possible to customize the kuma.io/service in Kubernetes.

If you want to have the same service running on both Universal and Kubernetes make sure to align the Universal’s data plane inbound to have the same kuma.io/service as the one in Kubernetes or leverage TrafficRoute.

Failure modes

Global control plane offline

  • Policy updates will be impossible
  • Change in service list between zones will not propagate:
    • New services will not be discoverable in other zones.
    • Services removed from a zone will still appear available in other zones.
  • You won’t be able to disable or delete a zone.

Note that both local and cross-zone application traffic is not impacted by this failure case. Data plane proxy changes will be propagated within their zones.

Zone control plane offline

  • New data plane proxies won’t be able to join the mesh.
  • Data plane proxy configuration will not be updated.
  • Communication between data plane proxies will still work.
  • Cross zone communication will still work.
  • Other zones are unaffected.

You can think of this failure case as “Freezing” the zone mesh configuration. Communication will still work but changes will not be reflected on existing data plane proxies.

Communication between Global and Zone control plane failing

This can happen with misconfiguration or network connectivity issues between control planes.

  • Operations inside the zone will happen correctly (data plane proxies can join, leave and all configuration will be updated and sent correctly).
  • Policy changes will not be propagated to the zone control plane.
  • ZoneIngress, ZoneEgress and Dataplane changes will not be propagated to the global control plane:
    • The global inventory view of the data plane proxies will be outdated (this only impacts observability).
    • Other zones will not see new services registered inside this zone.
    • Other zones will not see services no longer running inside this zone.
    • Other zones will not see changes in number of instances of each service running in the local zone.
  • Global control plane will not send changes from other zone ingress to the zone:
    • Local data plane proxies will not see new services registered in other zones.
    • Local data plane proxies will not see services no longer running in other zones.
    • Local data plane proxies will not see changes in number of instances of each service running in other zones.
  • Global control plane will not send changes from other zone ingress to the zone.

Note that both local and cross-zone application traffic is not impacted by this failure case.

Communication between 2 zones failing

This can happen if there are network connectivity issues:

  • Between control plane and zone ingress from other zone.
  • Between control plane and zone egress (when present).
  • Between zone egress (when present) and zone ingress from other zone.
  • All Zone egress instances of a zone (when present) are down.
  • All zone ingress instances of a zone are down.

When it happens:

  • Communication and operation within each zone is unaffected.
  • Communication across each zone will fail.

With the right resiliency setup (Retries, Probes, Locality Aware LoadBalancing, Circuit Breakers) the failing zone can be quickly severed and traffic re-routed to another zone.

Delete a zone

To delete a Zone we must first shut down the corresponding Kong Mesh zone control plane instances. As long as the Zone CP is running this will not be possible, and Kong Mesh returns a validation error like:

zone: unable to delete Zone, Zone CP is still connected, please shut it down first

When the Zone CP is fully disconnected and shut down, then the Zone can be deleted. All corresponding resources (like Dataplane and DataplaneInsight) will be deleted automatically as well.

Kubernetes
Universal
kubectl delete zone zone-1
kumactl delete zone zone-1

Disable a zone

Change the enabled property value to false in the global control plane:

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: Zone
metadata:
  name: zone-1
spec:
  enabled: false
type: Zone
name: zone-1
spec:
  enabled: false

With this setting, the global control plane will stop exchanging configuration with this zone. As a result, the zone’s ingress from zone-1 will be deleted from other zone and traffic won’t be routed to it anymore. The zone will show as Offline in the GUI and CLI.

Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023