Deploy a multi-zone global control plane

The global control plane on Kubernetes must reside on its own Kubernetes cluster, to keep its resources separate from the resources the zone control planes create during synchronization.

Run:

kumactl install control-plane \
  --set "kuma.controlPlane.mode=global" \
  | kubectl apply -f -

helm install --create-namespace --namespace kong-mesh-system \
  --set "kuma.controlPlane.mode=global" \
  kong-mesh kong-mesh/kong-mesh

Find the external IP and port of the global-zone-sync service in the kong-mesh-system namespace:

kubectl get services -n kong-mesh-system

NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
kong-mesh-system   global-zone-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
kong-mesh-system   kong-mesh-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s

By default, it’s exposed on port 5685. In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

Running global control plane in “Universal on Kubernetes” mode means using PostgreSQL as storage instead of Kubernetes. It means that failover / HA / reliability characteristics will change. Please read Kubernetes and PostgreSQL docs for more details.

Before using Kong Mesh with helm, please follow these steps to configure your local helm repo and learn the reference helm configuration values.yaml.

1. Define Kubernetes secrets with database sensitive information

   apiVersion: v1
   kind: Secret
   metadata:
     name: your-secret-name
   type: Opaque
   data:
     POSTGRES_DB: ...
     POSTGRES_HOST_RW: ...
     POSTGRES_USER: ...
     POSTGRES_PASSWORD: ...
   

2. Create a values.yaml file with: kuma.controlPlane.environment=universal and kuma.controlPlane.mode=global in the chart (values.yaml).

3. Set kuma.controlPlane.secrets with database sensitive information

   # ...
       secrets:
         postgresDb:
           Secret: your-secret-name
           Key: POSTGRES_DB
           Env: KUMA_STORE_POSTGRES_DB_NAME
         postgresHost:
           Secret: your-secret-name
           Key: POSTGRES_HOST_RW
           Env: KUMA_STORE_POSTGRES_HOST
         postgrestUser:
           Secret: your-secret-name
           Key: POSTGRES_USER
           Env: KUMA_STORE_POSTGRES_USER
         postgresPassword:
           Secret: your-secret-name
           Key: POSTGRES_PASSWORD
           Env: KUMA_STORE_POSTGRES_PASSWORD
   

4. Optionally set kuma.postgres with TLS settings

   # ...
     # Postgres' settings for universal control plane on k8s
     postgres:
       # -- Postgres port, password should be provided as a secret reference in "controlPlane.secrets"
       # with the Env value "KUMA_STORE_POSTGRES_PASSWORD".
       # Example:
       # controlPlane:
       #   secrets:
       #     - Secret: postgres-postgresql
       #       Key: postgresql-password
       #       Env: KUMA_STORE_POSTGRES_PASSWORD
       port: "5432"
       # TLS settings
       tls:
         # -- Mode of TLS connection. Available values are: "disable", "verifyNone", "verifyCa", "verifyFull"
         mode: disable # ENV: KUMA_STORE_POSTGRES_TLS_MODE
         # -- Whether to disable SNI the postgres `sslsni` option.
         disableSSLSNI: false # ENV: KUMA_STORE_POSTGRES_TLS_DISABLE_SSLSNI
         # -- Secret name that contains the ca.crt
         caSecretName:
         # -- Secret name that contains the client tls.crt, tls.key
         secretName:
   

5. Run helm install

    helm install kong-mesh \
      --create-namespace \
      --skip-crds \
      --namespace kong-mesh-system \
      --values values.yaml \
      kong-mesh/kong-mesh
   

6. Find the external IP and port of the global-zone-sync service in the kong-mesh-system namespace:

    kubectl get services -n kong-mesh-system
   

    NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
    kong-mesh-system   global-zone-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
    kong-mesh-system   kong-mesh-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s
   

   In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

When running the global control plane in Universal mode, a database must be used to persist state for production deployments. Ensure that migrations have been run against the database prior to running the global control plane.

1. Set up the global control plane, and add the global environment variable:

   KUMA_MODE=global \
   KUMA_ENVIRONMENT=universal \
   KUMA_STORE_TYPE=postgres \
   KUMA_STORE_POSTGRES_HOST=<postgres-host> \
   KUMA_STORE_POSTGRES_PORT=<postgres-port> \
   KUMA_STORE_POSTGRES_USER=<postgres-user> \
   KUMA_STORE_POSTGRES_PASSWORD=<postgres-password> \
   KUMA_STORE_POSTGRES_DB_NAME=<postgres-db-name> \
   kuma-cp run
   

For every individual zone create an own cluster and for each run:

kumactl install control-plane \
  --set "kuma.controlPlane.mode=zone" \
  --set "kuma.controlPlane.zone=<zone-name>" \
  --set "kuma.ingress.enabled=true" \
  --set "kuma.controlPlane.kdsGlobalAddress=grpcs://<global-kds-address>:5685" \
  --set "kuma.controlPlane.tls.kdsZoneClient.skipVerify=true" \
  | kubectl apply -f -

helm install --create-namespace --namespace kong-mesh-system \
  --set "kuma.controlPlane.mode=zone" \
  --set "kuma.controlPlane.zone=<zone-name>" \
  --set "kuma.ingress.enabled=true" \
  --set "kuma.controlPlane.kdsGlobalAddress=grpcs://<global-kds-address>:5685" \
  --set "kuma.controlPlane.tls.kdsZoneClient.skipVerify=true" \
  kong-mesh kong-mesh/kong-mesh

where kuma.controlPlane.zone is the same value for all zone control planes in the same zone.

Add --set kuma.egress.enabled=true to list of arguments if you want to deploy optional Zone Egress.

Set --set kuma.controlPlane.tls.kdsZoneClient.skipVerify=true because the default global control plane’s certificate is self-signed. For production use a certificate signed by a trusted CA. See Secure access across services page for more information.

After installing a zone control plane, make sure to restart the application pods that are already running such that the data plane proxies can be connected.

When running the zone control plane in Universal mode, a database must be used to persist state for production deployments. Ensure that migrations have been run against the database prior to running the zone control plane.

1. On each zone control plane, run:

    KUMA_MODE=zone \
    KUMA_MULTIZONE_ZONE_NAME=<zone-name> \
    KUMA_ENVIRONMENT=universal \
    KUMA_STORE_TYPE=postgres \
    KUMA_STORE_POSTGRES_HOST=<postgres-host> \
    KUMA_STORE_POSTGRES_PORT=<postgres-port> \
    KUMA_STORE_POSTGRES_USER=<postgres-user> \
    KUMA_STORE_POSTGRES_PASSWORD=<postgres-password> \
    KUMA_STORE_POSTGRES_DB_NAME=<postgres-db-name> \
    KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS=grpcs://<global-kds-address>:5685 \
    kuma-cp run
   

   where KUMA_MULTIZONE_ZONE_NAME is the same value for all zone control planes in the same zone.

   KUMA_MULTIZONE_ZONE_KDS_TLS_SKIP_VERIFY is required because the default global control plane’s certificate is self-signed. It’s recommended to use a certificate signed by a trusted CA in production. See Secure access across services page for more information.

2. Generate the zone proxy token:

   To register the zone ingress and zone egress with the zone control plane, we need to generate a token first

   kumactl generate zone-token --zone=<zone-name> --scope egress --scope ingress > /tmp/zone-token
   

   You can also generate the token with the REST API. Alternatively, you could generate separate tokens for ingress and egress.

3. Create an ingress data plane proxy configuration to allow kuma-cp services to be exposed for cross-zone communication:

   echo "type: ZoneIngress
   name: ingress-01
   networking:
     address: 127.0.0.1 # address that is routable within the zone
     port: 10000
     advertisedAddress: 10.0.0.1 # an address which other zones can use to consume this zone-ingress
     advertisedPort: 10000 # a port which other zones can use to consume this zone-ingress" > ingress-dp.yaml
   

4. Apply the ingress config, passing the IP address of the zone control plane to cp-address:

   kuma-dp run \
     --proxy-type=ingress \
     --cp-address=https://<kuma-cp-address>:5678 \
     --dataplane-token-file=/tmp/zone-token \
     --dataplane-file=ingress-dp.yaml
   

   If zone-ingress is running on a different machine than zone-cp you need to copy CA cert file from zone-cp (located in ~/.kuma/kuma-cp.crt) to somewhere accessible by zone-ingress (e.g. /tmp/kuma-cp.crt). Modify the above command and provide the certificate path in --ca-cert-file argument.

   kuma-dp run \
     --proxy-type=ingress \
     --cp-address=https://<kuma-cp-address>:5678 \
     --dataplane-token-file=/tmp/zone-token \
     --ca-cert-file=/tmp/kuma-cp.crt \
     --dataplane-file=ingress-dp.yaml
   

5. Optional: if you want to deploy zone egress

   Create a ZoneEgress data plane proxy configuration to allow kuma-cp services to be configured to proxy traffic to other zones or external services through zone egress:

   echo "type: ZoneEgress
   name: zoneegress-01
   networking:
     address: 127.0.0.1 # address that is routable within the zone
     port: 10002" > zoneegress-dataplane.yaml
   

6. Apply the egress config, passing the IP address of the zone control plane to cp-address:

    kuma-dp run \
      --proxy-type=egress \
      --cp-address=https://<kuma-cp-address>:5678 \
      --dataplane-token-file=/tmp/zone-token \
      --dataplane-file=zoneegress-dataplane.yaml
   

To view the list of service names available, run:

kubectl get serviceinsight all-services-default -oyaml

apiVersion: kuma.io/v1alpha1
kind: ServiceInsight
mesh: default
metadata:
  name: all-services-default
spec:
  services:
    echo-server_echo-example_svc_1010:
      dataplanes:
        online: 1
        total: 1
      issuedBackends:
        ca-1: 1
      status: online

The following are some examples of different ways to address echo-server in the echo-example Namespace in a multi-zone mesh.

To send a request in the same zone, you can rely on Kubernetes DNS and use the usual Kubernetes hostnames and ports:

curl http://echo-server:1010

Requests are distributed round-robin between zones. You can use locality-aware load balancing to keep requests in the same zone.

To send a request to any zone, you can use the generated kuma.io/service and Kong Mesh DNS:

curl http://echo-server_echo-example_svc_1010.mesh:80

Kong Mesh DNS also supports RFC 1123 compatible names, where underscores are replaced with dots:

curl http://echo-server.echo-example.svc.1010.mesh:80

kumactl inspect services

SERVICE                                  STATUS               DATAPLANES
echo-service_echo-example_svc_1010       Online               1/1

To consume the service in a Universal deployment without transparent proxy add the following outbound to your dataplane configuration:

outbound:
  - port: 20012
    tags:
      kuma.io/service: echo-server_echo-example_svc_1010

From the data plane running you will now be able to reach the service using localhost:20012.

Alternatively, if you configure transparent proxy you can just call echo-server_echo-example_svc_1010.mesh without defining an outbound section.