Multi-zone authentication

To add to the security of your deployments, Kong Mesh provides token generation for authenticating zone control planes to the global control plane.

The control plane token is a JWT that contains:

• The name of the zone the token is generated for
• Expiration date (10 years by default if not specified)

The control plane token is signed by a signing key that is autogenerated on the global control plane. The signing key is SHA256 encrypted.

You can check for the signing key:

$ kumactl get global-secrets

which returns something like:

NAME                             AGE
control-plane-signing-key-0001   36m

Set up tokens

To generate the tokens you need and configure your clusters:

• Generate a token for each zone control plane.
• Add the token to the configuration for each zone.
• Enable authentication on the global control plane.

Generate token for each zone

On the global control plane, authenticate and run the following command:

$ kumactl generate control-plane-token --zone=west --valid-for=720h > /tmp/token
$ cat /tmp/token

The generated token looks like:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJab25lIjoid2VzdCIsIlRva2VuU2VyaWFsTnVtYmVyIjoxfQ.kIrS5W0CPMkEVhuRXcUxk3F_uUoeI3XK1Gw-uguWMpQ

For authentication to the global control plane on Kubernetes, you can port-forward port 5681 to access the API.

Add token to each zone configuration

$ kumactl install control-plane \
  --mode=zone \
  --zone=<zone name> \
  --cp-token-path=/tmp/token \
  --ingress-enabled \
  --kds-global-address grpcs://`<global-kds-address>` | kubectl apply -f - 

$ kubectl create secret generic cp-token -n kong-mesh-system --from-file=/tmp/token

kuma:
  controlPlane:
    secrets:
      - Env: "KMESH_MULTIZONE_ZONE_KDS_AUTH_CP_TOKEN_INLINE"
        Secret: "cp-token"
        Key: "token"

$ KUMA_MODE=zone \
  KUMA_MULTIZONE_ZONE_NAME=<zone-name> \
  KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS=grpcs://<global-kds-address> \
  KMESH_MULTIZONE_ZONE_KDS_AUTH_CP_TOKEN_INLINE="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJab25lIjoid2VzdCIsIlRva2VuU2VyaWFsTnVtYmVyIjoxfQ.kIrS5W0CPMkEVhuRXcUxk3F_uUoeI3XK1Gw-uguWMpQ" \
  ./kuma-cp run

Enable authentication on the global control plane

If you are starting from scratch and not securing existing Kong Mesh deployment, you can do this as a first step.

$ kumactl install control-plane \
  --mode=global \
  --cp-auth=cpToken | kubectl apply -f -

kuma:
  controlPlane:
    envVars:
      KMESH_MULTIZONE_GLOBAL_KDS_AUTH_TYPE: cpToken

$ KUMA_MODE=global \
  KMESH_MULTIZONE_GLOBAL_KDS_AUTH_TYPE=cpToken \
  ./kuma-cp run

Verify the zone control plane is connected with authentication by looking at the global control plane logs:

2021-02-24T14:30:38.596+0100	INFO	kds.auth	Zone CP successfully authenticated using Control Plane Token	{"tokenSerialNumber": 1, "zone": "cluster-2"}

Revoke token

Kong Mesh does not keep a list of issued tokens. Whenever a single token is compromised, you can add it to revocation list so the token is no longer valid.

Every token has its own ID, which is available in the payload under the jti key. You can extract an ID from the token using jwt.io or the jwt-cli tool. Here is an example of a jti key:

0e120ec9-6b42-495d-9758-07b59fe86fb9

Specify a list of revoked IDs separated by commas (,) and store it as a GlobalSecret object named control-plane-token-revocations:

REVOCATIONS=$(echo '0e120ec9-6b42-495d-9758-07b59fe86fb9' | base64) && echo "apiVersion: v1
kind: Secret
metadata:
  name: control-plane-token-revocations
  namespace: kuma-system 
data:
  value: $REVOCATIONS
type: system.kuma.io/global-secret" | kubectl apply -f -

echo "
type: GlobalSecret
name: control-plane-token-revocations
data: " | kumactl apply --var revocations=$(echo '0e120ec9-6b42-495d-9758-07b59fe86fb9' | base64) -f -

Rotate signing key

If a signing key is compromised, you must rotate it and all the tokens. When the signing key is rotated, all tokens signed with the key are no longer valid. You do not need to add the tokens manually to a revocation list.

Generate new signing key

The signing key is stored as a GlobalSecret with a name that looks like control-plane-signing-key-{serialNumber}.

Make sure to generate the new signing key with a serial number greater than the serial number of the current signing key.

$ kubectl get secrets -n kong-mesh-system --field-selector='type=system.kuma.io/global-secret'
NAME                             TYPE                           DATA   AGE
control-plane-signing-key-0001   system.kuma.io/global-secret   1      25m

$ TOKEN="$(kumactl generate signing-key)" && echo "
apiVersion: v1
data:
  value: $TOKEN
kind: Secret
metadata:
  name: control-plane-signing-key-0002
  namespace: kong-mesh-system
type: system.kuma.io/global-secret
" | kubectl apply -f - 

$ kumactl get global-secrets
NAME                             AGE
control-plane-signing-key-0001   36m

echo "
type: GlobalSecret
name: control-plane-signing-key-0002
data: 
" | kumactl apply --var key=$(kumactl generate signing-key) -f -

Regenerate control plane tokens

Create and add a new token for each zone control plane. These tokens are automatically created with the signing key that’s assigned the highest serial number, so they’re created with the new signing key.

Make sure the new signing key is available; otherwise old and new tokens are created with the same signing key and can both provide authentication.

Remove the old signing key

$ kubectl delete secret control-plane-signing-key-0001 -n kong-mesh-system

$ kumactl delete global-secret control-plane-signing-key-0001

All new connections to the global control plane now require tokens signed with the new signing key.

Restart the global control plane

Restart all instances of the global control plane. All connections are now authenticated with the new tokens.

Explore an example token

You can decode the tokens to validate the signature or explore details.

For example, run:

$ kumactl generate control-plane-token --zone=west

which returns:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJab25lIjoid2VzdCIsIlRva2VuU2VyaWFsTnVtYmVyIjoxfQ.kIrS5W0CPMkEVhuRXcUxk3F_uUoeI3XK1Gw-uguWMpQ

Paste the token into the UI at jwt.io, or run

$ kumactl generate control-plane-token --zone=west | jwt

The result looks like:

Additional security

By default, a connection from the zone control plane to the global control plane is secured with TLS. You should also configure the zone control plane to verify the certificate authority (CA) of the global control plane.