PermalinkHybrid Mode Deployment
PermalinkIntroduction
Traditionally, Kong has always required a database, which could be either Postgres or Cassandra, to store its configured entities such as Routes, Services and Plugins. In Kong 1.1 we added the capability to run Kong without a database, using only in-memory storage for entities: we call this DB-less mode.
In Kong 2.0, we introduced a new method of deploying Kong which is called the Hybrid mode, also known as Control Plane / Data Plane Separation (CP/DP).
In this mode Kong nodes in a cluster are separated into two roles: Control Plane (CP), where configuration is managed and the Admin API is served from, and Data Plane (DP), which serves traffic for the proxy. Each DP node is connected to one of the CP nodes. Instead of accessing the database contents directly in the traditional deployment method, the DP nodes maintains connection with CP nodes, and receives the latest configuration.
Deploying using Hybrid mode has a number of benefits:
- Drastically reduce the amount of traffic on the database, since only CP nodes need a direct connection to the database.
- Increased security, in an event where one of the DP nodes gets intruded, an attacker won’t be able to affect other nodes in the Kong cluster.
- Easiness of management, since an admin will only need to interact with the CP nodes to control and monitor the status of the entire Kong cluster.
PermalinkConfiguration Properties
Hybrid Mode introduces the following configuration properties:
| Parameter | Description |
|---|---|
cluster_listen Optional |
List of addresses and ports on which the Control Plane will listen for incoming Data Plane connections. Defaults to 0.0.0.0:8005. Note this port is always protected with Mutual TLS (mTLS) encryption. Ignored on Data Plane nodes. |
cluster_control_plane Required |
Address and port that the Data Plane nodes use to connect to the Control Plane. Must point to the port configured using the cluster_listen property on the Control Plane node. Ignored on Control Plane nodes. |
cluster_mtls Optional |
One of "shared" or "pki". Indicates whether Hybrid Mode will use a shared certificate/key pair for CP/DP mTLS or if PKI mode will be used. Defaults to "shared". See below sections for differences in mTLS modes. |
The following properties are used differently between “shared” and “PKI” modes:
| Parameter | Description | Shared Mode | PKI Mode |
|---|---|---|---|
cluster_cert and cluster_cert_key Required |
Certificate/key pair used for mTLS between CP/DP nodes. | Same between CP/DP nodes. | Unique certificate for each node, generated from the CA specified by cluster_ca_cert. |
cluster_ca_cert Required in PKI mode |
The trusted CA certificate file in PEM format used to verify the cluster_cert. |
Ignored | CA certificate used to verify cluster_cert, same between CP/DP nodes. Required |
cluster_server_name Required in PKI mode |
The SNI Server Name presented by the DP node mTLS handshake. | Ignored | In PKI mode the DP nodes will also verify that the Common Name (CN) or Subject Alternative Name (SAN) inside certificate presented by CP matches the cluster_server_name value. |
PermalinkTopology
PermalinkGenerating Certificate/Key Pair
Before using the Hybrid Mode, it is necessary to have a shared certificate/key pair generated so that the communication security between CP and DP nodes can be established.
This certificate/key pair is shared by both CP and DP nodes, mutual TLS handshake (mTLS) is used for authentication so the actual private key is never transferred on the network.
To create a certificate/key pair:
kong hybrid gen_cert
This will generate cluster.crt and cluster.key files and save them to the current directory.
By default it is valid for 3 years, but can be set longer or shorter with the --days option.
See kong hybrid --help for more usage information.
The cluster.crt and cluster.key files need to be transferred to both Kong CP and DP nodes.
Observe proper permission setting on the key file to ensure it can only be read by Kong.
PermalinkSetting Up Kong Control Plane Nodes
Starting the Control Plane is fairly simple. Aside from the database configuration
which is the same as today, we need to specify the “role” of the node to “control_plane”.
This will cause Kong to listen on 0.0.0.0:8005 by default for Data Plane
connections. The 8005 port on the Control Plane will need to be
accessible by all the Data Plane it controls through any firewalls you may have
in place.
In addition, the cluster_cert and cluster_cert_key configuration need to point to
the certificate/key pair that was generated above.
KONG_ROLE=control_plane KONG_CLUSTER_CERT=cluster.crt KONG_CLUSTER_CERT_KEY=cluster.key kong start
Or in kong.conf:
role = control_plane
cluster_cert = cluster.crt
cluster_cert_key = cluster.key
Note that Control Plane still needs a database (Postgres or Cassandra) to store the “source of truth” configurations, although the database never needs to be access by Data Plane nodes. You may run more than a single Control Plane nodes to provide load balancing and redundancy as long as they points to the same backend database.
PermalinkPKI mode
Starting in Kong 2.1, the Hybrid cluster can use certificates signed by a central certificate authority (CA).
This mode can be activated by setting cluster_mtls to "pki" in kong.conf. The default value is "shared".
In PKI mode, the Control Plane and Data Plane don’t need to use the same cluster_cert and cluster_cert_key.
Instead, Kong validates both sides by checking if they are issued by configured CA. This eliminates the risk of
transporting private keys around.
Note: Certificates on CP and DP must contain the
TLS Web Server AuthenticationandTLS Web Client Authenticationas X509v3 Extended Key Usage extension, respectively.
Kong doesn’t validate the CommonName (CN) in the DP certificate; it can take an arbitrary value.
Set the following configuration parameters in kong.conf on the Control Plane:
cluster_mtls = pki
cluster_ca_cert = /path/to/ca-cert.crt
cluster_cert = control-plane.crt
cluster_cert_key = control-plane.key
cluster_ca_cert specifies the root CA certificate for cluster_cert and cluster_cert_key. This
certificate must be the root CA certificate and not any of an intermediate CA.
Kong allows at most 3 levels of intermediate CAs to be used between the root CA and the cluster certificate.
Set the following configuration parameters in kong.conf on the Data Plane:
cluster_mtls = pki
cluster_server_name = control-plane.kong.yourcorp.tld
cluster_cert = data-plane.crt
cluster_cert_key = data-plane.crt
cluster_server_name specifies the SNI (Server Name Indication extension) to use for Data Plane
connections to the Control Plane through TLS. When not set, Data Plane will use kong_clustering as the SNI.
PermalinkStarting Data Plane Nodes
Now we have a Control Plane running, it is not much useful if no Data Plane nodes are talking to it and serving traffic (remember Control Plane nodes can not be used for proxying). To start the Data Plane, all we need to do is to specify the “role” to “data_plane”, give it the address and port of where the Control Plane can be reached and the node automatically connects and syncs itself up with the current configuration.
Similar to the CP config above, cluster_cert and cluster_cert_key configuration need to
point to the same files as the CP has. In addition the cluster.crt file need to be listed
as trusted by OpenResty through the lua_ssl_trusted_certificate configuration. If you
have already specified a different lua_ssl_trusted_certificate, then adding the content
of cluster.crt into that file will achieve the same result.
Note: In this release of the Hybrid Mode, the Data Plane receives updates from the Control
Plane via a format that is similar to the Declarative Config, therefore the database
property has to be set to off for Kong to start up properly.
KONG_ROLE=data_plane KONG_CLUSTER_CONTROL_PLANE=control-plane.example.com:8005 KONG_CLUSTER_CERT=cluster.crt KONG_CLUSTER_CERT_KEY=cluster.key KONG_LUA_SSL_TRUSTED_CERTIFICATE=cluster.crt KONG_DATABASE=off kong start
Or in kong.conf:
role = data_plane
cluster_control_plane = control-plane.example.com:8005
database = off
cluster_cert = cluster.crt
cluster_cert_key = cluster.key
lua_ssl_trusted_certificate = cluster.crt
PermalinkChecking the status of the cluster
You may want to check the status of the Kong cluster from time to time, such as checking to see the which nodes are actively receiving config updates from Control Plane, or when was it last updated. This can be achieved by using the Control Plane’s new Cluster Status API:
# on Control Plane node
http :8001/clustering/status
{
"a08f5bf2-43b8-4f1c-bdf5-0a0ffb421c21": {
"config_hash": "64d661f505f7e1de5b4c5e5faa1797dd",
"hostname": "data-plane-2",
"ip": "192.168.10.3",
"last_seen": 1571197860
},
"e1fd4970-6d24-4dfb-b2a7-5a832a5de6e1": {
"config_hash": "64d661f505f7e1de5b4c5e5faa1797dd",
"hostname": "data-plane-1",
"ip": "192.168.10.4",
"last_seen": 1571197866
}
}
The Cluster Status API provides helpful information such as the name of the node and last time it synced with the Control Plane, as well as config version currently running on them.
PermalinkManaging the cluster using Control Plane nodes
Once the nodes are setup, use the Admin API on the Control Plane as usual, those changes will be synced and updated on the Data Plane nodes automatically within seconds.
PermalinkFault tolerance
A valid question you may ask is: What would happen if Control Plane nodes are down, will the Data Plane keep functioning? The answer is yes. Data plane caches the latest configuration it received from the Control Plane on the local disk. In case Control Plane stops working, Data Plane will keep serving requests using cached configurations. It does so while constantly trying to reestablish communication with the Control Plane.
This means that the Data Plane nodes can be restarted while the Control Plane is down, and still proxy traffic normally.
PermalinkLimitations
PermalinkPlugins compatibility
This version of Hybrid Mode uses declarative config as the config sync format which means it has the same limitations as declarative config as of today. Please refer to the Plugin Compatibility section of declarative config documentation for more information.