Kafka Log

Publish request and response logs to an Apache Kafka topic. For more information, see Kafka topics.

Kong also provides a Kafka plugin for request transformations. See Kafka Upstream.

Configuration Reference

This plugin is compatible with DB-less mode.

In DB-less mode, you configure Kong Gateway declaratively. Therefore, the Admin API is mostly read-only. The only tasks it can perform are all related to handling the declarative config, including:

• Setting a target's health status in the load balancer
• Validating configurations against schemas
• Uploading the declarative configuration using the /config endpoint

Example plugin configuration

A plugin which is not associated to any service, route, or consumer is considered global, and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

The following examples provide some typical configurations for enabling the kafka-log plugin globally.

curl -X POST http://localhost:8001/plugins/ \
  --data "name=kafka-log"  \
    --data-urlencode "config.bootstrap_servers=BOOTSTRAP_SERVERS" \
    --data-urlencode "config.topic=TOPIC" \
    --data-urlencode "config.authentication.strategy=sasl" \
    --data-urlencode "config.authentication.mechanism=PLAIN" \
    --data-urlencode "config.authentication.user=admin" \
    --data-urlencode "config.authentication.password=admin-secret" \
    --data "config.authentication.tokenauth=false" \
    --data "config.security.ssl=false" \
    --data-urlencode "config.security.certificate_id=nil" \
    --data "config.timeout=10000" \
    --data "config.keepalive=60000" \
    --data "config.producer_request_acks=-1" \
    --data "config.producer_request_timeout=2000" \
    --data "config.producer_request_limits_messages_per_request=200" \
    --data "config.producer_request_limits_bytes_per_request=1048576" \
    --data "config.producer_request_retries_max_attempts=10"

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-kafka-log>
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: \"true\"
config: 
  bootstrap_servers:
  - BOOTSTRAP_SERVERS
  topic:
  - TOPIC
  authentication:
    strategy: sasl
  authentication:
    mechanism: PLAIN
  authentication:
    user: admin
  authentication:
    password: admin-secret
  authentication:
    tokenauth: false
  security:
    ssl: false
  security:
    certificate_id: nil
  timeout: 10000
  keepalive: 60000
  producer_request_acks: -1
  producer_request_timeout: 2000
  producer_request_limits_messages_per_request: 200
  producer_request_limits_bytes_per_request: 1048576
  producer_request_retries_max_attempts: 10
plugin: kafka-log

plugins:
- name: kafka-log
  config: 
    bootstrap_servers:
    - BOOTSTRAP_SERVERS
    topic:
    - TOPIC
    authentication:
      strategy: sasl
    authentication:
      mechanism: PLAIN
    authentication:
      user: admin
    authentication:
      password: admin-secret
    authentication:
      tokenauth: false
    security:
      ssl: false
    security:
      certificate_id: nil
    timeout: 10000
    keepalive: 60000
    producer_request_acks: -1
    producer_request_timeout: 2000
    producer_request_limits_messages_per_request: 200
    producer_request_limits_bytes_per_request: 1048576
    producer_request_retries_max_attempts: 10

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Quickstart

The following guidelines assume that both Kong Enterprise and Kafka have been installed on your local machine.

Note: We use zookeeper in the following example, which is not required or has been removed on some Kafka versions. Refer to the Kafka ZooKeeper documentation for more information.

1. Create a kong-log topic in your Kafka cluster:

    ${KAFKA_HOME}/bin/kafka-topics.sh --create \
        --zookeeper localhost:2181 \
        --replication-factor 1 \
        --partitions 10 \
        --topic kong-log
   

2. Add the kafka-log plugin globally:

    curl -X POST http://localhost:8001/plugins \
        --data "name=kafka-log" \
        --data "config.bootstrap_servers[1].host=localhost" \
        --data "config.bootstrap_servers[1].port=9092" \
        --data "config.topic=kong-log"
   

3. Make sample requests:

    for i in {1..50} ; do curl http://localhost:8000/request/$i ; done
   

4. Verify the contents of the Kafka kong-log topic:

    ${KAFKA_HOME}/bin/kafka-console-consumer.sh \
        --bootstrap-server localhost:9092 \
        --topic kong-log \
        --partition 0 \
        --from-beginning \
        --timeout-ms 1000
   

Log format

Similar to the HTTP Log Plugin.

Implementation details

This plugin uses the lua-resty-kafka client.

When encoding request bodies, several things happen:

• For requests with a content-type header of application/x-www-form-urlencoded, multipart/form-data, or application/json, this plugin passes the raw request body in the body attribute, and tries to return a parsed version of those arguments in body_args. If this parsing fails, an error message is returned and the message is not sent.
• If the content-type is not text/plain, text/html, application/xml, text/xml, or application/soap+xml, then the body will be base64-encoded to ensure that the message can be sent as JSON. In such a case, the message has an extra attribute called body_base64 set to true.

TLS

Enable TLS by setting config.security.ssl to true.

mTLS

Enable mTLS by setting a valid UUID of a certificate in config.security.certificate_id.

Note that this option needs config.security.ssl set to true. See Certificate Object in the Admin API documentation for information on how to set up Certificates.

SASL Authentication

This plugin supports the following authentication mechanisms:

• PLAIN: Enable this mechanism by setting config.authentication.mechanism to PLAIN. You also need to provide a username and password with the config options config.authentication.user and config.authentication.password respectively.

• SCRAM: In cryptography, the Salted Challenge Response Authentication Mechanism (SCRAM) is a family of modern, password-based challenge–response authentication mechanisms providing authentication of a user to a server. The Kafka Log plugin supports the following:

  • SCRAM-SHA-256: Enable this mechanism by setting config.authentication.mechanism to SCRAM-SHA-256. You also need to provide a username and password with the config options config.authentication.user and config.authentication.password respectively.

  • SCRAM-SHA-512: Enable this mechanism by setting config.authentication.mechanism to SCRAM-SHA-512. You also need to provide a username and password with the config options config.authentication.user and config.authentication.password respectively.

• Delegation Tokens: Delegation Tokens can be generated in Kafka and then used to authenticate this plugin. Delegation Tokens leverage the SCRAM-SHA-256 authentication mechanism. The tokenID is provided with the config.authentication.user field and the token-hmac is provided with the config.authentication.password field. To indicate that a token is used you have to set the config.authentication.tokenauth setting to true.

  Read more on how to create, renew, and revoke delegation tokens.

Known issues and limitations

Known limitations:

1. Message compression is not supported.
2. The message format is not customizable.

Changelog

Kong Gateway 2.8.x

• Added support for the SCRAM-SHA-512 authentication mechanism.

• Added the cluster_name configuration parameter.

• The authentication.user and authentication.password configuration fields are now marked as referenceable, which means they can be securely stored as secrets in a vault. References must follow a specific format.

Kong Gateway 2.7.x

• Starting with Kong Gateway 2.7.0.0, if keyring encryption is enabled, the config.authentication.user and config.authentication.password parameter values will be encrypted.

  There’s a bug in Kong Gateway that prevents keyring encryption from working on deeply nested fields, so the encrypted=true setting does not currently have any effect in this plugin.

Kong Gateway 2.6.x

• The Kafka Log plugin now supports TLS, mTLS, and SASL auth. SASL auth includes support for PLAIN, SCRAM-SHA-256, and delegation tokens.