In order to give you better service we use cookies. By continuing to use our website, you agree to the use of cookies as described in our Cookie Policy

Kong Logo
header icon

Kafka Log

Kong Enterprise only: This plugin is only available with a Kong Enterprise subscription. Please inquire about Kong Enterprise by contacting us, or start a free trial today.

Publish request and response logs to a Kafka topic.

Configuration Reference

You can configure this plugin using the Kong Admin API or through declarative configuration, which involves directly editing the Kong configuration file.

Enabling the plugin globally

A plugin which is not associated to any Service, Route, or Consumer is considered global, and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

Kong Admin API
Kubernetes
Declarative (YAML)

For example, configure this plugin globally with:

$ curl -X POST http://<admin-hostname>:8001/plugins/ \
    --data "name=kafka-log"  \
    --data-urlencode "config.bootstrap_servers=BOOTSTRAP_SERVERS" \
    --data-urlencode "config.topic=TOPIC" \
    --data "config.timeout=TIMEOUT" \
    --data "config.keepalive=KEEPALIVE" \
    --data "config.producer_request_acks=PRODUCER_REQUEST_ACKS" \
    --data "config.producer_request_timeout=PRODUCER_REQUEST_TIMEOUT" \
    --data "config.producer_request_limits_messages_per_request=PRODUCER_REQUEST_LIMITS_MESSAGES_PER_REQUEST" \
    --data "config.producer_request_limits_bytes_per_request=PRODUCER_REQUEST_LIMITS_BYTES_PER_REQUEST" \
    --data "config.producer_request_retries_max_attempts=PRODUCER_REQUEST_RETRIES_MAX_ATTEMPTS"

Create a KongClusterPlugin resource and label it as global:

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-kafka-log>
  labels:
    global: \"true\"
config: 
  bootstrap_servers: BOOTSTRAP_SERVERS
  topic: TOPIC
  timeout: TIMEOUT
  keepalive: KEEPALIVE
  producer_request_acks: PRODUCER_REQUEST_ACKS
  producer_request_timeout: PRODUCER_REQUEST_TIMEOUT
  producer_request_limits_messages_per_request: PRODUCER_REQUEST_LIMITS_MESSAGES_PER_REQUEST
  producer_request_limits_bytes_per_request: PRODUCER_REQUEST_LIMITS_BYTES_PER_REQUEST
  producer_request_retries_max_attempts: PRODUCER_REQUEST_RETRIES_MAX_ATTEMPTS
plugin: kafka-log

For example, configure this plugin using the plugins: entry in the declarative configuration file:

plugins:
- name: kafka-log
  config: 
    bootstrap_servers: BOOTSTRAP_SERVERS
    topic: TOPIC
    timeout: TIMEOUT
    keepalive: KEEPALIVE
    producer_request_acks: PRODUCER_REQUEST_ACKS
    producer_request_timeout: PRODUCER_REQUEST_TIMEOUT
    producer_request_limits_messages_per_request: PRODUCER_REQUEST_LIMITS_MESSAGES_PER_REQUEST
    producer_request_limits_bytes_per_request: PRODUCER_REQUEST_LIMITS_BYTES_PER_REQUEST
    producer_request_retries_max_attempts: PRODUCER_REQUEST_RETRIES_MAX_ATTEMPTS

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Form ParameterDescription
nameThe name of the plugin to use, in this case kafka-log.
enabled

default value: true		Whether this plugin will be applied.
config.bootstrap_servers
required

List of bootstrap brokers in a {host: host, port: port} format.
config.topic
required

The Kafka topic to publish to.
config.timeout
optional

default value:

10000

Socket timeout in milliseconds.
config.keepalive
optional

default value:

60000

Keepalive timeout in milliseconds.
config.producer_request_acks
optional

default value:

1

The number of acknowledgments the producer requires the leader to have received before considering a request complete. Allowed values: 0 for no acknowledgments, 1 for only the leader, and -1 for the full ISR.
config.producer_request_timeout
optional

default value:

2000

Time to wait for a Produce response in milliseconds.
config.producer_request_limits_messages_per_request
optional

default value:

200

Maximum number of messages to include in a single Produce request.
config.producer_request_limits_bytes_per_request
optional

default value:

1048576

Maximum size of a Produce request in bytes.
config.producer_request_retries_max_attempts
optional

default value:

10

Maximum number of retry attempts per single Produce request.
config.producer_request_retries_backoff_timeout
optional

default value:

100

Backoff interval between Produce retry attempts in milliseconds.
config.producer_async
optional

default value:

true

Flag to enable asynchronous mode.
config.producer_async_flush_timeout
optional

default value:

1000

Maximum time interval in milliseconds between buffer flushes in asynchronous mode.
config.producer_async_buffering_limits_messages_in_memory
optional

default value:

50000

Maximum number of messages that can be buffered in memory in asynchronous mode.
config.api_version
optional

default value:

0

API version of a Produce request. Allowed values: 0, 1, or 2.

Installation

Manually download and install:

$ git clone https://github.com/kong/kong-plugin-kafka-log.git /path/to/kong/plugins/kong-plugin-kafka-log
$ cd /path/to/kong/plugins/kong-plugin-kafka-log
$ luarocks make *.rockspec

In both cases, you need to change your Kong plugins configuration option to include this plugin:

plugins = bundled,kong-plugin-kafka-log

Or, if you don’t want to activate any of the bundled plugins:

plugins = kong-plugin-kafka-log

Then reload kong:

kong reload

Configuration

Enabling globally

$ curl -X POST http://kong:8001/plugins \
    --data "name=kafka-log" \
    --data "config.bootstrap_servers[1].host=localhost" \
    --data "config.bootstrap_servers[1].port=9092" \
    --data "config.topic=kong-log" \
    --data "config.timeout=10000" \
    --data "config.keepalive=60000" \
    --data "config.producer_request_acks=1" \
    --data "config.producer_request_timeout=2000" \
    --data "config.producer_request_limits_messages_per_request=200" \
    --data "config.producer_request_limits_bytes_per_request=1048576" \
    --data "config.producer_request_retries_max_attempts=10" \
    --data "config.producer_request_retries_backoff_timeout=100" \
    --data "config.producer_async=true" \
    --data "config.producer_async_flush_timeout=1000" \
    --data "config.producer_async_buffering_limits_messages_in_memory=50000"

Log Format

Similar to HTTP Log Plugin.

Implementation details

This plugin makes use of lua-resty-kafka client under the hood.

Known issues and limitations

Known limitations:

  1. There is no support for TLS
  2. There is no support for Authentication
  3. There is no support for message compression

Quickstart

The following guidelines assume that both Kong and Kafka have been installed on your local machine:

  1. Install kong-plugin-kafka-log as specified in the “Installation” section above.

  2. Create kong-log topic in your Kafka cluster:

     ${KAFKA_HOME}/bin/kafka-topics.sh --create \
     --zookeeper localhost:2181 \
     --replication-factor 1 \
     --partitions 10 \
     --topic kong-log

  3. Add kong-plugin-kafka-log plugin globally:

     curl -X POST http://localhost:8001/plugins \
     --data "name=kafka-log" \
     --data "config.bootstrap_servers[1].host=localhost \
     --data "config.bootstrap_servers[1].port=9092 \
     --data "config.topic=kong-log"

    Alternatively, create a Service with a Route, and attach the plugin to that Service.

  4. Make sample requests:

     for i in {1..50} ; do curl http://localhost:8000/request/$i ; done

  5. Verify the contents of Kafka topic:

     ${KAFKA_HOME}/bin/kafka-console-consumer.sh \
     --bootstrap-server localhost:9092 \
     --topic kong-log \
     --partition 0 \
     --from-beginning \
     --timeout-ms 1000