Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 3.2.x (latest)
  • 3.1.x
  • 3.0.x
  • 2.8.x
  • 2.7.x
  • 2.6.x
  • Older Enterprise versions (2.1-2.5)
  • Older OSS versions (2.1-2.5)
  • Archive (pre-2.1)
    • Overview of Kong Gateway
      • Version Support Policy
      • Third Party Dependencies
      • Browser Support
    • Stability
    • Release Notes
      • Services
        • Overview
        • Configure Routes with Expressions
      • Upstreams
      • Plugins
      • Routing Traffic
      • Load Balancing
      • Health Checks and Circuit Breakers
      • Kong Performance Testing
    • Glossary
    • Get Kong
    • Services and Routes
    • Rate Limiting
    • Proxy Caching
    • Key Authentication
    • Load-Balancing
      • Overview
        • Overview
        • Deploy Kong Gateway in Hybrid mode
      • DB-less Deployment
      • Traditional
      • Overview
        • Helm
        • OpenShift with Helm
        • kubectl apply
        • Kubernetes Deployment Options
        • Using docker run
        • Build your own Docker images
        • Amazon Linux
        • Debian
        • Red Hat
        • Ubuntu
      • Running Kong as a non-root user
      • Securing the Admin API
      • Using systemd
      • Start Kong Gateway Securely
      • Programatically Creating Admins
      • Enabling RBAC
      • Overview
      • Download your License
      • Deploy Enterprise License
      • Using the License API
      • Monitor Licenses Usage
      • Default Ports
      • DNS Considerations
      • Network and Firewall
      • CP/DP Communication through a Forward Proxy
        • Configure PostgreSQL TLS
        • Troubleshooting PostgreSQL TLS
    • Kong Configuration File
    • Environment Variables
    • Serving a Website and APIs from Kong
      • Overview
      • Prometheus
      • StatsD
      • Datadog
      • Overview
      • Writing a Custom Trace Exporter
      • Tracing API Reference
    • Resource Sizing Guidelines
    • Security Update Process
    • Blue-Green Deployments
    • Canary Deployments
    • Clustering Reference
      • Log Reference
      • Dynamic log level updates
      • Customize Gateway Logs
      • Upgrade Kong Gateway 3.x.x
      • Migrate from OSS to Enterprise
    • Overview
      • Overview
      • Metrics
      • Analytics with InfluxDB
      • Analytics with Prometheus
      • Estimate Analytics Storage in PostgreSQL
      • Overview
      • Getting Started
      • Advanced Usage
        • Overview
        • Environment Variables
        • AWS Secrets Manager
        • Google Secrets Manager
        • Hashicorp Vault
        • Securing the Database with AWS Secrets Manager
      • Reference Format
      • Overview
      • Get Started with Dynamic Plugin Ordering
      • Overview
      • Enable the Dev Portal
      • Publish an OpenAPI Spec
      • Structure and File Types
      • Themes Files
      • Working with Templates
      • Using the Editor
        • Basic Auth
        • Key Auth
        • OIDC
        • Sessions
        • Adding Custom Registration Fields
        • Manage Developers
        • Developer Roles and Content Permissions
        • Authorization Provider Strategy
        • Enable Application Registration
        • Enable Key Authentication for Application Registration
          • External OAuth2 Support
          • Set up Okta and Kong for External Oauth
          • Set up Azure AD and Kong for External Authentication
        • Manage Applications
        • Theme Editing
        • Migrating Templates Between Workspaces
        • Markdown Rendering Module
        • Customizing Portal Emails
        • Adding and Using JavaScript Assets
        • Single Page App in Dev Portal
        • Alternate OpenAPI Renderer
      • SMTP
      • Workspaces
      • Helpers CLI
      • Portal API Documentation
    • Audit Logging
    • Keyring and Data Encryption
    • Workspaces
    • Consumer Groups
    • Event Hooks
    • Configure Data Plane Resilience
    • About Control Plane Outage Management
      • Overview
      • Install the FIPS Compliant Package
      • FIPS 140-2 Compliant Plugins
    • Overview
    • Enable Kong Manager
      • Services and Routes
      • Rate Limiting
      • Proxy Caching
      • Authentication with Consumers
      • Load Balancing
      • Overview
      • Create a Super Admin
      • Workspaces and Teams
      • Reset Passwords and RBAC Tokens
      • Basic Auth
        • Configure LDAP
        • LDAP Service Directory Mapping
        • Configure OIDC
        • OIDC Authenticated Group Mapping
      • Sessions
        • Overview
        • Enable RBAC
        • Add a Role and Permissions
        • Create a User
        • Create an Admin
    • Networking Configuration
    • Workspaces
    • Create Consumer Groups
    • Sending Email
    • Overview
    • File Structure
    • Implementing Custom Logic
    • Plugin Configuration
    • Accessing the Data Store
    • Storing Custom Entities
    • Caching Custom Entities
    • Extending the Admin API
    • Writing Tests
    • (un)Installing your Plugin
      • Overview
      • kong.client
      • kong.client.tls
      • kong.cluster
      • kong.ctx
      • kong.ip
      • kong.jwe
      • kong.log
      • kong.nginx
      • kong.node
      • kong.request
      • kong.response
      • kong.router
      • kong.service
      • kong.service.request
      • kong.service.response
      • kong.table
      • kong.tracing
      • kong.vault
      • kong.websocket.client
      • kong.websocket.upstream
      • Go
      • Javascript
      • Python
      • Running Plugins in Containers
      • External Plugin Performance
    • Overview
        • Overview
        • OpenID Connect with Curity
        • OpenID Connect with Azure AD
        • OpenID Connect with Google
        • OpenID Connect with Okta
        • OpenID Connect with Auth0
        • OpenID Connect with Cognito
      • Authentication Reference
      • Allow Multiple Authentication Plugins
    • Rate Limiting Plugin
      • Add a Body Value
    • GraphQL
      • gRPC Plugins
      • Configure a gRPC service
    • Overview
    • Information Routes
    • Health Routes
    • Tags
    • Debug Routes
    • Services
    • Routes
    • Consumers
    • Plugins
    • Certificates
    • CA Certificates
    • SNIs
    • Upstreams
    • Targets
    • Vaults
    • Keys
    • Licenses
    • Workspaces
    • RBAC
    • Admins
    • Developers
    • Consumer Groups
    • Event Hooks
    • Keyring and Data Encryption
    • Audit Logs
    • kong.conf
    • Injecting Nginx Directives
    • CLI
    • Key Management
    • Performance Testing Framework
    • Router Expressions Language
    • FAQ

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • Enable audit logging
  • Request audits
    • Generating and viewing audit logs
    • Workspaces and RBAC
    • Kong Manager authentication
    • Limiting audit log generation
  • Database audits
    • Generating and viewing audit logs
    • Limiting audit log generation
  • Audit log retention
  • Digital signatures
    • Setting up log signing
    • Validating signatures
  • More information
Kong Gateway
3.2.x (latest)
  • Home
  • Kong Gateway
  • Kong Enterprise
  • Audit Logging in Kong Gateway

Audit Logging in Kong Gateway

Kong Gateway provides a granular logging facility through its Admin API. This allows cluster administrators to keep detailed track of changes made to the cluster configuration throughout its lifetime, aiding in compliance efforts and providing valuable data points during forensic investigations. Generated audit log trails are Workspace and RBAC-aware, providing Kong operators a deep and wide look into changes happening within the cluster.

Enable audit logging

Audit logging is disabled by default. Configure it through Kong Gateway configuration in kong.conf:

audit_log = on # audit logging is enabled
audit_log = off # audit logging is disabled

or via environment variables:

export KONG_AUDIT_LOG=on
export KONG_AUDIT_LOG=off

As with other Kong configurations, changes take effect on kong reload or kong restart.

Request audits

Generating and viewing audit logs

Audit logging provides granular details of each HTTP request that was handled by Kong’s Admin API. Audit log data is written to Kong’s database. As a result, request audit logs are available via the Admin API (in addition to via direct database query).

For example, consider a query to the Admin API to the /status endpoint:

http :8001/status

You get the following response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 13 Nov 2018 17:32:47 GMT
Server: kong/ kong/3.2.2.0-enterprise-edition
Transfer-Encoding: chunked
X-Kong-Admin-Request-ID: ZuUfPfnxNn7D2OTU6Xi4zCnQkavzMUNM

{
    "database": {
        "reachable": true
    },
    
    "memory": {
        "lua_shared_dicts": {
            ...
        },
         "workers_lua_vms": [
            ...
         ]
    }

    "server": {
        "connections_accepted": 1,
        "connections_active": 1,
        "connections_handled": 1,
        "connections_reading": 0,
        "connections_waiting": 0,
        "connections_writing": 1,
        "total_requests": 1
    }
}

The above interaction with the Admin API generates a correlating entry in the audit log table. Querying the audit log via the Admin API returns the details of the previous interaction:

http :8001/audit/requests
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 13 Nov 2018 17:35:24 GMT
Server: kong/3.2.2.0-enterprise-edition
Transfer-Encoding: chunked
X-Kong-Admin-Request-ID: VXgMG1Y3rZKbjrzVYlSdLNPw8asVwhET
{
    "data": [
        {
            "client_ip": "127.0.0.1",
            "method": "GET",
            "path": "/status",
            "payload": null,
            "rbac_user_id": null,
            "rbac_user_name": null,
            "removed_from_payload": null,
            "request_id": "OjOcUBvt6q6XJlX3dd6BSpy1uUkTyctC",
            "request_source": null,
            "request_timestamp": 1676424547,
            "signature": null,
            "status": 200,
            "ttl": 2591997,
            "workspace": "1065b6d6-219f-4002-b3e9-334fc3eff46c"
        }
    ],
    "total": 1
}

Note the value of the request_id field. This is tied to the X-Kong-Admin-Request-ID response header received in the first transaction. This allows close association of client requests and audit log records within the Kong cluster.

Because every audit log entry is made available via Kong’s Admin API, it is possible to transport audit log entries into existing logging warehouses, SIEM solutions, or other remote services for duplication and inspection.

Workspaces and RBAC

Audit log entries are written with an awareness of the requested workspace, and the RBAC user (if present). When RBAC is enforced, the RBAC user’s UUID will be written to the rbac_user_id field in the audit log entry, and the username will be written to the rbac_user_name field:

{
    "data": [
        {
            "client_ip": "127.0.0.1",
            "method": "GET",
            "path": "/status",
            "payload": null,
            "rbac_user_id": "2e959b45-0053-41cc-9c2c-5458d0964331",
            "rbac_user_name": "admin",
            "request_id": "QUtUa3RMbRLxomqcL68ilOjjl68h56xr",
            "request_source": "kong-manager",
            "request_timestamp": 1581617463,
            "signature": null,
            "status": 200,
            "ttl": 2591995,
            "workspace": "0da4afe7-44ad-4e81-a953-5d2923ce68ae"
        }
    ],
    "total": 1
}

Note the presence of the workspace field. This is the UUID of the workspace with which the request is associated.

Kong Manager authentication

You can track login and logout events for the Kong Manager through the path, method, and request_source audit log fields.

For example, review the following audit log entry:

{
    "data": [
        {
            "client_ip": "127.0.0.1",
            "method": "GET",
            "path": "/auth",
            "payload": null,
            "rbac_user_id": "2e959b45-0053-41cc-9c2c-5458d0964331",
            "rbac_user_name": "admin",
            "request_id": "QUtUa3RMbRLxomqcL68ilOjjl68h56xr",
            "request_source": "kong-manager",
            "request_timestamp": 1581617463,
            "signature": null,
            "status": 200,
            "ttl": 2591995,
            "workspace": "0da4afe7-44ad-4e81-a953-5d2923ce68ae"
        }
    ],
    "total": 1
}

The request_source field tells you that the action occurred in Kong Manager.

The method and path fields correspond either to a login or logout event:

  • Login event: "method": "GET", "path": "/auth"
  • Logout event: "method": "DELETE", "path": "/auth?session_logout=true"

Limiting audit log generation

You may want to ignore audit log generation for certain Admin API requests, such as requests to the /status endpoint for health checking, or to ignore requests to a specific path prefix, for example, a given workspace.

Use the audit_log_ignore_methods and audit_log_ignore_paths configuration options:

audit_log_ignore_methods = GET,OPTIONS
# do not generate an audit log entry for GET or OPTIONS HTTP requests
audit_log_ignore_paths = /foo,/status,^/services,/routes$,/one/.+/two,/upstreams/
# do not generate an audit log entry for requests that match the above regular expressions

The values of audit_log_ignore_paths are matched via a Perl-compatible regular expression.

For example, when audit_log_ignore_paths = /foo,/status,^/services,/routes$,/one/.+/two,/upstreams/, the following request paths do not generate an audit log entry in the database:

  • /status
  • /status/
  • /foo
  • /foo/
  • /services
  • /services/example/
  • /one/services/two
  • /one/test/two
  • /routes
  • /plugins/routes
  • /one/routes/two
  • /upstreams/
  • bad400request

The following request paths generate an audit log entry in the database:

  • /example/services
  • /routes/plugins
  • /one/two
  • /routes/
  • /upstreams

Database audits

Generating and viewing audit logs

In addition to Admin API request data, Kong can generate granular audit log entries for all insertions, updates, and deletions to the cluster database. Database update audit logs are also associated with Admin API request unique IDs. Consider the following request to create a consumer:

http :8001/consumers username=bob

Response:

HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 13 Nov 2018 17:50:18 GMT
Server: kong/3.2.2.0-enterprise-edition
Transfer-Encoding: chunked
X-Kong-Admin-Request-ID: 59fpTWlpUtHJ0qnAWBzQRHRDv7i5DwK2

{
    "created_at": 1542131418000,
    "id": "16787ed7-d805-434a-9cec-5e5a3e5c9e4f",
    "type": 0,
    "username": "bob"
}

As seen before, a request audit log is generated with details about the request. Note the presence of the payload field, recorded when the request body is present:

http :8001/audit/requests
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 13 Nov 2018 17:52:41 GMT
Server: kong/3.2.2.0-dev-enterprise-edition
Transfer-Encoding: chunked
X-Kong-Admin-Request-ID: SpPaxLTkDNndzKaYiWuZl3xrxDUIiGRR

{
    "data": [
        {
            "client_ip": "127.0.0.1",
            "method": "POST",
            "path": "/consumers",
            "payload": "{\"username\": \"bob\"}",
            "request_id": "59fpTWlpUtHJ0qnAWBzQRHRDv7i5DwK2",
            "request_timestamp": 1581617463,
            "signature": null,
            "status": 201,
            "ttl": 2591995,
            "workspace": "fd51ce6e-59c0-4b6b-b991-aa708a9ff4d2"
        }
    ],
    "total": 1
}

Additionally, audit logs are generated to track the creation of the database entity:

http :8001/audit/objects

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 13 Nov 2018 17:53:27 GMT
Server: kong/3.2.2.0-dev-enterprise-edition
Transfer-Encoding: chunked
X-Kong-Admin-Request-ID: ZKra3QT0d3eJKl96jOUXYueLumo0ck8c

{
    "data": [
        {
            "dao_name": "consumers",
            "entity": "{\"created_at\":1542131418000,\"id\":\"16787ed7-d805-434a-9cec-5e5a3e5c9e4f\",\"username\":\"bob\",\"type\":0}",
            "entity_key": "16787ed7-d805-434a-9cec-5e5a3e5c9e4f",
            "expire": 1544723418009,
            "id": "7ebabee7-2b09-445d-bc1f-2092c4ddc4be",
            "operation": "create",
            "request_id": "59fpTWlpUtHJ0qnAWBzQRHRDv7i5DwK2",
            "request_timestamp": 1581617463,
        },
  ],
  "total": 1
}

Object audit entries contain information about the entity updated, including the entity body itself, its database primary key, and the type of operation performed (create, update, or delete). Note also the associated request_id field.

Limiting audit log generation

As with request audit logs, you may want to skip generation of audit logs for certain database tables. This is configurable via the audit_log_ignore_tables Kong config option:

audit_log_ignore_tables = consumers
# do not generate database audit logs for changes to the consumers table

Audit log retention

Audit log records are kept in the database for a duration defined by the audit_log_record_ttl Kong configuration property. Records in the database older than audit_log_record_ttl seconds are automatically purged.

PostgreSQL and Cassandra handle record deletion in different ways:

  • In Cassandra databases, record deletion is handled automatically via the Cassandra TTL mechanism.
  • In PostgreSQL databases, records are purged via the stored procedure that is executed on insert into the record database.

Therefore, request audit records may exist in the database longer than the configured TTL, if no new records are inserted to the audit table following the expiration timestamp.

Digital signatures

To provide non-repudiation, audit logs may be signed with a private RSA key. When enabled, a lexically sorted representation of each audit log entry is signed by the defined private key; the signature is stored in an additional field within the record itself. The public key should be stored elsewhere and can be used later to validate the signature of the record.

Setting up log signing

  1. Generate a private key via the openssl tool:

     openssl genrsa -out private.pem 2048
    
  2. Extract the public key for future audit verification:

     openssl rsa -in private.pem -outform PEM -pubout -out public.pem
    
  3. Configure Kong to sign audit log records:

     audit_log_signing_key = /path/to/private.pem
    
  4. Audit log entries will now contain the field signature:

     {
         "client_ip": "127.0.0.1",
         "method": "GET",
         "path": "/status",
         "payload": null,
         "rbac_user_id": "2e959b45-0053-41cc-9c2c-5458d0964331",
         "rbac_user_name": null,
         "request_id": "Ka2GeB13RkRIbMwBHw0xqe2EEfY0uZG0",
         "request_source": null,
         "request_timestamp": 1581617463,
         "signature": "l2LWYaRIHfXglFa5ehFc2j9ijfERazxisKVtJnYa+QUz2ckcytxfOLuA4VKEWHgY7cCLdn5C7uRJzE6es5V2SoOV59NOpskkr5lTt9kzao64UEw5UNOdeZYZKwyhG9Ge7IsxTK6haW0iG3a9dHqlKlwvnHZTbFM8TUV/umg8sJ1QJ/5ivXecbyHYtD5luKAI6oEgIdZPtQexRkwxlzvfR8lzeC/dDc2slSrjWRbBxNFlgfRKhDdVzVzgu8pEucgKggu67PKLkJ+bQEkxX1+Yg3czIpJyC3t6cgoggb0UNtBq1uUpswe0wdueKh6G5Gzz6XrmOjlv7zSz4gtVyEHZgg==",
         "status": 200,
         "ttl": 2591995,
         "workspace": "fd51ce6e-59c0-4b6b-b991-aa708a9ff4d2"
     }
    

Validating signatures

To verify record signatures, use the openssl utility, or other cryptographic tools that are capable of validating RSA digital signatures.

Signatures are generated using a 256-bit SHA digest. The following example demonstrates how to verify the audit log record shown above.

  1. First, store the record signature on disk after stripping the Base64 encoding:

     cat <<EOF | base64 -d > record_signature
     > l2LWYaRIHfXglFa5ehFc2j9ijfERazxisKVtJnYa+QUz2ckcytxfOLuA4VKEWHgY7cCLdn5C7uRJzE6es5V2SoOV59NOpskkr5lTt9kzao64UEw5UNOdeZYZKwyhG9Ge7IsxTK6haW0iG3a9dHqlKlwvnHZTbFM8TUV/umg8sJ1QJ/5ivXecbyHYtD5luKAI6oEgIdZPtQexRkwxlzvfR8lzeC/dDc2slSrjWRbBxNFlgfRKhDdVzVzgu8pEucgKggu67PKLkJ+bQEkxX1+Yg3czIpJyC3t6cgoggb0UNtBq1uUpswe0wdueKh6G5Gzz6XrmOjlv7zSz4gtVyEHZgg==
     > EOF
    
  2. Next, the audit record must be transformed into its canonical format used for signature generation. This transformation requires serializing the record into a string format that can be verified. The format is a lexically-sorted, pipe-delimited string of each audit log record part, without the signature, ttl, or expire fields. The following is a canonical implementation written in Lua:

     local cjson = require "cjson"
     local pl_sort = require "pl.tablex".sort
    
     local function serialize(data)
     local p = {}
    
     data.signature = nil
     data.expire = nil
     data.ttl = nil
    
     for k, v in pl_sort(data) do
         if type(v) == "table" then
         p[#p + 1] = serialize(v)
         elseif v ~= cjson.null then
         p[#p + 1] = v
         end
     end
    
     return p
     end
    
     table.concat(serialize(data), "|")
    

    For example, the canonical format of the audit record above is:

     cat canonical_record.txt
     127.0.0.1|1544724298663|GET|/status|Ka2GeB13RkRIbMwBHw0xqe2EEfY0uZG0|1542132298664|200|fd51ce6e-59c0-4b6b-b991-aa708a9ff4d2
    

    Ensure that the contents of the canonical record file on disk match the expected canonical record format exactly. The presence of any additional bytes, such as a trailing newline \n, will cause a validation failure in the next step.

  3. Once these two elements are in place, the signature can be verified:

     openssl dgst -sha256 -verify public.pem -signature record_signature canonical_record.txt
     Verified OK
    

More information

  • For Kong Gateway kong.conf options, see the Data & Admin Audit section of the Configuration Property Reference.
  • For the /audit API reference, see Audit Logs.
Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023