Storing Custom Entities

While not all plugins need it, your plugin might need to store more than its configuration in the database. In that case, Kong provides you with an abstraction on top of its primary data stores which allows you to store custom entities.

As explained in the previous chapter, Kong interacts with the model layer through classes we refer to as “DAOs”, and available on a singleton often referred to as the “DAO Factory”. This chapter will explain how to provide an abstraction for your own entities.

Modules

kong.plugins.<plugin_name>.daos
kong.plugins.<plugin_name>.migrations.init
kong.plugins.<plugin_name>.migrations.000_base_<plugin_name>
kong.plugins.<plugin_name>.migrations.001_<from-version>_to_<to_version>
kong.plugins.<plugin_name>.migrations.002_<from-version>_to_<to_version>

Create the migrations folder

Once you have defined your model, you must create your migration modules which will be executed by Kong to create the table in which your records of your entity will be stored.

If your plugin doesn’t have it already, you should add a <plugin_name>/migrations folder to it. If there is no init.lua file inside already, you should create one. This is where all the migrations for your plugin will be referenced.

The initial version of your migrations/init.lua file will point to a single migration.

In this case we have called it 000_base_my_plugin.

-- `migrations/init.lua`
return {
  "000_base_my_plugin",
}

This means that there will be a file in <plugin_name>/migrations/000_base_my_plugin.lua containing the initial migrations. We’ll see how this is done in a minute.

Add a new migration to an existing plugin

Sometimes it is necessary to introduce changes after a version of a plugin has already been released. A new functionality might be needed. A database table row might need changing.

When this happens, you must create a new migrations file. You must not modify the existing migration files once they are published (you can still make them more robust and bulletproof if you want, e.g. always try to write the migrations reentrant).

While there is no strict rule for naming your migration files, there is a convention that the initial one is prefixed by 000, the next one by 001, and so on.

Following with our previous example, if we wanted to release a new version of the plugin with changes in the database (for example, a table was needed called foo) we would insert it by adding a file called <plugin_name>/migrations/001_100_to_110.lua, and referencing it on the migrations init file like so (where 100 is the previous version of the plugin 1.0.0 and 110 is the version to which plugin is migrated to 1.1.0:

-- `<plugin_name>/migrations/init.lua`
return {
  "000_base_my_plugin",
  "001_100_to_110",
}

Migration file syntax

A migration file is a Lua file which returns a table with the following structure:

-- `<plugin_name>/migrations/000_base_my_plugin.lua`
return {
  postgres = {
    up = [[
      CREATE TABLE IF NOT EXISTS "my_plugin_table" (
        "id"           UUID                         PRIMARY KEY,
        "created_at"   TIMESTAMP WITHOUT TIME ZONE,
        "col1"         TEXT
      );

      DO $$
      BEGIN
        CREATE INDEX IF NOT EXISTS "my_plugin_table_col1"
                                ON "my_plugin_table" ("col1");
      EXCEPTION WHEN UNDEFINED_COLUMN THEN
        -- Do nothing, accept existing state
      END$$;
    ]],
  }
}

-- `<plugin_name>/migrations/001_100_to_110.lua`
return {
  postgres = {
    up = [[
      DO $$
      BEGIN
        ALTER TABLE IF EXISTS ONLY "my_plugin_table" ADD "cache_key" TEXT UNIQUE;
      EXCEPTION WHEN DUPLICATE_COLUMN THEN
        -- Do nothing, accept existing state
      END$$;
    ]],
    teardown = function(connector, helpers)
      assert(connector:connect_migrations())
      assert(connector:query([[
        DO $$
        BEGIN
          ALTER TABLE IF EXISTS ONLY "my_plugin_table" DROP "col1";
        EXCEPTION WHEN UNDEFINED_COLUMN THEN
          -- Do nothing, accept existing state
        END$$;
      ]]))
    end,
  }
}

Each strategy section has two parts, up and teardown.

• up is an optional string of raw SQL statements. Those statements are executed when kong migrations up is executed.

  We recommend that all the non-destructive operations, such as creation of new tables and addition of new records, are done on the up sections.

• teardown is an optional Lua function, which takes a connector parameter. The connector can invoke the query method to execute SQL queries. Teardown is triggered by kong migrations finish.

  We recommend that destructive operations, such as removal of data, changing row types, and insertion of new data, are done on the teardown sections.

All SQL statements should be written so that they are as reentrant as possible. For example, use DROP TABLE IF EXISTS instead of DROP TABLE, CREATE INDEX IF NOT EXIST instead of CREATE INDEX, and so on. If a migration fails for some reason, it is expected that the first attempt at fixing the problem will be simply re-running the migrations.

If your schema uses a unique constraint, you must set this constraint in the migrations for PostgreSQL.

To see a real-life example, give a look at the Key-Auth plugin migrations.

Define a schema

The first step to using custom entities in a custom plugin is defining one or more schemas.

A schema is a Lua table which describes entities. There’s structural information like how are the different fields of the entity named and what are their types, which is similar to the fields describing your plugin configuration). Compared to plugin configuration schemas, custom entity schemas require additional metadata (e.g. which field, or fields, constitute the entities’ primary key).

Schemas are to be defined in a module named:

kong.plugins.<plugin_name>.daos

Meaning that there should be a file called <plugin_name>/daos.lua inside your plugin folder. The daos.lua file should return a table containing one or more schemas. For example:

-- daos.lua
local typedefs = require "kong.db.schema.typedefs"


return {
  -- this plugin only results in one custom DAO, named `keyauth_credentials`:
  {
    name                  = "keyauth_credentials", -- the actual table in the database
    endpoint_key          = "key",
    primary_key           = { "id" },
    cache_key             = { "key" },
    generate_admin_api    = true,
    admin_api_name        = "key-auths",
    admin_api_nested_name = "key-auth",
    fields = {
      {
        -- a value to be inserted by the DAO itself
        -- (think of serial id and the uniqueness of such required here)
        id = typedefs.uuid,
      },
      {
        -- also inserted by the DAO itself
        created_at = typedefs.auto_timestamp_s,
      },
      {
        -- a foreign key to a consumer's id
        consumer = {
          type      = "foreign",
          reference = "consumers",
          default   = ngx.null,
          on_delete = "cascade",
        },
      },
      {
        -- a unique API key
        key = {
          type      = "string",
          required  = false,
          unique    = true,
          auto      = true,
        },
      },
    },
  },
}

This example daos.lua file introduces a single schema called keyauth_credentials.

Here is a description of some top-level properties:

Many field attributes encode validation rules. When attempting to insert or update entities using the DAO, these validations will be checked, and an error returned if the provided input doesn’t conform to them.

The typedefs variable (obtained by requiring kong.db.schema.typedefs) is a table containing a lot of useful type definitions and aliases, including typedefs.uuid, the most usual type for the primary key, and typedefs.auto_timestamp_s, for created_at fields. It is used extensively when defining fields.

Here’s a non-exhaustive explanation of some of the field attributes available:

To learn more about schemas, see:

• The source code of typedefs.lua to get an idea of what’s provided there by default.
• The Core Schemas to see examples of some other field attributes not discussed here.
• All the daos.lua files for embedded plugins, especially the key-auth one, which was used for this guide as an example.

The custom DAO

The schemas are not used directly to interact with the database. Instead, a DAO is built for each valid schema. A DAO takes the name of the schema it wraps, and is accessible through the kong.db interface.

For the example schema above, the DAO generated would be available for plugins via kong.db.keyauth_credentials.

Select an entity

local entity, err, err_t = kong.db.<name>:select(primary_key)

Attempts to find an entity in the database and return it. Three things can happen:

• The entity was found. In this case, it is returned as a regular Lua table.
• An error occurred - for example the connection with the database was lost. In that case the first returned value will be nil, the second one will be a string describing the error, and the last one will be the same error in table form.
• An error does not occur but the entity is not found. Then the function will just return nil, with no error.

Example of usage:

local entity, err = kong.db.keyauth_credentials:select({
  id = "c77c50d2-5947-4904-9f37-fa36182a71a9"
})

if err then
  kong.log.err("Error when selecting keyauth credential: " .. err)
  return nil
end

if not entity then
  kong.log.err("Could not find credential.")
  return nil
end

Iterate over all the entities

for entity, err in kong.db.<name>:each(entities_per_page) do
  if err then
    ...
  end
  ...
end

This method efficiently iterates over all the entities in the database by making paginated requests. The entities_per_page parameter, which defaults to 100, controls how many entities per page are returned.

On each iteration, a new entity will be returned or, if there is any error, the err variable will be filled up with an error. The recommended way to iterate is checking err first, and otherwise assume that entity is present.

Example of usage:

for credential, err in kong.db.keyauth_credentials:each(1000) do
  if err then
    kong.log.err("Error when iterating over keyauth credentials: " .. err)
    return nil
  end

  kong.log("id: " .. credential.id)
end

This example iterates over the credentials in pages of 1000 items, logging their ids unless an error happens.

Insert an entity

local entity, err, err_t = kong.db.<name>:insert(<values>)

Inserts an entity in the database, and returns a copy of the inserted entity, or nil, an error message (a string) and a table describing the error in table form.

When the insert is successful, the returned entity contains the extra values produced by default and auto.

The following example uses the keyauth_credentials DAO to insert a credential for a given Consumer, setting its key to "secret". Notice the syntax for referencing foreign keys.

local entity, err = kong.db.keyauth_credentials:insert({
  consumer = { id = "c77c50d2-5947-4904-9f37-fa36182a71a9" },
  key = "secret",
})

if not entity then
  kong.log.err("Error when inserting keyauth credential: " .. err)
  return nil
end

The returned entity, assuming no error happened will have auto-filled fields, like id and created_at.

Update an entity

local entity, err, err_t = kong.db.<name>:update(primary_key, <values>)

Updates an existing entity, provided it can be found using the provided primary key and a set of values.

The returned entity will be the entity after the update takes place, or nil + an error message + an error table.

The following example modifies the key field of an existing credential given the credential’s id:

local entity, err = kong.db.keyauth_credentials:update(
  { id = "2b6a2022-770a-49df-874d-11e2bf2634f5" },
  { key = "updated_secret" },
)

if not entity then
  kong.log.err("Error when updating keyauth credential: " .. err)
  return nil
end

Notice how the syntax for specifying a primary key is similar to the one used to specify a foreign key.

Upsert an entity

local entity, err, err_t = kong.db.<name>:upsert(primary_key, <values>)

upsert is a mixture of insert and update:

• When the provided primary_key identifies an existing entity, it works like update.
• When the provided primary_key does not identify an existing entity, it works like insert

Given this code:

local entity, err = kong.db.keyauth_credentials:upsert(
  { id = "2b6a2022-770a-49df-874d-11e2bf2634f5" },
  { consumer = { id = "a96145fb-d71e-4c88-8a5a-2c8b1947534c" } }
)

if not entity then
  kong.log.err("Error when upserting keyauth credential: " .. err)
  return nil
end

Two things can happen:

• If a credential with id 2b6a2022-770a-49df-874d-11e2bf2634f5 exists, then this code will attempt to set its Consumer to the provided one.
• If the credential does not exist, then this code is attempting to create a new credential, with the given id and Consumer.

Delete an entity

local ok, err, err_t = kong.db.<name>:delete(primary_key)

Attempts to delete the entity identified by primary_key. It returns true if the entity doesn’t exist after calling this method, or nil + error + error table if an error is detected.

Notice that calling delete will succeed if the entity didn’t exist before calling it. This is for performance reasons - we want to avoid doing a read-before-delete if we can avoid it. If you want to do this check, you must do it manually, by checking with select before invoking delete.

Example:

local ok, err = kong.db.keyauth_credentials:delete({
  id = "2b6a2022-770a-49df-874d-11e2bf2634f5"
})

if not ok then
  kong.log.err("Error when deleting keyauth credential: " .. err)
  return nil
end

Cache custom entities

Sometimes custom entities are required on every request/response, which in turn triggers a query on the data stores every time. This is very inefficient because querying the data stores adds latency and slows the request/response down, and the resulting increased load on the data stores could affect the data stores performance itself and, in turn, other Kong nodes.

When a custom entity is required on every request/response it is good practice to cache it in-memory by leveraging the in-memory cache API provided by Kong.

The next chapter will focus on caching custom entities, and invalidating them when they change in the data stores: Caching custom entities.