Lua

CAUTION

The lua transform is ~60% slower than the remap transform; we recommend that you use the remap transform whenever possible. The lua transform is designed solely for edge cases not covered by the remap transform and not as a go-to option. If the remap transform doesn't cover your use case, please open an issue and let us know.

Example Configuration

Add, rename, and remove log fields

Config

Input

Output

1[transforms.my_transform_id]
2type = "lua"
3version = "2"
4
5  [transforms.my_transform_id.hooks]
6  process = """
7function (event, emit)
8\t-- Add root level field
9\tevent.log.field = "new value"
10\t-- Add nested field
11\tevent.log.nested.field = "nested value"
12\t-- Rename field
13\tevent.log.renamed_field = event.log.field_to_rename
14\tevent.log.field_to_rename = nil
15\t-- Remove fields
16\tevent.log.field_to_remove = nil
17\temit(event)
18end"""

1{
2  "log": {
3    "field_to_rename": "old value",
4    "field_to_remove": "remove me"
5  }
6}

1{
2  "log": {
3    "field": "new value",
4    "nested": {
5      "field": "nested value"
6    },
7    "renamed_field": "old value"
8  }
9}

Add, rename, remove metric tags

Config

Input

Output

1[transforms.my_transform_id]
2type = "lua"
3version = "2"
4
5  [transforms.my_transform_id.hooks]
6  process = """
7function (event, emit)
8\t-- Add tag
9\tevent.metric.tags.tag = "new value"
10\t-- Rename tag
11\tevent.metric.tags.renamed_tag = event.log.tag_to_rename
12\tevent.metric.tags.tag_to_rename = nil
13\t-- Remove tag
14\tevent.metric.tags.tag_to_remove = nil
15\temit(event)
16end"""

1{
2  "metric": {
3    "kind": "incremental",
4    "name": "logins",
5    "counter": {
6      "value": 2
7    },
8    "tags": {
9      "tag_to_rename": "old value",
10      "tag_to_remove": "remove me"
11    }
12  }
13}

1{
2  "metric": {
3    "kind": "incremental",
4    "name": "logins",
5    "counter": {
6      "value": 2
7    },
8    "tags": {
9      "tag": "new value",
10      "renamed_tag": "old value"
11    }
12  }
13}

Drop an event

Config

Input

Output

1[transforms.my_transform_id]
2type = "lua"
3version = "2"
4
5  [transforms.my_transform_id.hooks]
6  process = """
7function (event, emit)
8\t-- Drop event entirely by not calling the `emit` function
9end"""

1{
2  "log": {
3    "field_to_rename": "old value",
4    "field_to_remove": "remove me"
5  }
6}

1null

Iterate over log fields

Config

Input

Output

1[transforms.my_transform_id]
2type = "lua"
3version = "2"
4
5  [transforms.my_transform_id.hooks]
6  process = """
7function (event, emit)
8\t-- Remove all fields where the value is "-"
9\tfor f, v in pairs(event) do
10\t\tif v == "-" then
11\t\t\tevent[f] = nil
12\t\tend
13\tend
14\temit(event)
15end"""

1{
2  "log": {
3    "value_to_remove": "-",
4    "value_to_keep": "keep"
5  }
6}

1{
2  "log": {
3    "value_to_keep": "keep"
4  }
5}

Parse timestamps

Config

Input

Output

1[transforms.my_transform_id]
2type = "lua"
3version = "2"
4
5  [transforms.my_transform_id.hooks]
6  source = """
7  timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
8  function parse_timestamp(str)
9\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
10\tlocal ms = 0
11\tif millis and millis ~= "" then
12\t\tms = tonumber(millis)
13\tend
14\treturn {
15\t\tyear    = tonumber(year),
16\t\tmonth   = tonumber(month),
17\t\tday     = tonumber(day),
18\t\thour    = tonumber(hour),
19\t\tmin     = tonumber(min),
20\t\tsec     = tonumber(sec),
21\t\tnanosec = ms * 1000000
22\t}
23  end
24  function process(event, emit)
25\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)
26\temit(event)
27  end"""
28  process = "process"

1{
2  "log": {
3    "timestamp_string": "2020-04-07 06:26:02.643"
4  }
5}

1{
2  "log": {
3    "timestamp_string": "2020-04-07 06:26:02.643",
4    "timestamp": "2020-04-07 06:26:02.643"
5  }
6}

Count the number of logs

Config

Input

Output

1[transforms.my_transform_id]
2type = "lua"
3source = """
4function init()
5\tcount = 0
6end
7function process()
8\tcount = count + 1
9end
10function timer_handler(emit)
11\temit(make_counter(count))
12\tcount = 0
13end
14function shutdown(emit)
15\temit(make_counter(count))
16end
17function make_counter(value)
18\treturn metric = {
19\t\tname = "event_counter",
20\t\tkind = "incremental",
21\t\ttimestamp = os.date("!*t"),
22\t\tcounter = {
23\t\t\tvalue = value
24\t\t}
25\t}
26end"""
27version = "2"
28
29  [[transforms.my_transform_id.timers]]
30  interval_seconds = 5
31  handler = "timer_handler"
32
33  [transforms.my_transform_id.hooks]
34  init = "init"
35  process = "process"
36  shutdown = "shutdown"

1{
2  "log": {}
3}

1{
2  "metric": {
3    "kind": "incremental",
4    "name": "event_counter",
5    "counter": {
6      "value": 1
7    },
8    "tags": {
9      "tag": "new value",
10      "renamed_tag": "old value"
11    }
12  }
13}

Configuration Options

Required Options

hooks(required)

Configures hooks handlers.

Type	Syntax	Default	Example
hash	literal		[]

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

Type	Syntax	Default	Example
array	literal		["my-source-or-transform-id","prefix-*"]

version(required)

Transform API version. Specifying this version ensures that Vector does not break backward compatibility.

Type	Syntax	Default	Example
string	literal		["1","2"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["lua"]

Advanced Options

search_dirs(optional)

A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector's configs.

Type	Syntax	Default	Example
array	literal		["/etc/vector/lua"]

source(optional)

The source which is evaluated when the transform is created.

Type	Syntax	Default	Example
string	literal		["function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t\tsyntax: \"literal\"\n\t\t}\n \t}\nend","-- external file with hooks and timers defined\nrequire('custom_module')"]

timers(optional)

Configures timers which are executed periodically at given interval.

Type	Syntax	Default	Example
array	literal		[]

How it Works

Event Data Model

The process hook takes an event as its first argument. Events are represented as tables in Lua and follow Vector's data model exactly. Please refer to Vector's data model reference for the event schema. How Vector's types map to Lua's type are covered below.

Learning Lua

In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.

State

This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.

Search Directories

Vector provides a search_dirs option that allows you to specify absolute paths that will be searched when using the Lua require function. If this option is not set, the directories of the configuration files will be used instead.