Remap

Example Configuration

Parse Syslog logs

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = ". |= parse_syslog!(.message)"

1{
2  "log": {
3    "message": "<102>1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
4  }
5}

1{
2  "log": {
3    "appname": "su",
4    "facility": "ntp",
5    "hostname": "vector-user.biz",
6    "message": "Something went wrong",
7    "msgid": "ID389",
8    "procid": 2666,
9    "severity": "info",
10    "timestamp": "2020-12-22T15:22:31.111Z",
11    "version": 1
12  }
13}

Parse key/value (logfmt) logs

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = ". = parse_key_value!(.message)"

1{
2  "log": {
3    "message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
4  }
5}

1{
2  "log": {
3    "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
4    "level": "info",
5    "msg": "Stopping all fetchers",
6    "tag#production": "stopping_fetchers",
7    "id": "ConsumerFetcherManager-1382721708341",
8    "module": "kafka.consumer.ConsumerFetcherManager"
9  }
10}

Parse custom logs

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = """
4. |= parse_regex!(.message, r'^(?P<timestamp>\\d+/\\d+/\\d+ \\d+:\\d+:\\d+ \\+\\d+) \\[(?P<severity>\\w+)\\] (?P<pid>\\d+)#(?P<tid>\\d+):(?: \\*(?P<connid>\\d+))? (?P<message>.*)$')
5
6# Coerce parsed fields
7.timestamp = parse_timestamp(.timestamp, "%Y/%m/%d %H:%M:%S %z") ?? now()
8.pid = to_int!(.pid)
9.tid = to_int!(.tid)
10
11# Extract structured data
12message_parts = split(.message, ", ", limit: 2)
13structured = parse_key_value(message_parts[1], key_value_delimiter: ":", field_delimiter: ",") ?? {}
14.message = message_parts[0]
15. = merge(., structured)"""

1{
2  "log": {
3    "message": "2021/01/20 06:39:15 +0000 [error] 17755#17755: *3569904 open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: \"GET /test.php HTTP/1.1\", host: \"yyy.yyy.yyy.yyy\""
4  }
5}

1{
2  "log": {
3    "timestamp": "2021-01-20T06:39:15Z",
4    "severity": "error",
5    "pid": 17755,
6    "tid": 17755,
7    "connid": "3569904",
8    "message": "open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory)",
9    "client": "xxx.xxx.xxx.xxx",
10    "server": "localhost",
11    "request": "GET /test.php HTTP/1.1",
12    "host": "yyy.yyy.yyy.yyy"
13  }
14}

Multiple parsing strategies

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = """
4structured =
5  parse_syslog(.message) ??
6  parse_common_log(.message) ??
7  parse_regex!(.message, r'^(?P<timestamp>\\d+/\\d+/\\d+ \\d+:\\d+:\\d+) \\[(?P<severity>\\w+)\\] (?P<pid>\\d+)#(?P<tid>\\d+):(?: \\*(?P<connid>\\d+))? (?P<message>.*)$')
8. = merge(., structured)"""

1{
2  "log": {
3    "message": "<102>1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
4  }
5}

1{
2  "log": {
3    "appname": "su",
4    "facility": "ntp",
5    "hostname": "vector-user.biz",
6    "message": "Something went wrong",
7    "msgid": "ID389",
8    "procid": 2666,
9    "severity": "info",
10    "timestamp": "2020-12-22T15:22:31.111Z",
11    "version": 1
12  }
13}

Modify metric tags

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = """
4.tags.environment = get_env_var!("ENV") # add
5.tags.hostname = del(.tags.host) # rename
6del(.tags.email)"""

1{
2  "metric": {
3    "kind": "incremental",
4    "name": "user_login_total",
5    "counter": {
6      "value": 102
7    },
8    "tags": {
9      "host": "my.host.com",
10      "instance_id": "abcd1234",
11      "email": "vic@vector.dev"
12    }
13  }
14}

1{
2  "metric": {
3    "kind": "incremental",
4    "name": "user_login_total",
5    "counter": {
6      "value": 102
7    },
8    "tags": {
9      "environment": "production",
10      "hostname": "my.host.com",
11      "instance_id": "abcd1234"
12    }
13  }
14}

Emitting multiple logs from JSON

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = ". = parse_json!(.message) # sets `.` to an array of objects"

1{
2  "log": {
3    "message": "[{\"message\": \"first_log\"}, {\"message\": \"second_log\"}]"
4  }
5}

1[
2  {
3    "log": {
4      "message": "first_log"
5    }
6  },
7  {
8    "log": {
9      "message": "second_log"
10    }
11  }
12]

Emitting multiple non-object logs from JSON

Config

Input

Output

1[transforms.my_transform_id]
2type = "remap"
3source = ". = parse_json!(.message) # sets `.` to an array"

1{
2  "log": {
3    "message": "[5, true, \"hello\"]"
4  }
5}

1[
2  {
3    "log": {
4      "message": 5
5    }
6  },
7  {
8    "log": {
9      "message": true
10    }
11  },
12  {
13    "log": {
14      "message": "hello"
15    }
16  }
17]

Configuration Options

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

Type	Syntax	Default	Example
array	literal		["my-source-or-transform-id","prefix-*"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["remap"]

Advanced Options

source(optional)

The Vector Remap Language (VRL) program to execute for each event.

Required if file is missing.

Type	Syntax	Default	Example
string	remap_program		[". = parse_json!(.message)\n.new_field = \"new value\"\n.status = to_int!(.status)\n.duration = parse_duration!(.duration, \"s\")\n.new_name = del(.old_name)"]

file(optional)

File path to the Vector Remap Language (VRL) program to execute for each event.

If a relative path is provided, its root is the current working directory.

Required if source is missing.

Type	Syntax	Default	Example
string	literal		["./my/program.vrl"]

drop_on_error(optional)

Drop the event if the VRL program returns an error at runtime.

Type	Syntax	Default	Example
bool

drop_on_abort(optional)

Drop the event if the VRL program is manually aborted through the abort statement.

Type	Syntax	Default	Example
bool

How it Works

Vector Remap Language

The Vector Remap Language (VRL) is a restrictive, fast, and safe language we designed specifically for mapping observability data. It avoids the need to chain together many fundamental Vector transforms to accomplish rudimentary reshaping of data.

The intent is to offer the same robustness of full language runtime (ex: Lua) without paying the performance or safety penalty.

Learn more about Vector's Remap Language in the Vector Remap Language reference.

Lazy Event Mutation

When you make changes to an event through VRL's path assignment syntax, the change isn't immediately applied to the actual event. If the program fails to run to completion, any changes made until that point are dropped and the event is kept in its original state.

If you want to make sure your event is changed as expected, you have to rewrite your program to never fail at runtime (the compiler can help you with this).

Alternatively, if you want to ignore/drop events that caused the program to fail, you can set the drop_on_error configuration value to true.

Learn more about runtime errors in the Vector Remap Language reference.

State

This component is stateless, meaning its behavior is consistent across each input.

Emitting multiple log events

Multiple log events can be emitted from remap by assigning an array to the root path .. One log event is emitted for each input element of the array.

If any of the array elements isn't an object, a log event is created that uses the element's value as the message key. For example, 123 is emitted as:

{
  "message": 123
}