Key-value Parser

CAUTION

This transform has been deprecated in favor of the remap transform, which enables you to use Vector Remap Language (VRL for short) to create transform logic of any degree of complexity. The examples below show how you can use VRL to replace this transform's functionality.

.message = parse_key_value(.message)
TIP

It is likely that the key_value transform will replace the logfmt_parser transform in the future since it offers a more flexible superset of that transform.

Example Configuration

Firewall log message

Config
Input
Output
1[transforms.my_transform_id]

2type = "key_value_parser"

3trim_value = '"'

4trim_key = '"'

5target_field = "data"

6separator = ";"

7field_split = ":"

8field = "message"

1{

2  "log": {

3    "message": "action:\"Accept\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"eth2-05\"; logid:\"6\"; loguid:\"{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}\";"

4  }

5}

1{

2  "log": {

3    "message": "action:\"Accept\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"eth2-05\"; logid:\"6\"; loguid:\"{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}\";",

4    "data": {

5      "action": "Accept",

6      "flags": "802832",

7      "ifdir": "inbound",

8      "ifname": "eth2-05",

9      "logid": "6",

10      "loguid": "{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}"

11    }

12  }

13}

Configuration Options

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["key_value_parser"]

Advanced Options

drop_field(optional)

If true will drop the specified field after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The log field containing key/value pairs to parse. Must be a string value.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child","array[0]"]
field_split(optional)

The character(s) to split a key/value pair on which results in a new field with an associated value. Must be a string value.

TypeSyntaxDefaultExample
stringliteral=[":","="]
overwrite_target(optional)

If target_field is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true.

TypeSyntaxDefaultExample
bool
separator(optional)

The character(s) that separate key/value pairs. Must be a string value.

TypeSyntaxDefaultExample
stringliteral[whitespace][",",";","|"]
target_field(optional)

If this setting is present, the parsed JSON will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.

TypeSyntaxDefaultExample
stringliteral["root_field","parent.child"]
trim_key(optional)

Removes characters from the beginning and end of a key until a character that is not listed. ex: <key>=value would result in key: value with this option set to <>.

TypeSyntaxDefaultExample
stringliteral["<>","{}"]
trim_value(optional)

Removes characters from the beginning and end of a value until a character that is not listed. ex: key=<<>value>> would result in key: value with this option set to <>.

TypeSyntaxDefaultExample
stringliteral["<>","{}"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]

How it Works

Description

The Key Value Parser accepts structured data that can be split on a character, or group of characters, and extracts it into a json object (dictionary) of key/value pairs. The separator option allows you to define the character(s) to perform the initial splitting of the message into pairs. The field_split option allows you to define the character(s) which split the key from the value.

State

This component is stateless, meaning its behavior is consistent across each input.