Logfmt Parser

CAUTION

This transform has been deprecated in favor of the remap transform, which enables you to use Vector Remap Language (VRL for short) to create transform logic of any degree of complexity. The examples below show how you can use VRL to replace this transform's functionality.

.message = parse_key_value(.message)

Example Configuration

Heroku Router Log

Config

Input

Output

1[transforms.my_transform_id]
2type = "logfmt_parser"
3drop_field = true
4field = "message"
5
6  [transforms.my_transform_id.types]
7  bytes = "int"
8  status = "int"

1{
2  "log": {
3    "message": "at=info method=GET path=/ host=myapp.herokuapp.com request_id=8601b555-6a83-4c12-8269-97c8e32cdb22 fwd=\"204.204.204.204\" dyno=web.1 connect=1ms service=18ms status=200 bytes=13 tls_version=tls1.1 protocol=http"
4  }
5}

1{
2  "log": {
3    "at": "info",
4    "method": "GET",
5    "path": "/",
6    "host": "myapp.herokuapp.com",
7    "request_id": "8601b555-6a83-4c12-8269-97c8e32cdb22",
8    "fwd": "204.204.204.204",
9    "dyno": "web.1",
10    "connect": "1ms",
11    "service": "18ms",
12    "status": 200,
13    "bytes": 13,
14    "tls_version": "tls1.1",
15    "protocol": "http"
16  }
17}

Loosely Structured

Config

Input

Output

1[transforms.my_transform_id]
2type = "logfmt_parser"
3drop_field = false
4field = "message"
5
6  [transforms.my_transform_id.types]
7  status = "int"

1{
2  "log": {
3    "message": "info | Sent 200 in 54.2ms duration=54.2ms status=200"
4  }
5}

1{
2  "log": {
3    "message": "info | Sent 200 in 54.2ms duration=54.2ms status=200",
4    "duration": "54.2ms",
5    "status": 200
6  }
7}

Configuration Options

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

Type	Syntax	Default	Example
array	literal		["my-source-or-transform-id","prefix-*"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["logfmt_parser"]

Advanced Options

drop_field(optional)

If the specified field should be dropped (removed) after parsing.

Type	Syntax	Default	Example
bool

field(optional)

The log field to parse.

Type	Syntax	Default	Example
string	literal	message	["message","parent.child","array[0]"]

timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

Type	Syntax	Default	Example
string	literal	local	["local","America/NewYork","EST5EDT"]

types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

One of the built-in-formats listed in the Timestamp Formats table below.
The time format specifiers from Rust's chrono library.

Types

array
bool
bytes
float
int
map
null
timestamp (see the table below for formats)

Timestamp Formats

Format	Description	Example
`%F %T`	`YYYY-MM-DD HH:MM:SS`	`2020-12-01 02:37:54`
`%v %T`	`DD-Mmm-YYYY HH:MM:SS`	`01-Dec-2020 02:37:54`
`%FT%T`	ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone	`2020-12-01T02:37:54`
`%a, %d %b %Y %T`	RFC 822/2822 without time zone	`Tue, 01 Dec 2020 02:37:54`
`%a %d %b %T %Y`	`date` command output without time zone	`Tue 01 Dec 02:37:54 2020`
`%a %b %e %T %Y`	ctime format	`Tue Dec 1 02:37:54 2020`
`%s`	UNIX timestamp	`1606790274`
`%FT%TZ`	ISO 8601/RFC 3339 UTC	`2020-12-01T09:37:54Z`
`%+`	ISO 8601/RFC 3339 UTC with time zone	`2020-12-01T02:37:54-07:00`
`%a %d %b %T %Z %Y`	`date` command output with time zone	`Tue 01 Dec 02:37:54 PST 2020`
`%a %d %b %T %z %Y`	`date` command output with numeric time zone	`Tue 01 Dec 02:37:54 -0700 2020`
`%a %d %b %T %#z %Y`	`date` command output with numeric time zone (minutes can be missing or present)	`Tue 01 Dec 02:37:54 -07 2020`

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

Type	Syntax	Default	Example
hash			[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp\|%F","timestamp_custom":"timestamp\|%a %b %e %T %Y","timestamp_unix":"timestamp\|%F %T","parent":{"child":"int"}}]

How it Works

Key/Value Parsing

This transform can be used for key/value parsing. Logfmt refers to a loosely defined spec that parses a key/value pair delimited by a = character. This section, and it's keywords, is primarily added to assist users in finding this transform for these terms.

Quoting Values

Values can be quoted to capture spaces, and quotes can be escaped with \. For example, this

key1="value with spaces" key2="value with spaces and \""

Would result in the following log event:

{
  "key1": "value with spaces",
  "key2": "value with spaces and \""
}

State

This component is stateless, meaning its behavior is consistent across each input.

Format Specification

Logfmt is, unfortunately, a very loosely defined format. There is no official specification for the format and Vector makes a best effort to parse key/value pairs delimited with a =. It works by splitting the field's value on non-quoted whitespace and then splitting each token by a non-quoted = character. This makes the parsing process somewhat flexible in that the string does not need to be strictly formatted.

For example, the following log line:

{
  "message": "Hello world duration=2s user-agent="Firefox/47.3 Mozilla/5.0""
}

Will be successfully parsed into:

{
  "message": "Hello world duration=2s user-agent="Firefox/47.3 Mozilla/5.0"",
  "duration": "2s",
  "user-agent": "Firefox/47.3 Mozilla/5.0"
}