Merge

CAUTION

This component has been deprecated in favor of the new reduce transform. The reduce transform provides a simple syntax for robust data merging. Let us know what you think!

Example Configuration

Default

Config

Input

Output

1[transforms.my_transform_id]
2type = "merge"

1[
2  {
3    "log": {
4      "message": "First",
5      "_partial": true,
6      "custom_string_field": "value1",
7      "custom_int_field": 1
8    }
9  },
10  {
11    "log": {
12      "message": "Second",
13      "_partial": true,
14      "custom_string_field": "value2",
15      "custom_int_field": 2
16    }
17  },
18  {
19    "log": {
20      "message": "Third",
21      "custom_string_field": "value3",
22      "custom_int_field": 3
23    }
24  }
25]

1{
2  "log": {
3    "message": "FirstSecondThird",
4    "custom_string_field": "value1",
5    "custom_int_field": 1
6  }
7}

With Merge Fields

Config

Input

Output

1[transforms.my_transform_id]
2type = "merge"
3fields = [ "message", "custom_string_field", "custom_int_field" ]

1[
2  {
3    "log": {
4      "message": "First",
5      "_partial": true,
6      "custom_string_field": "value1",
7      "custom_int_field": 1
8    }
9  },
10  {
11    "log": {
12      "message": "Second",
13      "_partial": true,
14      "custom_string_field": "value2",
15      "custom_int_field": 2
16    }
17  },
18  {
19    "log": {
20      "message": "Third",
21      "custom_string_field": "value3",
22      "custom_int_field": 3
23    }
24  }
25]

1{
2  "log": {
3    "message": "FirstSecondThird",
4    "custom_string_field": "value1value2value3",
5    "custom_int_field": 3
6  }
7}

Configuration Options

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

Type	Syntax	Default	Example
array	literal		["my-source-or-transform-id","prefix-*"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["merge"]

Advanced Options

fields(optional)

Fields to merge. The values of these fields will be merged into the first partial event. Fields not specified here will be ignored. Merging process takes the first partial event and the base, then it merges in the fields from each successive partial event, until a non-partial event arrives. Finally, the non-partial event fields are merged in, producing the resulting merged event.

Type	Syntax	Default	Example
array	literal	message	["message","parent.child"]

partial_event_marker_field(optional)

The field that indicates that the event is partial. A consequent stream of partial events along with the first non-partial event will be merged together.

Type	Syntax	Default	Example
string	literal	_partial	["_partial","parent.child"]

stream_discriminant_fields(optional)

An ordered list of fields to distinguish streams by. Each stream has a separate partial event merging state. Should be used to prevent events from unrelated sources from mixing together, as this affects partial event processing.

Type	Syntax	Default	Example
array	literal		["host","parent.child"]

How it Works

State

This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.