AWS CloudWatch Logs Subscription Parser

CAUTION

This transform has been deprecated in favor of the remap transform, which enables you to use Vector Remap Language (VRL for short) to create transform logic of any degree of complexity. The examples below show how you can use VRL to replace this transform's functionality.

.message = parse_aws_cloudwatch_log_subscription_message(.message)

Example Configuration

Default

Config

Input

Output

1[transforms.my_transform_id]
2type = "aws_cloudwatch_logs_subscription_parser"
3field = "message"

1{
2  "log": {
3    "message": "\t{\n\t  \"messageType\": \"DATA_MESSAGE\",\n\t  \"owner\": \"111111111111\",\n\t  \"logGroup\": \"test\",\n\t  \"logStream\": \"test\",\n\t  \"subscriptionFilters\": [\n\t\t\"Destination\"\n\t  ],\n\t  \"logEvents\": [\n\t\t{\n\t\t  \"id\": \"35683658089614582423604394983260738922885519999578275840\",\n\t\t  \"timestamp\": 1600110569039,\n\t\t  \"message\": \"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}\"\n\t\t},\n\t\t{\n\t\t  \"id\": \"35683658089659183914001456229543810359430816722590236673\",\n\t\t  \"timestamp\": 1600110569041,\n\t\t  \"message\": \"{\"bytes\":17707,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"109.81.244.252\",\"method\":\"GET\",\"protocol\":\"HTTP/2.0\",\"referer\":\"http://www.investormission-critical.io/24/7/vortals\",\"request\":\"/scale/functionalities/optimize\",\"source_type\":\"stdin\",\"status\":502,\"user-identifier\":\"feeney1708\"}\"\n\t\t}\n\t  ]\n\t}"
4  }
5}

1{
2  "log": {
3    "id": "35683658089614582423604394983260738922885519999578275840",
4    "log_group": "test",
5    "log_stream": "test",
6    "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-latform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
7    "owner": "111111111111",
8    "timestamp": "2020-09-14T19:09:29.039Z",
9    "subscription_filters": [
10      "Destination"
11    ]
12  }
13}

Configuration Options

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

Type	Syntax	Default	Example
array	literal		["my-source-or-transform-id","prefix-*"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["aws_cloudwatch_logs_subscription_parser"]

Advanced Options

field(optional)

The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.

Type	Syntax	Default	Example
string	literal	message

How it Works

Structured Log Events

Note that the events themselves are not parsed. If they are structured data, you will typically want to pass them through a parsing transform.

State

This component is stateless, meaning its behavior is consistent across each input.