AWS Kinesis Firehose

Requirements

AWS Kinesis Firehose can only deliver data over HTTP. You will need to solve TLS termination by fronting Vector with a load balaner or configuring the tls.* options.

Example Configuration

AWS CloudWatch Subscription message

Config

Input

Output

1[sources.my_source_id]
2type = "aws_kinesis_firehose"
3address = "0.0.0.0:443"

1"```json\n\t{\n\t\t\"requestId\": \"ed1d787c-b9e2-4631-92dc-8e7c9d26d804\",\n\t\t\"timestamp\": 1600110760138,\n\t\t\"records\": [\n\t\t\t{\n\t\t\t\t\"data\": \"H4sIABk1bV8AA52TzW7bMBCE734KQ2db/JdI3QzETS8FAtg91UGgyOuEqCQq5Mqua+TdS8lu0hYNUpQHAdoZDcn9tKfJdJo0EEL5AOtjB0kxTa4W68Xdp+VqtbheJrPB4A4t+EFiv6yzVLuHa+/6blARAr5UV+ihbH4vh/4+VN52aF37wdYIPkTDlyhF8SrabFsOWhIrtz+Dlnto8dV3Gp9RstshXKhMi0xpqk3GpNJccpFRKYw0WvCM5kIbzrVWipm4VK55rrSk44HGHLTx/lg2wxVYRiljVGWGCvPiuPRn2O60Se6P8UKbpOBZrulsk2xLhCEjljYJk2QFHeGU04KxQqpCsumcSko3SfQ+uoBnn8pTJmjKWZYyI0axAXx021G++bweS5136CpXj8WP6/UNYek5ycMOPPhReETsQkHI4XBIO2/bynZlXXkXwryrS9w536TWkab0XwED6e/tU2/R9eGS9NTD5VgEvnWwtQikcu0e/AO0FYyu4HpfwR3Gf2R0Btza9qxgiUNUISiLr30AP7fbyMzu7OWA803ynIzdfJ69B1EZpoVhsWMRZ8a5UVJoRoUyUlDNspxzZWiEnOXiXYiSvQOR5TnN/xsiNalmKZcy5Yr/yfB6+RZD/gbDC0IbOx8wQrMhxGGYx4lBW5X1wJBLkpO981jWf6EXogvIrm+rYYrKOn4Hgbg4b439/s8cFeVvcNwBtHBkOdWvQIdRnTxPfgCXvyEgSQQAAA==\"\n\t\t\t}\n\t\t]\n\t}\n```"

1[
2  {
3    "log": {
4      "request_id": "ed1d787c-b9e2-4631-92dc-8e7c9d26d804",
5      "source_arn": "arn:aws:firehose:us-east-1:111111111111:deliverystream/test",
6      "timestamp": "2020-09-14T19:12:40.138Z",
7      "message": "{\"messageType\":\"DATA_MESSAGE\",\"owner\":\"111111111111\",\"logGroup\":\"test\",\"logStream\":\"test\",\"subscriptionFilters\":[\"Destination\"],\"logEvents\":[{\"id\":\"35683658089614582423604394983260738922885519999578275840\",\"timestamp\":1600110569039,\"message\":\"{\\\"bytes\\\":26780,\\\"datetime\\\":\\\"14/Sep/2020:11:45:41 -0400\\\",\\\"host\\\":\\\"157.130.216.193\\\",\\\"method\\\":\\\"PUT\\\",\\\"protocol\\\":\\\"HTTP/1.0\\\",\\\"referer\\\":\\\"https://www.principalcross-platform.io/markets/ubiquitous\\\",\\\"request\\\":\\\"/expedite/convergence\\\",\\\"source_type\\\":\\\"stdin\\\",\\\"status\\\":301,\\\"user-identifier\\\":\\\"-\\\"}\"},{\"id\":\"35683658089659183914001456229543810359430816722590236673\",\"timestamp\":1600110569041,\"message\":\"{\\\"bytes\\\":17707,\\\"datetime\\\":\\\"14/Sep/2020:11:45:41 -0400\\\",\\\"host\\\":\\\"109.81.244.252\\\",\\\"method\\\":\\\"GET\\\",\\\"protocol\\\":\\\"HTTP/2.0\\\",\\\"referer\\\":\\\"http://www.investormission-critical.io/24/7/vortals\\\",\\\"request\\\":\\\"/scale/functionalities/optimize\\\",\\\"source_type\\\":\\\"stdin\\\",\\\"status\\\":502,\\\"user-identifier\\\":\\\"feeney1708\\\"}\"}]}"
8    }
9  }
10]

Configuration Options

Required Options

address(required)

The address to listen for connections on

Type	Syntax	Default	Example
string	literal		["0.0.0.0:443","localhost:443"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["aws_kinesis_firehose"]

Advanced Options

access_key(optional)

AWS Kinesis Firehose can be configured to pass along an access key to authenticate requests. If configured, access_key should be set to the same value. If not specified, vector will treat all requests as authenticated.

Type	Syntax	Default	Example
string	literal		["A94A8FE5CCB19BA61C4C08"]

tls(optional)

Configures the TLS options for incoming connections.

Type	Syntax	Default	Example
hash	literal		[]

record_compression(optional)

The compression of records within the Firehose message.

Some services, like AWS CloudWatch Logs, will compress the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.

Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.

Type	Syntax	Default	Example
string	literal	text

How it Works

Forwarding CloudWatch Log events

This source is the recommended way to ingest logs from AWS CloudWatch logs via AWS CloudWatch Log subscriptions. To set this up:

Deploy vector with a publicly exposed HTTP endpoint using this source. You will likely also want to use the aws_cloudwatch_logs_subscription_parser transform to extract the log events. Make sure to set the access_key to secure this endpoint. Your configuration might look something like:

 [sources.firehose]
 # General
 type = "aws_kinesis_firehose"
 address = "127.0.0.1:9000"
 access_key = "secret"

 [transforms.cloudwatch]
 type = "aws_cloudwatch_logs_subscription_parser"
 inputs = ["firehose"]

 [sinks.console]
 type = "console"
 inputs = ["cloudwatch"]
 encoding.codec = "json"

Create a Kinesis Firewatch delivery stream in the region where the CloudWatch Logs groups exist that you want to ingest.
Set the stream to forward to your Vector instance via its HTTP Endpoint destination. Make sure to configure the same access_key you set earlier.
Setup a CloudWatch Logs subscription to forward the events to your delivery stream

State

This component is stateless, meaning its behavior is consistent across each input.

Transport Layer Security (TLS)

Vector uses OpenSSL for TLS protocols. You can adjust TLS behavior via the tls.* options.

Context

By default, the aws_kinesis_firehose source augments events with helpful context keys.