Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

acknowledgements

If offsets for consumer group do not exist, set them using this strategy. See the
[librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) for the `auto.offset.reset` option for further
clarification.

auto_offset_reset

A comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.

bootstrap_servers

The frequency that the consumer offsets are committed (written) to offset storage.

commit_interval_ms

Maximum time the broker may wait to fill the response.

fetch_wait_max_ms

The consumer group name to be used to consume events from Kafka.

group_id

The log field name to use for the Kafka headers.

headers_key

The log field name to use for the Kafka message key.

key_field

Advanced options. See [librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) for details.


librdkafka_options

The log field name to use for the Kafka offset.

offset_key

The log field name to use for the Kafka partition name.

partition_key

Options for SASL/SCRAM authentication support.

sasl

Enable SASL/SCRAM authentication to the remote (not supported on Windows at this time).

enabled

mechanism

The Kafka SASL/SCRAM authentication password.

password

The Kafka SASL/SCRAM authentication username.

username

The Kafka session timeout in milliseconds.

session_timeout_ms

socket_timeout_ms

Configures the TLS options for incoming connections.

Absolute path to an additional CA certificate file, in DER or PEM format (X.509), or an inline CA certificate in PEM format.

ca_file

Absolute path to a certificate file used to identify this connection, in DER or PEM format (X.509) or PKCS#12, or an inline certificate in PEM format. If this is set and is not a PKCS#12 archive, `key_file` must also be set.

crt_file

Enable TLS during connections to the remote.

Absolute path to a private key file used to identify this connection, in DER or PEM format (PKCS#8), or an inline private key in PEM format. If this is set, `crt_file` must also be set.

key_file

Pass phrase used to unlock the encrypted key file. This has no effect unless `key_file` is set.

key_pass

If `true` (the default), Vector will validate the TLS certificate of the remote host.

verify_certificate

The log field name to use for the Kafka topic.

topic_key

The Kafka topics names to read events from. Regex is supported if the topic begins with `^`.

topics

The component type. This is a required field for all components and tells Vector which component to use.

type

Kafka protocol

[Apache Kafka](https://kafka.apache.org/) is an open-source project for a distributed publish-subscribe messaging system rethought as a distributed commit log. Kafka stores messages in topics that are partitioned and replicated across multiple brokers in a cluster. Producers send messages to topics from which consumers read. These features make it an excellent candidate for durably storing logs and metrics data.

Kafka

By default, the `kafka` source augments events with helpful
context keys.

context

Context

The `kafka` sink uses [`librdkafka`](https://github.com/edenhill/librdkafka) under the hood. This
is a battle-tested, high performance, and reliable library that facilitates
communication with Kafka. As Vector produces static MUSL builds,
this dependency is packaged with Vector, meaning you do not need to install it.

librdkafka

This component is stateless, meaning its behavior is consistent across each input.

state

State

Vector uses [OpenSSL](https://www.openssl.org/) for TLS protocols. You can
adjust TLS behavior via the `tls.*` options.

Transport Layer Security (TLS)

kafka

message

The Kafka offset at the time the record was retrieved.

offset

The Kafka partition that the record came from.

partition

The timestamp encoded in the Kafka message or the current time if it cannot be fetched.

timestamp

The Kafka topic that the record came from.

topic

record

The total number of failures to update a Kafka consumer offset.

consumer_offset_updates_failed_total

The total number of failures to read a Kafka message.

events_failed_total

The number of events accepted by this component either from tagged
origin like file and uri, or cumulatively from other origins.

events_in_total

The total number of events emitted by this component.

events_out_total

Total number of message bytes (including framing) received from Kafka brokers.

kafka_consumed_messages_bytes_total

Total number of messages consumed, not including ignored messages (due to offset, etc), from Kafka brokers.

kafka_consumed_messages_total

Total number of message bytes (including framing, such as per-Message framing and MessageSet/batch framing) transmitted to Kafka brokers.

kafka_produced_messages_bytes_total

Total number of messages transmitted (produced) to Kafka brokers.

kafka_produced_messages_total

Current number of messages in producer queues.

kafka_queue_messages

Current total size of messages in producer queues.

kafka_queue_messages_bytes

Total number of bytes transmitted to Kafka brokers.

kafka_requests_bytes_total

Total number of requests sent to Kafka brokers.

kafka_requests_total

Total number of bytes received from Kafka brokers.

kafka_responses_bytes_total

Total number of responses received from Kafka brokers.

kafka_responses_total

The number of bytes processed by the component.

processed_bytes_total

The total number of events processed by this component.
This metric is deprecated in place of using
[`events_in_total`](/docs/reference/configuration/sources/internal_metrics/#events_in_total) and
[`events_out_total`](/docs/reference/configuration/sources/internal_metrics/#events_out_total) metrics.

processed_events_total

A ratio from 0 to 1 of the load on a component. A value of 0 would indicate a completely idle component that is simply waiting for input. A value of 1 would indicate a that is never idle. This value is updated every 5 seconds.

Kafka

Configuration Options

Required Options

bootstrap_servers(required)

group_id(required)

topics(required)

type(required)

Advanced Options

acknowledgements(optional)

auto_offset_reset(optional)

commit_interval_ms(optional)

fetch_wait_max_ms(optional)

key_field(optional)

topic_key(optional)

partition_key(optional)

offset_key(optional)

headers_key(optional)

librdkafka_options(optional)

sasl(optional)

session_timeout_ms(optional)

socket_timeout_ms(optional)

tls(optional)

How it Works

librdkafka

State

Transport Layer Security (TLS)

Context