Filedaemonsidecar

  • Best Effort
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

include(required)

Array of file patterns to include. Globbing is supported.

TypeSyntaxDefaultExample
arrayliteral["/var/log/**/*.log"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["file"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
exclude(optional)

Array of file patterns to exclude. Globbing is supported.Takes precedence over the include option.

TypeSyntaxDefaultExample
arrayliteral["/var/log/binary-file.log"]
file_key(optional)

The key name added to each event with the full path of the file.

TypeSyntaxDefaultExample
stringliteralfile["file"]
fingerprint(optional)

Configuration for how the file source should identify files.

TypeSyntaxDefaultExample
hashliteral[]
glob_minimum_cooldown_ms(optional)

Delay between file discovery calls. This controls the interval at which Vector searches for files. Higher value result in greater chances of some short living files being missed between searches, but lower value increases the performance impact of file discovery.

TypeSyntaxDefaultExample
uint1000
host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

TypeSyntaxDefaultExample
stringliteralhost
ignore_not_found(optional)

Ignore missing files when fingerprinting. This may be useful when used with source directories containing dangling symlinks.

TypeSyntaxDefaultExample
bool
ignore_older_secs(optional)

Ignore files with a data modification date older than the specified number of seconds.

TypeSyntaxDefaultExample
uint[600]
line_delimiter(optional)

String sequence used to separate one file line from another

TypeSyntaxDefaultExample
stringliteral ["\r\n"]
max_line_bytes(optional)

The maximum number of a bytes a line can contain before being discarded. This protects against malformed lines or tailing incorrect files.

TypeSyntaxDefaultExample
uint102400
max_read_bytes(optional)

An approximate limit on the amount of data read from a single file at a given time.

TypeSyntaxDefaultExample
uint[2048]
oldest_first(optional)

Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from younger files.

TypeSyntaxDefaultExample
bool
remove_after_secs(optional)

Timeout from reaching eof after which file will be removed from filesystem, unless new data is written in the meantime. If not specified, files will not be removed.

TypeSyntaxDefaultExample
uint[0,5,60]
read_from(optional)

In the absence of a checkpoint, this setting tells Vector where to start reading files that are present at startup.

TypeSyntaxDefaultExample
stringliteralbeginning
multiline(optional)

Multiline parsing configuration. If not specified, multiline parsing is disabled.

TypeSyntaxDefaultExample
hashregex[]
data_dir(optional)

The directory used to persist file checkpoint positions. By default, the global data_dir option is used. Please make sure the Vector project has write permissions to this dir.

TypeSyntaxDefaultExample
stringfile_system_path["/var/lib/vector"]
encoding(optional)

Configures the encoding specific source behavior.

TypeSyntaxDefaultExample
hashliteral[]
ignore_checkpoints(optional)

This causes Vector to ignore existing checkpoints when determining where to start reading a file. Checkpoints are still written normally.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["file"]
3include = ["/var/log/**/*.log"]

Apache HTTP Server (HTTPD) Metricsdaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

endpoints(required)

mod_status endpoints to scrape metrics from.

TypeSyntaxDefaultExample
arrayliteral["http://localhost:8080/server-status/?auto"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["apache_metrics"]

Advanced Options

scrape_interval_secs(optional)

The interval between scrapes.

TypeSyntaxDefaultExample
uint15
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
namespace(optional)

The namespace of the metric. Disabled if empty.

TypeSyntaxDefaultExample
stringliteralapache
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["apache_metrics"]
3endpoints = ["http://localhost:8080/server-status/?auto"]

Prometheus Remote Writedaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to accept connections on. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:9090"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["prometheus_remote_write"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
auth(optional)

Options for HTTP Basic Authentication.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["prometheus_remote_write"]
3address = ["0.0.0.0:9090"]

AWS Kinesis Firehoseaggregator

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to listen for connections on

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:443","localhost:443"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_kinesis_firehose"]

Advanced Options

access_key(optional)

AWS Kinesis Firehose can be configured to pass along an access key to authenticate requests. If configured, access_key should be set to the same value. If not specified, vector will treat all requests as authenticated.

TypeSyntaxDefaultExample
stringliteral["A94A8FE5CCB19BA61C4C08"]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
record_compression(optional)

The compression of records within the Firehose message.

Some services, like AWS CloudWatch Logs, will compress the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.

Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.

TypeSyntaxDefaultExample
stringliteraltext
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_kinesis_firehose"]
3address = ["0.0.0.0:443","localhost:443"]

AWS S3aggregator

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_s3"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
strategy(optional)

The strategy to use to consume objects from AWS S3.

TypeSyntaxDefaultExample
stringliteralsqs
compression(optional)

The compression format of the S3 objects..

TypeSyntaxDefaultExample
stringliteraltext
multiline(optional)

Multiline parsing configuration. If not specified, multiline parsing is disabled.

TypeSyntaxDefaultExample
hashregex[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
sqs(optional)

SQS strategy options. Required if strategy=sqs.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_s3"]
3region = ["us-east-1"]

Datadog Agentaggregatorsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to accept connections on. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:80","localhost:80"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["datadog_agent"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
store_api_key(optional)

When incoming events contain a Datadog API key, if this setting is set to true the key will kept in the event metadata and will be used if the event is sent to a Datadog sink.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["datadog_agent"]
3address = ["0.0.0.0:80","localhost:80"]

AWS ECS Metricssidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_ecs_metrics"]

Advanced Options

endpoint(optional)

Base URI of the task metadata endpoint. If empty, the URI will be automatically discovered based on the latest version detected. The version 2 endpoint base URI is 169.254.170.2/v2/. The version 3 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI. The version 4 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI_V4.

TypeSyntaxDefaultExample
stringliteral${ECS_CONTAINER_METADATA_URI_V4}
namespace(optional)

The namespace of the metric. Disabled if empty.

TypeSyntaxDefaultExample
stringliteralawsecs
scrape_interval_secs(optional)

The interval between scrapes, in seconds.

TypeSyntaxDefaultExample
uint15
version(optional)

The version of the metadata endpoint. If empty, the version is automatically discovered based on environment variables.

TypeSyntaxDefaultExample
stringliteralv4
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_ecs_metrics"]

Dockerdaemon

  • Best Effort
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["docker_logs"]

Advanced Options

docker_host(optional)

The Docker host to connect to. Use an HTTPS URL to enable TLS encryption. If absent, Vector will try to use DOCKER_HOST enviroment variable. If DOCKER_HOST is also absent, Vector will use default Docker local socket (/var/run/docker.sock on Unix flatforms, //./pipe/docker_engine on Windows).

TypeSyntaxDefaultExample
stringliteral["http://localhost:2375","https://localhost:2376","unix:///var/run/docker.sock","npipe:////./pipe/docker_engine","/var/run/docker.sock","//./pipe/docker_engine"]
tls(optional)

TLS options to connect to the Docker daemon. This has no effect unless docker_host is an HTTPS URL. If absent, Vector will try to use environment variable DOCKER_CERT_PATH and then DOCKER_CONFIG. If both environment variables are absent, Vector will try to read certificates in ~/.docker/.

TypeSyntaxDefaultExample
hashliteral[]
auto_partial_merge(optional)

Setting this to false will disable the automatic merging of partial events.

TypeSyntaxDefaultExample
bool
exclude_containers(optional)

A list of container IDs or names to match against for containers you don't want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to exclude. This can be used in conjunction with include_containers.

TypeSyntaxDefaultExample
arrayliteral["exclude_","exclude_me_0","ad08cc418cf9"]
include_containers(optional)

A list of container IDs or names to match against for containers you want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to include. This can be used in conjunction with exclude_containers.

TypeSyntaxDefaultExample
arrayliteral["include_","include_me_0","ad08cc418cf9"]
include_labels(optional)

A list of container object labels to match against when filtering running containers. This should follow the described label's syntax in docker object labels docs.

TypeSyntaxDefaultExample
arrayliteral["com.example.vendor=Timber Inc.","com.example.name=Vector"]
include_images(optional)

A list of image names to match against. If not provided, all images will be included.

TypeSyntaxDefaultExample
arrayliteral["httpd","redis"]
retry_backoff_secs(optional)

The amount of time to wait before retrying after an error.

TypeSyntaxDefaultExample
uint1
multiline(optional)

Multiline parsing configuration. If not specified, multiline parsing is disabled.

TypeSyntaxDefaultExample
hashregex[]
host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

TypeSyntaxDefaultExample
stringliteralhost
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["docker_logs"]

EventStoreDB Metricsdaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["eventstoredb_metrics"]

Advanced Options

endpoints(optional)

Endpoints to scrape stats from.

TypeSyntaxDefaultExample
stringliteralhttps://localhost:2113/stats["https://localhost:2113/stats"]
scrape_interval_secs(optional)

The interval between scrapes, in seconds.

TypeSyntaxDefaultExample
uint15
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
default_namespace(optional)

The namespace used otherwise will be defaulted to eventstoredb.

TypeSyntaxDefaultExample
stringliteraleventstoredb["app-123-eventstoredb"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["eventstoredb_metrics"]

Execsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

mode(required)

The type of exec mechanism.

TypeSyntaxDefaultExample
stringliteral["scheduled","streaming"]
command(required)

The command to be run, plus any arguments required.

TypeSyntaxDefaultExample
arrayliteral[["echo","Hello World!"],["ls","-la"]]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["exec"]

Advanced Options

working_directory(optional)

The directory in which to run the command.

TypeSyntaxDefaultExample
stringliteral
include_stderr(optional)

Include the output of stderr when generating events.

TypeSyntaxDefaultExample
bool
event_per_line(optional)

Determine if events should be generated per line or buffered and output as a single event when script execution finishes.

TypeSyntaxDefaultExample
bool
maximum_buffer_size_bytes(optional)

The maximum buffer size allowed before a log event will be generated.

TypeSyntaxDefaultExample
uint1000000
scheduled(optional)

The scheduled options.

TypeSyntaxDefaultExample
hash[]
streaming(optional)

The streaming options.

TypeSyntaxDefaultExample
hash[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["exec"]
3mode = ["scheduled","streaming"]
4command = [ ["echo","HelloWorld!"], ["ls","-la"]]

Syslogaggregatorsidecar

  • Best Effort
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to listen for connections on, or systemd#N to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:9000","systemd","systemd#3"]
mode(required)

The type of socket to use.

TypeSyntaxDefaultExample
stringliteral["tcp","udp","unix_datagram","unix_stream"]
path(required)

The unix socket path. This should be an absolute path.

TypeSyntaxDefaultExample
stringliteral["/path/to/socket"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["syslog","socket"]

Advanced Options

host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

TypeSyntaxDefaultExample
stringliteralhost
max_length(optional)

The maximum bytes size of incoming messages before they are discarded.

TypeSyntaxDefaultExample
uint102400
receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
shutdown_timeout_secs(optional)

The timeout before a connection is forcefully closed during shutdown.

TypeSyntaxDefaultExample
uint30
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["syslog","socket"]
3address = ["0.0.0.0:9000","systemd","systemd#3"]
4mode = ["tcp","udp","unix_datagram","unix_stream"]
5path = ["/path/to/socket"]

Fluentsidecaraggregator

  • Best Effort
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to listen for TCP connections on.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:24224"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["fluent"]

Advanced Options

receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["fluent"]
3address = ["0.0.0.0:24224"]

Generatordaemonsidecar

  • At Least Once
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

format(required)

The format of the randomly generated output.

TypeSyntaxDefaultExample
stringliteral["shuffle","apache_common","apache_error","syslog","bsd_syslog","json"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["generator"]

Advanced Options

interval(optional)

The amount of time, in seconds, to pause between each batch of output lines. If not set, there is no delay.

TypeSyntaxDefaultExample
float[1,0.1,0.01]
count(optional)

The total number of lines to output. By default the source continuously prints logs (infinitely).

TypeSyntaxDefaultExample
uint
lines(optional)

The list of lines to output.

TypeSyntaxDefaultExample
arrayliteral["Line 1","Line 2"]
sequence(optional)

If true, each output line starts with an increasing sequence number, beginning with 0.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["generator"]
3format = [
4 "shuffle",
5 "apache_common",
6 "apache_error",
7 "syslog",
8 "bsd_syslog",
9 "json"
10]

Heroku Logplexaggregator

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to accept connections on. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:80","localhost:80"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["heroku_logs"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
auth(optional)

Options for HTTP Basic Authentication.

TypeSyntaxDefaultExample
hashliteral[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
query_parameters(optional)

A list of URL query parameters to include in the log event. These will override any values included in the body with conflicting names.

TypeSyntaxDefaultExample
arrayliteral["application","source"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["heroku_logs"]
3address = ["0.0.0.0:80","localhost:80"]

Host Metricsdaemon

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["host_metrics"]

Advanced Options

collectors(optional)

The list of host metric collector services to use. Defaults to all collectors.

TypeSyntaxDefaultExample
arrayliteralcgroupscpudiskfilesystemloadhostmemorynetwork["cgroups","cpu","disk","filesystem","load","host","memory","network"]
namespace(optional)

The namespace of metrics. Disabled if empty.

TypeSyntaxDefaultExample
stringliteralhost
scrape_interval_secs(optional)

The interval between metric gathering, in seconds.

TypeSyntaxDefaultExample
uint15
cgroups(optional)

Options for the "cgroups" (controller groups) metrics collector.

Note: this collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.

TypeSyntaxDefaultExample
hashliteral[]
disk(optional)

Options for the "disk" metrics collector.

TypeSyntaxDefaultExample
hashliteral[]
filesystem(optional)

Options for the "filesystem" metrics collector.

TypeSyntaxDefaultExample
hashliteral[]
network(optional)

Options for the "network" metrics collector.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["host_metrics"]

HTTPaggregatorsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to accept connections on. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:80","localhost:80"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["http"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
encoding(optional)

The expected encoding of received data. Note that for json and ndjson encodings, the fields of the JSON objects are output as separate fields.

TypeSyntaxDefaultExample
stringliteraltext
headers(optional)

A list of HTTP headers to include in the log event. These will override any values included in the JSON payload with conflicting names.

TypeSyntaxDefaultExample
arrayliteral["User-Agent","X-My-Custom-Header"]
auth(optional)

Options for HTTP Basic Authentication.

TypeSyntaxDefaultExample
hashliteral[]
query_parameters(optional)

A list of URL query parameters to include in the log event. These will override any values included in the body with conflicting names.

TypeSyntaxDefaultExample
arrayliteral["application","source"]
path(optional)

The URL path on which log event POST requests shall be sent.

TypeSyntaxDefaultExample
stringliteral/["/event/path","/logs"]
strict_path(optional)

If set to true, only requests using the exact URL path specified in path will be accepted; otherwise requests sent to a URL path that starts with the value of path will be accepted. With strict_path set to false and path set to "", the configured HTTP source will accept requests from any URL path.

TypeSyntaxDefaultExample
bool
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
path_key(optional)

The event key in which the requested URL path used to send the request will be stored.

TypeSyntaxDefaultExample
stringliteralpath["vector_http_path"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["http"]
3address = ["0.0.0.0:80","localhost:80"]

Internal Logsaggregatordaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["internal_logs"]

Advanced Options

host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

Set to "" to suppress this key.

TypeSyntaxDefaultExample
stringliteralhost
pid_key(optional)

The key name added to each event representing the current process ID.

Set to "" to suppress this key.

TypeSyntaxDefaultExample
stringliteralpid
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["internal_logs"]

Internal Metricsaggregatordaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["internal_metrics"]

Advanced Options

namespace(optional)

The namespace of the metric.

TypeSyntaxDefaultExample
stringliteralvector
scrape_interval_secs(optional)

The interval between metric gathering, in seconds.

TypeSyntaxDefaultExample
uint2
tags(optional)

Metric tag options.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["internal_metrics"]

Journalddaemon

  • At Least Once
  • Stable
  • Linux

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["journald"]

Advanced Options

batch_size(optional)

The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.

TypeSyntaxDefaultExample
uint16
current_boot_only(optional)

Include only entries from the current boot.

TypeSyntaxDefaultExample
bool
exclude_units(optional)

The list of unit names to exclude from monitoring. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

TypeSyntaxDefaultExample
arrayliteral["badservice","sysinit.target"]
include_units(optional)

The list of unit names to monitor. If empty or not present, all units are accepted. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

TypeSyntaxDefaultExample
arrayliteral["ntpd","sysinit.target"]
journalctl_path(optional)

The full path of the journalctl executable. If not set, Vector will search the path for journalctl.

TypeSyntaxDefaultExample
stringliteraljournalctl["/usr/local/bin/journalctl"]
data_dir(optional)

The directory used to persist file checkpoint positions. By default, the global data_dir option is used. Please make sure the Vector project has write permissions to this dir.

TypeSyntaxDefaultExample
stringfile_system_path["/var/lib/vector"]
journal_directory(optional)

The full path of the journal directory. If not set, journalctl will use the default system journal paths

TypeSyntaxDefaultExample
stringliteral["/run/log/journal"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["journald"]

Kafkaaggregator

  • At Least Once
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

bootstrap_servers(required)

A comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.

TypeSyntaxDefaultExample
stringliteral["10.14.22.123:9092,10.14.23.332:9092"]
group_id(required)

The consumer group name to be used to consume events from Kafka.

TypeSyntaxDefaultExample
stringliteral["consumer-group-name"]
topics(required)

The Kafka topics names to read events from. Regex is supported if the topic begins with ^.

TypeSyntaxDefaultExample
arrayliteral["^(prefix1|prefix2)-.+","topic-1","topic-2"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["kafka"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
auto_offset_reset(optional)

If offsets for consumer group do not exist, set them using this strategy. See the librdkafka documentation for the auto.offset.reset option for further clarification.

TypeSyntaxDefaultExample
stringliterallargest["smallest","earliest","beginning","largest","latest","end","error"]
commit_interval_ms(optional)

The frequency that the consumer offsets are committed (written) to offset storage.

TypeSyntaxDefaultExample
uint5000[5000,10000]
fetch_wait_max_ms(optional)

Maximum time the broker may wait to fill the response.

TypeSyntaxDefaultExample
uint100[50,100]
key_field(optional)

The log field name to use for the Kafka message key.

TypeSyntaxDefaultExample
stringliteralmessage_key["message_key"]
topic_key(optional)

The log field name to use for the Kafka topic.

TypeSyntaxDefaultExample
stringliteraltopic["topic"]
partition_key(optional)

The log field name to use for the Kafka partition name.

TypeSyntaxDefaultExample
stringliteralpartition["partition"]
offset_key(optional)

The log field name to use for the Kafka offset.

TypeSyntaxDefaultExample
stringliteraloffset["offset"]
headers_key(optional)

The log field name to use for the Kafka headers.

TypeSyntaxDefaultExample
stringliteral["headers"]
librdkafka_options(optional)

Advanced options. See librdkafka documentation for details.

TypeSyntaxDefaultExample
hash[{"client.id":"${ENV_VAR}","fetch.error.backoff.ms":"1000","socket.send.buffer.bytes":"100"}]
sasl(optional)

Options for SASL/SCRAM authentication support.

TypeSyntaxDefaultExample
hashliteral[]
session_timeout_ms(optional)

The Kafka session timeout in milliseconds.

TypeSyntaxDefaultExample
uint10000[5000,10000]
socket_timeout_ms(optional)

Default timeout for network requests.

TypeSyntaxDefaultExample
uint60000[30000,60000]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["kafka"]
3bootstrap_servers = ["10.14.22.123:9092,10.14.23.332:9092"]
4group_id = ["consumer-group-name"]
5topics = ["^(prefix1|prefix2)-.+","topic-1","topic-2"]

Kubernetes Logsdaemon

  • Best Effort
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["kubernetes_logs"]

Advanced Options

pod_annotation_fields(optional)

Configuration for how the events are annotated with Pod metadata.

TypeSyntaxDefaultExample
hashliteral[]
namespace_annotation_fields(optional)

Configuration for how the events are annotated with Namespace metadata.

TypeSyntaxDefaultExample
hashliteral[]
auto_partial_merge(optional)

Automatically merge partial messages into a single event. Partial here is in respect to messages that were split by the Kubernetes Container Runtime log driver.

TypeSyntaxDefaultExample
bool
ingestion_timestamp_field(optional)

The exact time the event was ingested into Vector.

TypeSyntaxDefaultExample
stringliteral
kube_config_file(optional)

Optional path to a kubeconfig file readable by Vector. If not set, Vector will try to connect to Kubernetes using in-cluster configuration.

TypeSyntaxDefaultExample
stringliteral
self_node_name(optional)

The name of the Kubernetes Node this Vector instance runs at. Configured to use an env var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.

TypeSyntaxDefaultExample
stringliteral${VECTOR_SELF_NODE_NAME}
exclude_paths_glob_patterns(optional)

A list of glob patterns to exclude from reading the files.

TypeSyntaxDefaultExample
arrayliteral**/*.gz**/*.tmp["**/exclude/**"]
extra_field_selector(optional)

Specifies the field selector to filter Pods with, to be used in addition to the built-in Node filter. The name of the Kubernetes Node this Vector instance runs at. Configured to use an env var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.

TypeSyntaxDefaultExample
stringliteral["metadata.name!=pod-name-to-exclude","metadata.name!=pod-name-to-exclude,metadata.name=mypod"]
extra_label_selector(optional)

Specifies the label selector to filter Pods with, to be used in addition to the built-in vector.dev/exclude filter.

TypeSyntaxDefaultExample
stringliteral["my_custom_label!=my_value","my_custom_label!=my_value,my_other_custom_label=my_value"]
max_line_bytes(optional)

The maximum number of a bytes a line can contain before being discarded. This protects against malformed lines or tailing incorrect files.

TypeSyntaxDefaultExample
uint32768
fingerprint_lines(optional)

The number of lines to read when generating a unique fingerprint of a log file. This is helpful when some containers share common first log lines. WARNING: If the file has less than this amount of lines then it won't be read at all. This is important since container logs are broken up into several files, so the greater lines value is, the greater the chance of it not reading the last file/logs of the container.

TypeSyntaxDefaultExample
uint1
glob_minimum_cooldown_ms(optional)

Delay between file discovery calls. This controls the interval at which Vector searches for files within a single pod.

TypeSyntaxDefaultExample
uint60000
data_dir(optional)

The directory used to persist file checkpoint positions. By default, the global data_dir option is used. Please make sure the Vector project has write permissions to this dir.

TypeSyntaxDefaultExample
stringfile_system_path["/var/lib/vector"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["kubernetes_logs"]

Logstashsidecaraggregator

  • Best Effort
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to listen for TCP connections on.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:5044"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["logstash"]

Advanced Options

receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["logstash"]
3address = ["0.0.0.0:5044"]

MongoDB Metricsdaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

endpoints(required)
TypeSyntaxDefaultExample
arrayliteral["mongodb://localhost:27017"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["mongodb_metrics"]

Advanced Options

scrape_interval_secs(optional)

The interval between scrapes.

TypeSyntaxDefaultExample
uint15
namespace(optional)

The namespace of metrics. Disabled if empty.

TypeSyntaxDefaultExample
stringliteralmongodb
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["mongodb_metrics"]
3endpoints = ["mongodb://localhost:27017"]

NATSaggregator

  • Best Effort
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

url(required)

The NATS URL to connect to. The url must take the form of nats://server:port.

TypeSyntaxDefaultExample
stringliteral["nats://demo.nats.io","nats://127.0.0.1:4222"]
subject(required)

The NATS subject to publish messages to.

TypeSyntaxDefaultExample
stringtemplate["{{ host }}","foo","time.us.east","time.*.east","time.>",">"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["nats"]

Advanced Options

queue(optional)

NATS Queue Group to join

TypeSyntaxDefaultExample
stringliteralvector["foo","API Name Option Example"]
connection_name(optional)

A name assigned to the NATS connection.

TypeSyntaxDefaultExample
stringliteralvector["foo","API Name Option Example"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["nats"]
3url = ["nats://demo.nats.io","nats://127.0.0.1:4222"]
4subject = [
5 "{{ host }}",
6 "foo",
7 "time.us.east",
8 "time.*.east",
9 "time.>",
10 ">"
11]

Vectoraggregator

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The HTTP address to listen for connections on. It must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:9000"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["vector"]

Advanced Options

acknowledgements(optional)

Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.

TypeSyntaxDefaultExample
bool
shutdown_timeout_secs(optional)

The timeout before a connection is forcefully closed during shutdown.

TypeSyntaxDefaultExample
uint30
receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
version(optional)

Source API version. Specifying this version ensures that Vector does not break backward compatibility.

TypeSyntaxDefaultExample
stringliteral1
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["vector"]
3address = ["0.0.0.0:9000"]

Nginx Metricsdaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

endpoints(required)

HTTP/HTTPS endpoint to Nginx server with enabled ngx_http_stub_status_module module.

TypeSyntaxDefaultExample
arrayliteral["http://localhost:8000/basic_status"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["nginx_metrics"]

Advanced Options

scrape_interval_secs(optional)

The interval between scrapes.

TypeSyntaxDefaultExample
uint15
namespace(optional)

The namespace of metrics. Disabled if empty.

TypeSyntaxDefaultExample
stringliteralnginx
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
auth(optional)

Configures the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["nginx_metrics"]
3endpoints = ["http://localhost:8000/basic_status"]

PostgreSQL Metricsdaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

endpoints(required)

PostgreSQL server endpoint in libpq-style connection strings.

TypeSyntaxDefaultExample
arrayliteral["postgresql://postgres:vector@localhost:5432/postgres"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["postgresql_metrics"]

Advanced Options

scrape_interval_secs(optional)

The interval between scrapes.

TypeSyntaxDefaultExample
uint15
namespace(optional)

The namespace of metrics. Disabled if empty.

TypeSyntaxDefaultExample
stringliteralpostgresql
include_databases(optional)

A list of databases to match (by using POSIX Regular Expressions) against the datname column for which you want to collect metrics from. If not set, metrics are collected from all databases. Specifying "" will include metrics where datname is NULL. This can be used in conjunction with exclude_databases.

TypeSyntaxDefaultExample
arrayliteral["^postgres$","^vector$","^foo"]
exclude_databases(optional)

A list of databases to match (by using POSIX Regular Expressions) against the datname column for which you don't want to collect metrics from. Specifying "" will include metrics where datname is NULL. This can be used in conjunction with include_databases.

TypeSyntaxDefaultExample
arrayliteral["^postgres$","^template.*",""]
tls(optional)

TLS options to connect to the PostgreSQL Server.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["postgresql_metrics"]
3endpoints = ["postgresql://postgres:vector@localhost:5432/postgres"]

Prometheus Scrapedaemonsidecar

  • At Least Once
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

endpoints(required)

Endpoints to scrape metrics from.

TypeSyntaxDefaultExample
arrayliteral["http://localhost:9090/metrics"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["prometheus_scrape"]

Advanced Options

scrape_interval_secs(optional)

The interval between scrapes, in seconds.

TypeSyntaxDefaultExample
uint15
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
auth(optional)

Configures the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["prometheus_scrape"]
3endpoints = ["http://localhost:9090/metrics"]

Socketaggregatorsidecar

  • Best Effort
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to listen for connections on, or systemd#N to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:9000","systemd","systemd#3"]
mode(required)

The type of socket to use.

TypeSyntaxDefaultExample
stringliteral["tcp","udp","unix_datagram","unix_stream"]
path(required)

The unix socket path. This should be an absolute path.

TypeSyntaxDefaultExample
stringliteral["/path/to/socket"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["socket"]

Advanced Options

host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

TypeSyntaxDefaultExample
stringliteralhost
max_length(optional)

The maximum bytes size of incoming messages before they are discarded.

TypeSyntaxDefaultExample
uint102400
receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
shutdown_timeout_secs(optional)

The timeout before a connection is forcefully closed during shutdown.

TypeSyntaxDefaultExample
uint30
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["socket"]
3address = ["0.0.0.0:9000","systemd","systemd#3"]
4mode = ["tcp","udp","unix_datagram","unix_stream"]
5path = ["/path/to/socket"]

Splunk HTTP Event Collector (HEC)aggregator

  • At Least Once
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["splunk_hec"]

Advanced Options

address(optional)

The address to accept connections on.

TypeSyntaxDefaultExample
stringliteral0.0.0.0:8080
token(optional)

If supplied, incoming requests must supply this token in the Authorization header, just as a client would if it was communicating with the Splunk HEC endpoint directly. If not supplied, the Authorization header will be ignored and requests will not be authenticated.

TypeSyntaxDefaultExample
stringliteral["A94A8FE5CCB19BA61C4C08"]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
valid_tokens(optional)

If supplied, incoming requests must supply one of these tokens in the Authorization header, just as a client would if it was communicating with the Splunk HEC endpoint directly. If not supplied, the Authorization header will be ignored and requests will not be authenticated.

TypeSyntaxDefaultExample
arrayliteral["A94A8FE5CCB19BA61C4C08"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["splunk_hec"]

StatsDaggregator

  • Best Effort
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

address(required)

The address to listen for connections on, or systemd#N to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:8125","systemd","systemd#3"]
mode(required)

The type of socket to use.

TypeSyntaxDefaultExample
stringliteral["tcp","udp","unix"]
path(required)

The unix socket path. This should be an absolute path.

TypeSyntaxDefaultExample
stringliteral["/path/to/socket"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["statsd"]

Advanced Options

receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
shutdown_timeout_secs(optional)

The timeout before a connection is forcefully closed during shutdown.

TypeSyntaxDefaultExample
uint30
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["statsd"]
3address = ["0.0.0.0:8125","systemd","systemd#3"]
4mode = ["tcp","udp","unix"]
5path = ["/path/to/socket"]

STDINsidecar

  • At Least Once
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["stdin"]

Advanced Options

host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

TypeSyntaxDefaultExample
stringliteralhost
max_length(optional)

The maximum bytes size of a message before rest of it will be discarded.

TypeSyntaxDefaultExample
uint102400
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["stdin"]

Dnstapdaemon

  • Best Effort
  • Beta
  • Linux, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.

Required Options

socket_path(required)

Absolute path of server socket file to which the DNS server is configured to send dnstap data. The socket file will be created by dnstap source component automatically upon startup.

TypeSyntaxDefaultExample
stringfile_system_path["/run/bind/dnstap.sock"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["dnstap"]

Advanced Options

max_frame_length(optional)

Max dnstap frame length that the dnstap source can handle.

TypeSyntaxDefaultExample
uint102400
socket_file_mode(optional)

Unix file mode bits to be applied to server socket file as its designated file permissions. Note that the file mode value can be specified in any numeric format supported by TOML, but it'd be more intuitive to use an octal number. Also note that the value specified must be between 0o700 and 0o777.

TypeSyntaxDefaultExample
uint[511,492,508]
socket_receive_buffer_size(optional)

Set receive buffer size of server Unix socket if specified. No change to the default size if omitted.

TypeSyntaxDefaultExample
uint
socket_send_buffer_size(optional)

Set send buffer size of server Unix socket if specified. No change to the default size if omitted.

TypeSyntaxDefaultExample
uint
raw_data_only(optional)

Whether or not to write out raw dnstap frame data directly (to be encoded in Base64) without any parsing and formatting.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["dnstap"]
3socket_path = ["/run/bind/dnstap.sock"]
  • Deprecated
  • Linux, Windows, MacOS

Splits a string field on a defined separator.

Required Options

field_names(required)

The field names assigned to the resulting tokens, in order.

TypeSyntaxDefaultExample
arrayliteral["timestamp","level","message","parent.child"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["split"]

Advanced Options

drop_field(optional)

If true the field will be dropped after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The field to apply the split on.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child"]
separator(optional)

The separator to split the field on. If no separator is given, it will split on all whitespace. 'Whitespace' is defined according to the terms of the Unicode Derived Core Property White_Space.

TypeSyntaxDefaultExample
stringliteral[whitespace][","]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["split"]
3field_names = ["timestamp","level","message","parent.child"]
4inputs = ["my-source-or-transform-id","prefix-*"]

Aggregate

  • Beta
  • Linux, Windows, MacOS

Aggregates multiple metric events into a single metric event based
on a defined interval window. This helps to reduce metric volume at
the cost of granularity.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aggregate"]

Advanced Options

interval_ms(optional)

The interval over which metrics are aggregated in milliseconds. Over this period metrics with the same series data (name, namespace, tags, ...) will be aggregated.

TypeSyntaxDefaultExample
uint10000
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aggregate"]
3inputs = ["my-source-or-transform-id","prefix-*"]

ANSI Stripper

  • Deprecated
  • Linux, Windows, MacOS

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["ansi_stripper"]

Advanced Options

field(optional)

The target field to strip ANSI escape sequences from.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child","array[0]"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["ansi_stripper"]
3inputs = ["my-source-or-transform-id","prefix-*"]

AWS CloudWatch Logs Subscription Parser

  • Deprecated
  • Linux, Windows, MacOS

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) coming from the aws_kinesis_firehose source.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_cloudwatch_logs_subscription_parser"]

Advanced Options

field(optional)

The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.

TypeSyntaxDefaultExample
stringliteralmessage
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_cloudwatch_logs_subscription_parser"]
3inputs = ["my-source-or-transform-id","prefix-*"]

AWS EC2 Metadata

  • Stable
  • Linux, Windows, MacOS

Enriches events with AWS EC2 environment metadata.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_ec2_metadata"]

Advanced Options

endpoint(optional)

Override the default EC2 Metadata endpoint.

TypeSyntaxDefaultExample
stringliteralhttp://169.254.169.254
fields(optional)

A list of fields to include in each event.

TypeSyntaxDefaultExample
arrayliteralinstance-idlocal-hostnamelocal-ipv4public-hostnamepublic-ipv4ami-idavailability-zonevpc-idsubnet-idregion["instance-id","local-hostname"]
namespace(optional)

Prepend a namespace to each field's key.

TypeSyntaxDefaultExample
stringliteral["","ec2","aws.ec2"]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
refresh_interval_secs(optional)

The interval in seconds at which the EC2 Metadata api will be called.

TypeSyntaxDefaultExample
uint10
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_ec2_metadata"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Coercer

  • Deprecated
  • Linux, Windows, MacOS

Coerces log fields into typed values.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["coercer"]

Advanced Options

drop_unspecified(optional)

Set to true to drop all fields that are not specified in the types table. Make sure both message and timestamp are specified in the types table as their absense will cause the original message data to be dropped along with other extraneous fields.

TypeSyntaxDefaultExample
bool
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["coercer"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Compound

  • Beta
  • Linux, Windows, MacOS

Defines an ordered chain of child tranforms that will be applied sequentially on incoming events.

Required Options

nested(required)

A table of transforms configurations' representing the chain of transforms to be applied on incoming events. All transforms in the chain can then be referenced as an input by other components with the name <transform_name>.<nested_transform_name>. All transforms in the chain also generate internal metrics as if they were configured separately.

TypeSyntaxDefaultExample
hash[]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["compound"]

Advanced Options

vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["compound"]
3nested = []
4inputs = ["my-source-or-transform-id","prefix-*"]
  • Deprecated
  • Linux, Windows, MacOS

Slices log string fields and joins them into a single field.

Required Options

items(required)

A list of substring definitons in the format of source_field[start..end]. For both start and end negative values are counted from the end of the string.

TypeSyntaxDefaultExample
arrayliteral["first[..3]","second[-5..]","third[3..6]"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
target(required)

The name for the new label.

TypeSyntaxDefaultExample
stringliteral["root_field_name","parent.child","array[0]"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["concat"]

Advanced Options

joiner(optional)

The string that is used to join all items.

TypeSyntaxDefaultExample
stringliteral [" ",",","_","+"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["concat"]
3items = [ "first[..3]", "second[-5..]", "third[3..6]"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5target = [ "root_field_name", "parent.child", "array[0]"]

Dedupe events

  • Stable
  • Linux, Windows, MacOS

Deduplicates events to reduce data volume by eliminating copies of data.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
fields(required)

Options controlling what fields to match against.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["dedupe"]

Advanced Options

cache(optional)

Options controlling how we cache recent Events for future duplicate checking.

TypeSyntaxDefaultExample
hash[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["dedupe"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4fields = []

Remove Tags

  • Deprecated
  • Linux, Windows, MacOS

Removes one or more metric tags.

Required Options

tags(required)

The tag names to drop.

TypeSyntaxDefaultExample
arrayliteral["tag1","tag2"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["remove_tags"]

Advanced Options

vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["remove_tags"]
3tags = ["tag1","tag2"]
4inputs = ["my-source-or-transform-id","prefix-*"]
  • Stable
  • Linux, Windows, MacOS

Enrich events with geolocation data from the MaxMind GeoIP2-City, GeoLite2-City, GeoIP2-ISP and GeoLite2-ASN databases.

Required Options

database(required)

Path to the MaxMind GeoIP2 or GeoLite2 binary city database file (GeoLite2-City.mmdb). Other databases, such as the the country database, are not supported.

TypeSyntaxDefaultExample
stringliteral["/path/to/GeoLite2-City.mmdb","/path/to/GeoLite2-ISP.mmdb"]
source(required)

The field name that contains the IP address. This field should contain a valid IPv4 or IPv6 address.

TypeSyntaxDefaultExample
stringliteral["ip_address","x-forwarded-for","parent.child","array[0]"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["geoip"]

Advanced Options

target(optional)

The default field to insert the resulting GeoIP data into. See output for more info.

TypeSyntaxDefaultExample
stringliteralgeoip["geoip","parent.child"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["geoip"]
3database = ["/path/to/GeoLite2-City.mmdb","/path/to/GeoLite2-ISP.mmdb"]
4source = [ "ip_address", "x-forwarded-for", "parent.child", "array[0]"]
5inputs = ["my-source-or-transform-id","prefix-*"]

Grok Parser

  • Deprecated
  • Linux, Windows, MacOS

Parses a log field value with Grok.

Required Options

pattern(required)
TypeSyntaxDefaultExample
stringliteral["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["grok_parser"]

Advanced Options

drop_field(optional)

If true will drop the specified field after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The log field to execute the pattern against. Must be a string value.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child","array[0]"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["grok_parser"]
3pattern = [
4 "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
5]
6inputs = ["my-source-or-transform-id","prefix-*"]

JSON Parser

  • Deprecated
  • Linux, Windows, MacOS

Parses a log field value as JSON.

Required Options

drop_invalid(required)

If true events with invalid JSON will be dropped, otherwise the event will be kept and passed through.

TypeSyntaxDefaultExample
bool
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["json_parser"]

Advanced Options

drop_field(optional)

If the specified field should be dropped (removed) after parsing. If parsing fails, the field will not be removed, irrespective of this setting.

TypeSyntaxDefaultExample
bool
field(optional)

The log field to decode as JSON. Must be a string value type.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child","array[0]"]
overwrite_target(optional)

If target_field is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true.

TypeSyntaxDefaultExample
bool
target_field(optional)

If this setting is present, the parsed JSON will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.

TypeSyntaxDefaultExample
stringliteral["root_field","parent.child"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["json_parser"]
3drop_invalid = ""
4inputs = ["my-source-or-transform-id","prefix-*"]

Key-value Parser

  • Deprecated
  • Linux, Windows, MacOS

Loosely parses a log field's value in key-value format.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["key_value_parser"]

Advanced Options

drop_field(optional)

If true will drop the specified field after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The log field containing key/value pairs to parse. Must be a string value.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child","array[0]"]
field_split(optional)

The character(s) to split a key/value pair on which results in a new field with an associated value. Must be a string value.

TypeSyntaxDefaultExample
stringliteral=[":","="]
overwrite_target(optional)

If target_field is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true.

TypeSyntaxDefaultExample
bool
separator(optional)

The character(s) that separate key/value pairs. Must be a string value.

TypeSyntaxDefaultExample
stringliteral[whitespace][",",";","|"]
target_field(optional)

If this setting is present, the parsed JSON will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.

TypeSyntaxDefaultExample
stringliteral["root_field","parent.child"]
trim_key(optional)

Removes characters from the beginning and end of a key until a character that is not listed. ex: <key>=value would result in key: value with this option set to <>.

TypeSyntaxDefaultExample
stringliteral["<>","{}"]
trim_value(optional)

Removes characters from the beginning and end of a value until a character that is not listed. ex: key=<<>value>> would result in key: value with this option set to <>.

TypeSyntaxDefaultExample
stringliteral["<>","{}"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["key_value_parser"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Log to Metric

  • Stable
  • Linux, Windows, MacOS

Derives one or more metric events from a log event.

Required Options

metrics(required)

A table of key/value pairs representing the keys to be added to the event.

TypeSyntaxDefaultExample
arrayliteral[]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["log_to_metric"]

Advanced Options

vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["log_to_metric"]
3metrics = []
4inputs = ["my-source-or-transform-id","prefix-*"]

Logfmt Parser

  • Deprecated
  • Linux, Windows, MacOS

Parses a log field's value in the logfmt format.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["logfmt_parser"]

Advanced Options

drop_field(optional)

If the specified field should be dropped (removed) after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The log field to parse.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child","array[0]"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["logfmt_parser"]
3inputs = ["my-source-or-transform-id","prefix-*"]
  • Stable
  • Linux, Windows, MacOS

Transform events with a full embedded Lua 5.4 engine.

Required Options

hooks(required)

Configures hooks handlers.

TypeSyntaxDefaultExample
hashliteral[]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
version(required)

Transform API version. Specifying this version ensures that Vector does not break backward compatibility.

TypeSyntaxDefaultExample
stringliteral["1","2"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["lua"]

Advanced Options

search_dirs(optional)

A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector's configs.

TypeSyntaxDefaultExample
arrayliteral["/etc/vector/lua"]
source(optional)

The source which is evaluated when the transform is created.

TypeSyntaxDefaultExample
stringliteral["function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t\tsyntax: \"literal\"\n\t\t}\n \t}\nend","-- external file with hooks and timers defined\nrequire('custom_module')"]
timers(optional)

Configures timers which are executed periodically at given interval.

TypeSyntaxDefaultExample
arrayliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["lua"]
3hooks = []
4inputs = ["my-source-or-transform-id","prefix-*"]
5version = ["1","2"]
  • Deprecated
  • Linux, Windows, MacOS

Merges partial log events into a single event.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["merge"]

Advanced Options

fields(optional)

Fields to merge. The values of these fields will be merged into the first partial event. Fields not specified here will be ignored. Merging process takes the first partial event and the base, then it merges in the fields from each successive partial event, until a non-partial event arrives. Finally, the non-partial event fields are merged in, producing the resulting merged event.

TypeSyntaxDefaultExample
arrayliteralmessage["message","parent.child"]
partial_event_marker_field(optional)

The field that indicates that the event is partial. A consequent stream of partial events along with the first non-partial event will be merged together.

TypeSyntaxDefaultExample
stringliteral_partial["_partial","parent.child"]
stream_discriminant_fields(optional)

An ordered list of fields to distinguish streams by. Each stream has a separate partial event merging state. Should be used to prevent events from unrelated sources from mixing together, as this affects partial event processing.

TypeSyntaxDefaultExample
arrayliteral["host","parent.child"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["merge"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Metric to Log

  • Stable
  • Linux, Windows, MacOS

Converts a metric event into a log event, which can be useful for sending metrics to log-support downstream components.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["metric_to_log"]

Advanced Options

host_tag(optional)

Tag key that identifies the source host.

TypeSyntaxDefaultExample
stringliteralhostname["host","hostname"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["metric_to_log"]
3inputs = ["my-source-or-transform-id","prefix-*"]
  • Beta
  • Linux, Windows, MacOS

Reduces multiple log events into a single log event based on a set of conditions and merge strategies.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["reduce"]

Advanced Options

ends_when(optional)

A condition used to distinguish the final event of a transaction. If this condition resolves to true for an event, the current transaction is immediately flushed with this event.

TypeSyntaxDefaultExample
stringliteral[".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"]
expire_after_ms(optional)

A maximum period of time to wait after the last event is received before a combined event should be considered complete.

TypeSyntaxDefaultExample
uint30000
flush_period_ms(optional)

Controls the frequency that Vector checks for (and flushes) expired events.

TypeSyntaxDefaultExample
uint1000
group_by(optional)

An ordered list of fields by which to group events. Each group is combined independently, allowing you to keep independent events separate. When no fields are specified, all events will be combined in a single group. Events missing a specified field will be combined in their own group.

TypeSyntaxDefaultExample
arrayliteral["request_id","user_id","transaction_id"]
merge_strategies(optional)

A map of field names to custom merge strategies. For each field specified this strategy will be used for combining events rather than the default behavior.

The default behavior is as follows:

  1. The first value of a string field is kept, subsequent values are discarded.
  2. For timestamp fields the first is kept and a new field [field-name]_end is added with the last received timestamp value.
  3. Numeric values are summed.
TypeSyntaxDefaultExample
hashliteral[{"method":"discard","path":"discard","duration_ms":"sum","query":"array"}]
starts_when(optional)

A condition used to distinguish the first event of a transaction. If this condition resolves to true for an event, the previous transaction is flushed (without this event) and a new transaction is started.

TypeSyntaxDefaultExample
stringliteral[".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["reduce"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Tag Cardinality Limit

  • Beta
  • Linux, Windows, MacOS

Limits the cardinality of tags on metric events, protecting against accidental high cardinality usage that can commonly disrupt the stability of metrics storages.

Required Options

mode(required)

Controls what approach is used internally to keep track of previously seen tags and deterime when a tag on an incoming metric exceeds the limit.

TypeSyntaxDefaultExample
stringliteral["exact","probabilistic"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["tag_cardinality_limit"]

Advanced Options

cache_size_per_tag(optional)

The size of the cache in bytes to use to detect duplicate tags. The bigger the cache the less likely it is to have a 'false positive' or a case where we allow a new value for tag even after we have reached the configured limits.

TypeSyntaxDefaultExample
uint5120000
limit_exceeded_action(optional)

Controls what should happen when a metric comes in with a tag that would exceed the configured limit on cardinality.

TypeSyntaxDefaultExample
stringliteraldrop_tag
value_limit(optional)

How many distinct values to accept for any given key.

TypeSyntaxDefaultExample
uint500
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["tag_cardinality_limit"]
3mode = ["exact","probabilistic"]
4inputs = ["my-source-or-transform-id","prefix-*"]

Tokenizer

  • Deprecated
  • Linux, Windows, MacOS

Tokenizes a field's value by splitting on white space, ignoring special wrapping characters, and zip the tokens into ordered field names.

Required Options

field_names(required)

The log field names assigned to the resulting tokens, in order.

TypeSyntaxDefaultExample
arrayliteral["timestamp","level","message","parent.child"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["tokenizer"]

Advanced Options

drop_field(optional)

If true the field will be dropped after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The log field to tokenize.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["tokenizer"]
3field_names = ["timestamp","level","message","parent.child"]
4inputs = ["my-source-or-transform-id","prefix-*"]
  • Beta
  • Linux, Windows, MacOS

Samples events at a configurable rate.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
rate(required)

The rate at which events will be forwarded, expressed as 1/N. For example, rate = 10 means 1 out of every 10 events will be forwarded and the rest will be dropped.

TypeSyntaxDefaultExample
uint[10]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["sample"]

Advanced Options

key_field(optional)

The name of the log field whose value will be hashed to determine if the event should be passed.

Consistently samples the same events. Actual rate of sampling may differ from the configured one if values in the field are not uniformly distributed. If left unspecified, or if the event doesn't have key_field, events will be count rated.

TypeSyntaxDefaultExample
stringliteral["message"]
exclude(optional)

The set of logical conditions to exclude events from sampling.

TypeSyntaxDefaultExample
stringremap_boolean_expression[".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["sample"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4rate = [10]

Regex Parser

  • Deprecated
  • Linux, Windows, MacOS

Parses a log field's value with a Regular Expression.

Required Options

patterns(required)

The Regular Expressions to apply. Do not include the leading or trailing / in any of the expressions.

TypeSyntaxDefaultExample
arrayliteral["^(?P<timestamp>[\\\\w\\\\-:\\\\+]+) (?P<level>\\\\w+) (?P<message>.*)$"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["regex_parser"]

Advanced Options

drop_failed(optional)

If the event should be dropped if parsing fails.

TypeSyntaxDefaultExample
bool
drop_field(optional)

If the specified field should be dropped (removed) after parsing.

TypeSyntaxDefaultExample
bool
field(optional)

The log field to parse.

TypeSyntaxDefaultExample
stringliteralmessage["message","parent.child"]
overwrite_target(optional)

If target_field is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true.

TypeSyntaxDefaultExample
bool
target_field(optional)

If this setting is present, the parsed fields will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.

TypeSyntaxDefaultExample
stringliteral["root_field","parent.child"]
timezone(optional)

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

TypeSyntaxDefaultExample
stringliterallocal["local","America/NewYork","EST5EDT"]
types(optional)

Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone2020-12-01T02:37:54
%a, %d %b %Y %TRFC 822/2822 without time zoneTue, 01 Dec 2020 02:37:54
%a %d %b %T %Ydate command output without time zoneTue 01 Dec 02:37:54 2020
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%FT%TZISO 8601/RFC 3339 UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339 UTC with time zone2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Ydate command output with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command output with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command output with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020

Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

TypeSyntaxDefaultExample
hash[{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["regex_parser"]
3patterns = [
4 "^(?P<timestamp>[\\\\w\\\\-:\\\\+]+) (?P<level>\\\\w+) (?P<message>.*)$"
5]
6inputs = ["my-source-or-transform-id","prefix-*"]

Add Tags

  • Deprecated
  • Linux, Windows, MacOS

Adds tags to metric events.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["add_tags"]

Advanced Options

overwrite(optional)

By default, fields will be overridden. Set this to false to avoid overwriting values.

TypeSyntaxDefaultExample
bool
tags(optional)

A table of key/value pairs representing the tags to be added to the metric.

TypeSyntaxDefaultExample
hash[{"static_tag":"my value","env_tag":"${ENV_VAR}"}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["add_tags"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Add Fields

  • Deprecated
  • Linux, Windows, MacOS

Adds fields to log events.

Required Options

fields(required)

A table of key/value pairs representing the keys to be added to the event.

TypeSyntaxDefaultExample
hash[{"string_field":"string value","env_var_field":"${ENV_VAR}","templated_field":"{{ my_other_field }}","int_field":1,"float_field":1.2,"bool_field":true,"timestamp_field":"1979-05-27T00:32:00-0700","parent":{"child_field":"child_value"},"list_field":["first","second","third"]}]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["add_fields"]

Advanced Options

overwrite(optional)

By default, fields will be overridden. Set this to false to avoid overwriting values.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["add_fields"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4[[fields]]
5string_field = "string value"
6env_var_field = "${ENV_VAR}"
7templated_field = "{{ my_other_field }}"
8int_field = 1
9float_field = 1.2
10bool_field = true
11timestamp_field = "1979-05-27T00:32:00-0700"
12list_field = ["first","second","third"]
13 [fields.parent]
14 child_field = "child_value"
  • Beta
  • Linux, Windows, MacOS

Is the recommended transform for parsing, shaping, and transforming data in Vector. It implements the Vector Remap Language (VRL), an expression-oriented language designed for processing observability data (logs and metrics) in a safe and performant manner.

Please refer to the VRL reference when writing VRL scripts.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["remap"]

Advanced Options

source(optional)

The Vector Remap Language (VRL) program to execute for each event.

Required if file is missing.

TypeSyntaxDefaultExample
stringremap_program[". = parse_json!(.message)\n.new_field = \"new value\"\n.status = to_int!(.status)\n.duration = parse_duration!(.duration, \"s\")\n.new_name = del(.old_name)"]
file(optional)

File path to the Vector Remap Language (VRL) program to execute for each event.

If a relative path is provided, its root is the current working directory.

Required if source is missing.

TypeSyntaxDefaultExample
stringliteral["./my/program.vrl"]
drop_on_error(optional)

Drop the event if the VRL program returns an error at runtime.

TypeSyntaxDefaultExample
bool
drop_on_abort(optional)

Drop the event if the VRL program is manually aborted through the abort statement.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["remap"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Remove Fields

  • Deprecated
  • Linux, Windows, MacOS

Removes one or more log fields.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
fields(required)

The log field names to drop.

TypeSyntaxDefaultExample
arrayliteral["field1","field2","parent.child"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["remove_fields"]

Advanced Options

drop_empty(optional)

If set to true, after removing fields, remove any parent objects that are now empty.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["remove_fields"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4fields = ["field1","field2","parent.child"]
  • Stable
  • Linux, Windows, MacOS

Splits a stream of events into multiple sub-streams based on a set of conditions.

Required Options

route(required)

A table of route identifiers to logical conditions representing the filter of the route. Each route can then be referenced as an input by other components with the name <transform_name>.<route_id>.

TypeSyntaxDefaultExample
hashremap_boolean_expression[]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["route"]

Advanced Options

vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["route"]
3route = []
4inputs = ["my-source-or-transform-id","prefix-*"]

Rename Fields

  • Deprecated
  • Linux, Windows, MacOS

Renames one or more log fields.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
fields(required)

A table of old-key/new-key pairs representing the keys to be moved in the event.

TypeSyntaxDefaultExample
hash[{"old_field_name":"new_field_name","parent":{"old_child_name":"parent.new_child_name"}}]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["rename_fields"]

Advanced Options

drop_empty(optional)

If set to true, after renaming fields, remove any parent objects of the old field that are now empty.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["rename_fields"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4[[fields]]
5old_field_name = "new_field_name"
6 [fields.parent]
7 old_child_name = "parent.new_child_name"
  • Stable
  • Linux, Windows, MacOS

Filters events based on a set of conditions.

Required Options

condition(required)

The condition to be matched against every input event. Only messages that pass the condition will be forwarded.

TypeSyntaxDefaultExample
stringremap_boolean_expression[".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["filter"]

Advanced Options

vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["filter"]
3condition = [
4 ".status_code != 200 && !includes([\"info\",\"debug\"], .severity)"
5]
6inputs = ["my-source-or-transform-id","prefix-*"]
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

address(required)

The address to connect to. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["92.12.333.224:5000"]
mode(required)

The type of socket to use.

TypeSyntaxDefaultExample
stringliteral["tcp","udp","unix"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
path(required)

The unix socket path. This should be the absolute path.

TypeSyntaxDefaultExample
stringliteral["/path/to/socket"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["socket"]

Advanced Options

buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
send_buffer_bytes(optional)

Configures the send buffer size using the SO_SNDBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["socket"]
3address = ["92.12.333.224:5000"]
4mode = ["tcp","udp","unix"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7path = ["/path/to/socket"]

Humio Logs

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
token(required)

Your Humio ingestion token.

TypeSyntaxDefaultExample
stringliteral["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["humio_logs"]

Advanced Options

endpoint(optional)

The base URL of the Humio instance.

TypeSyntaxDefaultExample
stringliteralhttps://cloud.humio.com["http://127.0.0.1","http://example.com"]
event_type(optional)

The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.

If unset, Humio will default it to none.

TypeSyntaxDefaultExample
stringtemplate["json","none"]
host_key(optional)

The name of the log field to be used as the hostname sent to Humio. This overrides the global host_key option.

TypeSyntaxDefaultExample
stringliteral["hostname"]
source(optional)

The source of events sent to this sink. Typically the filename the logs originated from. Maps to @source in Humio.

TypeSyntaxDefaultExample
stringtemplate["{{file}}","/var/log/syslog","UDP:514"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["humio_logs"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5token = ["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"]

Humio Metrics

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
token(required)

Your Humio ingestion token.

TypeSyntaxDefaultExample
stringliteral["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["humio_metrics"]

Advanced Options

endpoint(optional)

The base URL of the Humio instance.

TypeSyntaxDefaultExample
stringliteralhttps://cloud.humio.com["http://127.0.0.1","http://example.com"]
event_type(optional)

The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.

If unset, Humio will default it to none.

TypeSyntaxDefaultExample
stringtemplate["json","none"]
host_key(optional)

The name of the log field to be used as the hostname sent to Humio. This overrides the global host_key option.

TypeSyntaxDefaultExample
stringliteral["hostname"]
source(optional)

The source of events sent to this sink. Typically the filename the logs originated from. Maps to @source in Humio.

TypeSyntaxDefaultExample
stringtemplate["{{file}}","/var/log/syslog","UDP:514"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["humio_metrics"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5token = ["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"]

InfluxDB Logs

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

bucket(required)

The destination bucket for writes into InfluxDB 2.

TypeSyntaxDefaultExample
stringliteral["vector-bucket","4d2225e4d3d49f75"]
database(required)

Sets the target database for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["vector-database","iot-store"]
endpoint(required)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["http://localhost:8086/","https://us-west-2-1.aws.cloud1.influxdata.com","https://us-west-2-1.aws.cloud2.influxdata.com"]
org(required)

Specifies the destination organization for writes into InfluxDB 2.

TypeSyntaxDefaultExample
stringliteral["my-org","33f2cff0a28e5b63"]
token(required)

Authentication token for InfluxDB 2.

TypeSyntaxDefaultExample
stringliteral["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"]
namespace(required)

A prefix that will be added to all logs names.

TypeSyntaxDefaultExample
stringliteral["service"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["influxdb_logs"]

Advanced Options

consistency(optional)

Sets the write consistency for the point for InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["any","one","quorum","all"]
password(optional)

Sets the password for authentication if you’ve enabled authentication for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["${INFLUXDB_PASSWORD}","influxdb4ever"]
retention_policy_name(optional)

Sets the target retention policy for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["autogen","one_day_only"]
tags(optional)

A set of additional fields that will be attached to each LineProtocol as a tag. Note: If the set of tag values has high cardinality this also increase cardinality in InfluxDB.

TypeSyntaxDefaultExample
arrayfield_path["field1","parent.child_field"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
username(optional)

Sets the username for authentication if you’ve enabled authentication for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["todd","vector-source"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["influxdb_logs"]
3bucket = ["vector-bucket","4d2225e4d3d49f75"]
4database = ["vector-database","iot-store"]
5endpoint = [
6 "http://localhost:8086/",
7 "https://us-west-2-1.aws.cloud1.influxdata.com",
8 "https://us-west-2-1.aws.cloud2.influxdata.com"
9]
10org = ["my-org","33f2cff0a28e5b63"]
11token = ["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"]
12namespace = ["service"]
13inputs = ["my-source-or-transform-id","prefix-*"]
14encoding = []

InfluxDB Metrics

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

bucket(required)

The destination bucket for writes into InfluxDB 2.

TypeSyntaxDefaultExample
stringliteral["vector-bucket","4d2225e4d3d49f75"]
database(required)

Sets the target database for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["vector-database","iot-store"]
endpoint(required)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["http://localhost:8086/","https://us-west-2-1.aws.cloud1.influxdata.com","https://us-west-2-1.aws.cloud2.influxdata.com"]
org(required)

Specifies the destination organization for writes into InfluxDB 2.

TypeSyntaxDefaultExample
stringliteral["my-org","33f2cff0a28e5b63"]
token(required)

Authentication token for InfluxDB 2.

TypeSyntaxDefaultExample
stringliteral["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["influxdb_metrics"]

Advanced Options

consistency(optional)

Sets the write consistency for the point for InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["any","one","quorum","all"]
password(optional)

Sets the password for authentication if you’ve enabled authentication for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["${INFLUXDB_PASSWORD}","influxdb4ever"]
retention_policy_name(optional)

Sets the target retention policy for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["autogen","one_day_only"]
tags(optional)

A set of additional fields that will be attached to each LineProtocol as a tag. Note: If the set of tag values has high cardinality this also increase cardinality in InfluxDB.

TypeSyntaxDefaultExample
arrayfield_path["field1","parent.child_field"]
default_namespace(optional)

Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name.

TypeSyntaxDefaultExample
stringliteral["service"]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
username(optional)

Sets the username for authentication if you’ve enabled authentication for the write into InfluxDB 1.

TypeSyntaxDefaultExample
stringliteral["todd","vector-source"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["influxdb_metrics"]
3bucket = ["vector-bucket","4d2225e4d3d49f75"]
4database = ["vector-database","iot-store"]
5endpoint = [
6 "http://localhost:8086/",
7 "https://us-west-2-1.aws.cloud1.influxdata.com",
8 "https://us-west-2-1.aws.cloud2.influxdata.com"
9]
10org = ["my-org","33f2cff0a28e5b63"]
11token = ["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"]
12inputs = ["my-source-or-transform-id","prefix-*"]
13encoding = []
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

bootstrap_servers(required)

A comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.

TypeSyntaxDefaultExample
stringliteral["10.14.22.123:9092,10.14.23.332:9092"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
topic(required)

The Kafka topic name to write events to.

TypeSyntaxDefaultExample
stringliteral["topic-1234","logs-{{unit}}-%Y-%m-%d"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["kafka"]

Advanced Options

key_field(optional)

The log field name or tags key to use for the topic key. If the field does not exist in the log or in tags, a blank value will be used. If unspecified, the key is not sent. Kafka uses a hash of the key to choose the partition or uses round-robin if the record has no key.

TypeSyntaxDefaultExample
stringliteral["user_id"]
librdkafka_options(optional)

Advanced options. See librdkafka documentation for details.

TypeSyntaxDefaultExample
hash[{"client.id":"${ENV_VAR}","fetch.error.backoff.ms":"1000","socket.send.buffer.bytes":"100"}]
message_timeout_ms(optional)

Local message timeout.

TypeSyntaxDefaultExample
uint300000[150000,450000]
sasl(optional)

Options for SASL/SCRAM authentication support.

TypeSyntaxDefaultExample
hashliteral[]
socket_timeout_ms(optional)

Default timeout for network requests.

TypeSyntaxDefaultExample
uint60000[30000,60000]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["kafka"]
3bootstrap_servers = ["10.14.22.123:9092,10.14.23.332:9092"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6topic = ["topic-1234","logs-{{unit}}-%Y-%m-%d"]
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

api_key(required)

The Ingestion API key.

TypeSyntaxDefaultExample
stringliteral["${LOGDNA_API_KEY}","ef8d5de700e7989468166c40fc8a0ccd"]
hostname(required)

The hostname that will be attached to each batch of events.

TypeSyntaxDefaultExample
stringliteral["${HOSTNAME}","my-local-machine"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["logdna"]

Advanced Options

default_app(optional)

The default app that will be set for events that do not contain a file or app field.

TypeSyntaxDefaultExample
stringliteralvector["vector","myapp"]
default_env(optional)

The default environment that will be set for events that do not contain an env field.

TypeSyntaxDefaultExample
stringliteralproduction["staging","production"]
endpoint(optional)

The endpoint to send logs to.

TypeSyntaxDefaultExample
stringliteralhttps://logs.logdna.com/logs/ingest["http://127.0.0.1","http://example.com"]
ip(optional)

The IP address that will be attached to each batch of events.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0"]
mac(optional)

The mac address that will be attached to each batch of events.

TypeSyntaxDefaultExample
stringliteral["my-mac-address"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
tags(optional)

The tags that will be attached to each batch of events.

TypeSyntaxDefaultExample
arrayliteral["tag1","tag2"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["logdna"]
3api_key = ["${LOGDNA_API_KEY}","ef8d5de700e7989468166c40fc8a0ccd"]
4hostname = ["${HOSTNAME}","my-local-machine"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

The base URL of the Loki instance.

TypeSyntaxDefaultExample
stringliteral["http://localhost:3100"]
labels(required)

A set of labels that are attached to each batch of events. Both keys and values are templatable, which enables you to attach dynamic labels to events. Note: If the set of labels has high cardinality, this can cause drastic performance issues with Loki. To prevent this from happening, reduce the number of unique label keys and values.

TypeSyntaxDefaultExample
hashtemplate[{"forwarder":"vector","event":"{{ event_field }}","key":"value","\"{{ event_field }}\"":"{{ another_event_field }}"}]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["loki"]

Advanced Options

auth(optional)

Configures the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
out_of_order_action(optional)

Some sources may generate events with timestamps that aren't in strictly chronological order. The Loki service can't accept a stream of such events. Vector sorts events before sending them to Loki, however some late events might arrive after a batch has been sent. This option specifies what Vector should do with those events.

TypeSyntaxDefaultExample
stringliteraldrop
remove_label_fields(optional)

If this is set to true then when labels are collected from events those fields will also get removed from the event.

TypeSyntaxDefaultExample
bool
remove_timestamp(optional)

If this is set to true then the timestamp will be removed from the event payload. Note the event timestamp will still be sent as metadata to Loki for indexing.

TypeSyntaxDefaultExample
bool
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
tenant_id(optional)

The tenant id that's sent with every request, by default this is not required since a proxy should set this header. When running Loki locally a tenant id is not required either.

You can read more about tenant id's here.

TypeSyntaxDefaultExample
stringtemplate["some_tenant_id","{{ event_field }}"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["loki"]
3endpoint = ["http://localhost:3100"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6[[labels]]
7forwarder = "vector"
8event = "{{ event_field }}"
9key = "value"
10"\"{{ event_field }}\"" = "{{ another_event_field }}"
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

url(required)

The NATS URL to connect to. The url must take the form of nats://server:port.

TypeSyntaxDefaultExample
stringliteral["nats://demo.nats.io","nats://127.0.0.1:4222"]
subject(required)

The NATS subject to publish messages to.

TypeSyntaxDefaultExample
stringtemplate["{{ host }}","foo","time.us.east","time.*.east","time.>",">"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["nats"]

Advanced Options

healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
connection_name(optional)

A name assigned to the NATS connection.

TypeSyntaxDefaultExample
stringliteralvector["foo","API Name Option Example"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["nats"]
3url = ["nats://demo.nats.io","nats://127.0.0.1:4222"]
4subject = [
5 "{{ host }}",
6 "foo",
7 "time.us.east",
8 "time.*.east",
9 "time.>",
10 ">"
11]
12inputs = ["my-source-or-transform-id","prefix-*"]
13encoding = []

New Relic Logs

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["new_relic_logs"]

Advanced Options

insert_key(optional)

Your New Relic insert key (if applicable).

TypeSyntaxDefaultExample
stringliteral["xxxx","${NEW_RELIC_INSERT_KEY}"]
license_key(optional)

Your New Relic license key (if applicable).

TypeSyntaxDefaultExample
stringliteral["xxxx","${NEW_RELIC_LICENSE_KEY}"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
region(optional)

The region to send data to.

TypeSyntaxDefaultExample
stringliteralus
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["new_relic_logs"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []

Papertrail

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

The endpoint to send logs to.

TypeSyntaxDefaultExample
stringliteral["logs.papertrailapp.com:12345"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["papertrail"]

Advanced Options

buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
send_buffer_bytes(optional)

Configures the send buffer size using the SO_SNDBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["papertrail"]
3endpoint = ["logs.papertrailapp.com:12345"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []

Prometheus Exporter

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

address(required)

The address to expose for scraping.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:9598"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["prometheus_exporter"]

Advanced Options

buckets(optional)

Default buckets to use for aggregating distribution metrics into histograms.

TypeSyntaxDefaultExample
array0.0050.010.0250.050.10.250.512.5510[0.005,0.01]
flush_period_secs(optional)

Time interval between set values are reset.

TypeSyntaxDefaultExample
uint60
default_namespace(optional)

Used as a namespace for metrics that don't have it. Typically namespaces are set during ingestion (sources), but it is optional and when missing, we'll use this value. It should follow Prometheus naming conventions.

TypeSyntaxDefaultExample
stringliteral["service"]
quantiles(optional)

Quantiles to use for aggregating distribution metrics into a summary.

TypeSyntaxDefaultExample
array0.50.750.90.950.99[0.5,0.75,0.9,0.95,0.99]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["prometheus_exporter"]
3address = ["0.0.0.0:9598"]
4inputs = ["my-source-or-transform-id","prefix-*"]

Prometheus Remote Write

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

The endpoint URL to send data to.

TypeSyntaxDefaultExample
stringliteral["https://localhost:8087/"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["prometheus_remote_write"]

Advanced Options

auth(optional)

Configures the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
default_namespace(optional)

Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name. It should follow Prometheus naming conventions.

TypeSyntaxDefaultExample
stringliteral["service"]
buckets(optional)

Default buckets to use for aggregating distribution metrics into histograms.

TypeSyntaxDefaultExample
array0.0050.010.0250.050.10.250.512.5510[0.005,0.01]
quantiles(optional)

Quantiles to use for aggregating distribution metrics into a summary.

TypeSyntaxDefaultExample
array0.50.750.90.950.99[0.5,0.75,0.9,0.95,0.99]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
tenant_id(optional)

If set, a header named X-Scope-OrgID will be added to outgoing requests with the text of this setting. This may be used by Cortex or other remote services to identify the tenant making the request.

TypeSyntaxDefaultExample
stringtemplate["my-domain"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["prometheus_remote_write"]
3endpoint = ["https://localhost:8087/"]
4inputs = ["my-source-or-transform-id","prefix-*"]

Apache Pulsar

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

Endpoint to which the pulsar client should connect to.

TypeSyntaxDefaultExample
stringliteral["pulsar://127.0.0.1:6650"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
topic(required)

The Pulsar topic name to write events to.

TypeSyntaxDefaultExample
stringliteral["topic-1234"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["pulsar"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["pulsar"]
3endpoint = ["pulsar://127.0.0.1:6650"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6topic = ["topic-1234"]

Sematext Logs

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

region(required)

The region to send data to.

TypeSyntaxDefaultExample
stringliteral["us","eu"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
token(required)

The token that will be used to write to Sematext.

TypeSyntaxDefaultExample
stringliteral["${SEMATEXT_TOKEN}","some-sematext-token"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["sematext_logs"]

Advanced Options

endpoint(optional)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["http://127.0.0.1","http://example.com"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["sematext_logs"]
3region = ["us","eu"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6token = ["${SEMATEXT_TOKEN}","some-sematext-token"]

Sematext Metrics

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

region(required)

The region to send data to.

TypeSyntaxDefaultExample
stringliteral["us","eu"]
default_namespace(required)

Used as a namespace for metrics that don't have it.

TypeSyntaxDefaultExample
stringliteral["service"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
token(required)

The token that will be used to write to Sematext.

TypeSyntaxDefaultExample
stringliteral["${SEMATEXT_TOKEN}","some-sematext-token"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["sematext_metrics"]

Advanced Options

endpoint(optional)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["http://127.0.0.1","http://example.com"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["sematext_metrics"]
3region = ["us","eu"]
4default_namespace = ["service"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7token = ["${SEMATEXT_TOKEN}","some-sematext-token"]

Splunk HEC

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

The base URL of the Splunk instance.

TypeSyntaxDefaultExample
stringliteral["https://http-inputs-hec.splunkcloud.com","https://hec.splunk.com:8088","http://example.com"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
token(required)

Your Splunk HEC token.

TypeSyntaxDefaultExample
stringliteral["${SPLUNK_HEC_TOKEN}","A94A8FE5CCB19BA61C4C08"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["splunk_hec"]

Advanced Options

host_key(optional)

The name of the log field to be used as the hostname sent to Splunk HEC. This overrides the global host_key option.

TypeSyntaxDefaultExample
stringliteral["hostname"]
index(optional)

The name of the index where send the events to. If not specified, the default index is used.

TypeSyntaxDefaultExample
stringtemplate["{{ host }}","custom_index"]
indexed_fields(optional)

Fields to be added to Splunk index.

TypeSyntaxDefaultExample
arrayfield_path["field1","field2"]
source(optional)

The source of events sent to this sink. Typically the filename the logs originated from. If unset, the Splunk collector will set it.

TypeSyntaxDefaultExample
stringtemplate["{{ file }}","/var/log/syslog","UDP:514"]
sourcetype(optional)

The sourcetype of events sent to this sink. If unset, Splunk will default to httpevent.

TypeSyntaxDefaultExample
stringtemplate["{{ sourcetype }}","_json","httpevent"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["splunk_hec"]
3endpoint = [
4 "https://http-inputs-hec.splunkcloud.com",
5 "https://hec.splunk.com:8088",
6 "http://example.com"
7]
8inputs = ["my-source-or-transform-id","prefix-*"]
9encoding = []
10token = ["${SPLUNK_HEC_TOKEN}","A94A8FE5CCB19BA61C4C08"]
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

address(required)

The address to connect to. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["92.12.333.224:5000"]
mode(required)

The type of socket to use.

TypeSyntaxDefaultExample
stringliteral["tcp","udp","unix"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
path(required)

The unix socket path. This should be the absolute path.

TypeSyntaxDefaultExample
stringliteral["/path/to/socket"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["statsd","socket"]

Advanced Options

buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
send_buffer_bytes(optional)

Configures the send buffer size using the SO_SNDBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
default_namespace(optional)

Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name.

TypeSyntaxDefaultExample
stringliteral["service"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["statsd","socket"]
3address = ["92.12.333.224:5000"]
4mode = ["tcp","udp","unix"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7path = ["/path/to/socket"]
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

address(required)

The downstream Vector address to connect to. The address must include a port.

TypeSyntaxDefaultExample
stringliteral["92.12.333.224:6000"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["vector"]

Advanced Options

buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
send_buffer_bytes(optional)

Configures the send buffer size using the SO_SNDBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
version(optional)

Sink API version. Specifying this version ensures that Vector does not break backward compatibility.

TypeSyntaxDefaultExample
stringliteral1
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["vector"]
3address = ["92.12.333.224:6000"]
4inputs = ["my-source-or-transform-id","prefix-*"]

Honeycomb

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

api_key(required)

The team key that will be used to authenticate against Honeycomb.

TypeSyntaxDefaultExample
stringliteral["${HONEYCOMB_API_KEY}","some-api-key"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
dataset(required)

The dataset that Vector will send logs to.

TypeSyntaxDefaultExample
stringliteral["my-honeycomb-dataset"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["honeycomb"]

Advanced Options

buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["honeycomb"]
3api_key = ["${HONEYCOMB_API_KEY}","some-api-key"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6dataset = ["my-honeycomb-dataset"]

Datadog Metrics

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

api_key(required)

Datadog API key

TypeSyntaxDefaultExample
stringliteral["${DATADOG_API_KEY_ENV_VAR}","ef8d5de700e7989468166c40fc8a0ccd"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["datadog_metrics"]

Advanced Options

endpoint(optional)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["127.0.0.1:8080","example.com:12345"]
region(optional)

The region to send data to.

TypeSyntaxDefaultExample
stringliteral
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
default_namespace(optional)

Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name.

TypeSyntaxDefaultExample
stringliteral["service"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["datadog_metrics"]
3api_key = [
4 "${DATADOG_API_KEY_ENV_VAR}",
5 "ef8d5de700e7989468166c40fc8a0ccd"
6]
7inputs = ["my-source-or-transform-id","prefix-*"]
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

url(required)

The Redis URL to connect to. The url must take the form of protocol://server:port/db where the protocol can either be redis or rediss for connections secured via TLS.

TypeSyntaxDefaultExample
stringliteral["redis://127.0.0.1:6379/0"]
key(required)

The Redis key to publish messages to.

TypeSyntaxDefaultExample
stringtemplate["syslog:{{ app }}","vector"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["redis"]

Advanced Options

data_type(optional)

The Redis data type (list or channel) to use.

TypeSyntaxDefaultExample
stringliterallist
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
list(optional)

Options for the Redis list data type.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["redis"]
3url = ["redis://127.0.0.1:6379/0"]
4key = ["syslog:{{app}}","vector"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []

GCP Cloud Monitoring (formerly Stackdrive) Metrics

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

project_id(required)

The project ID to which to publish logs. See the Google Cloud Platform project management documentation for more details.

Exactly one of billing_account_id, folder_id, organization_id, or project_id must be set.

TypeSyntaxDefaultExample
stringliteral["vector-123456"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
resource(required)

Options for describing the logging resource.

TypeSyntaxDefaultExample
hashliteral[{"type":"global","projectId":"vector-123456","instanceId":"Twilight","zone":"us-central1-a"}]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["gcp_stackdriver_metrics"]

Advanced Options

credentials_path(optional)

The filename for a Google Cloud service account credentials JSON file used to authenticate access to the Stackdriver Logging API. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filename.

If no filename is named, Vector will attempt to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.

TypeSyntaxDefaultExample
stringliteral["/path/to/credentials.json"]
default_namespace(optional)

The namespace used if the metric we are going to send to GCP has no namespace.

TypeSyntaxDefaultExample
stringliteralnamespace["vector-123456"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = "global"
3type = ["gcp_stackdriver_metrics"]
4project_id = ["vector-123456"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7[[resource]]
8projectId = "vector-123456"
9instanceId = "Twilight"
10zone = "us-central1-a"

GCP Operations (formerly Stackdriver) Logs

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

log_id(required)

The log ID to which to publish logs. This is a name you create to identify this log stream.

TypeSyntaxDefaultExample
stringtemplate["vector-logs","{{ component_id }}"]
project_id(required)

The project ID to which to publish logs. See the Google Cloud Platform project management documentation for more details.

Exactly one of billing_account_id, folder_id, organization_id, or project_id must be set.

TypeSyntaxDefaultExample
stringliteral["vector-123456"]
resource(required)

Options for describing the logging resource.

TypeSyntaxDefaultExample
hashliteral[{"type":"gce_instance","instance_id":"Twilight","zone":"us-central1-a"}]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["gcp_stackdriver_logs"]

Advanced Options

billing_account_id(optional)

The billing account ID to which to publish logs.

Exactly one of must be set: billing_account_id, folder_id, organization_id, or project_id.

TypeSyntaxDefaultExample
stringliteral["012345-6789AB-CDEF01"]
credentials_path(optional)

The filename for a Google Cloud service account credentials JSON file used to authenticate access to the Stackdriver Logging API. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filename.

If no filename is named, Vector attempts to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.

TypeSyntaxDefaultExample
stringliteral["/path/to/credentials.json"]
folder_id(optional)

The folder ID to which to publish logs.

See the Google Cloud Platform folder documentation for more details.

Exactly one of billing_account_id, folder_id, organization_id, or project_id must be set.

TypeSyntaxDefaultExample
stringliteral["My Folder"]
organization_id(optional)

The organization ID to which to publish logs. This would be the identifier assigned to your organization on Google Cloud Platform.

Exactly one of billing_account_id, folder_id, organization_id, or project_id must be set.

TypeSyntaxDefaultExample
stringliteral["622418129737"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
severity_key(optional)

The field of the log event from which to take the outgoing log's severity field. The named field is removed from the log event if present, and must be either an integer between 0 and 800 or a string containing one of the severity level names (case is ignored) or a common prefix such as err.

If no severity key is specified, the severity of outgoing records is set to 0 (DEFAULT).

See the GCP Stackdriver Logging LogSeverity description for more details on the value of the severity field.

TypeSyntaxDefaultExample
stringliteral["severity"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = "gce_instance"
3type = ["gcp_stackdriver_logs"]
4log_id = ["vector-logs","{{component_id}}"]
5project_id = ["vector-123456"]
6inputs = ["my-source-or-transform-id","prefix-*"]
7encoding = []
8[[resource]]
9instance_id = "Twilight"
10zone = "us-central1-a"

GCP PubSub

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

project(required)

The project name to which to publish logs.

TypeSyntaxDefaultExample
stringliteral["vector-123456"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
topic(required)

The topic within the project to which to publish logs.

TypeSyntaxDefaultExample
stringliteral["this-is-a-topic"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["gcp_pubsub"]

Advanced Options

api_key(optional)

A Google Cloud API key used to authenticate access the pubsub project and topic. Either this or credentials_path must be set.

TypeSyntaxDefaultExample
stringliteral["${GCP_API_KEY}","ef8d5de700e7989468166c40fc8a0ccd"]
credentials_path(optional)

The filename for a Google Cloud service account credentials JSON file used to authenticate access to the pubsub project and topic. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filename.

If no filename is named, Vector will attempt to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.

TypeSyntaxDefaultExample
stringliteral["/path/to/credentials.json"]
endpoint(optional)

The endpoint to which to send data.

TypeSyntaxDefaultExample
stringliteralhttps://pubsub.googleapis.com["https://us-central1-pubsub.googleapis.com"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["gcp_pubsub"]
3project = ["vector-123456"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6topic = ["this-is-a-topic"]

GCP Cloud Storage (GCS)

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

bucket(required)

The GCS bucket name.

TypeSyntaxDefaultExample
stringliteral["my-bucket"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["gcp_cloud_storage"]

Advanced Options

acl(optional)

Predefined ACL to apply to the created objects. For more information, see Predefined ACLs. If this is not set, GCS will apply a default ACL when the object is created.

TypeSyntaxDefaultExample
stringliteral
credentials_path(optional)

The filename for a Google Cloud service account credentials JSON file used to authenticate access to the Cloud Storage API. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filename.

If no filename is named, Vector will attempt to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.

TypeSyntaxDefaultExample
stringliteral["/path/to/credentials.json"]
filename_append_uuid(optional)

Whether or not to append a UUID v4 token to the end of the file. This ensures there are no name collisions high volume use cases.

TypeSyntaxDefaultExample
bool
filename_extension(optional)

The filename extension to use in the object name.

TypeSyntaxDefaultExample
stringliterallog
filename_time_format(optional)

The format of the resulting object file name. strftime specifiers are supported.

TypeSyntaxDefaultExample
stringliteral%s
key_prefix(optional)

A prefix to apply to all object key names. This should be used to partition your objects, and it's important to end this value with a / if you want this to be the root GCS "folder".

TypeSyntaxDefaultExample
stringtemplatedate=%F/["date=%F/","date=%F/hour=%H/","year=%Y/month=%m/day=%d/","application_id={{ application_id }}/date=%F/"]
metadata(optional)

The set of metadata key:value pairs for the created objects. See the GCS custom metadata documentation for more details.

TypeSyntaxDefaultExample
stringliteral[]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
storage_class(optional)

The storage class for the created objects. See the GCP storage classes for more details.

TypeSyntaxDefaultExample
stringliteral
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["gcp_cloud_storage"]
3bucket = ["my-bucket"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
path(required)

File name to write events to.

TypeSyntaxDefaultExample
stringtemplate["/tmp/vector-%Y-%m-%d.log","/tmp/application-{{ application_id }}-%Y-%m-%d.log"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["file"]

Advanced Options

idle_timeout_secs(optional)

The amount of time a file can be idle and stay open. After not receiving any events for this timeout, the file will be flushed and closed.

TypeSyntaxDefaultExample
uint30
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["file"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5path = [
6 "/tmp/vector-%Y-%m-%d.log",
7 "/tmp/application-{{ application_id }}-%Y-%m-%d.log"
8]

Elasticsearch

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

The Elasticsearch endpoint to send logs to. This should be the full URL as shown in the example.

TypeSyntaxDefaultExample
stringliteral["http://10.24.32.122:9000","https://example.com","https://user:password@example.com"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["elasticsearch"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
aws(optional)

Options for the AWS connections.

TypeSyntaxDefaultExample
hashliteral[]
bulk_action(optional)

Action to use when making requests to the Elasticsearch Bulk API. Currently, Vector only supports index and create. update and delete actions are not supported.

TypeSyntaxDefaultExample
stringtemplateindex["index","create","{{ action }}"]
data_stream(optional)

Options for the data stream mode.

TypeSyntaxDefaultExample
hashtemplate[]
doc_type(optional)

The doc_type for your index data. This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.

TypeSyntaxDefaultExample
stringliteral_doc
id_key(optional)

The name of the event key that should map to Elasticsearch's _id field. By default, Vector does not set the _id field, which allows Elasticsearch to set this automatically. You should think carefully about setting your own Elasticsearch IDs, since this can hinder perofrmance.

TypeSyntaxDefaultExample
stringliteral["id","_id"]
index(optional)

Index name to write events to.

TypeSyntaxDefaultExample
stringtemplatevector-%F["application-{{ application_id }}-%Y-%m-%d","vector-%Y-%m-%d"]
metrics(optional)

Options for metrics.

TypeSyntaxDefaultExample
hashliteral[]
mode(optional)

The type of index mechanism. If data_stream mode is enabled, the bulk_action is set to create.

TypeSyntaxDefaultExample
stringliteralnormal["normal","data_stream"]
pipeline(optional)

Name of the pipeline to apply.

TypeSyntaxDefaultExample
stringliteral["pipeline-name"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
query(optional)

Custom parameters to Elasticsearch query string.

TypeSyntaxDefaultExample
hash[{"X-Powered-By":"Vector"}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["elasticsearch"]
3endpoint = [
4 "http://10.24.32.122:9000",
5 "https://example.com",
6 "https://user:password@example.com"
7]
8inputs = ["my-source-or-transform-id","prefix-*"]
9encoding = []

Datadog Logs

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

default_api_key(required)

Default Datadog API key, if an event has a key set in its metadata it will prevail over the one set here.

TypeSyntaxDefaultExample
stringliteral["${DATADOG_API_KEY_ENV_VAR}","ef8d5de700e7989468166c40fc8a0ccd"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["datadog_logs"]

Advanced Options

endpoint(optional)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["127.0.0.1:8080","example.com:12345"]
region(optional)

The region to send data to.

TypeSyntaxDefaultExample
stringliteral
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralgzip
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
site(optional)

The Datadog site to send data to.

TypeSyntaxDefaultExample
stringliteraldatadoghq.com["us3.datadoghq.com","datadoghq.com","datadoghq.eu"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["datadog_logs"]
3default_api_key = [
4 "${DATADOG_API_KEY_ENV_VAR}",
5 "ef8d5de700e7989468166c40fc8a0ccd"
6]
7inputs = ["my-source-or-transform-id","prefix-*"]
8encoding = []

Datadog Events

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

default_api_key(required)

Default Datadog API key, if an event has a key set in its metadata it will prevail over the one set here.

TypeSyntaxDefaultExample
stringliteral["${DATADOG_API_KEY_ENV_VAR}","ef8d5de700e7989468166c40fc8a0ccd"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["datadog_events"]

Advanced Options

endpoint(optional)

The endpoint to send data to.

TypeSyntaxDefaultExample
stringliteral["127.0.0.1:8080","example.com:12345"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
site(optional)

The Datadog site to send data to.

TypeSyntaxDefaultExample
stringliteraldatadoghq.com["us3.datadoghq.com","datadoghq.com","datadoghq.eu"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["datadog_events"]
3default_api_key = [
4 "${DATADOG_API_KEY_ENV_VAR}",
5 "ef8d5de700e7989468166c40fc8a0ccd"
6]
7inputs = ["my-source-or-transform-id","prefix-*"]

Console

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["console"]

Advanced Options

target(optional)

The standard stream to write to.

TypeSyntaxDefaultExample
stringliteralstdout
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["console"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []

Clickhouse

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

endpoint(required)

The endpoint of the Clickhouse server.

TypeSyntaxDefaultExample
stringliteral["http://localhost:8123"]
table(required)

The table that data will be inserted into.

TypeSyntaxDefaultExample
stringliteral["mytable"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["clickhouse"]

Advanced Options

auth(optional)

Configures the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
database(optional)

The database that contains the stable that data will be inserted into.

TypeSyntaxDefaultExample
stringliteral["mydatabase"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralgzip
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
skip_unknown_fields(optional)

Sets input_format_skip_unknown_fields, allowing Clickhouse to discard fields not present in the table schema.

TypeSyntaxDefaultExample
bool
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["clickhouse"]
3endpoint = ["http://localhost:8123"]
4table = ["mytable"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []

Blackhole

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["blackhole"]

Advanced Options

The number of events that must be received in order to print a summary of activity.

TypeSyntaxDefaultExample
uint1000[1000]
rate(optional)

Rates the amount of events that the sink can consume per second.

TypeSyntaxDefaultExample
uint[1000]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["blackhole"]
3inputs = ["my-source-or-transform-id","prefix-*"]

Azure Monitor Logs

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

customer_id(required)

The unique identifier for the Log Analytics workspace.

TypeSyntaxDefaultExample
stringliteral["5ce893d9-2c32-4b6c-91a9-b0887c2de2d6","97ce69d9-b4be-4241-8dbd-d265edcf06c4"]
log_type(required)

The record type of the data that is being submitted. Can only contain letters, numbers, and underscore (_), and may not exceed 100 characters.

TypeSyntaxDefaultExample
stringliteral["MyTableName","MyRecordType"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
shared_key(required)

The primary or the secondary key for the Log Analytics workspace.

TypeSyntaxDefaultExample
stringliteral["${AZURE_MONITOR_SHARED_KEY_ENV_VAR}","SERsIYhgMVlJB6uPsq49gCxNiruf6v0vhMYE+lfzbSGcXjdViZdV/e5pEMTYtw9f8SkVLf4LFlLCc2KxtRZfCA=="]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["azure_monitor_logs"]

Advanced Options

azure_resource_id(optional)

Resource ID of the Azure resource the data should be associated with.

TypeSyntaxDefaultExample
stringliteral["/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/otherResourceGroup/providers/Microsoft.Storage/storageAccounts/examplestorage","/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/examplegroup/providers/Microsoft.SQL/servers/serverName/databases/databaseName"]
host(optional)

Alternative host for dedicated Azure regions.

TypeSyntaxDefaultExample
stringliteralods.opinsights.azure.com["ods.opinsights.azure.us","ods.opinsights.azure.cn"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["azure_monitor_logs"]
3log_type = ["MyTableName","MyRecordType"]
4customer_id = [
5 "5ce893d9-2c32-4b6c-91a9-b0887c2de2d6",
6 "97ce69d9-b4be-4241-8dbd-d265edcf06c4"
7]
8inputs = ["my-source-or-transform-id","prefix-*"]
9encoding = []
10shared_key = [
11 "${AZURE_MONITOR_SHARED_KEY_ENV_VAR}",
12 "SERsIYhgMVlJB6uPsq49gCxNiruf6v0vhMYE+lfzbSGcXjdViZdV/e5pEMTYtw9f8SkVLf4LFlLCc2KxtRZfCA=="
13]

Azure Blob Storage

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

connection_string(required)

The Azure Blob Storage Account connection string. Only authentication with access key supported.

TypeSyntaxDefaultExample
stringliteral["DefaultEndpointsProtocol=https;AccountName=mylogstorage;AccountKey=storageaccountkeybase64encoded;EndpointSuffix=core.windows.net"]
container_name(required)

The Azure Blob Storage Account container name.

TypeSyntaxDefaultExample
stringliteral["my-logs"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["azure_blob"]

Advanced Options

blob_prefix(optional)

A prefix to apply to all object key names. This should be used to partition your objects, and it's important to end this value with a / if you want this to be the root azure storage "folder".

TypeSyntaxDefaultExample
stringtemplateblob/%F/["date/%F/","date/%F/hour/%H/","year=%Y/month=%m/day=%d/","kubernetes/{{ metadata.cluster }}/{{ metadata.application_name }}/"]
blob_append_uuid(optional)

Whether or not to append a UUID v4 token to the end of the file. This ensures there are no name collisions high volume use cases.

TypeSyntaxDefaultExample
bool
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralgzip
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
blob_time_format(optional)

The format of the resulting object file name. strftime specifiers are supported.

TypeSyntaxDefaultExample
stringstrftime%s
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["azure_blob"]
3connection_string = [
4 "DefaultEndpointsProtocol=https;AccountName=mylogstorage;AccountKey=storageaccountkeybase64encoded;EndpointSuffix=core.windows.net"
5]
6container_name = ["my-logs"]
7inputs = ["my-source-or-transform-id","prefix-*"]
8encoding = []
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

uri(required)

The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI.

TypeSyntaxDefaultExample
stringliteral["https://10.22.212.22:9000/endpoint"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["http"]

Advanced Options

auth(optional)

Configures the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["http"]
3uri = ["https://10.22.212.22:9000/endpoint"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

bucket(required)

The S3 bucket name. Do not include a leading s3:// or a trailing /.

TypeSyntaxDefaultExample
stringliteral["my-bucket"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_s3"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
acl(optional)

Canned ACL to apply to the created objects. For more information, see Canned ACL.

TypeSyntaxDefaultExample
stringliteral
content_encoding(optional)

Specifies what content encodings have been applied to the object and thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field. By default calculated from compression value.

TypeSyntaxDefaultExample
stringliteral["gzip"]
content_type(optional)

A standard MIME type describing the format of the contents.

TypeSyntaxDefaultExample
stringliteraltext/x-log
filename_append_uuid(optional)

Whether or not to append a UUID v4 token to the end of the file. This ensures there are no name collisions high volume use cases.

TypeSyntaxDefaultExample
bool
filename_extension(optional)

The filename extension to use in the object name.

TypeSyntaxDefaultExample
stringliterallog
filename_time_format(optional)

The format of the resulting object file name. strftime specifiers are supported.

TypeSyntaxDefaultExample
stringstrftime%s
grant_full_control(optional)

Gives the named grantee READ, READ_ACP, and WRITE_ACP permissions on the created objects.

TypeSyntaxDefaultExample
stringliteral["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"]
grant_read(optional)

Allows the named grantee to read the created objects and their metadata.

TypeSyntaxDefaultExample
stringliteral["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"]
grant_read_acp(optional)

Allows the named grantee to read the created objects' ACL.

TypeSyntaxDefaultExample
stringliteral["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"]
grant_write_acp(optional)

Allows the named grantee to write the created objects' ACL.

TypeSyntaxDefaultExample
stringliteral["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"]
key_prefix(optional)

A prefix to apply to all object key names. This should be used to partition your objects, and it's important to end this value with a / if you want this to be the root S3 "folder".

TypeSyntaxDefaultExample
stringtemplatedate=%F/["date=%F/","date=%F/hour=%H/","year=%Y/month=%m/day=%d/","application_id={{ application_id }}/date=%F/"]
server_side_encryption(optional)

The Server-side Encryption algorithm used when storing these objects.

TypeSyntaxDefaultExample
stringliteral
ssekms_key_id(optional)

If server_side_encryption has the value "aws.kms", this specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed customer master key (CMK) that will used for the created objects. If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.

TypeSyntaxDefaultExample
stringliteral["abcd1234"]
storage_class(optional)

The storage class for the created objects. See the S3 Storage Classes for more details.

TypeSyntaxDefaultExample
stringliteral
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralgzip
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
tags(optional)

The tag-set for the object.

TypeSyntaxDefaultExample
hash[{"Tag1":"Value1"}]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_s3"]
3bucket = ["my-bucket"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]

AWS Kinesis Data Streams

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
stream_name(required)

The stream name of the target Kinesis Logs stream.

TypeSyntaxDefaultExample
stringliteral["my-stream"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_kinesis_streams"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
partition_key_field(optional)

The log field used as the Kinesis record's partition key value.

TypeSyntaxDefaultExample
stringliteral["user_id"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_kinesis_streams"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5region = ["us-east-1"]
6stream_name = ["my-stream"]

AWS Cloudwatch Metrics

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

default_namespace(required)

A namespace that will isolate different metrics from each other. Used as a namespace for metrics that don't have it.

TypeSyntaxDefaultExample
stringliteral["service"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_cloudwatch_metrics"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_cloudwatch_metrics"]
3default_namespace = ["service"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5region = ["us-east-1"]

AWS Cloudwatch Logs

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

group_name(required)

The group name of the target CloudWatch Logs stream.

TypeSyntaxDefaultExample
stringtemplate["group-name","{{ file }}"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
stream_name(required)

The stream name of the target CloudWatch Logs stream. Note that there can only be one writer to a log stream at a time so if you are running multiple vectors all writing to the same log group, include a identifier in the stream name that is guaranteed to be unique by vector instance (for example, you might choose host)

TypeSyntaxDefaultExample
stringtemplate["{{ host }}","%Y-%m-%d","stream-name"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_cloudwatch_logs"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
create_missing_group(optional)

Dynamically create a log group if it does not already exist. This will ignore create_missing_stream directly after creating the group and will create the first stream.

TypeSyntaxDefaultExample
bool
create_missing_stream(optional)

Dynamically create a log stream if it does not already exist.

TypeSyntaxDefaultExample
bool
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_cloudwatch_logs"]
3group_name = ["group-name","{{file}}"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]
7stream_name = ["{{host}}","%Y-%m-%d","stream-name"]

Amazon Simple Queue Service (SQS)

  • Beta
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

queue_url(required)

The URL of the Amazon SQS queue to which messages are sent.

TypeSyntaxDefaultExample
stringliteral["https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_sqs"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
message_group_id(optional)

The tag that specifies that a message belongs to a specific message group. Can be applied only to FIFO queues.

TypeSyntaxDefaultExample
stringtemplate["vector","vector-%Y-%m-%d"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
message_deduplication_id(optional)

The message deduplication ID value to allow AWS to identify duplicate messages. This value is a template which should result in a unique string for each event.

See the AWS documentation for more about how AWS does message deduplication.

TypeSyntaxDefaultExample
stringtemplate["{{ transaction_id }}"]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_sqs"]
3queue_url = ["https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]

AWS Kinesis Firehose

  • Stable
  • Linux, Windows, MacOS

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.

Required Options

stream_name(required)

The stream name of the target Kinesis Firehose delivery stream.

TypeSyntaxDefaultExample
stringliteral["my-stream"]
inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

TypeSyntaxDefaultExample
arrayliteral["my-source-or-transform-id","prefix-*"]
encoding(required)

Configures the encoding specific sink behavior.

TypeSyntaxDefaultExample
hashliteral[]
region(required)

The AWS region of the target service. If endpoint is provided it will override this value since the endpoint includes the region.

TypeSyntaxDefaultExample
stringliteral["us-east-1"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["aws_kinesis_firehose"]

Advanced Options

auth(optional)

Options for the authentication strategy.

TypeSyntaxDefaultExample
hashliteral[]
endpoint(optional)

Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region moot.

TypeSyntaxDefaultExample
stringliteral["127.0.0.0:5000/path/to/service"]
buffer(optional)

Configures the sink specific buffer behavior.

TypeSyntaxDefaultExample
hashliteral[]
batch(optional)

Configures the sink batching behavior.

TypeSyntaxDefaultExample
hash[]
compression(optional)

The compression strategy used to compress the encoded event data before transmission.

Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.

TypeSyntaxDefaultExample
stringliteralnone
healthcheck(optional)

Health check options for the sink.

TypeSyntaxDefaultExample
hash[]
request(optional)

Configures the sink request behavior.

TypeSyntaxDefaultExample
hash[]
proxy(optional)

Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.

TypeSyntaxDefaultExample
hashliteral[]
vector.toml
SimpleAdvanced
1[sources.my_source_id]
2type = ["aws_kinesis_firehose"]
3stream_name = ["my-stream"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]