- Best Effort
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
include(required)
Array of file patterns to include. Globbing is supported.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["/var/log/**/*.log"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["file"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
exclude(optional)
Array of file patterns to exclude. Globbing is supported.Takes precedence over the include
option.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["/var/log/binary-file.log"] |
file_key(optional)
The key name added to each event with the full path of the file.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | file | ["file"] |
fingerprint(optional)
Configuration for how the file source should identify files.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
glob_minimum_cooldown_ms(optional)
Delay between file discovery calls. This controls the interval at which Vector searches for files. Higher value result in greater chances of some short living files being missed between searches, but lower value increases the performance impact of file discovery.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 1000 |
host_key(optional)
The key name added to each event representing the current host. This can also be globally set via the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
ignore_not_found(optional)
Ignore missing files when fingerprinting. This may be useful when used with source directories containing dangling symlinks.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
ignore_older_secs(optional)
Ignore files with a data modification date older than the specified number of seconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [600] |
line_delimiter(optional)
String sequence used to separate one file line from another
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["\r\n"] |
max_line_bytes(optional)
The maximum number of a bytes a line can contain before being discarded. This protects against malformed lines or tailing incorrect files.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 102400 |
max_read_bytes(optional)
An approximate limit on the amount of data read from a single file at a given time.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [2048] |
oldest_first(optional)
Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from younger files.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
remove_after_secs(optional)
Timeout from reaching eof
after which file will be removed from filesystem, unless new data is written in the meantime. If not specified, files will not be removed.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [0,5,60] |
read_from(optional)
In the absence of a checkpoint, this setting tells Vector where to start reading files that are present at startup.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | beginning |
multiline(optional)
Multiline parsing configuration. If not specified, multiline parsing is disabled.
Type | Syntax | Default | Example |
---|---|---|---|
hash | regex | [] |
data_dir(optional)
The directory used to persist file checkpoint positions. By default, the global data_dir
option is used. Please make sure the Vector project has write permissions to this dir.
Type | Syntax | Default | Example |
---|---|---|---|
string | file_system_path | ["/var/lib/vector"] |
encoding(optional)
Configures the encoding specific source behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
ignore_checkpoints(optional)
This causes Vector to ignore existing checkpoints when determining where to start reading a file. Checkpoints are still written normally.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["file"]
3include = ["/var/log/**/*.log"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
endpoints(required)
mod_status
endpoints to scrape metrics from.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["http://localhost:8080/server-status/?auto"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["apache_metrics"] |
Advanced Options
scrape_interval_secs(optional)
The interval between scrapes.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
namespace(optional)
The namespace of the metric. Disabled if empty.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | apache |
1[sources.my_source_id]
2type = ["apache_metrics"]
3endpoints = ["http://localhost:8080/server-status/?auto"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to accept connections on. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:9090"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["prometheus_remote_write"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
auth(optional)
Options for HTTP Basic Authentication.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["prometheus_remote_write"]
3address = ["0.0.0.0:9090"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to listen for connections on
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:443","localhost:443"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_kinesis_firehose"] |
Advanced Options
access_key(optional)
AWS Kinesis Firehose can be configured to pass along an access
key to authenticate requests. If configured, access_key
should
be set to the same value. If not specified, vector will treat
all requests as authenticated.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["A94A8FE5CCB19BA61C4C08"] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
record_compression(optional)
The compression of records within the Firehose message.
Some services, like AWS CloudWatch Logs, will compress the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.
Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | text |
1[sources.my_source_id]
2type = ["aws_kinesis_firehose"]
3address = ["0.0.0.0:443","localhost:443"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_s3"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
strategy(optional)
The strategy to use to consume objects from AWS S3.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | sqs |
compression(optional)
The compression format of the S3 objects..
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | text |
multiline(optional)
Multiline parsing configuration. If not specified, multiline parsing is disabled.
Type | Syntax | Default | Example |
---|---|---|---|
hash | regex | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
sqs(optional)
SQS strategy options. Required if strategy=sqs
.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["aws_s3"]
3region = ["us-east-1"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to accept connections on. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:80","localhost:80"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["datadog_agent"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
store_api_key(optional)
When incoming events contain a Datadog API key, if this setting is set to true
the key will kept in the event metadata and will be used if the event is sent to a Datadog sink.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["datadog_agent"]
3address = ["0.0.0.0:80","localhost:80"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_ecs_metrics"] |
Advanced Options
endpoint(optional)
Base URI of the task metadata endpoint.
If empty, the URI will be automatically discovered based on the latest version detected.
The version 2 endpoint base URI is 169.254.170.2/v2/
.
The version 3 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI
.
The version 4 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI_V4
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ${ECS_CONTAINER_METADATA_URI_V4} |
namespace(optional)
The namespace of the metric. Disabled if empty.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | awsecs |
scrape_interval_secs(optional)
The interval between scrapes, in seconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
version(optional)
The version of the metadata endpoint. If empty, the version is automatically discovered based on environment variables.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | v4 |
1[sources.my_source_id]
2type = ["aws_ecs_metrics"]
- Best Effort
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["docker_logs"] |
Advanced Options
docker_host(optional)
The Docker host to connect to. Use an HTTPS URL to enable TLS encryption.
If absent, Vector will try to use DOCKER_HOST
enviroment variable.
If DOCKER_HOST
is also absent, Vector will use default Docker local socket
(/var/run/docker.sock
on Unix flatforms, //./pipe/docker_engine
on Windows).
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://localhost:2375","https://localhost:2376","unix:///var/run/docker.sock","npipe:////./pipe/docker_engine","/var/run/docker.sock","//./pipe/docker_engine"] |
tls(optional)
TLS options to connect to the Docker daemon. This has no effect unless docker_host
is an HTTPS URL.
If absent, Vector will try to use environment variable DOCKER_CERT_PATH
and then DOCKER_CONFIG
.
If both environment variables are absent, Vector will try to read certificates in ~/.docker/
.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
auto_partial_merge(optional)
Setting this to false
will disable the automatic merging
of partial events.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
exclude_containers(optional)
A list of container IDs or names to match against for
containers you don't want to collect logs from. Prefix matches
are supported, so you can supply just the first few characters
of the ID or name of containers you want to exclude. This can be
used in conjunction with
include_containers
.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["exclude_","exclude_me_0","ad08cc418cf9"] |
include_containers(optional)
A list of container IDs or names to match against for
containers you want to collect logs from. Prefix matches are
supported, so you can supply just the first few characters of
the ID or name of containers you want to include. This can be
used in conjunction with
exclude_containers
.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["include_","include_me_0","ad08cc418cf9"] |
include_labels(optional)
A list of container object labels to match against when filtering running containers. This should follow the described label's syntax in docker object labels docs.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["com.example.vendor=Timber Inc.","com.example.name=Vector"] |
include_images(optional)
A list of image names to match against. If not provided, all images will be included.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["httpd","redis"] |
retry_backoff_secs(optional)
The amount of time to wait before retrying after an error.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 1 |
multiline(optional)
Multiline parsing configuration. If not specified, multiline parsing is disabled.
Type | Syntax | Default | Example |
---|---|---|---|
hash | regex | [] |
host_key(optional)
The key name added to each event representing the current host. This can also be globally set via the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
1[sources.my_source_id]
2type = ["docker_logs"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["eventstoredb_metrics"] |
Advanced Options
endpoints(optional)
Endpoints to scrape stats from.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | https://localhost:2113/stats | ["https://localhost:2113/stats"] |
scrape_interval_secs(optional)
The interval between scrapes, in seconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
default_namespace(optional)
The namespace used otherwise will be defaulted to eventstoredb.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | eventstoredb | ["app-123-eventstoredb"] |
1[sources.my_source_id]
2type = ["eventstoredb_metrics"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
mode(required)
The type of exec mechanism.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["scheduled","streaming"] |
command(required)
The command to be run, plus any arguments required.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | [["echo","Hello World!"],["ls","-la"]] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["exec"] |
Advanced Options
working_directory(optional)
The directory in which to run the command.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
include_stderr(optional)
Include the output of stderr when generating events.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
event_per_line(optional)
Determine if events should be generated per line or buffered and output as a single event when script execution finishes.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
maximum_buffer_size_bytes(optional)
The maximum buffer size allowed before a log event will be generated.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 1000000 |
scheduled(optional)
The scheduled options.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
streaming(optional)
The streaming options.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
1[sources.my_source_id]
2type = ["exec"]
3mode = ["scheduled","streaming"]
4command = [ ["echo","HelloWorld!"], ["ls","-la"]]
- Best Effort
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to listen for connections on, or systemd#N
to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:9000","systemd","systemd#3"] |
mode(required)
The type of socket to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tcp","udp","unix_datagram","unix_stream"] |
path(required)
The unix socket path. This should be an absolute path.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/socket"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["syslog","socket"] |
Advanced Options
host_key(optional)
The key name added to each event representing the current host. This can also be globally set via the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
max_length(optional)
The maximum bytes size of incoming messages before they are discarded.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 102400 |
receive_buffer_bytes(optional)
Configures the receive buffer size using the SO_RCVBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the source.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
shutdown_timeout_secs(optional)
The timeout before a connection is forcefully closed during shutdown.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 30 |
1[sources.my_source_id]
2type = ["syslog","socket"]
3address = ["0.0.0.0:9000","systemd","systemd#3"]
4mode = ["tcp","udp","unix_datagram","unix_stream"]
5path = ["/path/to/socket"]
- Best Effort
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to listen for TCP connections on.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:24224"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["fluent"] |
Advanced Options
receive_buffer_bytes(optional)
Configures the receive buffer size using the SO_RCVBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the source.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["fluent"]
3address = ["0.0.0.0:24224"]
- At Least Once
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
format(required)
The format of the randomly generated output.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["shuffle","apache_common","apache_error","syslog","bsd_syslog","json"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["generator"] |
Advanced Options
interval(optional)
The amount of time, in seconds, to pause between each batch of output lines. If not set, there is no delay.
Type | Syntax | Default | Example |
---|---|---|---|
float | [1,0.1,0.01] |
count(optional)
The total number of lines to output. By default the source continuously prints logs (infinitely).
Type | Syntax | Default | Example |
---|---|---|---|
uint |
lines(optional)
The list of lines to output.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["Line 1","Line 2"] |
sequence(optional)
If true
, each output line starts with an increasing sequence number, beginning with 0.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["generator"]
3format = [
4 "shuffle",
5 "apache_common",
6 "apache_error",
7 "syslog",
8 "bsd_syslog",
9 "json"
10]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to accept connections on. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:80","localhost:80"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["heroku_logs"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
auth(optional)
Options for HTTP Basic Authentication.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
query_parameters(optional)
A list of URL query parameters to include in the log event. These will override any values included in the body with conflicting names.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["application","source"] |
1[sources.my_source_id]
2type = ["heroku_logs"]
3address = ["0.0.0.0:80","localhost:80"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["host_metrics"] |
Advanced Options
collectors(optional)
The list of host metric collector services to use. Defaults to all collectors.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | cgroupscpudiskfilesystemloadhostmemorynetwork | ["cgroups","cpu","disk","filesystem","load","host","memory","network"] |
namespace(optional)
The namespace of metrics. Disabled if empty.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
scrape_interval_secs(optional)
The interval between metric gathering, in seconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
cgroups(optional)
Options for the "cgroups" (controller groups) metrics collector.
Note: this collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
disk(optional)
Options for the "disk" metrics collector.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
filesystem(optional)
Options for the "filesystem" metrics collector.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
network(optional)
Options for the "network" metrics collector.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["host_metrics"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to accept connections on. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:80","localhost:80"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
encoding(optional)
The expected encoding of received data. Note that for json
and ndjson
encodings, the fields of the JSON objects are output as separate fields.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | text |
headers(optional)
A list of HTTP headers to include in the log event. These will override any values included in the JSON payload with conflicting names.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["User-Agent","X-My-Custom-Header"] |
auth(optional)
Options for HTTP Basic Authentication.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
query_parameters(optional)
A list of URL query parameters to include in the log event. These will override any values included in the body with conflicting names.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["application","source"] |
path(optional)
The URL path on which log event POST requests shall be sent.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | / | ["/event/path","/logs"] |
strict_path(optional)
If set to true
, only requests using the exact URL path specified in path
will be accepted;
otherwise requests sent to a URL path that starts with the value of path
will be accepted.
With strict_path
set to false
and path
set to ""
, the configured HTTP source will
accept requests from any URL path.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
path_key(optional)
The event key in which the requested URL path used to send the request will be stored.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | path | ["vector_http_path"] |
1[sources.my_source_id]
2type = ["http"]
3address = ["0.0.0.0:80","localhost:80"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["internal_logs"] |
Advanced Options
host_key(optional)
The key name added to each event representing the current host. This can also be globally set via the
global host_key
option.
Set to "" to suppress this key.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
pid_key(optional)
The key name added to each event representing the current process ID.
Set to "" to suppress this key.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | pid |
1[sources.my_source_id]
2type = ["internal_logs"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["internal_metrics"] |
Advanced Options
namespace(optional)
The namespace of the metric.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | vector |
scrape_interval_secs(optional)
The interval between metric gathering, in seconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 2 |
tags(optional)
Metric tag options.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["internal_metrics"]
- At Least Once
- Stable
- Linux
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["journald"] |
Advanced Options
batch_size(optional)
The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 16 |
current_boot_only(optional)
Include only entries from the current boot.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
exclude_units(optional)
The list of unit names to exclude from monitoring. Unit names lacking a "."
will have ".service"
appended to make them a valid service unit name.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["badservice","sysinit.target"] |
include_units(optional)
The list of unit names to monitor. If empty or not present, all units are accepted. Unit names lacking a "."
will have ".service"
appended to make them a valid service unit name.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["ntpd","sysinit.target"] |
journalctl_path(optional)
The full path of the journalctl
executable. If not set, Vector will search the path for journalctl
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | journalctl | ["/usr/local/bin/journalctl"] |
data_dir(optional)
The directory used to persist file checkpoint positions. By default, the global data_dir
option is used. Please make sure the Vector project has write permissions to this dir.
Type | Syntax | Default | Example |
---|---|---|---|
string | file_system_path | ["/var/lib/vector"] |
journal_directory(optional)
The full path of the journal directory. If not set, journalctl
will use the default system journal paths
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/run/log/journal"] |
1[sources.my_source_id]
2type = ["journald"]
- At Least Once
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
bootstrap_servers(required)
A comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["10.14.22.123:9092,10.14.23.332:9092"] |
group_id(required)
The consumer group name to be used to consume events from Kafka.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["consumer-group-name"] |
topics(required)
The Kafka topics names to read events from. Regex is supported if the topic begins with ^
.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["^(prefix1|prefix2)-.+","topic-1","topic-2"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["kafka"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
auto_offset_reset(optional)
If offsets for consumer group do not exist, set them using this strategy. See the
librdkafka documentation for the auto.offset.reset
option for further
clarification.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | largest | ["smallest","earliest","beginning","largest","latest","end","error"] |
commit_interval_ms(optional)
The frequency that the consumer offsets are committed (written) to offset storage.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 5000 | [5000,10000] |
fetch_wait_max_ms(optional)
Maximum time the broker may wait to fill the response.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 100 | [50,100] |
key_field(optional)
The log field name to use for the Kafka message key.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message_key | ["message_key"] |
topic_key(optional)
The log field name to use for the Kafka topic.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | topic | ["topic"] |
partition_key(optional)
The log field name to use for the Kafka partition name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | partition | ["partition"] |
offset_key(optional)
The log field name to use for the Kafka offset.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | offset | ["offset"] |
headers_key(optional)
The log field name to use for the Kafka headers.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["headers"] |
librdkafka_options(optional)
Advanced options. See librdkafka documentation for details.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"client.id":"${ENV_VAR}","fetch.error.backoff.ms":"1000","socket.send.buffer.bytes":"100"}] |
sasl(optional)
Options for SASL/SCRAM authentication support.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
session_timeout_ms(optional)
The Kafka session timeout in milliseconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 10000 | [5000,10000] |
socket_timeout_ms(optional)
Default timeout for network requests.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 60000 | [30000,60000] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["kafka"]
3bootstrap_servers = ["10.14.22.123:9092,10.14.23.332:9092"]
4group_id = ["consumer-group-name"]
5topics = ["^(prefix1|prefix2)-.+","topic-1","topic-2"]
- Best Effort
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["kubernetes_logs"] |
Advanced Options
pod_annotation_fields(optional)
Configuration for how the events are annotated with Pod metadata.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
namespace_annotation_fields(optional)
Configuration for how the events are annotated with Namespace metadata.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
auto_partial_merge(optional)
Automatically merge partial messages into a single event. Partial here is in respect to messages that were split by the Kubernetes Container Runtime log driver.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
ingestion_timestamp_field(optional)
The exact time the event was ingested into Vector.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
kube_config_file(optional)
Optional path to a kubeconfig file readable by Vector. If not set, Vector will try to connect to Kubernetes using in-cluster configuration.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
self_node_name(optional)
The name of the Kubernetes Node
this Vector instance runs at. Configured to use an env var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ${VECTOR_SELF_NODE_NAME} |
exclude_paths_glob_patterns(optional)
A list of glob patterns to exclude from reading the files.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | **/*.gz**/*.tmp | ["**/exclude/**"] |
extra_field_selector(optional)
Specifies the field selector to filter Pod
s with, to be used in addition to the built-in Node
filter.
The name of the Kubernetes Node
this Vector instance runs at. Configured to use an env var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["metadata.name!=pod-name-to-exclude","metadata.name!=pod-name-to-exclude,metadata.name=mypod"] |
extra_label_selector(optional)
Specifies the label selector to filter Pod
s with, to be used in
addition to the built-in vector.dev/exclude
filter.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my_custom_label!=my_value","my_custom_label!=my_value,my_other_custom_label=my_value"] |
max_line_bytes(optional)
The maximum number of a bytes a line can contain before being discarded. This protects against malformed lines or tailing incorrect files.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 32768 |
fingerprint_lines(optional)
The number of lines to read when generating a unique fingerprint of a log file.
This is helpful when some containers share common first log lines.
WARNING: If the file has less than this amount of lines then it won't be read at all.
This is important since container logs are broken up into several files, so the greater
lines
value is, the greater the chance of it not reading the last file/logs of
the container.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 1 |
glob_minimum_cooldown_ms(optional)
Delay between file discovery calls. This controls the interval at which Vector searches for files within a single pod.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 60000 |
data_dir(optional)
The directory used to persist file checkpoint positions. By default, the global data_dir
option is used. Please make sure the Vector project has write permissions to this dir.
Type | Syntax | Default | Example |
---|---|---|---|
string | file_system_path | ["/var/lib/vector"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
1[sources.my_source_id]
2type = ["kubernetes_logs"]
- Best Effort
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to listen for TCP connections on.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:5044"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["logstash"] |
Advanced Options
receive_buffer_bytes(optional)
Configures the receive buffer size using the SO_RCVBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the source.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["logstash"]
3address = ["0.0.0.0:5044"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
endpoints(required)
MongoDB Connection String URI Format
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["mongodb://localhost:27017"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["mongodb_metrics"] |
Advanced Options
scrape_interval_secs(optional)
The interval between scrapes.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
namespace(optional)
The namespace of metrics. Disabled if empty.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | mongodb |
1[sources.my_source_id]
2type = ["mongodb_metrics"]
3endpoints = ["mongodb://localhost:27017"]
- Best Effort
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
url(required)
The NATS URL to connect to. The url must take the form of nats://server:port
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["nats://demo.nats.io","nats://127.0.0.1:4222"] |
subject(required)
The NATS subject to publish messages to.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ host }}","foo","time.us.east","time.*.east","time.>",">"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["nats"] |
Advanced Options
queue(optional)
NATS Queue Group to join
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | vector | ["foo","API Name Option Example"] |
connection_name(optional)
A name assigned to the NATS connection.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | vector | ["foo","API Name Option Example"] |
1[sources.my_source_id]
2type = ["nats"]
3url = ["nats://demo.nats.io","nats://127.0.0.1:4222"]
4subject = [
5 "{{ host }}",
6 "foo",
7 "time.us.east",
8 "time.*.east",
9 "time.>",
10 ">"
11]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The HTTP address to listen for connections on. It must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:9000"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector"] |
Advanced Options
acknowledgements(optional)
Controls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
shutdown_timeout_secs(optional)
The timeout before a connection is forcefully closed during shutdown.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 30 |
receive_buffer_bytes(optional)
Configures the receive buffer size using the SO_RCVBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the source.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
version(optional)
Source API version. Specifying this version ensures that Vector does not break backward compatibility.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | 1 |
1[sources.my_source_id]
2type = ["vector"]
3address = ["0.0.0.0:9000"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
endpoints(required)
HTTP/HTTPS endpoint to Nginx server with enabled ngx_http_stub_status_module
module.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["http://localhost:8000/basic_status"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["nginx_metrics"] |
Advanced Options
scrape_interval_secs(optional)
The interval between scrapes.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
namespace(optional)
The namespace of metrics. Disabled if empty.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | nginx |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
auth(optional)
Configures the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["nginx_metrics"]
3endpoints = ["http://localhost:8000/basic_status"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
endpoints(required)
PostgreSQL server endpoint in libpq-style connection strings.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["postgresql://postgres:vector@localhost:5432/postgres"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["postgresql_metrics"] |
Advanced Options
scrape_interval_secs(optional)
The interval between scrapes.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
namespace(optional)
The namespace of metrics. Disabled if empty.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | postgresql |
include_databases(optional)
A list of databases to match (by using POSIX Regular Expressions) against
the datname
column for which you want to collect metrics from. If not set, metrics are collected from
all databases. Specifying ""
will include metrics where datname
is NULL
. This can be used in
conjunction with exclude_databases
.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["^postgres$","^vector$","^foo"] |
exclude_databases(optional)
A list of databases to match (by using POSIX Regular Expressions) against
the datname
column for which you don't want to collect metrics from.
Specifying ""
will include metrics where datname
is NULL
.
This can be used in conjunction with include_databases
.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["^postgres$","^template.*",""] |
tls(optional)
TLS options to connect to the PostgreSQL Server.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["postgresql_metrics"]
3endpoints = ["postgresql://postgres:vector@localhost:5432/postgres"]
- At Least Once
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
endpoints(required)
Endpoints to scrape metrics from.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["http://localhost:9090/metrics"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["prometheus_scrape"] |
Advanced Options
scrape_interval_secs(optional)
The interval between scrapes, in seconds.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 15 |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
auth(optional)
Configures the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["prometheus_scrape"]
3endpoints = ["http://localhost:9090/metrics"]
- Best Effort
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to listen for connections on, or systemd#N
to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:9000","systemd","systemd#3"] |
mode(required)
The type of socket to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tcp","udp","unix_datagram","unix_stream"] |
path(required)
The unix socket path. This should be an absolute path.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/socket"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["socket"] |
Advanced Options
host_key(optional)
The key name added to each event representing the current host. This can also be globally set via the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
max_length(optional)
The maximum bytes size of incoming messages before they are discarded.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 102400 |
receive_buffer_bytes(optional)
Configures the receive buffer size using the SO_RCVBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the source.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
shutdown_timeout_secs(optional)
The timeout before a connection is forcefully closed during shutdown.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 30 |
1[sources.my_source_id]
2type = ["socket"]
3address = ["0.0.0.0:9000","systemd","systemd#3"]
4mode = ["tcp","udp","unix_datagram","unix_stream"]
5path = ["/path/to/socket"]
- At Least Once
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["splunk_hec"] |
Advanced Options
address(optional)
The address to accept connections on.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | 0.0.0.0:8080 |
token(optional)
If supplied, incoming requests must supply this token in the Authorization
header, just as a client would if it was communicating with the Splunk HEC endpoint directly. If not supplied, the Authorization
header will be ignored and requests will not be authenticated.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["A94A8FE5CCB19BA61C4C08"] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
valid_tokens(optional)
If supplied, incoming requests must supply one of these tokens in the Authorization
header, just as a client would if it was communicating with the Splunk HEC endpoint directly. If not supplied, the Authorization
header will be ignored and requests will not be authenticated.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["A94A8FE5CCB19BA61C4C08"] |
1[sources.my_source_id]
2type = ["splunk_hec"]
- Best Effort
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
address(required)
The address to listen for connections on, or systemd#N
to use the Nth socket passed by systemd socket activation. If an address is used it must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:8125","systemd","systemd#3"] |
mode(required)
The type of socket to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tcp","udp","unix"] |
path(required)
The unix socket path. This should be an absolute path.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/socket"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["statsd"] |
Advanced Options
receive_buffer_bytes(optional)
Configures the receive buffer size using the SO_RCVBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the source.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
shutdown_timeout_secs(optional)
The timeout before a connection is forcefully closed during shutdown.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 30 |
1[sources.my_source_id]
2type = ["statsd"]
3address = ["0.0.0.0:8125","systemd","systemd#3"]
4mode = ["tcp","udp","unix"]
5path = ["/path/to/socket"]
- At Least Once
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["stdin"] |
Advanced Options
host_key(optional)
The key name added to each event representing the current host. This can also be globally set via the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | host |
max_length(optional)
The maximum bytes size of a message before rest of it will be discarded.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 102400 |
1[sources.my_source_id]
2type = ["stdin"]
- Best Effort
- Beta
- Linux, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type.
Required Options
socket_path(required)
Absolute path of server socket file to which the DNS server is configured to send dnstap data. The socket file will be created by dnstap source component automatically upon startup.
Type | Syntax | Default | Example |
---|---|---|---|
string | file_system_path | ["/run/bind/dnstap.sock"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["dnstap"] |
Advanced Options
max_frame_length(optional)
Max dnstap frame length that the dnstap source can handle.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 102400 |
socket_file_mode(optional)
Unix file mode bits to be applied to server socket file
as its designated file permissions.
Note that the file mode value can be specified in any numeric format
supported by TOML, but it'd be more intuitive to use an octal number.
Also note that the value specified must be between 0o700
and 0o777
.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [511,492,508] |
socket_receive_buffer_size(optional)
Set receive buffer size of server Unix socket if specified. No change to the default size if omitted.
Type | Syntax | Default | Example |
---|---|---|---|
uint |
socket_send_buffer_size(optional)
Set send buffer size of server Unix socket if specified. No change to the default size if omitted.
Type | Syntax | Default | Example |
---|---|---|---|
uint |
raw_data_only(optional)
Whether or not to write out raw dnstap frame data directly (to be encoded in Base64) without any parsing and formatting.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["dnstap"]
3socket_path = ["/run/bind/dnstap.sock"]
- Deprecated
- Linux, Windows, MacOS
Splits a string field on a defined separator.
Splits a string field on a defined separator.
Required Options
field_names(required)
The field names assigned to the resulting tokens, in order.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["timestamp","level","message","parent.child"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["split"] |
Advanced Options
drop_field(optional)
If true
the field
will be dropped after parsing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The field to apply the split on.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child"] |
separator(optional)
The separator to split the field on. If no separator is given, it will split on all whitespace. 'Whitespace' is defined according to the terms of the Unicode Derived Core Property White_Space
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | [whitespace] | [","] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["split"]
3field_names = ["timestamp","level","message","parent.child"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Aggregates multiple metric events into a single metric event based
on a defined interval window. This helps to reduce metric volume at
the cost of granularity.
Aggregates multiple metric events into a single metric event based
on a defined interval window. This helps to reduce metric volume at
the cost of granularity.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aggregate"] |
Advanced Options
interval_ms(optional)
The interval over which metrics are aggregated in milliseconds. Over this period metrics with the same series data (name, namespace, tags, ...) will be aggregated.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 10000 |
1[sources.my_source_id]
2type = ["aggregate"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Strips ANSI escape sequences.
Strips ANSI escape sequences.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["ansi_stripper"] |
Advanced Options
field(optional)
The target field to strip ANSI escape sequences from.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child","array[0]"] |
1[sources.my_source_id]
2type = ["ansi_stripper"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch
subscriptions) coming from the aws_kinesis_firehose
source.
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch
subscriptions) coming from the aws_kinesis_firehose
source.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_cloudwatch_logs_subscription_parser"] |
Advanced Options
field(optional)
The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message |
1[sources.my_source_id]
2type = ["aws_cloudwatch_logs_subscription_parser"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Enriches events with AWS EC2 environment metadata.
Enriches events with AWS EC2 environment metadata.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_ec2_metadata"] |
Advanced Options
endpoint(optional)
Override the default EC2 Metadata endpoint.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | http://169.254.169.254 |
fields(optional)
A list of fields to include in each event.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | instance-idlocal-hostnamelocal-ipv4public-hostnamepublic-ipv4ami-idavailability-zonevpc-idsubnet-idregion | ["instance-id","local-hostname"] |
namespace(optional)
Prepend a namespace to each field's key.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["","ec2","aws.ec2"] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
refresh_interval_secs(optional)
The interval in seconds at which the EC2 Metadata api will be called.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 10 |
1[sources.my_source_id]
2type = ["aws_ec2_metadata"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Coerces log fields into typed values.
Coerces log fields into typed values.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["coercer"] |
Advanced Options
drop_unspecified(optional)
Set to true
to drop all fields that are not specified in the types
table. Make sure both message
and timestamp
are specified in the types
table as their absense will cause the original message data to be dropped along with other extraneous fields.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["coercer"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Defines an ordered chain of child tranforms that will be applied sequentially
on incoming events.
Defines an ordered chain of child tranforms that will be applied sequentially on incoming events.
Required Options
nested(required)
A table of transforms configurations' representing the chain of transforms to be applied on incoming
events. All transforms in the chain can then be referenced as an input by other components with the name
<transform_name>.<nested_transform_name>
. All transforms in the chain also generate internal metrics
as if they were configured separately.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["compound"] |
Advanced Options
1[sources.my_source_id]
2type = ["compound"]
3nested = []
4inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Slices log string fields and joins them into a single field.
Slices log string fields and joins them into a single field.
Required Options
items(required)
A list of substring definitons in the format of source_field[start..end]. For both start and end negative values are counted from the end of the string.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["first[..3]","second[-5..]","third[3..6]"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
target(required)
The name for the new label.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["root_field_name","parent.child","array[0]"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["concat"] |
Advanced Options
joiner(optional)
The string that is used to join all items.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | [" ",",","_","+"] |
1[sources.my_source_id]
2type = ["concat"]
3items = [ "first[..3]", "second[-5..]", "third[3..6]"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5target = [ "root_field_name", "parent.child", "array[0]"]
- Stable
- Linux, Windows, MacOS
Deduplicates events to reduce data volume by eliminating copies of data.
Deduplicates events to reduce data volume by eliminating copies of data.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
fields(required)
Options controlling what fields to match against.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["dedupe"] |
Advanced Options
cache(optional)
Options controlling how we cache recent Events for future duplicate checking.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
1[sources.my_source_id]
2type = ["dedupe"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4fields = []
- Deprecated
- Linux, Windows, MacOS
Removes one or more metric tags.
Removes one or more metric tags.
Required Options
tags(required)
The tag names to drop.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["tag1","tag2"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["remove_tags"] |
Advanced Options
1[sources.my_source_id]
2type = ["remove_tags"]
3tags = ["tag1","tag2"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Enrich events with geolocation data from the MaxMind GeoIP2-City,
GeoLite2-City, GeoIP2-ISP and GeoLite2-ASN databases.
Enrich events with geolocation data from the MaxMind GeoIP2-City, GeoLite2-City, GeoIP2-ISP and GeoLite2-ASN databases.
Required Options
database(required)
Path to the MaxMind GeoIP2 or GeoLite2 binary city
database file (GeoLite2-City.mmdb
). Other
databases, such as the the country database, are not supported.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/GeoLite2-City.mmdb","/path/to/GeoLite2-ISP.mmdb"] |
source(required)
The field name that contains the IP address. This field should contain a valid IPv4 or IPv6 address.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["ip_address","x-forwarded-for","parent.child","array[0]"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["geoip"] |
Advanced Options
target(optional)
The default field to insert the resulting GeoIP data into. See output for more info.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | geoip | ["geoip","parent.child"] |
1[sources.my_source_id]
2type = ["geoip"]
3database = ["/path/to/GeoLite2-City.mmdb","/path/to/GeoLite2-ISP.mmdb"]
4source = [ "ip_address", "x-forwarded-for", "parent.child", "array[0]"]
5inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Parses a log field value with Grok.
Parses a log field value with Grok.
Required Options
pattern(required)
The Grok pattern
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["grok_parser"] |
Advanced Options
drop_field(optional)
If true
will drop the specified field
after parsing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The log field to execute the pattern
against. Must be a string
value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child","array[0]"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["grok_parser"]
3pattern = [
4 "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
5]
6inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Parses a log field value as JSON.
Parses a log field value as JSON.
Required Options
drop_invalid(required)
If true
events with invalid JSON will be dropped, otherwise the event will be kept and passed through.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["json_parser"] |
Advanced Options
drop_field(optional)
If the specified field
should be dropped (removed) after parsing. If parsing fails, the field will not be removed, irrespective of this setting.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The log field to decode as JSON. Must be a string
value type.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child","array[0]"] |
overwrite_target(optional)
If target_field
is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true
.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
target_field(optional)
If this setting is present, the parsed JSON will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["root_field","parent.child"] |
1[sources.my_source_id]
2type = ["json_parser"]
3drop_invalid = ""
4inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Loosely parses a log field's value in key-value format.
Loosely parses a log field's value in key-value format.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["key_value_parser"] |
Advanced Options
drop_field(optional)
If true
will drop the specified field
after parsing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The log field containing key/value pairs to parse. Must be a string
value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child","array[0]"] |
field_split(optional)
The character(s) to split a key/value pair on which results in a new field with an associated value. Must be a string
value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | = | [":","="] |
overwrite_target(optional)
If target_field
is set and the log contains a field of the same name
as the target, it will only be overwritten if this is set to true
.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
separator(optional)
The character(s) that separate key/value pairs. Must be a string
value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | [whitespace] | [",",";","|"] |
target_field(optional)
If this setting is present, the parsed JSON will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["root_field","parent.child"] |
trim_key(optional)
Removes characters from the beginning and end of a key until a character that is not listed.
ex: <key>=value
would result in key: value
with this option set to <>
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["<>","{}"] |
trim_value(optional)
Removes characters from the beginning and end of a value until a character that is not listed.
ex: key=<<>value>>
would result in key: value
with this option set to <>
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["<>","{}"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["key_value_parser"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Derives one or more metric events from a log event.
Derives one or more metric events from a log event.
Required Options
metrics(required)
A table of key/value pairs representing the keys to be added to the event.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | [] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["log_to_metric"] |
Advanced Options
1[sources.my_source_id]
2type = ["log_to_metric"]
3metrics = []
4inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Parses a log field's value in the logfmt format.
Parses a log field's value in the logfmt format.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["logfmt_parser"] |
Advanced Options
drop_field(optional)
If the specified field
should be dropped (removed) after parsing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The log field to parse.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child","array[0]"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["logfmt_parser"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Transform events with a full embedded Lua 5.4 engine.
Transform events with a full embedded Lua 5.4 engine.
Required Options
hooks(required)
Configures hooks handlers.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
version(required)
Transform API version. Specifying this version ensures that Vector does not break backward compatibility.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["1","2"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["lua"] |
Advanced Options
search_dirs(optional)
A list of directories to search when loading a Lua file via the require
function. If not specified, the modules are looked up in the directories of Vector's configs.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["/etc/vector/lua"] |
source(optional)
The source which is evaluated when the transform is created.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t\tsyntax: \"literal\"\n\t\t}\n \t}\nend","-- external file with hooks and timers defined\nrequire('custom_module')"] |
timers(optional)
Configures timers which are executed periodically at given interval.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | [] |
1[sources.my_source_id]
2type = ["lua"]
3hooks = []
4inputs = ["my-source-or-transform-id","prefix-*"]
5version = ["1","2"]
- Deprecated
- Linux, Windows, MacOS
Merges partial log events into a single event.
Merges partial log events into a single event.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["merge"] |
Advanced Options
fields(optional)
Fields to merge. The values of these fields will be merged into the first partial event. Fields not specified here will be ignored. Merging process takes the first partial event and the base, then it merges in the fields from each successive partial event, until a non-partial event arrives. Finally, the non-partial event fields are merged in, producing the resulting merged event.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | message | ["message","parent.child"] |
partial_event_marker_field(optional)
The field that indicates that the event is partial. A consequent stream of partial events along with the first non-partial event will be merged together.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | _partial | ["_partial","parent.child"] |
stream_discriminant_fields(optional)
An ordered list of fields to distinguish streams by. Each stream has a separate partial event merging state. Should be used to prevent events from unrelated sources from mixing together, as this affects partial event processing.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["host","parent.child"] |
1[sources.my_source_id]
2type = ["merge"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Converts a metric event into a log event, which can be useful for sending metrics
to log-support downstream components.
Converts a metric event into a log event, which can be useful for sending metrics to log-support downstream components.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["metric_to_log"] |
Advanced Options
host_tag(optional)
Tag key that identifies the source host.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | hostname | ["host","hostname"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
1[sources.my_source_id]
2type = ["metric_to_log"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Reduces multiple log events into a single log event based on a set of
conditions and merge strategies.
Reduces multiple log events into a single log event based on a set of conditions and merge strategies.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["reduce"] |
Advanced Options
ends_when(optional)
A condition used to distinguish the final event of a transaction. If this condition resolves to true
for an event, the current transaction is immediately flushed with this event.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | [".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"] |
expire_after_ms(optional)
A maximum period of time to wait after the last event is received before a combined event should be considered complete.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 30000 |
flush_period_ms(optional)
Controls the frequency that Vector checks for (and flushes) expired events.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 1000 |
group_by(optional)
An ordered list of fields by which to group events. Each group is combined independently, allowing you to keep independent events separate. When no fields are specified, all events will be combined in a single group. Events missing a specified field will be combined in their own group.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["request_id","user_id","transaction_id"] |
merge_strategies(optional)
A map of field names to custom merge strategies. For each field specified this strategy will be used for combining events rather than the default behavior.
The default behavior is as follows:
- The first value of a string field is kept, subsequent values are discarded.
- For timestamp fields the first is kept and a new field
[field-name]_end
is added with the last received timestamp value. - Numeric values are summed.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [{"method":"discard","path":"discard","duration_ms":"sum","query":"array"}] |
starts_when(optional)
A condition used to distinguish the first event of a transaction. If this condition resolves to true
for an event, the previous transaction is flushed (without this event) and a new transaction is started.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | [".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"] |
1[sources.my_source_id]
2type = ["reduce"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Limits the cardinality of tags on metric events, protecting against
accidental high cardinality usage that can commonly disrupt the stability
of metrics storages.
Limits the cardinality of tags on metric events, protecting against accidental high cardinality usage that can commonly disrupt the stability of metrics storages.
Required Options
mode(required)
Controls what approach is used internally to keep track of previously seen tags and deterime when a tag on an incoming metric exceeds the limit.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["exact","probabilistic"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tag_cardinality_limit"] |
Advanced Options
cache_size_per_tag(optional)
The size of the cache in bytes to use to detect duplicate tags. The bigger the cache the less likely it is to have a 'false positive' or a case where we allow a new value for tag even after we have reached the configured limits.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 5120000 |
limit_exceeded_action(optional)
Controls what should happen when a metric comes in with a tag that would exceed the configured limit on cardinality.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | drop_tag |
value_limit(optional)
How many distinct values to accept for any given key.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 500 |
1[sources.my_source_id]
2type = ["tag_cardinality_limit"]
3mode = ["exact","probabilistic"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Tokenizes a field's value by splitting on white space, ignoring special
wrapping characters, and zip the tokens into ordered field names.
Tokenizes a field's value by splitting on white space, ignoring special wrapping characters, and zip the tokens into ordered field names.
Required Options
field_names(required)
The log field names assigned to the resulting tokens, in order.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["timestamp","level","message","parent.child"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tokenizer"] |
Advanced Options
drop_field(optional)
If true
the field
will be dropped after parsing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The log field to tokenize.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["tokenizer"]
3field_names = ["timestamp","level","message","parent.child"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Samples events at a configurable rate.
Samples events at a configurable rate.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
rate(required)
The rate at which events will be forwarded, expressed as 1/N. For example,
rate = 10
means 1 out of every 10 events will be forwarded and the rest will be dropped.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [10] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["sample"] |
Advanced Options
key_field(optional)
The name of the log field whose value will be hashed to determine if the event should be passed.
Consistently samples the same events. Actual rate of sampling may differ from the configured one if
values in the field are not uniformly distributed. If left unspecified, or if the event doesn't have
key_field
, events will be count rated.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["message"] |
exclude(optional)
The set of logical conditions to exclude events from sampling.
Type | Syntax | Default | Example |
---|---|---|---|
string | remap_boolean_expression | [".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"] |
1[sources.my_source_id]
2type = ["sample"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4rate = [10]
- Deprecated
- Linux, Windows, MacOS
Parses a log field's value with a Regular Expression.
Parses a log field's value with a Regular Expression.
Required Options
patterns(required)
The Regular Expressions to apply. Do not include the leading or trailing /
in any of the expressions.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["^(?P<timestamp>[\\\\w\\\\-:\\\\+]+) (?P<level>\\\\w+) (?P<message>.*)$"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["regex_parser"] |
Advanced Options
drop_failed(optional)
If the event should be dropped if parsing fails.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
drop_field(optional)
If the specified field
should be dropped (removed) after parsing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
field(optional)
The log field to parse.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | message | ["message","parent.child"] |
overwrite_target(optional)
If target_field
is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true
.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
target_field(optional)
If this setting is present, the parsed fields will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["root_field","parent.child"] |
timezone(optional)
The name of the time zone to apply to timestamp conversions that do not contain an explicit time
zone. This overrides the global timezone
option.
The time zone name may be any name in the TZ database, or local
to
indicate system local time.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | local | ["local","America/NewYork","EST5EDT"] |
types(optional)
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|
, for example
"timestamp|%F"
. Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format specifiers from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format | Description | Example |
---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone | 2020-12-01T02:37:54 |
%a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 |
%a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"status":"int","duration":"float","success":"bool","timestamp_iso8601":"timestamp|%F","timestamp_custom":"timestamp|%a %b %e %T %Y","timestamp_unix":"timestamp|%F %T","parent":{"child":"int"}}] |
1[sources.my_source_id]
2type = ["regex_parser"]
3patterns = [
4 "^(?P<timestamp>[\\\\w\\\\-:\\\\+]+) (?P<level>\\\\w+) (?P<message>.*)$"
5]
6inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Adds tags to metric events.
Adds tags to metric events.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["add_tags"] |
Advanced Options
overwrite(optional)
By default, fields will be overridden. Set this to false
to avoid overwriting values.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
tags(optional)
A table of key/value pairs representing the tags to be added to the metric.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"static_tag":"my value","env_tag":"${ENV_VAR}"}] |
1[sources.my_source_id]
2type = ["add_tags"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Adds fields to log events.
Adds fields to log events.
Required Options
fields(required)
A table of key/value pairs representing the keys to be added to the event.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"string_field":"string value","env_var_field":"${ENV_VAR}","templated_field":"{{ my_other_field }}","int_field":1,"float_field":1.2,"bool_field":true,"timestamp_field":"1979-05-27T00:32:00-0700","parent":{"child_field":"child_value"},"list_field":["first","second","third"]}] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["add_fields"] |
Advanced Options
overwrite(optional)
By default, fields will be overridden. Set this to false
to avoid overwriting values.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["add_fields"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4[[fields]]
5string_field = "string value"
6env_var_field = "${ENV_VAR}"
7templated_field = "{{ my_other_field }}"
8int_field = 1
9float_field = 1.2
10bool_field = true
11timestamp_field = "1979-05-27T00:32:00-0700"
12list_field = ["first","second","third"]
13 [fields.parent]
14 child_field = "child_value"
- Beta
- Linux, Windows, MacOS
Is the recommended transform for parsing, shaping, and transforming data in Vector. It implements the
Vector Remap Language (VRL), an expression-oriented language designed for processing
observability data (logs and metrics) in a safe and performant manner.
Please refer to the VRL reference when writing VRL scripts.
Is the recommended transform for parsing, shaping, and transforming data in Vector. It implements the Vector Remap Language (VRL), an expression-oriented language designed for processing observability data (logs and metrics) in a safe and performant manner.
Please refer to the VRL reference when writing VRL scripts.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["remap"] |
Advanced Options
source(optional)
The Vector Remap Language (VRL) program to execute for each event.
Required if file
is missing.
Type | Syntax | Default | Example |
---|---|---|---|
string | remap_program | [". = parse_json!(.message)\n.new_field = \"new value\"\n.status = to_int!(.status)\n.duration = parse_duration!(.duration, \"s\")\n.new_name = del(.old_name)"] |
file(optional)
File path to the Vector Remap Language (VRL) program to execute for each event.
If a relative path is provided, its root is the current working directory.
Required if source
is missing.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["./my/program.vrl"] |
drop_on_error(optional)
Drop the event if the VRL program returns an error at runtime.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
drop_on_abort(optional)
Drop the event if the VRL program is manually aborted through the abort
statement.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["remap"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Removes one or more log fields.
Removes one or more log fields.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
fields(required)
The log field names to drop.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["field1","field2","parent.child"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["remove_fields"] |
Advanced Options
drop_empty(optional)
If set to true
, after removing fields, remove any parent objects that are now empty.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["remove_fields"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4fields = ["field1","field2","parent.child"]
- Stable
- Linux, Windows, MacOS
Splits a stream of events into multiple sub-streams based on a set of
conditions.
Splits a stream of events into multiple sub-streams based on a set of conditions.
Required Options
route(required)
A table of route identifiers to logical conditions representing the filter of the route. Each route
can then be referenced as an input by other components with the name <transform_name>.<route_id>
.
Type | Syntax | Default | Example |
---|---|---|---|
hash | remap_boolean_expression | [] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["route"] |
Advanced Options
1[sources.my_source_id]
2type = ["route"]
3route = []
4inputs = ["my-source-or-transform-id","prefix-*"]
- Deprecated
- Linux, Windows, MacOS
Renames one or more log fields.
Renames one or more log fields.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
fields(required)
A table of old-key/new-key pairs representing the keys to be moved in the event.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"old_field_name":"new_field_name","parent":{"old_child_name":"parent.new_child_name"}}] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["rename_fields"] |
Advanced Options
drop_empty(optional)
If set to true
, after renaming fields, remove any parent objects of the old field that are now empty.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["rename_fields"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4[[fields]]
5old_field_name = "new_field_name"
6 [fields.parent]
7 old_child_name = "parent.new_child_name"
- Stable
- Linux, Windows, MacOS
Filters events based on a set of conditions.
Filters events based on a set of conditions.
Required Options
condition(required)
The condition to be matched against every input event. Only messages that pass the condition will be forwarded.
Type | Syntax | Default | Example |
---|---|---|---|
string | remap_boolean_expression | [".status_code != 200 && !includes([\"info\", \"debug\"], .severity)"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["filter"] |
Advanced Options
1[sources.my_source_id]
2type = ["filter"]
3condition = [
4 ".status_code != 200 && !includes([\"info\",\"debug\"], .severity)"
5]
6inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
address(required)
The address to connect to. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["92.12.333.224:5000"] |
mode(required)
The type of socket to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tcp","udp","unix"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
path(required)
The unix socket path. This should be the absolute path.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/socket"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["socket"] |
Advanced Options
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
send_buffer_bytes(optional)
Configures the send buffer size using the SO_SNDBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["socket"]
3address = ["92.12.333.224:5000"]
4mode = ["tcp","udp","unix"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7path = ["/path/to/socket"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
token(required)
Your Humio ingestion token.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["humio_logs"] |
Advanced Options
endpoint(optional)
The base URL of the Humio instance.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | https://cloud.humio.com | ["http://127.0.0.1","http://example.com"] |
event_type(optional)
The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.
If unset, Humio will default it to none.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["json","none"] |
host_key(optional)
The name of the log field to be used as the hostname sent to Humio. This overrides the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["hostname"] |
source(optional)
The source of events sent to this sink. Typically the filename the logs originated from. Maps to
@source
in Humio.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{file}}","/var/log/syslog","UDP:514"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["humio_logs"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5token = ["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
token(required)
Your Humio ingestion token.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["humio_metrics"] |
Advanced Options
endpoint(optional)
The base URL of the Humio instance.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | https://cloud.humio.com | ["http://127.0.0.1","http://example.com"] |
event_type(optional)
The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.
If unset, Humio will default it to none.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["json","none"] |
host_key(optional)
The name of the log field to be used as the hostname sent to Humio. This overrides the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["hostname"] |
source(optional)
The source of events sent to this sink. Typically the filename the logs originated from. Maps to
@source
in Humio.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{file}}","/var/log/syslog","UDP:514"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["humio_metrics"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5token = ["${HUMIO_TOKEN}","A94A8FE5CCB19BA61C4C08"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
bucket(required)
The destination bucket for writes into InfluxDB 2.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-bucket","4d2225e4d3d49f75"] |
database(required)
Sets the target database for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-database","iot-store"] |
endpoint(required)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://localhost:8086/","https://us-west-2-1.aws.cloud1.influxdata.com","https://us-west-2-1.aws.cloud2.influxdata.com"] |
org(required)
Specifies the destination organization for writes into InfluxDB 2.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-org","33f2cff0a28e5b63"] |
token(required)
Authentication token for InfluxDB 2.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"] |
namespace(required)
A prefix that will be added to all logs names.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["influxdb_logs"] |
Advanced Options
consistency(optional)
Sets the write consistency for the point for InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["any","one","quorum","all"] |
password(optional)
Sets the password for authentication if you’ve enabled authentication for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${INFLUXDB_PASSWORD}","influxdb4ever"] |
retention_policy_name(optional)
Sets the target retention policy for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["autogen","one_day_only"] |
tags(optional)
A set of additional fields that will be attached to each LineProtocol as a tag. Note: If the set of tag values has high cardinality this also increase cardinality in InfluxDB.
Type | Syntax | Default | Example |
---|---|---|---|
array | field_path | ["field1","parent.child_field"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
username(optional)
Sets the username for authentication if you’ve enabled authentication for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["todd","vector-source"] |
1[sources.my_source_id]
2type = ["influxdb_logs"]
3bucket = ["vector-bucket","4d2225e4d3d49f75"]
4database = ["vector-database","iot-store"]
5endpoint = [
6 "http://localhost:8086/",
7 "https://us-west-2-1.aws.cloud1.influxdata.com",
8 "https://us-west-2-1.aws.cloud2.influxdata.com"
9]
10org = ["my-org","33f2cff0a28e5b63"]
11token = ["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"]
12namespace = ["service"]
13inputs = ["my-source-or-transform-id","prefix-*"]
14encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
bucket(required)
The destination bucket for writes into InfluxDB 2.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-bucket","4d2225e4d3d49f75"] |
database(required)
Sets the target database for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-database","iot-store"] |
endpoint(required)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://localhost:8086/","https://us-west-2-1.aws.cloud1.influxdata.com","https://us-west-2-1.aws.cloud2.influxdata.com"] |
org(required)
Specifies the destination organization for writes into InfluxDB 2.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-org","33f2cff0a28e5b63"] |
token(required)
Authentication token for InfluxDB 2.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["influxdb_metrics"] |
Advanced Options
consistency(optional)
Sets the write consistency for the point for InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["any","one","quorum","all"] |
password(optional)
Sets the password for authentication if you’ve enabled authentication for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${INFLUXDB_PASSWORD}","influxdb4ever"] |
retention_policy_name(optional)
Sets the target retention policy for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["autogen","one_day_only"] |
tags(optional)
A set of additional fields that will be attached to each LineProtocol as a tag. Note: If the set of tag values has high cardinality this also increase cardinality in InfluxDB.
Type | Syntax | Default | Example |
---|---|---|---|
array | field_path | ["field1","parent.child_field"] |
default_namespace(optional)
Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
username(optional)
Sets the username for authentication if you’ve enabled authentication for the write into InfluxDB 1.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["todd","vector-source"] |
1[sources.my_source_id]
2type = ["influxdb_metrics"]
3bucket = ["vector-bucket","4d2225e4d3d49f75"]
4database = ["vector-database","iot-store"]
5endpoint = [
6 "http://localhost:8086/",
7 "https://us-west-2-1.aws.cloud1.influxdata.com",
8 "https://us-west-2-1.aws.cloud2.influxdata.com"
9]
10org = ["my-org","33f2cff0a28e5b63"]
11token = ["${INFLUXDB_TOKEN}","ef8d5de700e7989468166c40fc8a0ccd"]
12inputs = ["my-source-or-transform-id","prefix-*"]
13encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
bootstrap_servers(required)
A comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["10.14.22.123:9092,10.14.23.332:9092"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
topic(required)
The Kafka topic name to write events to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["topic-1234","logs-{{unit}}-%Y-%m-%d"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["kafka"] |
Advanced Options
key_field(optional)
The log field name or tags key to use for the topic key. If the field does not exist in the log or in tags, a blank value will be used. If unspecified, the key is not sent. Kafka uses a hash of the key to choose the partition or uses round-robin if the record has no key.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["user_id"] |
librdkafka_options(optional)
Advanced options. See librdkafka documentation for details.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"client.id":"${ENV_VAR}","fetch.error.backoff.ms":"1000","socket.send.buffer.bytes":"100"}] |
message_timeout_ms(optional)
Local message timeout.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 300000 | [150000,450000] |
sasl(optional)
Options for SASL/SCRAM authentication support.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
socket_timeout_ms(optional)
Default timeout for network requests.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 60000 | [30000,60000] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["kafka"]
3bootstrap_servers = ["10.14.22.123:9092,10.14.23.332:9092"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6topic = ["topic-1234","logs-{{unit}}-%Y-%m-%d"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
api_key(required)
The Ingestion API key.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${LOGDNA_API_KEY}","ef8d5de700e7989468166c40fc8a0ccd"] |
hostname(required)
The hostname that will be attached to each batch of events.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${HOSTNAME}","my-local-machine"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["logdna"] |
Advanced Options
default_app(optional)
The default app that will be set for events that do not contain a file
or app
field.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | vector | ["vector","myapp"] |
default_env(optional)
The default environment that will be set for events that do not contain an env
field.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | production | ["staging","production"] |
endpoint(optional)
The endpoint to send logs to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | https://logs.logdna.com/logs/ingest | ["http://127.0.0.1","http://example.com"] |
ip(optional)
The IP address that will be attached to each batch of events.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0"] |
mac(optional)
The mac address that will be attached to each batch of events.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-mac-address"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
tags(optional)
The tags that will be attached to each batch of events.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["tag1","tag2"] |
1[sources.my_source_id]
2type = ["logdna"]
3api_key = ["${LOGDNA_API_KEY}","ef8d5de700e7989468166c40fc8a0ccd"]
4hostname = ["${HOSTNAME}","my-local-machine"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
The base URL of the Loki instance.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://localhost:3100"] |
labels(required)
A set of labels that are attached to each batch of events. Both keys and values are templatable, which enables you to attach dynamic labels to events. Note: If the set of labels has high cardinality, this can cause drastic performance issues with Loki. To prevent this from happening, reduce the number of unique label keys and values.
Type | Syntax | Default | Example |
---|---|---|---|
hash | template | [{"forwarder":"vector","event":"{{ event_field }}","key":"value","\"{{ event_field }}\"":"{{ another_event_field }}"}] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["loki"] |
Advanced Options
auth(optional)
Configures the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
out_of_order_action(optional)
Some sources may generate events with timestamps that aren't in strictly chronological order. The Loki service can't accept a stream of such events. Vector sorts events before sending them to Loki, however some late events might arrive after a batch has been sent. This option specifies what Vector should do with those events.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | drop |
remove_label_fields(optional)
If this is set to true
then when labels are collected from events those fields will also get removed from the event.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
remove_timestamp(optional)
If this is set to true
then the timestamp will be removed from the event payload. Note the event timestamp will still be sent as metadata to Loki for indexing.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
tenant_id(optional)
The tenant id that's sent with every request, by default this is not required since a proxy should set this header. When running Loki locally a tenant id is not required either.
You can read more about tenant id's here.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["some_tenant_id","{{ event_field }}"] |
1[sources.my_source_id]
2type = ["loki"]
3endpoint = ["http://localhost:3100"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6[[labels]]
7forwarder = "vector"
8event = "{{ event_field }}"
9key = "value"
10"\"{{ event_field }}\"" = "{{ another_event_field }}"
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
url(required)
The NATS URL to connect to. The url must take the form of nats://server:port
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["nats://demo.nats.io","nats://127.0.0.1:4222"] |
subject(required)
The NATS subject to publish messages to.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ host }}","foo","time.us.east","time.*.east","time.>",">"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["nats"] |
Advanced Options
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
connection_name(optional)
A name assigned to the NATS connection.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | vector | ["foo","API Name Option Example"] |
1[sources.my_source_id]
2type = ["nats"]
3url = ["nats://demo.nats.io","nats://127.0.0.1:4222"]
4subject = [
5 "{{ host }}",
6 "foo",
7 "time.us.east",
8 "time.*.east",
9 "time.>",
10 ">"
11]
12inputs = ["my-source-or-transform-id","prefix-*"]
13encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["new_relic_logs"] |
Advanced Options
insert_key(optional)
Your New Relic insert key (if applicable).
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["xxxx","${NEW_RELIC_INSERT_KEY}"] |
license_key(optional)
Your New Relic license key (if applicable).
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["xxxx","${NEW_RELIC_LICENSE_KEY}"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
region(optional)
The region to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | us |
1[sources.my_source_id]
2type = ["new_relic_logs"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
The endpoint to send logs to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["logs.papertrailapp.com:12345"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["papertrail"] |
Advanced Options
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
send_buffer_bytes(optional)
Configures the send buffer size using the SO_SNDBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["papertrail"]
3endpoint = ["logs.papertrailapp.com:12345"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
address(required)
The address to expose for scraping.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["0.0.0.0:9598"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["prometheus_exporter"] |
Advanced Options
buckets(optional)
Default buckets to use for aggregating distribution metrics into histograms.
Type | Syntax | Default | Example |
---|---|---|---|
array | 0.0050.010.0250.050.10.250.512.5510 | [0.005,0.01] |
flush_period_secs(optional)
Time interval between set values are reset.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 60 |
default_namespace(optional)
Used as a namespace for metrics that don't have it. Typically namespaces are set during ingestion (sources), but it is optional and when missing, we'll use this value. It should follow Prometheus naming conventions.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
quantiles(optional)
Quantiles to use for aggregating distribution metrics into a summary.
Type | Syntax | Default | Example |
---|---|---|---|
array | 0.50.750.90.950.99 | [0.5,0.75,0.9,0.95,0.99] |
1[sources.my_source_id]
2type = ["prometheus_exporter"]
3address = ["0.0.0.0:9598"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
The endpoint URL to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["https://localhost:8087/"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["prometheus_remote_write"] |
Advanced Options
auth(optional)
Configures the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
default_namespace(optional)
Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name. It should follow Prometheus naming conventions.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
buckets(optional)
Default buckets to use for aggregating distribution metrics into histograms.
Type | Syntax | Default | Example |
---|---|---|---|
array | 0.0050.010.0250.050.10.250.512.5510 | [0.005,0.01] |
quantiles(optional)
Quantiles to use for aggregating distribution metrics into a summary.
Type | Syntax | Default | Example |
---|---|---|---|
array | 0.50.750.90.950.99 | [0.5,0.75,0.9,0.95,0.99] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
tenant_id(optional)
If set, a header named X-Scope-OrgID
will be added to outgoing requests with the text of this setting. This may be used by Cortex or other remote services to identify the tenant making the request.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["my-domain"] |
1[sources.my_source_id]
2type = ["prometheus_remote_write"]
3endpoint = ["https://localhost:8087/"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
Endpoint to which the pulsar client should connect to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["pulsar://127.0.0.1:6650"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
topic(required)
The Pulsar topic name to write events to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["topic-1234"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["pulsar"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
1[sources.my_source_id]
2type = ["pulsar"]
3endpoint = ["pulsar://127.0.0.1:6650"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6topic = ["topic-1234"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
region(required)
The region to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us","eu"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
token(required)
The token that will be used to write to Sematext.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${SEMATEXT_TOKEN}","some-sematext-token"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["sematext_logs"] |
Advanced Options
endpoint(optional)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://127.0.0.1","http://example.com"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["sematext_logs"]
3region = ["us","eu"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6token = ["${SEMATEXT_TOKEN}","some-sematext-token"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
region(required)
The region to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us","eu"] |
default_namespace(required)
Used as a namespace for metrics that don't have it.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
token(required)
The token that will be used to write to Sematext.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${SEMATEXT_TOKEN}","some-sematext-token"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["sematext_metrics"] |
Advanced Options
endpoint(optional)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://127.0.0.1","http://example.com"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["sematext_metrics"]
3region = ["us","eu"]
4default_namespace = ["service"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7token = ["${SEMATEXT_TOKEN}","some-sematext-token"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
The base URL of the Splunk instance.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["https://http-inputs-hec.splunkcloud.com","https://hec.splunk.com:8088","http://example.com"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
token(required)
Your Splunk HEC token.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${SPLUNK_HEC_TOKEN}","A94A8FE5CCB19BA61C4C08"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["splunk_hec"] |
Advanced Options
host_key(optional)
The name of the log field to be used as the hostname sent to Splunk HEC. This overrides the
global host_key
option.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["hostname"] |
index(optional)
The name of the index where send the events to. If not specified, the default index is used.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ host }}","custom_index"] |
indexed_fields(optional)
Fields to be added to Splunk index.
Type | Syntax | Default | Example |
---|---|---|---|
array | field_path | ["field1","field2"] |
source(optional)
The source of events sent to this sink. Typically the filename the logs originated from. If unset, the Splunk collector will set it.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ file }}","/var/log/syslog","UDP:514"] |
sourcetype(optional)
The sourcetype of events sent to this sink. If unset, Splunk will default to httpevent.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ sourcetype }}","_json","httpevent"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["splunk_hec"]
3endpoint = [
4 "https://http-inputs-hec.splunkcloud.com",
5 "https://hec.splunk.com:8088",
6 "http://example.com"
7]
8inputs = ["my-source-or-transform-id","prefix-*"]
9encoding = []
10token = ["${SPLUNK_HEC_TOKEN}","A94A8FE5CCB19BA61C4C08"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
address(required)
The address to connect to. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["92.12.333.224:5000"] |
mode(required)
The type of socket to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["tcp","udp","unix"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
path(required)
The unix socket path. This should be the absolute path.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/socket"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["statsd","socket"] |
Advanced Options
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
send_buffer_bytes(optional)
Configures the send buffer size using the SO_SNDBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
default_namespace(optional)
Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
1[sources.my_source_id]
2type = ["statsd","socket"]
3address = ["92.12.333.224:5000"]
4mode = ["tcp","udp","unix"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7path = ["/path/to/socket"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
address(required)
The downstream Vector address to connect to. The address must include a port.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["92.12.333.224:6000"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector"] |
Advanced Options
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
send_buffer_bytes(optional)
Configures the send buffer size using the SO_SNDBUF
option on the socket.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [65536] |
keepalive(optional)
Configures the TCP keepalive behavior for the connection to the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
version(optional)
Sink API version. Specifying this version ensures that Vector does not break backward compatibility.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | 1 |
1[sources.my_source_id]
2type = ["vector"]
3address = ["92.12.333.224:6000"]
4inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
api_key(required)
The team key that will be used to authenticate against Honeycomb.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${HONEYCOMB_API_KEY}","some-api-key"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
dataset(required)
The dataset that Vector will send logs to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-honeycomb-dataset"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["honeycomb"] |
Advanced Options
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["honeycomb"]
3api_key = ["${HONEYCOMB_API_KEY}","some-api-key"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6dataset = ["my-honeycomb-dataset"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
api_key(required)
Datadog API key
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${DATADOG_API_KEY_ENV_VAR}","ef8d5de700e7989468166c40fc8a0ccd"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["datadog_metrics"] |
Advanced Options
endpoint(optional)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.1:8080","example.com:12345"] |
region(optional)
The region to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
default_namespace(optional)
Used as a namespace for metrics that don't have it. A namespace will be prefixed to a metric's name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
1[sources.my_source_id]
2type = ["datadog_metrics"]
3api_key = [
4 "${DATADOG_API_KEY_ENV_VAR}",
5 "ef8d5de700e7989468166c40fc8a0ccd"
6]
7inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
url(required)
The Redis URL to connect to. The url must take the form of protocol://server:port/db
where the protocol can either be redis
or rediss
for connections secured via TLS.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["redis://127.0.0.1:6379/0"] |
key(required)
The Redis key to publish messages to.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["syslog:{{ app }}","vector"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["redis"] |
Advanced Options
data_type(optional)
The Redis data type (list
or channel
) to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | list |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
list(optional)
Options for the Redis list
data type.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["redis"]
3url = ["redis://127.0.0.1:6379/0"]
4key = ["syslog:{{app}}","vector"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
project_id(required)
The project ID to which to publish logs. See the Google Cloud Platform project management documentation for more details.
Exactly one of billing_account_id
, folder_id
, organization_id
, or project_id
must be set.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-123456"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
resource(required)
Options for describing the logging resource.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [{"type":"global","projectId":"vector-123456","instanceId":"Twilight","zone":"us-central1-a"}] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["gcp_stackdriver_metrics"] |
Advanced Options
credentials_path(optional)
The filename for a Google Cloud service account credentials JSON file used to authenticate access to the Stackdriver Logging API. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS
environment variable for a filename.
If no filename is named, Vector will attempt to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/credentials.json"] |
default_namespace(optional)
The namespace used if the metric we are going to send to GCP has no namespace.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | namespace | ["vector-123456"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = "global"
3type = ["gcp_stackdriver_metrics"]
4project_id = ["vector-123456"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
7[[resource]]
8projectId = "vector-123456"
9instanceId = "Twilight"
10zone = "us-central1-a"
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
log_id(required)
The log ID to which to publish logs. This is a name you create to identify this log stream.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["vector-logs","{{ component_id }}"] |
project_id(required)
The project ID to which to publish logs. See the Google Cloud Platform project management documentation for more details.
Exactly one of billing_account_id
, folder_id
, organization_id
, or project_id
must be set.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-123456"] |
resource(required)
Options for describing the logging resource.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [{"type":"gce_instance","instance_id":"Twilight","zone":"us-central1-a"}] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["gcp_stackdriver_logs"] |
Advanced Options
billing_account_id(optional)
The billing account ID to which to publish logs.
Exactly one of must be set: billing_account_id
, folder_id
, organization_id
,
or project_id
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["012345-6789AB-CDEF01"] |
credentials_path(optional)
The filename for a Google Cloud service account credentials JSON file used to authenticate access to the
Stackdriver Logging API. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS
environment variable for a filename.
If no filename is named, Vector attempts to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/credentials.json"] |
folder_id(optional)
The folder ID to which to publish logs.
See the Google Cloud Platform folder documentation for more details.
Exactly one of billing_account_id
, folder_id
, organization_id
, or project_id
must be set.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["My Folder"] |
organization_id(optional)
The organization ID to which to publish logs. This would be the identifier assigned to your organization on Google Cloud Platform.
Exactly one of billing_account_id
, folder_id
, organization_id
, or project_id
must be set.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["622418129737"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
severity_key(optional)
The field of the log event from which to take the outgoing log's severity
field. The named field is
removed from the log event if present, and must be either an integer between 0 and 800 or a string
containing one of the severity level names (case is ignored) or a
common prefix such as err
.
If no severity key is specified, the severity of outgoing records is set to 0 (DEFAULT
).
See the GCP Stackdriver Logging LogSeverity description for more
details on the value of the severity
field.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["severity"] |
1[sources.my_source_id]
2type = "gce_instance"
3type = ["gcp_stackdriver_logs"]
4log_id = ["vector-logs","{{component_id}}"]
5project_id = ["vector-123456"]
6inputs = ["my-source-or-transform-id","prefix-*"]
7encoding = []
8[[resource]]
9instance_id = "Twilight"
10zone = "us-central1-a"
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
project(required)
The project name to which to publish logs.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["vector-123456"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
topic(required)
The topic within the project to which to publish logs.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["this-is-a-topic"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["gcp_pubsub"] |
Advanced Options
api_key(optional)
A Google Cloud API key used to authenticate access the pubsub project and topic. Either this or credentials_path
must be set.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${GCP_API_KEY}","ef8d5de700e7989468166c40fc8a0ccd"] |
credentials_path(optional)
The filename for a Google Cloud service account credentials JSON file used to authenticate access to the pubsub project and topic. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS
environment variable for a filename.
If no filename is named, Vector will attempt to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/credentials.json"] |
endpoint(optional)
The endpoint to which to send data.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | https://pubsub.googleapis.com | ["https://us-central1-pubsub.googleapis.com"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["gcp_pubsub"]
3project = ["vector-123456"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6topic = ["this-is-a-topic"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
bucket(required)
The GCS bucket name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-bucket"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["gcp_cloud_storage"] |
Advanced Options
acl(optional)
Predefined ACL to apply to the created objects. For more information, see Predefined ACLs. If this is not set, GCS will apply a default ACL when the object is created.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
credentials_path(optional)
The filename for a Google Cloud service account credentials JSON file used to authenticate access to the Cloud Storage API. If this is unset, Vector checks the GOOGLE_APPLICATION_CREDENTIALS
environment variable for a filename.
If no filename is named, Vector will attempt to fetch an instance service account for the compute instance the program is running on. If Vector is not running on a GCE instance, you must define a credentials file as above.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/path/to/credentials.json"] |
filename_append_uuid(optional)
Whether or not to append a UUID v4 token to the end of the file. This ensures there are no name collisions high volume use cases.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
filename_extension(optional)
The filename extension to use in the object name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | log |
filename_time_format(optional)
The format of the resulting object file name. strftime
specifiers are supported.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | %s |
key_prefix(optional)
A prefix to apply to all object key names. This should be used to partition your objects, and it's important to end this value with a /
if you want this to be the root GCS "folder".
Type | Syntax | Default | Example |
---|---|---|---|
string | template | date=%F/ | ["date=%F/","date=%F/hour=%H/","year=%Y/month=%m/day=%d/","application_id={{ application_id }}/date=%F/"] |
metadata(optional)
The set of metadata key:value
pairs for the created objects. See the GCS custom metadata documentation for more details.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | [] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
storage_class(optional)
The storage class for the created objects. See the GCP storage classes for more details.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
1[sources.my_source_id]
2type = ["gcp_cloud_storage"]
3bucket = ["my-bucket"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
path(required)
File name to write events to.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["/tmp/vector-%Y-%m-%d.log","/tmp/application-{{ application_id }}-%Y-%m-%d.log"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["file"] |
Advanced Options
idle_timeout_secs(optional)
The amount of time a file can be idle and stay open. After not receiving any events for this timeout, the file will be flushed and closed.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 30 |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
1[sources.my_source_id]
2type = ["file"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5path = [
6 "/tmp/vector-%Y-%m-%d.log",
7 "/tmp/application-{{ application_id }}-%Y-%m-%d.log"
8]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
The Elasticsearch endpoint to send logs to. This should be the full URL as shown in the example.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://10.24.32.122:9000","https://example.com","https://user:password@example.com"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["elasticsearch"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
aws(optional)
Options for the AWS connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
bulk_action(optional)
Action to use when making requests to the Elasticsearch Bulk API.
Currently, Vector only supports index
and create
. update
and delete
actions are not supported.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | index | ["index","create","{{ action }}"] |
data_stream(optional)
Options for the data stream mode.
Type | Syntax | Default | Example |
---|---|---|---|
hash | template | [] |
doc_type(optional)
The doc_type
for your index data. This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | _doc |
id_key(optional)
The name of the event key that should map to Elasticsearch's _id
field. By default, Vector does not set the _id
field, which allows Elasticsearch to set this automatically. You should think carefully about setting your own Elasticsearch IDs, since this can hinder perofrmance.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["id","_id"] |
index(optional)
Index name to write events to.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | vector-%F | ["application-{{ application_id }}-%Y-%m-%d","vector-%Y-%m-%d"] |
metrics(optional)
Options for metrics.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
mode(optional)
The type of index mechanism. If data_stream
mode is enabled, the bulk_action
is set to create
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | normal | ["normal","data_stream"] |
pipeline(optional)
Name of the pipeline to apply.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["pipeline-name"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
query(optional)
Custom parameters to Elasticsearch query string.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"X-Powered-By":"Vector"}] |
1[sources.my_source_id]
2type = ["elasticsearch"]
3endpoint = [
4 "http://10.24.32.122:9000",
5 "https://example.com",
6 "https://user:password@example.com"
7]
8inputs = ["my-source-or-transform-id","prefix-*"]
9encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
default_api_key(required)
Default Datadog API key, if an event has a key set in its metadata it will prevail over the one set here.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${DATADOG_API_KEY_ENV_VAR}","ef8d5de700e7989468166c40fc8a0ccd"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["datadog_logs"] |
Advanced Options
endpoint(optional)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.1:8080","example.com:12345"] |
region(optional)
The region to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | gzip |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
site(optional)
The Datadog site to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | datadoghq.com | ["us3.datadoghq.com","datadoghq.com","datadoghq.eu"] |
1[sources.my_source_id]
2type = ["datadog_logs"]
3default_api_key = [
4 "${DATADOG_API_KEY_ENV_VAR}",
5 "ef8d5de700e7989468166c40fc8a0ccd"
6]
7inputs = ["my-source-or-transform-id","prefix-*"]
8encoding = []
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
default_api_key(required)
Default Datadog API key, if an event has a key set in its metadata it will prevail over the one set here.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${DATADOG_API_KEY_ENV_VAR}","ef8d5de700e7989468166c40fc8a0ccd"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["datadog_events"] |
Advanced Options
endpoint(optional)
The endpoint to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.1:8080","example.com:12345"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
site(optional)
The Datadog site to send data to.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | datadoghq.com | ["us3.datadoghq.com","datadoghq.com","datadoghq.eu"] |
1[sources.my_source_id]
2type = ["datadog_events"]
3default_api_key = [
4 "${DATADOG_API_KEY_ENV_VAR}",
5 "ef8d5de700e7989468166c40fc8a0ccd"
6]
7inputs = ["my-source-or-transform-id","prefix-*"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["console"] |
Advanced Options
1[sources.my_source_id]
2type = ["console"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
endpoint(required)
The endpoint of the Clickhouse server.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http://localhost:8123"] |
table(required)
The table that data will be inserted into.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["mytable"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["clickhouse"] |
Advanced Options
auth(optional)
Configures the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
database(optional)
The database that contains the stable that data will be inserted into.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["mydatabase"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | gzip |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
skip_unknown_fields(optional)
Sets input_format_skip_unknown_fields
, allowing Clickhouse to discard fields not present in the table schema.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
1[sources.my_source_id]
2type = ["clickhouse"]
3endpoint = ["http://localhost:8123"]
4table = ["mytable"]
5inputs = ["my-source-or-transform-id","prefix-*"]
6encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["blackhole"] |
Advanced Options
print_amount(optional)
The number of events that must be received in order to print a summary of activity.
Type | Syntax | Default | Example |
---|---|---|---|
uint | 1000 | [1000] |
rate(optional)
Rates the amount of events that the sink can consume per second.
Type | Syntax | Default | Example |
---|---|---|---|
uint | [1000] |
1[sources.my_source_id]
2type = ["blackhole"]
3inputs = ["my-source-or-transform-id","prefix-*"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
customer_id(required)
The unique identifier for the Log Analytics workspace.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["5ce893d9-2c32-4b6c-91a9-b0887c2de2d6","97ce69d9-b4be-4241-8dbd-d265edcf06c4"] |
log_type(required)
The record type of the data that is being submitted. Can only contain letters, numbers, and underscore (_), and may not exceed 100 characters.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["MyTableName","MyRecordType"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
shared_key(required)
The primary or the secondary key for the Log Analytics workspace.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["${AZURE_MONITOR_SHARED_KEY_ENV_VAR}","SERsIYhgMVlJB6uPsq49gCxNiruf6v0vhMYE+lfzbSGcXjdViZdV/e5pEMTYtw9f8SkVLf4LFlLCc2KxtRZfCA=="] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["azure_monitor_logs"] |
Advanced Options
azure_resource_id(optional)
Resource ID of the Azure resource the data should be associated with.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/otherResourceGroup/providers/Microsoft.Storage/storageAccounts/examplestorage","/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/examplegroup/providers/Microsoft.SQL/servers/serverName/databases/databaseName"] |
host(optional)
Alternative host for dedicated Azure regions.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ods.opinsights.azure.com | ["ods.opinsights.azure.us","ods.opinsights.azure.cn"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["azure_monitor_logs"]
3log_type = ["MyTableName","MyRecordType"]
4customer_id = [
5 "5ce893d9-2c32-4b6c-91a9-b0887c2de2d6",
6 "97ce69d9-b4be-4241-8dbd-d265edcf06c4"
7]
8inputs = ["my-source-or-transform-id","prefix-*"]
9encoding = []
10shared_key = [
11 "${AZURE_MONITOR_SHARED_KEY_ENV_VAR}",
12 "SERsIYhgMVlJB6uPsq49gCxNiruf6v0vhMYE+lfzbSGcXjdViZdV/e5pEMTYtw9f8SkVLf4LFlLCc2KxtRZfCA=="
13]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
connection_string(required)
The Azure Blob Storage Account connection string. Only authentication with access key supported.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["DefaultEndpointsProtocol=https;AccountName=mylogstorage;AccountKey=storageaccountkeybase64encoded;EndpointSuffix=core.windows.net"] |
container_name(required)
The Azure Blob Storage Account container name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-logs"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["azure_blob"] |
Advanced Options
blob_prefix(optional)
A prefix to apply to all object key names. This should be used to partition your objects, and it's important to end this value with a /
if you want this to be the root azure storage "folder".
Type | Syntax | Default | Example |
---|---|---|---|
string | template | blob/%F/ | ["date/%F/","date/%F/hour/%H/","year=%Y/month=%m/day=%d/","kubernetes/{{ metadata.cluster }}/{{ metadata.application_name }}/"] |
blob_append_uuid(optional)
Whether or not to append a UUID v4 token to the end of the file. This ensures there are no name collisions high volume use cases.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | gzip |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
blob_time_format(optional)
The format of the resulting object file name. strftime
specifiers are supported.
Type | Syntax | Default | Example |
---|---|---|---|
string | strftime | %s |
1[sources.my_source_id]
2type = ["azure_blob"]
3connection_string = [
4 "DefaultEndpointsProtocol=https;AccountName=mylogstorage;AccountKey=storageaccountkeybase64encoded;EndpointSuffix=core.windows.net"
5]
6container_name = ["my-logs"]
7inputs = ["my-source-or-transform-id","prefix-*"]
8encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
uri(required)
The full URI to make HTTP requests to. This should include the protocol and host, but can also include the port, path, and any other valid part of a URI.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["https://10.22.212.22:9000/endpoint"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["http"] |
Advanced Options
auth(optional)
Configures the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
tls(optional)
Configures the TLS options for incoming connections.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["http"]
3uri = ["https://10.22.212.22:9000/endpoint"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
bucket(required)
The S3 bucket name. Do not include a leading s3://
or a trailing /
.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-bucket"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_s3"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
acl(optional)
Canned ACL to apply to the created objects. For more information, see Canned ACL.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
content_encoding(optional)
Specifies what content encodings have been applied to the object and thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field. By default calculated from compression
value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["gzip"] |
content_type(optional)
A standard MIME type describing the format of the contents.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | text/x-log |
filename_append_uuid(optional)
Whether or not to append a UUID v4 token to the end of the file. This ensures there are no name collisions high volume use cases.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
filename_extension(optional)
The filename extension to use in the object name.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | log |
filename_time_format(optional)
The format of the resulting object file name. strftime
specifiers are supported.
Type | Syntax | Default | Example |
---|---|---|---|
string | strftime | %s |
grant_full_control(optional)
Gives the named grantee READ, READ_ACP, and WRITE_ACP permissions on the created objects.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"] |
grant_read(optional)
Allows the named grantee to read the created objects and their metadata.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"] |
grant_read_acp(optional)
Allows the named grantee to read the created objects' ACL.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"] |
grant_write_acp(optional)
Allows the named grantee to write the created objects' ACL.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","person@email.com","http://acs.amazonaws.com/groups/global/AllUsers"] |
key_prefix(optional)
A prefix to apply to all object key names. This should be used to partition your objects, and it's important to end this value with a /
if you want this to be the root S3 "folder".
Type | Syntax | Default | Example |
---|---|---|---|
string | template | date=%F/ | ["date=%F/","date=%F/hour=%H/","year=%Y/month=%m/day=%d/","application_id={{ application_id }}/date=%F/"] |
server_side_encryption(optional)
The Server-side Encryption algorithm used when storing these objects.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
ssekms_key_id(optional)
If server_side_encryption
has the value "aws.kms"
, this specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed customer master key (CMK) that will used for the created objects. If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["abcd1234"] |
storage_class(optional)
The storage class for the created objects. See the S3 Storage Classes for more details.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | gzip |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
tags(optional)
The tag-set for the object.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [{"Tag1":"Value1"}] |
1[sources.my_source_id]
2type = ["aws_s3"]
3bucket = ["my-bucket"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
stream_name(required)
The stream name of the target Kinesis Logs stream.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-stream"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_kinesis_streams"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
partition_key_field(optional)
The log field used as the Kinesis record's partition key value.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["user_id"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["aws_kinesis_streams"]
3inputs = ["my-source-or-transform-id","prefix-*"]
4encoding = []
5region = ["us-east-1"]
6stream_name = ["my-stream"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
default_namespace(required)
A namespace that will isolate different metrics from each other. Used as a namespace for metrics that don't have it.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["service"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_cloudwatch_metrics"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["aws_cloudwatch_metrics"]
3default_namespace = ["service"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5region = ["us-east-1"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
group_name(required)
The group name of the target CloudWatch Logs stream.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["group-name","{{ file }}"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
stream_name(required)
The stream name of the target CloudWatch Logs stream. Note that there can only be one writer to a log stream at a time so if you are running multiple vectors all writing to the same log group, include a identifier in the stream name that is guaranteed to be unique by vector instance (for example, you might choose host
)
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ host }}","%Y-%m-%d","stream-name"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_cloudwatch_logs"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
create_missing_group(optional)
Dynamically create a log group if it does not already exist. This will ignore create_missing_stream
directly after creating the group and will create the first stream.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
create_missing_stream(optional)
Dynamically create a log stream if it does not already exist.
Type | Syntax | Default | Example |
---|---|---|---|
bool |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["aws_cloudwatch_logs"]
3group_name = ["group-name","{{file}}"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]
7stream_name = ["{{host}}","%Y-%m-%d","stream-name"]
- Beta
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
queue_url(required)
The URL of the Amazon SQS queue to which messages are sent.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_sqs"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
message_group_id(optional)
The tag that specifies that a message belongs to a specific message group. Can be applied only to FIFO queues.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["vector","vector-%Y-%m-%d"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
message_deduplication_id(optional)
The message deduplication ID value to allow AWS to identify duplicate messages. This value is a template which should result in a unique string for each event.
See the AWS documentation for more about how AWS does message deduplication.
Type | Syntax | Default | Example |
---|---|---|---|
string | template | ["{{ transaction_id }}"] |
1[sources.my_source_id]
2type = ["aws_sqs"]
3queue_url = ["https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]
- Stable
- Linux, Windows, MacOS
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.
Required Options
stream_name(required)
The stream name of the target Kinesis Firehose delivery stream.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["my-stream"] |
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
Type | Syntax | Default | Example |
---|---|---|---|
array | literal | ["my-source-or-transform-id","prefix-*"] |
encoding(required)
Configures the encoding specific sink behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
region(required)
The AWS region of the target service. If endpoint
is provided it will override this value since the endpoint includes the region.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["us-east-1"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["aws_kinesis_firehose"] |
Advanced Options
auth(optional)
Options for the authentication strategy.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
endpoint(optional)
Custom endpoint for use with AWS-compatible services. Providing a value for this option will make region
moot.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | ["127.0.0.0:5000/path/to/service"] |
buffer(optional)
Configures the sink specific buffer behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
batch(optional)
Configures the sink batching behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
compression(optional)
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
Type | Syntax | Default | Example |
---|---|---|---|
string | literal | none |
healthcheck(optional)
Health check options for the sink.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
request(optional)
Configures the sink request behavior.
Type | Syntax | Default | Example |
---|---|---|---|
hash | [] |
proxy(optional)
Configures an HTTP(S) proxy for Vector to use. By default, the globally configured proxy is used.
Type | Syntax | Default | Example |
---|---|---|---|
hash | literal | [] |
1[sources.my_source_id]
2type = ["aws_kinesis_firehose"]
3stream_name = ["my-stream"]
4inputs = ["my-source-or-transform-id","prefix-*"]
5encoding = []
6region = ["us-east-1"]