Configuring Vector
Vector is configured using a configuration file. This section contains a comprehensive reference of all Vector configuration options.
Example
The following is an example of a popular Vector configuration that ingests logs from a file and routes them to both Elasticsearch and AWS S3. Your configuration will differ based on your needs.
{{< tabs default="vector.toml" >}} {{< tab title="vector.toml" >}}
1# Set global options
2data_dir = "/var/lib/vector"
3
4# Vector's API (disabled by default)
5# Enable and try it out with the `vector top` command
6[api]
7enabled = false
8# address = "127.0.0.1:8686"
9
10# Ingest data by tailing one or more files
11[sources.apache_logs]
12type = "file"
13include = ["/var/log/apache2/*.log"] # supports globbing
14ignore_older = 86400 # 1 day
15
16# Structure and parse via Timber's Remap Language
17[transforms.remap]
18inputs = ["apache_logs"]
19type = "remap"
20source = '''
21. = parse_apache_log(.message)
22'''
23
24# Sample the data to save on cost
25[transforms.apache_sampler]
26inputs = ["apache_parser"]
27type = "sampler"
28rate = 50 # only keep 50%
29
30# Send structured data to a short-term storage
31[sinks.es_cluster]
32inputs = ["apache_sampler"] # only take sampled data
33type = "elasticsearch"
34host = "http://79.12.221.222:9200" # local or external host
35index = "vector-%Y-%m-%d" # daily indices
36
37# Send structured data to a cost-effective long-term storage
38[sinks.s3_archives]
39inputs = ["apache_parser"] # don't sample for S3
40type = "aws_s3"
41region = "us-east-1"
42bucket = "my-log-archives"
43key_prefix = "date=%Y-%m-%d" # daily partitions, hive friendly format
44compression = "gzip" # compress final objects
45encoding = "ndjson" # new line delimited JSON
46batch.max_size = 10000000 # 10mb uncompressed
{{< /tab >}} {{< tab title="vector.yaml" >}}
1data_dir: /var/lib/vector
2sources:
3 apache_logs:
4 type: file
5 include:
6 - /var/log/apache2/*.log
7 ignore_older: 86400
8transforms:
9 remap:
10 inputs:
11 - apache_logs
12 type: remap
13 source: |
14 . = parse_apache_log(.message)
15 apache_sampler:
16 inputs:
17 - apache_parser
18 type: sampler
19 rate: 50
20sinks:
21 es_cluster:
22 inputs:
23 - apache_sampler
24 type: elasticsearch
25 host: 'http://79.12.221.222:9200'
26 index: vector-%Y-%m-%d
27 s3_archives:
28 inputs:
29 - apache_parser
30 type: aws_s3
31 region: us-east-1
32 bucket: my-log-archives
33 key_prefix: date=%Y-%m-%d
34 compression: gzip
35 encoding: ndjson
36 batch:
37 max_size: 10000000
{{< /tab >}} {{< tab title="vector.json" >}}
1{
2 "data_dir": "/var/lib/vector",
3 "sources": {
4 "apache_logs": {
5 "type": "file",
6 "include": [
7 "/var/log/apache2/*.log"
8 ],
9 "ignore_older": 86400
10 }
11 },
12 "transforms": {
13 "remap": {
14 "inputs": [
15 "apache_logs"
16 ],
17 "type": "remap",
18 "source": ". = parse_apache_log(.message)"
19 },
20 "apache_sampler": {
21 "inputs": [
22 "apache_parser"
23 ],
24 "type": "sampler",
25 "rate": 50
26 }
27 },
28 "sinks": {
29 "es_cluster": {
30 "inputs": [
31 "apache_sampler"
32 ],
33 "type": "elasticsearch",
34 "host": "http://79.12.221.222:9200",
35 "index": "vector-%Y-%m-%d"
36 },
37 "s3_archives": {
38 "inputs": [
39 "apache_parser"
40 ],
41 "type": "aws_s3",
42 "region": "us-east-1",
43 "bucket": "my-log-archives",
44 "key_prefix": "date=%Y-%m-%d",
45 "compression": "gzip",
46 "encoding": "ndjson",
47 "batch": {
48 "max_size": 10000000
49 }
50 }
51 }
52}
{{< /tab >}} {{< /tabs >}}
To use this configuration file, specify it with the --config flag when starting Vector:
{{< tabs default="TOML" >}} {{< tab title="TOML" >}}
1vector --config /etc/vector/vector.toml
{{< /tab >}} {{< tab title="YAML" >}}
1vector --config /etc/vector/vector.yaml
{{< /tab >}} {{< tab title="JSON" >}}
1vector --config /etc/vector/vector.json
{{< /tab >}} {{< /tabs >}}
Reference
Components
{{< jump "/docs/reference/configuration/sources" >}} {{< jump "/docs/reference/configuration/transforms" >}} {{< jump "/docs/reference/configuration/sinks" >}}
Advanced
{{< jump "/docs/reference/configuration/global-options" >}} {{< jump "/docs/reference/configuration/template-syntax" >}}
How it works
Environment variables
Vector interpolates environment variables within your configuration file with the following syntax:
1[transforms.add_host]
2type = "add_fields"
3
4[transforms.add_host.fields]
5host = "${HOSTNAME}" # or "$HOSTNAME"
6environment = "${ENV:-development}" # default value when not present
Default values
Default values can be supplied using :- syntax:
1option = "${ENV_VAR:-default}"
Escaping
You can escape environment variables by prefacing them with a $ character. For example $${HOSTNAME} or $$HOSTNAME is treated literally in the above environment variable example.
Formats
Vector supports TOML, YAML, and JSON to ensure that Vector fits into your workflow. A side benefit of supporting JSON is that it enables you to use JSON-outputting data templating languages like Jsonnet and Cue.
Location
The location of your Vector configuration file depends on your installation method. For most Linux-based systems, the file can be found at /etc/vector/vector.toml.
Multiple files
You can pass multiple configuration files when starting Vector:
1vector --config vector1.toml --config vector2.toml
Or using a globbing syntax:
1vector --config /etc/vector/*.toml
Wilcards in component IDs
Vector supports wildcards (*) in component IDs when building your topology. For example:
1[sources.app1_logs]
2type = "file"
3includes = ["/var/log/app1.log"]
4
5[sources.app2_logs]
6type = "file"
7includes = ["/var/log/app.log"]
8
9[sources.system_logs]
10type = "file"
11includes = ["/var/log/system.log"]
12
13[sinks.app_logs]
14type = "datadog_logs"
15inputs = ["app*"]
16
17[sinks.archive]
18type = "aws_s3"
19inputs = ["app*", "system_logs"]
Sections
{{< sections >}}
Pages
{{< pages >}}