Docker

CAUTION

Collecting logs directly from the Docker Engine is known to have performance problems for very large setups. If you have a large setup, please consider alternative collection methods, such as the Docker syslog or Docker journald driver drivers. To avoid collecting logs from itself when deployed as a container, the Docker source uses current hostname to find out which container it is inside. If a container's ID matches the hostname, that container will be excluded. If you change container's hostname, consider manually excluding Vector container using exclude_containers.

Example Configuration

Dummy Logs

Config

Input

Output

1[sources.my_source_id]
2type = "docker_logs"
3include_images = [ "mingrammer/flog" ]

1" ```json\n {\n   \"stream\": \"stdout\",\n   \"message\": \"150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] \"HEAD /initiatives HTTP/1.1\" 504 117\"\n }\n```"

1{
2  "log": {
3    "container_created_at": "2020-10-03T16:11:29.443232Z",
4    "container_id": "fecc98177eca7fb75a2b2186c418bf9a0cd3a05a1169f2e2293bf8987a9d96ab",
5    "container_name": "flog",
6    "image": "mingrammer/flog",
7    "message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] \"HEAD /initiatives HTTP/1.1\" 504 117",
8    "stream": "stdout",
9    "host": "my-host.local"
10  }
11}

Configuration Options

Required Options

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["docker_logs"]

Advanced Options

docker_host(optional)

The Docker host to connect to. Use an HTTPS URL to enable TLS encryption. If absent, Vector will try to use DOCKER_HOST enviroment variable. If DOCKER_HOST is also absent, Vector will use default Docker local socket (/var/run/docker.sock on Unix flatforms, //./pipe/docker_engine on Windows).

Type	Syntax	Default	Example
string	literal		["http://localhost:2375","https://localhost:2376","unix:///var/run/docker.sock","npipe:////./pipe/docker_engine","/var/run/docker.sock","//./pipe/docker_engine"]

tls(optional)

TLS options to connect to the Docker daemon. This has no effect unless docker_host is an HTTPS URL. If absent, Vector will try to use environment variable DOCKER_CERT_PATH and then DOCKER_CONFIG. If both environment variables are absent, Vector will try to read certificates in ~/.docker/.

Type	Syntax	Default	Example
hash	literal		[]

auto_partial_merge(optional)

Setting this to false will disable the automatic merging of partial events.

Type	Syntax	Default	Example
bool

exclude_containers(optional)

A list of container IDs or names to match against for containers you don't want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to exclude. This can be used in conjunction with include_containers.

Type	Syntax	Default	Example
array	literal		["exclude_","exclude_me_0","ad08cc418cf9"]

include_containers(optional)

A list of container IDs or names to match against for containers you want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to include. This can be used in conjunction with exclude_containers.

Type	Syntax	Default	Example
array	literal		["include_","include_me_0","ad08cc418cf9"]

include_labels(optional)

A list of container object labels to match against when filtering running containers. This should follow the described label's syntax in docker object labels docs.

Type	Syntax	Default	Example
array	literal		["com.example.vendor=Timber Inc.","com.example.name=Vector"]

include_images(optional)

A list of image names to match against. If not provided, all images will be included.

Type	Syntax	Default	Example
array	literal		["httpd","redis"]

retry_backoff_secs(optional)

The amount of time to wait before retrying after an error.

Type	Syntax	Default	Example
uint		1

multiline(optional)

Multiline parsing configuration. If not specified, multiline parsing is disabled.

Type	Syntax	Default	Example
hash	regex		[]

host_key(optional)

The key name added to each event representing the current host. This can also be globally set via the global host_key option.

Type	Syntax	Default	Example
string	literal	host

How it Works

Merging Split Messages

Docker, by default, splits log messages that exceed 16kb. This can be a rather frustrating problem because it produces malformed log messages that are difficult to work with. Vector's solves this by default, automatically merging these messages into a single message. You can turn this off via the auto_partial_merge option. Furthermore, you can adjust the marker that we use to determine if an event is partial via the partial_event_marker_field option.

State

This component is stateless, meaning its behavior is consistent across each input.

Context

By default, the docker_logs source augments events with helpful context keys.