AWS S3

Vector checks for AWS credentials in the following order:

1. The access_key_id and secret_access_key options.
2. The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
3. The credential_process command in the AWS config file (usually located at ~/.aws/config).
4. The AWS credentials file (usually located at ~/.aws/credentials).
5. The IAM instance profile (only works if running on an EC2 instance with an instance profile/role).

If no credentials are found, Vector's health check fails and an error is logged. If your AWS credentials expire, Vector will automatically search for up-to-date credentials in the places (and order) described above.

If you're using Vector to write objects across AWS accounts then you should consider setting the grant_full_control option to the bucket owner's canonical user ID. AWS provides a full tutorial for this use case. If don't know the bucket owner's canonical ID you can find it by following this tutorial.

AWS S3 supports access control lists (ACL) for buckets and objects. In the context of Vector, only object ACLs are relevant (Vector does not create or modify buckets). You can set the object level ACL by using one of the acl, grant_full_control, grant_read, grant_read_acp, or grant_write_acp options.

Vector uses two different naming schemes for S3 objects. If you set the compression parameter to true (this is the default), Vector uses this scheme:

<key_prefix><timestamp>-<uuidv4>.log.gz

If compression isn't enabled, Vector uses this scheme (only the file extension is different):

<key_prefix><timestamp>-<uuidv4>.log

Some sample S3 object names (with and without compression, respectively):

date=2019-06-18/1560886634-fddd7a0e-fad9-4f7e-9bce-00ae5debc563.log.gz
date=2019-06-18/1560886634-fddd7a0e-fad9-4f7e-9bce-00ae5debc563.log

Vector appends a UUIDV4 token to ensure there are no naming conflicts in the unlikely event that two Vector instances are writing data at the same time.

You can control the resulting name via the key_prefix, filename_time_format, and filename_append_uuid options.

Vector currently only supports AWS S3 object tags and does not support object metadata. If you require metadata support see issue #1694.

We believe tags are more flexible since they are separate from the actual S3 object. You can freely modify tags without modifying the object. Conversely, object metadata requires a full rewrite of the object to make changes.

AWS S3 offers server-side encryption. You can apply defaults at the bucket level or set the encryption at the object level. In the context, of Vector only the object level is relevant (Vector does not create or modify buckets). Although, we recommend setting defaults at the bucket level when possible. You can explicitly set the object level encryption via the server_side_encryption option.

This component is stateless, meaning its behavior is consistent across each input.

Health checks ensure that the downstream service is accessible and ready to accept data. This check is performed upon sink initialization. If the health check fails an error will be logged and Vector will proceed to start.

Vector supports dynamic configuration values through a simple template syntax. If an option supports templating, it will be noted with a badge and you can use event fields to create dynamic values. For example:

[sinks.my-sink]
dynamic_option = "application={{ application_id }}"

In the above example, the application_id for each event will be used to partition outgoing data.

This component buffers & batches data as shown in the diagram above. You'll notice that Vector treats these concepts differently, instead of treating them as global concepts, Vector treats them as sink specific concepts. This isolates sinks, ensuring services disruptions are contained and delivery guarantees are honored.

Batches are flushed when 1 of 2 conditions are met:

1. The batch age meets or exceeds the configured timeout_secs.
2. The batch size meets or exceeds the configured max_size or max_events.

Buffers are controlled via the buffer.* options.

AWS S3 offers storage classes. You can apply defaults, and rules, at the bucket level or set the storage class at the object level. In the context of Vector only the object level is relevant (Vector does not create or modify buckets). You can set the storage class via the storage_class option.

Vector will retry failed requests (status == 429, >= 500, and != 501). Other responses will not be retried. You can control the number of retry attempts and backoff rate with the request.retry_attempts and request.retry_backoff_secs options.