Monitoring and observing Vector

Although Vector is primarily used to handle observability data from from a wide variety of sources, we also strive to make Vector highly observable itself. To that end, Vector provides two sources, internal_logs and internal_metrics, that you can use to handle logs and metrics produced by Vector just like you would logs and metrics from any other source.

linkLogs

Vector provide clear, informative, well-structured logs via the internal_logs source. This section shows you how to use them in your Vector topology.

Which logs Vector pipes through the internal_logs source is determined by the log level, which defaults to info.

linkAccessing logs

You can access Vector's logs by adding an internal_logs source to your topology. Here's an example configuration that takes Vector's logs and pipes them to the console as plain text:

1[sources.vector_logs]
2type = "internal_logs"
3
4[sinks.console]
5type = "console"
6inputs = ["vector_logs"]

linkUsing Vector logs

Once Vector logs enter your topology through the internal_logs source, you can treat them like logs from any other system, i.e. you can transform them and send them off to any number of sinks. The configuration below, for example, transforms Vector's logs using the remap transform and Vector Remap Language and then stores those logs in Clickhouse:

1[sources.vector_logs]
2type = "internal_logs"
3
4[transforms.modify]
5type = "remap"
6inputs = ["vector_logs"]
7
8# Reformat the timestamp to Unix time
9source = '''
10  .timestamp = to_unix_timestamp!(to_timestamp!(.timestamp))
11'''
12
13[sinks.database]
14type = "clickhouse"
15inputs = ["modify"]
16host = "http://localhost:8123"
17table = "vector-log-data"

linkConfiguring logs

linkLevels

Vector logs at the info level by default. You can set a different level when starting up your instance using either command-line flags or the LOG environment variable. The table below details these options:

Method	Description
-v flag	Drops the log level to debug
-vv flag	Drops the log level to trace
-q flag	Raises the log level to warn
-qq flag	Raises the log level to error
-qqq flag	Disables logging
LOG=<level> environment variable	Set the log level. Must be one of trace, debug, info, warn, error, off.

linkStack traces

You can enable full error backtraces by setting the RUST_BACKTRACE=full environment variable. More on this in the Troubleshooting guide. You can

linkMetrics

You can monitor metrics produced by Vector using the internal_metrics source. As with Vector's internal logs, you can configure an internal_metrics source and use the piped-in metrics however you wish. Here's an example configuration that

linkMetrics catalogue

The table below provides a list of internal metrics provided by Vector. See the docs for the internal_metrics source for more detailed information about the available metrics.

{{< internal-metrics-list >}}

linkTroubleshooting

More information in our troubleshooting guide:

{{< jump "/guides/level-up/troubleshooting" >}}

linkHow it works

linkEvent-driven observability

Vector employs an event-driven observability strategy that ensures consistent and correlated telemetry data. You can read more about our approach in RFC 2064.

linkLog rate limiting

Vector rate limits log events in the hot path. This enables you to get granular insight without the risk of saturating IO and disrupting the service. The trade-off is that repetitive logs aren't logged.