
**Vector Remap Language** (VRL) is an [expression-oriented][expression_oriented]
language designed to work with observability data (logs and metrics) in a _safe_
and _performant_ manner. It features a [simple syntax][vrl_expressions], a rich
set of [built-in functions][vrl_functions] tailored to observability use cases,
and [numerous features](#features) that set it far apart from other options.
This [0.12][0_12] release of Vector marks the official release of the language.

VRL has been under intense but careful development for several months. We made
it available in beta beginning with the [0.11][0_11] release, but as of this
[0.12][0_12] release VRL is now generally available (GA). We're confident that
it's a _major_ step forward for Vector and its users and the observability space
in general. Special thanks to [Jean Mertz][jean] and [Stephen Wakely][stephen]
for spearheading the VRL project.

_To jump right into the language, [skip to the solution section](#solution).
Otherwise, read on for why we created the language._

## Preamble

I know what you may be thinking: another custom language? With so many out
there, some of them poorly done, challenging to learn, and just plain
unnecessary, you can be forgiven for being wary of another one on the pile. Rest
assured that we decided to create VRL only when it became clear that a new
language was the best way to improve upon our existing story for transforming
data.

But it's important to know at the outset that VRL is _not a programming
language_ in the fullest sense. It lacks a wide range of programming constructs
that you find in all-purpose languages, such as loops, classes, modules, custom
functions, and IO access. This simplicity is _intentional_. We're confident that
Vector users will not chafe at the limitations of VRL and will in fact embrace
them as guarantors of sound design.

## The problem

Up to the 0.12 release, Vector has provided two types of transforms: **static
transforms** and **runtime transforms**.

- Static transforms do exactly _one_ particular thing and are configuration
  based. The now-deprecated `remove_fields` transform, for example, would remove
  the fields that you specify in your Vector configuration.

- Runtime transforms enable you to modify event data using full-blown language
  runtimes, such as [Lua][lua] and [WebAssembly][wasm] (Wasm).

Both of these options have enabled our users to transform their data
successfully, but they have severe limitations that we needed to address with
VRL:

1. Static transforms, while fast, are rigid, limited, and hard to manage. They
   leverage a configuration syntax for expressing data transformations. As a
   result, we've witnessed otherwise simple pipelines turn into hundreds of
   lines of configuration.

2. Runtime transforms are very robust, but users pay a steep performance and
   safety cost when using them. The robustness of using full-blown programming
   languages makes it very easy to write slow and error-prone programs that are
   difficult to collaborate on across a team.

The classic quadrant analysis looks something like this:

![VRL Comparison](/img/blog/vrl-quadrant-comparison.svg)

VRL eliminates this trade-off. Let's dig deeper on each of these points.

### Configuration languages are bad at expressing data transformations {#config-languages}

It's commonplace for observability pipelines to offer a rigid list of _static_
transforms that leverage a configuration syntax. Some call them "transforms",
others call them "filters" or "functions". If you zoom in, they make more sense
-- they perform a single task and make it difficult for users to do the wrong
thing. If you zoom out, they start to look like poorly designed programming
languages trying to be both a configuration _and_ data transformation language.

:::quote
If you zoom out, they start to look like poorly designed programming languages,
trying to be both a configuration _and_ data transformation language.
:::

If you've worked with observability pipelines, it's not uncommon to have
hundreds of lines of configuration for relatively simple pipelines.

To demonstrate, let's transform a common-log (Apache) coming from Docker, a very
common use case:

```json
{
  "time": "2021-02-03T21:13:54.713161211Z",
  "stream": "stdout",
  "log": "5.86.210.12 - zieme4647 [03/Feb/2021:21:13:55 -0200] \"GET /embrace/supply-chains/dynamic/vertical HTTP/1.0\" 201 20574"
}
```

Parsing this log, without VRL, looks like this:

:::tabs

# Vector

```toml title="vector.toml"
# ... sources ...

# Parse the internal Syslog log
[transforms.parse_syslog]
type = "regex_parser"
inputs = ["parse_docker"]
patterns = ['^(?P<host>\S+) (?P<client>\S+) (?P<user>\S+) \[(?<timestamp>[\w:/]+\s[+\-]\d{4})\] "(?<method>\S+) (?<resource>.+?) (?<protocol>\S+)" (?<status>\d{3}) (?<bytes_out>\S+)$']
field = "log"

# Remove Docker time and log fields since they are duplicative
[transform.remove_log]
type = "remove_fields"
inputs = ["parse_syslog"]
fields = ["time", "log"]

# Coerce parsed fields
[transforms.coerce_fields]
type = "coercer"
inputs = ["remove_log"]
types.timestamp = "timestamp"
types.status = "int"
types.bytes_out = "int"

# ... sinks ...
```

# Logstash

```text title="logstash.conf"
# ... inputs ...

filter {
  grok {
    match => { "message" => ["%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?",
      "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"-\" %{NUMBER:[apache2][access][response_code]} -" ] }
    remove_field => "message"
  }
  mutate {
    remove_field => [ "time", "log" ]
  }
  date {
    match => [ "[apache2][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
    remove_field => "[apache2][access][time]"
  }
  mutate {
    coerce => {
      "[apache2][access][body_sent][bytes]" => "integer"
      "[apache2][access][response_code]" => "integer"
    }
  }
}

# ... outputs ...
```

# FluentD

```text title="fluentd.conf"
<source>
  # ... source options ...
  format apache2
  tag apache.access
</source>

<parse>
  @type regexp
  expression /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
  time_format %d/%b/%Y:%H:%M:%S %z
  types code:integer,size:integer
</parse>

<filter apache.access>
  @type record_transformer
  remove_keys time,log
</filter>

# ... outputs ...
```

:::

Which results in this general output:

```json
{
  "bytes_out": 20574,
  "host": "5.86.210.12",
  "method": "GET",
  "resource": "/embrace/supply-chains/dynamic/vertical",
  "protocol": "HTTP/1.0",
  "status": 201,
  "timestamp": "2021-02-03T23:13:55Z",
  "client": "-",
  "user": "zieme4647"
}
```

As you can see, configuration languages become verbose for even simple
pipelines, making them non-ideal for expressing data transformations. This is
because they have competing concerns: the better a language is at configuration,
the worse it is at data transformation.

:::quote
The better a language is at configuration, the worse it is at data
transformation.
:::

Instead of conflating these concerns, like Logstash, Fluentd, and others, Vector
separates them, allowing you to choose your preferred configuration language
(TOML, YAML, or JSON) while offering a purpose-built language for data
transformation (VRL). But is VRL really necessary? Couldn't you leverage Lua,
Javascript, or any other existing language?

### Runtime transforms are slow and unsafe {#runtime-problems}

**Runtime** transforms enable you to modify event data using the full power of a
programming language runtime, such as Lua or Javascript. They're robust enough
to handle even the thorniest use cases, but they have significant downsides that
make them high-risk for critical infrastructure like observability pipelines:

1. First, you pay a **significant performance penalty**. For example, on
   average, Lua is about 60% slower than Vector's Rust-based static transforms,
   and other runtimes like Javascript are even slower.

2. Second, there are **severe security and safety risks** that make them show
   stoppers for critical infrastructure like observability pipelines. Things
   like the lack of memory-safety, dynamic code evaluation, access to IO,
   unaudited dependency trees, lack of sandboxing, and all of the security risks
   that come with that language runtime.

3. Finally, the extreme optionality presents **easy foot-guns** that manifest
   into real-world performance and reliability problems, making them difficult
   to manage.

Runtime transforms acted as robust escape hatches for Vector users, unblocking
exotic use cases and allowing us to learn more about the many ways people use
Vector. Still, we've seen the dark side of using them:

- Unexpected malformed data bringing pipelines down and waking operators up in
  the middle of the night.

- A poorly managed dependency tree introducing severe security risks,
  compromising their most sensitive data.

- Slow performance unable to keep up with data volume fluctuations.

- Complicated code leaving pipelines in an unmanageable state.

Because of these risks we recommend static transforms if given the choice, but
you shouldn't have to choose...

## Our solution: Vector Remap Language {#solution}

Before I dive into the specifics of VRL, let's look at the Docker, common-log
(Apache) example from above. If you skipped to this section we'll be parsing the
following log:

```json
{
  "time": "2021-02-03T21:13:54.713161211Z",
  "stream": "stdout",
  "log": "5.86.210.12 - zieme4647 [03/Feb/2021:21:13:55 -0200] \"GET /embrace/supply-chains/dynamic/vertical HTTP/1.0\" 201 20574"
}
```

Which can be achieved with the following VRL program:

```perl
. = parse_common_log!(.log)
.total_bytes = del(.size)
.internal_request = ip_cidr_contains("5.86.0.0/16", .host) ?? false
```

Resulting in this output:

```json
{
  "host": "5.86.210.12",
  "internal_request": true,
  "user": "zieme4647",
  "timestamp": "2021-02-03T23:13:55Z",
  "message": "GET /embrace/supply-chains/dynamic/vertical HTTP/1.0",
  "method": "GET",
  "path": "/embrace/supply-chains/dynamic/vertical",
  "protocol": "HTTP/1.0",
  "total_bytes": 20574,
  "status": 201
}
```

As you can see, the above configuration is _significantly_ easier to write,
read, and manage. Also, it maintains optimal performance and doesn't present the
[various security and safety](#runtime-problems) problems that come with runtime
transforms like Lua and Javascript. You'll notice we threw in an extra field,
`internal_request`, to demonstrate how VRL solves otherwise difficult tasks for
static transforms.

The syntax was originally inspired by [jq] (note the `.` based paths), but
evolved into a simple multi-line language. For a deep dive into the language
constructs, check out the [reference documentation][vrl_reference]. Here you
will find a comprehensive list of [expressions][vrl_expressions],
[functions][vrl_functions], and [examples][vrl_examples].

### VRL principles and features {#features}

VRL was designed to exploit two principles: _safety_ and _performance_, while
maintaining flexibility. This makes VRL ideal for always-up,
performance-sensitive infrastructure like observability pipelines. To illustrate
how we achieve this, below is a VRL feature matrix across these two principles:

| Feature                   | Safety | Performance |
| :------------------------ | :----: | :---------: |
| [Progressive type safety] |   ✅   |             |
| [Fail safety]             |   ✅   |             |
| [Memory safety]           |   ✅   |             |
| [Ergonomic safety]        |   ✅   |     ✅      |
| [Vector/Rust native]      |   ✅   |     ✅      |
| [Stateless]               |   ✅   |     ✅      |

For more info on each click on the feature, but for the purposes of
demonstrating how VRL is unique, let's touch on the first two: _progressive type
safety_ and _fail safety_.

### Progressive type and fail safety {#type-safety}

A unique design decision behind VRL is its implementation of type and fail
safety. After seeing many users deal with pipeline instability due to runtime
errors, we made type and fail safety cornerstones. This makes VRL programs
**infallible**, ensuring that they will work as expected in production. Let's
demonstrate with an example, using the same Docker, common-log (Apache) example
from above.

Given this log:

```json
{
  "time": "2021-02-03T21:13:54.713161211Z",
  "stream": "stdout",
  "log": "5.86.210.12 - zieme4647 [03/Feb/2021:21:13:55 -0200] \"GET /embrace/supply-chains/dynamic/vertical HTTP/1.0\" 201 20574"
}
```

We want to parse it into this result:

```json
{
  "host": "5.86.210.12",
  "user": "zieme4647",
  "timestamp": "2021-02-03T23:13:55Z",
  "message": "GET /embrace/supply-chains/dynamic/vertical HTTP/1.0",
  "method": "GET",
  "path": "/embrace/supply-chains/dynamic/vertical",
  "protocol": "HTTP/1.0",
  "total_bytes": 20574,
  "status": 201
}
```

Someone new to VRL might write the following VRL program:

```perl
. = parse_common_log(.log)
.total_bytes = del(.size)
```

And they'll be greeted with this thoughtful error message at compile-time
(Vector boot):

```rust
error[E103]: unhandled fallible assignment
  ┌─ :1:5
  │
1 │ . = parse_common_log(.log)
  │ --- ^^^^^^^^^^^^^^^^^^^^^^
  │ │   │
  │ │   this expression is fallible
  │ │   update the expression to be infallible
  │ or change this to an infallible assignment:
  │ ., err = parse_common_log(.log)
  │
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 103 at https://errors.vrl.dev/103
  = see language documentation at https://vrl.dev
```

As you can see, VRL requires the user to handle any expression that can result
in a runtime error. In our case, `.log` might not be a string, so we need to
either specify the type of the `.log` field or handle the error in the event
that `.log` is not a string. To resolve this error, the user must do one of
three things:

1.  **Handle the error**

    ```perl
    ., err = parse_common_log(.log)
    if err != null {
      .malformed = true
      log("Failed to parse common-log: " + err, level: "error")
    } else {
      .total_bytes = del(.size)
    }
    ```

    While subtle, this has valuable implications! If the event is malformed
    (`.log` is not a formatted common-log string), then we forgo log an error
    and add a `.malformed` field. This preserves the original data and makes it
    easy to [route][route_transform] the malformed data for inspection. Not
    handling this error is a _very_ common mistake that would otherwise result
    in data loss and downtime.

2.  **Raise the error and abort**

    ```perl
    . = parse_common_log!(.log)
    .total_bytes = del(.size)
    ```

    In some cases, malformed data is unacceptable and the program must be
    aborted. In this case VRL offers _fallible function variants_. This is a
    fancy way of saying that suffixing your function call with `!` will raise an
    error upon failure and abort the program. Vector itself will continue on to
    process the next event, but will also log an error message to alert
    operators. Again, this forces the users to decide how to handle errors
    instead of being surprised by them.

3.  **Specify types**

    ```perl
    .log = to_string!(.log)

    ., err = parse_common_log(.log)
    if err != null {
      # This error only occurs for malformed *strings*
      log("Failed to parse common-log: " + err, level: "error")
    } else {
      .total_bytes = del(.size)
    }
    ```

    Finally, a caveat of the two examples above is that it's hard to discern
    between a type error and a parsing error. Perhaps you want to abort on type
    errors and handle parsing errors? You can achieve this through _progressive
    type safety_. As a VRL program evaluates, it builds up type information
    about your fields. Once the type is known, subsequent usage of that field
    forgoes runtime type errors. Therefore, in the above example we know that
    the only possible way an error can occur is if parsing fails.

We're just scratching the surface on the language. For more information, check
out the VRL [reference][vrl_reference], [expressions][vrl_expressions],
[functions][vrl_functions], and [examples][vrl_examples].

## Should I start using VRL? {#should-you-start-using?}

The short answer is an emphatic **yes**. For the use cases that VRL can
cover—and that should be most use cases—it should be seen as the preferred
option beginning with this release. If you aren't sure if your use case is
covered, don't ever hesitate to visit us [on Discord][chat].

## Coming soon

Although VRL should be considered ready for production use cases, there's more
on the way in the next few releases:

- The initial lineup of VRL functions can ably cover many use cases but we do
  have more functions in the pipeline.

- Expect major new features for simple object traversal.

- A web playground that enables you to experiment with VRL in the browser.

- Improved type safety and ergonomics through event schemas.

So stay tuned for much more work in this area. But for now, we strongly
encourage to explore the [VRL documentation][vrl_reference] and get in touch
with us [on Discord][chat] if you have any issues, comments, or suggestions.

[0_11]: /releases/0.11.0
[0_12]: /releases/0.12.0
[affine type system]: /docs/reference/vrl/#affine-type-system
[chat]: https://chat.vector.dev
[ergonomic safety]: /docs/reference/vrl/#ergonomic-safety
[expression_oriented]: https://en.wikipedia.org/wiki/Expression-oriented_programming_language
[fail safety]: /docs/reference/vrl/#fail-safety
[jean]: https://github.com/JeanMertz
[jq]: https://stedolan.github.io/jq/
[lua]: https://vector.dev/docs/reference/transforms/lua/
[memory safety]: /docs/reference/vrl/#memory-safety
[progressive type safety]: /docs/reference/vrl/#progressive-type-safety
[route_transform]: /docs/reference/configuration/transforms/route/
[wasm]: /docs/reference/configuration/transforms/wasm/
[stateless]: /docs/reference/vrl/#stateless
[stephen]: https://github.com/FungusHumungus
[vector/rust native]: /docs/reference/vrl/#vector-rust-native
[vrl_examples]: /docs/reference/vrl/examples/
[vrl_expressions]: /docs/reference/vrl/expressions/
[vrl_functions]: /docs/reference/vrl/functions/
[vrl_reference]: /docs/reference/vrl/


Vector Remap Language


## Table of Contents

You can automatically generate a table of contents by adding `toc: true` to the frontmatter of any blog post. Headings will be automatically hyperlinked to the appropriate element and deault to using the header text as a value.This can be customized by appending `{#custom-id}` to the end of the text. This id will be slugified and used as both in the table of contents as well as the `id` value for the href in the link.

## Variables

Here is a [link][link_example] using a variable defined in the file.

## Tables

| Feature                   | Safety | Performance |
| :------------------------ | :----: | :---------: |
| [Progressive type safety] |   ✅   |             |
| [Fail safety]             |   ✅   |             |
| [Memory safety]           |   ✅   |             |
| [Ergonomic safety]        |   ✅   |     ✅      |
| [Vector/Rust native]      |   ✅   |     ✅      |
| [Stateless]               |   ✅   |     ✅      |

## Syntax Highlighting

```rust
error[E103]: unhandled fallible assignment
  ┌─ :1:5
  │
1 │ . = parse_common_log(.log)
  │ --- ^^^^^^^^^^^^^^^^^^^^^^
  │ │   │
  │ │   this expression is fallible
  │ │   update the expression to be infallible
  │ or change this to an infallible assignment:
  │ ., err = parse_common_log(.log)
  │
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 103 at https://errors.vrl.dev/103
  = see language documentation at https://vrl.dev
```

```toml
[prop]
  test = true
    thing = false
```

```javascript hide_line_numbers
function test() {
  if (thing) {
    return true;

    if (false) {
      return true;
    }
  }
}
```

## Tabs

:::tabs

## Test

What happens?

## Test 2

What happens?

:::

## Block Quotes

> Good timber does not grow with ease:
> The stronger wind, the stronger trees;
> The further sky, the greater length;
> The more the storm, the more the strength.

## Jumps

Jump components give you a stylish way to link to other documents by using a customer directive like so:

::jump{to="/docs/setup/quickstart" title="Transformation"}

## Admonitions

:::note
This is used to provide some helpful info to the user about the current doc but should be a litle less attention grabbing.
:::

:::tip
Welcome to the thunderdome! This is used to provide some helpful info to the user about the current doc.
:::

:::success
This can be used to show an example of something in the document while specifically calling attention to it.
:::

:::warning
This is a warning message, and can be used to warn a user about something they might need to look out for.
:::

:::caution
This is a very important caution message, and can be used to show that something is very important to pay attention to.
:::

:::danger
This is a very important caution message, and can be used to show that something has gone wrong or is critical to pay attention to.
:::

:::example
This can be used to show an example of something in the document while specifically calling attention to it.
:::

[link_example]: /docs/reference/vrl/
[0_11]: /releases/0.11.0
[0_12]: /releases/0.12.0
[affine type system]: /docs/reference/vrl/#affine-type-system
[chat]: https://chat.vector.dev
[ergonomic safety]: /docs/reference/vrl/#ergonomic-safety
[expression_oriented]: https://en.wikipedia.org/wiki/Expression-oriented_programming_language
[fail safety]: /docs/reference/vrl/#fail-safety
[jean]: https://github.com/JeanMertz
[jq]: https://stedolan.github.io/jq/
[lua]: https://vector.dev/docs/reference/transforms/lua/
[memory safety]: /docs/reference/vrl/#memory-safety
[progressive type safety]: /docs/reference/vrl/#progressive-type-safety
[route_transform]: /docs/reference/configuration/transforms/route/
[wasm]: /docs/reference/configuration/transforms/wasm/
[stateless]: /docs/reference/vrl/#stateless
[stephen]: https://github.com/FungusHumungus
[vector/rust native]: /docs/reference/vrl/#vector-rust-native
[vrl_examples]: /docs/reference/vrl/examples/
[vrl_expressions]: /docs/reference/vrl/expressions/
[vrl_functions]: /docs/reference/vrl/functions/
[vrl_reference]: /docs/reference/vrl/


Using Markdown


# This is an h1

## This is an h2

### This is an h3

# This is another h1

:::tabs

# First Tab

```javascript
function test() {
  if (thing) {
    return true;

    if (false) {
      return true;
    }
  }
}
```

# Second Tab

```javascript
function hello() {
  if (thing) {
    return true;

    if (false) {
      return true;
    }
  }
}
```

### Third Tab

```java
public class MainActivity extends AppCompatActivity {
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
    }

    /** Called when the user taps the Send button */
    public void sendMessage(View view) {
        // Do something in response to button
    }
}
```

:::


Markdown Testing


We're excited to announce that Timber and Vector are now officially part of [Datadog][datadog]!

When we started building Vector, we set out to provide an entirely new kind of data pipeline. We were guided by a belief that when it comes to collecting, transforming, and routing observability data, teams shouldn’t have to accept trade-offs between cost, performance, reliability, and security. They also shouldn’t have to stitch together disparate tools just to get their data from point A to B; the ideal solution should be simple, yet robust, and unify the entire pipeline. We’ve dedicated ourselves to that challenge, and we’re proud of the platform we’ve created.

We started talking to the team at Datadog back in 2017. They have a unique ability to take deep technical innovation and make it accessible to engineering teams through an intuitive and seamless user experience. We’ve always admired their focus on solving real customer problems and engineering excellence, and the breadth of their quality integrations—something we’ve sought to emulate through our work with Vector.

Since we launched the project in 2019, Vector has grown beyond our expectations. It is currently downloaded millions of times per month and used by hundreds of forward-thinking enterprises to collect, process, and take control of their observability data. This growth has significantly intensified the demands on the project and our team, dramatically increasing our integration count, creating unprecedented performance challenges, and adding complex enterprise requirements.

Joining Datadog means that Vector now has a safe, sustainable, long-term home where we can continue to evolve the project to meet the ever-changing needs of modern engineering teams at scale. We will have the resources to expand the team and execute on an ambitious roadmap, all while drawing on the years of observability experience that the Datadog team brings to the table. We plan to double down on Vector’s reliability and performance, solidifying it as the fastest and most reliable observability data platform, and to introduce numerous features designed for enterprise scale deployments of mission-critical observability pipelines.

Without the Vector community, this project would not be what it is today. You all have contributed heavily to the life of this project, providing us with more feedback, GitHub issues, pull requests, feature requests, and lively Discord discussions than we can count. We give our sincere thanks to everyone who has taken part in building something that we can all be proud of, and hope to have you with us as we enter this next chapter.

[datadog]: https://www.datadoghq.com


Accelerating Vector with Datadog


Although Vector is an observability tool, it's nonetheless important to be able
to observe Vector itself, especially in production environments where it serves
as critical infrastructure. That's why we're excited to announce the new [Vector
GraphQL API](https://vector.dev/docs/reference/api/), available in
[v0.11.0](https://vector.dev/releases/0.11.0/).

The API enables ad-hoc querying of:

- Your Vector topology, including
  [`sources`](https://vector.dev/docs/reference/sources/),
  [`transforms`](https://vector.dev/docs/reference/transforms/) and
  [`sinks`](https://vector.dev/docs/reference/sinks/).
- Uptime and health information.
- Event processing, byte processing, and error metrics, both per component and
  in aggregate across the Vector instance.
- Changes to your pipeline configuration, in real time.

This is just the beginning. In the near future, the API will enable you to both
observe _and_ control Vector remotely, programmatically, and on demand. Stay
tuned!

## Take it for a spin in our live playground

Try our new [**GraphQL public playground
today**](https://playground.vector.dev:8686/playground), which enables you to
execute queries against a live Vector instance that we host and manage:

![Vector GraphQL API Public Playground](/img/blog/vector-api-public-playground.png)

Here are a few queries you can try:

### Health and uptime

See how long it's been since Vector last restarted, with streaming uptime that
updates every second:

```graphql
subscription {
  uptime {
    seconds
    timestamp
  }
}
```

### Get configured topology + metrics

Get sources, transforms, and sinks configured by
[`vector.toml`](https://vector.dev/docs/setup/configuration/), along with total
events, bytes, and errors processed per component:

```graphql
query {
  components {
    __typename
    componentId
    processedEventsTotal {
      processedEventsTotal
    }
    ... on Sink {
      processedBytesTotal {
        processedBytesTotal
      }
    }
  }
}
```

### Hardware metrics

Get insight into the hardware Vector's running on, from memory and CPU usage to
network activity and more.

```graphql
query {
  hostMetrics {
    memory {
      totalBytes
      freeBytes
      usedBytes
      activeBytes
      availableBytes
      inactiveBytes
      buffersBytes
      cachedBytes
      sharedBytes
      wiredBytes
    }
    network {
      receiveBytesTotal
      receiveErrsTotal
      receivePacketsTotal
      transmitBytesTotal
      transmitErrsTotal
      transmitPacketsDropTotal
      transmitPacketsTotal
    }
    filesystem {
      freeBytes
      totalBytes
      usedBytes
    }
    cpu {
      cpuSecondsTotal
    }
    swap {
      freeBytes
      totalBytes
      usedBytes
      swappedInBytesTotal
      swappedOutBytesTotal
    }
    loadAverage {
      load1
      load5
      load15
    }
    disk {
      readBytesTotal
      readsCompletedTotal
      writtenBytesTotal
      writesCompletedTotal
    }
  }
}
```

## Enabling the Vector GraphQL API

The API is an opt-in feature that spawns an HTTP server to accept queries.

To enable, open your `vector.toml`, and add the following:

```toml
[api]
  enabled = true
  address = "127.0.0.1:8686" # optional. Change IP/port if required
```

After restarting Vector, you can access an API playground at
[http://localhost:8686/playground](http://localhost:8686/playground).

## Why we chose GraphQL

The Vector API uses [GraphQL](https://graphql.org), a simple, type-safe,
flexible query language.

We chose GraphQL over REST or gRPC for a few reasons:

- It's type safe. Clients can introspect schema and discern whether a query is
  valid before it's executed.
- Relational data (like Vector topology) is natural to model and query for.
- Data can be streamed via subscriptions over WebSockets, making it ideal for
  high-frequency metrics and a pub/sub pattern for observing topology changes.
- Data is returned in pure JSON, making it simple, readable, and easy to parse.
- There's [great language/library
  tooling](https://github.com/chentsulin/awesome-graphql), making it trivial to
  interop with existing applications.
- Compile-time type safety in Rust, via
  [async-graphql](https://github.com/async-graphql/async-graphql) (server) and
  [graphql-client](https://github.com/graphql-rust/graphql-client).
- Universal browser support, making it an ideal protocol for web usage (or
  apps!).

## Our first API clien: `top`

To help you visualize your Vector instance in action,
[v0.11.0](https://vector.dev/releases/0.11.0/) also brings you `vector top`, our
new terminal dashboard CLI tool powered by the Vector GraphQL API.

To enable it, run:

```bash
vector top # pass --url <http://path/to/graphql> for remote observability
```

On Linux, Windows and macOS, you'll get an interface like this:

![Vector top](/img/blog/vector-top.png)

This will display your configured components and metrics, updating
every second (pass a millisecond `--interval` to adjust). If you run Vector in
[watch mode](https://vector.dev/docs/reference/cli/#vector_watch_config), it'll
even pick up topology changes automatically.

I'll be telling you more about `vector top` soon in a dedicated blog post.

## What's next?

This release introduces the API and a sample client, but we're just getting
started. In future releases, we'll be expanding the API's capabilities to
include:

- Validating and changing configuration programmatically. So far, all of our
  queries have been read-only. Future versions will include GraphQL mutations
  that allow changes to a running Vector pipeline (think: HTTP POST-style
  requests).

- An exciting new web UI for visualising and interacting with topology.

- Component-specific types and metrics that can provide much deeper and more
  granular insight into what's happening in your topology

Follow us at [@vectordotdev](https://twitter.com/vectordotdev) to be notified of
updates.

## Further reading

GraphQL is an API protocol introduced by Facebook. You can learn more at
[https://graphql.org](https://graphql.org)

For clients and language support, check out [Awesome
GraphQL](https://github.com/chentsulin/awesome-graphql), a curated list of
libraries and other helpers.

For a deeper dive into the currently supported fields in the Vector API, your
best source of information is the "Docs" sidebar in the API playground. As a
type-safe interface, GraphQL is self-documenting. Review the inline comments to
learn more about what a field returns. We'll eventually add to this with
dedicated API docs, but this is a great way to get started.

In future posts we'll be exploring the GraphQL ecosystem in more detail and
explaining the mechanics of what made GraphQL a winning choice for the Vector
API, over some of the alternatives such as REST or gRPC &dash; including a deep
dive into Rust libs and code snippets.

Stay tuned!


The new GraphQL API for Vector


After months of development, [over 100 pull requests][pulls], and intensive QA in clusters producing over 20 terabytes of event data a day, we're pleased to announce our first-class [Kubernetes] integration for Vector in [version 0.11][v_0_11]. We strove to make even this initial integration rock solid and production ready because we aim to make Vector the default pipeline for all Kubernetes observability data.

To cut straight to the chase, checkout the [installation instructions][install], otherwise read on for the details.

## A new source for Kubernetes log data

The crux of our Kubernetes integration for Vector is the new [`kubernetes_logs`][kubernetes_logs] source. This source does all of the following:

- Automatically collects logs from all [Nodes][node_logging] in your cluster
- Automatically merges logs that are split due to the 16kb limit imposed by Docker
- Communicates with the Kubernetes API to [enrich][enriching] logs with Kubernetes metadata, including:
  - Container name and image
  - Pod uid, name, namespace, and labels
  - Node name
- Provides robust filtering options for including and excluding Pods and individual containers within Pods

This simple Vector configuration, for example, would suffice to send logs from _all_ of your Nodes/Pods to [Elasticsearch]:

```toml title="vector.toml"
[sources.k8s_all]
type = "kubernetes_logs"

[sinks.es_out]
type = "elasticsearch"
inputs = ["k8s_all"]
host = "http://your-elasticsearch-cluster:9200"
index = "vector-k8s-%F"
```

Using this simple topology as a starting point, you can [transform][transforms] your log data in a wide variety of ways, ship it to [other sinks][sinks], and much more.

## Adaptive concurrency

Vector 0.11 also includes a feature called **Adaptive Request Concurrency** (ARC). ARC uses a battle-tested algorithm to automatically optimize concurrency limits for HTTP [sinks] based on downstream service responses. The benefit of ARC is that it enables you to safely move beyond static rate limiting when using systems like [Elasticsearch] and [Clickhouse]. Simply enable ARC with a single line of configuration and you can fully utilize your services' bandwidth while ensuring that Vector doesn't over-tax those systems.

ARC isn't specific to Kubernetes but it _is_ particularly useful in highly elastic Kubernetes environments, where static rate limiting runs the risk of underutilizing system resources and/or jeopardizing system reliability. For more info on ARC, check out our [announcement post][arc].

## Installing Vector in your Kubernetes cluster

To install Vector in your Kubernetes cluster, check out our [Kubernetes installation documentation][install], which includes instructions for:

- Standard installation using [kubectl] and [Kustomize]

- Deployment using [Helm]

Both installation options enable you to easily deploy Vector [agents][agent] to your Nodes. In an upcoming release, we'll also provide support for deploying Vector as a centralized [aggregator].

## Why Vector for your Kubernetes cluster?

We all know that the Kubernetes ecosystem is _packed_ with options. The now-infamous CNCF landscape includes [over 90][tools] observability tools. Amidst such plenty, it's not surprising that many Kubernetes clusters run multiple observability tools side by side, often on the same Node or even in the same Pod. But when you run ever-more resource-hungry processes side by side, that can lead to **agent fatigue**, a costly problem that Vector tackles head-on.

To give a real-world example of agent fatigue, let's look at a telecom enterprise before and after adopting Vector.

### Before Vector

Prior to adopting Vector, the company deployed four or more agents (!) on each of their Kubernetes Nodes:

![The crowded Kubernetes Node](/img/blog/k8s-before.png)

That's one agent for each of the following: logs, metrics, traces, and security (e.g. breach detection). And then on top of four agents per Node, they needed to run a cluster of observability data aggregators for cross-Node analysis and batching/archiving as well as a Kafka cluster in front of those aggregators (to address reliability and durability concerns).

All together, those four agents consumed ~30% of the Node's computing resources, while Kafka/aggregator combination was the company's largest services in terms of resource usage (what!?). In total, a whopping ~40% of their resources were dedicated to observability infrastructure. And this resource intensity doesn't even include in the engineering time and effort required to deploy and manage all of it:

![The crowded Kubernetes Node](/img/blog/k8s-resources-before.png)

Although this is an extreme case, we'd bet that you can relate to some—though hopefully not all!—of these troubles. What makes this problem so pernicious is that it happens over time. This heavy resource usage was not the result of an elaborate master plan by these engineers; it was the byproduct of evolving company needs, fast-paced engineering, fragmentation in the observability space, and vendor lock-in. That last one can be a real killer.

### After Vector

By switching to Vector, the telecom company from the "before" scenario was able to consolidate the logs, metrics, and tracing agents:

![The quiet Kubernetes Node](/img/blog/k8s-after.png)

This reduced observability resource usage by 90%. This brought resource usage from 40% of the total down to 5%. In addition, Vector reduced Splunk data volume through sampling and cleaning, afforded by using AWS S3 as the system of record. All in all, Vector significantly reduced the overall observability costs. The result looked like this:

![The quiet Kubernetes Node](/img/blog/k8s-resources-after.png)

To summarize, adopting the new Vector-based architecture meant:

- One observability tool
- One workflow to deploy and manage that tool
- Reduced infrastructure cost
- Reduced Splunk cost
- Reduced engineering and operations time
- Reduced lock-in, as the company can now modify their observability setup by re-configuring Vector rather than swapping out agents

## More to come

We're confident that this initial integration is up to the task of dramatically reducing observability costs and headaches in your Kubernetes cluster. But we do have more on the way here, including native tracing support for [OpenTelemetry]-compliant traces and a variety of new [Prometheus]-related features.

We want Vector to become the backbone of observability in Kubernetes. If you share our vision of a single, ultra-fast, ultra-reliable, open-source observability platform, give us a try! As a user-first company, our entire engineering team is [actively engaged][chat] with our community and eager to work with you.

::jump{to=/docs/setup/installation/platforms/kubernetes title="Install Vector on Kubernetes"}

[agent]: /docs/setup/deployment/topologies/#distributed
[aggregator]: /docs/setup/deployment/topologies/#centralized
[arc]: /blog/adaptive-request-concurrency
[chat]: https://chat.vector.dev
[clickhouse]: /docs/reference/configuration/sinks/clickhouse
[enriching]: /docs/reference/configuration/sources/kubernetes_logs/#output
[elasticsearch]: /docs/reference/configuration/sinks/elasticsearch
[helm]: https://helm.sh
[install]: /docs/setup/installation/platforms/kubernetes/#install
[kubectl]: https://kubernetes.io/docs/reference/kubectl/kubectl
[kubernetes]: https://kubernetes.io
[kubernetes_logs]: /docs/reference/configuration/sources/kubernetes_logs/
[kubernetes_logs_config]: /docs/reference/configuration/sources/kubernetes_logs/#configuration
[kustomize]: https://kustomize.io
[node_logging]: https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level
[opentelemetry]: https://opentelemetry.io
[prometheus]: https://prometheus.io
[pulls]: https://github.com/timberio/vector/pulls?q=is%3Apr+sort%3Aupdated-desc+kubernetes+is%3Aclosed
[sinks]: /docs/reference/configuration/sinks/
[tools]: https://landscape.cncf.io/card-mode?category=observability-and-analysis
[transforms]: /docs/reference/configuration/transforms/
[v_0_11]: /releases/0.11.0/


First-class Kubernetes integration for Vector


Observability pipelines have become critical infrastructure in the current technological landscape, which is why we've built [Vector] to provide extremely high throughput with the tiniest resource footprint we can manage ([Rust] is a huge help here). But this is not enough in the real world: your observability pipeline needs to provide optimal performance and efficiency while _also_ being a good infrastructure citizen and playing nicely with services like [Elasticsearch] and [Clickhouse].

And so we're excited to announce that Vector version 0.11 includes support for **Adaptive Request Concurrency** (ARC) in all of its HTTP-based [sinks]. This feature does away with static rate limits and automatically optimizes HTTP concurrency limits based on downstream service responses. The underlying [mechanism](#how-it-works) is a feedback loop inspired by TCP congestion control algorithms.

## The lead-up

One of the most common support questions we get about Vector involves logs like this:

```text
TRACE tower_limit::rate::service: rate limit exceeded, disabling service
```

Users typically have two questions about this:

1. What does it mean?
1. How can I fix it?

The answer to the first question is simple: Vector has _internally_ rate-limited processing to respect user-configured limits—[`request.rate_limit_duration_secs`][rate_limit_duration_secs] and [`request.rate_limit_num`][rate_limit_num]—for that particular [sink][sinks]. In other words, Vector has intentionally reduced performance to stay within static limits.

The answer to the second question—how to fix it—is more complex because it depends on a variety of factors that change over time (covered in more detail [below](#rate-limiting-problem)). Telling the user to raise their limits would be irresponsible since we'd then risk overwhelming the downstream service and causing an outage; but not changing them could mean limiting performance in a dramatic way.

:::quote
In one case, we found that rate limits were limiting performance by over 80%.
:::

The crux of the matter is that Vector's high throughput presents a major challenge for HTTP-based sinks like [Elasticsearch] because those services can't always handle event payloads as quickly as Vector can send them. And when data services are heavily interdependent—which is almost always!—letting Vector overhwelm one of them can lead to system-wide performance degradation or even cascading failures.

In versions of Vector prior to 0.11, you could address this problem by setting [**rate limits**][rate limit] on outbound HTTP traffic to downstream services. Rate limiting certainly _does_ help prevent certain worst-case scenarios but customer feedback and our own internal QA has revealed that this approach also has deep limitations.

## The problem: rate limiting is not a panacea {#rate-limiting-problem}

Rate limiting is nice to have as a fallback but it's a blunt instrument, a static half-solution to a dynamic problem. The core problem is that configuring your own rate limits locks you into a perpetual loop:

![The rate limiting decision loop](/img/blog/rate-limiting-loop.png)

Within this vicious loop, you need to constantly avoid two outcomes:

- You set the limit too high and thus **compromise system reliability** by overwhelming your services. Time to lower the limit and re-assess.
- You set the limit too low and **waste resources**. Your Elasticsearch cluster may be capable of handling more concurrency than you're providing—at least for now. Time to raise the limit and re-assess.

Not only do you need to perform this balancing act on a per-sink basis and on each Vector instance—that may be a _lot_ of application points in your system—but the optimal rate is an elusive target that shifts along with changes in a number of factors:

- The number of Vector instances currently sending traffic
- The current capacity of downstream services
- The volume of data you're currently sending

These changes are especially pronounced in highly elastic environments, like [Kubernetes], that are essentially _designed_ to let you tweak cluster topologies, configuration, and much more with very little friction, which compounds the problem.

And don't forget, of course, that this chasing-the-dragon decision loop has its own cognitive and operational costs.

## The solution: Adaptive Request Concurrency

We feel strongly that Vector's **Adaptive Request Concurrency** (ARC) feature provides a qualitatively better path than rate limiting. With ARC [enabled](#the-role-of-configuration) on any given sink, Vector determines the optimal network concurrency based on current environment conditions and continuously re-adjusts in light of new information.

Here's how that plays out in some example scenarios:

| Change                                                        |     | Response                                                                                                                                       |
| :------------------------------------------------------------ | :-: | :--------------------------------------------------------------------------------------------------------------------------------------------- |
| **You deploy more Vector instances**                          |  ➔  | Vector automatically redistributes HTTP throughput across both current and new instances                                                       |
| **You scale up your Elasticsearch cluster**                   |  ➔  | Vector automatically increases concurrency to take full advantage of the new capacity                                                          |
| **You scale your Elasticsearch cluster back down**            |  ➔  | Vector lowers concurrency to avoid any risk of destabilizing the cluster (while still taking full of advantage of the now-decreased bandwidth) |
| **Your Elasticsearch cluster experiences a temporary outage** |  ➔  | Vector lowers concurrency dramatically and provides backpressure by [buffering][buffer] events                                                 |

With ARC, these scenarios require no human intervention. Vector quietly hums along making these decisions for you with a speed and granularity that rate limits simply cannot provide.

## How it works

ARC in Vector is based on a decision-making process that’s fairly simple at a high level. When Vector POSTs data to downstream services via HTTP, it continuously keeps track of downstream service performance and uses that information to make precise concurrency decisions.

The diagram below shows Vector's decision chart:

![The Adaptive Request Concurrency decision chart](/img/shared/adaptive-concurrency.png)

With ARC enabled, Vector watches for significant movements in two things: the round-trip time (RTT) of requests and HTTP response codes (failure vs. success).

- If the RTT is declining/constant and/or response codes are consistently successful (200-299), Vector sees 🟢 and increases the throughput linearly. This is the "additive increase" in AIMD.
- If the RTT is increasing and/or response codes consistently indicate failure—codes like `429 Too Many Requests` and `503 Service Unavailable`—Vector sees 🟡 and exponentially decreases concurrency. This is the "multiplicative decrease" in AIMD.

This decision tree is always active and Vector always "knows" what to do, even in extreme cases like total service failure.

### The role of configuration

Vector never stops quietly making the linear up vs. exponential down decision in the background, and it works out of the box with zero configuration beyond enabling the feature, which is currently on an opt-in basis in version 0.11. You can enable ARC in an HTTP sink by setting the [`request.concurrency`][request_concurrency] parameter to `adaptive`. Here's an example for a Clickhouse sink:

```toml
[sinks.clickhouse_internal]
type = "clickhouse"
inputs = ["log_stream_1", "log_stream_2"]
host = "http://clickhouse-prod:8123"
table = "prod-log-data"
request.concurrency = "adaptive"
```

There's also room for fine-tuning if you find yourself needing additional knobs:

- **Buffering**. What happens when Vector needs to lower concurrency and thus throttle the output? What happens to data that needs to be sent later? Vector lets you choose between an on-disk and an in-memory [buffer] and to set a [max_size] for that buffer. The `memory` buffer is the default, which maximizes performance, but you can always choose `disk` if your use case requires stronger durability guarantees. As always, this can be configured on a per-sink basis.
- **The adaptive concurrency algorithm itself**. In general, you shouldn't need to adjust the algorithm, but in case you need to resort to that, there are three parameters available:
  - `decrease_ratio` — This determines how rapidly Vector lowers the limit in response to failures or higher latency.
  - `ewma_alpha` — Vector uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. The `ewma_alpha` parameter determines how heavily new measurements are weighted compared to older ones.
  - `rtt_threshold_ratio` — The minimal change in RTT necessary for the algorithm to respond and adjust concurrency; changes below that threshold are ignored.

The defaults should work just fine for these parameters in most cases, but we know that some scenarios may call for a highly targeted approach.

## How we built it

The development process behind ARC was highly methodical and data-driven. To summarize:

- Customer feedback has pinpointed concurrency management as a pain point since very early in the life of Vector.
- The initial foray in addressing the problem came in [RFC 1858], which called for a qualitatively better option for users and [gestured toward][prior_art] some possible inspirations.
- Our engineers ultimately opted for a solution deeply inspired by analogous work on the Netflix engineering team, which is beautifully summarized in the [Performance Under Load] piece on their blog. Our respective approaches utilize an [additive-increase/multiplicative-decrease][aimd] (AIMD) algorithm inspired by TCP [congestion control] algorithms. We'll have a lot more to say about this here on the blog next week. If you want to see the in-depth discussion that drove this process, see GitHub [issue #3255][issue_325] on the Vector repo. There you'll see a pretty epic back and forth within our engineering team along with a slew of visualizations. It's quite the read.
- As Netflix is largely a Java shop and there was nothing immediately usable "off the shelf" in the Rust ecosystem, we needed to create our own Rust implementation, which you can see in the [adaptive_concurrency] module. Of special importance is the concurrency [Controller], which is responsible for the linear up/exponential down decision that I alluded to above.
- For testing, the team mostly relied on our in-house [http_test_server], a pretty straightforward but highly customizable HTTP server written in Go.

It took several months, some hefty PRs, and even a handful of [dead ends], but we think that both the process and the end result are wholly consistent with the fastidious approach we strive for in building Vector.

## More to come

Next week, we'll follow up on this announcement with a post from Timber's [Bruce Guenter], the lead engineer behind ARC, that provides a far more in-depth look at how this feature was implemented. Bruce has quite an intricate story to tell and some great visualizations, so we urge you to tune in.

Going forward, we'll continue listening to Vector users and incorporating their feedback on concurrency management into Vector's roadmap. We're fully open to refining the underlying algorithm and providing more configuration knobs in a future release if that serves our users. There's currently an [open issue], for example, that calls for exploration of an alternative gradient algorithm (also inspired by Netflix's work), and some lively internal discussions are already pointing the way to next steps.

For now, we're quite confident that ARC in Vector 0.11, even in its initial state, should immediately improve the experience of users that rely on downstream HTTP services.

[aimd]: https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease
[adaptive_concurrency]: https://github.com/timberio/vector/tree/master/src/sinks/util/adaptive_concurrency
[bruce guenter]: https://github.com/bruceg
[buffer]: /docs/reference/configuration/sinks/http/#buffer
[clickhouse]: /docs/reference/configuration/sinks/clickhouse
[congestion control]: https://en.wikipedia.org/wiki/TCP_congestion_control
[controller]: https://github.com/timberio/vector/blob/master/src/sinks/util/adaptive_concurrency/controller.rs#L23-L31
[dead ends]: https://github.com/timberio/vector/pull/3671
[elasticsearch]: /docs/reference/configuration/sinks/elasticsearch
[http_test_server]: https://github.com/timberio/http_test_server
[issue_325]: https://github.com/timberio/vector/issues/3255
[kubernetes]: https://kubernetes.io
[max_size]: /docs/reference/configuration/sinks/http/#buffer.max_size
[open issue]: https://github.com/timberio/vector/issues/3887
[performance under load]: https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581
[prior_art]: https://github.com/timberio/vector/blob/master/rfcs/2020-04-06-1858-automatically-adjust-request-limits.md#prior-art
[rate limit]: /docs/reference/configuration/sinks/http/#rate-limits-adapative-concurrency
[rate_limit_duration_secs]: /docs/reference/configuration/sinks/http/#request.rate_limit_duration_secs
[rate_limit_num]: /docs/reference/configuration/sinks/http/#request.rate_limit_num
[request_concurrency]: /docs/reference/configuration/sinks/http/#request.concurrency
[rfc 1858]: https://github.com/timberio/vector/blob/master/rfcs/2020-04-06-1858-automatically-adjust-request-limits.md
[rust]: https://rust-lang.org
[sinks]: /docs/reference/configuration/sinks
[sources]: /docs/reference/configuration/sources
[splunk]: https://splunk.com
[transforms]: /docs/reference/configuration/transforms
[vector]: /


Adaptive request concurrency. Resilient observability at scale.

Today we're very excited to open source the Vector project! Vector is a tool for building flexible and robust pipelines for your logs and metrics data. We're still in the early stages, but our goal with Vector is to dramatically simplify your observability infrastructure while making it easy to get more value from your data.



## Why build Vector?

We decided to build Vector because we believe that **existing tools take too narrow a view of the problem of building observability infrastructure**. Whether they're limited to working with a single storage system, not performant enough to handle your volume of data, or simply not flexible enough to let you do all the things you want with your data, the tools available today inevitably end up as a small part of a larger pipeline that your team must build and maintain.

Assembling and maintaining these pipelines piecemeal can be a tough and unrewarding job. There's a dizzying array of tools that all seem to cover unique subsets of the overall capabilities you're looking to implement and there's no guarantee that any given two of them will work well together. Faced with this, teams tend towards one (or a combination!) of three rough directions: (1) vendor-first, (2) trusted open source stacks, and (3) build your own.

**Vendor-first** is tempting, but usually expensive and invasive. The vendor provides their custom collectors and client libraries for you to integrate, and you simply ship them all your data. In the end, you're limited to their capabilities, locked in to their ecosystem, and paying more and more every month as your data volume grows.

**Trusted open source stacks** give you a bit more flexibility, but with a significantly higher maintenance burden. You have to pick the right implementation of each sub-component (eight years ago there were already at least [20 implementations of statsd][statsd]), ensure they'll work together well, and then maintain and scale them for the lifetime of the infrastructure. Adding new capabilities or changing storage backends generally means adding a whole new set of components to run, often overlapping with some you're already running.

Finally, there are teams with the resources and organizational willpower to **build their own solutions**. This gives you an enormous amount of flexibility and power, but with an equally impressive price tag in engineering time.

## What is Vector?

So what exactly is Vector and how is it a better solution to this problem? At first glance, Vector looks a lot like some other logging infrastructure tools you might be familiar with. It can ingest data in a number of ways (tailing files, syslog messages over the network, etc), it can process that data in flight (regex and JSON parsing, filtering, sampling, etc), and it can send the raw or processed data to a variety of external systems for storage and querying. Whether it's rsyslog, logstash, fluentd, or another example, it's more likely than not that something with this basic set of features is running in your infrastructure.

On top of this basic functionality, Vector adds a few important enhancements:

1. A **richer data model**, supporting not only logs but aggregated metrics, fully structured events, etc
2. **Programmable transforms** written in lua (or eventually wasm) that let you parse, filter, aggregate, and otherwise manipulate your data in arbitrary ways
3. **Uncompromising performance and efficiency** that enables a huge variety of deployment strategies

The end result is a tool that can **be** your pipeline, rather than just another component in it. We hope to enable the full flexibility and power of a custom built solution with a tiny fraction of the required investment. There have been a few other projects aimed in this direction (heka and cernan are two big inspirations), but with Vector we think we can take the ease of use and breadth of capabilities to another level.

## Try it out!

We're still very early in the process of building Vector, but it's already at a stage where we believe it's competitive with existing tools and ready to be trialled for production use. Not every feature mentioned above is present yet, but we're happy with the foundation and working quickly to flesh everything out. If you're interested in giving it a try, check out the project on GitHub, our docs, and please reach out! We'd love to hear about your use cases and are excited to see where we can take this project together.

[statsd]: https://joemiller.me/2011/09/list-of-statsd-server-implementations

Introducing Vector

When we set out to build Vector, we knew that reliability and performance were
two of our top priorities. We also knew that even the best of intentions would
not be enough to make certain those qualities were realized and reflected in our
users' production deployments. Since then, we've been continuously evolving and
expanding our approach to achieving that level of quality.



There are a few factors that make ensuring robustness a particularly difficult
task for software like Vector:

1. It's relatively new and not as "battle tested" as more widely deployed
   software.
1. The vast majority of its functionality lives at the edges, interfacing with
   various external systems.
1. Instead of a monolithic application designed for a single task, it is
   a collection of components that can be assembled into a near-infinite number
   of configurations.

This challenge has given us a unique opportunity to apply wide variety of
testing techniques:

- Example-based testing
  - Unit tests
  - Integration tests
- Generative testing
  - Property-based testing
  - Model-based testing
  - Fuzz testing
- Black-box testing
  - Performance tests
  - Correctness tests
  - Reliability tests

While there's no one perfect solution to overcome these difficulties, we've
found that the combination of these techniques at different layers of the stack
has given us a good level of confidence in Vector's behavior.

In this post, we'll discuss briefly how we're using each of these types of
tests, their strengths and weaknesses, as well as any tips we have for using
them effectively.

## Example-based testing

We'll start off with the types of tests you're likely most familiar with, the
humble unit and integration tests. These are the bread and butter of almost
every test suite, and for good reason.

We group them into "example-based" because they both follow the same general
pattern. As the developer, you come up with an example input, you have your code
process that example, and then you assert that the outcome is what you expected.

A great deal has been written on these types of tests already, so we'll try to
keep this section brief and focused on situations where these techniques start
to break down.

### Unit tests

A unit test is generally defined by the idea of isolation. This can mean
different things to different people, but a common definition is that the test
is both isolated from the outside world (i.e. no network calls, reading files,
etc) and exercises a single component of the system (e.g. one function, class,
etc).

In Vector, we've found unit tests to be a particularly great fit for our
transforms. This makes sense, since transforms are effectively isolated
functions that take events in and return events out. For the same reason, we
tend to use unit tests extensively around the encoder portions of our sinks.

For other types of components, so much of the functionality is focused on
communication with external systems that it can be difficult to isolate logic
enough for a unit test. As much as possible, we try to critically assess that
difficulty and use it to help find ways we can refactor the design to be more
modular and therefore amenable to unit testing. Unit tests are an excellent
source of design feedback, so we want to feel that pain and refactor when
they're difficult to write.

That being said, there are two specific places we often run into the limitations
of unit tests. The first and more obvious is around pieces of code that are
fundamentally not isolated. The second situation has to do with the size of the
input space and number of potential paths through the component under test. As
an example-based testing strategy, the effectiveness of unit tests comes down to
the developer's ability to provide a thorough set of example inputs. This
becomes exponentially more difficult with each logical branch, and requires
recognizing paths you weren't considering when you initially wrote the code.

Takeaways:

- Isolation makes unit tests simple, fast, and reliable.
- If something is difficult to unit test, refactor until it's easy.
- As a human, be wary of your ability to think up exhaustive example inputs.

### Integration tests

The category of integration tests is a bit of a catch-all. Roughly defined,
they're example-based tests that are explicitly not isolated and focus on the
interaction between two or more components.

Given Vector's focus on integrating with a wide variety of external systems, we
have a higher ratio of integration tests than your average system. Even once
we've done all we can to isolate logic into small, unit-testable functions, we
still need to ensure the component as a whole does what it's supposed to. If
unit tests tell you that something works in theory, integration tests tell you
that it _should_ work in practice.

Similar to unit tests, there are two major downsides to integration tests. The
first is the same problem unit tests have with exhaustiveness, but intensely
magnified. Because they often cover the interaction of multiple whole systems,
integration tests will virtually never cover the combinatorial explosion of
potential execution paths. Second, integration tests are often simply a pain to
write and maintain. Given their dependence on external systems, they can be
tedious to write, slow to run, and often flaky if their environment isn't set up
just so.

Takeaways:

- While they can be a pain, integration tests are a valuable sanity check that
  your system will actually work for users.
- Don't overuse them trying to cover every possible scenario, because you can't.

## Generative testing

The idea of generative testing is a direct response to the shared shortcoming of
example-based strategies like unit and integration testing. Humans are bad at
visualizing very large state spaces, and the state space of your code's possible
inputs and execution paths is enormous. You also bring all the same biases when
writing tests as you did when writing the original code.

Generative tests address these issues by taking the human out of the equation
and having the computer generate millions of example inputs. The trade-off is
that since you aren't simply hard-coding a list of inputs and their expected
outputs anymore, you're forced to come up with more creative ways of identifying
failures.

### Property-based tests

A simple way to think about property-based tests is that they are unit tests
where the computer comes up with random example inputs. Since the test author
can no longer provide the expected output along with those example inputs, they
instead declare certain _properties_ that must hold true across any combination
of input and output. A classic example is testing a function that reverses
a list against the property that any list reversed twice must be equal to
itself.

While this is something you could do with no tooling support at all, there are
some pretty advanced libraries that make things easier (e.g. the venerable
[quickcheck][1] and more recent [hypothesis][2]). Beyond simple random input
generation, these tools often include valuable features like customizable
generators and shrinking, which attempts to simplify failing inputs as much as
possible before presenting them to you.

In Vector, we use property-based tests to exercise our internal data
serialization logic. We want to make sure that arbitrary input events can go
through the full serialization and deserialization process without losing any
information along the way. We use the Rust [proptest][3] library to generate
input events and then test that the output event is equal to the input after
a full round trip through the system. Since this is an isolated, deterministic
function, we can quickly and easily run millions of iterations of this test.

While property-based tests are certainly better at covering the input space than
unit tests, simple random generation of possible inputs can become a limitation.
Without some sense of what makes an input "interesting", property-based tests
have no way of intelligently exploring the space. In some cases, this can
meaning burning a lot of CPU without necessarily finding any new failures.

Takeaways:

- Property-based tests help uncover more edge cases in your system's logic.
- Like unit tests, they're most effective when applied to isolated components.
- They can't directly test "correctness", only that your given set of invariants
  is upheld.

### Model-based testing

One particularly interesting application of property-based test tooling is
something we've seen called [model-based testing][4]. The basic idea is that you
implement a simplified model of your system (e.g. a hashmap is a simple model of
a key-value store), and then assert that for all possible inputs, your system
should produce the same output as the model.

Vector inherited some of these tests from [cernan][5], where its file tailing
module originated (thanks [Brian][6]!). It works by generating random sequences
of file writes, reads, rotations, truncations, etc, and applying them to both
a simple in-memory model of a file system as well as the actual file system being
tailed by our file watcher implementation. It then verifies that the lines
returned by our watcher are the same as those returned from the simplified
simulation.

In this strategy, the model is acting as an oracle and the quality of the test
depends on that oracle actually behaving correctly. That makes it a good match
for components with a relatively simple API but deeper inner complexity due to
performance optimizations, persistence, etc. Like normal property-based tests,
they may have trouble efficiently exploring the state space of especially
complex components.

Takeaways:

- Model-based tests are a good match for components with deep implementations
  but relatively shallow APIs.
- They rely on a model implementation simple enough to be "obviously correct",
  which is not possible for all systems.

### Fuzz testing

At its most simplistic, fuzz testing is just feeding your program random data
and seeing if it breaks. In that sense, you can think of it as a kind of
external property-based testing, where the property is that your system should
not crash. This might not sound terribly interesting on its own, but modern
tools (e.g. [american fuzzy lop][7]) have developed a superpower that gives them
a massive advantage over traditional property-based testing: using code-coverage
information to guide input generation.

With this critical feedback loop in place, tools can see when a particular input
led to a new execution path. They can then intelligently evolve these
interesting inputs to prioritize finding even more new paths, zeroing in far
more efficiently on potential edge cases.

This is a particularly powerful technique for testing parsers. Where normal
property-based tests might repeatedly attempt to parse random strings and never
happen upon anything remotely valid, a fuzz testing tool can gradually "learn"
the format being parsed and spend far more time exploring productive areas of
the input space.

Many of the parsers we use in Vector are pre-built for various data formats and
have seen some fuzz testing in their upstream library. We did, however, write
our `tokenizer` parser from scratch and it's unique in that it's not for
a specific format. Instead, it gives a best-effort attempt at breaking the input
up into logical fields. We've found it to be a great fit for fuzz testing
because the way that it handles strange and misshapen inputs is less important
than that fact that it will not panic and crash the program.

One of the limitations of AFL-style fuzzing is the focus on random byte strings
as inputs. This matches up really well with parsers, but maybe not that many
other components in your system. The idea of [structure-aware][12] fuzzers looks
to address this. One such tool is [fuzzcheck][8], which we've been starting to
explore. Instead of byte strings, it works directly with the actual types of
your program. It also runs in-process with your system, making it simpler to
detect not just panics but also things like simple test failures. In many ways,
it has the potential to combine the best of both fuzz testing and property-based
testing.

Takeaways:

- Feedback loops allow fuzz testing to efficiently explore extremely large input
  spaces, like those of a parser.
- Tools are advancing rapidly, making fuzz tests more convenient for more types
  of situations.

## Black-box testing

Even if all of the above testing strategies worked flawlessly and got us to 100%
branch coverage, we still wouldn't know for certain that Vector was performing
at the level we expect. To answer that question, we need to run it as users run
it and observe things like throughput, memory usage, CPU usage, etc.

This is where the [`vector-test-harness`][9] comes in. These are high-level,
black-box tests where we run various Vector configurations on deployed hardware,
generating load and capturing metrics about its performance. And since they're
black-box tests (i.e. they require no access to or knowledge of Vector
internals), we can also provide configurations for similar tools to see how they
compare.

### Performance tests

The performance tests in our harness focus on generating as much load as the
given configuration can handle and measuring throughput, memory use, etc. These
tests capture our real-world performance in way that micro-benchmarks can't, and
they give us a very useful point of comparison with other tools that may have
made different design decisions. If one of the metrics looks way off, that gives
us a starting point to investigate why we're not performing as well as we think
we should.

Since these tests are almost completely automated, we'll soon be looking to
start running them on a nightly basis and graphing the results over time. This
should give us an early warning signal in the case of a serious performance
regression, and help us visualize our progress in making Vector faster and more
efficient over time.

Takeaways:

- Behavior under load is an important part of the user experience and deserves
  a significant testing investment.
- Regular, automated testing can generate valuable data for catching performance
  issues before they reach users.

### Correctness tests

Alongside those performance tests, we also have a set of tests we call
correctness tests. The setup is quite similar to the performance tests, but the
focus is different. Instead of generating as much load as we can and watching
things like throughput and system resource use, we instead run each
configuration through different interesting scenarios to see how they behave.

For example, we have correctness tests around various flavors of file rotation,
disk persistence across restarts, nested JSON messages, etc. While these are
behaviors that we also test at various lower levels (e.g. unit and integration
tests), covering a handful of important cases at this level of abstraction gives
us some extra confidence that we are seeing exactly what our users will see.

The ability to compare behaviors across competing tools is another bonus. Going
through the process of setting up those tests gets us valuable experience
working with those other tools. We can see what works well in their
configuration, documentation, etc, and identify areas where we can improve
Vector as a result.

Takeaways:

- Taking time to "zoom out" and test your system as a user would can help
  uncover blind spots and sanity-check behavior.
- Evaluating similar tools can help build a better understanding of user
  expectations.

### Reliability tests

A third category that we're currently working to integrate into
`vector-test-harness` is something we're calling reliability tests. These are
similar to performance and correctness tests, except that they're designed to
run continuously and flush out errors that may occur only in rare environmental
circumstances.

In a way, they're like simple, integration-level fuzz tests where changes in the
environment over time provide input randomness. For example, running a week-long
reliability test of our S3 sink exposed a bug where a specific kind of network
failure could lead to duplicate data when the retried request crossed
a timestamp boundary. That is not the type of failure we expect to be able to
induce in a local integration test, and the relevant factors (time and network
conditions) were not those exercised by standard fuzzing or property-based
testing.

The main challenge with these kinds of tests, aside from getting the requisite
environment and harnesses up and running, is capturing sufficient context about
the environment at the time of the failure that you stand a chance at
understanding and reproducing it. This task itself is a great test for our
internal observability, and any issue we can't reproduce is a sign that our
logging and metrics data needs to be improved.

Another issue with these tests is that the vast majority of the time, nothing
particularly interesting is happening. Since we want to find bugs as quickly as
possible, we can supplement the randomness of the environment by injecting
various types of faults on our own. There are a variety of tools for this, such
as [Toxiproxy][10] and [Namazu][11].

Takeaways:

- The environment is an important source of uncertainty in your system that is
  difficult to simulate accurately.
- Observing bugs from a user's perspective incentivizes good internal
  observability tooling

## Conclusion

Even with all of the above in place, we're continuously exploring ways to
further increase our confidence in the reliability and performance of Vector.
That could mean anything from expanding our current test suites to be more
thorough to adopting entirely new techniques to help cover more possible
executions (e.g. [simulation][15] or [metamorphic][16] testing).

With some users running a Vector process on nearly every host in their
infrastructure, ensuring an extremely high level of robustness and efficiency is
paramount. At the same time, those needs must be balanced with increasing
Vector's functional capabilities. Finding the right balance is an ongoing
challenge as the project grows and matures.

[1]: http://www.cse.chalmers.se/~rjmh/QuickCheck/manual.html
[2]: https://hypothesis.works/articles/what-is-property-based-testing/
[3]: https://github.com/AltSysrq/proptest
[4]: https://medium.com/@tylerneely/reliable-systems-series-model-based-property-testing-e89a433b360
[5]: https://github.com/postmates/cernan
[6]: https://github.com/blt
[7]: http://lcamtuf.coredump.cx/afl/
[8]: https://github.com/loiclec/fuzzcheck-rs
[9]: https://github.com/timberio/vector-test-harness/
[10]: https://github.com/Shopify/toxiproxy
[11]: https://github.com/osrg/namazu
[12]: https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md
[13]: https://github.com/rust-fuzz/cargo-fuzz
[14]: https://github.com/timberio/vector/blob/9fe1eeb4786b27843673c05ff012f6b5cf5c3e45/src/transforms/tokenizer.rs#L240-L249
[15]: https://www.youtube.com/watch?v=4fFDFbi3toc
[16]: https://www.hillelwayne.com/post/metamorphic-testing/

Blog