Tag Cardinality Limit

Example Configuration

Drop high-cardinality tag

Config

Input

Output

1[transforms.my_transform_id]
2type = "tag_cardinality_limit"
3
4  [transforms.my_transform_id.fields]
5  value_limit = 1
6  limit_exceeded_action = "drop_tag"

1[
2  {
3    "metric": {
4      "kind": "incremental",
5      "name": "logins",
6      "counter": {
7        "value": 2
8      },
9      "tags": {
10        "user_id": "user_id_1"
11      }
12    }
13  },
14  {
15    "metric": {
16      "kind": "incremental",
17      "name": "logins",
18      "counter": {
19        "value": 2
20      },
21      "tags": {
22        "user_id": "user_id_2"
23      }
24    }
25  }
26]

1[
2  {
3    "metric": {
4      "kind": "incremental",
5      "name": "logins",
6      "counter": {
7        "value": 2
8      },
9      "tags": {
10        "user_id": "user_id_1"
11      }
12    }
13  },
14  {
15    "metric": {
16      "kind": "incremental",
17      "name": "logins",
18      "counter": {
19        "value": 2
20      },
21      "tags": {}
22    }
23  }
24]

Configuration Options

Required Options

mode(required)

Controls what approach is used internally to keep track of previously seen tags and deterime when a tag on an incoming metric exceeds the limit.

Type	Syntax	Default	Example
string	literal		["exact","probabilistic"]

inputs(required)

A list of upstream source or transform IDs. Wildcards (*) are supported.

See configuration for more info.

Type	Syntax	Default	Example
array	literal		["my-source-or-transform-id","prefix-*"]

type(required)

The component type. This is a required field for all components and tells Vector which component to use.

Type	Syntax	Default	Example
string	literal		["tag_cardinality_limit"]

Advanced Options

cache_size_per_tag(optional)

The size of the cache in bytes to use to detect duplicate tags. The bigger the cache the less likely it is to have a 'false positive' or a case where we allow a new value for tag even after we have reached the configured limits.

Type	Syntax	Default	Example
uint		5120000

limit_exceeded_action(optional)

Controls what should happen when a metric comes in with a tag that would exceed the configured limit on cardinality.

Type	Syntax	Default	Example
string	literal	drop_tag

value_limit(optional)

How many distinct values to accept for any given key.

Type	Syntax	Default	Example
uint		500

How it Works

Intended Usage

This transform is intended to be used as a protection mechanism to prevent upstream mistakes. Such as a developer accidentally adding a request_id tag. When this is happens, it is recommended to fix the upstream error as soon as possible. This is because Vector's cardinality cache is held in memory and it will be erased when Vector is restarted. This will cause new tag values to pass through until the cardinality limit is reached again. For normal usage this should not be a common problem since Vector processes are normally long-lived.

Failed Parsing

This transform stores in memory a copy of the key for every tag on every metric event seen by this transform. In mode exact, a copy of every distinct value for each key is also kept in memory, until value_limit distinct values have been seen for a given key, at which point new values for that key will be rejected. So to estimate the memory usage of this transform in mode exact you can use the following formula:

(number of distinct field names in the tags for your metrics * average length of
the field names for the tags) + (number of distinct field names in the tags of
your metrics * `value_limit` * average length of the values of tags for your
metrics)

In mode probabilistic, rather than storing all values seen for each key, each distinct key has a bloom filter which can probabilistically determine whether a given value has been seen for that key. The formula for estimating memory usage in mode probabilistic is:

(number of distinct field names in the tags for your metrics * average length of
the field names for the tags) + (number of distinct field names in the tags of
-your metrics * `cache_size_per_tag`)

The cache_size_per_tag option controls the size of the bloom filter used for storing the set of acceptable values for any single key. The larger the bloom filter the lower the false positive rate, which in our case means the less likely we are to allow a new tag value that would otherwise violate a configured limit. If you want to know the exact false positive rate for a given cache_size_per_tag and value_limit, there are many free on-line bloom filter calculators that can answer this. The formula is generally presented in terms of 'n', 'p', 'k', and 'm' where 'n' is the number of items in the filter (value_limit in our case), 'p' is the probability of false positives (what we want to solve for), 'k' is the number of hash functions used internally, and 'm' is the number of bits in the bloom filter. You should be able to provide values for just 'n' and 'm' and get back the value for 'p' with an optimal 'k' selected for you. Remember when converting from value_limit to the 'm' value to plug into the calculator that value_limit is in bytes, and 'm' is often presented in bits (1/8 of a byte).

State

This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.

Restarts

This transform's cache is held in memory, and therefore, restarting Vector will reset the cache. This means that new values will be passed through until the cardinality limit is reached again. See intended usage for more info.