Aggregate
Example Configuration
Aggregate over 5 seconds
1[transforms.my_transform_id]
2type = "aggregate"
3interval_ms = 5_000
1[
2 {
3 "metric": {
4 "kind": "incremental",
5 "name": "counter.1",
6 "timestamp": "2021-07-12T07:58:44.223543Z",
7 "tags": {
8 "host": "my.host.com"
9 },
10 "counter": {
11 "value": 1.1
12 }
13 }
14 },
15 {
16 "metric": {
17 "kind": "incremental",
18 "name": "counter.1",
19 "timestamp": "2021-07-12T07:58:45.223543Z",
20 "tags": {
21 "host": "my.host.com"
22 },
23 "counter": {
24 "value": 2.2
25 }
26 }
27 },
28 {
29 "metric": {
30 "kind": "incremental",
31 "name": "counter.1",
32 "timestamp": "2021-07-12T07:58:45.223543Z",
33 "tags": {
34 "host": "different.host.com"
35 },
36 "counter": {
37 "value": 1.1
38 }
39 }
40 },
41 {
42 "metric": {
43 "kind": "absolute",
44 "name": "gauge.1",
45 "timestamp": "2021-07-12T07:58:47.223543Z",
46 "tags": {
47 "host": "my.host.com"
48 },
49 "counter": {
50 "value": 22.33
51 }
52 }
53 },
54 {
55 "metric": {
56 "kind": "absolute",
57 "name": "gauge.1",
58 "timestamp": "2021-07-12T07:58:45.223543Z",
59 "tags": {
60 "host": "my.host.com"
61 },
62 "counter": {
63 "value": 44.55
64 }
65 }
66 }
67]
1[
2 {
3 "metric": {
4 "kind": "incremental",
5 "name": "counter.1",
6 "timestamp": "2021-07-12T07:58:45.223543Z",
7 "tags": {
8 "host": "my.host.com"
9 },
10 "counter": {
11 "value": 3.3
12 }
13 }
14 },
15 {
16 "metric": {
17 "kind": "incremental",
18 "name": "counter.1",
19 "timestamp": "2021-07-12T07:58:45.223543Z",
20 "tags": {
21 "host": "different.host.com"
22 },
23 "counter": {
24 "value": 1.1
25 }
26 }
27 },
28 {
29 "metric": {
30 "kind": "absolute",
31 "name": "gauge.1",
32 "timestamp": "2021-07-12T07:58:45.223543Z",
33 "tags": {
34 "host": "my.host.com"
35 },
36 "counter": {
37 "value": 44.55
38 }
39 }
40 }
41]
Configuration Options
Required Options
inputs(required)
A list of upstream source or transform
IDs. Wildcards (*) are supported.
See configuration for more info.
| Type | Syntax | Default | Example |
|---|---|---|---|
| array | literal | ["my-source-or-transform-id","prefix-*"] |
type(required)
The component type. This is a required field for all components and tells Vector which component to use.
| Type | Syntax | Default | Example |
|---|---|---|---|
| string | literal | ["aggregate"] |
Advanced Options
interval_ms(optional)
The interval over which metrics are aggregated in milliseconds. Over this period metrics with the same series data (name, namespace, tags, ...) will be aggregated.
| Type | Syntax | Default | Example |
|---|---|---|---|
| uint | 10000 |
How it Works
Aggregation Behavior
Metrics are aggregated based on their kind. During an interval, incremental metrics
are "added" and newer absolute metrics replace older ones in the same series. This results in a reduction
of volume and less granularity, while maintaining numerical correctness. As an example, two
incremental counter metrics with values 10 and 13 processed by the transform during a period would be
aggregated into a single incremental counter with a value of 23. Two absolute gauge metrics with
values 93 and 95 would result in a single absolute gauge with the value of 95. More complex
types like distribution, histogram, set, and summary behave similarly with incremental
values being combined in a manner that makes sense based on their type.
State
This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.
Advantages of Use
The major advantage to aggregation is the reduction of volume. It may reduce costs directly in situations that charge by metric event volume, or indirectly by requiring less CPU to process and/or less network bandwidth to transmit and receive. In systems that are constrained by the processing required to ingest metric events it may help to reduce the processing overhead. This may apply to transforms and sinks downstream of the aggregate transform as well.