Logstash

Example Configuration

Logstash message from generator input

Config
Input
Output
1[sources.my_source_id]

2type = "logstash"

1"Logstash input config:\n\n```text\ninput {\n\tgenerator {\n\t\tcount => 1\n\t}\n}\n\nOutput if sent to stdout logstash output:\n\n```text\n{ \"@version\" => \"1\", \"@timestamp\" => 2021-06-14T20:57:14.230Z, \"host\" => \"c082bb583445\", \"sequence\" => 0, \"message\" => \"Hello world!\" }\n```"

1{

2  "log": {

3    "host": "34.33.222.212",

4    "line": "2021-06-14T20:57:14.230Z c082bb583445 hello world"

5  }

6}

Message from Elastic Beat Heartbeat agent

Config
Input
Output
1[sources.my_source_id]

2type = "logstash"

1"Heartbeat input config:\n\n```yaml\nheartbeat.config.monitors:\n\tpath: ${path.config}/monitors.d/*.yml\n\treload.enabled: false\n\treload.period: 5s\n\nheartbeat.monitors:\n- type: http\n\tschedule: '@every 5s'\n\turls:\n\t- http://google.com\n```\n\nOutput if sent to stdout output:\n\n```json\n{\"@timestamp\":\"2021-06-14T21:25:37.058Z\",\"@metadata\":{\"beat\":\"heartbeat\",\"type\":\"_doc\",\"version\":\"7.12.1\"},\"url\":{\"full\":\"http://google.com\",\"scheme\":\"http\",\"domain\":\"google.com\",\"port\":80},\"tcp\":{\"rtt\":{\"connect\":{\"us\":18504}}},\"event\":{\"dataset\":\"uptime\"},\"ecs\":{\"version\":\"1.8.0\"},\"resolve\":{\"rtt\":{\"us\":7200},\"ip\":\"172.217.4.174\"},\"summary\":{\"up\":1,\"down\":0},\"http\":{\"response\":{\"mime_type\":\"text/html; charset=utf-8\",\"headers\":{\"Content-Length\":\"219\",\"Date\":\"Mon, 14 Jun 2021 21:25:37 GMT\",\"Server\":\"gws\",\"X-Xss-Protection\":\"0\",\"Location\":\"http://www.google.com/\",\"Expires\":\"Wed, 14 Jul 2021 21:25:37 GMT\",\"Content-Type\":\"text/html; charset=UTF-8\",\"Cache-Control\":\"public, max-age=2592000\",\"X-Frame-Options\":\"SAMEORIGIN\"},\"status_code\":301,\"body\":{\"hash\":\"2178eedd5723a6ac22e94ec59bdcd99229c87f3623753f5e199678242f0e90de\",\"bytes\":219}},\"rtt\":{\"response_header\":{\"us\":51481},\"validate\":{\"us\":52664},\"content\":{\"us\":1182},\"total\":{\"us\":71585},\"write_request\":{\"us\":134}}},\"monitor\":{\"type\":\"http\",\"status\":\"up\",\"duration\":{\"us\":79517},\"check_group\":\"0c8c908a-cd57-11eb-85a4-025000000001\",\"ip\":\"172.217.4.174\",\"timespan\":{\"gte\":\"2021-06-14T21:25:37.137Z\",\"lt\":\"2021-06-14T21:25:42.137Z\"},\"id\":\"auto-http-0X993E1F882355CFD2\",\"name\":\"\"},\"agent\":{\"hostname\":\"docker-desktop\",\"ephemeral_id\":\"9e15e5bc-86d6-4d47-9067-4262b00c5cce\",\"id\":\"404c8975-a41b-45bd-8d93-3f6c4449e973\",\"name\":\"docker-desktop\",\"type\":\"heartbeat\",\"version\":\"7.12.1\"}}\n```"

1{

2  "log": {

3    "host": "34.33.222.212",

4    "timestamp": "2021-06-14T21:25:37.058Z",

5    "@timestamp": "2021-06-14T21:25:37.058Z",

6    "@metadata": {

7      "beat": "heartbeat",

8      "type": "_doc",

9      "version": "7.12.1"

10    },

11    "url": {

12      "full": "http://google.com",

13      "scheme": "http",

14      "domain": "google.com",

15      "port": 80

16    },

17    "tcp": {

18      "rtt": {

19        "connect": {

20          "us": 18504

21        }

22      }

23    },

24    "event": {

25      "dataset": "uptime"

26    },

27    "ecs": {

28      "version": "1.8.0"

29    },

30    "resolve": {

31      "rtt": {

32        "us": 7200

33      },

34      "ip": "172.217.4.174"

35    },

36    "summary": {

37      "up": 1,

38      "down": 0

39    },

40    "http": {

41      "response": {

42        "mime_type": "text/html; charset=utf-8",

43        "headers": {

44          "Content-Length": "219",

45          "Date": "Mon, 14 Jun 2021 21:25:37 GMT",

46          "Server": "gws",

47          "X-Xss-Protection": "0",

48          "Location": "http://www.google.com/",

49          "Expires": "Wed, 14 Jul 2021 21:25:37 GMT",

50          "Content-Type": "text/html; charset=UTF-8",

51          "Cache-Control": "public, max-age=2592000",

52          "X-Frame-Options": "SAMEORIGIN"

53        },

54        "status_code": 301,

55        "body": {

56          "hash": "2178eedd5723a6ac22e94ec59bdcd99229c87f3623753f5e199678242f0e90de",

57          "bytes": 219

58        }

59      },

60      "rtt": {

61        "response_header": {

62          "us": 51481

63        },

64        "validate": {

65          "us": 52664

66        },

67        "content": {

68          "us": 1182

69        },

70        "total": {

71          "us": 71585

72        },

73        "write_request": {

74          "us": 134

75        }

76      }

77    },

78    "monitor": {

79      "type": "http",

80      "status": "up",

81      "duration": {

82        "us": 79517

83      },

84      "check_group": "0c8c908a-cd57-11eb-85a4-025000000001",

85      "ip": "172.217.4.174",

86      "timespan": {

87        "gte": "2021-06-14T21:25:37.137Z",

88        "lt": "2021-06-14T21:25:42.137Z"

89      },

90      "id": "auto-http-0X993E1F882355CFD2",

91      "name": ""

92    },

93    "agent": {

94      "hostname": "docker-desktop",

95      "ephemeral_id": "9e15e5bc-86d6-4d47-9067-4262b00c5cce",

96      "id": "404c8975-a41b-45bd-8d93-3f6c4449e973",

97      "name": "docker-desktop",

98      "type": "heartbeat",

99      "version": "7.12.1"

100    }

101  }

102}

Configuration Options

Required Options

address(required)

The address to listen for TCP connections on.

TypeSyntaxDefaultExample
stringliteral["0.0.0.0:5044"]
type(required)

The component type. This is a required field for all components and tells Vector which component to use.

TypeSyntaxDefaultExample
stringliteral["logstash"]

Advanced Options

receive_buffer_bytes(optional)

Configures the receive buffer size using the SO_RCVBUF option on the socket.

TypeSyntaxDefaultExample
uint[65536]
keepalive(optional)

Configures the TCP keepalive behavior for the connection to the source.

TypeSyntaxDefaultExample
hash[]
tls(optional)

Configures the TLS options for incoming connections.

TypeSyntaxDefaultExample
hashliteral[]

How it Works

Sending data from logstash agents to Vector aggregators

If you are already running an Elastic agent (Logstash or Elastic Beats) in your infrastructure, this source can make it easy to start getting that data into Vector.

Logstash configuration

To configure Logstash to forward to a Vector instance, you can use the following output configuration:

output {
		lumberjack {
				# update these to point to your vector instance
				hosts => ["127.0.0.1"]
				port => 5044
				ssl_certificate => "/path/to/certificate.crt"
		}
}

Note that Logstash requires SSL to be configured.

Elastic Beats configuration

To configure one of the Elastic Beats agents to forward to a Vector instance, you can use the following output configuration:

	output.logstash:
	  # update these to point to your vector instance
	  hosts: ["127.0.0.1:5044"]

State

This component is stateless, meaning its behavior is consistent across each input.

Transport Layer Security (TLS)

Vector uses OpenSSL for TLS protocols. You can adjust TLS behavior via the tls.* options.

Acknowledgement support

Currently, this source will acknowledge events to the sender once the event has been sent to the next component in the topology. In the future, this source will utilize Vector's support for end-to-end acknowledgements.

Context

By default, the logstash source augments events with helpful context keys.