Syslog Ingestion Troubleshooting.

noel_david — Fri, 20 Dec 2024 10:26:10 GMT

This article shows the common issues and fixes for issues with ingestion of Syslog into Dynatrace via ActiveGate.
Make sure you have followed the steps on the page below.

https://docs.dynatrace.com/docs/shortlink/lma-ingestion-syslog#enable-syslog-ingestion

The logs will be visible in the below location

/var/lib/dynatrace/remotepluginmodule/log/extensions/datasources/otelSyslog/dynatraceotelcollector.log

Case 1

If you see the warning below in the log:

[otelSyslog][otelSyslog][147494][err]Error: cannot start pipelines: start stanza: failed to listen on interface: failed to configure tcp listener: listen tcp 0.0.0.0:601: bind: permission denied [otelSyslog][otelSyslog][147494][err]2024/10/10 16:59:09 collector server run finished with error: cannot start pipelines: start stanza: failed to listen on interface: failed to configure tcp listener: listen tcp 0.0.0.0:601: bind: permission denied

If the filesystem mountpoint is using the ‘nosuid’ option. We need to remove the nosuid option from /opt . If SELinux is there, please disable it.

Case 2

If you see the warning below in the log:

[otelSyslog][otelSyslog][126924][err]Error: cannot start pipelines: start stanza: failed to open connection: listen udp 0.0.0.0:514: bind: address already in use [otelSyslog][otelSyslog][126924][err]2024/11/20 12:36:46 collector server run finished with error: cannot start pipelines: start stanza: failed to open connection: listen udp 0.0.0.0:514: bind: address already in use [otelSyslog][otelSyslog][129642][err]2024-11-20T13:36:49.760+0100 info service@v0.99.0/service.go:99 Setting up own telemetry… [otelSyslog][otelSyslog][129642][err]2024-11-20T13:36:49.760+0100 info service@v0.99.0/service.go:140 Skipped telemetry setup. {"address": ":8888", "level": "None"}

This means port 514 is already used by something else on the system, so it cannot be bound twice. Either change the port configuration or kill the process already occupying this port.

Case3

If you see the warning below in the log:

[otelSyslog][otelSyslog][68537][err]2024-11-22T17:01:39.689+0100 error helper/transformer.go:101 Failed to process entry {"kind": "receiver", "name": "syslog/udp", "data_type": "logs", "operator_id": "syslog_input_internal_parser", "operator_type": "syslog_parser", "error": "expecting a version value in the range 1-999 [col 4]", "action": "send"}

Adjust the configuration as described here (Syslog ingestion)
Set the used protocol to rfc3164

Example:

receivers: syslog/udp: udp: listen_address: '0.0.0.0:514' add_attributes: true protocol: rfc3164 operators: - type: syslog_parser protocol: rfc3164

Case 4

If you see the warning below in the log:

[otelSyslog][otelSyslog][2579615][err]2024-10-24T13:01:00.269Z error helper/transformer.go:101 Failed to process entry {"kind": "receiver", "name": "syslog/tcp", "data_type": "logs", "operator_id": "syslog_parser", "operator_type": "syslog_parser", "error": "expecting a priority value within angle brackets [col 0]", "action": "send"}

This can occur if you send a syslog message without a pre-header.

article Syslog Ingestion Troubleshooting. in Troubleshooting

Syslog Ingestion Troubleshooting.

Case 1

Case 2

Case3

Case 4