Syslog Ingestion via ActiveGate Troubleshooting Guide

[noel_david]

Summary

This article provides a practical troubleshooting guide for resolving common issues when ingesting Syslog via Dynatrace ActiveGate.

• Summary
• Setup
• Deploy Environment ActiveGate
• Enable syslog ingestion on ActiveGate
• Edit the syslog receiver configuration.
• Troubleshooting
• Check the protocol used in theYAML file configuration
• Check theOtel Collector logs
• Verify the Otel collector process is listening on the expected port
• Analyze logs and traffic using TCP dump
• Validate SELInux and firewall
• What's next
• Related reading

Setup

Deploy Environment ActiveGate

Installation steps 

Enable syslog ingestion on ActiveGate

Edit the /var/lib/dynatrace/remotepluginmodule/agent/conf/extensionsuser.conf file and add the following flag:

syslogenabled=true

For more details, please check the documentation 

Syslog ingestion with ActiveGate — Dynatrace Docs

Edit the syslog receiver configuration.

Syslog ingestion configuration in Dynatrace uses the following YAML setup

receivers:
  syslog/udp:
     udp:
       listen_address: "0.0.0.0:514"
       add_attributes: true
       protocol: rfc5424
     operators:
       - type: syslog_parser
       protocol: rfc5424
  syslog/tcp:
     tcp:
       listen_address: "0.0.0.0:601"
       add_attributes: true
       protocol: rfc5424
     operators:
       - type: syslog_parser
       protocol: rfc5424

 

Troubleshooting

Check the protocol used in the YAML file configuration

By default, the ingested syslog must be in the format defined by RFC3164 and RFC5424. 

RFC 3164 log example

<134>Jul 3 15:15:06 DOE-APL-WAF-01.doe.local ASM:CEF:0
F5 ASM 17.1.2 Successful Request
dvchost=DOE-APL-WAF-01.doe.local dvc=10.20.20.3 cs1=/Common/UNIFIEDPLATFORM_PROD_SEC_POL.app/UNIFIEDPLATFORM_PROD_SEC_POL cs1Label=policy_name .

Structure of an RFC 3164 Message

<PRI>TIMESTAMP HOSTNAME TAG: MESSAGE

• Priority value combining facility and severity (e.g., <134>)
• TIMESTAMP: Format MMM DD HH:MM:SS (e.g., Jul 3 15:15:06)
• HOSTNAME: Originating device or system (e.g., DOE-APL-WAF-01.doe.local)
• TAG: Application or process name (e.g., ASM:CEF:0)
• MESSAGE: The actual log content

RFC5424 log example  

<34>1 2025-07-02T11:20:20.003Z mymachine.example.com myapp 12345 ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...

Structure of an RFC 5424 Message

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE

• Priority value (facility + severity)
• VERSION: Syslog version (always 1)
• TIMESTAMP: ISO 8601 format (e.g., 2025-07-02T11:20:20.003Z)
• HOSTNAME: Originating system (e.g., mymachine.example.com)
• APP-NAME: Application name (e.g., myapp)
• PROCID: Process ID (e.g., 12345)
• MSGID: Message ID (e.g., ID47)
• STRUCTURED-DATA: Optional metadata in key-value format
• MESSAGE: Free-form text message

If the logs are meeting this standard, ingestion can still be done with the help of Dynatrace OpenPipeline.
Troubleshooting Syslog Ingestion When Standards Aren’t Strictly Followed. 

Check the Otel Collector logs  

If ActiveGate receives the log records, you should see entries as in the example below:

[otelSyslog][otelSyslog][37448][err]LogRecord #3
[otelSyslog][oteiSyslog][37448][err]ObservedTimestamp: 2024-05-06 @9:52:10.6748723 +8000 UTC
[otelSyslog][otelSyslog][37448][err]Timestamp: 2624-05-@6 11:52:16 +90e0 UTC
[otelSyslog][otelsyslog][37448][err]SeverityText: info
[otelSyslog][otelSyslog][37443][err]SeverityNumber: Info(9)
[otelSyslog][otelSyslog][37448][err]Body: Str(<30>May 6 11:52:10 SOME-HOST systemd[1]: Finished Load Kernel Module fuse.)
[otelSyslog][otelSyslog][37448][err]Attributes:
[otelSyslog][otelSyslog][37448][err] -> priority: Int(3)
[otelSyslog][otelSyslog][37448][err] -> facility: Int(3)
[otelSyslog][otelSyslog][37448][err] -> appname: Str(systemd)
[otelSyslog][otelSyslog][37448][err] -> proc_id: Str(1)
[otelSyslog][otelSyslog][37443][err] -> log: Map({“source": “syslog"})
[otelSyslog][otelSyslog][37443][err] -> hostname: Str(SOME-HOST)
[otelSyslog][otelSyslog][37443][err] -> message: Str(Finished Load Kernel Module fuse.)
[otelSyslog][otelSyslog][37448][err]Trace ID:
[otelSyslog][otelSyslog][37448][err]Span ID:
[otelSyslog][otelSyslog][37443][err]Flags: 0

 

Verify the Otel collector process is listening on the expected  port

 This can be verified with the below command so  that the otel collector process is listening on the expected  port, and no other application uses the port

sudo netstat -tulnp | grep otel

 

This command lists all TCP/UDP listening ports and filters for processes related to OTel. You’ll see output like:

udp        0      0 0.0.0.0:514           0.0.0.0:*       1234/otelcol
tcp        0      0 0.0.0.0:4317          0.0.0.0:*       1234/otelcol

• 514 might be used for syslog ingestion.
• 4317 is the default port for OTLP over gRPC.
• 1234/otelcol indicates the process ID and name.

Analyze logs and traffic using TCP dump

Use the below command to verify that the logs are coming

sudo tcpdump -A -i any port <port>

A sample output might look like:

15:15:06.123456 IP 10.20.20.3.514 > 10.20.20.4.514: SYSLOG
<134>Jul 3 15:15:06 DOE-APL-WAF-01.doe.local ASM:CEF:0 F5 ASM 17.1.2 Successful Request dvchost=DOE-APL-WAF-01.doe.local dvc=10.20.20.3 cs1=/...

This shows:

• Timestamp
• Source and destination IPs and ports
• Protocol (e.g., SYSLOG)
• Message content (e.g., CEF-formatted log from an F5 device)

 

Validate SELInux and firewall

Review the firewall settings and ensure that incoming traffic is allowed on the specified port.
The below is an example command that can be used to verify that

sudo firewall-cmd --zone=public --query-port=514/udp

If the  output is:

yes

This confirmed that UDP traffic on port 514 was allowed

The SELinux configuration was checked using be below example command:

sudo semanage port -l | grep http_port | grep 514

If the result is as below, then it's configured correctly :

http_port_t udp 514 

As an additional test, configure syslog to listen on a higher port, like 8514, and try sending data on that port. If data is received, that means the problem still lies somewhere in the internal environment.

Another verification that can be done is to check if the binary is allowed to bind:

The below command can be used to verify this

getcap/opy/dynatrace/remotepluginmodule/agent/res/dsruntime/otc/dynatraceotelcollector

Example output

getcap/opy/dynatrace/remotepluginmodule/agent/res/dsruntime/otc/dynatraceotelcollector= cap_net_bind_service+ep

Its also possible that nftables can drop traffic silently, on systems using nftables, verify that traffic on port 514 is allowed, it can be tested using the command below

sudo nft list ruleset | grep 'dport 514'
​

Expected results

udp dport 514 accept​

or

tcp dport 514 accept​

What's next

Opening a support ticket - 

Create a support ticket with all the steps mentioned above with the logs, also provide

• Link to the tenant
• Link to ActiveGate configured.

Related reading

📖  Syslog Ingestion Troubleshooting

📖  Installation steps   

📖  Syslog ingestion with ActiveGate