Multi-Language Log Ingest for Dynatrace via FluentD

bsnurka — Wed, 22 Jan 2025 17:51:02 GMT

Use Case

We need to ingest multi-language logs into Dynatrace via FluentD Integration as the OneAgent native log ingest is not supported due to one of the reasons listed on this table: https://docs.dynatrace.com/docs/shortlink/oneagent-support-matrix#other-modules

Issue/Symptoms

When setting up FluentD with Dynatrace to ship container logs, one may notice that multi-language logs, such as logs including both English and Chinese, will not properly ingest into Dynatrace. This results in the non-English text displaying as �/UTF Character U+FFFD within the Dynatrace UI. A deeper review of the environment will show that the logs coming into Dynatrace from FluentD already have the non-English text swapped out for �, meaning the text is being altered within FluentD and not at the Dynatrace Ingest.

Solution

This can be caused by how FluentD’s Tail plugin (https://docs.fluentd.org/input/tail#encoding-from_encoding) works with encoding.

If we only have the `encoding` parameter defined, FluentD is likely attempting to ship the logs in the default ASCII-7 format to the UTF-8 endpoint within Dynatrace. This leads to the unknown language characters being swapped for �.

Adding the `from_encoding utf8` parameter to the FluentD.conf file will correspondingly force FluentD to ensure the native logs are left in their UTF8 format, and do not ever get encoded into ASCII-7. This change will resolve the multi-language log ingest issue, assuming the non-English Language in question is included in the UTF-8 encoding set (https://www.w3schools.com/charsets/ref_html_utf8.asp).

article Multi-Language Log Ingest for Dynatrace via FluentD in Troubleshooting

Multi-Language Log Ingest for Dynatrace via FluentD

Use Case

Issue/Symptoms

Solution