Troubleshooting logs ingested via Fluent Bit

[noel_david]

• Summary
• Troubleshooting steps
• What's next

 

Summary

This troubleshooting article will help resolve some common issues faced while ingesting logs into Dynatrace via Fluent Bit.

 

Troubleshooting steps

1. Make sure the token is a  Dynatrace API token with the logs.ingest (Ingest Logs) scope.
2. Make sure  Deploy Fluent Bit is deployed successfully.
3. Once deployed, ensure the Fluent Bit pod is running. This can be checked by running the command 

   kubectl get pods -n <namespace> ​

4.  Ensure the HTTP output plugin, which sends logs to the Dynatrace generic logs ingest endpoint through the configuration file, is configured correctly.
   1. Edit the configmap by running the command below and adding the contents below to it.
      

      kubectl edit configmap fluent-bit -n <namespace> ​

      [OUTPUT]
          name  http
          match *
          header Content-Type application/json; charset=utf-8
          header Authorization Api-Token {your-API-token-here}
          allow_duplicated_headers false
          host  {your-environment-id}.live.dynatrace.com
          Port  443
          URI   /api/v2/logs/ingest
          Format json
          json_date_format iso8601
          json_date_key timestamp
          tls On
          tls.verify On​

   2. Run the command below to check if the logs are sent successfully to Dynatrace.

      kubectl logs <fluent-bit pod-name>  -n <namespace> | grep  {your-environment-id}  

      You'll get similar logs like this repeating.
      

      [timestamp][ info] [output:http:http.2] {your-environment-id}:443, HTTP status=204​

An HTTP status of 204 means the request was accepted.
Other possible responses include '401 Unauthorized' if you used a bad token, or any of the responses listed here.

What's next

If none of the steps in this article resolved the issue, open a chat or support ticket and provide the following details:  

• Sample logs created by the pods.
• Output of the following commands:
  

  kubectl logs <fluent-bit pod-name> -n <namespace> > pod-logs.txt
  kubectl describe pod <fluent-bit pod-name> -n <namespace> > pod-describe.txt
  kubectl get configmap <fluent-bit configmap-name> -n <namespace> -o yaml > configmap.yaml​