NGINX logs enrichment whether Dynatrace is enabled or not

[ahmad]

Hello all,

I've been following the article to enable Nginx logs enrichment and to include the DT in the Nginx logs however, I have Dynatrace implemented in one environment and it works fine but when I try to run Nginx on my local machine I get the following error:

[emerg] 1#1: unknown "dt_trace_id" variable
nginx: [emerg] unknown "dt_trace_id" variable

My config looks like this:

worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /tmp/nginx.pid;


events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  custom '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '[!dt dt.trace_id=$dt_trace_id,dt.span_id=$dt_span_id,dt.trace_sampled=$dt_trace_sampled] '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  custom;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Anyone can advise on how I can do Nginx logs enrichment that can work whether Dynatrace is enabled or not?

Thank you.

[ChadTurner]

@ahmad were you able to solve this issue or are you still experiencing it? If it was solved, what was the underlying problem? 

[danmichael]

Is nginx instrumented locally? What version of nginx is being run? Do you have the oneagent installed?

Nginx logs enrichment that can work whether Dynatrace is enabled or not? -- using this method depends on the nginx process being instrumented by the OneAgent

[jasonbutler22]

We also have dev and test environments without Dynatrace. My strategy is to run a startup script to detect if Dynatrace is instrumenting the Docker container my nginx server is running in. The script tests if the /etc/ld.so.preload is set to load the dynatrace one agent library. If it is detected then I update the logging config with the Dynatrace specific logging patterns. e.g.

# Check if the Dynatrace OneAgent is installed by checking for the presence of
# liboneagentproc in the /etc/ld.so.preload file.
if grep -q "liboneagentproc" "/etc/ld.so.preload" ; then 
echo "Enabling Dynatrace logging instrumentation"
# Update or replace nginx.conf to insert Dynatrace pattern [!dt dt.trace_id=$dt_trace_id]
# into the logging config here.
fi

[radek_jasinski]

Hi,

You can try solving this by defining fallback values for the Dynatrace variables using the map directive in NGINX. This is useful when Dynatrace OneAgent is not installed (e.g., in local environments), and NGINX would otherwise throw an error due to unknown variables like $dt_trace_id.

map "" $dt_trace_id_safe {
default "-";
}

map "" $dt_span_id_safe {
default "-";
}

map "" $dt_trace_sampled_safe {
default "-";
}

then update your log_format.

Radek

[andre_vdveen]

Hello @radek_jasinski, do you have a working example of this that you can share? I'd like to see where you placed that and in which file, the nginx.conf or another? I also have a need for Nginx log enrichment, but I get the same error as @ahmad did.

I have the agent monitoring the nginx process in full stack, the OneAgent module dt_one_agent shows up in the modules list.

[radek_jasinski]

Yes, i will share later some example