Enrich Fargate Logs with `dt.entity.process_group`

[eprogers]

I'm trying to enrich my Fargate logs with the `dt.entity.process_group`. The end goal is that I want my team to be able to view all logs related to a service.

Here is a screen shot of the feature I'm trying to use:

 And when I click the link above I expect to see logs associated with my process group:

As it is, Oneagent is enriching my logs with various other fields I need including: `dt.entity.process_group_instance`, `dt.entity.host`, `dt.entity.process_group`, `dt.span_id`, `dt.trace_sampled`, and `dt.trace_id`. However, I cannot get the the `dt.entity.process_group` to be included in the log, even though Oneagent is aware of that what that group. I know Oneagent is aware of this id because I can see the process group ID in Oneagent's logs when when I set the `DT_LOGLEVELCON` env to `info` and when I exec into the container and inspect Oneagent's conf files, it actually uses the process group id as part of its internal file structure that it sets up at runtime.

I've scoured docs and settings options and it doesn't appear as though there is a setting I can change to toggle this. But maybe I'm missing something.

Interestingly, when I run Oneagent on a Linux or Windows EC2 instances, the `dt.entity.process_group` is included in the logs. We are committed as an organization to running our apps on Fargate so simply running our apps on linux is not an option. We're about to write our own logic to query Dynatrace at deploy time, manually look up the process group id ourselves, and enrich out logs with this manually. This seems like a lot of work for looks like its supposed to be an off-the-shelf feature.

Has anyone run into this problem before? Any suggestions? Thanks!

[ari_polavarapu · Accepted solution]

Hello,

This is expected by default since Fargate deployments are app-only and there is no Log Agent deployed (like it is with full stack deployments). Try enabling this OneAgent feature (it should start adding the process group metadata to the logs) to see if you get any better results

This change does require a restart.

Thanks,

Ari

[eprogers · Accepted solution]

Hi Ari, its good to know that this is an expected behavior from Dynatrace and not just a configuration issue on my end. My team put together a custom workaround for this which I'll share in case it benefits anyone else. Our solution is roughly:

1. Determine our app's process group ID while we are creating/deploying our infrastructure
   1. If our app is being deployed for the first time and hasn't registered in Dynatrace or we can't determine the app's process group ID or any reason we just put in a dummy place holder value
2. Set our app's process group ID as an environment variable for our log_router/fluentbit container
3. Modify the logs in our log_router/fluentbit to include a `dt.entity.process_group` field in each log and use the process Group ID environment variable as its value.

We are able to achieve step #1 pretty easily because our IaC tool Pulumi has a provider for AWS that sets up our infrastructure and another provider for Dynatrace we can use to determine our process group. We can use both providers in the same stack so when we deploy our IaC it makes API calls to Dynatrace to determine process group ID, then passes that value into the code that sets up our ECS app.

Pulumi ECS Docs

Pulumi Dynatrace Docs 

[ASE]

Hello, @ari_polavarapu.

We are having a similar issue in our environment. Link added below.

We are using ECS/Docker/Fargate.

Would enabling the Metadata context enrichment for logs [Opt-In] setting resolve our issue?

https://community.dynatrace.com/t5/Container-platforms/Container-Logs-Not-Found-Containers-App/m-p/2...