Prevent Dynatrace to create new process group instance

[AK]

Hi Folks,

How can I prevent Dynatrace to create new process group instance if command line argument changes.

We tried to add process instance extraction rule under advance process group detection but that didn't worked.

Does anyone came across this situation and resolve it successfully?

Regards,

AK

[DanielS]

Hello @AK if I understand you well, you can create an exception, with the option to not monitor a Process Group. I leave you the guide.

[AK]

@DanielS, thanks for your comment. However, my dilemma is,

whenever application team does any update in command line argument of process, Dynatrace creates new process group instance. This trigger an alert for old instance and it stays in dead state forever.

We have applied below advance detection rule,

Regards,

AK

[AntonPineiro]

Hi,

Regarding that alert. Is process availability enabled?

Just thinking that you can use process group alert or OS services monitoring if that process is in relation to some service.

Best regards

[fstekelenburg]

@AK , question: is the change performed in a Maintenance Windows set in Dynatrace, configured with "Disable problem detection" (Do Not Monitor, Do Not Alert)?

I have noticed something similar recently with a (DT_CLUSTER_ID) change that changed the process(group) names.
I assume that because Dynatrace did not monitor, it did not notice the clean stop of the application's processes. After the maintenance window ended, the new process names were monitored fine, but availability Problems were created for the old processes, that only could be manually closed.
If that indeed is the case, the MW might better be set to 'Detect problems but don't alert' for such changes.

[AK]

Has anyone came across this situation?

Regards,

AK

[Julius_Loman]

@AK  It's always a cat-and-mouse problem if you are trying to keep up with the monitoring definitions and the operations team makes sudden changes you are not aware of in advance.

The way I recommend customers to migitage such scenarios is to use environment variables for determining the process group and instance. For example if your company is ACME, then having something like:

• ACME_APP - determines process group (required)
• ACME_APP_INSTANCE - determines instance within the process group (optional)
• ... others if required

Then have a generic set of process group detection rules for that. You can always change the process group name afterwards by process group naming rules or manually in the UI.

Using the legacy builtin variables DT_CLUSTER_ID and DT_NODE_ID is not recommended and should be avoided. You should set up your own variables and rules instead. 
Using this approach, your operations them will be in charge of the process identification and it's up to them to have it correctly.

[AK]

@Julius_Loman, thanks for the response.

Will hardware migration can cause new process group instance to be created because, team recently did Power 9 to Power 10 migration on hosts.

The query because, If you look at the original question that is posted, I just can see only command line argument changed between old instance and new instance.

Regards,

AK

[Julius_Loman]

@AK  command line arguments typically won't have effect on it. Since you mention migration - any chance you have the hosts in different host groups (Power9 and Power10)? That is a common reason for a process group split and it is by design splitter. When your host group names won't match, processes running on them are in separate process groups.

[AK · Accepted solution]

I don't see any difference in the host group. The only difference I can see that is in command line argument.

Now, I have added the rule for process group extraction with the property that would remain constant throughout the process group instances. Technically this should work.

Thanks again @Julius_Loman for all the recommendations and inputs.

Regards,

AK

[Julius_Loman]

Well, it depends on the current setting of process group detection. Yes, command line arguments can be a reason if some rule matches.

Anyway, you always can look into the oneagent logs for the particular process (for example for java in the java directory) and after the process is started - there is info about how exactly the process group id and other ids have been calculated:

2023-11-30 14:18:37.156 UTC [001ad680] info    [native] Process group ID ............ 0x8fa541213cab1187
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     PGID item ............... JAVA_JAR_FILE: "xxxx.jar" (ignored) (removedIds)
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     PGID item ............... SPRINGBOOT_STARTUP_CLASS: "xxxx.Application"
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     PGID item ............... EXE_NAME: "java" (ignored) (removedIds)
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     PGID item ............... HOST_GROUP: "xxx"
2023-11-30 14:18:37.156 UTC [001ad680] info    [native] OSI ID ...................... 0xac0325fe2cc8184b
2023-11-30 14:18:37.156 UTC [001ad680] info    [native] Node ID ..................... 0x0000000000000000
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     Node ID item ............ JAVA_JAR_PATH: "xxxxx.jar" (ignored) (removedIds)
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     Node ID item ............ EXE_PATH: "/usr/lib/jvm/java-*-openjdk-amd*/bin/java" (ignored) (removedIds)
2023-11-30 14:18:37.156 UTC [001ad680] info    [native]     Node ID item ............ COMMAND_LINE_ARGS: "/usr/bin/java -jar /xxxx.jar" (ignored)
2023-11-30 14:18:37.156 UTC [001ad680] info    [native] Process group instance ID ... 0x23a664df106309cc
2023-11-30 14:18:37.156 UTC [001ad680] info    [native] Container group ID .......... 0x0000000000000000
2023-11-30 14:18:37.156 UTC [001ad680] info    [native] Container group instance ID . 0x254ba3d2a697aefb
2023-11-30 14:18:37.156 UTC [001ad680] info    [native] Container ID ................