Help with Process Group Naming Rule

[ct_27]

All,   I'm have a process that runs on all of our Windows machines,  "ProcessName1".   When I go to the Technologies view in DT I see a very long list of all the locations this process is running, so imagine the name "ProcessName1" repeated 100's of times with no other distinguishable characteristic.  It's like having 100 people in a room name "Joe".

Every one of our hosts has a HostGroup Name.  To be able to determine which process belongs to which host or hostgroup I created a 'Process Group Naming" rule shown below.

The problem I'm having is whenever i turn this on a process group is created that lists all of my HostGroupName values in alphabetical order until a character limitation is met.

What I expected was that on each host....if the Host Group Name EXISTS the rule would have pre-fixed the Process Group Detected Name with the HostGroup Name.

{HostGroup:Name} - {ProcessGroup:DetectedName}

Can anyone help explain why my logic is off and maybe even help me reach my objective?  Thank you in advance to the community.

Also, this is directly from Dynatrace online documentation:

"Additionally, host groups affect how process groups are detected. When the same process is running in two different host groups, Dynatrace will create one process group for each host group. This means you can also configure process groups differently depending on which host group they run in. Consequently, services are also grouped per host group. So you can configure services differently per host group."

[Yosi_Neuman]

Hi @ct_27 

I can help with the logic - You have used Proccess Group naming rule but you are looking for Proccess Instance naming rule , which as far as I know is not present.

Check this thread and this one too 

HTH

Yos 

[ct_27]

I read through the two threads, wow, that is some complex stuff. Many of which look like they have other consequences as the rules are applied, that it overlaps, has exceptions, or might have to be applied multiple times for each need.  I feel like this should be much easier to accomplish.

I'm actually looking primarily for the process groups to become better organized.  Once the process groups split up correctly, the process group instances will naturally fall in line.  The rule I applied works for the most part....out of the 700+ only 2 process groups are pulling in a bunch of servers.  You can see in the screenshot many others are naming correctly.   The problematic process groups, one is pulling in 172 servers, the other 44 servers. So, of the 1,300 servers monitored this is happening only to 216.  

Some of these machines are newer, some have been on OneAgent for a year.  All of our servers are re-started every month.  

I feel the configuration/logic is right but there is something in Dynatrace that's just not acting consistently.  This happens most frequently with Go processes.  Not sure if it's related to the Go technology or just coincidence that this technology is used for a widely used process.

@Yosi_Neuman thanks for the input.  I'll spend more time reading through the threads but I think my issue is simpler than that.

[adam_gardner · Accepted solution]

You're correct. There is nothing wrong with your logic and you're right that host groups automatically create separations between process groups.

Were the host groups applied after the processes were started? Processes remain in their groups until they're restarted.

If that's the case, then the solution is to restart the processes and / or wait until your natural restart cadence. You'll gradually see new groups as the processes "move" from the combined group to their individual groups (after the process restart).

To demonstrate, in my demo setup (screenshot) I:

1. Created 2 VMs and installed OneAgent with no host groups
2. Installed apache2 on both
3. Modified the host group while apache was still running and used --restart-service to restart OneAgent
4. Both apaches are in the combined group
5. Restarted only apache process on group1 host
6. Notice a new group + old one (which will age away eventually)

Edit: You mentioned that your OneAgents have been there for a while. Are they up-to-date? Let's not rule out an old OneAgent as a potential issue.

If this isn't the case, then speak to a DTONE representative via chat as a support ticket might be necessary?

[ct_27]

@adam_gardner thanks so much for breaking it down. I think you nailed it with the the aging out part and the old OneAgent part.  

     We, (my admin and I) thought we were following all the rules when we did the --restart-service, not realizing this required pretty much a full system reboot because there are so many monitored processes on each machine.  This began our frustration.  

     Next we did reboot some machines (with our Guardian by our side) but then we saw the old and new groups. We knew we were on to something but it felt buggy.

     Finally we left it for a few months and yes most stuff started to clear but some stuff was still mixed up. This is where we were struggling to consistently recreate the issue and explain root cause.

    After your explanation and re-thinking the timeline it was a combination of a bunch of stuff.  New machines being deployed, not all old machines fully rebooting, the waiting game for old groups to phase out, and having some old OneAgents around.

    I'm going to go straighten all this out, give it time to settle, and will come back to this post to update the community with our results.

  Thank you again. 

[ct_27]

Adam's response is correct.  It's unfortunate that DT does this because it causes a moment of panic and concern.  We reacted the first time based on DT suggestion to clear out the rule but the correct solution is to wait it out and let DT self-resolve.  

[MarwanC]

This is great thread. But there are still some clarifications what will be the resulting merge on the service that are served via the Apache. I have exactly the same setup. Two web servers that are running on two separate hosts. Both serving the the requests via 4 backends application servers. 

I have the following:

1. Both web servers have already the same host group (in this case we call  _UAT.

2. These hosts appears as a separate Services  Dynatrace (and below that all the services that they are called)

Based on what you described, we should have different host group on each server (group 1 and group 2)

Where do you define the rule is it under this or in another place?

Settings -> Container and Prosess -> Group Naming

 my question at the moment is that I am trying to create SLOs combined for both web servers, not only the web server for any of the services appearing under this web server? I still have the problem that I have to do the SLO for leg1 and leg 2 as I do not see these groups combined into one.

Can you please shed some light on this to see if this solution you proposed is applicable to my case.

Currently is very frustrating to do these SLOS for each web server and below

[MarwanC]

Here are the procedures that I used but it is still not combining the two apache in one which are running on different hosts

1. On both hosts I use the same host group (identical name). We one host group per env (QA and PROD)

2. There is no need to restart one agent (as both host groups are the same?).

3. Define a new process naming group (see above)  the preview also comes correctly and shows that the two apache are showing grouped together (leg 1 and leg 2) - This is accessed from : 

Settings -> Container and Prosess -> Group Naming

4. Restarted both Apache httpd on both hosts.

5. going to hosts in hosts classics page still showing same names of processes.

6. going to services is still showing two separate services one for each host.

Any idea? I appreciate your support to see why this procedures is still not working?