We recently migrated all of our java services to new hosts with an upgraded OS version. Instead of updating the hosts/properties on our existing services, Dynatrace automatically created new process groups and services and did not migrate our settings. This created a host of issues.
None of our service settings are showing on the duplicate services. Request naming rules are not working. Many request attributes were tied to a process group and are now all broken.
All of our multi-dimension analysis rules are also not present, as they were on the original service and did not get auto-migrated to the new.
When I tried to pin attributes to a service or process group, I now see two of every service in the list, and don't know which is the correct one without trial and error on every single service.
Why did Dynatrace create duplicates of all our services, and is there something we could have done to prevent it or ease the migration pain?
In general if for Example command linę arguments, binary location, host group or other things changed during migration, this is behavior that should may be expected. This is because Dynatrace detects it as some new process that has nothing in common with old one. There is no option for merging process groups. Not all services may be merged. In general new process group always results with separate set of services. You can use configuration API in dynatrace to migrate some of configurations. Not all endpoints that you may need exists, but there is always something.Old process should not be visible when You have timeframe that not covers their existence. If you don’t have agents on old hosts that, they should disappear almost instant. Only archival data will be possible.
That is disappointing.
Looking through the API docs, it looks like it would help maybe with naming rules, request attributes, and dashboards. That is a still a large effort to document all the old vs new service/process group IDs and write code to cover everything. For others, many additional hours of work would be needed to rebuild the multi-analysis rules, custom error detections, etc., by hand. As we roll out dynatrace to cover hundreds or thousands of microservices, I don't know how we would manage these tasks.
Also, you are correct that old services drop off from Transactions and Services when we change the timeframe. However, process groups are still visible for things like Request Attribute criteria regardless of our time setting. I can't tell which process group is which since they share the exact same name.
Dynatrace Devs, any ideas on how to approach these kinds of changes? In Appmon, there is zero effort on our part when migrating apps to a new cluster, unless we used a host filter in the agent mapping.
Did you try to adapt the process group detection rules so the new processes will match the old process groups you have already settings on? That would be probably the first thing I'd try to do. Alternatively, you can force a process to be a member of a process group by setting the DT_CLUSTER_ID (and DT_NODE_ID) for each process.
Don't forget you need the process group detection rules, not process group naming rules for that. Also, host groups affect those settings. Processes that would be normally members of one process group will on hosts with different host group end up in two separate process groups.
Often overlooked, but getting the right process group names / IDs is one of the most important things with Dynatrace deployment. If your process groups are not wisely named and used from the beginning, you might find your self in a very uncomfortable situation.
We couldn't find anything wrong with the process group detection rules. Both sets of groups were identified exactly the same way with the same result.
We did see that the new servers are in a different host group. Would that cause the process/service groups to split up?
Exactly! Dynatrace splits processes groups by host group. Processes in a process group must be on hosts within the same host group.
Just change the host group for the new hosts to the host group for the old ones and restart the processes. I don't know your reasons for the host groups in your environment - the change of a host group might have other implications in your environment (tagging rules, management zones, etc...)