Tagging of shared services/infrastructure components

[ADawson]

We have a common scenario in our environment where we have servers and databases that underpin more than one application.

What approach do people take in relationship to tagging their shared entities from an application tagging point of view ?

Dynatrace appears to be unique compared to other platforms like AWS in that it supports duplicate tag keys against entities like hosts. eg. Application:App1 Application:App2 Application:App3

However the usage of duplicate tag keys does not seem like a good practice especially when we look at our whole environment such as tagging in AWS and support for tag based service mapping in ServiceNow.

We would like to automate grouping/mapping where possible through approaches like using tags to avoid having to manually map application to shared infrastructure relationships.

[Maheedhar_T]

Hey @ADawson ,
I might not be the right person for answering this or I might not be having the right experience on microservices stuff but I can hint you on few things which we do in our organization that might help. You can go over and name your sevices/host groups to contain the names of applications they are running. Something like naming your host groups 
A1_<Application1>_A2_<Application2> and the use wild card pattern in the Auto-Tagging 

{HostGroup:Name/.*_A2_([A-Za-z0-9\.\-]++)_?}

I'm not sure if this is the right pattern but if you search for jinga2 you would find how to form these.

Apologies if this isn't what you were looking for.

Regards,
@Maheedhar_T 

[ADawson]

We have already implemented a hostgroup naming convention to align with Services in our ServiceNow CMDB and limitations around hostgroup names like length and that they only apply to OneAgent hosts may impact the approach in our environment.
Thanks for idea, I will keep the flexibility of using regex in naming rules in mind.

[islam_zidan]

Hi @ADawson 

I am usually use some selector that get the service from the calling transactions, this will help to tag the flow with the right tag, please check the below example for that.

type(SERVICE),toRelationships.calls(type(SERVICE),tag("xxxxx","yyyyy"))

Thanks,

Islam

[ADawson]

The challenge for us is to implement a consistent approach across Dynatrace, AWS and ServiceNow that all platforms can support. Being able to utilise relationships in Dynatrace is an approach that would be great but we want them to translate into related service maps in ServiceNow in an automated way if we can.

Thanks for the idea, I will keep the relationships entity selector in mind.

[Kor02]

We have a similar situation going on with shared hosts/databases and haven't really landed on a great way to handle it.

Currently, we just add duplicate tags for everything associated with that entity (app:thing1 app:thing2, etc). If that host has a problem, all could be impacted. Alerting Profiles are configured by team, and we have a process to open incidents in PagerDuty based on a tag value. This means everyone tagged gets an alert.

For us, shared infrastructure usually means a DBA or Systems Engineer with server access needs to do the actual work, so app teams may get alerts for something they have no way to take action. This has led to us really tuning infrastructure alerts, and adding some filtering/rules in PagerDuty to reduce alert fatigue. You may be able to get creative with Alerting Profiles to do the filtering if you're not using something like PagerDuty.

So, yeah, curious on other replies!

[ADawson]

Yes similar to us. Appreciate the response.

Long term we are hoping to get problems from Dynatrace raised as Incidents in ServiceNow and after-hours notifications that are automatically assigned/sent to the appropriate support team – App, DBA, Infrastructure.
Current state is just assigning ServiceNow incidents manually via a central ops team/roster.
A rostering and notification capability like you have with Pager duty is planned in the future.
In preparation we are trying to line up AWS, ServiceNow and Dynatrace such as the using same approach to tagging Shared Infrastructure across all platforms to support things like automating tagged based service mapping in ServiceNow, hence the trying to avoid the duplicate tag approach if we can.

We are fairly new to Dynatrace. Tuning to reduce the high alert noise/fatigue issues are work in progress for us.
We are focusing on using the new Davis Anomaly Detection app for most of our alerts going forward as it provides us a number of benefits such as –
- Use a consistent naming convention to help people track an alert back to what rule created it.
- Include support in the query so that entities can be excluded from the default alerts by setting a similarly named tag key with a value of Disabled.
- Set properties so additional details/notes can be included in problems.
- Does not appear to have any scaling issues like we hit with Metric Events.
Im hoping to be able to avoid having to manage numerous alerting profiles if we can through –
- Tuning alerts
- Using specific entity tags/alert properties to filter alerts via a single alert profile
- Future – use Event Management in ServiceNow similarly to how you are using PagerDuty to filter noise and assigning to the appropriate support team.

Thanks