Setting up custom alerts in Dynatrace for AWS resources

[devinmarco]

Hello Everyone

I hope this message finds you well. I'm currently working on configuring custom alerts in Dynatrace for our AWS resources, and I could use some guidance and advice from those who have experience with this tool. If you have expertise in Dynatrace or have faced similar challenges, your input would be greatly appreciated.

Here's a bit more context about my situation:

Objective: I want to set up custom alerts in Dynatrace to monitor specific AWS resources and trigger notifications when certain conditions are met. This will help us proactively address issues and optimize our AWS environment.

Challenges:

1. Understanding Metrics: I'm struggling with identifying the relevant metrics and thresholds to create effective alerts. Which AWS metrics should I be monitoring, and how do I determine the right thresholds?

2. Alert Configuration: Once I have identified the metrics, I'm not entirely sure how to configure the alerts within Dynatrace. What are the best practices for alert configuration?

3. Notification Integration: We use multiple communication channels for alerts, including email, Slack, and SMS. How can I ensure that Dynatrace integrates smoothly with these platforms for timely notifications?

4. False Positives: I want to avoid being flooded with false positives. Any tips on fine-tuning alert conditions to minimize unnecessary alerts would be incredibly helpful.

If you've successfully set up custom alerts in Dynatrace for AWS resources or have insights into any of the challenges I mentioned, please share your knowledge. Your experience will undoubtedly help not only me but also others in the community who might be facing similar issues.

Thank you in advance

[Fin_Ubels · Accepted solution]

Hey Devinmarco,

This is a very open ended request and I would highly recommend first narrowing it down.

1. Without knowing what is important to you/your team this is difficult to answer. One team might care about network metrics, another about hardware, another about users. Also with over 100s of metrics to pick from relating to many different services (like 80+ supported) there is a lot. However, to get an idea of things that are important, Dynatrace has OOTB alerting available. This is only for a very small few services and metrics but it is there and can give you ideas. You can also look at other alerting options under the same settings to get an idea.

Also, to see what metrics are available you can head over to the docs here: Amazon Web Services | Dynatrace Docs

2. Try and focus on what someone would consider a problem. There are many different options, different conditions, models, etc so determining the best way to create alerts is very subjective. One alert may require completely different configurations to another. Also I would recommend starting small, ensuring what you have created works, makes sense, doesn't over alert, and then expand.

3. Dynatrace supports Slack and Email notifications just fine. For SMS notifications it supports other services that can send via SMS or if your carrier supports it, there are options for sending custom messages to an API or email which can be configured to send via SMS.

4. While setting up custom alerts you can see the last 7 days and whether or not those custom alerts would have alerted. This is incredibly helpful as well for setting up alerts just after a problem has occurred because you can tune the alert to not fire except during that problem scenario since you can visualize it right there in the UI.

I would really recommend focusing on mission critical services and then expanding the alerts. Try not to look too far out just yet. Find something like Lambdas, or EC2 or something else and focus on just one service first.

Hopefully this helps to move things in the right direction. If you have any specific questions follow up and hopefully people can help out!