HTTP Synthetic Monitor Reporting Location Unavailability Despite Active Usage

[lwaldrop]

I have a synthetic monitor set up for one location, and it works fine. Now, I want to add a second location to the monitor. However, after a few minutes, the newly added locations report as unavailable.

I’ve gone through the Dynatrace settings but haven’t been able to determine whether this is a configuration issue within Dynatrace or something external affecting availability.

Here’s what I’ve done so far:

1. I navigated to ServiceName > Settings > Frequency and Locations to add the new location(s).
2. In the Availability pane, the currently configured location shows 100% availability, but the new locations consistently show 0% availability.

I have full admin capabilities, so I should have access to all relevant configuration settings.

Any suggestions on what might be causing this issue or areas to investigate further would be greatly appreciated.

[Fin_Ubels]

Hey @lwaldrop 

By the sounds of things, the synthetic is unable to execute successfully from the new location but there could be many reasons for this.

Just some questions to get a better idea of what could be going wrong and what needs to be looked into:

• What type of location was the original, public or private?
• What type of location is the newly applied synthetic location, public or private?
• Have you tried different new locations to see if it is an issue with a specific new location?
• Is the API endpoint that is being tested on a private network or behind a firewall and if so, have the correct rules been put in place that the synthetic location will be able to access it?
• If the API endpoint is publicly accessible, does it have region restrictions or automatic anti scraping measures in place? Could there have been rules previously set up to ignore restrictions or rules for the addresses of the previous synthetic location? 
• When you go to the synthetic execution details, what errors show up in the failed execution? Is there a http error code? Did it time out?

Hopefully some of these questions help you get going in the right direction!

[HannahM]

I second everything Fin mentions. I would start with checking the Synthetic execution details  as that should point you in the right direction.  There's a troubleshooting guide for investigating this kind of issue here. 

[lwaldrop]

@Fin_Ubels @HannahM  Thanks for responding and giving those starting points. I'll try to answer the questions as best I can:

What type of location was the original, public or private?

• The original location was public.

What type of location is the newly applied synthetic location, public or private?

• All locations attempted have been public.

Have you tried different new locations to see if it is an issue with a specific new location?

• I've tried several locations in the same or different region and different cloud providers.

Is the API endpoint that is being tested on a private network or behind a firewall and if so, have the correct rules been put in place that the synthetic location will be able to access it?

• The endpoint is behind a firewall but what rules are in place I'm not sure, I don't have visibility in that area. The relevant team members are out on vacation till after the New Year so it might be until then when I have a better answer for this.

If the API endpoint is publicly accessible, does it have region restrictions or automatic anti scraping measures in place? Could there have been rules previously set up to ignore restrictions or rules for the addresses of the previous synthetic location?

• The endpoint is public. I don't believe there is any restrictions in place, but I'll have to follow up on that.

When you go to the synthetic execution details, what errors show up in the failed execution? Is there a http error code? Did it time out?

•  The specific error is a 403 forbidden error. That has been consistent across the locations I've attempted. What prompted looking into adding a second location was we've been receiving global outage alerts for timeouts that are lasting  ~1 minute before it recovers. It looks like this has been an ongoing (but ignored) problem since it's creation date. Because its set to generate a problem only when a monitor is unavailable at all locations, I wanted to add the second to confirm if its the service itself or just a 'blip' that could occur for any reason.

[lwaldrop · Accepted solution]

@HannahM @Fin_Ubels I located where the issue was. There were IP address restrictions for the service on IIS. 🙂