Random Connection refused occuring in the HTTP Monitor

[SabineR]

Hello all,

I created several http Monitors, each with 10 to 20 HTTP Requests. These http requests are linked between them and simulate http requests of some mobile applications.

All http monitors are run from a public location.

We already use this public location for other synthetic browser monitors.

The issue is that we have a recurring error, "Connection refused" coming every day, completely randomly.

Never the same HTTP Request, never the same timeline. Currently, impossible to find a pattern.

So, I have some questions:

- As the public location has several IP addresses, during one run, is it always one unique IP address used or not?

- How can we get which IP address is used during the run? (In execution detail, we have the public location name and id and that's all). 

- Is it possible to get the execution detail of other failed run than only the last one? (Or if we can receive it by mail, via notification menu...)

- Any idea on how to resolve it?

Many thanks for your help.

Sabine

[DanielS]

Hi @SabineR for me a good starting point is this great guide from @HannahM . Please check if the public IP's are whitelisted in the app?

[SabineR]

I will check on our side, yes.

>If a public location, is this endpoint available? 

What is the consequence of not having an endpoint?

[HannahM]

• During 1 execution, all the requests will be run by a single engine using a single IP address. 
• We do only use a subset of the IPs published for each public location, as we hold some in hand for expansion etc. Each engine is tied to a specific IP address, and several engines may be tied to that same IP address. You can see EngineId for each execution in the execution detail, so if you always see different ones for the failures then it is unlikely to be related to a single IP. If needed, support can tell you what IP relates to the EngineIds you are interested in, but, as @DanielS mentions, you should ensure all the listed IPs are whitelisted. 
• Only the last failed, and last successful, execution details can be seen at any point in time. 
  You can use the GET HTTP Monitor  API to retrieve the last execution results as often as you like. 
  Support can provide basic details for other executions but only for specific troubleshooting. 

[SabineR]

Thank you for your answer.

For the engineID, I just checked some case (failed run I downloaded and the last one) : unfortunately, there is no unique engineID which is responsible of all the errors.

The engineID of the last successfull run had returned an error previously.

>Only the last failed, and last successful, execution details can be seen at any point in time. 

It would be great to have more...

[HannahM]

I recommend that you vote for this Product Idea if you would like to see the details for more executions. 🙂 

[AntonioSousa]

@SabineR,

Those are really tricky to find. Here's my reasoning.

• I would start with that Dynatrace's synthetic monitoring is very stable. I have a measurement that I have been running for two years, from two locations, and that gave me only a handful of errors in that timeframe.
• Given several reasoning's in the post, it is very unlikely that the reason for something doing a "connection refused" is related to Dynatrace.
• So, I would point out to a problem nearer your location.
• A "connection refused" is something very clear: something is actively refusing a connection, to port 443, I assume.
• Given the last one, I would track down if it's at the server/process level, or at a security network level.

So, if you have the chance to have a OneAgent server side, check connectivity of the process. If not, might be better to diagnose network level. See if you can get a network trace of some type, any connection refused will be immediately visible.

[SabineR]

As I have more questions linked directly to our environment/network, I'm creating a ticket...

Thank you all!

[SabineR]

Hello all,

We have still issues with this.
We are investigating on our network side, but it's long and we must track all other errors (we get timeout, unauthorized etc...).
I have 2 http calls (POST mode) which generate the most these "Connection Refused" errors.
And I would like to do a re-run of these 2 http calls, when it happens.

Here is the scenario, I want to put:
I duplicate the request, and in 1st post execution script:
if status 200, we skip the next request.
if status 0, we do the next request (ie the same request)
if status >=400, run is in error.

Do you think it's possible? 
Any other idea?

Many thanks for your help

Sabine

[HannahM]

If the error is a network error, like connection refused, then no, this will not work. You can ignore status codes and continue but not network level errors. 

You can increase the threshold that you have outage handling set for, so you don't alert as early, or you could use Workflows to do this. 

[SabineR]

Thank you for the confirmation, Hannah.

[SabineR · Accepted solution]

Issue resolved: in HTTP Monitor scripts, connections are default automatically closed.

Or persistent connections are the default behavior of any connection in HTTP 1.1.

I forced connection to be persistent for some HTTP requests and the number of connection refused has drastically regressed.