topic Re: Disabling bunch of synthetic monitors for particular time period in a week in Synthetic Monitoring

Automated way to disable synthetic monitors for a particular time period

AK — Fri, 10 Mar 2023 15:28:34 GMT

Hi Folks,

Is there a way we can disable the few selected synthetic monitors for the day of week and also enabled them back?

Basically, I'm looking to automate this task. I checked on synthetic API but we need to pass complete synthetic monitor JSON with "enabled" parameter as "false" or "true".

I don't want to use Maintenance window as it has some drawbacks.

Has anyone did this?

Regards,

Re: Disabling bunch of synthetic monitors for particular time period in a week

ChadTurner — Wed, 11 Nov 2020 19:57:55 GMT

What you could do is repost your synthetic and pass the parameter of false instead of true:

Re: Disabling bunch of synthetic monitors for particular time period in a week

AntonioSousa — Wed, 11 Nov 2020 20:46:12 GMT

At the moment, you should be able to do this through the API.

Please check out:

https://github.com/Dynatrace/snippets/tree/master/api/synthetic/bulk-enable-disable-monitors

Re: Disabling bunch of synthetic monitors for particular time period in a week

AK — Fri, 13 Nov 2020 09:52:07 GMT

Exellent. Thanks for this Antonio. It worked like a charm.

Re: Disabling bunch of synthetic monitors for particular time period in a week

AK — Fri, 13 Nov 2020 09:52:42 GMT

Thank You Chad for your comments. I'm looking for some automated way to do it.

Re: Disabling bunch of synthetic monitors for particular time period in a week

jshelby — Tue, 17 Nov 2020 15:31:30 GMT

We accomplished this by creating a Jenkins job that runs the commands to disable / enable at the desired times. We use a tag on designated the synthetics that are disables/enables.

Re: Disabling bunch of synthetic monitors for particular time period in a week

rmeli — Tue, 17 Nov 2020 19:33:12 GMT

We also use tags to specify when a synthetic should be enabled/disabled and another job uses the API to do the enable/disable.

Re: Disabling bunch of synthetic monitors for particular time period in a week

AK — Tue, 16 Feb 2021 14:51:02 GMT

@Antonio S.

Script was working very well till now but it is started failing. Can you please help here?

Below is the error I'm receiving,

"E:\DynaTrace\Scipt.py", line 98, in <module>
    response = restApiGet(domain+"api/v1/synthetic/monitors?tag="+tag+"&Api-Token="+apiToken)
  File "E:\DynaTrace\Script.py", line 15, in restApiGet
    if int(response.headers['X-RateLimit-Remaining']) == 0:
  File "C:\Program Files\Python39\lib\site-packages\requests\structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'x-ratelimit-remaining'
waiting for 5 seconds
Reading contents of Error.log file

Is it the case that dynatrace API are not passing/using x-ratelimit-remaining header is now?

Regards,

Re: Disabling bunch of synthetic monitors for particular time period in a week

AntonioSousa — Tue, 16 Feb 2021 15:38:41 GMT

It looks like you might be experiencing some rate limiting. Take a look at:

https://www.dynatrace.com/support/help/dynatrace-api/basics/access-limit/

https://www.dynatrace.com/news/blog/best-practices-for-accelerating-dynatrace-apis-within-large-monitoring-environments/

Re: Disabling bunch of synthetic monitors for particular time period in a week

AK — Tue, 16 Feb 2021 17:52:20 GMT

@Antonio S.

I tried it for just four monitors but its failing for them too.

Re: Disabling bunch of synthetic monitors for particular time period in a week

AntonioSousa — Tue, 16 Feb 2021 17:56:31 GMT

Please beware that the rate limit applies for all API calls. Please check you are not using too much of them... If not, you should probably open a support case, or at least try the Chat.

Re: Disabling bunch of synthetic monitors for particular time period in a week

AK — Thu, 18 Feb 2021 14:28:17 GMT

Hi Antonio,

We came to know from support that,

Those throttling headers (X-RateLimit) will no longer in use from SaaS 1.210 version and new implementation now relies on a rapidly refreshing thread pool that processes API rather than a threshold of calls that is used per minute or hour.
And sleep time must be added based on the HTTP 429 response and then try again shortly after

In our python script (The one which is present on git), sleeping is added based on the X-RateLimit-Reset threshold.

I believe, adding random time.sleep won't be appropriate.

Can you help or suggest, how we can adopt or modify the time.sleep based on HTTP 429 response or when throttling headers are not available?

Regards,

Re: Disabling bunch of synthetic monitors for particular time period in a week

AntonioSousa — Thu, 18 Feb 2021 22:12:59 GMT

Not sure about this new implementation... Did Support make any suggestion?

Re: Disabling bunch of synthetic monitors for particular time period in a week

AK — Wed, 24 Feb 2021 06:12:46 GMT

It's resolved. Support suggested to add,

if response.status_code == 429;
time.sleep(10);

Queue will get completed within 10 sec.

Re: Disabling bunch of synthetic monitors for particular time period in a week

adam_gardner — Wed, 24 Feb 2021 07:18:44 GMT

This is now available when you create a maintenance window (notice the toggle at the bottom of my image).

Re: Disabling bunch of synthetic monitors for particular time period in a week

adam_gardner — Wed, 24 Feb 2021 07:20:20 GMT

I'd highly recommend using our open source tool monaco rather than troubling yourself with scripting against our APIs 🙂 #makeyourlifeeasier

That said, see my other comment about this being available OOTB when you create maintenance windows now.
https://github.com/dynatrace-oss/dynatrace-monitoring-as-code