Automated way to disable synthetic monitors for a particular time period

[AK]

Hi Folks,

Is there a way we can disable the few selected synthetic monitors for the day of week and also enabled them back?

Basically, I'm looking to automate this task. I checked on synthetic API but we need to pass complete synthetic monitor JSON with "enabled" parameter as "false" or "true".

I don't want to use Maintenance window as it has some drawbacks.

Has anyone did this?

Regards,

AK

[ChadTurner]

What you could do is repost your synthetic and pass the parameter of false instead of true:

[AK]

Thank You Chad for your comments. I'm looking for some automated way to do it.

[AntonioSousa · Accepted solution]

At the moment, you should be able to do this through the API.

Please check out:

https://github.com/Dynatrace/snippets/tree/master/api/synthetic/bulk-enable-disable-monitors

[AK]

Exellent. Thanks for this Antonio. It worked like a charm.

[AK]

@Antonio S.

Script was working very well till now but it is started failing. Can you please help here?

Below is the error I'm receiving,

"E:\DynaTrace\Scipt.py", line 98, in <module>
    response = restApiGet(domain+"api/v1/synthetic/monitors?tag="+tag+"&Api-Token="+apiToken)
  File "E:\DynaTrace\Script.py", line 15, in restApiGet
    if int(response.headers['X-RateLimit-Remaining']) == 0:
  File "C:\Program Files\Python39\lib\site-packages\requests\structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'x-ratelimit-remaining'
waiting for 5 seconds
Reading contents of Error.log file

Is it the case that dynatrace API are not passing/using x-ratelimit-remaining header is now?

Regards,

AK

[AntonioSousa]

It looks like you might be experiencing some rate limiting. Take a look at:

https://www.dynatrace.com/support/help/dynatrace-api/basics/access-limit/

https://www.dynatrace.com/news/blog/best-practices-for-accelerating-dynatrace-apis-within-large-moni...

[AK]

@Antonio S.

I tried it for just four monitors but its failing for them too.

[AntonioSousa]

Please beware that the rate limit applies for all API calls. Please check you are not using too much of them... If not, you should probably open a support case, or at least try the Chat.

[AK]

Hi Antonio,

We came to know from support that,

Those throttling headers (X-RateLimit) will no longer in use from SaaS 1.210 version and new implementation now relies on a rapidly refreshing thread pool that processes API rather than a threshold of calls that is used per minute or hour.
And sleep time must be added based on the HTTP 429 response and then try again shortly after

In our python script (The one which is present on git), sleeping is added based on the X-RateLimit-Reset threshold.

I believe, adding random time.sleep won't be appropriate.

Can you help or suggest, how we can adopt or modify the time.sleep based on HTTP 429 response or when throttling headers are not available?

Regards,

AK

[AntonioSousa]

Not sure about this new implementation... Did Support make any suggestion?

[AK]

It's resolved. Support suggested to add,

if response.status_code == 429;
time.sleep(10);

Queue will get completed within 10 sec.

[adam_gardner · Accepted solution]

I'd highly recommend using our open source tool monaco rather than troubling yourself with scripting against our APIs 🙂 #makeyourlifeeasier

That said, see my other comment about this being available OOTB when you create maintenance windows now.
https://github.com/dynatrace-oss/dynatrace-monitoring-as-code

[jshelby]

We accomplished this by creating a Jenkins job that runs the commands to disable / enable at the desired times. We use a tag on designated the synthetics that are disables/enables.

[rmeli]

We also use tags to specify when a synthetic should be enabled/disabled and another job uses the API to do the enable/disable.

[adam_gardner · Accepted solution]

This is now available when you create a maintenance window (notice the toggle at the bottom of my image).