I have synthetic tests that create problems as soon as 2/3 locations fail. These problems immediately send emails to support teams as an FYI.
We have a secondary Alerting profile that will send the problem to our NOC team if the problem stays open for 15 minutes.
The problem we have with this system is that if the tests are being a bit intermittent, the problem will resolve itself and then re-open again. This doesn't reset the clock on the problem timer, so an alert goes to the NOC which is going to start MI processes.
We don't want alerts going to the NOC if the test failure is intermittent, only if it's hard down.
(Yes I know we could set up 2/3 locations failing 3 times consecutively, but that prevents us from sending the email alerts to the support team for intermittent failures (fail, fail, pass, fail, pass, pass, fail, pass etc)
My question is, do you know how long a problem has to be closed for before a new one is raised rather than re-opening the 1st one?