We are using the SNMP Action Plugin to integrate IncidentRules with Omnibus. We have logic in the SNMP collector that will send different values to Omnibus based on the Severity that is sent from DT. After doing some testing, I've found some behavior that was unexpected, so hoping someone can confirm a few of the quirky details. I wasn't able to find these details in the provided doc.
1. The "Severity" value that is sent over SNMP (and defined in the MIB file) comes from the "Action Severity" - that is the Severity that appears in the SNMP action. The Incident Rule Severity is NOT sent in the SNMP trap.
2. When I set the Incident Rule severity to "Informational", it doesn't appear to fire the SNMP action. So for example, I'm using one condition: "Preparation Count (Database)". I then trigger start and end messages by moving the thresholds to extreme values, so that the current value is well above or below the threshold. If the Incident Rule's severity is Information, then it never seems to trigger the action, regardless of how everything is setup (Action Severity, which threshold is exceeded, etc.)
The net result is that, if I want to send an alert that has "Severity=Informational", then I need to set the following:
- Incident Rule Severity = Warning or Severe
- SNMP Action Severity = Informational
Does this seem correct, or could I be doing something wrong?
I'm running into the same issue myself right now. I'm curious to know the answer to this as well.
The SNMP trap sent takes the severity from the "Action severity" and not the incident severity or even better, the violation threshold. Right now, it is always hardcoded.
The trap I receive does not contain the value or threshold violated so I can at least calculate the severity myself with those values.
Looking forward to seeing the answers to your questions above.
For the first item you are correct, the severity sent in via the SNMP Trap is that of the Action Severity. This is the same behavior for any actions (i.e. email alerts contain the Action Severity). For your second question I suspect this is due to the Informational Incident's being Auto-Confirmed. Check your Incident's Dashlet (ensure you change the filter to see all Incidents and not just Warning/Severe and unconfirmed) to validate.
@Diego M - I believe the violation threshold should be sent - please make sure you are not truncating the trap too soon in the Action settings (Maximum Number of Message Octets). For testing you can set this value to 0 or -1 to avoid any truncation.
Markie - thanks very much for your response. After disabling the severity and status filters, I do see a series of Confirmed incidents in the dashlet.
So if I understand correctly, when an incident is auto-confirmed, it does not trigger the associated actions, correct?
Is there a way to disable auto-confirm for Informational incidents?