Metric Ingestion - Query

[Hasan]

Hi There,

Using metric ingest we have created a metric which is configured for 2 servers (1 live server and 1 is fallback server ), the client wants alert to be generated only when both the nodes are not fulfilling the requests, refer screen below for a sample metric created, kindly advise how we can fulfil this.
Please note: with the current configured metric we are getting alerts for both the nodes as separate alert.

Previously posted this in slack so far no reply 

https://dynatrace.slack.com/archives/C01P9N2RZQE/p1693290399106039

[mark_bley]

Hi @Hasan ,

I am assuming both nodes are will send the same metric key.

When both of them fail no metric data points will be sent into Dynatrace so data will be missing

-> "alertOnNoData": true,

Using a metric event like following would alert you if the metric is completely missing, i.e. primary and secondary both have failed.

[{
    "schemaId": "builtin:anomaly-detection.metric-events",
    "schemaVersion": "1.0.15",
    "scope": "environment",
    "value": {
      "enabled": true,
      "summary": "Important Metric is missing",
      "queryDefinition": {
        "type": "METRIC_SELECTOR",
        "metricSelector": "very.important.metric.that.has.to.be.there",
        "managementZone": null,
        "queryOffset": null
      },
      "modelProperties": {
        "type": "STATIC_THRESHOLD",
        "threshold": -1,
        "alertOnNoData": true,
        "alertCondition": "BELOW",
        "violatingSamples": 3,
        "samples": 5,
        "dealertingSamples": 5
      },
      "eventTemplate": {
        "title": "Very important metric is missing",
        "description": "The {metricname} value was {alert_condition} normal behavior.",
        "eventType": "AVAILABILITY",
        "davisMerge": true,
        "metadata": []
      },
      "eventEntityDimensionKey": null,
      "legacyId": null
    }
  }
]

[Hasan]

Hi Mark,
Both nodes are sending separate metric key sharing the same below along with screenshots.
Metric in Live server

icardonline.prod.host119.FALCON.55382

Metric in Fallback server

icardonline.prodfb.host118.FALCON.55382

[mark_bley]

Hi @Hasan 

I cannot recommend using dimensions in the metric key, that is what metric dimensions are for.

https://www.dynatrace.com/support/help/shortlink/metric-ingestion-protocol

You would send a metric line like this and both hosts would send this with different dimension values e.g.:

icardonline,stage=prodfb,host=host118.FALCON.5538 <metric data>
icardonline,stage=prod,host=host119.FALCON.55382 <metric data>

If you do not want or cannot use above's format, you still could use a metric query in the metric event adding up both metrics and alerting if data is missing, but again this is not recommended.

BR,

Mark