Pinging 100 K devices?

[AntonioSousa]

I have a challenge to ping 100K devices with Dynatrace.

I'm expecting to hit several Dynatrace limits...

Has anyone gone so far? Any ideas on how to tackle this efficiently?

[sujit_k_singh]

Hi @AntonioSousa 

100K devices is extreme scale for Dynatrace ICMP monitoring. You'll hit limits on synthetic monitors (default 10K), DEM unit consumption, and ActiveGate capacity.

My approach would be  

• Extension Framework 2.0 — Build a custom ICMP extension that pings devices in bulk from ActiveGate. Avoids synthetic monitor limits entirely, no DEM unit consumption, and is far more scalable. You'd need multiple ActiveGates distributed to handle the load.
• Metrics API v2 ingest — I'd write a lightweight script (Python) that handles the bulk ICMP pings externally and pushes the results into Dynatrace via Metrics API v2. You still get everything in Dynatrace — dashboards, alerting, without hitting synthetic limits or burning DEM units. 

Extension 2.0 or Metrics API v2 ingest is probably the cleanest path for this scale — all within Dynatrace, no DEM cost.

Thanks,

Sujit

[AntonioSousa]

@sujit_k_singh ,

NAM currently has a 5000 monitor limit per environment. Not an option. BTW, NAM consumes DDUs, not DEMs.

Yes, I'm going the extension way, as I have developed several ping extensions over time (eg. Ping Every Second).

The first limitation that I'm considering is really anomaly detection around metric events, and the associated throttling. I'm considering launching the events directly from the extension 😊

Pretty sure there are other limits waiting me along the way...

[sujit_k_singh]

Hi @AntonioSousa 

Thanks for the correction on NAM/DDUs — my mistake.

Good to hear you're going the extension route — makes sense given your experience with Ping Every Second.

Curious how you'll handle the AG distribution/batching — that's the key architectural decision here.

[AntonPineiro]

Hi,

I do not have that use case in mind, but it would be a challenge.

Looks like Prometheus + black exporter or Zabbix + Zabbix proxy are more capable for this. I am curious to know your results.

Best regards

[AntonioSousa]

@AntonPineiro @sujit_k_singh 

Just to satisfy some curiosity, what we are managing to obtain at the moment:

• 120 pings per milissecond, that's 120K pings/sec
• Agregated processing of all this data
• CPU usage, in a 4 core Xeon Gold 6448Y VM, 10Gbit/s:
  System: 11% 
  User: 1%
• Memory usage
  VmSize: 27960 kB
  VmHWM: 8272 kB
  VmRSS: 8272 kB
  RssAnon: 6060 kB
  RssFile: 2212 kB

[sujit_k_singh]

Hi @AntonioSousa 

Thanks for circling back with actual numbers and honestly the results are better than I expected. 120K pings/sec at 11% system CPU on a 4-core box is a strong signal that the raw-socket extension approach was the right call over NAM/synthetic for this scale.

A few things I'd love to understand better now that you have real data:

• What poll interval are you running per device at 100K scale, is 120K pings/sec your sustained steady-state rate, or a burst capacity number?
• Are you doing anything special for send batching (e.g. sendmmsg) to get CPU that low, or is this just an efficient single-socket event loop?
• How are you handling ingestion/cardinality on the Dynatrace side per-device metrics, or aggregated by subnet/site before it lands?

This is shaping up to be a great reference architecture for anyone hitting the NAM/synthetic ceiling.

Thanks,

Sujit

[AntonioSousa]

@sujit_k_singh ,

sendmmsg of course, this run with 64 packets for each kernel invocation, as I'm more interested in availability than RTT. It does introduce a 0.5 ms RTT penalty, as I'm getting 7 μs per packet. At the moment, I'm doing one ping per minute, but finishing under one second, before timeout/retransmission. Not yet at 100K IPs, but close. 
So, to be clear, I'm doing 64 pings/milissecond, and extrapolating results, both in terms of ping capacity and resource usage. These 64 pings complete clearly under half a milissecond. 
Going to run it with 500K Internet IP addresses. At the moment, I believe there are kernel issues (routing, security, etc...) that are limiting me, as I cannot get more than 60 Mbps, in a 10Gbps interface, so clearly a lot to improve!

Regarding Dynatrace ingestion, I do have some issues. First, it's Managed. And second, I have a lot of fails. Network related. More than 3%. So, at the moment, aggregating at 15 minutes, and analyzing events at 6 hours. But still too much issues. Just to give you an idea, p50 RTT is at 800 ms. Lot's of issues forward!

[sujit_k_singh]

Hi @AntonioSousa 

Understood, thanks for sharing insights. 

Thanks,

Sujit