Dynatrace Outage - impacting Customer application

[SachinJindal]

We have got Dynatrace classic oneagent deployed on our application stack and we can see latency issues and also getting page distortion and 404 error for beacons. 

This has happened when we had DT global outage ..

Q - How any monitoring tool can slowdown or impact customer application when monitoring tool has any outage/issue..

see RUM errors for reference .. 

[SachinJindal]

Can someone help to understand this?

[AntonioSousa]

@SachinJindal ,
I would suggest that the best way forward is submitting a ticket, so you can get help from Dynatrace in your issue.

[SachinJindal]

@sujit_k_singh  will you be able to help?

[sujit_k_singh]

Hi @SachinJindal  

This is classic question, let me explain in depth "How a Monitoring Agent Can Impact an Application During an Outage"

The honest answer is: it depends on the monitoring architecture. There are two fundamentally different ways a monitoring tool can affect your application, and Dynatrace uses both which is why both symptoms you're seeing (latency + beacon 404s) have different root causes.

Root Cause 1 — In-Process Agent (Latency Impact)

OneAgent works by injecting bytecode instrumentation directly into your application's JVM/process at runtime. This means:

• OneAgent code runs inside your application process — in the same thread, on the same heap
• Every instrumented method call passes through OneAgent hooks before and after execution
• When OneAgent loses connectivity to the Dynatrace cluster, it must decide what to do with buffered telemetry data

As you've already noticed the RUM-related errors, I won't go into further detail on that. Instead, I'll come straight to your issue - RUM JavaScript Beacon (404 + Page Distortion)

Let me break down the actual error step by step.

rb_bf96606mkq - The beacon/RUM JS endpoint path — rb_ prefix = OneAgent-served RUM endpoint, bf96606mkq is your application's monitoring ID
?type=js3 -  The browser is requesting the RUM JavaScript payload (type js3 = the full monitoring script bundle) — this is not a data beacon, it's the script download itself.
sn=v_4_srv_11_sn_51FFE545DD -  Session/server context — srv_11 identifies which OneAgent server node was handling this

and finally 

ruxitagentjs_ICA15789NPQRTUVXf - The specific RUM JS filename — ICA15789NPQRTUVXf is your feature flag/config fingerprint encoding which monitoring features are enabled
404 repeated 3 times - The same request failing three consecutive times the browser is retrying

The critical insight: type=js3 means this is the script download request, not a data beacon. The browser injected <script src="rb_bf96606mkq?type=js3&..."> into your page and that URL is returning 404. This means the RUM JavaScript itself cannot load so no monitoring runs at all, and the failed script request is what's causing your page distortion.

What's Actually Happening

Browser loads your page

• OneAgent injects: <script src="/rb_bf96....?type=js3&sn=v_4_srv_11_sn_51.......">
• Browser fetches that URL from your app server
• OneAgent on your server tries to serve ruxitagentjs_ICA15789...... -> 
  • OneAgent is lost cluster connectivity during outage
  • Returns 404 (cannot serve the JS bundle)
• Browser retries the same URL → 404 again (x3 in your logs)
• RUM JavaScript never loads → no monitoring

Why It Happened During the Outage Specifically
The rb_bf96606mkq?type=js3 endpoint is served dynamically by OneAgent it's not a static file. OneAgent generates the JS bundle on-the-fly based on your application's current monitoring configuration pulled from the cluster. During a cluster outage:

OneAgent on your server → tries to fetch current JS config from Dynatrace cluster
                                           → cluster unreachable
                                           → OneAgent cannot assemble the JS bundle
                                           → returns 404 instead of the script

Summary of Your Specific Situation
Your symptoms map exactly to this: OneAgent auto-injects a synchronous <script> tag pointing to a dynamically-served RUM JS endpoint (rb_bf96606mkq?type=js3). During the cluster outage, OneAgent couldn't assemble that JS bundle and returned 404. The synchronous script tag then blocked or partially disrupted page rendering, causing both the latency and the visual distortion. 

I hope this answered your question and provided the information you were looking for. 😊

Thanks,

Sujit

[SachinJindal]

Many thanks Sujit for step by step explanation .. How this can be avoided when same situation happens? i am fine with not to have monitoring during this time however major concerns will be,  not to have any impact on application .. do you think by changing beacon endpoint from default to cluster activegate or any other way..

[sujit_k_singh]

Hi @SachinJindal 

I would recommend two fixes, Fix 1 ensures your application is never impacted even if monitoring fails, and Fix 2 reduces the likelihood of monitoring failing in the first place.

Fix 1 — Break the Render-Blocking Dependency (Async Loading)
By default, OneAgent injects the RUM script synchronously. You can change this to asynchronous injection, which means a 404 or slow response will not block page rendering.

This is the single most important change. With async, if the RUM JS returns a 404 during a cluster outage, your page will still render normally the browser won't block on the failed script.

Fix 2 — Reduce 404 Probability (Beacon Endpoint) 

You partially answered your own question here, yes, routing through an ActiveGate helps. In "Beacon endpoint section"  Change Type from Default settings to Cluster ActiveGate (or enter your ActiveGate address if using a custom one). This routes the RUM JS bundle request through your ActiveGate, which can serve a cached version during a cluster outage instead of returning 404.

Thanks,

Sujit

[SachinJindal]

Many thanks Sujit .. just one more question what will be the best way to replicate same kind of situation where we can see latency/rendering issue on the pages and than we apply the fix and see the behavior and give more confidence in terms of Prod  ..

[sujit_k_singh]

Hi Sachin,

There are multiple options to replicate the issue like block the beacon endpoint via browser DevTools, you can configure proxy to intercept requests matching through proxy/interceptor tool or temporarily modify hosts file or firewall rule.

I would recommend use emporarily modify hosts file or firewall rule.

• Block the Dynatrace cluster IP at the application server level (not browser level)
• This forces OneAgent itself to lose cluster connectivity, exactly replicating the outage condition
• OneAgent will fail to assemble the JS bundle -> return 404 -> browser impact
  
  

Thanks,

Sujit

[AntonPineiro]

Hi,

Maybe you can configure a custom beacon endpoint.

Best regards