Take the Tracing Fan Challenge! ‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌💚‌

[MichalGebacki]

On Valentine’s Day one year ago, we asked you to share the Dynatrace features you love, and received so much astonishing feedback in response! This time, we’re returning to explore Dynatrace user stories from a bit different perspective!

Share your unique use cases for Dynatrace’s Trace Intelligence tools, such as TraceView, ResponseTimeAnalysis, MethodHotspots, MemoryAllocations, ServiceFlow, and FailureAnalysis. What do you like the most? Did it help you to find a critical bug in your software? Did you get information about something critical? We’d like to hear it!

And this month, we’ve joined forces with @josef_schiessl and his team, preparing something special for you! Every participant will receive the usual 100 bonus points and a unique “Tracer” forum badge, but, additionally, the top three use cases (shared before March 10th) of our development team’s choice will get 1 x $75, and 2 x $50 gift codes to utilize in the Dynatrace Swag Store!

Looking forward to your great use cases!

[josef_schiessl]

Hello everyone! :fist_bump:
As an initiator of this challenge, I want to actually also contribute / kick it off now :slightly_smiling_face:

A cool thing about developing our tracing tools is that we can actually use them ourselves!
One of my favourite tools is the memory analysis (heres a recent very interesting blog post about it).

Whenever we touch any code that is highly critical from a performance standpoint, we always make sure to compare the before and after using memory analysis.
We use the “method” filter and put the class name or package where the changed happened to only analyze the memory consumption of our change. Tip: The “method filter” can filter for packages, classnames and even methods.
Sometimes very small changes in the code / adaptions can lead to lots of allocated heap / performance loss. On the other hand sometimes refactorings might not have any impact on heap because you never know what the JVM does behind the scenes when it executes the code.

Its awesome to compare the raw numbers but also see how the flamegraph changes.

Looking forward to hearing about your usecases / tips and tricks!

Cheers,
Josef

[dannemca]

In my case was a new feature introduced few weeks ago in Synthetics, where we can now compare, side by side, the execution results of 2 different locations or status, and point out their differences easily.

This feature helped me to find out the reason of too many inconsistent execution result from different monitors to an external application not managed by the client, where his app is hosted.

Checking the executions result, I could point to a bad IP from the hosting DNS Load Balancer. We have then opened a ticket to hosting service, which confirmed the IP issues after our investigation and finally have fixed their services.

That’s pure AIOps, where AI stands for Actionable Insights.

That’s really useful data, meant to help and guide troubleshooting.

[AntonioSousa]

All of the tools are great! So many I could say, but just the first that came to mind:

I have a client that has machines where you can pay and get a receipt.  Lately clients were complaining it was slower. Quickly, traces showed a big 5 second gap, each! Very rapidly Method Hotspots revealed the culprit: a sleep(5) left in the code, in a recent deployment! Immediate fix and immediate improvement of response time!

[ChadTurner]

So I have two unique use cases of Dynatrace and the traces and Service flows. THe first one has been turned into a blog post so you might have read it: Staying ahead of your competition with Dynatrace Synthetics

Using Synthetics to test your competitors.  At a previous employer we started to synthetically test our direct competitors. We were a insurance company that wrote homes, renters and auto. We tested a director competitor to record and track the customer experience to see if we provide a faster and better quality experience/service at login. Now if we failed to beat the competitor, we then had that service flow and tons of transaction data surrounding that test which we could adopt at the organization.

Using backtracing Via SmartScape to validate Application ownership.  It saddens me to say this but man many times when we onboarded a new set of hosts the application owners/requestors had no idea what their front facing app was, let alone their URL. If a user did have that information it also tended to be incorrect as the URL had changed or they were just totally wrong about it. Management came to us and said we are paying for Dynatrace, YOU need to tell me what their app URL is because we cant rely on the customer knowing it. That night when i should have been sleeping i racked my brain over and over again. How can we work it backwards from the URL to the Hosts, and that’s when it hit me. Use all the metric data that Dynatrace has, all the relationships and work it back via Smartscape. So I went to the catch all - My Web App, and started to break out these domains as their own app. I prefixed the URL with my name so I could have a bunch broken out but filter on my name “chad - <URL>”. Once it was broken out as its own Application we let it have some time to collect user sessions, then we went to smartscape and looked back to the Services level, from there we could see the metadata and tags in which were populated from host down to the processes down to the services. Not only did this allow us to be proactive and self reliant, but it also gave us 100% certainty that this App belongs to this host/HostGroup/Management Zone. Management was very please and strengthened our tooling at the organization as Dynatrace was a single point of truth by laying the direct relationships based off the actual communication and not something that a human made in Visio.

[Kenny_Gillette]

My story:

Our team was brought in to troubleshoot high CPU on a server.  Dynatrace was immediately blamed for high CPU.  Don’t we always get blamed?  Many were on the call trying to figure out with no luck.

I told the team to let me look.  Within a few minutes I figured out the issue.  What did I do?

I went to offending process then went to “View detailed CPU breakdown”

Next, I look at highest CPU usage and went to Method HotSpots

Then went to Hotspots and found that a logging application was causing high cpu, not Dynatrace.  Go figure.

It is fun to show the power of Dynatrace.

[Ana.Kuzmenchuk]

We love to see Dynatrace shine a light on the true root cause! Thanks for sharing!

[MichalGebacki]

What a solid piece of a complex user journey, sounds like a great source of inspiration!

[mizso74]

Hi Folks,

I really love use Dynatrace for dig into the traces. My recent story with traces was a rescue of a project (watermelone project - three weeks before the release was green, two weeks before the release was yellow, one week before the release was full red. :wink:). So one week before the release the BAs and developers found me becasue there were huge performance issues (huge response times) at a new green field kubernetes app. Quickly I checked a random individual trace between two meetings:

I immadiately checked the method hotspots and in the call hierarchy the bcrypto indetified as a source of the huge response time at this single request:

Then I asked the projects to generate load on the services and my first observation was perfect (Dynatrace observation). At each services I could see code execution perfromance issues becasue of the bcrypto during the whole load.

After that the bcrypto was replaced by developers there were new perfromance tests. These performance tests were checked quickly in DT again with compare function:

So the project, the release was saved becasue received the green light in time thanks to DT (and to me ).

Best regards,

Mizső

[MichalGebacki]

Impressive workaround, thank you @Mizső for so much details!

[Shirai]

Hi everyone.

I like Dynatrace’s Backtrace feature. Requests to the DB often result in Response time hotspots. The Backtrace feature allows you to trace back from slow SQL statements all the way back to Useractions. Application development members and DB administrators will share this information and proceed with troubleshooting.

Toshimasa Shirai.

[MichalGebacki]

Great choice to highlight! Big thanks Toshimasa for your reply, it’s truly enriching this thread!

[Kenny_Gillette]

Yes, love how Dynatrace proved it wasn’t Dynatrace and it was another app.

[Ana.Kuzmenchuk]

A watermelon project? Love it! Thank you for taking time to share it with us in such a detail!

[PedroDeodato]

This one’s made for me: I’m a true Tracing Fan!

As far as Observability Pillars go, Tracing is absolutely the best one ❤️

I got to know Tracing long before I knew Dynatrace: from my academic times, when I was doing my Master’s Thesis on OpenTelemetry

Back then, all of Tracing was manual for me: instrument application code to create trace, setup context, create root span, create child spans, end spans, end trace, send tracing info to X, …

When I first saw Dynatrace, and saw Automatic Tracing with so much detail, with no effort at all… I mean, like magic

I think this generational leap really makes you treasure and appreciate the massive load of work that Dynatrace gets out of your way!

On the other hand, beginning with an open-source and manual approach also brought me the base knowledge to leverage Dynatrace’s tracing capabilities at its fullest: it helped me to avoid being overwhelmed with so many great information, as I already knew what I was looking for

Speaking of leveraging Dynatrace’s tracing capabilites (there are so many!), a funny story that I remember was of a client who was complaining of Service A when it invoked Service B, regardless of Service B showing pretty fast response times.

When we looked at the trace examples, we immediately saw what was going on: indeed Service B had fast Server-Side Response Time, but - from Service A perspective - there was a VERY long Client-Side Response Time!

Unfortunately, I do not have screenshot on me of what exactly happened at the time, but I found something similar on another post:

What was great is that no significant analysis was needed to point us on the right way: we just saw the huge difference between the whole span bar (light blue + dark blue) and only the Server-Side response time (only dark blue), because of the great Client-Side Response time (light blue) overhead.

This was only possible, at least in such a simple way, because of Tracing: instead of just having the Response Time from Service A and Service B, we got traces from Service A that showed us when the request was sent out, when did it reach Service B, when did Service B sent out the response and when did that response reached Service A again!

Knowing this, we immediately suspected of a network issue and, after some investigation we were right: a request from Service A to Service B would travel through a hugely inneficient route, instead of going straight to the final destination, which lead to a huge network time overhead.

This was why Client-Side Response Time (Server-Side + Network Time), as seen from Service A, was a lot greater than the Server-Side Response Time reported by Service B itself: Service B handled the request and sent out a response very quickly, it was the travel time between the two that was causing impact.

Think about the amount of data you would have to go through to figure this one out without Tracing!

[andreas_grabner]

What a story. While its great to see Dynatrace can capture this I would advocate for also using static code scanning tools in CI to detect artificial sleeps

[andreas_grabner]

Great story and a great reminder that there is a lot going on when a request leaves one service until it makes it to the other service.

What I have seen in my life of analyzing PurePaths is that - while the network is most likely to blame in the scenario above - it might also be other components that are not part of the trace, e.g: serializing or deserializing the data package (also called marshalling). Or - there might be component such as queues between A and B which end up delaying message delivery. This is why its always important to ask about which components you have in a distributed systems and how can you monitor them. For network or other components such as queues you can look at key metrics such as throughput, package loss or queue length

Keep up the great work and please keep sharing those stories

[andreas_grabner]

Wow. I have seen plugins like this in the past that either consume all the CPU for doing tasks that might not be necessary - OR - that are simply configured wrong and can only handle a small amount of concurrent requests. Thanks for sharing. I am curious though → why was this library added in the first place? Was it by accident? or was there a real reason for adding this crypto hashing algorithm?

[andreas_grabner]

Hey Kenny. Great to see those examples - well - great might be the wrong word as in 2023 I havent expect seeing logging as the main performance issues in apps. Just curious. Was this due to a bad logging framework? Or did somebody turn on debug logging under heavy load?

[andreas_grabner]

I remember writing that blog post with you