cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

This product reached the end of support date on March 31, 2021.

Monitoring Apache Spark?

paul_brebner
Inactive

Hi, has anyone tried monitoring Apache Spark? (or similar?) How would I set up a personal copy of Dynatrace to monitor it please (just one a single machine initially), and what would it pick up? (i.e. what would the PurePaths looks like I wonder?) Spark appears to support a rich set of programming models and also has a potentially complex/dynamic execution model, so I'm trying to work out if Dynatrace will reveal interesting performance and scalability features (eventually hopefully to populate our predictive modelling tool), Thanks, Paul

8 REPLIES 8

rick_boyd
Mentor

We are moving from MapReduce to Spark and I am hoping for future support consideration for Spark. If you try to work with it using custom instrumentation please post your findings here

Thanks,

Rick B

paul_brebner
Inactive

Hi Rick, have you successfully monitored MapReduce with Dynatrace? Did it show anything interesting? It appears that Spark is java based so DT should work ok, currently just trying to work out where to put the DT java -agentpath string option... Paul

Hi Paul,

Finally revisiting this. Sorry i missed your question. MapReduce went away as of 6.3, and there's no official support for Spark now or announced for the future. Did you have any luck hooking spark? what components did you get instrumented?

Thanks,

Rick B

Hi Rick,

It looks like MapReduce is back in beta form. But I think it's just hitting the YARN resource manager for data instead of instrumenting the running apps. Hard to say though.

steve_swor
Newcomer

There's a beta release of Spark monitoring now, but I haven't tried it yet. The documentation is a bit light but from looking at the screenshots and the description on that blog post, I think it's just grabbing the metrics from the app primary (as opposed to instrumenting running code like the java agent does). It's hard to say for sure, though.

dominik_stadler
Dynatrace Pro
Dynatrace Pro

Please note that the link mentioned is for the Dynatrace SaaS monitoring, not Dynatrace AppMon.

However it should be even easier to try it as you don't need to setup a server locally, but just install the agent on the machines running Apache Spark after you started a free trial at https://www.dynatrace.com/trial/

Dominik.

chris
Newcomer

Has anyone been able to instrument Spark with AppMon? Unfortunately, what I've seen on Dynatrace is too high-level for our needs and it would be fantastic to have a better view into Spark jobs to identify performance bottlenecks!

Joe_Hoffman
Dynatrace Leader
Dynatrace Leader

Christopher, Can you clarify why Dynatrace is too high level? Perhaps you need to drill down into Purepath view? Or define new Services (think: purepath starting points in AppMon Terms) to capture the relevant Spark details?

But to answer your question: I have not instrumented Spark with either, but both products should be able to give you useful information.