Apache Hadoop FastPack
FastPack Contents dynaTrace 5.5
Fastpack Download for dynaTrace 5.5 contains:
Fastpack Download for dynaTrace 5.6 or higher additionally contains support for:
If you have any questions, please don't hesitate to ask in our Plugins and Extension Forum.
The MapReduce Dashboard gives an overview of the currently running Jobs and how much the Hadoop Environment is utilized. In addition it shows the specific running and completed jobs (lower pane). This dashboard relies on the Job Monitor.
The HDFS (NameNode) Statistics Dashboard gives an overview over the most important HDFS Nodes. It shows the summary as seen by the name node. In addition the user can put the metrics for specific data nodes on separate dashboards if that is desired.
The Hadoop Cluster Overview shows a consolidated view of the cluster resource usage over time. It gives a sense of how much CPU is consumed and the load average of the whole cluster. It does the same for Disk and Network I/I as well as memory. It serves the same purpose as Ganglia dashboards do.
dynaTrace 5.5. has Hadoop MapReduce support built in. This fastpack contains three additional Dashboards, a Job Monitor and two Metric Groups for the Name Node and Data Node respectively. dynaTrace can monitor any JMX attribute available withing the Hadoop nodes in addition to those supplied in this fastpack.
You should first inject Agents into your Hadoop environment and create a new system profile. See the documentation on how to do this.
Next just download and import the FastPack on your Dynatrace Server (see Plugin Management).
As a last step setup the Job Monitor, give it the name "Apache Hadoop Job Monitor" to make all dashboards work out of the box.
Feel free to contribute any changes on Github
This is really cool. Nice job Mike!
Am trying to install this at a PoC. They don't have TaskTracker, but do have SecondaryNameNode and Balancer.
Any comments on how this affects using the dashboards or the sensors? I'm not clear on how to name these tiers.
the main benefit of dynaTrace is to analyze the MapReduce Jobs, without a Task Tracker there would be no MapReduce Job? Without this the Dashboards would only show HDFS statistics.
The problem we were having is that none of the names of their systems sounded like the Task Tracker. They were using Cloudera and not very familiar with the setup.
I also wasn't clear on the role that the other systems play - SecondaryNameNode, Balancer, etc..
Even Cloudera has all those things.
Best is to use the https://apmcommunity.compuware.com/community/display/DOCDT55/Hadoop+MapReduce proposed changes this all works in cloudera as well. Its now documented really well and should work in cloudera.
just add those things to the env and mapred fine and you should be fine.
Does this FastPack include the Hadoop sensor pack as this is no longer provided by default within AppMon?
I have just tested it and it looks like the answer is no.As such I have copied the information from the previous one and created a sensor pack:
Thanks for providing this. I have added the sensor pack to the download section.
Does AppMon 6.5 still work with Hadoop if we manually add the sensor pack? I'm trying to get visibility to my team's Hadoop's cluster with 6.5 (only available download)
Hi Jeff. I think I just answered this question via email you sent me. Here is my response: You should still be able to do this by injecting a Java Agent into your Hadoop Nodes. The reason why we official stopped pushing for Hadoop is because we havent seen that many of our users actively using Dynatrace AppMon to monitor their Hadoop Clusters. Give it a try though as I think you will get a lot of good insight