Icon

Comments have been closed on this page. Please use AppMon & UEM Plugins forum for questions about this plugin.

As of version 5.5 Dynatrace AppMon has built-in support for Hadoop MapReduce. This FastPack provides additional dashboards and a Job Monitor for further monitoring and analysis.

Fast Pack Details

Name

Apache Hadoop FastPack

Version

5.5.0.5225, 5.6.0.5802

dynaTrace Version

>= 5.5

Author

Michael Kopp

License

dynaTrace BSD

Support

Not Supported

FastPack Contents dynaTrace 5.5

Sensorpack Download

Fastpack Download for dynaTrace 5.5 contains:

  • MapReduce Jobs Dashboard
  • Hadoop HDFS Statistics Dashboard
  • Hadoop Cluster Overview Dashboard
  • Two Hadoop Metric Groups for the Data and Name Nodes
  • Job Monitor that works for all major Hadoop Versions
  • Sensor and Profile are built-in with dynaTrace 5.5

Fastpack Download for dynaTrace 5.6 or higher additionally contains support for:

  • Amazon EMR 1.0.3
  • Cloudera 4.6.0 (MR1)
  • Cloudera 4.3.0 (MR1 + MR2)
  • Cloudera 4.0.0 (MR1)

If you have any questions, please don't hesitate to ask in our Plugins and Extension Forum.

Hadoop MapReduce Jobs

The MapReduce Dashboard gives an overview of the currently running Jobs and how much the Hadoop Environment is utilized. In addition it shows the specific running and completed jobs (lower pane). This dashboard relies on the Job Monitor.

HDFS (NameNode) Statistics

The HDFS (NameNode) Statistics Dashboard gives an overview over the most important HDFS Nodes. It shows the summary as seen by the name node. In addition the user can put the metrics for specific data nodes on separate dashboards if that is desired.

Hadoop Cluster Overview

The Hadoop Cluster Overview shows a consolidated view of the cluster resource usage over time. It gives a sense of how much CPU is consumed and the load average of the whole cluster. It does the same for Disk and Network I/I as well as memory. It serves the same purpose as Ganglia dashboards do.

FastPack Information

dynaTrace 5.5. has Hadoop MapReduce support built in. This fastpack contains three additional Dashboards, a Job Monitor and two Metric Groups for the Name Node and Data Node respectively. dynaTrace can monitor any JMX attribute available withing the Hadoop nodes in addition to those supplied in this fastpack.

Installation

You should first inject Agents into your Hadoop environment and create a new system profile. See the documentation on how to do this.

Next just download and import the FastPack on your Dynatrace Server (see Plugin Management).

As a last step setup the Job Monitor, give it the name "Apache Hadoop Job Monitor" to make all dashboards work out of the box.

Contribution

Feel free to contribute any changes on Github

  1. Anonymous (login to see details)

    This is really cool. Nice job Mike!

  2. Anonymous (login to see details)

    Am trying to install this at a PoC. They don't have TaskTracker, but do have SecondaryNameNode and Balancer.

    Any comments on how this affects using the dashboards or the sensors? I'm not clear on how to name these tiers.

    1. Anonymous (login to see details)

      Eric,

      the main benefit of dynaTrace is to analyze the MapReduce Jobs, without a Task Tracker there would be no MapReduce Job? Without this the Dashboards would only show HDFS statistics.

      1. Anonymous (login to see details)

        The problem we were having is that none of the names of their systems sounded like the Task Tracker. They were using Cloudera and not very familiar with the setup.

        I also wasn't clear on the role that the other systems play - SecondaryNameNode, Balancer, etc..

        1. Anonymous (login to see details)

          Even Cloudera has all those things.

          Best is to use the https://apmcommunity.compuware.com/community/display/DOCDT55/Hadoop+MapReduce proposed changes this all works in cloudera as well. Its now documented really well and should work in cloudera.

          just add those things to the env and mapred fine and you should be fine.

  3. Anonymous (login to see details)

    Does this FastPack include the Hadoop sensor pack as this is no longer provided by default within AppMon?

    1. Anonymous (login to see details)

      I have just tested it and it looks like the answer is no.
      As such I have copied the information from the previous one and created a sensor pack:

      com.dynatrace.Big_Data.HadoopNew.dtcs

      1. Anonymous (login to see details)

        Thanks for providing this. I have added the sensor pack to the download section.

        Ingo

  4. Anonymous (login to see details)

    Does AppMon 6.5 still work with Hadoop if we manually add the sensor pack?  I'm trying to get visibility to my team's Hadoop's cluster with 6.5 (only available download)

    1. Anonymous (login to see details)

      Hi Jeff. I think I just answered this question via email you sent me. Here is my response: You should still be able to do this by injecting a Java Agent into your Hadoop Nodes. The reason why we official stopped pushing for Hadoop is because we havent seen that many of our users actively using Dynatrace AppMon to monitor their Hadoop Clusters. Give it a try though as I think you will get a lot of good insight