cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Spark cluster monitoring

suresh230591
Advisor

We have oneagent installed on our spark cluster but I can't see Spark cluster specific metrics on Dynatrace. Spark plugin under technology is already enabled.

Is there any configuration needs to done separately. kindly advise

Regards
Suresh

4 REPLIES 4

Jakub_Mierzewsk
Inactive

Hi,

https://www.dynatrace.com/news/blog/introducing-sp...

Please check prerequisites:


  • Dynatrace OneAgent version 1.105+
  • Linux OS or Windows
  • Spark Standalone cluster manager
  • Spark version 1.6
  • Enabled JMX monitoring metrics. Turn on JmxSink for all instances by class names: org.apache.spark.metrics.sink.JmxSink conf/metrics.properties For more details, see Spark documentation.
  • To recognize your cluster, SparkSubmit must be executed with the –master parameter and master host address. See example below:

spark-submit \
--class de.codecentric.SparkPi \
--master spark://192.168.33.100:7077 \
--conf spark.eventLog.enabled=true \
/vagrant/jars/spark-pi-example-1.0.jar 100

Is this only work for Spark version 1.6 or Higher version also supported.

suresh230591
Advisor

Thanks Jakub. I will try the way you suggested.

suresh230591
Advisor

Hi Jacub , we executed the above command what it does not work for us. We are using Spark 2.6
If you can provide bit details about following parameter in above command that would be great.

1) class de.codecentric.SparkPi
2) What is the IP address and port number here
3) conf spark.eventLog.enabled=true
4) /vagrant/jars/spark-pi-example-1.0.jar 100