cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Azure - Azure Databricks Data Factory

alberto-jesus_1
Contributor

Hi,


is there any insight available un Dynatrace brought from Azure Monitor related to Azure databricks and Data factory ? Do you have any experience getting this info from Azure?

Thanks in advance.


Alberto.

16 REPLIES 16

Karolina_Ruszko
Inactive

Hi Alberto,

Not yet, however support for Azure Data Factories should be available very soon. In fact this service is under active development right now.

Best,

Karolina

luis_mpa
Helper

Hi Karolina R. What is the status of support for Azure Databricks? Does Dynatrace plan to monitor Azure Databricks?

sia_h
Dynatrace Pro
Dynatrace Pro

Hi Luis,

Azure Data Factory monitoring is available as part of the Azure Monitor integration:

https://www.dynatrace.com/support/help/technology-support/cloud-platforms/microsoft-azure-services/s...

Sia

Hi @Siavash H.., thank you for your reply. But what I need to know is if Dynatrace has planned to monitor Azure Databricks. And in that case, when would it be available and what can we expect from such monitoring (what aspects of Databricks would it monitor?).

Thank you, good day!.

Hi Luis,

I am unable to find anything on the roadmap for Azure Databricks.

However, I had a look on the Azure documentation and it mentioned the ability to send Azure Databricks application logs using Log4j. In the new Log Monitor, Dynatrace offers generic log ingestion and log4j2 can be easily integrated directly to stream log directly from Databricks to Dynatrace.

I hope this helps. If you're looking for another ability to monitor Databricks I recommend to raise a product idea when the new Community is launched.

Sia

Interesting, @Siavash H. Thanks a lot.

Can you share with me documentation about the new Log Monitor? I'm really interested in this new functionality.

Nice weekend, regards!

Thanks, and you too Luis!

With regard to the new Log Monitor capabilities I think we'll announce something very soon.

Sia

Any updates as to the new log monitor capabilities for Azure Databricks?

CMolina
Newcomer

Hello everyone, i need to bring this topic back to life again, ¿any update?

@sia_h 

 

wojciech_grajew
Inactive

@apasoquen1 @CMolina as far as I know, Azure Databricks as a supporting service is still not on the roadmap. However, when it comes to log monitoring, our recent solution should support it, please check: https://www.dynatrace.com/support/help/how-to-use-dynatrace/infrastructure-monitoring/cloud-platform...

Many Thanks wojciech_grajew

 

Please any technical document about Dynatrace capacities on Databricks monitoring (currently) will very useful.

On the other hand Any updates about Dynatrace roadmap?

 

Many Thanks

Damián

 

Dynatracer

Hello Damián,

Unfortunately right now I don't have any more documentation to share except for the link above. Regarding the roadmap, this service is not on it yet.

Many Thanks wojciech_grajew for your feedback.

 

Please one additional question: my understanting is that Apache Spark platforms are supported by Dynatrace... so my question here are

 

- Dynatrace capacities for monitoring Apache Spark platforms are equivalents and available for Databricks or not ?

 

- Which are the monitoring capacities to be deployed by Dynatrace in order to consider a plataform (Databrick for instance) as "supporting service" ?  

 

Kind regards

Damián

 

Hi Damián,

 

I'm not an expert on Azure Databricks unfortunately. From what I see, Dynatrace supports monitoring Apache Spark through OneAgent:
https://www.dynatrace.com/support/help/technology-support/dynatrace-extensions/supported-out-of-the-...

As Databricks is a cloud service, I'd expect it to be a managed offering where you don't have access to the underlying instances where you could install OneAgent. You'd need to research it though.

 

Regarding the second question -- I'm not sure I understand. Supporting services is the name we give to the cloud services that Dynatrace can monitor through their public cloud APIs (making requests to Azure Resource Graph and Azure Monitor for example). They are monitored from ActiveGate and don't require OneAgent installation. Right now we plan to implement more supporting services but I don't think Databricks has high priority in this queue:
https://www.dynatrace.com/support/help/how-to-use-dynatrace/infrastructure-monitoring/cloud-platform... 

joseph_bannert
Dynatrace Enthusiast
Dynatrace Enthusiast

Hi alberto-jesus_1,

 

You can monitor the Azure Databricks cluster nodes using OneAgent.

 

Steps:
Navigate to the “Download Dynatrace OneAgent for Linux” screen in your Dynatrace environment and copy the 2 commands highlighted below to create a OneAgent install script.
step1.png

Paste the commands into a file to save for later, let’s refer to it as installdynatraceagent.sh. Cut out the 2 places where the agent version is specified so the latest version will always be used. Your script should look like this:

 

wget -O Dynatrace-OneAgent-Linux.sh "https://<domain>/e/xxxxxxx/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default" --header="Authorization: Api-Token xxxxxxxx"

 /bin/sh Dynatrace-OneAgent-Linux.sh --set-infra-only=true --set-app-log-content-access=true --set-host-group=<host_group>

 

Navigate to your Azure Databricks workspace and go to the Admin Console.

step2.png

Select Global Init Scripts and click add.

step3.png

Paste the name installdynatraceagent.sh and script we created earlier, select enabled and click add.

step4.png

Now your clusters will automatically install OneAgent when they are created, for the OneAgent to be installed on clusters already running they will need to be restarted for the script to run.

step5.png

Done! You now should be able to see your Databricks cluster nodes as shown below in the deployment status.

step6.png

Advisory notes: 

  • Your script should include a host group for the nodes, then in Dynatrace you need to turn off "Detect host or monitoring connection lost problems" on the host group level, this is to avoid an alert storm caused by frequent node termination.
  • OneAgent should be used in infra only mode to save license because Databricks nodes typically have high RAM and full stack mode does not give much extra value.

Regards, Joe.

This is an awesome walkthrough for the OneAgent deployment to Databricks! We can also get deeper metric insight with various splittings via the Ganglia Extension as well for enhanced Datbricks AIOps from alerts and dashboard reports. The extension and instruction can be accessed from the in-product Dynatrace Hub or from the Public Dynatrace Hub. https://www.dynatrace.com/hub/detail/databricks/?query=databricks

lawrence_cuneaz_1-1658152076843.png