22 Jun 2020 07:29 AM - last edited on 17 Mar 2023 07:16 AM by Ana_Kuzmenchuk
is there any insight available on Dynatrace brought from Azure Monitor related to Azure Databricks and Data Factory? Do you have any experience getting this info from Azure?
Thanks in advance.
Solved! Go to Solution.
Not yet, however support for Azure Data Factories should be available very soon. In fact this service is under active development right now.
Hi Karolina R. What is the status of support for Azure Databricks? Does Dynatrace plan to monitor Azure Databricks?
Azure Data Factory monitoring is available as part of the Azure Monitor integration:
Hi @Siavash H.., thank you for your reply. But what I need to know is if Dynatrace has planned to monitor Azure Databricks. And in that case, when would it be available and what can we expect from such monitoring (what aspects of Databricks would it monitor?).
Thank you, good day!.
I am unable to find anything on the roadmap for Azure Databricks.
However, I had a look on the Azure documentation and it mentioned the ability to send Azure Databricks application logs using Log4j. In the new Log Monitor, Dynatrace offers generic log ingestion and log4j2 can be easily integrated directly to stream log directly from Databricks to Dynatrace.
I hope this helps. If you're looking for another ability to monitor Databricks I recommend to raise a product idea when the new Community is launched.
Interesting, @Siavash H. Thanks a lot.
Can you share with me documentation about the new Log Monitor? I'm really interested in this new functionality.
Nice weekend, regards!
Thanks, and you too Luis!
With regard to the new Log Monitor capabilities I think we'll announce something very soon.
Any updates as to the new log monitor capabilities for Azure Databricks?
@apasoquen1 @CMolina as far as I know, Azure Databricks as a supporting service is still not on the roadmap. However, when it comes to log monitoring, our recent solution should support it, please check: https://www.dynatrace.com/support/help/how-to-use-dynatrace/infrastructure-monitoring/cloud-platform...
Many Thanks wojciech_grajew
Please any technical document about Dynatrace capacities on Databricks monitoring (currently) will very useful.
On the other hand Any updates about Dynatrace roadmap?
Unfortunately right now I don't have any more documentation to share except for the link above. Regarding the roadmap, this service is not on it yet.
Many Thanks wojciech_grajew for your feedback.
Please one additional question: my understanting is that Apache Spark platforms are supported by Dynatrace... so my question here are
- Dynatrace capacities for monitoring Apache Spark platforms are equivalents and available for Databricks or not ?
- Which are the monitoring capacities to be deployed by Dynatrace in order to consider a plataform (Databrick for instance) as "supporting service" ?
I'm not an expert on Azure Databricks unfortunately. From what I see, Dynatrace supports monitoring Apache Spark through OneAgent:
As Databricks is a cloud service, I'd expect it to be a managed offering where you don't have access to the underlying instances where you could install OneAgent. You'd need to research it though.
Regarding the second question -- I'm not sure I understand. Supporting services is the name we give to the cloud services that Dynatrace can monitor through their public cloud APIs (making requests to Azure Resource Graph and Azure Monitor for example). They are monitored from ActiveGate and don't require OneAgent installation. Right now we plan to implement more supporting services but I don't think Databricks has high priority in this queue:
You can monitor the Azure Databricks cluster nodes using OneAgent.
Navigate to the “Download Dynatrace OneAgent for Linux” screen in your Dynatrace environment and copy the 2 commands highlighted below to create a OneAgent install script.
Paste the commands into a file to save for later, let’s refer to it as installdynatraceagent.sh. Cut out the 2 places where the agent version is specified so the latest version will always be used. Your script should look like this:
wget -O Dynatrace-OneAgent-Linux.sh "https://<domain>/e/xxxxxxx/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default" --header="Authorization: Api-Token xxxxxxxx" /bin/sh Dynatrace-OneAgent-Linux.sh --set-infra-only=true --set-app-log-content-access=true --set-host-group=<host_group>
Navigate to your Azure Databricks workspace and go to the Admin Console.
Select Global Init Scripts and click add.
Paste the name installdynatraceagent.sh and script we created earlier, select enabled and click add.
Now your clusters will automatically install OneAgent when they are created, for the OneAgent to be installed on clusters already running they will need to be restarted for the script to run.
Done! You now should be able to see your Databricks cluster nodes as shown below in the deployment status.
This is an awesome walkthrough for the OneAgent deployment to Databricks! We can also get deeper metric insight with various splittings via the Ganglia Extension as well for enhanced Datbricks AIOps from alerts and dashboard reports. The extension and instruction can be accessed from the in-product Dynatrace Hub or from the Public Dynatrace Hub. https://www.dynatrace.com/hub/detail/databricks/?query=databricks
I have followed the instructions for Databricks extension v1.0.4 and I am not seeing any Ganglia metrics either in dashboard or in Metrics page
All steps completed from the Details page, correct?
Yes , the extension is configured in Dynatrace and GlobalInitscripts also updated in Databricks as per instructions. The cluster is visible in Dynatrace with OneAgent installed but no metrics for ganglia showing up