I am unable to find anything on the roadmap for Azure Databricks.
However, I had a look on the Azure documentation and it mentioned the ability to send Azure Databricks application logs using Log4j. In the new Log Monitor, Dynatrace offers generic log ingestion and log4j2 can be easily integrated directly to stream log directly from Databricks to Dynatrace.
I hope this helps. If you're looking for another ability to monitor Databricks I recommend to raise a product idea when the new Community is launched.
@apasoquen1 @CMolina as far as I know, Azure Databricks as a supporting service is still not on the roadmap. However, when it comes to log monitoring, our recent solution should support it, please check: https://www.dynatrace.com/support/help/how-to-use-dynatrace/infrastructure-monitoring/cloud-platform...
Many Thanks wojciech_grajew
Please any technical document about Dynatrace capacities on Databricks monitoring (currently) will very useful.
On the other hand Any updates about Dynatrace roadmap?
Many Thanks wojciech_grajew for your feedback.
Please one additional question: my understanting is that Apache Spark platforms are supported by Dynatrace... so my question here are
- Dynatrace capacities for monitoring Apache Spark platforms are equivalents and available for Databricks or not ?
- Which are the monitoring capacities to be deployed by Dynatrace in order to consider a plataform (Databrick for instance) as "supporting service" ?
I'm not an expert on Azure Databricks unfortunately. From what I see, Dynatrace supports monitoring Apache Spark through OneAgent:
As Databricks is a cloud service, I'd expect it to be a managed offering where you don't have access to the underlying instances where you could install OneAgent. You'd need to research it though.
Regarding the second question -- I'm not sure I understand. Supporting services is the name we give to the cloud services that Dynatrace can monitor through their public cloud APIs (making requests to Azure Resource Graph and Azure Monitor for example). They are monitored from ActiveGate and don't require OneAgent installation. Right now we plan to implement more supporting services but I don't think Databricks has high priority in this queue:
You can monitor the Azure Databricks cluster nodes using OneAgent.
Navigate to the “Download Dynatrace OneAgent for Linux” screen in your Dynatrace environment and copy the 2 commands highlighted below to create a OneAgent install script.
Paste the commands into a file to save for later, let’s refer to it as installdynatraceagent.sh. Cut out the 2 places where the agent version is specified so the latest version will always be used. Your script should look like this:
wget -O Dynatrace-OneAgent-Linux.sh "https://<domain>/e/xxxxxxx/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default" --header="Authorization: Api-Token xxxxxxxx" /bin/sh Dynatrace-OneAgent-Linux.sh --set-infra-only=true --set-app-log-content-access=true --set-host-group=<host_group>
Navigate to your Azure Databricks workspace and go to the Admin Console.
Select Global Init Scripts and click add.
Paste the name installdynatraceagent.sh and script we created earlier, select enabled and click add.
Now your clusters will automatically install OneAgent when they are created, for the OneAgent to be installed on clusters already running they will need to be restarted for the script to run.
Done! You now should be able to see your Databricks cluster nodes as shown below in the deployment status.
This is an awesome walkthrough for the OneAgent deployment to Databricks! We can also get deeper metric insight with various splittings via the Ganglia Extension as well for enhanced Datbricks AIOps from alerts and dashboard reports. The extension and instruction can be accessed from the in-product Dynatrace Hub or from the Public Dynatrace Hub. https://www.dynatrace.com/hub/detail/databricks/?query=databricks