Hi,
Since Dynatrace can monitor ADF metrics, apart from the metrics listed in the document (Azure Data Factory (V1, V2) monitoring — Dynatrace Docs) Can it also monitor the below-described use case?
Use case: The client is using ADF as an ETL to schedule some jobs in DB. The pain point is the long-running pipelines and it’s execution cycle. Even if the current scheduled pipeline is running state the next scheduled pipe line gets triggered and started running lead to overlapping of pipelines, jobs and data inconsistency. Can DT monitoring help here?
For Example : It should notify that the next pipeline from ADF has been stared running despite of the previous one is still in running mode so that the team can get into it for checking the reasons for long running pipe lines.
Thanks,
Chandni
Hey @Chandnipatel!
Sorry for the delay. Our Community gets lots of activity every day, and unfortunately, sometimes, some posts don’t get as much attention as they deserve. Did you find an answer to your question, or would you like me to seek further assistance for you?
Hello!
I have the exact use case with @Chandnipatel.
Is it something at least on the roadmap?
Thank you.
Hello,
We run ADF pipelines from our Autosys scheduler. Our Autosys logic (box jobs, dependencies, etc) would cover the use cases noted above. Furthermore, we use Autosys maxrun alarms for notifications of pipeline runs that may be problematic.
How about using AZURE logs ingestion. If we enable diagnostics on the DF and ingest logs through application insights can we get some value there? For example start and finish time of individual activities.
BR
same pipeline triggered by multiple trigger at same time , if we few succeeded and one failed , is there a way to differentiate