Extension for Aviatrix Network Insights

[steventorrestam]

We are using Aviatrix at my company for our complex network cloud monitoring. Aviatrix has a blog regarding using their APIs for ingestion Prometheus JSON into dashboard monitoring tools like Datadog, Grafana etc - https://aviatrix.com/blog/bridging-cloudops-devops-the-key-to-application-network-performance/

Is Dynatrace looking to setup an extension that would allow the ingestion of key metrics using their APIs? 

[AurelienGravier]

Hello @steventorrestam ,

There is no public Dynatrace extension on Dynatrace HUB : https://www.dynatrace.com/hub/?query=aviatrix&filter=all

But you can easily develop your proper extension especially with a prometheus datasource : https://docs.dynatrace.com/docs/shortlink/prometheus-extension

I recommend watching the practical guides to SQL Extensions or SNMP Datasource (closer than prometheus data source): 
https://www.youtube.com/watch?v=t9kHPKL9yKY&t=779s
https://www.youtube.com/watch?v=JVObrl8CaPI

Regards Aurélien.

[mihaime]

Bonjour Aurelien,

I am also with Aviatrix and have a question on this topic...just that unlike Bryan I'm more the reverse engineering all rounder type:)

We expose the metrics in OpenMetrics format.

We can also have a Prometheus installation that imports OpenMetrics from our custom URL.

This raises the next question:)

a) Can I integrate Dynatrace directly with OpenMetrics?

if yes, then do I just write that custom yaml file (plugin definition) and with what metrics it would find? (I looked very superficially at it some minutes ago so excuse any discrepancies, I will read more during my train ride now)

b) Same question for Prometheus

I did find your article that is K8S dedicated and requires also some annotations (not my case)

For a)  metrics are exposed by querying an URL like this:

https://<copilot_public_ip>/metrics-api/v1/gateways?format=json

and the result is in this format

```
memory_cached{gateway="Controller"} 5375148 1680115793000 memory_buf{gateway="Controller"} 332689 1680115793000 memory_swpd{gateway="Controller"} 0 1680115793000 memory_free{gateway="Controller"} 624738 1680115793000 cpu_idle{gateway="Controller"} 98 1680115793000

```

[bashley8]

Hi Steven, my name is Bryan Ashley, and I am the VP of PMM at Aviatrix.  While we are still working through additional integration partners, we are thrilled that you want to leverage Dynatrace to expose this additional telemetry and insights.  We would be happy to work with you and any Dynatrace representatives to see what we could make work for your organization.  Please don't hesitate to reach out to me and I can connect you to some supporting resources who can work directly with you.  My email address is bashley@aviatrix.com. 

[steventorrestam]

Hello Bryan, I will reach out to you via email, as my team internally wants to look into a solution with extensions. 

[rob_jahn]

Hi Bryan -- I am in the Technical Alliance team at Dynatrace and would like to understand more about your solution and will reach out to you via the email you shared above.

[bashley8]

Great, look forward to connecting Rob!

[AurelienGravier · Accepted solution]

Hello @mihaime ,

In my opinion, you have two options:

1. The simplest one, Dynatrace provides out-of-the-box support for ingesting metrics from Prometheus exporters in Kubernetes. In this case, you use Prometheus to expose your metrics in the OpenMetrics format, but for that, you need a K8S cluster, and annotations on the pods are a prerequisite.

2. If you don't have a Kubernetes env, you can create a custom extension V2 using the Prometheus data source, which is native in the Dynatrace extension framework. You can choose to run it on an ActiveGate or a OneAgent :
https://docs.dynatrace.com/docs/shortlink/extensions-concepts
https://docs.dynatrace.com/docs/shortlink/prometheus-extension

Regards Aurélien. 

[mihaime]

Hi Aurelien,

The first option with K8S is no go at the moment for me, a bit too overkill just for this purpose.

Option 2 would it work with OpenMetrics also or do I need to install Prometheus? (I know they are similar)

Prometheus is able to fetch the OpenMetrics from my URL, I tested it in the past and wo issues.

Thank you,
Mihai

[AurelienGravier]

Hi,

The Dynatrace Prometheus datasource may indeed support OpenMetrics, but you'll need to test it to confirm. This compatibility is plausible since Prometheus uses OpenMetrics as its standard for metric exposition, and the Prometheus client library for Python can capture metrics in this format.

If it doesn’t work natively, you could develop a Dynatrace Python extension using the Prometheus client library for Python.

Regards,

[mihaime]

Thank you/Merci beaucoup Aurelien!

Last dumb question, do you have any examples from 0 to hero type on this topic?

I even tried setting up the Extension in VSCode and I got some errors there so anything like a walkthrough with 1-2 examples would be magic and save me from the initial banging my head against the walls experience.

[Mike_L]

This video of the creator of the vscode extension would be my go-to for that: https://www.youtube.com/watch?v=Qy4Ge8HIzEk

[mihaime]

Thanks. Really good video.

Was trying to skip through it last time because of lack of time but after watching it e2e today without any disturbance, I can say it did unblock me and I managed to get everything to work (Except the extension simulator but that is probably because I am on a mac).

[steventorrestam]

Hello @mihaime if you happen to make a successful Aviatrix extension. Do you mind sharing your extension with others? I would love to try it out within my company too. We have it on the internal roadmap for design but other efforts are currently in the works in front of it. 

[mihaime · Accepted solution]

Hi Steve,

Will do once it's ready. In fact I am working for Aviatrix but not in the engineering department or in PS.

I wanted to get this one working for a demo and I did miss looking under the hood of things and trying to figure out how they work.
I just need to be more patient and not think that I can get it in just a few hours while also multi-tasking. My bad.

[Mike_L]

I don’t think we tested it on Mac indeed. 

[mihaime]

I managed to get it to read my metrics so far 🙂 so did not need the simulator anymore.

Now while watching the Dynatrace Extension 2.0 video in repeat I was wondering if I need the Topology or not as I saw I can create Dashboards on my own (but yeah they do not hold any relational logic in them, true).

Do you happen to have a magic link that would also explain how to define the yaml structure for topology?

So far I reached this one while looking up what the properties defined in the video are:

https://docs.dynatrace.com/docs/dynatrace-api/environment-api/settings/schemas/builtin-monitoredenti...

and then googled a bit..but feel there must be an E2E tutorial that goes through everything in a single article somewhere.

Do you happen to have a magic link?

If not, then I manage, no worries. It does not seem to be rocket science 🙂 

I just tried too fast to be from 0 to hero in 1 day.

[Mike_L]

This one explains topology in the best way:

https://youtu.be/tIIVaC9iWPM?si=hPtJk40NIxJQ-M3h It starts at 22:30

[steventorrestam]

@mihaime do you work for @bashley8 ? I know I have reached out to Brad on email and he introduced me to some associates. We should all look to regroup together to get this going. As this would be nice to have at the end a GA release version to share with all of the Dynatrace Community that uses Aviatrix and DT. Today my team is currently ingesting the API info into Splunk for some base dashboard of Aviatrix performance but i would love to walk away from that and do it all in Dynatrace, since that is my company #1 enterprise monitoring tool. 🙂

[mihaime]

No but we know each other pretty well 🙂 I'm far far away..in Europe 🙂 I asked @bashley8 to put me in contact with you and will probably reach out toward Monday.

Still need to do some experiments but I'm slowly getting the hang of Dynatrace. Quite cool technology, can't believe I did not spot it earlier.
What I still miss is an E2E tutorial or a logical schema approach with Steps and where/what fits in the puzzle to get the high level overview and then to have the nitty gritty details and different doc sections for each. 

(I know what you're saying with Splunk and API 🙂 past me played with Splunk and Cisco ACI ingestion of data...though that part was not so well developed or desired by the 2nd vendor)

[mihaime]

@AurelienGravier it did work with OpenMetrics which was good as I did not have to add Prometheus 🙂

I need now to find some spare hours to look through the docs and get an understanding about the various other elements: Topology, Scenes and what is different vs self made Dashboards (which indeed have no parent-child relations and are just showing values).
Any good link up your sleeve that groups all info together and goes through each?

I did find separate articles but at some point ended up with too many tabs open:)

[mihaime]

@steventorrestam 

I made a small article here with just importing Aviatrix Metrics and adding them to a Dashboard:

https://blog.mihai.tech/2024/dynatrace-monitor-your-aviatrix-environment/

Also have to thanks @Radu for having explained 1-2 things to me to get me unstuck.

[steventorrestam]

Hello does the what is your API endpoint, end with its the env /metrics-api or you using any other endpoint to call and get the metadata back. When i am in my VScode and put my endpoint https://abc123.com/metrics-api/v1, i am not getting the response showing it pulled that metadata back for the metric definitions.

[AurelienGravier]

Good Job @mihaime !