Custom charting deprecation and Data Explorer gaps

[AntonioSousa]

I started seeing this for the first time today:

I can't believe this for several reasons:

• Data explorer is missing a lot of functionality that Custom charts has.
• Evolution of Data Explorer has been minimal since introduced
• There is no idea when this is going to happen?

There are a lot of RFEs around Data Explorer, and Dynatrace should make sure that Custom Charts are not dropped before Data Explorer is enhanced to support these functionalities. I'll be doing a list of things that don't exist and I believe are critical in Data Explorer, and that exist in Custom Charts.

In the meantime, if someone from Dynatrace could reply to the third point above, I believe we would all appreciate!

[RomanW]

Hi Antonio!

First and foremost, thank you for your feedback!

Our goal with the recent introduction of the Data explorer (by ending its "early adopter" program) was to provide a more powerful query and visualization tool for data in Dynatrace. In the recent months we focused to provide many capabilities that solve long running feature requests for charts on dashboards, which previously translated to "custom charts". For instance, the Data explorer now allows to utilize Metric transformations  and Metric expressions directly in queries/charts on dashboards.

It's correct that there are some capabilities that we still have to add to the Data explorer in order to really cover all that custom charts offered so far. One example is the capability to assign specific colors to series in a chart.

On the other hand, Data explorer already offers vastly improved query capabilities, like filtering metrics on dimension, entity tag and property level. We'll soon start to more actively communicate these particular capabilities to also allow our customer to make use of those new capabilities.

Custom charts not going to be removed entirely for the foreseeable future. This should just communicate that all our efforts go into improving the Data explorer to solve requirements around query and visualization power on dashboards.

With that in mind, could you please provide me with a list of aspects that you'd like to see addressed to fulfill your specific use cases?

Thanks,

Roman

[Julius_Loman]

@RomanW Some feedback from me:

• looks like the Split By was even removed from Data Explorer if you access it directly in the menu, see how it looks like in 1.220. I hope this is just a bug and gets updated soon.
  
  It works if you access it from the Metrics menu.
• The warning sign for old widgets should not be displayed (yet) until the Data Explorer includes all the features. It really causes confusion for customers. No one wants to see warning signs on their dashboards. Please add a transition period (3-6 months) and then start displaying warning signs.
• It's a huge effort to migrate widgets from custom chart to Data Explorer. You might consider a migration path (semi-automatic?)
• Data explorer does not display correct timestamps for graphs. Please look at ticket 5334 for details.
• From the features assigning colours for individual time-series is really a must-have one 

[AntonioSousa]

@RomanW 

We are preparing the list but what is greatly frustrating is that in the new menu, which has so many useless/repeated items, Dynatrace has managed to take out "Custom charts"!!! You have to go a great length to get back to it! This is definitely not the way to do things!

[PedroDeodato]

These are the list of aspects that should be addressed to fulfill the specific use cases of our clients. These are all real cases of usage of Custom Charts in existing dashboards:

• It only allows for color choosing based on a palette
  • In most of my dashboards, I use color coding!
  • So not being able to choose a specific color for each line also renders most of my tiles useless

• Not supporting graph exporting to images, CSV & XLS 
• Not possible to custom define the axis, with the units/min/max values
• Not possible to remove the legend
• Losing axis labels on smaller tiles

• For the same tile size, Custom Charts shows labels, Data Explorer does not 
• Not possible to define custom percentiles for certain metrics in the UI, despite the fact that you can “code” it
• A CPU chart where the 100% line is in the middle of the graph, and the scale goes up to 200%.
  • RFE: https://community.dynatrace.com/t5/Dynatrace-product-ideas/limit-Y-axis-on-graph/idi-p/169873
  • 

And I must say that I do not think that Data Explorer is bad, I like and use many features of it, in other use cases! I just think that it is still a lot underdeveloped, when compared to basic (and needed) features on custom charts... It should fulfill the most important features provided by Custom Charts!

I could also talk about the major headache that will be switching all the tiles by hand, one by one, on each dashboard, one by one, on each client, one by one... talk about overtime!

[joshua_dtrace]

@Julius_Loman  The "Split By" and other features right now are hidden on the "+" icon on the right-side of the metric row on the Data explorer. I believe there are planned improvements to make that callout larger / more intuitive in an upcoming version of Dynatrace.

[Julius_Loman]

@joshua_dtrace thanks! Still, I consider this confusing. This version should have at least some "what's changed" notice when the user first accessed the changed Data Explorer.

[rastislav_danis]

I wonder, how will Sergio recreate this nice graph in Data Explorer (from Why Devs Love Dynatrace tenant):

[sean_mcconville]

Hello,
The value of metrics reported in the data-explorer seems to give a rounded value:

When compared with the same metric shown in a custom chart:

[sean_mcconville]

The chart migration tool does not seem to work when you have a filter selected for a specific device.

I presume this is because the data-explorer filter allows to filter by tag/name/etc.

It seems the migration tool could be improved by migrating current "Custom device tags" to "custom device:name tags" rather than just skipping the tag (or at least could give a warning that migration was not fully successful) 

Custom chart:

Data-explorer

Currently this tag gets totally stripped and causes a lot of confusion for users when the data values change. To resolve it must be manually re-added to the data-explorer.

Since it is the same tag it would be nice if the migration tool could handle it. 

[sean_mcconville]

It would be nice to see the series stacked consistently.
Sometimes we see them side by side (within the same minute), other times they are stacked.

Ideally it would be best if it was possible to set a custom resolution/rollup for the datapoints

[larry_roberts]

Kinda a big change that we did not see coming. Communication on this type of stuff needs too greatly improve. 

[AntonioSousa]

Every time I go to Custom Charts, the following is appearing:

I have given feedback, but for some reason it doesn't care! We shouldn't be annoyed by a message we also know is not true at the moment. I will happily move when the critical existing features in Custom Charts, that have been documented in this thread, are implemented in Data Explorer.

[hmdai]

I have encountered issue when I convert our existing charts from custom chart to Data Explorer. In custom chart we can use multiple tags as filtering options. However, in Data Explorer, only the first tags is taken into consideration and the second one will got mess up if I press "Run Query"

I have raised a ticket to Dynatrace and the support replies that the default behavior is using "or" instead of "and" for filtering criteria. That is the reason why multiple tags cannot be applied. The only way to include multiple tags as filter is using code mode. This seems to be in contrast to easy-to-use direction of Data Explorer.

As mentioned by @PedroDeodato, we cannot customize the coloring of the chart. The default color palette  is too purplerish and not looking good under dark background.

Another issue is the tile does not automatically scaled. The following is the comparison, the left hand side chart is created by Data Exploere while the right hand side is created by custom chart. The percentage figure can be displayed normally under 8x8 size. However, if I shrink it to 6x6. The figures from Data Explorer disappears while custom chart is unaffected.

Data Explorer is a great tool for chart creating. However, it is not mature enough to replace the custom charts.

[larry_roberts]

I at last converted every single custom chart on every dashboard to the new Data Explorer. Only had issues with 2 of them that would not convert for some reason, so I just deleted them and created new ones. 

That was a bit painful going through each and every one of them to convert in terms of time. Hopefully they come up with a cleaner process in the future if you need to mass update like this.

I will say though, I am very happy to at long last be able to hover the mouse of charts on a dashboard and see metric data without the need to click into the chart! HUGE PLUS! 

[AlanZ]

Hi all,

We were looking at how to "convert" all of our dashboards so we do not loose charts in our dashboards. Doing this manually seems to work fine but since we have hundreds of dashboards and potentially thousands of charts to convert, we need an automated way to do this.

@Anonymous: any suggestion ?

[larry_roberts]

I did create an RFE for this very thing the other day. Just an FYI...

https://community.dynatrace.com/t5/Dynatrace-product-ideas/RFE-Add-ability-to-migrate-all-custom-cha...

I feel your pain 🙂

[zietho]

Dear all,

first off thanks to @AntonioSousa , @larry_roberts , @hmdai , @sean_mcconville , and @AlanZ  for providing all the valuable feedback and acknowledging a lot we also receive internally.

And sorry for our late second reply!

Let's me try to answer the two most important questions in the room...

1. What are you going to add before a deprecation?

- configurable whether you want to connect null values aka stepped charts (out with 1.230)

- coloring per metric time series (1.233) 

- ability to pick a desired resolution (1.234)

- stacked column charts (1.235 - see later update in thread for screenshots)

- stacked area charts (1.236)

- being able to use absolute values (1.238)

- export per tile to CSV (planned 1.240) 

- decimals, absolute values (planned 1.240/1.241)

- configurable units (planned 1.241/1.242)  

When I last checked that should cover most if not all of the feedback provided here so far. 

I will update this list along the lines.

2. What about the migration? 

I for sure acknowledge the need for this. But to be honest, we spent the time until now gathering feedback, and refining the list I provided above + starting to work on the first items. We accepted the related product idea, which means we will provide an improved migration path! 

I will come back to you in Q4 and to provide you with details about what such improvements may look like and what we can ship in a reasonable time so that you can make use of it.

Stay tuned and please reach out to me/us should there be more feedback that we need to consider before a deprecation.

regards

Thomas

[AntonioSousa]

@zietho 

I greatly appreciate all your effort! I'll also be checking if the suggestions we have advanced, are covered, but I believe so!

@PedroDeodatohad already noticed today that you have changed the alert message that comes with the tiles in the dashboards. Much better than the one that was there before 😉

[zietho]

Hey, 

with 1.229 I hope we got the single tile migration path (almost) right. Feedback appreciated!

Hope you like it :). 

regards

Thomas

[ct_27]

@AntonioSousa Would it be a good ideas to rename this thread?  IT appears to have gotten taken over as the place to go for Data Explorer updates.  It's pretty much become the Data Explorer Feedback forum.

[zietho]

A bit of sneak peek into 1.235 seeing the power of:

* resolution

* color override

* stacked columns

* metric grouping for columns and :timeshift to compare this week (first metric/column per time bucket) to last week (second metric / column per time bucket)

We hope you like it! 

Next up getting better than custom charting topics: 

* stacked area

* export 

* filter improvements

[Kenny_Gillette]

Great stuff.  What is estimate in timeline for export?

[nikkip]

Hi @zietho,

Just wondering, do you know if there is a way of getting the actual count in Multi-Dimensional analysis rather than a rounded number?

[AntonioSousa · Accepted solution]

@nikkip,

Yes it has been announced, and I believe it will be out in 1.238:
https://community.dynatrace.com/t5/Dynatrace-product-ideas/Unit-Configuration-on-Data-Explorer/idi-p...

[zietho]

Exactly, thx @AntonioSousa 

* controlling decimals will follow around 1.240/1.241 and

* picking units will follow around 1.241/1.242

regards

[zietho]

@AntonioSousa is it ok if I rename the thread to e.g., "Custom charting deprecation and Data explorer gaps"

[AntonioSousa]

Done. I think it reveals the present situation of the ticket 🙂

[zietho]

Dear all, since it has been awfully silent, let me spoiler the related part what my beloved LPE @EnriqueOriol and I gonna present in tomorrow's observability clinic so that the update reaches most of you. 

Yes, I am sorry for being still a bit vague but let's first make sure all of you can successfully migrate and then officially RIP/EOL Custom charting. I hope you all agree! 

kind regards

Thomas 

[AntonioSousa]

@zietho,

I can't believe that the last functionality that I'm missing is coming already in 1.239. I hope it includes all those available in Custom Charts 😃

BTW, is there an easy way to track which dashboards use Custom Charts? I suppose that looking at the JSON files is probably the fastest way?

[zietho]

Regarding Export: Yes, you will all burn me but speed > perfection. For a start all except table, single value, honeycombs. However, we will certainly iterate and first add table because feedback already showed that it is important. Another potential iteration could be an alternative format, the one from the API but only if feedback shows that its needed. Otherwise we rather invest in more needed things such as heatmaps, more visualization context (such as baselines, problems, events...) or interaction with charts (improved drill downs that pertain more filters, consistency, zoom, etc...). I hope you agree. 

Regarding migration: love the question, great point! Thx for raising that! In the simplest case whenever you have an  upgrade button in the menu but maybe we can do some more sugar there. We haven't yet started, so let's see ;). 

[AntonioSousa]

@zietho,

Regarding export, we have one environment where usage of SVG export is heavily used. That's the most critical functionality lacking on our side.

Regarding migration, we've got a lot of them to track down. Automating that would be great!

Additionally, just checked in a Managed environment, and charts don't have the yellow triangle in Dashboards. Checked in version 1.236.

[fstekelenburg]

Working heavily with Data explorer and API to get overview results of (external) synthetic request got me to this topic. By what I can see a lot of issues has been addressed, yet I think there are still some wishes and issues that remain unsolved.
Wondering about the current Data Explorer Gaps and Issues - status July 2022, I actually found this topic that might be a better place to address these further : Data explorer + dashboarding 🗣 feedback 🛋 lounge - Dynatrace Community

(so mods might close this one?)

[zietho]

Hey @fstekelenburg, yes the mentioned channel makes more sense.

However, I owe you and all others an update regarding the planned deprecation and the next steps.

I already promised this end of June / beginning of July back then. Since the planned migration capabilities aren't yet in place (they slipped) I have to move the update to August. Then I can officially provide you with a final migration path and dates and timelines.

Sorry for the delay but please stay tuned!

kind regards

Thomas