cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

This product reached the end of support date on March 31, 2021.

re-process past zdata files

sylvian_lam1
Organizer

Hi All,

Is it possible to re-process the zdata files which in past time stamp? I disable the data-feed of a AMD (say 11:00am) and enable back after 6 hours (say at 5:00pm), from the CAS report I only see data since 5:00pm, it seems it didn't process the zdata between 11:00am to 5:00pm, I check those zdata files are still on the AMD.

Much appreciate for any comment/suggestion.

Thx

Sylvian

15 REPLIES 15

adam_piotrowicz
Dynatrace Pro
Dynatrace Pro

CAS can process data only if it's newer that latest interval already processed by CAS.

This means that it's not possible to "fill" any "data gaps".

But if you have disabled data processing at 11am and then started it on 5pm, then CAs should start processing 11:05 data, then 11:10, 11:15, etc ...

The only problem could be that if you have skipped 11am-5pm data either in CAS or deleted on the AMD.

Please share output of http://your-CAs/rtmdebug screen.

Erik_Soderquist
Dynatrace Pro
Dynatrace Pro

If the report server is reading from multiple AMDs and you disabled one of them, your only option to process the effectively skipped data files is to shutdown the report server and restore the database from a backup taken before you disabled the datafeed, then let the report server reprocess everything from all AMDs to catch up again.

-- Erik

Hello Erik,

If this is the case so what will happen if we have only CAS
server and that goes down.

According to documents AMD can keep data for the 420 Minutes or
7 Days, so I thought that whenever we will bring back our CAS server before 7
Days the data will be reprocessed again automatically without any gap in the
monitored data. It means my understanding was wrong.

Can you please clear my concept
in the case of failure of reporting server?

Babar

sylvian_lam1
Organizer

Erik, yes I'm using multiple AMD. I thought enable/disable data feed is per AMD so it should process the "time-gap" zdata when data feed is enabled back. In my case, if I don't disable the data feed of problematic AMD, it caused CAS not process the normal AMD's zdata and the user interface will show "delay in performance data processing is x hours .....". If problem happen again, should I delete the problematic AMD from the CAS config instead of disable data feed? Or the product design won't trace back those old zdata even AMD connect back to CAS?

Badly I don't have any DB backup.

Sylvian

Again I ask, how exactly is this AMD "problematic"?

-- Erik

Erik_Soderquist
Dynatrace Pro
Dynatrace Pro

The AMD does buffer the data, but that is only for a CAS down
situation; zdata is a synchronous data type, so whatever zdata files are
available from all data sources when the CAS processes the data are
merged into a single data and then processed by the CAS. Once that data
set is processed, there is no "going back" short of restoring the
database to a backup taken before the data was processed.

Yes,
enable/disable is per data feed, but disabling a data feed excludes
that data from processing entirely. Deleting the data feed would have
the same effect.

How is this AMD "problematic"?
Normally a delay in data processing like you are describing is caused by
overloading the CAS, and you should consider adding CAS hosts as secondary
nodes to better distribute the data load or reconfigure the Software
Services to reduce the data load.

-- Erik

@Erik Soderquist

Do you have anywhere where you can find out about the data flow and processing of incoming data to the AMD and how it traverses the DT components

chris_v
Dynatrace Pro
Dynatrace Pro

If you have an AMD that is important to be in the CAS data set, but is problematic, you can mark the AMD as 'Priority' in the RUM Console (CAS Data Source configuration), the CAS will not move forward with processing until the priority AMD(s) are back, this will result in processing delays (no processing occurs) if a priority AMD is unavailable.

Once the priority AMD is back, the CAS will continue on form where it left off. Leaving you with a complete data set from all AMDs, but with some delays to catch up on.

Hi Chris,

Does this setting make any difference if a CAS is using an AMD and Synthetic data as two data sources? Or is it only between multiple AMDs? For example in a situation where the connection between the AMD and CAS is disrupted, the Synthetic data keeps coming in and the AMD is still monitoring.

@Anton L

I'm not sure on the answer to that. The synthetic data is stored differently to the AMD data in the CAS, but the CAS processes them by time interval in the same way.

It may require you to test such a configuration, unless someone from product management can respond to this question.

sylvian_lam1
Organizer

Thanks Guys your valuable responses. To consolidate the above, it seems the issue I met is related to product design: one processing time stamp for al AMDs.

Actually I have two AMDs and both rtm are running fine, the only "problematic" component is the unstable network connection between the regional AMD. The heartbeat between AMD and CAS lost then lead to both AMD stop processing data (over an hour), thus I disable datafeed of the regional AMD to let the local AMD process data as usual.....so after datafeed enabled back on the regional AMD , CAS only process the time stamp base on local AMD.

Well, if the above theory is true, my next question will be "how to overcome"? What if CAS can separate the processing time stamp of individual AMD? ^^

Sylvian

ulf_thornander3
Inactive

@Sylvian Lam Hypothetically - If you want to have the data from both AMD's ASAP, and you don't want any wait for the other while you have a shaky network, you would need 2 CAS.

If you do like Chris is suggesting above, you'll have delayed processing as soon as you have any network problems, but I still think that is the best approach.

sylvian_lam1
Organizer

Good question, Anton. I'm using CAS to received from ESM, SaaS and AMD.

@Ulf, yes, 2 CAS is good but too costly.....;)

Erik_Soderquist
Dynatrace Pro
Dynatrace Pro

In that scenario your options are

  1. fix the faulty network link.
  2. as Ulf said, two separate CAS hosts so the faulty network link is irrelevant to CAS data processing.
  3. or as Chris said, set the remote AMD as a 'priority' AMD and simply accept processing delays when the network link is failed.

-- Erik

sylvian_lam1
Organizer

Thanks Erik.

Thanks Guys.