First of all, yes, if the AMD already started to have data feed into it way earlier than CAS (instead of deployed both CAS and AMD at almost the same time) then of course there would be processing delay, this is expected as the CAS has to process data the have already been collected by AMD hours/days earlier.
However, in one of the POC I've done, I try to clear all the data in var/spool/adlex/rtm and restart the AMD, which means there would be nothing (or if there is something somehow, the timestamp/date-modified would be of few minutes ago) after restart. But still, the data processing delay didn't gone or get any shorter. In that POC, I have to tell my client I can only go over there the next day to check on it.
Also, another reason I think this might be a bug/a feature for product enhancement is that, my colleague has even experienced the delay processing of more than 7 days.
Correct me if I am wrong, but AFAIK the AMD can only store around 7 days of data, so a delay processing of >7 days is kind of confusing for me.
Anyone has any idea of what I might've gone wrong about it? Or let's say if the persistence of delay processing even after the data in the directory is purged, anyone has any workaround to at least, shorten the delay processing?
I don't know what the behaviour is for the CAS actually if you delete the contents of /var/spool/adlex/rtm. However you might need to use controlled data processing to tell the CAS to skip certain data files.
If you navigate to your CAS and go to Tools > Advanced > Administration Console.
and then click System Management > Controlled Data Processing
(or head straight to http://[servername]/DiagConsole#/diag/DATA%20PROCESSING%20CONTROL )
From this dialog you can tell the CAS server to give up processing historical data files and to jump forwards. Please be aware however that once you have skipped a data file, you can't go back and re-process it.
The AMD by default stores ten days worth of data, so that could explain your 7 days of processing delay.
If you are having problems with your delay, have a browse to your server.log file and tail it. It might help you determine what is causing the delay. Sometimes we have found in the field that it is waiting on something, certain file types, an AMD that isn't responding, slow download of data files, change in monitoring config, etc.
The processing status report (I think is the name) will also help you look back in time to observe you processing delay so you can help pinpoint when it happened.
If it isn't waiting on anything and it comes down to sheer volume of records to process, you could look to tune your configuration to filter out noise, aggregate unnecessary data, etc. and improve those processing times.
Hope that helps!