cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Server cache limit has-been exceeded

jose_baeza
Participant

Good afternoon

I have the following warning:
Server cache limit has-been exceeded.
There are 50000 Servers in the cache limit is September to 50000 (50000).


This happened after adding static url with url parameter part groups

and this is not plotting any data

Central Analysis Server
version : 12.1.2.56

Any suggestions to solve

thanks for your suggestions

33 REPLIES 33

ulf_thorn222
Inactive

 

Hola Jose

The limit you hit is set for performance reasons. Depending on your hardware, you might have some capacity left and you can tweak this. But the best way to do this is to go through you configuration and validate that you don't have things monitored/configured that you don't explicitly need.

You should also check your "Server IP Range". This is done by going to the advanced properties and changing those  HTTP://YOURSERVER.COM/ATSCON and selecting "Advanced Properties Editor" and then specifying the "Accepted Server IP address range".

If this doesn't help and you DO have extra capacity in your hardware then you go to HTTP://YOURSERVER.COM/ATSCON and select "Capacity Settings Editor" and slightly increase the server settings.

If you add to much, you run the risk of slowing down the server database to much by having more data than it can manage.

jose_baeza
Participant

Hola  Ulf

 Thank you for your answer

pawel_brzoska
Inactive

Server limit also includes distinct urls, so that explains why it happened after reconfiguration of url parameter. Most likely there are too many values of this parameter resulting in thousands different urls populating the database. 50k is a lot, so i dont advise increasing this limit, rather take a look at how dynamic is the parameter you configured and tweak its definition to result in more aggressive aggregation of parameter values to single operations.

brian_chandler
Dynatrace Organizer
Dynatrace Organizer

I also seem to be running into a repeated issue of exceeding the server cache limit. Today I just exceeded 30000 and had to bump it to 40000... The weird thing is that this is a very small RUM environment. It is just monitoring a single cluster of an SAP application. I would say there are no more than 15 servers this AMD is receiving traffic on (12.1, soon to be 12.2).

Could the SAP GUI traffic it is analyzing at the moment really be creating this many unique URLs and causing it to continue to hit this limit? Any guidance would be great... Thanks.

Please go to http://<CAS>/modulestatus?advanced=1 page of your CAS and copy/paste here rows with Module name column set to "Advanced DB Statistics".

Hi Adam, thanks for the reply. Is this what you were referring to?

  Advanced DB StatisticsRaw sessions246298
  Advanced DB StatisticsManual sites
0
  Advanced DB StatisticsUsers
3829
  Advanced DB StatisticsFile defined sites
56
  Advanced DB StatisticsApplications
1751
  Advanced DB StatisticsCIDR sites
0
  Advanced DB StatisticsAS sites
0
  Advanced DB StatisticsPredefined sites
1
  Advanced DB StatisticsServers
24
  Advanced DB StatisticsURLs on servers
30070
  Advanced DB StatisticsServices (incl URLs)
31247
  Advanced DB StatisticsUnique URLs
20451
  Advanced DB StatisticsServices (excl URLs)
49
  Advanced DB StatisticsAggregated Sessions (AI)
26
  Advanced DB StatisticsAggregated Sessions (AS+AL)
31284
  Advanced DB StatisticsServer samples
11524189
  Advanced DB StatisticsNetwork samples
11524189
  Advanced DB StatisticsServer aggregated samples
1223015
  Advanced DB StatisticsUsage samples
0
  Advanced DB StatisticsNetwork aggregated samples
1223015
  Advanced DB StatisticsLast sample retrieved
5/13/14 14:05

Yes, it looks like the number of sessions is made by operations. Can you share your applications.xml? Also please run the following SQL script that will show what Software Services, Servers, URLs and Sites have the most sessions occupied. Result should be similar to attached image.

Hi Adam, the SQL script appears no longer to be available. Is it possible to make it available permanently?

Can you see it when you go on this very page to Tools -> Attachments

or:

https://community.compuwareapm.com/community/pages/viewpageattachments.action?pageId=158632223

I may be overlooking it, but I do not see Tools as an option. Maybe you have more rights?
But the link/url you posted works, so thanks! (I got a Not Permitted on the original link)

dennis_mispelba
Inactive

Hi,

reecently I get these errors quite often as well. I told the customer that we would need to go through the software service and check if we can get rid of any configured regular expression URL monitors. I was told they would need all the URLs which are configured there. Customer wanted to know what the impact of this warning/error is (Server cache limit exceeded). What is the impact? Performance data is lost? Would be great to get some insight in what the result of the error is... I couldnt find anything in the documentation.

Best regards,
Dennis

Which DCRUM version are you running ? We got those errors at a time when we used 12.1.2 (I guess this was12.1.2.56). After the error was triggered no data - at least for our defined software services that use the HTTP Analyzer - was processed.

We got a patch for 12.1.2 and the advertising that the patch will be included in 12.1.3 (APMODCRUM-12273). Never checked it because meanwhile we migrated to 12.2.1 where the fix is included for sure.

 

 

dennis_mispelba
Inactive

Hi Ralph-Peter,

Thanks for your answer, interesting point. I am facing it at a customer having version 12.1.1.73 (CAS).
I still have data shown in the start page of the CAS. However, I don't know if any data might be lost.

Best regards,
Dennis

Unfortunately I can't remember how the start page looked in our environment. We regularly don't use the start page and instead assign custom made reports to the CAS users as personal starting pages. As far as I can remember no Real User monitoring data was processed. In any case 12.1.1.73 (that should be DCRUM 12.1 + SP1) is a version prior to the version we ran into the problem with.Therefore there is a chance that the error is there, too.
The patch I got from Compuware to solve the issue APMODCRUM-12273 was named "cas_is_not_saving_data_when_server_cache_limit_exceeds.jar". I assume you should have a look if maybe this patch is installed in your customer's environment but contact Compuware anyway to clarify if you suffer from data loss.

Krzysztof_Ziemi
Dynatrace Pro
Dynatrace Pro

When server cache limit is exceeded, several things happen:

  • no new servers and URLs will be added to the CAS database - so it is possible that data loss will occur (you won't see new URLs or servers that have been configured for monitoring)
  • existing URLs and servers that CAS already knows are processed as usual (measurement data for those is recoded by CAS)

When a new data file (5-min zdata package) with new servers/URLs is processed by the CAS and cache limit will be exceeded in the middle of it - servers are processed first, then URLs are processed, so chances are that servers will be recorded and some (randomly chosen) URLs may be ignored.

Summarizing, if you have an existing server already known to CAS and you expanded number of URLs monitored to the extent that cache has been exceeded - you should still be getting complete measurements for the server level and complete measurements for some URLs (those that have been known to the CAS already plus, perhaps, some of the newly added URLs). Remaining URLs data will be rolled up into All Other Operations.

Please note that in the upcoming 12.3 DC RUM release we redesigned the way how auto-learning of servers is performed on CAS, so you will always be able to discover all software services on the network and report on them on the software service level plus aggregate of servers to a location level ("servers from location abc"), and then drill down to the list of serves that comprise the server location.

So what you are saying is that in the case of "Server cache limit ,,,," the total bytes for a server will be correctly recorded but the details for all URL's might be skewed?

So then the data for the URL's that can''t be added in detail will be collected under "All other" ?

Krzysztof, thank you for the insight.

Best regards,
Dennis

erik_pepermans
Participant

So I ran into the same issue. I tried to configure monitored url's by rewriting the url as an 'URL as regular expression'. 

Can I 'clear' the 'server-cache' somehow ? Since it is a quite fresh installation, I could restart capturing with a 'fresh' cache ?

Disscussed with Support I got the response that since we are running with the default value for the server cache, it could be possible to increase this if we had enough resources available, which looking at the system, they say we don't.
"The CAS on it's own should have 32GB RAM, this system only have 16GB with only 7GB assigned to the CAS"

So if the CAS was given more resources we could easily up the server cache limit, but that as it appears our only option is probably going to be to reduce the amount of data in the CAS by means of aggregation perhaps.

Pawel Brzoska, above stated that server limit also includes distinct urls. So is this part here overlooked?

 

Hi,

 

We have a customer that now hits the 50k Server Cache limit.  The "Advanced DB Statistics" are below.

Obviously it's the URLs on servers, and unique URLs that are filling mostly the case.
In use are HTTP, SSL but also SAP decodes. In what way can we quickly get insight in which software services are generating the most URLs in use. And what the URLs are?

Raw sessions1718390
Applications143
Users41234
File defined sites0
AS sites445
Manual sites131
CIDR sites0
Predefined sites1
Services (incl URLs)28472
Servers350
Services (excl URLs)173
URLs on servers49650
Unique URLs36272
Aggregated Sessions (AI)111
Aggregated Sessions (AS+AL)30389
Server samples27779417
Server aggregated samples2995216
Network samples27779417
Usage samples1867619
Network aggregated samples2995216

I ran that script, indeed. Pointed me somewhat in the right direction. But to determine what URLs actually are filling up one needs to dive deeper in the results. What is the definition of sessions? It is not the URL entries in the Cache right? As I see it those are te end user sessions connecting to the software server/URL.

It is with a query like for instance below that gives a certain insight regarding the task generating a lot of URLs:

  select task,count(*) as tel from (
     select url,alias,task,service from dbo.rtmurl
     where url like 'https://portal.customer.com%'
  ) as sub
  GROUP BY task
  ORDER BY tel desc

 

The output of your script:

 

Software Service    Number of sessions
Portal_Web466052
Sharepoint_LB315613
SAP-XYZ_AS_SAPGUI268109
SERVICES_Web250746
TLT_LB162408
SKY_LB110713
AMGR_LB95796
NOD345_LB80711
Train_LB

59350

TLT_FMX51482


,Query result:

Operation nameNumber of sessions
 541839
All other operations208781
S000 123430550
http://intranet.customer.com/pages/default.aspx29010
http://intranet.customer.com/_vti_bin/search.asmx  27926
http://intranet.customer.com/27384
SESSION_MANAGER SESSION 27005
https://plz.customer.com/nlpp/idkk/sso20181
http://search.customer.com/pages/peopleresults.aspx14454
https://portal.customer.com/irj/portalapps/com.sap.portal.pagebuilder/html/emptydocument.html11668

<...>

What would here be the next approach?

 

 

Frans,

Indeed this script is not answer which URLs are filling server cache (that is the most often reason) but only a direction which Software Services have the most sessions (that is usually related to number of URLs) so we know which one should be investigated in DMI by making very simple report with Software Service and Operation dimensions and Operations metric.

As an alternative you can remove top directive from SQL that queries for URLs.

Both here and in "DMI approach" you will be looking for unique URLs that has very small (usually just 1) operation/session. This used to happen when monitored application uses non standard cut separators, incorrect URL monitoring regexes that introduces unique/temporary/1-operation URLs.

And next step is fixing the configuration or if configuration is OK and it's just a matter of number of non-unique operations increasing server cache limit if CAS health permits.

Thank you, Adam. This now has become a good reference in the approach to attack the Server Cache Limit exceeded phenomenon.
Indeed with the simple DMI the URLs can further be investigated.

In our case we needed to fix an application with the right regexp that was generating 30K URLs, because an unique identifier was enclosed in the URL.

One question remains, that I think I have seen asked on the forum somewhere else. Can we somehow delete the server cache entries?

Only in brutal or patient way (smile)

Brutal: purge you DB

Patient: wait for AMD_STORAGE_PERIOD to remove it day by day. Or rather night by night (smile)

sylvian_lam1
Organizer

Hi Gents,

Just read this thread, am running 12.3.2 with same issues. Server cache keep increasing during the new year. From the SQL console result --I found most of sessions belongs to "Other TCP Proto" and "Other UDP Proto" while the largest contributor of "Operation name" is <empty>, yes, it consumes over 1 million session but the column name show nothing. Is there a way to decrease the server cache? Ideas are welcome. Many thanks.







































Advanced DB StatisticsRaw sessions
Advanced DB StatisticsManual sites25
Advanced DB StatisticsAS sites0
Advanced DB StatisticsFile defined sites0
Advanced DB StatisticsCIDR sites43544
Advanced DB StatisticsPredefined sites1
Advanced DB StatisticsUsers6416
Advanced DB StatisticsApplications3380
Advanced DB StatisticsServers338879
Advanced DB StatisticsURLs on servers403325
Advanced DB StatisticsUnique URLs31541
Advanced DB StatisticsServices (incl URLs)65867
Advanced DB StatisticsServices (excl URLs)26897
Advanced DB StatisticsAggregated Sessions (AS+AL)1484995
Advanced DB StatisticsAggregated Sessions (AI)133075
Advanced DB StatisticsServer samples41635511
Advanced DB StatisticsNetwork samples54095004
Advanced DB StatisticsServer aggregated samples42795493
Advanced DB StatisticsNetwork aggregated samples59443316
Advanced DB StatisticsLast sample retrieved1/19/16 11:35
Advanced DB StatisticsUsage samples77911072

Sylvian,

The solution is only one: SP7!

SP3 and SP6 contain fixes that will help you.

Thanks Adam. I browsed thru the release note but no idea which item resolve my issue, could you show me which fix related to my issues? Thanks a lot.

genesius_jarom1
Organizer

Referencing this post is some good background info on sever cache limit exceeded.

https://answers.dynatrace.com/questions/124034/ser...

However, I do have a question about this posting

https://answers.dynatrace.com/content/idea/129080/...

now that I have this information, what do I do with it? How do I use it to resolve the cache limit.

I apologize for asking a question that is probably pretty obvious to most here.

Thanks and God bless,

Genesius

Genesius,

The question is very good, as I'm not sure if there is any topic speaking about what to do next ...

I would encourage to create new topic like "Server cache limit - how to resolve?" so we coudl group all answers there and give it a better visibility than page X in this thread ...

Karolina_Linda
Community Team
Community Team

Hi All,

if you see the "Server cache limit has been exceeded" warning, please check the following solutions:

Server cache limit exceeded - solutions checklist

If none of them resolves the problem, you should contact our support team.

Regards,

Karolina

Keep calm and build Community!

@Karolina Linda

Thank you. I have been searching for a procedural list. While I have not had the opportunity to review it, I hope it will help when this error returns, which it will.

Thanks and God bless,

Genesius