Hello again Abhishek,
You can use as much memory as you like as long as your leave space for the OS and some extra space for headroom. I'd see how much your OS is taking when it's not running the CAS, add a couple GB for extra space, and then give the CAS the rest.
Ex: 16 GB machine - Maybe 2-3GB OS, 2GB overhead, 12 GB CAS
Ex: 40 GB machine - Maybe 2-3GB OS, 2GB overhead, 36 GB CAS
Should one be careful about over-allocating memory? If I have 128 GB in the system I probably do not want to allocate too much either. The CAS is a JVM - aren't there limitations to how much memory a JVM should use to run efficiently, or is the sky the limit?
John, In my experience, there's a no-mans land around 32GB you want to avoid using.
JVM starts handling memory differently at 32GB - it has to start using larger pointers - which degrades performance.
I find setting -Xmx to 32000M is the sweet spot (it's a little bit smaller than the full 32GB) if you have about that much memory to allocate (after assigning some to the OS/SQL/overhead etc.).
If however you have heaps more say 64+ to allocate, go for it. The (significantly) extra memory exceeds the performance loss from the heavier pointers.
If you've got say 32-40GB to allocate, keep it to 32000M, it's not worth the overhead for only a few GB extra memory.
As covered, ensure there's some left over the the OS, SQL (and set SQL's limit too), and remember to add 2GB each for RUMC and CSS if they are also running on the same server.
You've gotten some good recommendations from some field experts already but thought I'd add the rule of thumb I use in addition to the recommendations.
1) Always reserve 4GB for the Operating System
2) Always reserve 4GB for RUM Console (if installed)
3) Balance the remaining RAM 50/50 between the DB and CAS JVM
4) After a few days check the memory allocation, Observe the max amount of memory the JVM uses.. if it is not even close to using the max adjust it so you have roughly a 20% ceiling and allocate the rest to the DB which will almost always consume the max you give to it eventually.
In my experience I haven't seen a need to go over the recommended 32GB of RAM for the CAS JVM. What normally happens is clients forget to reserve the 4GB for the OS or they let SQL run wild and consume all the memory. Both of these things will degrade performance.