DT is failing to start. We have tried several time and getting the below error while try to start.
SEVERE [ApplicationSession] Could not start continuous session recording for profile XXX: com.dynatrace.diagnostics.server.datacenter.shared.ApplicationSession startContinuousRecording:3519
java.io.FileNotFoundException: /data/sessions/stored/xxxx/20180117173412_0.t1.session/dcrum.dtsf (Too many open files
dynaTrace Server NOT started successfully: com.dynatrace.diagnostics.server.ServerBundleActivator start:634
folk we need urgent help...
Note that for urgent assistance a support ticket is recommended.
The issue looks to be that the user running the server has too many files open and is reaching the ulimit which is preventing the server from starting up. An immediate fix should be to increase the ulimit on your server but you may also want to investigate why that limit was reached and if it is the case of the ulimit being too low or some activity on the server such as excessive report generation is causing to many files to remain open. To drill into that you may want to reach out to support.
cat /proc/<appmon server pid>/limits
Should show you the current limits. Normally 4096 seems to be a good ulimit but in servers with large amounts of system profiles 8192 can sometimes be necesary.