cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Appmon Server not starting up : osgi writable issue

himanshu_mor
Advisor

HI Guys,

my appmon server is not starting up and giving below errors for the backend and frontend servers



Backend server error:

CompilerOracle: exclude java/io/ObjectInputStream.readO*

2019-08-12 06:04:30 UTC [31f208c5] info [java ] <title>Invalid Configuration Location</title>The configuration area at '/apps/dynatrace-7.1/server/osgi' is not writ

able. Please choose a writable location using the '-configuration' command line option.

2019-08-12 06:04:39 UTC [33437885] info [native] Process 14233 terminated, exitcode: 0

2019-08-12 06:04:39 UTC [33437885] info [native] Dynatrace Server terminated



frontend server error

<title>Invalid Configuration Location</title>The configuration area at '/apps/dynatrace-7.1/server/osgi.frontend' is

not writable. Please choose a writable location using the '-configuration' command line option.

2019-08-12 06:09:27 UTC [da5738f9] info [native] Process 14802 terminated, exitcode: 0

2019-08-12 06:09:27 UTC [da5738f9] info [native] Dynatrace Frontend Server terminated


Note that the server was working fine . i just did a restart after editing user.permissions.xml and it stopped working


Thanks

Himanshu Mor


12 REPLIES 12

It looks like issue with permissions to this file. Go to that location and check if you didn't change owner of user.permissions.xml. I suppose that you've edited it as root user, but appmon had it's own user.

Sebastian


Hey @sebastian k. , Thanks for he response!

But as i remember , the owner of user.permissions.xml file was root and not the specific user we are using to run dynatrace.

Just to check , i changes the user and group ownership of user.permissions.xml to DT user but no luck


Thanks

Himanshu Mor


do you have backup of this file from time before changes?

Sebastian


Yes i restored it from the backup !


also i saw that the ownership of directories dt_home/server/osgi.frontend and dt_home/server/osgi was root.

I changed the ownership to the dedicated DT user for these two osgi directories(as it was mentioend in logs that this directory is not writable)


After this , the error has changed . My backend server is restarting continuously


2019-08-12 09:50:20 UTC [87ff88b7] info [java ] ... last message repeated 2 times ...

2019-08-12 09:50:20 UTC [87ff88b7] info [java ] ### Excluding compile: java.io.ObjectInputStream::readObject0

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] Process 20320 terminated, exitcode: 102

2019-08-12 09:50:21 UTC [8da7e8f7] severe [native] Process terminated unexpectedly!

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] Restarting Dynatrace Server...

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] 1 update(s) found in /apps/dynatrace-7.1/server

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] update /apps/dynatrace-7.1/server/updates/update_7.1.19.1020 Version 7.1.19.1020 is ACTIVE

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] newest active update is 7.1.19.1020

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] merging entries from ini file /apps/dynatrace-7.1/server/updates/update_7.1.19.1020/dtserver.update.ini

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] Number of max open files to set to 8192

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] Effective memory model: unsupported

2019-08-12 09:50:21 UTC [8da7e8f7] info [native] Effective sizing model: demo

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Launcher configuration summary for Dynatrace Server:

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Launcher: /apps/dynatrace-7.1/dtserver

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Java VM: /apps/dynatrace-7.1/jre/bin/java

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Java version: 1.8.0_162 64-Bit

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Startup jar: /apps/dynatrace-7.1/server/plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Working directory: /apps/dynatrace-7.1/server

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Configuration file: /apps/dynatrace-7.1/dtserver.ini

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Pid file: /apps/dynatrace-7.1/server/dtserver71.pid

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Native watchdog enabled, port: 50000, timeout: 300 sec (timeout for graceful shutdown: 300 sec)

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Max open files: 8192 (minimum required: 8192)

2019-08-12 09:50:22 UTC [8da7e8f7] info [native] Process created successfully, pid: 20434

2019-08-12 09:50:33 UTC [877f98b7] info [java ] ServerHelperAgent loaded

2019-08-12 09:50:33 UTC [877f98b7] info [java ] CompilerOracle: exclude java/io/ObjectInputStream.readO*

2019-08-12 09:50:36 UTC [877f98b7] info [java ]

2019-08-12 09:50:47 UTC [877f98b7] info [java ] ... last message repeated 2 times ...

2019-08-12 09:50:47 UTC [877f98b7] info [java ] ### Excluding compile: java.io.ObjectInputStream::readObject0

2019-08-12 09:50:49 UTC [8da7e8f7] info [native] Process 20434 terminated, exitcode: 102

2019-08-12 09:50:49 UTC [8da7e8f7] severe [native] Process terminated unexpectedly!

2019-08-12 09:50:49 UTC [8da7e8f7] severe [native] Restart counter exceeded the maximum of 3 restarts in sequence (time delay between restarts less than 60 seconds)

2019-08-12 09:50:49 UTC [8da7e8f7] info [native] Dynatrace Server terminated



Max open files limit is exhausted. You have to extend it.

Sebastian


DT doesn't have permissions to write according to log you have. As soon as you will restore proper permissions (CHMOD/CHOWN) to "/apps/dynatrace-7.1/server/osgi.frontend" it should start working again.

Sebastian


permissions.JPG

i have given the permissions. . now the frontend server starts , but backend server restarts 3 times and then terminates



I've made another comment about limit of open files. Your server reaches it and restarts. You have to increase it


Hi @sebastian k. , i have made the changes to ulimit and ulimit -Hnis giving me 12288 now.

But in logs i still see the old ulimit. do i also need to specify ulimit in appmon config somewhere?

2019-08-12 10:50:52 UTC [443b98fd] severe [native] Process terminated unexpectedly!

2019-08-12 10:50:52 UTC [443b98fd] info [native] Restarting Dynatrace Server...

2019-08-12 10:50:52 UTC [443b98fd] info [native] 1 update(s) found in /apps/dynatrace-7.1/server

2019-08-12 10:50:52 UTC [443b98fd] info [native] update /apps/dynatrace-7.1/server/updates/update_7.1.19.1020 Version 7.1.19.1020 is ACTIVE

2019-08-12 10:50:52 UTC [443b98fd] info [native] newest active update is 7.1.19.1020

2019-08-12 10:50:52 UTC [443b98fd] info [native] merging entries from ini file /apps/dynatrace-7.1/server/updates/update_7.1.19.1020/dtserver.update.ini

2019-08-12 10:50:52 UTC [443b98fd] info [native] Number of max open files to set to 8192

2019-08-12 10:50:52 UTC [443b98fd] info [native] Effective memory model: unsupported

2019-08-12 10:50:52 UTC [443b98fd] info [native] Effective sizing model: demo

2019-08-12 10:50:53 UTC [443b98fd] info [native] Launcher configuration summary for Dynatrace Server:

2019-08-12 10:50:53 UTC [443b98fd] info [native] Launcher: /apps/dynatrace-7.1/dtserver

2019-08-12 10:50:53 UTC [443b98fd] info [native] Java VM: /apps/dynatrace-7.1/jre/bin/java

2019-08-12 10:50:53 UTC [443b98fd] info [native] Java version: 1.8.0_162 64-Bit

2019-08-12 10:50:53 UTC [443b98fd] info [native] Startup jar: /apps/dynatrace-7.1/server/plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar

2019-08-12 10:50:53 UTC [443b98fd] info [native] Working directory: /apps/dynatrace-7.1/server

2019-08-12 10:50:53 UTC [443b98fd] info [native] Configuration file: /apps/dynatrace-7.1/dtserver.ini



I suggest openning support ticket. Paste your logs there. I didn't have such issue before. You can always try using migration tool for extracting configuration, reinstalling server and restoring it via the same tool. It may be quicker than looking for issues in current installation.


Thanks @sebastian k. for your help!

issue was resolved as there were problems with DT_RUNAS account.