START_APP_CMD for Tomcat on Linux App Service

[g_kat]

Hi,

I've been trying to setup OA on a simple Tomcat process with. I have a Linux App service configured for Apache Tomcat 8.5 with Java 8, and I have created a startup scrip based on the documentation

https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-cloud-platforms/microsoft-azure-ser...:

I put this script in /home/startup.sh and declared it on the Startup Command field in Web App Settings. I also configured the DT_ variables in the App Settings.

From what I've seen on the raw script, it seems that after the LD_PRELOAD it expects to run something:

# Inject variable into the proccess and run the actual application proccess LD_PRELOAD="/opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so" $START_APP_CMD

So, what is the command I need to put there in order for the OA to inject on the Tomcat process?

Regards,

George 

[g_kat]

Also, here is the log since I can't seem to make it work in 1 message:

2024-04-25T09:30:38.811521134Z STARTUP_COMMAND=
2024-04-25T09:30:38.813521392Z Copying /home/startup.sh to /tmp/startup.sh and fixing EOL characters in /tmp/startup.sh
2024-04-25T09:30:38.834581906Z Running STARTUP_FILE: /tmp/startup.sh
2024-04-25T09:30:38.948488027Z + readonly 'LIB_MUSL=musl'
2024-04-25T09:30:38.956471860Z + readonly 'LIB_GCLIB=libc'
2024-04-25T09:30:38.957476289Z + readonly 'LIB_DEFAULT=default'
2024-04-25T09:30:38.957489789Z + readonly 'ALPINE_RELEASE_FILE=/etc/alpine-release'
2024-04-25T09:30:38.957495590Z + readonly 'INSTALLER_DOWNLOAD_PATH=/tmp/installer.sh'
2024-04-25T09:30:38.957500590Z + readonly 'INSTALLER_URL_SUFFIX=api/v1/deployment/installer/agent/unix/paas-sh/latest'
2024-04-25T09:30:38.957899501Z + main
2024-04-25T09:30:38.958134608Z + lib=
2024-04-25T09:30:38.958374715Z + check_ldd
2024-04-25T09:30:38.959259841Z + ldd /bin/echo
2024-04-25T09:30:38.976444842Z + ldd_result=' /lib/ld-musl-x86_64.so.1 (0x7e11e2275000)
2024-04-25T09:30:38.978284196Z libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7e11e2275000)'
2024-04-25T09:30:38.979618635Z + readonly ldd_result
2024-04-25T09:30:38.988081481Z + grep -qi musl
2024-04-25T09:30:38.998547586Z + echo ' /lib/ld-musl-x86_64.so.1 (0x7e11e2275000)
2024-04-25T09:30:38.999750121Z libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7e11e2275000)'
2024-04-25T09:30:39.007408745Z + lib=musl
2024-04-25T09:30:39.008726683Z + test -z musl
2024-04-25T09:30:39.015904292Z + test -z musl
2024-04-25T09:30:39.019697803Z + DT_FLAVOR=musl
2024-04-25T09:30:39.019720504Z + run
2024-04-25T09:30:39.019726604Z + wget -O /tmp/installer.sh -q 'https://{environmentid}.live.dynatrace.com/api/v1/deployment/installer/agent/unix/paas-sh/latest?Api...'
2024-04-25T09:30:50.824393846Z + sh /tmp/installer.sh
2024-04-25T09:30:50.851764421Z 09:30:50 Installing to /opt
2024-04-25T09:30:50.893107091Z 09:30:50 Extracting...
2024-04-25T09:30:53.829458527Z 09:30:53 Unpacking to '/opt/dynatrace/oneagent' ...
2024-04-25T09:31:14.980992878Z 09:31:14 Unpacking complete.
2024-04-25T09:31:15.042335979Z + LD_PRELOAD=/opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
2024-04-25T09:31:15.050484818Z Finished running startup file '/tmp/startup.sh'. Exit code: '0'.
2024-04-25T09:31:15.051052835Z No STARTUP_COMMAND defined.
2024-04-25T09:31:15.051712854Z Starting Tomcat with CATALINA_BASE set to ""
2024-04-25T09:31:15.052701283Z Launched child process with pid: 209
2024-04-25T09:31:15.060163903Z Waiting for main process to exit. GLOBAL_PID_MAIN=209
2024-04-25T09:31:15.060765220Z Waiting for GLOBAL_PID_MAIN == 209
2024-04-25T09:31:15.146008823Z Picked up JAVA_TOOL_OPTIONS: -Xmx1150M -Djava.net.preferIPv4Stack=true

[AntonioSousa]

@g_kat,

Where do you get these logs?

[g_kat]

@AntonioSousa 

I'm not 100% sure because it's been some time, but I think they were from the docker logs I downloaded through Kudu.

[jose_araya]

Hello George, by looking at the log entries it seems that what is missing is the actual Start up command for your Tomcat Java application. It should point to the location and name of the startup script.sh that you normally use to start, it can even be the Environment variable in which you already store this script in the app service settings. 

Normally I configure all this inside the Azure portal App Service Configuration section, where it says Startup command:

You can add something like this, you need to replace all variables directly here or by declaring them before in the app service configuration

wget -O /tmp/installer-wrapper.sh -q https://raw.githubusercontent.com/dynatrace-oss/cloud-snippets/main/azure/linux-app-service/oneagent-installer.sh && DT_ENDPOINT=$DT_ENDPOINT DT_API_TOKEN=$DT_API_TOKEN DT_INCLUDE=$DT_INCLUDE START_APP_CMD=$START_APP_CMD sh /tmp/installer-wrapper.sh

Best regards,

Jose

[jose_araya]

 This is the section I'm talking about.

[g_kat]

Hi Jose,

Yes, that's the script I'm trying to figure out what to put in. In the past I have used pm2 or java there for Node and Java stack, but for Tomcat I don't get what I need to input there. Based on the MS documentation(https://learn.microsoft.com/en-us/troubleshoot/azure/app-service/faqs-app-service-linux#what-are-the...) it seems to expect a Tomcat configuration file but not necessarily a startup command. From this blog post(https://techcommunity.microsoft.com/t5/apps-on-azure-blog/customize-tomcat-configuration-in-linux-ap...) I gathered that you can use the tomcat startup script to make changes to the server, but I didn't see any startup command there like ./catalina.sh etc.

[g_kat · Accepted solution]

With

START_APP_CMD= /usr/local/tomcat/bin/startup.sh

I manage to get the OA to monitor the tomcat process.

[jose_araya]

Great! I was about to suggest the same.

If you don't see the traces yet you may need to restart again (double restart).

Good luck!

[AntonioSousa]

A note to everyone that might be trying to use an ActiveGate in this context: the script that is launched has a wget, and if you try it against an AG without a valid cert, it will fail.

I had to change, using tenant URL.

[benjamin_mullan]

My finding after few frustrating weeks is that when using the AzureRmWebAppDeployment@4 deployment method the tomcat instance is already running and deployed PRIOR to the Dynatrace installation. 

Using START_APP_CMD= /usr/local/tomcat/bin/startup.sh merely spawns additional instances of tomcat but traffic is not hitting these instances so you see the process but never the service itself. 

changing the command to   START_APP_CMD= /usr/local/tomcat/bin/catalina.sh run

allows the application to be redeployed (and essentially restarted) so you get the install and monitoring all happening on the initial bootup.

[g_kat]

I dropped this some time ago, but theoretically if you put the /usr/local/tomcat/bin/catalina.sh run command in the startup.sh script and then specify the script in START_APP_CMD it should have the same effect.

[AntonioSousa]

@benjamin_mullan,
I too am getting 2 tomcat processes, and the one that Dynatrace instruments also doesn't get any traffic.
I have been using essentially "ps -ef" during the container launch to figure out what is happening with each scripts being executed.

I followed your suggestion and replaced the startup.sh with the catalina.sh, but it is still spawning the two tomcat processes 😐

I'm also dealing with Support, so we can figure out what might be happening.

Also trying to get a hold on log files, but still trying to figure out why I don't have any.

[benjamin_mullan]

I'm specifically using  Azure App service plan - Linux | Runtime Stack Java 17 Tomcat

The code is built in Azure Devops into a .war and deployed to the app service with the AzureRmWebAppDeployment@4  task.

To troubleshoot I suggest going to the /appsvctmp/volatile/logs/runtime and checking the log files with the largest size (best to do a restart to get a new log generated). This is a copy of what the log stream feature shows in the portal but the amount of times that stream disconnects you are better off looking here.

In my case i could see the Azure start up followed immediately by a configuring tomcat and cp command for the war to tomcat hosting dir.

I then saw the Dynatrace install occur. with the start up command. Originally this was the startup.sh in /usr/local/tomcat/bin and this was spawning a new process with injection, but no traffic will hit this. ps -ef can be used to confirm the Prescence of one or more "/usr/lib/jdk/bin/java" processes. you only want one.

I found that using the Dynatrace provided startup command (in the app service config) actually fails with my setup so i copied the calling-only script below, which works for all Linux images. found on 
https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-cloud-platforms/microsoft-azure-ser....

I put this script in /home by removing #!/bin/sh and using echo "script here" > dynatrace.sh

I then used /home/dynatrace.sh in the app service startup. this allowed Dynatrace to install but again it was picking up a tom cat process but no service. 

I changed the app service environment variable START_APP_CMD 
from /usr/local/tomcat/bin/startup.sh
to /usr/local/tomcat/bin/catalina.sh run

and finally the deep service monitoring was showing.


My advice would be to see if removing the Dynatrace startup script all together only spawns one "/usr/lib/jdk/bin/java" process. if it does re introduce the steps as i have done above. using both the logs mentioned and the ps -ef command. One other thing to make sure is i notice the deployment task AzureRmWebAppDeployment@4  also supports a start up command in our case we remove it entirely and only have it on the app service infra code not the application code deployment

Additionally you can run all the commands manually in the ssh terminal, so run the dynatrace install if its successful you can go into /opt/dynatrace/oneagent/log and check the installer log for anything. if the process is injected there should be a /java/ dir aswell. Then try the various start up scripts at this point. Hoping that no additional process is spawned and that the tomcat restart isnt instructing the underlying container to exit and restart