Dynatrace Managed home page blank

[sami_bensalah]

Hi,

I have a Dynatrace Managed On-Prem version, and I cannot access the home page anymore as it is constantly loading.

And I have the below error message in the log:

2018-10-01 06:10:57 UTC SEVERE [<default,0x1,node01e5a4-internal,tab-cxWicLFVXSFS>] [WebApplicationExceptionHandler] Server Side Exception | admin (tab-cxWicLFVXSFS) | https://xxxxxxxxxxxxx:8021/rest/clustermanagement... | NullPointerException | UNHANDLED_SERVERSIDE_ERROR(10010) | || PKYYI96F

And also this, (seem to be an update issue)

2018-10-01 05:50:31 UTC SEVERE [<default,0x1>] [PeriodicServiceWorkerManager] Tenant: xxxxxxxxxxxxxxxxxxxxxxxxxxx Periodic worker: [node] updates worker encountered an exception. Will continue to run (throttled for 1 hour):
java.lang.NullPointerException
at com.compuware.apm.server.onprem.rest.management.api.updates.impl.UpdateBatchManagementUtilServiceImpl.getBatchIdentifier(UpdateBatchManagementUtilServiceImpl.java:210)
at com.compuware.apm.server.onprem.updates.UpdatesManagementServiceImpl.matchesWithLocalBatch(UpdatesManagementServiceImpl.java:254)
at com.compuware.apm.server.onprem.updates.UpdatesManagementServiceImpl.lambda$synchronizeWithUpdatesEntryList$3(UpdatesManagementServiceImpl.java:82)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.compuware.apm.server.onprem.updates.UpdatesManagementServiceImpl.synchronizeWithUpdatesEntryList(UpdatesManagementServiceImpl.java:83)
at com.compuware.apm.server.onprem.updates.UpdatesWorker.run(UpdatesWorker.java:118)
at com.compuware.apm.platform.PeriodicServiceWorkerManager.run(PeriodicServiceWorkerManager.java:130)
at com.compuware.apm.util.api.threadpool.RunnableThrowableFilterDecorator.run(RunnableThrowableFilterDecorator.java:48)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at com.compuware.apm.util.api.threadpool.AccessibleRunnableScheduledFuture.run(AccessibleRunnableScheduledFuture.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Can someone please help?

[gary_spencer]

Hi,

Sorry to hear this, are you accessing the CMC, an environment, or a particular page bookmarked?

I have seen two similar cases on sites

1. One was due to the font after a RHEL upgrade a customer performed where a default font is moved and is solved with this.
2. Another I seen is on user session drilldown in some small amount of cases, for which we have issued a fix in 1.154. Check with support if your problem is related.

If you do not have the first issue (RHEL/java font) then please open a support ticket since we will need further information.

Thanks;
Gary.

[sami_bensalah]

Other pages are view-able, issue is only on the cluster management home page.

If I click on the Settings > Automatic Update, I get the following

We’ve run into
technical difficulties
Don’t worry though — your data is secure
and we’ve logged this issue.

[gary_spencer]

Ok thanks for the update. Can I ask your browser?

[sami_bensalah]

tried on

IE 11.0.9600.19002

Chrome 66.0.3359.139

[gary_spencer]

Ok thanks for trying, both browsers are supported. I advise log a ticket here for this.

[paulina_rudnick]

Hi, which version of Dynatrace Managed are you using? There was a bug which produced exactly the same error log, but it was fixed in 1.146

[sami_bensalah]

Hi,

1.140 but upgrading per step to 1.152 but it seem I have to wait 24 hours between upgrades

Can we reduce this timeframe to a 4 hours?

Regards,

[paulina_rudnick]

This timeframe is for automatic upgrades, normally you can also trigger the upgrade manually at any time by clicking the "Upgrade now" button on the Automatic Update page. However, this page is currently not working because of the bug... And this bug may also prevent automatic upgrades from starting.

There is a couple of things we can do, but first can you please tell me if this is an online cluster (connected to Mission Control, automatically downloads update files) or an offline cluster (not connected, update files are uploaded manually)? It's important, because this bug could be caused by update files being only partially uploaded (e.g. copying .sh files without corresponding .config files). Also, do you have access to machines with Dynatrace Managed installed (or in other words, is executing installation files manually an option?)

[sami_bensalah]

I can install files manually

It is indeed offline

[sami_bensalah]

I also did launch a rest command through upgradeManagement and I can see that some product are in REMOVING status for 1 day

[
{
"type": "JS_AGENT",
"version": "1.151.206.20180821-210005",
"status": "READY",
"fileSizeInBytes": 692179,
"deleteEnabled": true,
"readyNodeIds": [1,3,5
]
},
{
"type": "SERVER",
"version": "1.146.102.20180730-082136",
"status": "REMOVING",
"fileSizeInBytes": 765795881,
"deleteEnabled": false,
"readyNodeIds": [3,5
]
},
{
"type": "AGENT",
"version": "1.145.241.20180702-133131",
"status": "READY",
"fileSizeInBytes": 770044352,
"deleteEnabled": true,
"readyNodeIds": [1,3,5
]
},
{
"type": "SG",
"version": "1.145.244.20180704-091933",
"status": "REMOVING",
"fileSizeInBytes": 466420362,
"deleteEnabled": false,
"readyNodeIds": [3,5
]
},
{
"type": "AGENT",
"version": "1.151.314.20180920-141612",
"status": "PROGRESS",
"fileSizeInBytes": 836665729,
"deleteEnabled": true,
"readyNodeIds": [3,5
]
},
{
"type": "NGINX",
"version": "1.1.0.20180926-001200",
"status": "READY",
"fileSizeInBytes": 7185159,
"deleteEnabled": true,
"readyNodeIds": [1,3,5
]
},
{
"type": "JS_AGENT",
"version": "1.145.161.20180613-161859",
"status": "REMOVING",
"fileSizeInBytes": 4156888,
"deleteEnabled": false,
"readyNodeIds": [3,5
]
},
{
"type": "NGINX",
"version": "1.1.0.20180731-001200",
"status": "REMOVING",
"fileSizeInBytes": 7060311,
"deleteEnabled": false,
"readyNodeIds": [3,5
]
}
]

[paulina_rudnick]

Could be because 1.146 installation package is not valid for 1.140 (you cannot skip 142 and 144) and this is a pretty old version, which maybe didn't handle such situations well. In newer versions this problem should not occur (e.g. you can upload 148, 150 and 152 while in version 146)

As for agents etc, their version cannot be higher than server version, which is why they're treated as invalid, too

[paulina_rudnick · Accepted solution]

Then the easiest way to upgrade this cluster with that bug in place would probably be manually executing installation files on each node. If you have already uploaded them via UI before, they're located by default in "/opt/dynatrace-managed/installer/upgrade" (could be different if you're using custom paths). But you can unpack your server installation package and run the file anywhere on the machine. The file you need to execute will be named "dynatrace-managed-1.142.xxx.sh". No parameters are necessary, as this is an upgrade and will use existing configuration. After you successfully upgrade each server node to 142, you will need to do the same for version 144 and 146.

Starting from 146, you should be able to use the UI again, because the bug from before will be fixed. You can then upload all remaining installation packages (148, 150 and 152) on the Automatic Update page and trigger upgrades with a button on that page. I hope this helps.