Impact of log4j zero day vulnerability

[Enrico_F]

Today a high severity zero day vulnerability impacting the very popular log4j package has been published:

https://www.randori.com/blog/cve-2021-44228/

https://www.lunasec.io/docs/blog/log4j-zero-day/

I would be interested to know if any Dynatrace components are known to be affected and if so, how exactly, what's the risk of compromise and if there is anything that can be done from a user/customer perspective to help minimize the risk of exploits.

I've already approached support but haven't received any response yet.

Any feedback is appreciated.

[Enrico_F]

My own digging on our managed cluster nodes (1.226) has revealed this:

$ ls -l /opt/dynatrace-managed/elasticsearch/lib | grep log4j
-rw-r--r--. 1 dynatrace dynatrace 264060 May 11 2020 log4j-api-2.11.1.jar
-rw-r--r--. 1 dynatrace dynatrace 1607947 May 11 2020 log4j-core-2.11.1.jar

 So it seems the bundled ES component might be affected...

[AntonioSousa]

If the sha256 hash is the one below, it most certainly is:

d4485176aea67cc85f5ccc45bb66166f8bfc715ae4a695f0d870a1f8d848cc3d log4j-core-2.11.1.jar

Source: https://github.com/mubix/CVE-2021-44228-Log4Shell-Hashes/blob/main/sha256sums.txt

[ben_davidson]

Hi Enrico, 

Our team is aware and actively investigating this issue. We've released the following statement:

Dynatrace is currently investigating potential exposure of the Log4Shell (CVE-2021-44228) vulnerability. Dynatrace is treating this as a critical vulnerability and follows its incident response procedures. In case Dynatrace identifies an exposure of this vulnerability to our customers, we are going to proactively inform affected customers within the next 72 hours.

[ChadTurner]

Thanks @Enrico_F . Waiting on confirmation for SaaS Customers. I'll post any news we get. 

[melorymonie]

Hi,

is it possible to use Dynatrace to find all servers/services that use log4j?

[deon_allen]

We found that only Synthetic use it.  So if you have an activegate with synthetics on it will have Log4J.

[AntonioSousa]

The new Application Security offering should be able to point this out in your applications.

[ben_wrightson]

Yes, it does.

[kb00563121]

we are using Activegate for our Synthetic Monitoring , will our Activegate and other systems were also get impacted and also we are using SaaS tenant.

[jonathanp]

Hi All,

Does anyone have any updates on this?
Synthetics have log4j2, but what I can't tell is the version that it is running.  -- anyone else ?

Would be good to know what Active Gate versions are affected..

[AntonioSousa]

I have not confirmed, yes or no, that Synthetic Activegates are affected.

But looking at the description of the vulnerability, it will only happen if an attacker can inject some malicious code in synthetic measurements being done. I would say that most of the private synthetic use cases that I can imagine are not easily attackable.

For anyone involved, probably one of the best security measures would be cancelling LDAP traffic, or other JNDI related endpoints, out of the organization, in firewalls. Well, that should be something in place by default. It doesn't protect though against internal attackers.  There are other mitigations possible, as referenced in https://logging.apache.org/log4j/2.x/security.html

[tarjei]

We have been scanned multiple times from different clients. There are also multiple log4j components in dynatrace managed.

Would be nice to get some feed back one whether the different important flags are set, and whether DT Managed is vulnerable.

[anton_goosen · Accepted solution]

Dear valued customer, 

We want to inform you that in response to the recently published vulnerability of log4j (CVE-2021-44228) the Dynatrace team has rigorously reviewed any potential exposure and risks arising from the vulnerability. 

We have updated all components of our cloud services where a risk could not be ruled out and our cloud services are currently not exposed to the vulnerability. We have equally ruled out any potential risk for the Dynatrace Managed cluster and OneAgent. 

While investigations are still ongoing, we have, out of an abundance of caution, provided updates for the Synthetic Chromium component (used only in ActiveGates running Private Synthetic tests) with an updated version of log4j which is not affected by this vulnerability.  

By default, this upgrade is applied automatically. In case the auto update functionality for ActiveGates has been disabled, we recommend customers manually trigger the upgrade. The following Synthetic Chromium versions use the updated log4j version: 

• 1.229.49.20211210-165018 
• 1.227.31.20211210-164955 
• 1.225.29.20211210-164930 
• 1.223.30.20211210-164926 

What do I do if I have questions?  

If you have any questions, please reach out to us using our in-product assistance from within your Dynatrace environment or open a Ticket in our Support-Portal. 

Sincerely, 
Dynatrace ONE 

[tarjei]

Where did you get this? I have not gotten this.

[stefan_lexow]

Hey Tarjei, Dynatrace targeted this message to customers which actively use Private Synthetic locations. Let me know if there is anything else I can help you with.

[tarjei]

We use and manage private synthetic for at least 10 different customers, so I find it a bit strange that this has not reached us?

[stefan_lexow]

Hey Tarjei, can you confirm that you are signed up as an "emergency contact" for these clusters as documented here?
https://www.dynatrace.com/support/help/how-to-use-dynatrace/user-management-and-sso/configure-emerge...

[Henmar]

Hi Stefan. I can confirm that I'm emergency contact on one of our clusters and no email has been received.

/Thomas

[Enrico_F]

We are actively using several synthetic private locations but so far none of our emergency contacts have received this message, either.

In our case auto-update for our ActiveGates is turned off but it seems the "Synthetic" module has already been updated automatically without our intervention regardless:

Perhaps that explains why we didn't receive any info?

[jungwan_kim]

Hi Team,

Thanks for the answer.

The updated version of log4j is 2.15. Right?

Regards

[anton_goosen]

Yes, 2.15.0.

[Babar_Qayyum]

Hello @anton_goosen 

Our security team is complaining about the following:

• /dyna/dm_binaries/elasticsearch/lib/log4j-api-2.11.1.jar
  /dyna/dm_binaries/elasticsearch/lib/log4j-core-2.11.1.jar

The cluster version: 1.228.128.20211118-160244

Regards,

Babar

[tarjei]

We are also worried about these.

Would be nice with:

- Are these vulnerable?

- When will log4j be updated on these components?

[sureshKumarK]

Hi, Could someone confirm whether any other component than Synthetic monitoring/Synthetic enabled with Activegate is also impacted because of this issue? Our security team is concerned about ongoing vulnerability. Appreciate if someone from Dynatrace confirm whether Dynatrace SaaS and/or Dynatrace Managed are impacted. Thanks, Suresh.

[ben_wrightson]

Hi Suresh,

Anton has mentioned the other components are not affected:
"We have updated all components of our cloud services where a risk could not be ruled out and our cloud services are currently not exposed to the vulnerability. We have equally ruled out any potential risk for the Dynatrace Managed cluster and OneAgent."
Will that help your security team?
If they have any more questions, I'd recommend to use the in-product chat or contacting Support.

[sureshKumarK]

Thanks for the update @ben_wrightson and @anton_goosen your latest updates puts our team in a comfortable spot ATM 😊 But, please post further updates on this incident as soon as you have. Have a nice weekend!

[Karolina_Linda]

Yesterday, Dynatrace has also published a blog post on this vulnerability: https://www.dynatrace.com/news/blog/log4j-vulnerability-identifying-and-minimizing-production-risk/  

[chki]

HI.

elasticsearch has log4j 2.11.

And log4j is loaded in the process.

Restart the service by adding -Dlog4j2.formatMsgNoLookups=true to jvm.options in elasticsearch/config.

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-es...

[kalle_lahtinen]

@anton_goosen or someone from Dynatrace - can you please comment on elasticsearch running on Managed cluster nodes? This appears to conflict the earlier statement "We have equally ruled out any potential risk for the Dynatrace Managed cluster and OneAgent."

[chki]

My Managed server version is 1.230.120.

And built-in elasticsearch version is 7.10.0.

JVM version : 1.8.0_292

Elasticsearch says "Elasticsearch running on JDK8 or below is susceptible to an information leak via DNS which is fixable by the JVM property identified below. The JVM option identified below is effective for Elasticsearch versions 5.5+, 6.5+, and 7+. Soon we will make available Elasticsearch 6.8.21 and 7.16.1 which will set the JVM option identified below and remove the vulnerable Log4j component out of an abundance of caution."

[anton_goosen]

Hi Chki

Our security team is still assessing the impact if that Elastic search version. If you are concerned, I can confirm that the Mitigation step (setting the JVM option) that is suggested in your post is safe to use on the DT Elastic server. We will update everyone as soon as we have confirmation. 

Anton

[chris_labrado]

Great, is there a broader communication being prepared for all customers?

[nada_ezzeldeen]

HI @Kalle,

According to https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-es..., what I can confirm is Elasticsearch is not susceptible to remote code execution with this vulnerability due to the use of the Java Security Manager.

[Nizar]

Hi Nada,

How to check that we use Java Security Manager or is already activated by the module ?

[tylerj]

Is the DynaTrace Android Mobile App impacted??

[Nizar]

Hi all,

We are not installing active gate 

It does it mean we are not using this module and we are not impacted by the vulnearbility so we can remove the jar file ?

[markus_saul]

If you are referring to the jar file on cluster hosts, then this is not related to the ActiveGate Synthetic Module. The jar file on the cluster hosts belongs to ElasticSearch. According to  https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-es  There are the versions listed which are affected. The content was update for new version information: "[...]Supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) are not susceptible to either remote code execution or information leakage. This is due to Elasticsearch’s usage of the Java Security Manager. [...]". To check your ElasticSearch version you could use: curl -XGET 'http://localhost:9200' on a cluster node. 

[sureshKumarK]

Hi Team, I am seeing log4j 2.15 is also discredited. So, may I know what Dynatrace team suggesting now?
 Please see this page https://github.com/apache/logging-log4j2/blob/release-2.x/src/site/markdown/security.md#history

Older (discredited) mitigation measures

This page previously mentioned other mitigation measures, but we discovered that these measures only limit exposure while leaving some attack vectors open.

The 2.15.0 release was found to still be vulnerable when the configuration has a pattern layout containing a Context Lookup (for example, $${ctx:loginId}), or a Thread Context Map pattern %X, %mdc or %MDC. When an attacker can control Thread Context values, they may inject a JNDI Lookup pattern, which will be evaluated and result in a JNDI connection. Log4j 2.15.0 restricts JNDI connections to localhost by default, but this may still result in DOS (Denial of Service) attacks, or worse.

Other insufficient mitigation measures are: setting system property log4j2.formatMsgNoLookups or environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true for releases >= 2.10, or modifying the logging configuration to disable message lookups with %m{nolookups}, %msg{nolookups} or %message{nolookups} for releases >= 2.7 and <= 2.14.1.

The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.

The safest thing to do is to upgrade Log4j to a safe version, or remove the JndiLookup class from the log4j-core jar.

[anton_goosen]

Hi Suresh

Our security team are closely monitoring the situation and are working to mitigate any issues as we find them.  If you have any questions, please reach out to us via Chat or a Support Ticket
Kind Regards

Anton

[rastislav_danis]

Fixed versions of Dynatrace mentioned in official communication:

1.230.127.20211213-130244, 1.228.131.20211213-130253, 1.226.128.20211213-130354
contains same elastic/log4j versions as nonfixed, but with added elastic jvm parameter "-Dlog4j2.formatMsgNoLookups=true".

[DTAnalyst]

Hey DnyaMight Pro

This of course is impacting our Managed Cluster Nodes!  We're currently on version 1.230.127.20211213-130244.  Our infosec team would like know if the lower lo4j (2.11) can be removed without causing any impact as our scans will continue to highlight these libraries..

Can the log4j be upgraded?  If so, what are the steps?  

[stefan_lexow · Accepted solution]

Dear valued customers,

we would like to inform you that Dynatrace just published a website summarizing the current state and findings in regards to the current log4j situation. You can find the article here:
https://www.dynatrace.com/news/security-alert/log4shell-log4j-vulnerability/

Dynatrace expects to update this document as new information becomes available.

[MaciejNeumann]

As all official communication about this topic will be done from now on through the article Stefan posted, Dynatrace chat and support tickets, I'm closing this thread for now - as soon as I will get a green light again, it will be reopened (hopefully pretty shortly 🙂)

[ghaydtner · Accepted solution]

To learn more about how the Log4Shell vulnerability itself works and how to mitigate it, check out the following resources:

• What is Log4j vulnerability? Log4Shell explained blog
• Log4Shell: Identifying and minimizing production risk blog
• Log4Shell vulnerability: Identifying and minimizing production risk webinar
• Log4Shell vulnerability discovery and mitigation require automatic and intelligent observability blog
• Log4j vulnerability (Log4Shell) Dynatrace security alert
• Dynatrace Log4Shell resource center webpage