By now everyone has likely come across the recent log4j exploit called "log4shell", recorded as CVE-2021-44228. Later, two more vulnerabilities were discovered, CVE-2021-45046 and CVE-2021-45105. We wanted to share our findings and recommended immediate remediation steps. We will update this post as new information becomes available.
Whenever we update information in this blog post you can find a summary of the changes here.
|2021-12-21||add note on patching live system|
|2021-12-21||add info about Cloud & CDN security services|
|2021-12-21||published public version|
|2021-12-15||add notes on 3rd party software remediation|
|2021-12-15||add info on new CVE-2021-45046|
What happened and who is affected?
A vulnerability in the
log4j-core Java library was found that allows Remote Code Execution (RCE) and/or extracting information from affected systems. It is considered one of the most serious bugs in recent software history. Every application using the
log4j-core library directly or indirectly might be affected. Worse, the vulnerability seems to have been around for a few years(!).
log4j-core (i.e. the implementation of the log4j v2 logging engine) is affected.
log4j-api is not affected.
Immediate Mitigation Steps
Upgrade every affected application to log4j v2.17.0 ASAP. (no, there's no way around it!)
- most other fixes previously circulated have been discredited
rotate all secrets after upgrading!
- Secrets used by the application may have been compromised
perform a clean slate redeployment of your services
- hackers might have already installed malicious software
If you can't upgrade & redeploy, there's a possibility to temporarily patch a running live system as documented by LunaSec here.
In a soon-to-follow-up post, we will describe a list of ways to upgrade applications.
Note: there are some other recommendations out there - only the first in the list below is still considered a valid remediation:
For releases from 2.0 to 2.10.0: you may want to remove the LDAP class from log4j completely by issuing the following command: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
For version >=2.10: set log4j2.formatMsgNoLookups to true
For certain JVM Versions, it is possible to set com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase to false to mitigate the vulnerability. Some JVM versions already have this as default setting
How to identify affected applications?
IMPORTANT: internal systems may be affected as well: Malicious data may be sent via messages or stored in a database and may trigger the exploit when this data is processed at a later point by such internal systems!
mvn dependency:tree | grep log4j-core to check if your application has
log4j-core on its classpath. If so -> force an upgrade (e.g. using
3rd party applications
Here is a list of applications known to use log4j v2. Check with the vendors for patch releases.
Scan using APM tools like NewRelic
APM tools like NewRelic, AppDynamics, DynaTrace and Datadog support querying your applications' inventories.
In our case, a lot of our services integrate with NewRelic. There's a script that allows to use NewRelic to scan software for usages of log4j-core libraries. This is also a great help for monitoring progress of mitigation efforts.
Use Security Scanners to scan your systems
Previously circulating superseded information
Previously at has been reported that some Java versions would be safe. A few other workarounds like setting environment variables to disable to problematic feature were suggested. All this information and workarounds have all been discredited in the meantime (see https://logging.apache.org/log4j/2.x/security.html)
log4j v1 is not affected
log4j v1 has a lot of other security holes, but incidently does NOT seem to be affected by log4shell.
Check entire JVM
Check any JVM agent you might be running and upgrade to a patched version. E.g. also NewRelic agent is affected and must be upgraded.
Logback is also affected, but low severity
logback is also affected but the exploit is much more difficult, hence the severity of logback's bug is not critical. Upgrade to v1.2.8 is still recommended though.
How the Attack works
Lunasec did a great job explaining the attack in detail in their blog post.
Identify successful attacks
Since the exploit is already around for a while, noone can be sure, whether systems had been successfully attacked in the past. We are currently triaging systems under the assumption that they have been exploited in the past. The good news is, that we can not identify any attacks prior to 2021-12-10 ~14h CEST.
Leverage Cloud and CDN Provider Security Services
Check for posts of your providers about using their security services to protect your applications
- AWS: https://aws.amazon.com/blogs/security/using-aws-security-services-to-protect-against-detect-and-respond-to-the-log4j-vulnerability/
- CloudFlare: https://blog.cloudflare.com/tag/log4j/
- Azure: https://www.microsoft.com/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-log4j-2-exploitation/
Review emergency procedures
This includes reviewing runbooks etc. to coordinate and improve your response to future attacks. A thorough post-mortem is highly recommended!
Swap log4j for slf4j+logback
At this point it is advisable to replace log4j with logback by leveraging slf4j's "bridge" components - it's effectively a drop-in replacement: http://www.slf4j.org/legacy.html
- List of known affected software: https://github.com/NCSC-NL/log4shell/tree/main/software
- Latest log4j security news: https://logging.apache.org/log4j/2.x/security.html
- NewRelic agent: https://github.com/newrelic/newrelic-java-agent/releases/latest
- Logback news: http://logback.qos.ch/news.html
- Logback as drop-in replacement for log4j: http://www.slf4j.org/legacy.html
- Lunasec's detailled explanation of the exploit: https://www.lunasec.io/docs/blog/log4shell-live-patch-technical/
- Nuclei Security Scanner: https://nuclei.projectdiscovery.io/nuclei/get-started/
- Burp Security Suite: https://portswigger.net/burp