We’re midway through December 2021, right before the calendar year ends, and the log4j vulnerability has bubbled to the top of the to-do list for any company, especially if they have applications that talk to the internet.

This post aims to describe the Log4Shell flaw in digestible terms, and how you can engage our team to help.

‍

Summary:

1. Log4Shell needs to be taken seriously.

2. The vulnerability against log4j, dubbed “Log4Shell”, is officially NIST CVE-2021-44228.

3. Concise updates can be found on Github’s advisory here.

4. A RASP tool (Runtime Application Self-Protection) will help you protect yourself during remediation. AppDynamics customers can leverage Cisco Secure Application for immediate protection during remediation.

5. Organizations need a platform like Lacework that provides holistic cloud security visibility.

6. Once the Log4Shell vulnerability is remediated, organizations should examine their DevOps infrastructure and policies and ensure they are equipped to respond to these threats fluidly and decisively.

‍

Impacted Apache Log4j Versions: 2.0 through 2.14.1

Severity rating: Critical (10.0 Overall NIST CVSS)

NIST’s official listing: CVE-2021-44228

TALOS Threat Advisory: CVE-2021-44228

Apache’s Remediation Guidance: Remote code injection in Log4j

‍

The Fundamentals: What is Log4Shell?

Broader Context

The Apache Log4j vulnerability, called Log4Shell, has similarities to SolarWinds vulnerability. Sure they’re different, but they’re two very similar situations. Why? SolarWinds and Log4Shell are both Software Supply Chain Vulnerabilities. The SUNBURST vulnerability (the one that lead to the SolarWinds incident) was a malicious inclusion (an attack) in a DLL, and the Log4Shell vulnerability is currently believed to be an accidental inclusion in the log4j library, but both are vulnerabilities in the software supply chain.

‍

What is a “Software Supply Chain Vulnerability”?

Hidden deep inside all applications are code libraries that the application developer did not write, and that they did not personally vet. They are ubiquitous. These libraries are little (sometimes not-so-little) bundles of pre-written code that perform a specific function or set of functions.

Developers use public code libraries to not reinvent the wheel. Or in this case, to not reinvent a Java logging framework. Developers leverage libraries like log4j because the libraries have proven themselves as robust, powerful, and reliable solutions for common software tasks.

Therefore, a Software Supply Chain Vulnerability happens when a code library is compromised further up the supply chain from the Application.

‍

How do Software Supply Chain Vulnerabilities happen?

Problems happen when a vulnerability is found in software libraries. Whether a vulnerability ends up in the library code maliciously or accidentally, if you are using it, you are at risk. Software and Security Engineers can’t entirely insulate themselves from the threat of vulnerabilities in dependencies. Teams might pursue rigorous auditing or they might try to avoid using external libraries entirely, but those options aren’t  economically feasible, typically.

In addition to the threat of vulnerabilities being found against existing code, new vulnerable code might be published in the future. These libraries are constantly being updated. Updates come for bugfixes and security patches, which we are all happy to get, but new features also get added. These new features represent an ever growing attack surface against our applications. A development team could decide to pin their dependencies to a specific version with only the features they need, but then they miss out on bugfixes and security patches. In the worst cases, vulnerabilities can be added maliciously, which would make this a Software Supply Chain Attack.

‍

What is Log4Shell Specifically?

Log4Shell, officially CVE-2021-44228, is a Remote Code Execution (RCE) vulnerability in an ubiquitous Java logging framework, log4j (v2).

‍

There are three factors that make Log4Shell so dangerous:

1. The vulnerable library log4j is widely deployed.

2. The vulnerability is shockingly easy to exploit.

3. Exploiting gives attackers the ability to run arbitrary code remotely.

‍

The vulnerable library log4j is widely deployed: Log4J2 has existed as a library since 2001, and is nearly 21 years old. As of this writing, it has just shy of 11,500 commits on its git repo (https://github.com/apache/logging-log4j2), and it is totally free to use, licensed by the not-for-profit Apache Foundation. Developers could attempt to replicate 21 years of ongoing work, or they could use this free, readily available library that is already trusted and used nearly everywhere.

‍

The Log4Shell vulnerability is incredibly easy to exploit: All it takes is to be able to make a vulnerable application log something of your choice.

This is much easier than it sounds. Let’s say a user goes to an online storefront. They attempt to log in, and the webapp logs their attempt.  Their username is recorded along with the date they made the attempt, whether or not the attempt was successful, etc. So if that user is an attacker and decides to type a malicious string into the username field, it gets logged. The site need not even have an option to log in. As a user navigates to a page, it logs their visit. Often, their browser’s user-agent identifier gets logged. An attacker can spoof this user agent to a string of their choice, and then the application will log that custom string.

Logging an attacker’s custom string on its own wouldn’t be problematic. The issue comes in one of log4j’s many features: message lookup substitution. When message lookups are enabled, lookups give log4j the ability to enrich data by performing lookups against urls in the logged message.

‍

Now attackers can run arbitrary code remotely: An attacker hosts a url that services malicious code, and forces an application running log4j to log it. When logged, log4j will perform a lookup and connect to that url, downloading and running the malicious code. The malicious code exploits the Log4Shell vulnerability and injects a second stage of the attack into the jvm. The second stage then allows the attacker to execute their own code remotely. This is a simplified version of the full process, which Talos Intelligence describes here: (https://blog.talosintelligence.com/2021/12/apache-log4j-rce-vulnerability.html).

‍

What Action Steps Should I Take?

In my experience tackling Log4Shell over the past few days, the most reliable source for concise, accurate information has been here:

Github CVE-2021-44228 Advisory

‍

Immediate action steps are to:

- If you utilize a RASP, ensure it is configured to block exploits against this vulnerability.

- Take inventory of where your own apps are vulnerable via vulnerability scanning or other methods.

- Take inventory of your third party software and check their advisories.

- Remediate your applications by updating any log4j libraries in use to >=2.16.0.

- Remediate your third party software by following instructions in your vendors’ advisories.

- Hunt possible threats internally. The speed Log4Shell was exploited by attackers means even if an organization did everything right, there could still have been a breach.

‍

Lending a Helping Hand or Second Opinion

Organizations will not completely fix the Log4Shell vulnerability overnight, this week, or even this month. Security, DevOps, infrastructure, and IT Leadership teams all need to work together over both the short-term and the long-term.

Security and technology teams need ways to get visibility, simplicity, and usability to complex enterprise platforms to lead the digital transformation and addressing vulnerabilities quickly, without the resulting cascade of IT headaches.

‍

‍

Short-Term

‍

Lending a Helping Hand or Second Opinion

We can offer advisory calls with our experts to validate that our clients are addressing the vulnerabilities correctly. Typically we can help with identification of vulnerable applications and components within a client's landscape, through automated scanning. There are ways we can provide assistance with the actual mitigation, especially in more complex scenarios. And lastly, we can give advice on how to safeguard systems going forward, including suggestions on tooling and risk management.

‍

Cisco Secure Application

Cisco Secure Application is a RASP (Runtime Application Self Protection) tool developed alongside AppDynamics. Of the available RASP tools, Cisco Secure Application is the smoothest to deploy and use I’ve ever encountered.

It is built into all AppDynamics agents 21.3 or later. It won’t automate remediation for you, but it can help you track where you are vulnerable to Log4Shell and offer protection while you remediate. Because it’s built into the AppDynamics Agent, if you are already an AppDynamics user nothing is faster than adding Secure Application. You can have it enabled and actively protect your applications in as little as a day.

Secure Application tracks vulnerabilities by looking at the application's loaded libraries. It can see the vulnerable library, and see executions that use the vulnerable code path. It is then able to detect exploits running against that (and other) vulnerable code paths and alert you to them. It can optionally be configured to block that exploit execution as well.

If you are running 21.3 or later agents, Secure Application can be turned on without any additional deployment required on your part. The Log4Shell vulnerability and exploits against the vulnerability are detected automatically. The only configuration is to enable apps, and then to configure automatic blocking of the exploit if desired.

See This Blog Post On Cisco Secure Application In Action

‍

Shore Up Cloud Accounts | Lacework

Like Secure Application, Lacework's runtime container vulnerability detection can help you track where in your environment you're vulnerable to this type of threat. Going beyond Secure Application, Lacework can help you see at a host level every instance of a vulnerable library in your cloud accounts.

In addition, Lacework's container scanning helps you detect these vulnerabilities in your software before they are deployed. This is particularly helpful because it can help you avoid ever reverting back to a vulnerable version.

Where Lacework really shines though is in its holistic approach to cloud Security. Lacework’s industry leading anomaly detection allows you to quickly detect threats that may have successfully gained entry into your environment.

See Lacework's Analysis Of Log4Shell Attackers

With an initial deploy time in as little as a week, Lacework allows you to rapidly go from blind to total cloud visibility.

‍

Long-Term

‍

Patching and Deploying Updates | Harness

In the longer term, you should strongly consider making sure your engineering team is equipped with a mechanism for quickly patching and deploying software updates.

Harness enables customers to simplify complex modern DevOps environments. The winning strategy is threefold: increased speed to market, automated cloud optimization, and security-first developer empowerment.  Harness brings governance to IT change as customers deploy code, patches, and spin up new infrastructure. IT change creates business risk for customers, and the ROI on governance is massive, both in dollars and on morale.

‍

Conclusion

In summary, Log4Shell is a serious threat but it likely won’t be the last threat of its type that security teams encounter. It’s important to be prepared for these types of threats in the future, as well, because we will see more of them. As scary as these threats are, modern tooling (such as Cisco Secure Application) is well equipped to help you get protected and get visibility. Forward looking security professionals should continue to implement this tooling and governance before the next vulnerability, and Evolutio can help.

If you want to learn more, follow the "Contact Us" form and we can have a discussion.