On December 9th, 2021, security researchers published a 0-day exploit in the Java logging library log4j that results in Remote Code Execution (RCE) by logging a certain string. This exploit applies to versions 2.0 <= log4j <= 2.14.1, and is widely used across common software applications.
CVE-2021-44228, dubbed Log4Shell, could allow an attacker to control log messages or log message parameters, and execute arbitrary code when message lookup substitution is enabled. As soon as we learned about this vulnerability, Harness initiated a broad-sweeping discovery effort to identify and validate the presence of log4j in our codebase.
Remediation Efforts
The Incident Response process at Harness is designed to quickly notify relevant stakeholders, and enable rapid response to operational or security events. In this case, our IR process included representatives from Engineering, Security, Customer Success, and Legal.
We quickly discovered several instances of log4j-api:2.12.1 and log4j-to-slf4j:2.12.1 supporting components of our primary SaaS environment. Harness uses these libraries to collect and categorize a variety of messages generated by our platform. Our next steps were to determine priority for remediation and triggering our incident response process, with first priority to patch the vulnerability on external facing services.
Harness microservices deployment is done on Kubernetes clusters in Google Cloud Platform (GCP). As Harness continues to scale, we have our microservices deployed to multiple Kubernetes clusters in GCP, which horizontally partition customer data across those clusters to ensure performance of our application.
We also maintain an API Gateway microservice which works as a reverse proxy, and routes customer requests to correct clusters. This Gateway was our first priority for patching, and we were able to successfully apply a hotfix without any impact to performance or availability.
Shortly after, we systematically applied the log4j 2.15.0 patch to the backend service across our three production k8s clusters. You can read more about our deployment and production architecture in this blog post.
While Harness delegates do not support HTTP/HTTPS termination (or run any embedded web services) and this vulnerability does not have any direct impact to our delegate deployment in our customer environments, we will be upgrading our delegates with the same version of the library soon.
Update: 12/17/2021
Harness is aware of CVE-2021-45056, which could allow attackers with control of the Thread Context Map to bypass mitigations provided in the log4j update 2.15.0. Apache has classified this CVE as Low (3.7).
Apache has recommended an additional upgrade of the log4j library to version 2.16.0. This update is currently in our QA cycle, and our standard major release on Wednesday, 22 December will include log4j 2.16.0 across SaaS environments and the Harness delegate.
On 15 December, Harness published an updated version (73021) of our on-prem service, which contains the log4j 2.16.0 update in all management services and the delegate. Customers can obtain updated images through their normal mechanisms.
On 17 December UTC, Harness released version 73119 to the SaaS Prod 1 and Prod 2 instances. With this version, the Harness delegate for SaaS now includes log4j version 2.15.
Update: 12/20/2021
On 17 December, security researchers discovered a bypass for the log4j mitigation introduced in 2.15.0, allowing for remote code execution if successfully exploited.
Due to the increase in severity and impact, Harness immediately expedited release of the log4j 2.16.0 patch in our production environments. As of 18 December, all production clusters and the gateway service have been successfully patched to 2.16.0.
On 18 December, Apache released 2.17.0, which mitigates CVE-2021-4104 (affecting log4j 1.2) and CVE-2021-45105 (affecting the log4j-core JAR).
We have not detected use of the affected components in our environments. At this time, the risks identified in CVE-2021-4104 and CVE-2021-45105 are not applicable. We will continue to evaluate new patches for applicability, and monitor Apache’s security page for any updates.