Log4j exploit weekend fun

What did you do this weekend? A fun side project, time with the family? I mostly analyzed my company’s services and infrastructure to find out whether we are affected by the Log4j exploit published this week (Log4Shell, CVE-2021-44228).

We do not use Java as a backend language at DroidSolutions, so nothing we developed ourselves is directly exploitable. But that is only part of the picture. We run third-party services in our clusters and on our servers, we have internally managed tools, and we buy SaaS from vendors — Confluence and Jira, for example. Every one of those is a potential Log4j carrier.

What the audit looked like

A combination of firewall rules, patched startup scripts and environments, and deployed bugfix versions has hopefully moved us out of the direct danger zone. Some of the concrete tasks from the weekend:

• Keycloak (v15.1): no log4j 2 dependency — fine.
• Internal Jitsi: patched the startup scripts of JVB and Jicofo.
• Unifi Cloud Key: deployed v6.5.54.
• Checked firewall rules on several services, for example the internal-only Elasticsearch.
• Went through every server — all VMs in the Proxmox cluster, every service deployed in the Kubernetes clusters — looking for Java-based software we had not documented so far.
• Created a Confluence page to document the findings.
• Wrote up the state in our internal chat so the responsible people (ops, CEO) have everything available on Monday morning.

There is still work left for the coming days, and we will keep monitoring the situation, our logs, and the release feeds of everything we run.

What made it bearable

The audit went as quickly as it did because of work done long before this weekend: almost everything we run is documented, and updates are code routines rather than tribal knowledge. An inventory you can grep beats any emergency heroics — the weekend was spent patching, not searching. That discipline is also what I try to carry into my own infrastructure.