Post Views: 5

Andres Freund, a Microsoft engineer, was doing routine testing on a Linux system when he noticed something strange. His investigation into this unusual behavior led him to discover one of the most significant backdoors in the history of open source software. This is the true story of the discovery.

This article simplifies the technical aspects of the incident, but if you are interested in the technical details, please explore further.

What is Open Source Software?

Open source software is software whose source code is publicly available. This allows any developer to review code, identify vulnerabilities, suggest improvements, or contribute to the project. However, not everyone can update the code immediately. Most open source projects are hosted on platforms like GitHub, where trusted individuals called “maintainers” review contributions and decide which changes are safe to incorporate into the main source code.

Project: XZ-Utils

This story revolves around an open source project called “XZ-Utils”, a compression tool widely used in Linux systems. While Windows users typically rely on ZIP files for compression, Linux systems often use XZ. XZ-Utils maintainer Lasse Collin manages the project in his spare time, as it is not his full-time job. Unfortunately, its limited availability creates opportunities for attackers.

Attack Background

Two contributors, Kumar and Eng, were frustrated with the slow pace of change in the XZ-Utils project and Collin’s slow response. They expressed their dissatisfaction and even suggested that a new administrator might be needed. Aware of his struggles, Collin apologized, admitting that he was overwhelmed by the workload as he maintained the project as an unpaid hobby.

Needing help, Collin identified an active contributor named Jia Tan, who seemed like a good fit for the managing role. Unbeknownst to Collin, this was part of a larger plan designed by Kumar, Eng, and Tan. Their goal is to gain control of the project.

In June 2022, Jia was promoted to core manager of XZ-Utils. Rather than immediately launching an attack, Jia patiently built trust over time by fixing bugs and making legitimate contributions. Eventually, he managed to configure the project so that security alerts were routed to him instead of Collin.

Attack

In February 2024, Jia launched his attack. Rather than inserting malicious code directly into the source code, it takes a more covert approach. Because XZ-Utils is an open source project, anyone can inspect the code, and if malicious changes occur to the main codebase, they will be detected quickly. Instead, Jia hid his backdoor in test files that were typically less supervised.

Soon after, various Linux distributions, such as Debian, Kali, and MicroOS, started updating their systems with the latest versions of XZ-Utils. However, this update was initially rolled out in the unstable branch, which is used for testing purposes. After thorough testing, updates are usually promoted to the stable version used in production environments.

Invention

This is where our hero, Andres Freund, enters the story. When running microbenchmarks on a Debian Linux system with the new XZ version, he noticed something strange—the SSHD process (which handles secure shell connections for remote access) was consuming an unusually large amount of CPU resources. Even more concerning, SSH connections take about half a second longer than they should. While a half-second delay may not seem like much, in the world of computing, it is a significant red flag.

Andres investigated further and discovered that a backdoor had been cleverly inserted (malware that allows unauthorized access to the system by bypassing the normal authentication process). Realizing the gravity of the situation, he immediately reported the vulnerability to the Debian team. If undetected, these backdoors can compromise large portions of the Linux ecosystem, leaving many systems vulnerable to attack.

After that

After Andres published his findings, the cybersecurity world was stunned. The nightmare scenario is that if the backdoor goes unnoticed, countless Linux systems could be compromised. Affected Linux distributions were quickly reverted to previous secure versions of XZ-Utils. Investigators soon traced the malicious code back to Jia Tan, but it became clear that Jia was not working alone.

Interestingly, Kumar, Eng, and Jia Tan are just GitHub usernames, and it remains unclear whether they were individual attackers or part of a larger hacking group. Until now, the true identity behind the attack remains a mystery.