Last week, the world narrowly escaped what could have been one of the largest and most crippling cyber attacks in history. And yet, there has been minimal media coverage of what (almost) happened.
I’m speaking about the xz Utils backdoor.
Before I jump in to what happened: A quick plug to read Rachael Newberry’s thought piece on how close we are to truly personalized medicine. Then join her Tuesday, April 16, at Cary Founded to continue the discussion with local companies that are enabling a better future for health.
What is xz Utils?
xz Utils is a relatively small piece of code used for data compression and decompression. This is a common basic operation that software utilizes in nearly every application imaginable. xz Utils is significant in that it is lossless (data quality is robust) and it is generalized (not specific to only a single type of data). As such, xz Utils has become a core component of all major Linux operating systems and nearly all Linux-like systems.
How important is Linux?
Linux is an open source operating system, first released in 1991. It became wildly popular and is the most widely used operating system today, with numerous variants and derivatives. All Android phones and tablets utilize Linux (84% of mobile devices globally). Between half and two-thirds of cloud servers are Linux-based. Even competing proprietary operating systems like Microsoft Azure have a third of implementations augmented by Linux tools, and 100% of Apple’s iOS devices leverage Linux for elements of their operation. The majority of new, “smart” products – our tv’s, appliances, cars and wearables – are built with a linux operating system.
The point – the xz Utils data compression code is built into the operating system of just about everything.
So what happened?
About a week ago, it was discovered that a backdoor had been integrated into xz Utils. A backdoor is a segment of code that allows anyone to surreptitiously insert malware into a system or to extract sensitive data from it. More specifically, this backdoor was accessible via another piece of code called SSH, which is the most common code used to enable remote access to a device.
It appears this hack was years in the making. An open source contributor named Jia Tan (JiaT75 on GitHub) had been making code recommendations and implementations since 2021. More recently, he persistently requested and eventually was granted repository maintainer authority. He quietly added, obfuscated and approved the malicious backdoor into the primary xz Utils codebase.
In February of this year, Tan implored the major Linux operating systems to integrate this new revision. It was perhaps days away from being implemented into the two most widely distributed Liinux platforms – Red Hat and Debian.
Through good fortune, amazing diligence and a bit of luck, a single Microsoft developer discovered the backdoor and announced it in the proper open source forum. Once verified, the issue was patched quickly, and a massive threat was averted.
How much danger were we really in?
The nature of this vulnerability would have allowed anyone with knowledge of the backdoor to hack into just about any system we can imagine, without detection or a digital forensic path back to the origin after the fact. Because the backdoor was implemented via SSH, the hacker could operate remotely, from a position of relative safety.
They could have taken sensitive data from banking systems, healthcare providers or security databases. They could have inserted malicious code into industrial operations, civic infrastructure and defense systems. This hack would have been far more widespread than well-known hacks like Wannacry, which impacted more than 100,000 companies in 150 countries. This could have impacted millions of companies, causing trillions of dollars in damage.
Don’t think only about data breaches or ransomware attacks. This hack could have allowed malicious changes to public safety systems, critical infrastructure and emergency response. It would have opened the door to bypassing national security and altering military defense.
The bigger danger wasn’t the technical hack, but the social one
This kind of hack should have been recognized and caught before it got approved. So why didn’t it?
Like Linux, xz Utils is open source. This means that the code is available publicly and anyone can suggest changes and improvements over time. Most open source software is housed in code repositories like GitHub, and are managed by repository “maintainers”. Almost universally, these maintainers are volunteers who review code improvement suggestions from millions of open source contributors, and implement the best updates.
The theory, which has been proven time and time again, is that the intelligence of the crowd will result in a higher quality product than reliance on a small number of “experts”. By coding in the open, far more people can review code for errors and vulnerabilities. And through reputation and trust, also established in the open, maintainers oversee that high quality.
The problem is that nearly all maintainers are doing so on a volunteer, part-time basis while holding down their own careers, families and while managing competing priorities for their time and attention. Even the most fundamentally critical bits of code – like xz Utils – are often overseen by just one or two people, with very little recognition and almost never any financial compensation, for their effort.
This creates a massive ‘human vulnerability,’ as we are all prone to error from time to time. These “single points of failure” are susceptible to social engineering, as appears to have occurred in this case. Jia Tan, through a combination of feints (good code contributions) and persistence, eventually gained maintainer status. At some point, who is left to watch the watcher?
In a world of generative AI, will we see “AI code contributors” that build trust through positive contributions, seemingly coming from humans who are building trust in their community. Eventually that trust leads to authority and responsibility and a position of power. With so few existing gatekeepers, mostly overloaded and underappreciated, it is easy to see a few fall prey to bribery, blackmail or other means of persuasion to let a small snippet of code get integrated into an approved release.
Where do we go from here?
Open source provides far too many benefits compared to these risks for us to consider moving away from the paradigm. Certainly, better and stronger security verification tools, testing automation and other “tech” will continue to develop, combating bad actors with good tools. But those aren’t architected for this social hacking risk.
I don’t have a silver bullet or magic solution to the risks described here. But one component, I believe, is to increase awareness and to increase support.
It is time for the largest industries that are leveraging all this open source volunteerism to make a larger contribution. The largest tech companies in the world are all built on Linux. Many allow a few of their employees to contribute to open source projects, but it is not a corporate priority, despite the fact that their enterprise offerings would completely fail without those open source contributions.
If an open source contribution is eventually found to contain a vulnerability, big tech certainly would claim no liability for how it may have impacted their product. That should change. If companies leverage the code for free, they should be held financially to account, even if that is the source of a future failure. That would motivate them to more actively contribute and support open source contributors and maintainers, improving overall quality and human resource availability.
Few companies make any significant direct financial contribution back to the open source community. It is time to recognize the critical importance of good people, volunteering their time and expertise to keep the most critical infrastructure in a digital economy working smoothly. Companies that integrate open source segments into their products should financially support the repository maintainers and the thousands upon thousands of contributors.
I don’t have a great mechanism idea for how to implement this – and greatly doubt big tech would listen anyway. But the fact is – despite near-misses like xz Utils, no company can build software better or faster if they tried to do it all themselves.
If you want to learn more about the xz Utils hack, I highly recommend this informative piece by Dan Goodin at ARS Technica.
