One Upgrade Away: What the LiteLLM Compromise Teaches About Common-Cause Failure

On October 21, 2026, a malicious release of LiteLLM, a widely used open-source gateway for calling AI models, was published to PyPI and Docker. Version 1.107.0 was available on PyPI for about one hour and 44 minutes and on Docker for about two and a half hours, including under the tags "latest" and "main-stable." Socket estimates the package sees around 23 million downloads a week. It was the project's second supply chain compromise this year.

The payload was built to steal the things developers keep closest: cloud credentials, SSH and GPG keys, Git credentials, browser sessions, shell history, and the authentication files of AI coding tools. It also wrote itself into Python's startup files, so it would run every time Python started in that environment, whether or not LiteLLM was ever imported again.

Security teams will rightly focus on indicators of compromise. This article looks at a different question: how did a malicious build get through a release process that required a second person's approval?

The Case Study: How the Release Got Out

According to LiteLLM's account, as reported by Help Net Security, the attacker held stolen GitHub CLI OAuth tokens belonging to three of the project's engineers:

1. One account modified the release pipeline and triggered the release
2. A second account approved it
3. A third deleted workflow records, removing evidence investigators would need

The project believes the tokens were taken from the logs of an internal deployment that was exposed to the internet, possibly through a recently patched vulnerability. The investigation is ongoing. Researchers at Wiz later observed the attacker validating stolen AWS credentials from compromised CI runners and Kubernetes nodes, then checking access to AI models, roughly four and a half hours after data left victim systems.

One group was not affected: enterprise customers who installed from LiteLLM's private registry.

The Control That Looked Like Redundancy

A two-person release rule is meant to be redundancy. Two independent humans must both agree before code ships. In a safety analysis, that is exactly how it would be modelled: two barriers, each of which must fail.

But the two barriers were not independent. Both depended on the same thing: GitHub tokens that could be stolen from the same place. Once the attacker had them, "two people" was one attacker clicking twice.

Aerospace and nuclear engineers call this common-cause failure. Two hydraulic systems routed through the same section of fuselage are not two systems if one event can sever both. Dual sensors powered from the same bus are one sensor when the bus fails. The fix is never "add a third approver." The fix is to find the shared dependency and break it, for example with hardware-bound signing keys that cannot be lifted from a log file.

Software teams rarely do common-cause analysis, because their risk reviews, when they happen at all, look at components one at a time.

How FMEA Could Have Prevented This

FMEA would not have stopped the theft of tokens. It would have asked two questions earlier:

• What do our "independent" release controls have in common? If the answer is "they all trust a GitHub session," that shared dependency becomes its own failure mode with its own severity rating
• What happens downstream when a trusted dependency turns hostile? For consumers of LiteLLM, the exposure was not "a library had a bug." It was "a library ran with the full permissions of our CI runners and developer laptops"

For organisations that consume open-source packages, the second question is where the work is. Every unpinned dependency is a component you have agreed to replace automatically with whatever its publisher, or anyone holding its publisher's tokens, ships next.

Applying the AIAG-VDA 7-Step FMEA Process

1. Step 1: Planning and Preparation – Scope the analysis to "build and release," for your own software and for the third-party packages your builds pull in.
2. Step 2: Structure Analysis – List the release pipeline as a structure: source repository, CI runners, signing, approval, registry, consumers. Add critical third-party packages as elements, especially those that handle credentials, like an AI gateway that holds every model provider key.
3. Step 3: Function Analysis – State what each element is trusted to do. A CI runner "holds cloud credentials and executes any code in the build." That sentence alone raises the severity of everything it installs.
4. Step 4: Failure Analysis – Include hostile failure modes, not only accidental ones: "dependency replaced with malicious version," "approver identity used by someone else."
5. Step 5: Risk Analysis – Rate detection honestly. If your only control is that a researcher might notice and post about it, detection is poor.
6. Step 6: Optimization – Pin versions with hashes, install from a vetted internal mirror, separate build credentials from developer credentials, and require hardware-backed signing for releases.
7. Step 7: Results Documentation – Keep the analysis with the system. When the next compromised package is announced, you should be able to answer "were we exposed?" from your records in minutes.

Key Failure Modes to Document

Unpinned Dependency Pulls a Malicious Release

• Function: Supply a third-party library to builds and runtime environments
• Failure Mode: Floating version or "latest" tag resolves to a trojanized release
• Effects: Attacker code runs with the permissions of CI runners, servers and developer machines; credentials exfiltrated
• Causes: Version ranges or mutable container tags; installs straight from the public registry
• Current Controls: Public registry takedown after discovery, measured in hours
• Recommended Actions: Pin exact versions with hashes; install through an internal mirror with a short quarantine period for new releases; treat mutable tags such as "latest" as forbidden in production builds

Release Approval Defeated by a Shared Dependency

• Function: Require two independent people to approve each release
• Failure Mode: Both approvals performed by an attacker holding both approvers' tokens
• Effects: Malicious release shipped with every appearance of legitimacy; audit records deleted
• Causes: Approvals bound to reusable session tokens; tokens exposed in application logs
• Current Controls: Two-person rule (not independent in practice)
• Recommended Actions: Require hardware-bound keys for release approval and signing; scrub tokens from logs; alert on approvals from new devices or locations; protect workflow history from deletion

Removal Does Not Remove the Compromise

• Function: Restore a clean environment after a bad package is discovered
• Failure Mode: Package upgraded or uninstalled, but persistence in Python startup files keeps executing
• Effects: Teams believe they are clean while credentials continue to leak
• Causes: Remediation defined as "upgrade the package"; no rebuild from a known-good image
• Current Controls: Package upgrade
• Recommended Actions: Define remediation as rebuild plus full credential rotation for any environment that installed the bad version; verify with the published indicators of compromise

From Failure Modes to Requirements

Each recommended action becomes a traceable requirement, for example:

• "Production builds shall install third-party packages only at pinned versions with verified hashes." Verified by an automated build check.
• "Release approval shall require a hardware-bound credential that cannot be exported." Verified by configuration inspection.
• "Any environment that executed a compromised package shall be rebuilt from a known-good image and all its credentials rotated." Verified by an incident checklist.

Conclusion

The LiteLLM compromise is a software story with a very old engineering lesson. Redundancy only protects you when the redundant parts cannot fail for the same reason. Two approvers who can both be impersonated with stolen tokens are one approver. And a dependency you upgrade automatically is a component whose design you have handed to someone else.

NirmIQ brings the discipline safety-critical industries use for this, structure analysis, AIAG-VDA failure analysis and requirement-to-test traceability, to the software supply chain. If your risk register has never asked what your "independent" controls share, start that analysis today.

Further reading:

• FMEA for Software Teams: What the Knight Capital Disaster Teaches Us
• The Railway Breach: Your Secrets Store Is a Component. Is It in Your FMEA?

Sources: Help Net Security, "LiteLLM compromised in supply chain attack" (October 22, 2026), citing LiteLLM, Wiz, Socket and Aikido; The Hacker News and SecurityWeek coverage of the same incident. Figures are as reported at the time of writing and may change as the investigation continues.