---
title: "LiteLLM Supply Chain Breach: What to Check and Rotate"
description: "The LiteLLM supply chain breach exposed 434,000 CI/CD pipelines. How to check if you're affected, what to rotate first, and why rotation alone fails."
canonical: "https://axipro.co/litellm-supply-chain-breach/"
language: "en-US"
modified: "2026-08-14T03:33:11+00:00"
generator: "WordPress 7.1.1"
---

[Home](https://axipro.co)

/ [AI Security](https://axipro.co/category/ai-security/), [ISO 42001](https://axipro.co/category/iso-42001/)

/ LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

# LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

![Picture of Pedro Dias](https://axipro.co/wp-content/uploads/2026/05/pedro-passport-picture-scaled.jpg)

- Pedro Dias
- August 14, 2026

Copy Link

Two compromised versions of LiteLLM sat on PyPI for roughly 40 minutes on the morning of March 24, 2026. That window was enough to capture secrets from around **434,000 CI/CD pipeline runs across nearly 2,500 organizations**, including AWS, Samsung, Cisco, Salesforce, Siemens, and Deloitte. In August, researchers at CloudSEK and Hudson Rock confirmed they had obtained the raw exfiltrated data: a 153GB archive containing 433,909 files of environment variables, cloud keys, Kubernetes secrets, and API tokens harvested live from running pipelines, as covered by [Help Net Security’s reporting on the credential archive](https://www.helpnetsecurity.com/2026/08/13/litellm-breach-stolen-credentials-leak/).

If LiteLLM runs anywhere in your stack, or you touch any AI proxy infrastructure at all, you need answers to three things: whether you were exposed, what to rotate first, and whether the rotation you did back in March actually held. That last one matters more than it sounds, because “we rotated everything” has already burned at least one very large company.

## How the Breach Happened

The attack didn’t start with LiteLLM. On March 19, 2026, a threat group called **TeamPCP** compromised the build pipeline of Trivy, a vulnerability scanner half the industry runs, and pushed a poisoned release. LiteLLM’s own CI pipeline ran Trivy, so the poisoned scanner had legitimate read access to the project’s runner environment. The attackers used that to steal LiteLLM’s PyPI publishing tokens and ship two malicious releases of their own: **versions 1.82.7 and 1.82.8**. KICS and the Telnyx Python SDK got hit in the same campaign.

The payload design is the part worth studying. The malicious package dropped a *.pth* startup hook into site-packages, so the code ran the moment any Python interpreter started on the machine, **whether or not anything imported LiteLLM**. From there it harvested environment variables, read local credential files like .aws/credentials and .kube/config, tried to move laterally across Kubernetes clusters, and installed a systemd backdoor dressed up as a generic telemetry service. [InfoQ’s coverage of the PyPI compromise](https://infoq.com/news/2026/03/litellm-supply-chain-attack/) put downloads of the compromised release above 40,000. For scale, LiteLLM normally gets downloaded around 3 million times a day.

The exfiltration had a nasty fallback, too. According to CloudSEK, stolen data was encrypted and sent to a typosquatted domain, and when that failed, the malware created a public repository inside the victim’s own GitHub account and uploaded the loot as a release asset. Some companies were publishing their own secrets to the open internet and had no idea.

**Worth Knowing:** The malicious code only existed in the PyPI artifacts. The GitHub source repository stayed clean the whole time, so a developer reviewing the code on GitHub saw nothing wrong. Source review isn’t artifact verification. If you don’t check that what the registry serves matches the upstream source, this class of attack is invisible to you.

## How to Check If You Were Exposed

Three checks, from quickest to most involved.

### 1. Confirm whether the compromised versions ever ran

The malicious versions went live on PyPI at 10:39 UTC on March 24, 2026 and got quarantined about 40 minutes later. The project’s advice: treat any install from that day before 16:00 UTC as suspect. Search your lockfiles, pip caches, SBOMs, and container image histories for 1.82.7 and 1.82.8. And check your internal artifact mirrors. An Artifactory or Nexus proxy that cached the bad release in March can keep serving it internally long after PyPI pulled it.

Keep the .pth mechanism in mind when you scope this. The question isn’t “which applications import LiteLLM,” it’s “which machines had the package installed at all,” because every Python process on an infected machine triggered the payload.

### 2. Hunt for persistence

Rotation is pointless if the attacker still has a foothold. Check developer machines, CI runners, and containers for unauthorized .pth files in site-packages and for suspicious systemd units, especially anything posing as a system telemetry service. And review activity from March 24 onward, not just the 40-minute window. Persistence is there so the access outlives the infection.

**Pro Tip:** Don’t limit the persistence hunt to live machines. Base container images rebuilt in late March may have baked the payload into every image derived from them since. Scan your image registry for the affected LiteLLM versions and for unexpected .pth files, then trace which running workloads came from flagged images.

### 3. Check whether your secrets are in the dump

Hudson Rock has published a domain lookup tool and is running ethical disclosures for affected organizations, and CloudSEK maintains a high-confidence victim list. Use them, but know their limits. Attribution in this dataset is genuinely hard. One dump with a siriusxm.com committer email actually traced, through its self-hosted GitLab endpoints, to AdsWizz, a SiriusXM subsidiary. And a large share of the dumps are generic pipeline configurations with **no identifying domain, email, or server name at all**. Absence from a victim list is not evidence of absence. If your pipelines ran the compromised versions, assume exposure no matter what a lookup tool tells you.

## What to Rotate, in What Order

The guidance from both research teams is blunt: treat every secret the LiteLLM environment could reach as compromised. That covers secrets on disk, in memory, injected into CI jobs, and anything retrievable through instance metadata services. Work down by blast radius:

| Priority | Credential type | Why it comes first |
| --- | --- | --- |
| 1 | Cloud IAM keys (AWS, GCP, Azure) | Direct control of infrastructure, data stores, and billing. This is where attackers monetize fastest. |
| 2 | GitHub and GitLab PATs, package publishing tokens | These let an attacker poison your releases and turn your company into the next link in the supply chain. |
| 3 | Kubernetes service account tokens and kubeconfigs | Lateral movement across clusters was built into the payload, not a theoretical risk. |
| 4 | Database passwords and third-party API keys | Dumped in plain text in the archive, often with no attribution, so nobody will warn you they leaked. |
| 5 | AI provider API keys | Billing abuse, quota theft, and access to whatever data flows through your LLM routing layer. |

One word matters more than the rest of this article: **revoke, don’t just rotate**. That distinction is exactly how this breach grew in the first place. After the initial Trivy compromise, its maintainers rotated an automation token but didn’t fully revoke the old one for around 20 days, which handed the attackers a three-week window to force-push malicious code into third-party builds.

**Important:** Rotating a credential creates a new one. Revoking kills the old one. Plenty of platforms let both exist side by side, and CI systems, long-lived sessions, and cached tokens will happily keep honoring the old secret. After every rotation in this incident response, verify the old credential actually fails, then check audit logs for anything that used it after March 24.

This isn’t theoretical. Independent researcher Kevin Beaumont tested credentials from the dump, months after the attack, against an organization whose disclosure policy allowed it. The company had told him the leak was old news because everything had been rotated. His verdict: *“Almost every one worked.”* That was one of the largest US tech companies, not some resource-starved startup.

Two more steps before you call it done. Review AWS CloudTrail and Kubernetes API audit logs for anomalous activity going back to March 24, 2026, and put strict egress filtering on runner environments so the next payload has nowhere to send your secrets.

## What This Incident Says About AI Supply-Chain Controls

The uncomfortable part is that this wasn’t an AI failure. It was a DevOps failure, made worse by how fast teams have bolted AI tooling onto their pipelines. As Beaumont put it, the teens behind TeamPCP ran circles around organizations obsessed with shipping AI while their build security lagged behind. A [supply chain attack](https://en.wikipedia.org/wiki/Supply_chain_attack) of this shape is now the default threat model for any company whose product depends on open source. Which is every company.

The structural fixes aren’t new either. [NIST’s Secure Software Development Framework](https://axipro.co/nist-ai-rmf-1-0/) has warned for years that build pipelines are the soft underbelly of modern companies, and its practices map directly onto what failed here: verified artifact integrity, least-privilege pipeline credentials, and egress controls on build environments. The compliance frameworks are catching up too. SOC 2’s Trust Services Criteria added explicit focus areas on software supply chain security and vendor risk in the 2022 revision, and ISO 42001 extends that discipline to the AI layer: how you govern the AI tooling and third-party models your business now depends on. If this incident is what finally puts AI governance on your roadmap, [Axipro’s ISO 42001 implementation services](https://axipro.co/step-by-step-iso-42001-implementation-guide-axipro/) cover exactly this territory, from supplier controls to auditable AI asset inventories.

Expect the questionnaires to change as well. Enterprise buyers already probe vendors on SOC 2 and ISO 27001. After a breach that turned an AI proxy into a credential vacuum, questions about AI dependencies, artifact verification, and pipeline secret scoping are coming. Our guide to [security certifications for AI agent vendors](https://axipro.co/secure-ai-agent-vendor-certifications/) maps what buyers now ask for, and the pattern rhymes with what we covered in the [May 2026 GitHub breach](https://axipro.co/github-breach-may-2026/): developer tooling has become the highest-value target in most companies, and the security model around it hasn’t kept pace.

**Insider Note:** [REVIEW: confirm or replace with a real Axipro observation] In the gap analyses we run before ISO 27001 and SOC 2 engagements, over-scoped CI/CD credentials are the single most common finding: runners that build a marketing site holding organization-wide cloud admin keys, because scoping them properly was someone’s someday task. Every one of those environments would have leaked its full keyring in this breach. Least-privilege pipeline secrets are tedious to set up and boring to maintain, which is exactly why attackers count on you not doing it.

There’s a testing angle here as well. A payload that moved laterally through Kubernetes and planted systemd persistence is the kind of attack path a good offensive exercise should catch before a real adversary does. If your last test predates your AI infrastructure, our breakdown of [what auditors expect from ISO 27001 penetration testing](https://axipro.co/iso-27001-pentesting/) is a practical place to start scoping one.

The LiteLLM breach compressed the whole modern supply-chain threat into 40 minutes: one upstream compromise, hundreds of thousands of exposed pipelines, and secrets that stayed live for months because rotation got treated as a checkbox. Check whether the bad versions ever touched your environment, hunt for persistence before you rotate, revoke rather than rotate, and verify the old credentials are actually dead. Then fix what made the blast radius so big in the first place: pipelines holding far more privilege than the jobs they run ever needed.

## Frequently Asked Questions

### Which LiteLLM versions were compromised?

Versions 1.82.7 and 1.82.8, published to PyPI at 10:39 UTC on March 24, 2026 and quarantined roughly 40 minutes later. The project advises treating any LiteLLM install from that day before 16:00 UTC as suspect, and internal package mirrors may have cached the bad releases well past that window.

### My company is not on the victim list. Are we safe?

Not necessarily. Researchers could only attribute dumps that contained identifying markers like domains or committer emails, and a big share of the archive is generic pipeline data with no attribution at all. If your systems ran the compromised versions, assume your secrets are in the dataset and act on it.

### We rotated our credentials in March. Is that enough?

Only if the old credentials were fully revoked and you’ve verified they no longer work. One major tech company believed it had rotated everything, and a researcher found nearly all of its leaked credentials still worked months later. Test the old secrets, then audit logs for any use of them after March 24, 2026.

### We never import LiteLLM directly. Does this still affect us?

Possibly. LiteLLM shows up as a transitive dependency in a lot of AI stacks, and the malicious code ran through a .pth startup hook whenever any Python interpreter started on an infected machine, with no import required. Audit dependency trees and installed packages, not just your direct requirements files.

Axipro Author

![Picture of Pedro Dias](https://axipro.co/wp-content/uploads/2026/05/pedro-passport-picture-scaled.jpg)

### Pedro Dias

Pedro has been writing online for over 10 years. With experience in all things programming, cyber security, and compliance, he is our editor-in-chief at Axipro.

- August 14, 2026
- [AI Security](https://axipro.co/category/ai-security/), [ISO 42001](https://axipro.co/category/iso-42001/)

Copy Link

## Blog Highlights

## Explore More Articles

[Read More Blogs](https://axipro.co/blog/)

- [All Blog](https://axipro.co/category/blog/), [Customer Stories](https://axipro.co/category/stories/), [Denmark](https://axipro.co/category/denmark/), [ISO-27001](https://axipro.co/category/iso-27001/)

- September 19, 2026

#### [How Haime got through its first ISO 27001 internal and external audits in under four weeks with Axipro](https://axipro.co/haime-iso-27001-internal-external-audit/)

Haime, a Danish AI governance software company, completed independent ISO 27001 internal and external audits with Axipro in under four weeks in 2026.

[Read more](https://axipro.co/haime-iso-27001-internal-external-audit/)

- [ISO-9001](https://axipro.co/category/iso-9001/)

- September 18, 2026

#### [ISO 9001:2026 Changes: What’s New and How to Transition](https://axipro.co/iso-9001-2026-changes/)

ISO published ISO 9001:2026 on September 16, 2026, and the 2015 edition is now formally withdrawn. If you hold a certificate, the good news is that the structure and the process approach are the same, and the list of new requirements is short. Top management now has to promote a quality culture and ethical behavior. Risks and opportunities get handled separately, change management carries more weight, and the 2024 climate change amendment sits inside the core text. That’s most of it. Below, we go through each change clause by clause, cover what stayed where it was, set out the transition timeline, and list the work a certified company has to do before the deadline. Key Takeaways ISO 9001:2026 is the sixth edition of the standard and replaces ISO 9001:2015. Most of the new text is guidance, and only a small part of it adds requirements. The changes that carry audit weight are in Clause 5.1 (quality culture and ethical behavior), Clause 6.1 (risks and opportunities addressed separately), and Clause 6.3 (planning of changes). ISO 9001:2015 certificates stay valid during the transition period, which is expected to run for three years, until around September 2029. Your certification body confirms the exact date. Certification bodies need their own accreditation to the new edition before they can issue 2026 certificates, so nobody has to panic this quarter. A healthy 2015 system needs a gap analysis, some document updates, and better leadership evidence. You won’t have to rebuild it. ISO 9001:2026 Is Now Published: Where the Revision Stands On September 16, 2026, ISO announced the publication of ISO 9001:2026. ISO describes the edition as a set of targeted updates that make the standard clearer and easier to use, built on the framework more than one million organizations already work with. The official ISO 9001:2026 standard page is live. ISO’s page for ISO 9001:2015 now marks that edition as withdrawn and tells certified organizations to speak to their certification body about transition arrangements. It took longer to get here than planned. ISO’s quality committee first voted to leave the 2015 edition alone, then changed its mind in August 2023 after wider consultation. The Draft International Standard followed in August 2025, the final draft went to ballot in spring 2026, and publication hit the September target. Two companion documents came out earlier in the year. ISO 9000:2026, the fundamentals and vocabulary standard, was published in May 2026, and ISO 19011:2026, the auditing guideline, was updated around the same time. If your internal audit procedure cites either one by year, add it to the update list. Why ISO 9001:2015 Was Revised Eleven years is a long time for a management standard. Since 2015, supply chains have become more fragile, remote, and hybrid work has changed how processes run, and customers ask harder questions about ethics and data integrity than they used to. ISO reviews its standards on a regular cycle, and in 2023 the consensus was that a revision would be worth the effort. According to ISO/TC 176/SC 2, the subcommittee responsible for ISO 9001, 81 experts from 46 countries and liaison bodies took part. The result is still conservative, and that was a choice. A standard with a million-plus users can’t afford a rewrite every decade, so the committee went for clarification. ISO 9001:2026 vs ISO 9001:2015: Summary of Changes Area ISO 9001:2015 ISO 9001:2026 Structure Annex SL high-level structure, Clauses 4 to 10 Same clause layout, updated to the latest Harmonized Structure Clause 3, terms Points entirely to ISO 9000 Includes a limited set of core terms; ISO 9000:2026 remains the normative reference Climate change Added by Amendment 1 in 2024 Built into Clauses 4.1 and 4.2 Leadership (5.1) Commitment to the QMS and customer focus Adds promotion of quality culture and ethical behavior Risks and opportunities (6.1) Addressed together Addressed separately, with distinct actions for each Planning of changes (6.3) Brief requirement Reinforced to protect intended results Annex A Short clarification of structure and terms Expanded guidance on the intent of requirements, informative only Annex B Listed other ISO/TC 176 standards Removed; references moved to Annex A and the committee website Key Changes in ISO 9001:2026, Clause by Clause Clause 3: Core Terms Now Sit Inside the Standard The 2015 edition sent readers to ISO 9000 for every definition. The 2026 edition brings a limited number of core management system terms into Clause 3 itself, and ISO 9000:2026 remains the normative reference for the full vocabulary. There’s nothing to set up here. Just check that your quality manual and procedures don’t cite definitions by their old source or year. Clause 4: The Climate Change Amendment Is Now Core Text In February 2024, ISO amended every major management system standard. Organizations had to determine whether climate change is a relevant issue (4.1) and whether interested parties have related requirements (4.2). That amendment took effect immediately, with no transition period, and ISO 9001:2026 folds the same text into the body of the standard. If you handled the amendment properly in 2024, you have nothing new to do. If you wrote “not applicable” on a sticky note, go back to it, because auditors will now read this as a standing requirement. Not relevant is a perfectly acceptable conclusion for many businesses, as long as there’s a reason written down behind it. Clause 5.1: Quality Culture and Ethical Behavior Become Leadership Duties This is the change everyone is talking about, and it’s the hardest one to evidence. Top management now has to show leadership by promoting a quality culture and ethical behavior. The same themes turn up in the requirements for awareness (7.3) and the environment for the operation of processes (7.1.4). You don’t need a culture program for this, and you don’t strictly need a new code of conduct, although one helps. What the auditor wants is for top management to show what they do day to day. Management review minutes where quality problems get discussed without blame are good evidence. So is a working route

[Read more](https://axipro.co/iso-9001-2026-changes/)

- [ISO 42001](https://axipro.co/category/iso-42001/), [SOC-2](https://axipro.co/category/soc-2-2/)

- September 17, 2026

#### [AI Agent Audit Log Requirements: ISO 42001 & SOC 2](https://axipro.co/ai-agent-audit-log-requirements-iso-42001-soc-2/)

An AI agent reads a customer record, decides a refund is warranted, and calls the payments API. The trail it leaves looks nothing like a human doing the same job. The log says a user logged in, a service account made three API calls, and the transaction cleared. It doesn’t say why the agent decided on a refund, what it read first, which model version did the reasoning, or who gave the agent permission to act in the first place. That missing “why” is the whole audit problem. This article covers what ISO/IEC 42001:2023 and the SOC 2 Trust Services Criteria expect from AI agent audit logs, where the two overlap, the fields a log needs to satisfy both, how long to keep records, what you shouldn’t record, and how to package it all for an auditor. It’s written for the CTO, platform lead, or founder who owns compliance for a product that now ships with autonomous agents and needs a certification and a Type II report without running two separate logging programs. The Compliance Gap: Traditional Application Logs vs. AI Agent Audit Logs Why Standard Logs Fall Short for Autonomous Agents Application logs were built for deterministic software. Same input, same state, same output, so recording the input, the state change, and the result is enough to reconstruct what happened. A SOC 2 auditor sampling access logs can trace a database write back to a login, a role, and a change ticket without much effort. Agents break that chain in a few places. They usually run under a shared service account or a borrowed OAuth token, so the log pins the action to a machine identity with no link to the human who set the task. The action itself was picked at runtime by a model rather than fixed in code, so there’s no source line to point at. The same prompt can produce a different tool call tomorrow, so a single sampled log entry proves almost nothing about how the system behaves in general. The Shift from Deterministic State Logging to Intent and Reasoning Capture Traditional logs answer “what changed.” Agent audit logs also have to answer “what was the agent trying to do, what did it consider, and what held it back.” That means capturing the task as delegated, the context the model was handed, the reasoning or planning steps it produced, the tools it picked and the arguments it passed, and every point where a guardrail stepped in. The unit of audit moves from the event to the decision, and each decision needs enough surrounding context that a reviewer can judge whether it was reasonable. Unique Audit Challenges of Non-Deterministic AI Behavior Non-determinism is the part auditors struggle with most. In a normal control test, the auditor re-performs the control and expects the same result. Re-run the same input through an agent and you may get a different path. The practical answer is to stop trying to prove that any single output was correct and instead prove that every output was recorded, attributed, bounded by policy, and reviewable. Logs show that the management system works. They don’t show the model is infallible, and nobody expects them to. ISO 42001 accepts this framing outright. SOC 2 auditors are still catching up, and you’ll spend some time educating them. Insider Note: Auditors don’t expect you to explain the model’s weights. They expect you to show that when the agent did something unexpected, you could find it, see what it read, see what it did, and see who was accountable. Frame every logging decision around that reconstruction test. What ISO 42001 Requires for AI Agent Audit Logs ISO/IEC 42001:2023 is the certifiable standard for an AI Management System (AIMS). It follows the same Plan-Do-Check-Act structure as ISO 27001 and comes with 38 Annex A controls. The phrase “audit log” barely appears in it, but logging obligations run through the main clauses and at least three Annex A areas. Our ISO 42001 certification services map these to your existing controls where possible. Clause 8: Operational Logging and Documentation Requirements Clause 8 asks you to plan, run, and control the processes needed to meet your AI requirements, and to keep documented information showing those processes ran as planned. For an agent in production, the process is the runtime behavior, so documented evidence means logs of the agent operating, not a procedure document on its own. Clause 8.4 adds an AI system impact assessment whose results you have to retain. When an agent’s scope or toolset changes, the record of that change and the updated assessment are both Clause 8 evidence. Clause 9: Performance Evaluation and Evidence of Monitoring Clause 9.1 asks you to decide what to monitor and measure, how, and when, and to keep evidence of the results. An auditor will want the monitoring you defined for each agent (error rates, guardrail block rates, tool-call anomalies, how often humans override) and the records showing you reviewed it. Clause 9.2 internal audit and 9.3 management review both feed off those records. Without operational logs, there’s nothing to measure, and Clause 9 falls over. Annex A.6: AI System Lifecycle Logging Obligations Annex A.6 is where logging gets explicit. A.6.2.8, AI system recording of event logs, requires you to decide at which phases of the AI system lifecycle event logging is switched on, and the Annex B guidance ties this to traceability and anomaly detection. A.6.2.6, AI system operation and monitoring, requires ongoing monitoring in operation, including AI-specific threats like data poisoning and model theft. Read together, they mean logging can’t start at go-live. Design decisions, validation runs, deployment configs, and production behavior all need a record. Annex A.9: Logging Requirements for AI System Operation Annex A.9 covers responsible use: processes for responsible use (A.9.2), objectives for it (A.9.3), and intended use (A.9.4). The logging consequence is that you need to show the agent stayed inside its intended use. That takes logs of the tasks it was given, the actions it took,

[Read more](https://axipro.co/ai-agent-audit-log-requirements-iso-42001-soc-2/)

WhatsApp us