/ LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

Two compromised versions of LiteLLM sat on PyPI for roughly 40 minutes on the morning of March 24, 2026. That window was enough to capture secrets from around 434,000 CI/CD pipeline runs across nearly 2,500 organizations, including AWS, Samsung, Cisco, Salesforce, Siemens, and Deloitte. In August, researchers at CloudSEK and Hudson Rock confirmed they had obtained the raw exfiltrated data: a 153GB archive containing 433,909 files of environment variables, cloud keys, Kubernetes secrets, and API tokens harvested live from running pipelines, as covered by Help Net Security’s reporting on the credential archive.

If LiteLLM runs anywhere in your stack, or you touch any AI proxy infrastructure at all, you need answers to three things: whether you were exposed, what to rotate first, and whether the rotation you did back in March actually held. That last one matters more than it sounds, because “we rotated everything” has already burned at least one very large company.

How the Breach Happened

The attack didn’t start with LiteLLM. On March 19, 2026, a threat group called TeamPCP compromised the build pipeline of Trivy, a vulnerability scanner half the industry runs, and pushed a poisoned release. LiteLLM’s own CI pipeline ran Trivy, so the poisoned scanner had legitimate read access to the project’s runner environment. The attackers used that to steal LiteLLM’s PyPI publishing tokens and ship two malicious releases of their own: versions 1.82.7 and 1.82.8. KICS and the Telnyx Python SDK got hit in the same campaign.

The payload design is the part worth studying. The malicious package dropped a .pth startup hook into site-packages, so the code ran the moment any Python interpreter started on the machine, whether or not anything imported LiteLLM. From there it harvested environment variables, read local credential files like .aws/credentials and .kube/config, tried to move laterally across Kubernetes clusters, and installed a systemd backdoor dressed up as a generic telemetry service. InfoQ’s coverage of the PyPI compromise put downloads of the compromised release above 40,000. For scale, LiteLLM normally gets downloaded around 3 million times a day.

The exfiltration had a nasty fallback, too. According to CloudSEK, stolen data was encrypted and sent to a typosquatted domain, and when that failed, the malware created a public repository inside the victim’s own GitHub account and uploaded the loot as a release asset. Some companies were publishing their own secrets to the open internet and had no idea.

Worth Knowing: The malicious code only existed in the PyPI artifacts. The GitHub source repository stayed clean the whole time, so a developer reviewing the code on GitHub saw nothing wrong. Source review isn’t artifact verification. If you don’t check that what the registry serves matches the upstream source, this class of attack is invisible to you.

How to Check If You Were Exposed

Three checks, from quickest to most involved.

1. Confirm whether the compromised versions ever ran

The malicious versions went live on PyPI at 10:39 UTC on March 24, 2026 and got quarantined about 40 minutes later. The project’s advice: treat any install from that day before 16:00 UTC as suspect. Search your lockfiles, pip caches, SBOMs, and container image histories for 1.82.7 and 1.82.8. And check your internal artifact mirrors. An Artifactory or Nexus proxy that cached the bad release in March can keep serving it internally long after PyPI pulled it.

Keep the .pth mechanism in mind when you scope this. The question isn’t “which applications import LiteLLM,” it’s “which machines had the package installed at all,” because every Python process on an infected machine triggered the payload.

2. Hunt for persistence

Rotation is pointless if the attacker still has a foothold. Check developer machines, CI runners, and containers for unauthorized .pth files in site-packages and for suspicious systemd units, especially anything posing as a system telemetry service. And review activity from March 24 onward, not just the 40-minute window. Persistence is there so the access outlives the infection.

Pro Tip: Don’t limit the persistence hunt to live machines. Base container images rebuilt in late March may have baked the payload into every image derived from them since. Scan your image registry for the affected LiteLLM versions and for unexpected .pth files, then trace which running workloads came from flagged images.

3. Check whether your secrets are in the dump

Hudson Rock has published a domain lookup tool and is running ethical disclosures for affected organizations, and CloudSEK maintains a high-confidence victim list. Use them, but know their limits. Attribution in this dataset is genuinely hard. One dump with a siriusxm.com committer email actually traced, through its self-hosted GitLab endpoints, to AdsWizz, a SiriusXM subsidiary. And a large share of the dumps are generic pipeline configurations with no identifying domain, email, or server name at all. Absence from a victim list is not evidence of absence. If your pipelines ran the compromised versions, assume exposure no matter what a lookup tool tells you.

What to Rotate, in What Order

The guidance from both research teams is blunt: treat every secret the LiteLLM environment could reach as compromised. That covers secrets on disk, in memory, injected into CI jobs, and anything retrievable through instance metadata services. Work down by blast radius:

PriorityCredential typeWhy it comes first
1Cloud IAM keys (AWS, GCP, Azure)Direct control of infrastructure, data stores, and billing. This is where attackers monetize fastest.
2GitHub and GitLab PATs, package publishing tokensThese let an attacker poison your releases and turn your company into the next link in the supply chain.
3Kubernetes service account tokens and kubeconfigsLateral movement across clusters was built into the payload, not a theoretical risk.
4Database passwords and third-party API keysDumped in plain text in the archive, often with no attribution, so nobody will warn you they leaked.
5AI provider API keysBilling abuse, quota theft, and access to whatever data flows through your LLM routing layer.

One word matters more than the rest of this article: revoke, don’t just rotate. That distinction is exactly how this breach grew in the first place. After the initial Trivy compromise, its maintainers rotated an automation token but didn’t fully revoke the old one for around 20 days, which handed the attackers a three-week window to force-push malicious code into third-party builds.

Important: Rotating a credential creates a new one. Revoking kills the old one. Plenty of platforms let both exist side by side, and CI systems, long-lived sessions, and cached tokens will happily keep honoring the old secret. After every rotation in this incident response, verify the old credential actually fails, then check audit logs for anything that used it after March 24.

This isn’t theoretical. Independent researcher Kevin Beaumont tested credentials from the dump, months after the attack, against an organization whose disclosure policy allowed it. The company had told him the leak was old news because everything had been rotated. His verdict: “Almost every one worked.” That was one of the largest US tech companies, not some resource-starved startup.

Two more steps before you call it done. Review AWS CloudTrail and Kubernetes API audit logs for anomalous activity going back to March 24, 2026, and put strict egress filtering on runner environments so the next payload has nowhere to send your secrets.

What This Incident Says About AI Supply-Chain Controls

The uncomfortable part is that this wasn’t an AI failure. It was a DevOps failure, made worse by how fast teams have bolted AI tooling onto their pipelines. As Beaumont put it, the teens behind TeamPCP ran circles around organizations obsessed with shipping AI while their build security lagged behind. A supply chain attack of this shape is now the default threat model for any company whose product depends on open source. Which is every company.

The structural fixes aren’t new either. NIST’s Secure Software Development Framework has warned for years that build pipelines are the soft underbelly of modern companies, and its practices map directly onto what failed here: verified artifact integrity, least-privilege pipeline credentials, and egress controls on build environments. The compliance frameworks are catching up too. SOC 2’s Trust Services Criteria added explicit focus areas on software supply chain security and vendor risk in the 2022 revision, and ISO 42001 extends that discipline to the AI layer: how you govern the AI tooling and third-party models your business now depends on. If this incident is what finally puts AI governance on your roadmap, Axipro’s ISO 42001 implementation services cover exactly this territory, from supplier controls to auditable AI asset inventories.

Expect the questionnaires to change as well. Enterprise buyers already probe vendors on SOC 2 and ISO 27001. After a breach that turned an AI proxy into a credential vacuum, questions about AI dependencies, artifact verification, and pipeline secret scoping are coming. Our guide to security certifications for AI agent vendors maps what buyers now ask for, and the pattern rhymes with what we covered in the May 2026 GitHub breach: developer tooling has become the highest-value target in most companies, and the security model around it hasn’t kept pace.

Insider Note: [REVIEW: confirm or replace with a real Axipro observation] In the gap analyses we run before ISO 27001 and SOC 2 engagements, over-scoped CI/CD credentials are the single most common finding: runners that build a marketing site holding organization-wide cloud admin keys, because scoping them properly was someone’s someday task. Every one of those environments would have leaked its full keyring in this breach. Least-privilege pipeline secrets are tedious to set up and boring to maintain, which is exactly why attackers count on you not doing it.

There’s a testing angle here as well. A payload that moved laterally through Kubernetes and planted systemd persistence is the kind of attack path a good offensive exercise should catch before a real adversary does. If your last test predates your AI infrastructure, our breakdown of what auditors expect from ISO 27001 penetration testing is a practical place to start scoping one.

The LiteLLM breach compressed the whole modern supply-chain threat into 40 minutes: one upstream compromise, hundreds of thousands of exposed pipelines, and secrets that stayed live for months because rotation got treated as a checkbox. Check whether the bad versions ever touched your environment, hunt for persistence before you rotate, revoke rather than rotate, and verify the old credentials are actually dead. Then fix what made the blast radius so big in the first place: pipelines holding far more privilege than the jobs they run ever needed.

Frequently Asked Questions

Which LiteLLM versions were compromised?

Versions 1.82.7 and 1.82.8, published to PyPI at 10:39 UTC on March 24, 2026 and quarantined roughly 40 minutes later. The project advises treating any LiteLLM install from that day before 16:00 UTC as suspect, and internal package mirrors may have cached the bad releases well past that window.

My company is not on the victim list. Are we safe?

Not necessarily. Researchers could only attribute dumps that contained identifying markers like domains or committer emails, and a big share of the archive is generic pipeline data with no attribution at all. If your systems ran the compromised versions, assume your secrets are in the dataset and act on it.

We rotated our credentials in March. Is that enough?

Only if the old credentials were fully revoked and you’ve verified they no longer work. One major tech company believed it had rotated everything, and a researcher found nearly all of its leaked credentials still worked months later. Test the old secrets, then audit logs for any use of them after March 24, 2026.

We never import LiteLLM directly. Does this still affect us?

Possibly. LiteLLM shows up as a transitive dependency in a lot of AI stacks, and the malicious code ran through a .pth startup hook whenever any Python interpreter started on an infected machine, with no import required. Audit dependency trees and installed packages, not just your direct requirements files.

Axipro Author

Picture of Pedro Dias

Pedro Dias

Pedro has been writing online for over 10 years. With experience in all things programming, cyber security, and compliance, he is our editor-in-chief at Axipro.

Blog Highlights

Explore More Articles

A consultant-grade ISO 42001 gap analysis checklist has 38 Annex A controls, roughly 80 clause-level “shall” statements, and one question attached to every line: where is the evidence, and would a certification body accept it? That last question is what separates the checklists consultants use from the free self-assessment spreadsheets that rank for the same search. This article lays out the checklist itself: what a consultant checks before the engagement starts, the clause-by-clause and control-by-control checkpoints, how evidence gets sampled, how gaps get scored, what the deliverables look like, and what fails most often. Use it to run your own assessment, or to check whether the consultant you’re about to hire is doing the job properly. What Makes a Consultant-Grade ISO 42001 Gap Analysis Checklist Different​ Depth of Evidence Review vs. Self-Assessment Tools A self-assessment tool asks whether you have an AI policy. A consultant asks to see it, checks the approval date and version, reads clause 5.2 against it, and then asks three people in engineering whether they’ve read it. The checklist item is the same. The evidence standard is not. Consultants score every item on three levels: documented, implemented, and effective. A policy that exists but nobody follows scores as “ad hoc,” not “defined.” A control that runs but produces no record scores as unverifiable, which for audit purposes is the same as absent. Self-assessment tools collapse those three levels into a single yes/no, which is why companies that score 85% on a free tool routinely receive major nonconformities at Stage 2. Alignment with Certification Body Expectations Certification bodies auditing against ISO/IEC 42001:2023 now work under ISO/IEC 42006:2025, which sets competence, audit-time, and impartiality requirements for AIMS auditors and builds on ISO/IEC 17021-1. A consultant-grade checklist is written with 42006 in mind: it organizes findings by clause and control identifier, because that’s how the auditor works, and it records evidence locations, because that’s what the auditor will sample. The practical difference shows up in the report. A gap register that says “AI governance needs improvement” is useless in front of an auditor. One that says “A.5.2 not conformant: no documented impact assessment process; two of four in-scope systems have no assessment on file” maps directly to the audit plan. Risk-Weighted Scoring Methodology Self-assessments count gaps. Consultants weight them. A missing AI policy under clause 5.2 and an incomplete competence matrix under 7.2 are both gaps, but the first will block certification and the second will earn you a minor finding. A consultant-grade checklist carries two scores per line: a maturity rating (how far the control is from working) and a certification criticality (what happens at audit if it stays this way). Effort estimates live in the remediation plan, never in the gap score, because mixing them produces a roadmap that fixes easy things first rather than important ones. Insider Note: The fastest tell that a checklist is consultant-grade rather than a marketing download is whether it has a column for evidence location. Auditors don’t accept “yes” as evidence. If the checklist has nowhere to record where the proof lives, it wasn’t built by someone who has sat through a Stage 2. Pre-Engagement Preparation Consultants Complete Before the Gap Analysis Client AI Inventory and Use Case Cataloging Nothing in the checklist works without a complete AI inventory, and it’s the input clients get wrong most often. The inventory records every AI system in use: purpose, the role you play (developer, provider, deployer, or user), data consumed, outputs produced, whether a human sits between the output and the decision, and which third-party model or API it depends on. Consultants push hard on shadow AI here: SaaS tools that added AI features, agents running under employee credentials, and internal scripts calling model APIs. Every one of those is in scope until you document why it isn’t. Defining AIMS Scope Boundaries Clause 4.3 requires a scope statement naming which AI systems, business units, locations, and lifecycle stages the AIMS covers. Consultants draft this from the inventory, not before it. Scope discipline matters commercially too: certification bodies price audits by audit days, and audit days scale with scope. A narrow, well-justified first scope (the customer-facing AI product, say, rather than every internal tool) is usually the right call for a first certification. Stakeholder Interview Planning The checklist needs answers from people who don’t write policies. A typical interview plan covers the executive sponsor (clause 5), the AI or product lead (clauses 6 and 8), data engineering (A.7), procurement or vendor management (A.10), legal or privacy (A.5, A.8), and at least one front-line user of the AI system (A.9). Consultants interview the doers separately from the document owners, because the distance from what the procedure says to what actually happens is the finding. Document Request List (DRL) Consultants Send Clients The DRL goes out one to two weeks before fieldwork. A standard ISO 42001 DRL asks for the AI inventory; existing AI, security, and data policies; org chart with AI governance roles; any AI risk assessments or impact assessments; model documentation (model cards, system cards, or whatever exists); training-data provenance and data quality records; supplier contracts for third-party models; incident and change logs; training records; any ISO 27001 ISMS documentation; and the last internal audit and management review minutes if they exist. Missing items become findings rather than delays. Pro Tip: Return an Honest DRL Return the DRL with a column that says “does not exist” wherever that’s true. Consultants would rather know on day one than discover it in a workshop. An honest DRL shortens fieldwork by days and makes the maturity scores more accurate, which makes the remediation plan cheaper. Clause-by-Clause Checklist Consultants Use (ISO 42001 Clauses 4 to 10) ISO 42001 follows the Harmonized Structure shared with ISO 27001 and ISO 9001, so clauses 4 to 10 will look familiar to anyone who has run an ISMS. What’s different is the content each clause demands. Clause 4 – Context of the Organization Checkpoints Consultants check for a documented analysis of

Scigeniq, a UAE life sciences software vendor, completed SOC 2 Type 2 and ISO 27001 in one three-month engagement with Axipro and Vamu.

ISO/IEC 42001:2023 asks for three assessments, and most teams try to squeeze them into one spreadsheet: a gap analysis against clauses 4 to 10 and Annex A, an AI risk assessment under clause 6.1.2, and an AI system impact assessment under clause 6.1.4. Treat them as one exercise and the auditor pulls them apart for you at Stage 2. Treat them as three unrelated projects and you triple the workshops, the registers, and the remediation lists. What works is a single methodology with distinct outputs that share inputs, share a traceability matrix, and feed one remediation plan. This article lays out that methodology end to end: how gap analysis and risk assessment fit together under ISO 42001, how to prepare, the step-by-step process for each, how to merge the outputs into one risk treatment plan, the registers and templates you’ll need, and what a certification body expects to see when you’re done. Why Gap Analysis and Risk Assessment Must Work Together Under ISO 42001 A gap analysis measures distance from the standard. A risk assessment measures exposure from your AI systems. They answer different questions, and ISO 42001 makes them depend on each other in a way ISO 27001 only implies. Clause 6.1.3 requires you to compare the controls you select through risk treatment against Annex A, and to justify any Annex A control you leave out in the Statement of Applicability (SoA). So your Annex A gap analysis has no defensible baseline until the risk assessment tells you which controls you need. Run the gap analysis on its own, and you end up scoring yourself against all 38 controls, including ones your risk profile never called for. Run the risk assessment on its own, and you pick treatments with no idea what already exists to deliver them. The methodology below interleaves the two. A clause-level gap review sets the scope and evidence base, the risk and impact assessments decide which controls are required, and a control-level gap review then scores only what matters. How AI-specific risks shape the methodology Traditional information security risk works from confidentiality, integrity, and availability. AI risk adds categories that don’t map neatly onto any of those: model drift, bias in training data, outputs nobody can explain, automation bias in the humans doing the reviewing, and dependence on third-party foundation models whose behavior changes without warning. ISO/IEC 23894, the companion guidance on AI risk management, adapts the ISO 31000 cycle (establish context, identify, analyze, evaluate, treat) to these sources rather than inventing a new one. That’s why the methodology here keeps the familiar ISO 31000 shape and changes the inputs, not the process. Regulatory and business drivers for a formal methodology The commercial driver is procurement. Enterprise security questionnaires now ask whether you ran an AI impact assessment, whether a human reviews high-stakes outputs, and which third-party models touch customer data. A documented methodology answers those questions with evidence instead of assurances. The regulatory driver is the EU AI Act, and its timeline moved in July. Regulation (EU) 2026/1744, the Digital Omnibus on AI, entered into force on July 27, 2026, and pushed the high-risk obligations for standalone Annex III systems from August 2, 2026 to December 2, 2027. Annex I embedded systems moved to August 2, 2028. The Article 50 transparency obligations still kicked in on August 2, 2026, as originally planned. Article 9 of the AI Act text on EUR-Lex requires a risk management system for high-risk AI that runs continuously across the system lifecycle, which is exactly what an ISO 42001 methodology gives you. Sixteen extra months is time to build it properly, not a reason to shelve it. Core Principles of an ISO 42001 Gap Analysis and Risk Assessment Methodology Four principles keep the methodology defensible in front of a certification body. Alignment with clauses 4 to 10 and Annex A. Every finding in the gap register cites a clause or an Annex A control identifier. Auditors work clause by clause, so a gap register organized any other way forces a translation step during the audit that nobody enjoys. Integration with the AI system impact assessment. Clause 6.1.4 is what separates ISO 42001 from every other Annex SL standard. The impact assessment looks outward at individuals, groups, and society. The risk assessment under 6.1.2 looks inward at the organization. The standard wants both as separate documented outputs, and the consequences you find in the impact assessment have to feed back into the risk assessment. So the methodology runs the impact assessment as a scheduled input to risk analysis, not something bolted on the week before the audit. Risk-based thinking applied to the AIMS itself. Clause 6.1.1 also asks you to consider risks and opportunities to the management system: someone leaving the AI governance function, a vendor retiring a model, a regulator changing its classification rules. These go in the same register with a different category tag. Defined inputs, outputs, and success criteria. Inputs are the AI system inventory, the scope statement, existing policies, data flow diagrams, model documentation, and your risk criteria. Outputs are the gap register, the AI risk register, impact assessment reports, the SoA, and the risk treatment plan. Success means each output traces to the others, every gap and risk has an owner, and an internal auditor could repeat the process and land somewhere similar. Insider Note: Impact assessments are where certification auditors probe hardest, because they’re the most distinctive part of ISO 42001 compared with ISO 27001. A recycled security risk register with “AI” pasted into the risk titles gets picked apart in Stage 2. Build the impact assessment methodology properly the first time. It’s far cheaper than rebuilding it under a nonconformity deadline. Preparing for the Gap Analysis and Risk Assessment Preparation is where most of the calendar time goes, and where most later problems start. Define scope, boundaries, and the AI system inventory. Scope under clause 4.3 has to name which AI systems, business units, and lifecycle stages the AIMS covers. You can’t write