Free Assessment

Table of Contents

Reach SOC 2 Compliance in 6 Weeks or Less.

Request a Demo

Home

  / ,

  / LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

Picture of Pedro Dias
Copy Link

Two compromised versions of LiteLLM sat on PyPI for roughly 40 minutes on the morning of March 24, 2026. That window was enough to capture secrets from around 434,000 CI/CD pipeline runs across nearly 2,500 organizations, including AWS, Samsung, Cisco, Salesforce, Siemens, and Deloitte. In August, researchers at CloudSEK and Hudson Rock confirmed they had obtained the raw exfiltrated data: a 153GB archive containing 433,909 files of environment variables, cloud keys, Kubernetes secrets, and API tokens harvested live from running pipelines, as covered by Help Net Security’s reporting on the credential archive.

If LiteLLM runs anywhere in your stack, or you touch any AI proxy infrastructure at all, you need answers to three things: whether you were exposed, what to rotate first, and whether the rotation you did back in March actually held. That last one matters more than it sounds, because “we rotated everything” has already burned at least one very large company.

How the Breach Happened

The attack didn’t start with LiteLLM. On March 19, 2026, a threat group called TeamPCP compromised the build pipeline of Trivy, a vulnerability scanner half the industry runs, and pushed a poisoned release. LiteLLM’s own CI pipeline ran Trivy, so the poisoned scanner had legitimate read access to the project’s runner environment. The attackers used that to steal LiteLLM’s PyPI publishing tokens and ship two malicious releases of their own: versions 1.82.7 and 1.82.8. KICS and the Telnyx Python SDK got hit in the same campaign.

The payload design is the part worth studying. The malicious package dropped a .pth startup hook into site-packages, so the code ran the moment any Python interpreter started on the machine, whether or not anything imported LiteLLM. From there it harvested environment variables, read local credential files like .aws/credentials and .kube/config, tried to move laterally across Kubernetes clusters, and installed a systemd backdoor dressed up as a generic telemetry service. InfoQ’s coverage of the PyPI compromise put downloads of the compromised release above 40,000. For scale, LiteLLM normally gets downloaded around 3 million times a day.

The exfiltration had a nasty fallback, too. According to CloudSEK, stolen data was encrypted and sent to a typosquatted domain, and when that failed, the malware created a public repository inside the victim’s own GitHub account and uploaded the loot as a release asset. Some companies were publishing their own secrets to the open internet and had no idea.

Worth Knowing: The malicious code only existed in the PyPI artifacts. The GitHub source repository stayed clean the whole time, so a developer reviewing the code on GitHub saw nothing wrong. Source review isn’t artifact verification. If you don’t check that what the registry serves matches the upstream source, this class of attack is invisible to you.

How to Check If You Were Exposed

Three checks, from quickest to most involved.

1. Confirm whether the compromised versions ever ran

The malicious versions went live on PyPI at 10:39 UTC on March 24, 2026 and got quarantined about 40 minutes later. The project’s advice: treat any install from that day before 16:00 UTC as suspect. Search your lockfiles, pip caches, SBOMs, and container image histories for 1.82.7 and 1.82.8. And check your internal artifact mirrors. An Artifactory or Nexus proxy that cached the bad release in March can keep serving it internally long after PyPI pulled it.

Keep the .pth mechanism in mind when you scope this. The question isn’t “which applications import LiteLLM,” it’s “which machines had the package installed at all,” because every Python process on an infected machine triggered the payload.

2. Hunt for persistence

Rotation is pointless if the attacker still has a foothold. Check developer machines, CI runners, and containers for unauthorized .pth files in site-packages and for suspicious systemd units, especially anything posing as a system telemetry service. And review activity from March 24 onward, not just the 40-minute window. Persistence is there so the access outlives the infection.

Pro Tip: Don’t limit the persistence hunt to live machines. Base container images rebuilt in late March may have baked the payload into every image derived from them since. Scan your image registry for the affected LiteLLM versions and for unexpected .pth files, then trace which running workloads came from flagged images.

3. Check whether your secrets are in the dump

Hudson Rock has published a domain lookup tool and is running ethical disclosures for affected organizations, and CloudSEK maintains a high-confidence victim list. Use them, but know their limits. Attribution in this dataset is genuinely hard. One dump with a siriusxm.com committer email actually traced, through its self-hosted GitLab endpoints, to AdsWizz, a SiriusXM subsidiary. And a large share of the dumps are generic pipeline configurations with no identifying domain, email, or server name at all. Absence from a victim list is not evidence of absence. If your pipelines ran the compromised versions, assume exposure no matter what a lookup tool tells you.

What to Rotate, in What Order

The guidance from both research teams is blunt: treat every secret the LiteLLM environment could reach as compromised. That covers secrets on disk, in memory, injected into CI jobs, and anything retrievable through instance metadata services. Work down by blast radius:

PriorityCredential typeWhy it comes first
1Cloud IAM keys (AWS, GCP, Azure)Direct control of infrastructure, data stores, and billing. This is where attackers monetize fastest.
2GitHub and GitLab PATs, package publishing tokensThese let an attacker poison your releases and turn your company into the next link in the supply chain.
3Kubernetes service account tokens and kubeconfigsLateral movement across clusters was built into the payload, not a theoretical risk.
4Database passwords and third-party API keysDumped in plain text in the archive, often with no attribution, so nobody will warn you they leaked.
5AI provider API keysBilling abuse, quota theft, and access to whatever data flows through your LLM routing layer.

One word matters more than the rest of this article: revoke, don’t just rotate. That distinction is exactly how this breach grew in the first place. After the initial Trivy compromise, its maintainers rotated an automation token but didn’t fully revoke the old one for around 20 days, which handed the attackers a three-week window to force-push malicious code into third-party builds.

Important: Rotating a credential creates a new one. Revoking kills the old one. Plenty of platforms let both exist side by side, and CI systems, long-lived sessions, and cached tokens will happily keep honoring the old secret. After every rotation in this incident response, verify the old credential actually fails, then check audit logs for anything that used it after March 24.

This isn’t theoretical. Independent researcher Kevin Beaumont tested credentials from the dump, months after the attack, against an organization whose disclosure policy allowed it. The company had told him the leak was old news because everything had been rotated. His verdict: “Almost every one worked.” That was one of the largest US tech companies, not some resource-starved startup.

Two more steps before you call it done. Review AWS CloudTrail and Kubernetes API audit logs for anomalous activity going back to March 24, 2026, and put strict egress filtering on runner environments so the next payload has nowhere to send your secrets.

What This Incident Says About AI Supply-Chain Controls

The uncomfortable part is that this wasn’t an AI failure. It was a DevOps failure, made worse by how fast teams have bolted AI tooling onto their pipelines. As Beaumont put it, the teens behind TeamPCP ran circles around organizations obsessed with shipping AI while their build security lagged behind. A supply chain attack of this shape is now the default threat model for any company whose product depends on open source. Which is every company.

The structural fixes aren’t new either. NIST’s Secure Software Development Framework has warned for years that build pipelines are the soft underbelly of modern companies, and its practices map directly onto what failed here: verified artifact integrity, least-privilege pipeline credentials, and egress controls on build environments. The compliance frameworks are catching up too. SOC 2’s Trust Services Criteria added explicit focus areas on software supply chain security and vendor risk in the 2022 revision, and ISO 42001 extends that discipline to the AI layer: how you govern the AI tooling and third-party models your business now depends on. If this incident is what finally puts AI governance on your roadmap, Axipro’s ISO 42001 implementation services cover exactly this territory, from supplier controls to auditable AI asset inventories.

Expect the questionnaires to change as well. Enterprise buyers already probe vendors on SOC 2 and ISO 27001. After a breach that turned an AI proxy into a credential vacuum, questions about AI dependencies, artifact verification, and pipeline secret scoping are coming. Our guide to security certifications for AI agent vendors maps what buyers now ask for, and the pattern rhymes with what we covered in the May 2026 GitHub breach: developer tooling has become the highest-value target in most companies, and the security model around it hasn’t kept pace.

Insider Note: [REVIEW: confirm or replace with a real Axipro observation] In the gap analyses we run before ISO 27001 and SOC 2 engagements, over-scoped CI/CD credentials are the single most common finding: runners that build a marketing site holding organization-wide cloud admin keys, because scoping them properly was someone’s someday task. Every one of those environments would have leaked its full keyring in this breach. Least-privilege pipeline secrets are tedious to set up and boring to maintain, which is exactly why attackers count on you not doing it.

There’s a testing angle here as well. A payload that moved laterally through Kubernetes and planted systemd persistence is the kind of attack path a good offensive exercise should catch before a real adversary does. If your last test predates your AI infrastructure, our breakdown of what auditors expect from ISO 27001 penetration testing is a practical place to start scoping one.

The LiteLLM breach compressed the whole modern supply-chain threat into 40 minutes: one upstream compromise, hundreds of thousands of exposed pipelines, and secrets that stayed live for months because rotation got treated as a checkbox. Check whether the bad versions ever touched your environment, hunt for persistence before you rotate, revoke rather than rotate, and verify the old credentials are actually dead. Then fix what made the blast radius so big in the first place: pipelines holding far more privilege than the jobs they run ever needed.

Frequently Asked Questions

Which LiteLLM versions were compromised?

Versions 1.82.7 and 1.82.8, published to PyPI at 10:39 UTC on March 24, 2026 and quarantined roughly 40 minutes later. The project advises treating any LiteLLM install from that day before 16:00 UTC as suspect, and internal package mirrors may have cached the bad releases well past that window.

My company is not on the victim list. Are we safe?

Not necessarily. Researchers could only attribute dumps that contained identifying markers like domains or committer emails, and a big share of the archive is generic pipeline data with no attribution at all. If your systems ran the compromised versions, assume your secrets are in the dataset and act on it.

We rotated our credentials in March. Is that enough?

Only if the old credentials were fully revoked and you’ve verified they no longer work. One major tech company believed it had rotated everything, and a researcher found nearly all of its leaked credentials still worked months later. Test the old secrets, then audit logs for any use of them after March 24, 2026.

We never import LiteLLM directly. Does this still affect us?

Possibly. LiteLLM shows up as a transitive dependency in a lot of AI stacks, and the malicious code ran through a .pth startup hook whenever any Python interpreter started on an infected machine, with no import required. Audit dependency trees and installed packages, not just your direct requirements files.

Axipro Author

Picture of Pedro Dias

Pedro Dias

Pedro has been writing online for over 10 years. With experience in all things programming, cyber security, and compliance, he is our editor-in-chief at Axipro.
Copy Link

Blog Highlights

Explore More Articles

Read More Blogs

LiteLLM Supply-Chain Breach: How to Check If You’re Exposed and What to Rotate

Two compromised versions of LiteLLM sat on PyPI for roughly 40 minutes on the morning of March 24, 2026. That window was enough to capture secrets from around 434,000 CI/CD pipeline runs across nearly 2,500 organizations, including AWS, Samsung, Cisco, Salesforce, Siemens, and Deloitte. In August, researchers at CloudSEK and Hudson Rock confirmed they had obtained the raw exfiltrated data: a 153GB archive containing 433,909 files of environment variables, cloud keys, Kubernetes secrets, and API tokens harvested live from running pipelines, as covered by Help Net Security’s reporting on the credential archive. If LiteLLM runs anywhere in your stack, or you touch any AI proxy infrastructure at all, you need answers to three things: whether you were exposed, what to rotate first, and whether the rotation you did back in March actually held. That last one matters more than it sounds, because “we rotated everything” has already burned at least one very large company. How the Breach Happened The attack didn’t start with LiteLLM. On March 19, 2026, a threat group called TeamPCP compromised the build pipeline of Trivy, a vulnerability scanner half the industry runs, and pushed a poisoned release. LiteLLM’s own CI pipeline ran Trivy, so the poisoned scanner had legitimate read access to the project’s runner environment. The attackers used that to steal LiteLLM’s PyPI publishing tokens and ship two malicious releases of their own: versions 1.82.7 and 1.82.8. KICS and the Telnyx Python SDK got hit in the same campaign. The payload design is the part worth studying. The malicious package dropped a .pth startup hook into site-packages, so the code ran the moment any Python interpreter started on the machine, whether or not anything imported LiteLLM. From there it harvested environment variables, read local credential files like .aws/credentials and .kube/config, tried to move laterally across Kubernetes clusters, and installed a systemd backdoor dressed up as a generic telemetry service. InfoQ’s coverage of the PyPI compromise put downloads of the compromised release above 40,000. For scale, LiteLLM normally gets downloaded around 3 million times a day. The exfiltration had a nasty fallback, too. According to CloudSEK, stolen data was encrypted and sent to a typosquatted domain, and when that failed, the malware created a public repository inside the victim’s own GitHub account and uploaded the loot as a release asset. Some companies were publishing their own secrets to the open internet and had no idea. Worth Knowing: The malicious code only existed in the PyPI artifacts. The GitHub source repository stayed clean the whole time, so a developer reviewing the code on GitHub saw nothing wrong. Source review isn’t artifact verification. If you don’t check that what the registry serves matches the upstream source, this class of attack is invisible to you. How to Check If You Were Exposed Three checks, from quickest to most involved. 1. Confirm whether the compromised versions ever ran The malicious versions went live on PyPI at 10:39 UTC on March 24, 2026 and got quarantined about 40 minutes later. The project’s advice: treat any install from that day before 16:00 UTC as suspect. Search your lockfiles, pip caches, SBOMs, and container image histories for 1.82.7 and 1.82.8. And check your internal artifact mirrors. An Artifactory or Nexus proxy that cached the bad release in March can keep serving it internally long after PyPI pulled it. Keep the .pth mechanism in mind when you scope this. The question isn’t “which applications import LiteLLM,” it’s “which machines had the package installed at all,” because every Python process on an infected machine triggered the payload. 2. Hunt for persistence Rotation is pointless if the attacker still has a foothold. Check developer machines, CI runners, and containers for unauthorized .pth files in site-packages and for suspicious systemd units, especially anything posing as a system telemetry service. And review activity from March 24 onward, not just the 40-minute window. Persistence is there so the access outlives the infection. Pro Tip: Don’t limit the persistence hunt to live machines. Base container images rebuilt in late March may have baked the payload into every image derived from them since. Scan your image registry for the affected LiteLLM versions and for unexpected .pth files, then trace which running workloads came from flagged images. 3. Check whether your secrets are in the dump Hudson Rock has published a domain lookup tool and is running ethical disclosures for affected organizations, and CloudSEK maintains a high-confidence victim list. Use them, but know their limits. Attribution in this dataset is genuinely hard. One dump with a siriusxm.com committer email actually traced, through its self-hosted GitLab endpoints, to AdsWizz, a SiriusXM subsidiary. And a large share of the dumps are generic pipeline configurations with no identifying domain, email, or server name at all. Absence from a victim list is not evidence of absence. If your pipelines ran the compromised versions, assume exposure no matter what a lookup tool tells you. What to Rotate, in What Order The guidance from both research teams is blunt: treat every secret the LiteLLM environment could reach as compromised. That covers secrets on disk, in memory, injected into CI jobs, and anything retrievable through instance metadata services. Work down by blast radius: Priority Credential type Why it comes first 1 Cloud IAM keys (AWS, GCP, Azure) Direct control of infrastructure, data stores, and billing. This is where attackers monetize fastest. 2 GitHub and GitLab PATs, package publishing tokens These let an attacker poison your releases and turn your company into the next link in the supply chain. 3 Kubernetes service account tokens and kubeconfigs Lateral movement across clusters was built into the payload, not a theoretical risk. 4 Database passwords and third-party API keys Dumped in plain text in the archive, often with no attribution, so nobody will warn you they leaked. 5 AI provider API keys Billing abuse, quota theft, and access to whatever data flows through your LLM routing layer. One word matters more than the rest of this article: revoke, don’t just rotate. That

Read more

EU AI Act Recruitment Rules: What to Know for 2027

The EU AI Act names recruitment AI as high-risk. Annex III explicitly lists AI systems used for recruitment, candidate selection, and employment decisions, which pulls CV screeners, video interview platforms, and assessment tools into the most demanding compliance regime the Act contains. The original compliance date for these systems was August 2, 2026. In June 2026, the EU’s Digital Omnibus moved the deadline to December 2, 2027, a 16-month extension that has led many HR and talent teams to shelve the topic entirely. That’s a mistake, for two reasons. First, one rule that directly affects recruitment technology is already in force: the ban on emotion recognition in the workplace has applied since February 2, 2025, and it catches features still shipping in some video interview products today. Second, the deferred obligations didn’t shrink. Conformity assessments, human oversight design, bias monitoring, and documentation all still arrive in full, and the practical work of auditing a recruitment stack, renegotiating vendor contracts, and training hiring teams routinely takes a year or more. Here’s what the EU AI Act actually requires of employers and vendors using recruitment tools, on the timeline that now applies. Why Recruitment Tools Are Classified as High-Risk Under the EU AI Act​ Definition of High-Risk AI Systems in Hiring​ The Act takes a list-based approach. Annex III, point 4, designates as high-risk any AI system intended for the recruitment or selection of natural persons, including placing targeted job advertisements, analyzing and filtering applications, and evaluating candidates. The same point covers AI used for decisions on promotion, termination, task allocation, and monitoring of workers, so the classification follows the tool through the entire employment lifecycle, not just the hiring funnel. The reasoning is straightforward: hiring decisions shape access to livelihoods, and algorithmic discrimination in hiring is well documented. The European Commission’s regulatory framework for AI treats employment as one of the areas where an AI error or bias causes serious harm to fundamental rights. That’s the test for the high-risk tier. Types of Recruitment Tools Affected In practice, the high-risk classification captures most of the modern recruitment stack: CV and resume screeners that rank or filter applicants, video interview platforms that score responses or delivery, psychometric and skills assessment tools that produce scores feeding a hiring decision, sourcing and matching algorithms that decide which candidates a recruiter sees, and programmatic job ad targeting systems that determine who sees a vacancy at all. If the system’s output materially influences who advances and who does not, assume high-risk until proven otherwise. Important: Emotion recognition is not high-risk in the workplace. It is prohibited. Article 5 bans AI systems that infer emotions of people in the workplace (outside narrow medical and safety cases), and that ban has applied since February 2025 with the Act’s top penalty tier attached. If your video interview vendor markets “engagement scoring” or “sentiment analysis” of candidates, that feature needs to be switched off for EU hiring now, not in 2027. Recruitment Tools That May Fall Outside High-Risk Classification Not everything in the HR stack qualifies. The Act carves out systems performing narrow procedural tasks that do not materially influence decision outcomes. An applicant tracking system that stores applications, schedules interviews, and sends templated emails is a database with a workflow, not a high-risk AI system. The same goes for tools that transcribe interviews without scoring them, deduplicate candidate records, or generate first drafts of job descriptions for a human to edit. The line is decision influence: the moment a tool ranks, scores, filters, or recommends candidates, it crosses into Annex III territory. Deployers who rely on an exemption must be able to document that assessment, so “we decided it doesn’t count” needs to exist on paper. Extraterritorial Scope: Which Employers Are Covered The Act applies to providers placing AI systems on the EU market and to deployers established in the EU, but it also reaches further: it covers providers and deployers located outside the EU where the output of the system is used in the EU. For recruitment, the consequence is blunt. A US or UK company with no EU entity that uses an AI screener to filter applicants for roles based in Berlin or Dublin, or that screens candidates located in the EU, is using the system’s output in the Union. Brexit doesn’t move UK employers out of scope when they hire into or from the EU. Providers vs. Deployers of Recruitment AI Tools The Act splits obligations between the provider (the vendor that develops the tool and places it on the market) and the deployer (the employer using it). Most employers are deployers, and deployer obligations are lighter but real. One common trap: an employer that substantially modifies a high-risk system, or puts its own name on it, can be reclassified as a provider and inherit the full provider stack. Heavy customization of a screening model, or fine-tuning it on your own hiring data, can be enough to trigger this. Key Obligations for Employers Using AI Recruitment Tools Human Oversight in Automated Hiring Decisions Deployers must assign oversight of the system to people with the competence, training, and authority to intervene. That last word matters. A recruiter who rubber-stamps whatever the ranking algorithm produces, because nobody has time to review 800 rejected CVs, doesn’t count as oversight. Regulators and courts will look at whether the human could genuinely override the system and whether they ever did. Designing review checkpoints where a person can meaningfully change the outcome, and logging when they do, is the core of compliant deployment. Transparency Requirements Toward Candidates Employers must inform workers and their representatives before putting a high-risk AI system into use at work, and candidates subjected to such a system must be told it is being used. In countries with works councils, such as Germany, this obligation lands on top of existing co-determination rights, so employee representatives may need to be consulted before the tool goes live rather than just told afterward. Burying an AI disclosure in a privacy policy paragraph is unlikely to survive scrutiny.

Read more

4 SOC 2 Controls Auditors Reject Even When Your Compliance Tool Says Passing

A green dashboard is not an audit opinion. Compliance automation platforms like Vanta, Drata, Secureframe, and Hyperproof have made SOC 2 readiness faster and cheaper, but every audit cycle produces the same pattern: controls that sat at “passing” for months come back from the auditor with exceptions or requests for re-testing. The four controls below account for a disproportionate share of those rejections, and they all fail for the same underlying reason. The tool confirmed that evidence exists. The auditor tested whether the control actually operated. This article walks through each of the four: what auditors reject, why, and how to fix the evidence before fieldwork starts. Why Compliance Tools Show “Passing” But Auditors Still Reject Controls​ The Gap Between Automated Checks and Auditor Judgment Compliance platforms run continuous control monitoring: API calls that check whether a configuration exists, a document is uploaded, or a task is marked done. That’s real value. It catches drift, keeps evidence in one place, and saves weeks of screenshot collection. An audit is a different exercise. A SOC 2 examination is an attestation performed by a CPA firm under AICPA standards, and the auditor’s job is to form an independent opinion on whether your controls met the Trust Services Criteria. That opinion rests on professional judgment, not on whether an API integration returned a 200 response. What “Passing” Actually Means in Your Compliance Dashboard​ When a control shows “passing,” the platform is telling you one narrow thing: at the moment of the last scan, an automated test found the artifact or setting it was programmed to look for: MFA enforced in the identity provider, a policy document uploaded, a training campaign sitting at 100%. The test says nothing about whether the underlying process ran the way your control narrative claims it did, or whether it ran that way across the whole audit period. How Auditors Evaluate Controls Beyond the Checkbox Auditors test two dimensions. Design effectiveness asks whether the control, as described, would meet the criterion if it worked as intended. Operating effectiveness, the core of a SOC 2 Type 2 report, asks whether it actually did throughout the audit period. To answer that, the auditor pulls a population (every access review, every change, every new hire in the period), selects a sample, and inspects the evidence item by item. A dashboard status feeds into that process. It doesn’t replace it. Insider Note: Auditors increasingly ask for evidence outside the compliance platform precisely because they know what the platform auto-collects. If every artifact you produce comes from the same tool export, expect the auditor to independently pull the population from the source system and compare. Discrepancies between the two are one of the fastest routes to an exception. Control #1: Access Reviews That Automation Marks Complete but Auditors Reject Why Auditors Reject Automated Access Review Evidence​ User access reviews sit under the logical access criteria (CC6.1 through CC6.3), and they are the single most common source of audit exceptions we see. The typical failure: the platform generated a user list, someone clicked “complete,” and the dashboard turned green. The auditor then asks a simple question the evidence can’t answer: what did the reviewer actually decide? The Missing Element: Documented Reviewer Judgment​ An access review is a judgment control. Someone with knowledge of the system must look at each account and confirm the access is still appropriate for the person’s role. A timestamped task closure proves the task was closed. It doesn’t prove anyone assessed anything, and an “approve all” review completed in ninety seconds gets exactly the skepticism it deserves. What Auditors Actually Want to See in Access Review Evidence Auditors look for four things: The full population of accounts at the time of review (including service accounts and admin roles), Evidence of who reviewed it and when, explicit dispositions per account or group (retain, modify, revoke), and Proof that flagged access was actually removed. That last item, the deprovisioning ticket showing revocation within a defined window, is the piece most companies can’t produce. How to Fix Your Access Review Control Before the Audit​ Assign a named control owner per in-scope system, run reviews quarterly, and require reviewers to record a disposition for every line, not a blanket approval. When access is revoked, link the removal ticket to the review record. If a quarter was missed, don’t backfill it. Document it honestly and show the remediation, because auditors treat fabricated retroactive evidence far more severely than a disclosed gap. Control #2: Change Management Approvals That Pass Automated Scans​ Why Ticket Closure Isn’t Proof of Approval​ Change management (CC8.1) automation typically verifies that production changes link to a ticket and the ticket is closed. Auditors test something stricter: that each sampled change was approved by an authorized person before deployment. An approval added after the merge, or a ticket closed by the same engineer who wrote the code, fails that test even though every automated check came back green. The Segregation of Duties Problem Automation Misses Segregation of duties is the requirement that no single person can develop, approve, and deploy the same change. NIST’s SP 800-53 control catalog treats it as a foundational access control principle, and SOC 2 auditors apply the same logic. Small engineering teams trip on this constantly. Self-approved pull requests, admins who can bypass branch protection, direct pushes to main: a scanner sees “changes with tickets” while an auditor sees SoD violations. Emergency Changes and Retroactive Approvals: Common Rejection Triggers​ Every audit period contains hotfixes. Auditors don’t reject emergency changes. They reject emergency changes with no documented post-hoc review. If your policy says urgent changes get retroactive approval within two business days, the auditor will sample your emergency changes and check exactly that. No policy, or a policy nobody followed, produces an exception. Rebuilding Change Management Evidence Auditors Will Accept​ Enforce the control technically: branch protection requiring at least one independent reviewer, no admin bypass, and deploy pipelines that only run from protected branches. Then write the emergency change procedure down and generate the review artifact every time it fires.

Read more