Shadow AI Policy Template: Ready-to-Use IT Framework

A green dashboard is not an audit opinion. Compliance automation platforms like Vanta, Drata, Secureframe, and Hyperproof have made SOC 2 readiness faster and cheaper, but every audit cycle produces the same pattern: controls that sat at “passing” for months come back from the auditor with exceptions or requests for re-testing. The four controls below account for a disproportionate share of those rejections, and they all fail for the same underlying reason. The tool confirmed that evidence exists. The auditor tested whether the control actually operated. This article walks through each of the four: what auditors reject, why, and how to fix the evidence before fieldwork starts. Why Compliance Tools Show “Passing” But Auditors Still Reject Controls​ The Gap Between Automated Checks and Auditor Judgment Compliance platforms run continuous control monitoring: API calls that check whether a configuration exists, a document is uploaded, or a task is marked done. That’s real value. It catches drift, keeps evidence in one place, and saves weeks of screenshot collection. An audit is a different exercise. A SOC 2 examination is an attestation performed by a CPA firm under AICPA standards, and the auditor’s job is to form an independent opinion on whether your controls met the Trust Services Criteria. That opinion rests on professional judgment, not on whether an API integration returned a 200 response. What “Passing” Actually Means in Your Compliance Dashboard​ When a control shows “passing,” the platform is telling you one narrow thing: at the moment of the last scan, an automated test found the artifact or setting it was programmed to look for: MFA enforced in the identity provider, a policy document uploaded, a training campaign sitting at 100%. The test says nothing about whether the underlying process ran the way your control narrative claims it did, or whether it ran that way across the whole audit period. How Auditors Evaluate Controls Beyond the Checkbox Auditors test two dimensions. Design effectiveness asks whether the control, as described, would meet the criterion if it worked as intended. Operating effectiveness, the core of a SOC 2 Type 2 report, asks whether it actually did throughout the audit period. To answer that, the auditor pulls a population (every access review, every change, every new hire in the period), selects a sample, and inspects the evidence item by item. A dashboard status feeds into that process. It doesn’t replace it. Insider Note: Auditors increasingly ask for evidence outside the compliance platform precisely because they know what the platform auto-collects. If every artifact you produce comes from the same tool export, expect the auditor to independently pull the population from the source system and compare. Discrepancies between the two are one of the fastest routes to an exception. Control #1: Access Reviews That Automation Marks Complete but Auditors Reject Why Auditors Reject Automated Access Review Evidence​ User access reviews sit under the logical access criteria (CC6.1 through CC6.3), and they are the single most common source of audit exceptions we see. The typical failure: the platform generated a user list, someone clicked “complete,” and the dashboard turned green. The auditor then asks a simple question the evidence can’t answer: what did the reviewer actually decide? The Missing Element: Documented Reviewer Judgment​ An access review is a judgment control. Someone with knowledge of the system must look at each account and confirm the access is still appropriate for the person’s role. A timestamped task closure proves the task was closed. It doesn’t prove anyone assessed anything, and an “approve all” review completed in ninety seconds gets exactly the skepticism it deserves. What Auditors Actually Want to See in Access Review Evidence Auditors look for four things: The full population of accounts at the time of review (including service accounts and admin roles), Evidence of who reviewed it and when, explicit dispositions per account or group (retain, modify, revoke), and Proof that flagged access was actually removed. That last item, the deprovisioning ticket showing revocation within a defined window, is the piece most companies can’t produce. How to Fix Your Access Review Control Before the Audit​ Assign a named control owner per in-scope system, run reviews quarterly, and require reviewers to record a disposition for every line, not a blanket approval. When access is revoked, link the removal ticket to the review record. If a quarter was missed, don’t backfill it. Document it honestly and show the remediation, because auditors treat fabricated retroactive evidence far more severely than a disclosed gap. Control #2: Change Management Approvals That Pass Automated Scans​ Why Ticket Closure Isn’t Proof of Approval​ Change management (CC8.1) automation typically verifies that production changes link to a ticket and the ticket is closed. Auditors test something stricter: that each sampled change was approved by an authorized person before deployment. An approval added after the merge, or a ticket closed by the same engineer who wrote the code, fails that test even though every automated check came back green. The Segregation of Duties Problem Automation Misses Segregation of duties is the requirement that no single person can develop, approve, and deploy the same change. NIST’s SP 800-53 control catalog treats it as a foundational access control principle, and SOC 2 auditors apply the same logic. Small engineering teams trip on this constantly. Self-approved pull requests, admins who can bypass branch protection, direct pushes to main: a scanner sees “changes with tickets” while an auditor sees SoD violations. Emergency Changes and Retroactive Approvals: Common Rejection Triggers​ Every audit period contains hotfixes. Auditors don’t reject emergency changes. They reject emergency changes with no documented post-hoc review. If your policy says urgent changes get retroactive approval within two business days, the auditor will sample your emergency changes and check exactly that. No policy, or a policy nobody followed, produces an exception. Rebuilding Change Management Evidence Auditors Will Accept​ Enforce the control technically: branch protection requiring at least one independent reviewer, no admin bypass, and deploy pipelines that only run from protected branches. Then write the emergency change procedure down and generate the review artifact every time it fires.