Shadow AI Audit Checklist

Shadow AI is any AI tool, model, browser extension, or agent in use in your organization without a security review. It ranges from a chatbot tab holding pasted customer data to a no-code agent wired into your CRM with someone's personal API key. This checklist is for security leaders, IT governance teams, and internal auditors who need a repeatable way to find shadow AI, size the risk, and bring it under governance without pushing usage further underground.

Run it once as a baseline audit, then repeat it quarterly and whenever you adopt a new SaaS platform, identity provider, or AI vendor. The three phases line up with the Govern, Map, Measure, and Manage functions of the NIST AI Risk Management Framework, so findings can feed an existing AI governance program.

How to use this checklist

1. Assign one owner per phase: security engineering for discovery, risk for assessment, and a governance lead for phase 3.
2. Record every finding in a single inventory with the tool name, business owner, data it touches, and credentials it uses.
3. Frame discovery as an amnesty, not an enforcement exercise. People disclose far more when they are not afraid of losing the tool.

Phase 1: Discovery

• □ Review egress, proxy, and DNS logs for AI provider domains and API endpoints. Calls from servers, CI runners, or unexpected subnets usually mean an unregistered integration rather than casual browser use.
• □ Audit OAuth grants and service principals in your identity provider. Third-party apps holding mail, file, or calendar scopes are where AI assistants and meeting bots most often appear.
• □ Inventory browser extensions on managed endpoints. AI writing and summarization extensions can read page content, including internal web apps.
• □ Check SaaS admin consoles for embedded AI features. Many platforms ship AI features switched on; confirm what data they process and where it goes.
• □ Scan repositories and CI/CD pipelines for AI SDKs and hard-coded API keys. Secret scanning finds keys; dependency manifests reveal which SDKs are in use.
• □ Survey teams directly. A short, non-punitive questionnaire surfaces local models, personal accounts, and no-code agents that logs miss.

Phase 2: Risk assessment

• □ Map the data each tool can reach (public, internal, confidential, regulated). Data exposure drives impact more than the tool itself.
• □ Classify each agent by data sensitivity, autonomy level, and operational scope. A read-only assistant and an agent that can send email or change records need very different controls.
• □ Identify credential exposure: personal API keys, long-lived tokens, shared service accounts, and database connection strings stored in agent configurations.
• □ Review each vendor's data handling terms: retention, use of inputs for model training, sub-processors, and data residency.
• □ Assess prompt injection exposure for any tool that reads untrusted content (email, web pages, tickets) and can also take actions. The OWASP Top 10 for LLM Applications is a useful threat list for this step.

Phase 3: Governance

• □ Publish an approved-tools list with permitted use cases and data classes. A clear "yes" path removes most of the incentive to go around security.
• □ Offer a sanctioned alternative for each high-demand shadow tool before you block it.
• □ Define least-privilege scopes per tool category and replace personal keys with managed, rotated credentials.
• □ Add detections for new AI-related OAuth grants and unusual API volume to your SIEM.
• □ Extend incident response playbooks to cover sensitive data submitted to an external AI service.
• □ Schedule quarterly inventory reviews and re-run Phase 1 after major vendor or platform changes.

Common pitfalls

• Blanket blocking. It moves usage to personal devices and accounts, where you lose all visibility.
• Counting only chatbots. Agents, IDE assistants, meeting recorders, and AI features inside approved SaaS carry the same data risk.
• One-time audits. New AI features appear constantly; an inventory that is not refreshed is stale within a quarter.
• No named owner per tool. Every approved tool needs a business owner who answers for its access and data.