The easiest way to audit an AI program is to ignore the demos.

Demos show the happy path. The control plane shows whether the company can run the system when usage spreads, costs rise, tools expand, quality drifts, context gets messy, and someone asks what happened.

Use this audit when an AI initiative moves from experiment to operations.

1. Inventory the live AI systems

List the copilots, agents, automations, product features, internal workflows, scripts, vendor tools, and model integrations currently in use. Include unofficial ones if you can find them.

For each one, capture owner, purpose, users, data sources, model, tools, cost center, review path, and current status. If this inventory is hard, that is the first finding.

2. Check identity and ownership

Every AI system should have an owner. Every agent or workflow should have an identity. Shared keys, anonymous service accounts, and "owned by whoever built it" are control-plane debt.

Ask:

  • who owns this workflow?
  • who can run it?
  • what identity does it act under?
  • who can revoke access?
  • what happens when the owner leaves?

3. Map permissions and data scope

For each system, separate read, write, draft, send, approve, export, delete, and admin permissions. Identify the data it can see and whether that access is broader than the work requires.

Look especially for customer data, employee data, financial data, production systems, legal documents, security information, and cross-account context.

4. Inspect tool boundaries

Find every tool the AI can call. For each tool, ask what actions are automatic, staged, dry-run, approval-required, or blocked.

A healthy system has clear action boundaries. An unhealthy one has broad API access and a comforting sentence in a policy doc.

5. Review model routing

Identify which models are used where and why. Look for hardcoded defaults, premium models doing low-value work, cheap models handling risky work, and no fallback behavior.

A good routing policy considers task type, risk, quality bar, latency, privacy, and budget.

6. Review budgets and usage controls

Check spend by workflow instead of stopping at the vendor bill. Look for owners, limits, alerts, cost per accepted output, retry ceilings, token limits, batch limits, and approval thresholds for expensive runs.

If finance can only see the bill after the fact, the control plane is weak.

7. Audit memory and context

Find what the system stores, retrieves, summarizes, embeds, remembers, and reuses. Then ask who owns that memory, which source created it, who can read it, when it expires, and how it can be corrected.

Stale or over-broad memory is one of the quietest AI risks.

8. Test evals and release gates

For each production workflow, identify the eval set, owner, failure taxonomy, release threshold, and last regression result. If a prompt, model, tool, or retrieval change can ship without passing a relevant gate, quality control is mostly informal.

Ask what would block a release. If nobody knows, the gate is not real.

9. Inspect logs and observability

Pick three recent outputs and reconstruct what happened: actor, workflow, model, prompt version, context, tool calls, cost, approval, final action, and business object.

If reconstruction requires detective work, the logs are not good enough.

Then check operator views: acceptance rate, review load, override rate, escalation rate, cost per accepted output, tool failures, drift, and policy exceptions.

10. Review escalation and human review

Map the review queues. What triggers review? Who reviews? How much capacity exists? What gets escalated? Who owns decisions? What happens to reviewer corrections?

If humans are in the loop but the queue is overloaded, unclear, or unaudited, the control is weaker than it looks.

The scoring question

At the end, ask one blunt question:

Can this company increase AI usage by 10x without losing control of access, cost, quality, memory, tool actions, escalation, and auditability?

If the answer is no, the company does not need a bigger AI policy. It needs a better control plane.

The audit should produce a short backlog:

  • permissions to narrow
  • owners to assign
  • tools to sandbox
  • budgets to add
  • routing rules to define
  • eval gates to enforce
  • memory to expire or scope
  • logs to improve
  • review queues to redesign
  • escalation paths to clarify

Do not try to fix everything at once. Start where the blast radius is highest: customer-facing actions, financial impact, production systems, sensitive data, expensive workflows, and high-volume review queues.

A control plane is not a monument. It is the operating layer that lets AI become more useful without becoming harder to trust.

That is the work.

This is part 10 of 10 in The AI Control Plane.