AI agent governance for regulated industries: a practical framework

Agentic AI breaks the assumptions your model-risk framework was built on. A credit model produces a score; an agent produces actions — sequences of tool calls chosen at runtime that nobody enumerated in advance. This piece is a practical framework for governing that, written for people who have to face a validator, an auditor, or — from August 2026 — an EU AI Act conformity assessment.

Why existing model risk management doesn’t map cleanly

SR 11-7-style MRM assumes three things agents violate:

1. A fixed input→output surface. Agents compose tools dynamically; the “output” is a trajectory, not a prediction.
2. Validation before deployment. You can validate a score’s accuracy; you can’t pre-enumerate every tool-call sequence an agent might take.
3. One model, one owner. An agentic system is a model plus an orchestration harness, tools, prompts, and memory — each changing on its own release cycle.

The answer is not to abandon MRM discipline — it’s to move the control points.

The framework: five control layers

1. Capability boundary — what the agent can do. Enforced in the tool layer (MCP servers you own), not in the prompt. Entitlements per tool, per environment; deny-by-default.
2. Action authorization — what it may do unattended. Classify tool calls as read / reversible-write / irreversible-write, and require human approval above a materiality line you define in advance.
3. Trajectory logging — evidence. Every tool call, argument, and decision context captured in a structured, replayable log. If you can’t reconstruct why the agent acted, you can’t defend it.
4. Evaluation regime — ongoing, not point-in-time. Golden-task suites run on every change to model, prompt, or tools; drift monitoring on trajectory patterns in production.
5. Accountability map — one named owner per component (model, harness, each tool server), and an explicit answer to “who is accountable when the agent is wrong?”

Coming in an update: a worked, fully generic example (a customer-service refund agent) for each layer, plus a mapping table: control layer ↔ SR 11-7 concept ↔ EU AI Act article.