AI agent governance for regulated industries: a practical framework

Agentic AI breaks the assumptions most model-risk frameworks were built on. A credit model produces a score. An agent produces actions: sequences of tool calls chosen at runtime that nobody listed in advance. This piece is a practical framework for governing that, written for people who have to face a validator, an auditor or a regulator, and still useful if you just want your agents to be defensible.

The regulatory backdrop has shifted

Two changes in 2026 matter here.

• In the US, model-risk guidance now excludes agents. SR 11-7 was superseded on 17 April 2026 by SR 26-2 (OCC Bulletin 2026-13, FDIC FIL-15-2026), which states that generative and agentic AI “are not within the scope of this guidance.” Banks must govern agents through their broader risk-management practices instead. The SR 26-2 explainer covers what that means in detail.
• In the EU, the high-risk deadline moved. The Digital Omnibus (Regulation (EU) 2026/1744) pushed the Annex III high-risk obligations to 2 December 2027 and Annex I to 2 August 2028. The Article 50 transparency duties for systems that interact with people have applied since 2 August 2026. The EU AI Act timeline has the full table.

Neither change reduces the work. One removes the template you might have borrowed. The other gives you more time to build the evidence.

Why classic model risk management doesn’t map cleanly

Traditional model-risk practice assumes three things agents violate:

1. A fixed input-to-output surface. Agents compose tools dynamically, so the “output” is a trajectory, not a prediction.
2. Validation before deployment. You can validate a score’s accuracy. You can’t enumerate every tool-call sequence an agent might take.
3. One model, one owner. An agentic system is a model plus an orchestration harness, tools, prompts and memory, and each of those changes on its own release cycle.

The answer is not to abandon model-risk discipline. It’s to move the control points to where agents can actually be controlled.

The framework: five control layers

1. Capability boundary: what the agent can do. Enforced in the tool layer (MCP servers or APIs you own), not in the prompt. Entitlements per tool and per environment, deny by default.
2. Action authorization: what it may do unattended. Classify every tool call as read, reversible write or irreversible write, and require human approval above a materiality line you set in advance.
3. Trajectory logging: the evidence. Every tool call, argument and decision context captured in a structured, replayable log. If you can’t reconstruct why the agent acted, you can’t defend it.
4. Evaluation regime: ongoing, not point-in-time. Golden-task suites run on every change to the model, prompt or tools, plus monitoring of trajectory patterns in production.
5. Accountability map. One named owner per component (model, harness, each tool server) and an explicit answer to “who is accountable when the agent is wrong?”

Worked example: a customer-service refund agent

Here’s a generic agent that a retail bank, a card issuer or an insurer might build. It reads a customer’s message, looks up the account and recent transactions, decides whether a fee refund or goodwill credit is warranted, and either issues it or drafts a reply for a human. Nothing about it is exotic, which is why it’s useful.

Layer 1: capability boundary

The agent holds four tools, each served by an MCP server the team owns (see wrapping a REST API in MCP for the pattern):

Tool	Action class	Boundary enforced in the tool layer
get_account_summary	Read	Only the account linked to the authenticated conversation; masked account numbers
list_recent_transactions	Read	Last 90 days, that account only
issue_fee_refund	Reversible write	Only fee transaction types; amount must equal the original fee; one refund per fee
issue_goodwill_credit	Irreversible write (in practice)	Hard cap per credit and per customer per year; no other account

The agent has no tool to change limits, close accounts or move money between customers. It can’t be talked into doing those things because they don’t exist in its world. The caps are checked in the server code, so a prompt injection that says “refund $5,000” gets a validation error, not a refund.

Layer 2: action authorization

The team fills in the tool risk matrix and picks a pattern per action class from the human-in-the-loop patterns:

• Reads: autonomous.
• Fee refunds: autonomous inside bounds (pattern 4), with a 5% sample audited by a supervisor within two business days (pattern 3).
• Goodwill credits: autonomous below a small threshold; above it, or for any customer flagged as vulnerable, pre-approval by a human (pattern 1).
• Any write after the conversation contains pasted content from outside the bank (forwarded emails, links): pre-approval, regardless of amount. That rule comes straight from the prompt-injection threat model.

The approval request shows what the agent wants to do, why, what it looked at and whether untrusted content was involved, so the reviewer makes a decision instead of clicking through.

Layer 3: trajectory logging

Every conversation produces one trajectory record in the format from the trajectory-log template: run ID, model and prompt version, each tool call with arguments and a digest of the result, each approval decision with the approver’s ID, token counts and cost, and the outcome. Retention follows the bank’s existing rules for the equivalent human-handled complaint. The observability guide covers the full spec and alerting.

The same log feeds cost attribution: “cost per resolved complaint” is the number that keeps the agent funded.

Layer 4: evaluation regime

Before launch the team builds about 60 golden tasks with the evaluation harness:

• clear-cut refunds (duplicate fee, fee charged in error);
• clear-cut refusals (fee correctly charged, request outside policy);
• boundary cases right at the credit cap;
• injection attempts embedded in pasted text;
• tool failures (the transactions API times out), where the right answer is to escalate, not to guess.

Pass thresholds are set by tier: 100% on refusals and injection cases, and a documented minimum on the rest. The suite re-runs on every change to the model version, prompts, tool descriptions or MCP server code. In production, a weekly sample of trajectories is re-scored against the same criteria and compared with supervisors’ audit findings.

Layer 5: accountability map

Component	Owner	Accountable for
Use case and policy (refund rules, caps)	Head of customer operations	What the agent is allowed to decide
Harness and prompts	Agent product team lead	Behavior, evals, releases
Each MCP server	The owning system’s team	Entitlements and caps enforced in code
Model provider relationship	Vendor management + platform team	Contract, version pinning, fallback
Independent review	Risk / validation function	Design review and evaluation evidence

The answer to “who is accountable when the agent is wrong?” is the head of customer operations, exactly as it would be for a human agent following the same policy. The other owners are accountable to that person for their part.

The mapping table

Each control layer, the SR 26-2 concept it borrows from, and the EU AI Act article it supports if the use case is high-risk. SR 26-2 doesn’t apply to agents directly (footnote 3 excludes them), so the middle column is an analogy your model-risk colleagues will recognize, not a requirement.

Control layer	SR 26-2 concept (by analogy)	EU AI Act article (high-risk systems)	Evidence artifact
1. Capability boundary	Model use and limitations; use beyond intended purpose	Art. 9 risk management; Art. 15 robustness and cybersecurity	Tool inventory with action classes; server-side validation tests
2. Action authorization	Materiality (exposure and purpose)	Art. 14 human oversight	Risk matrix; approval records; override-rate reports
3. Trajectory logging	Ongoing monitoring; documentation	Art. 12 record-keeping; Art. 26(6) deployer log retention	Trajectory logs; retention policy
4. Evaluation regime	Validation (conceptual soundness, outcomes analysis); ongoing monitoring	Art. 9 testing; Art. 15 accuracy	Eval suite, results per release, production sampling
5. Accountability map	Roles and responsibilities; effective challenge; vendor products	Art. 26 deployer duties (assigning oversight); Art. 11 technical documentation	RACI; design-review record; vendor file

Two further obligations sit outside the five layers. Article 50 transparency (telling people they’re dealing with an AI system) applies to this refund agent now, whether or not it’s high-risk. And supply-chain integrity, meaning the packages and tool servers the agent runs on, belongs in your change-management process; the supply-chain guide covers it.

Getting started

You don’t need all five layers on day one. In order of payoff:

1. Capability boundary first. Moving entitlements into the tool layer is the single change that most limits how badly things can go wrong.
2. Trajectory logging second, because every other layer depends on the evidence it produces.
3. Action authorization for any irreversible write.
4. Evaluation regime, starting with 20 golden tasks and growing.
5. Accountability map, written down and signed off, before the agent reaches customers.