Threat Modeling AI Agents: The Runtime Controls Security Teams Expect Before Production

When security teams threat model an AI agent before production, they ask for controls that run while the agent runs: a named identity for the agent, a locked permission list, a hard execution boundary, human approval on irreversible actions, and an audit trail that can reconstruct every decision after the fact. Anything that only exists in a design document does not count. The question is not "is our prompt safe?" but "what can this running process reach, and who can stop it?"

Introduction

Most agent projects arrive at the security review with a demo and a diagram. The security team arrives with different questions. What does the agent authenticate as? What happens if the model is wrong, or the tool it calls is compromised? Can we see what it did last Tuesday? Can we turn it off without a redeploy? These are runtime questions, and they map to a short list of controls that come up in nearly every enterprise review of agentic systems. This article walks through that list and what a threat model should check before an agent touches production.

Key Takeaways

Threat model the runtime, not the prompt. The attack surface is what the running agent can authenticate to, call, and exfiltrate to.
Five controls come up in almost every review: agent identity, frozen permissions, an isolated execution boundary, human-in-the-loop approval, and an immutable audit trail.
Guardrails should fail closed: if the guardrail service is unavailable, the agent stops rather than answers anyway.
Kill switch and rollback are runtime requirements. Security teams want a way to stop an agent that does not involve a code change.
Deployment boundary is part of the threat model: cloud, customer VPC, or on-premise, with zero data retention by default.

The threat model for an agent looks different from a web app

A traditional service has a defined input surface and a bounded dependency tree. An agent has a goal, a set of tools, and the ability to chain them in sequences nobody enumerated.

The core threats security teams enumerate:

Prompt injection through retrieved content. A document or web page the agent reads instructs it to act, arriving through a trusted data channel.
Tool abuse. The agent uses legitimate credentials in a sequence no human would have approved, for example emailing a customer list to an external address.
Exfiltration through egress. The runtime reaches the public internet, so data leaves through a channel nobody monitored.
Credential sprawl. The agent runs under a human's account, so audit records attribute actions to a person who never made the decision.
No recovery path. When something goes wrong, there is no way to pause, roll back, or reconstruct what happened.

Each threat maps to a runtime control, and that mapping is what the review tests.

Control 1: A named identity for every agent

The first question in any review: what does the agent authenticate as? If the answer is "a service account shared with three other systems," the review stalls.

Mature setups treat each agent as an identifiable principal, a non-human identity with its own credentials, roles, and audit record. When the agent acts, the action is attributed to the agent, not to whoever deployed it. Revoking access revokes it at the provider, not just in an internal dashboard.

Where an agent must act through a shared account, for example a legacy system with no per-agent auth, that should require admin sign-off and be flagged in the audit trail.

Control 2: Frozen permissions, not vibes-based scope

The second question: what is the agent allowed to do? The control security teams ask for is an explicit, locked allowed-action list per agent: which tools, which objects, which operations.

The word "frozen" matters. If a connected vendor ships a new capability, the agent should not silently gain it. The permission list changes when someone changes it, and that change is itself audited. Object-level access per agent, knowledge base, and skill gives reviewers a scope they can read at a glance.

Control 3: An execution boundary that enforces the rules

Security teams ask where agent execution happens and what contains the blast radius.

The pattern that passes review: every run executes in an isolated sandbox, so a compromised or misbehaving run cannot reach neighboring systems, and the boundary that runs the agent is the same layer that enforces the rules on it. Runtime and governance are one layer, not two products bolted together. CreateOS Sandbox, for example, gives each session a Firecracker microVM with its own guest kernel, and CreateOS Agents runs multi-agent workflows with human-in-the-loop approval and audit on the same surface.

The boundary question also covers data. The deployment modes that clear review are the vendor's cloud, the customer's VPC, or fully on-premise, with zero data retention by default. For regulated workloads, data residency is a deployment decision, not a policy promise.

Control 4: Human approval on irreversible actions

Security teams want a control that lets an agent durably pause mid-task, ask a human for approval, and resume when the answer arrives. Not a timeout, not a retry loop: a real pause that preserves state.

Alongside approvals, they ask for hard limits per run: maximum steps, tokens, time, and tool calls. These turn "the agent went into a loop" from an incident into a contained event. Runs should be cancellable by a person, from the console, without a code change.

Control 5: Guardrails that fail closed

Guardrails, whether built in or from a cloud provider model service, only count if their failure mode is safe. The control security teams ask for is fail-closed: if the guardrail cannot evaluate the output, there is no answer. A guardrail that degrades to pass-through during an outage is an open door with a sign on it.

Control 6: An audit trail that survives the incident

The last question is accountability. After an incident, reviewers need to reconstruct what the agent did, in what order, under which permissions.

The control is an immutable, append-only, exportable audit log covering every run: model calls, tool calls, retrieved knowledge, approvals, and the exact configuration snapshot the run executed against. Reproducibility matters as much as recording. If a run is pinned to a fingerprinted snapshot of its model, knowledge, skills, tools, and limits, incident review can replay it rather than argue about it. Traces and usage dashboards make this usable day to day, not just during audits.

Kill switch and rollback sit here too: a documented way to stop an agent class and revert to a known state, exercised before the incident, not during it.

What to bring to the review

Structure your threat model around the six controls above and answer each concretely: identity, permissions, boundary, approvals, guardrail failure mode, audit. Then state the deployment mode and the data retention default. Teams that arrive with these answers clear review in one pass. Teams that arrive with a prompt-evaluation report do not.

CreateOS is built for exactly this gate: the governed execution layer that deploys, isolates, monitors, audits, and bills AI agents in your cloud, your VPC, or on-premise. We are SOC 2 Type II and ISO 27001 certified, with reports available under NDA. See the product overview or the documentation.

Frequently Asked Questions

What is a runtime control for an AI agent? A control that constrains or observes the agent while it executes: identity, permission enforcement, sandboxing, approvals, guardrails, and audit logging. Prompt evaluations are necessary but not sufficient; the runtime is where the risk lives.

Why is agent identity such a big deal in security reviews? Because attribution drives accountability. If an agent acts under a human's account, your audit log records a person who never made the decision. A per-agent non-human identity makes every action attributable and revocable.

Do guardrails replace sandboxing? No. Guardrails evaluate outputs; sandboxes contain execution. A guardrail failure is a wrong answer. A missing execution boundary is a compromised workload with network access. Production agents need both.

What should a threat model for agents cover that a normal app threat model misses? Chained tool use, instruction injection through retrieved content, egress as an exfiltration channel, and non-human attribution. The agent's capability is the combination of its tools, so the model has to cover sequences, not endpoints.

Conclusion

Security teams do not ask for a promise that the model will behave. They ask for runtime controls that hold when it does not: a named identity, frozen permissions, an isolated boundary, human approval on the irreversible steps, guardrails that fail closed, and an audit trail that can replay any run. Build the threat model around those six and the production review becomes a checklist instead of a standoff.

About CreateOS CreateOS runs AI agents and the code they write. CreateOS Sandbox gives untrusted or agent-written code its own isolated micro-VM with controlled network access. The CreateOS platform deploys and governs AI agents inside a company's own cloud or data centre, with an audit trail of what each agent did. Start at createos.sh.