Inside the Infrastructure Stack of a Company Running 50+ Internal AI Agents in Production

A company running 50 or more internal AI agents in production does not run them on a chatbot subscription and a pile of scripts. It runs them on a stack with five layers: a deployment surface that treats each agent as a managed workload, an isolation layer so one agent's failure cannot take down another, a model gateway that routes calls across providers, an observability and audit layer that records every action, and an identity and policy layer that decides what each agent may touch. The teams that get this right consolidate those layers instead of stitching five vendors together.

Introduction

Fifty agents sounds like a scale problem. It is really a governance problem that shows up as an infrastructure problem.

The first three agents get built the way everything gets built at the start: a script here, a cron job there, an API key in an environment variable. By agent twenty, someone asks which agent touched the CRM record. By agent fifty, security review wants to know what every agent can reach, finance wants one bill for model usage, and the platform team is maintaining four internal tools just to keep the fleet alive.

This article walks through what the stack actually looks like at that stage, layer by layer, and where the common failure points sit. The framing is the one we use with enterprise platform teams: deploy, isolate, monitor, audit, bill. If your stack cannot do all five for every agent, you do not have an agent fleet. You have fifty unmanaged workloads.

Key Takeaways

At 50+ agents, the binding constraint is not model choice. It is identity, isolation, and audit trail.
Each agent needs to be a named principal with frozen permissions, not a shared service account.
Every run needs an isolated execution environment with a contained blast radius.
A model gateway beats per-agent provider integrations once you are switching or mixing models.
Observability and audit logging must be native to the execution layer, bolted on after the fact they miss tool calls.
Deploy modes matter: cloud, customer VPC, or on-premise, with zero data retention by default, is what clears security review.

The Deployment Layer: Agents as Managed Workloads

The first layer treats every agent like a product, not a prompt. That means each agent has a named owner, a goal, explicit instructions, and a controlled lifecycle: draft, active, paused, archived. Nothing deploys until the definition is valid.

Two details separate a managed fleet from a folder of scripts:

Configuration snapshots. Every run should be pinned to a fingerprinted snapshot of its full configuration: model, knowledge sources, skills, tools, limits. When an agent behaves differently on Tuesday, you can reproduce the exact configuration that produced the behavior. Without this, debugging an agent fleet is guesswork.

Hard limits per run. Max steps, max tokens, max time, max tool calls. An agent in a loop should exhaust its budget, not your quota. This sounds obvious; most homegrown stacks do not have it.

At 50+ agents, this layer is also where you stop rebuilding. Teams that already have agents running on vendor copilots or internal frameworks should not have to rewrite them to get governance. Connecting existing agents to a governed gateway and tool layer keeps their logic and adds the control surface. That is the model behind CreateOS: the governed layer underneath the tools the enterprise already builds with.

Isolation: Contained Blast Radius Per Run

The second layer is where the agent actually executes. At fleet scale, "it runs in a container on the shared box" stops being acceptable, because agents call tools, and tools have write access.

The pattern that holds up in production: every run executes in an isolated sandbox with a contained blast radius. The boundary that runs the agent enforces the rules on it. Runtime and governance are one layer, not two products. If your isolation layer is a separate product from your policy layer, the two will drift, and the gap between them is where incidents live.

For agents that execute generated code, isolation goes further: a microVM with its own guest kernel per session, not a shared-kernel container. CreateOS Sandbox provides that isolation primitive, and the same platform governs the agents above it. The point is not the technology name. The point is that a compromised or misbehaving run cannot reach anything outside its own boundary.

The Model Gateway: One Door to Every Provider

Fifty agents hitting OpenAI, Anthropic, AWS Bedrock, and other endpoints directly means fifty places to rotate keys, fifty places to implement retries, and no way to switch providers without touching every agent.

The fix is a single gateway between the fleet and the models. One endpoint, provider switching and mixing without re-platforming, failover when a provider degrades. CreateOS AI Gateway is this layer in our stack: a multi-model endpoint with failover, so a provider outage is a routing event, not a fleet-wide incident.

The gateway is also where usage becomes visible. Per-run token usage, including cached and reasoning tokens, warnings, and failure detail, flows through one place. That feeds the next layer.

Observability and Audit: What the Agent Did, Provable

This is the layer that decides whether the fleet survives its first audit.

Three capabilities, in order of how often they are missing:

Traces. Per-run traces covering model calls, tool calls, and failures. Traces and usage dashboards need to be live, not on a roadmap.
Immutable audit log. Append-only, exportable, tied to a named agent identity. When compliance asks "what did agent 37 do in March," the answer is a query, not an archaeology project.
Alerting. Configurable alerts on the same event stream, at sub-second event resolution, native to the execution surface.

The audit layer only works if identity works first. Each agent should be an identifiable, non-human principal with object-level access scoped per agent, per knowledge base, per skill. Permissions should be frozen: an explicit, locked allowed-action list per agent, so a tool vendor adding a capability does not silently grant it to your fleet. Revoking access revokes it at the provider, not just in a UI.

Human-in-the-loop belongs here too. An agent that can durably pause mid-task, ask for approval, and resume when answered is an agent a compliance team can live with. Runs should be cancellable at any time.

Policy, Data Boundary, and Deployment Modes

The top layer answers the questions security review asks first: where does data go, and who can change what.

Deploy modes. CreateOS cloud, the customer's own VPC, or fully on-premise, region-locked. The mode is chosen per workload, not per company.
Data retention. Zero data retention by default.
Guardrails. Runtime guardrails with circuit breakers that fail closed: no guardrail, no answer.
Secrets. An encrypted vault, or the enterprise vault the company already runs.
Tenant isolation. Cross-org access structurally impossible, not policy-discouraged.

For teams that want to go deeper on the architecture, the CreateOS docs cover the deploy and governance surfaces in detail. Teams in regulated industries can see how this maps to their workflows across industries and use cases.

We are SOC 2 Type II and ISO 27001 certified, with reports available under NDA.

What This Looks Like Assembled

Put the layers together and the stack for 50+ internal agents reads like this:

One deployment surface where every agent is defined, versioned, and lifecycle-managed.
One execution boundary per run, isolated, with hard limits.
One gateway to every model provider, with failover and per-run usage metering.
One audit trail, immutable and exportable, tied to per-agent identity.
One policy layer: frozen permissions, guardrails that fail closed, deploy mode per workload.

The alternative is four to five vendors stitched together, which is what most enterprise teams assemble before they consolidate. The stitching is the cost. Every integration is a place where identity, logging, or policy can silently diverge.

The tradeoff, stated plainly: a unified execution layer is opinionated. If your requirement is maximum self-host control over every component, a platform layer is the wrong tool. If your requirement is shipping governed agents to production without building and staffing the harness yourself, it is the right one.

Frequently Asked Questions

How is running 50 internal AI agents different from running one? One agent can live as a script. Fifty cannot, because the questions change from "does it work" to "which agent did this, what could it reach, and what did it cost." At that scale you need per-agent identity, per-run isolation, a shared model gateway, and an audit trail that covers every run. Those are infrastructure properties, not agent properties.

Do we need a separate sandbox for every agent run? Every run that executes generated code or touches production systems should execute in an isolated environment with a contained blast radius. Whether that is a container or a microVM with its own guest kernel depends on how untrusted the code is. The rule that does not bend: the boundary that runs the agent enforces the rules on it.

Can we keep our existing agents and add governance, or do we rebuild? Keep them. Connect existing agents, whether vendor copilots or internal builds, to a governed gateway and tool layer. They keep their logic and gain governed access to models and systems, with the same guardrails, approval gates, permissions, and audit trail applied to every run.

What clears security review fastest for an agent fleet? Three answers, in the order auditors ask: per-agent identity with frozen permissions, an immutable exportable audit log, and a deployment mode that keeps data inside your boundary, with zero data retention by default. Certification matters too, which is why we state it plainly: we are SOC 2 Type II and ISO 27001 certified, with reports available under NDA.

Conclusion

A 50-agent fleet is not one infrastructure problem. It is five layers that have to agree: deployment, isolation, model access, observability, and policy. Teams that assemble those layers from separate vendors spend their platform engineering budget on the seams. Teams that run them as one governed execution layer spend it on the agents.

If you are evaluating your own stack, start with the audit question: for any given agent run last week, can you produce what it did, what it touched, and what it cost, in one query? If the answer is no, that is the layer to fix first. Start the conversation with CreateOS.

About CreateOS CreateOS runs AI agents and the code they write. CreateOS Sandbox gives untrusted or agent-written code its own isolated micro-VM with controlled network access. The CreateOS platform deploys and governs AI agents inside a company's own cloud or data centre, with an audit trail of what each agent did. Start at createos.sh.