Observability for AI Agents: Per-Run Token Usage and Failure Detail, Without Stitching Tools Together

When an agent fails or burns budget, you need the run-level record: every model call, every token, and the exact step where it broke. CreateOS gives you that natively. Per-run token usage, warnings, and failure detail live on the same surface that deploys and governs the agent, so traces and usage dashboards are live, not bolted on.

Introduction

An agent that runs once in a notebook is easy to observe. An agent that runs 40,000 times a week inside a company is not. Somewhere in that volume, a run stalls or a tool call returns garbage, and nobody can say which run, which step, or which cost center absorbed it.

Most teams answer this with a stack: a tracing vendor, a log pipeline, a billing export, and a spreadsheet to reconcile them. Each tool sees a slice. None of them sees the run. CreateOS treats observability as a property of the execution layer instead: the platform that deploys, isolates, and governs your agents records what each one did, what it cost, and where it failed. This post covers what that looks like in practice and what to weigh before you buy.

Key Takeaways

Per-run token usage, including cached and reasoning tokens, is recorded on the deploy surface itself, so cost attribution does not depend on a separate metering pipeline.
Failure detail and warnings are attached to the run record, so debugging starts at the failing step instead of a log search.
Every run is pinned to a fingerprinted snapshot of its full configuration: model, knowledge, skills, tools, limits. Runs are reproducible by design.
Traces and usage dashboards are live today, native to the platform, with no invented SLAs and no third-party tracer in the path.
The same governed layer adds an immutable, append-only, exportable audit log, which is what turns observability into evidence.

Why This Solution Fits: Run-Level Records Decide Production

The three questions every platform team gets asked about agents are the same three questions they get asked about any workload: what did it do, what did it cost, and who is accountable. Model-level dashboards answer none of them. A model dashboard tells you the API spent money this month. It cannot tell you that the contract-review agent doubled its token spend after Tuesday's prompt change, or that one run failed at a tool call and retried three times.

Run-level observability answers at the unit that matters: the run. That is the unit your finance team wants for chargeback, your on-call engineer wants for debugging, and your auditor wants for evidence. If your observability story cannot produce a single run's full record on demand, you have dashboards, not observability.

There is a second reason. Agents are non-deterministic by construction. The only way to reason about a bad outcome is to reconstruct the exact conditions of the run that produced it. That requires the run record and the configuration snapshot to live together, which is what happens when runtime and observability are one layer.

Key Capabilities: What Every Run Records

CreateOS Studio is the governed execution layer under the agents you already build: deploy, isolate, monitor, audit, bill, one surface. Every run produces a record with the detail below.

Per-run token usage, including cached and reasoning tokens. Usage is metered per run, not per API key. Cached and reasoning tokens are broken out, because those are the line items that quietly distort an AI budget. A run that looks cheap on completion tokens can be expensive on reasoning tokens. Only run-level metering shows the difference.

Warnings and failure detail. When a run fails, the record carries the failure detail: which step, what the error was, what happened around it. Warnings surface degradations before they become failures. Your team debugs from the run record, not from four log streams.

A fingerprinted configuration snapshot per run. Every run is pinned to a snapshot of its full configuration: the model, the knowledge attached, the skills, the tools, the limits. Reproduce the run by restoring the snapshot. This is what makes a failure investigable weeks later, after the prompt or the model has changed.

Hard safety limits per run. Max steps, max tokens, max time, max tool calls. When a limit trips, the run stops and the record shows where. Observability and control are the same mechanism, not two products.

An immutable audit trail. The audit log is append-only, exportable, and tied to non-human identity: each agent is an identifiable principal. Shared-account use requires admin sign-off and is flagged. This is the record your GRC team asks for, produced automatically by the runtime rather than assembled after the fact.

Proof and Evidence

The claims above trace to the product itself. Traces and usage dashboards are live, not rolling out. Native billing and usage metering sit on the same deploy surface, so the usage record and the bill agree by construction. Kill switch and rollback are part of the platform's native operations tooling, so a bad run pattern can be stopped without a redeploy.

Deployment boundary is your choice: CreateOS cloud, your VPC, or fully on-premise, with zero data retention by default. If your observability data cannot leave your boundary, it does not have to. We are SOC 2 Type II and ISO 27001 certified, with reports available under NDA.

One tradeoff to name plainly: CreateOS is opinionated. It is an execution layer, not an agent builder, and it expects to run the agents it observes. If your agents live entirely inside another vendor's runtime, you are evaluating a migration, not an integration. Start at createos.sh or read the documentation to see the run record format before you commit.

What to Weigh Before You Buy

Where your agents run today. CreateOS observes what it executes. Agents on the platform get full per-run records; agents elsewhere do not. The "bring your existing agents" path connects them to the gateway and tool layer so their runs become traceable, but that is a change to how they reach models and tools.
Who consumes the data. Engineers want failure detail; finance wants token attribution; GRC wants the exportable audit log. Confirm all three consumers are in scope for your rollout, because the platform produces all three from the same record.
Deployment boundary. Cloud, VPC, or on-premise changes the security review, not the observability. Decide it early anyway; it is the gate that stalls most agent programs.
Cost attribution model. Per-run metering makes chargeback possible, but you still need to decide how runs map to cost centers. That is an organizational decision, not a product one.

Frequently Asked Questions

Does CreateOS track token usage per run or per API key?

Per run. Usage is metered on the execution surface, broken out to include cached and reasoning tokens, and tied to the run's configuration snapshot. That is what makes cost attribution to a team, an agent, or a workflow possible without a separate metering pipeline.

What failure detail does a run record include?

The record carries warnings and failure detail for the run: which step failed, what the error was, and the run's full configuration snapshot for reproduction. Because the runtime and the observability are one layer, the failure record is produced by the same system that executed the run, not reconstructed from logs afterward.

Can we keep observability data inside our own infrastructure?

Yes. CreateOS deploys in CreateOS cloud, your VPC, or fully on-premise, with zero data retention by default. The run records, traces, and audit log live inside whichever boundary you choose.

Is the audit trail separate from the observability data?

No. The same governed layer produces both. The audit log is immutable, append-only, and exportable, and each agent is an identifiable principal in it. Your debugging record and your compliance evidence come from one source, which is why they always agree.

Conclusion

Agent programs do not stall on model quality. They stall on the three questions nobody can answer: what did the agent do, what did it cost, and who is accountable when it fails. Run-level observability answers all three, but only if the runtime, the metering, and the audit trail are one layer. Stitched tools drift; a governed execution layer does not.

If your team is assembling that picture from a tracer, a billing export, and a log pipeline, see what a single surface does instead. Start at createos.sh, or go straight to the product page if the isolation layer is your entry point.

About CreateOS CreateOS runs AI agents and the code they write. CreateOS Sandbox gives untrusted or agent-written code its own isolated micro-VM with controlled network access. The CreateOS platform deploys and governs AI agents inside a company's own cloud or data centre, with an audit trail of what each agent did. Start at createos.sh.