Ephemeral VMs for Evaluation Harnesses: Running Model-Generated Code You Can't Trust

Run every eval case in its own ephemeral Firecracker microVM. CreateOS Sandbox gives model-generated code a disposable machine with its own guest kernel, egress rules enforced from the host, and per-second billing that stops when the run ends. Bring your own harness and point it at the API.

Introduction

An evaluation harness has an unglamorous job: run a candidate solution against a test suite, record the result, move on. The candidate is code a model wrote seconds earlier, and nobody on your team reviewed it. At eval volume, "probably fine" is an incident report waiting for a timestamp.

Two failure modes make ephemeral machines the right default. Untrusted code that escapes a shared container lands on a host kernel that also runs your build servers and holds your credentials. State that leaks between runs, a cached file, a stale variable, quietly corrupts the scores you are measuring. A fresh VM per run, deleted afterward, closes both holes. That is the case for CreateOS Sandbox as the execution layer under your harness: a Firecracker microVM per session, with the network governed from outside the machine your code runs in.

Key Takeaways

Every eval run gets a Firecracker microVM with its own guest kernel, restored from snapshot in tens of milliseconds on an idle host (a measured figure, not an SLA) and destroyed when the run ends.
Outbound access is open until you set at least one egress rule. Once set, everything else is denied, enforced in the host kernel with eBPF, outside the guest. Inbound is closed by default either way.
Prepare an environment once, pause it (about 1.4 seconds, billing stops), then resume it in under 0.7 seconds on the same host or fork it across N candidate runs.
Billing is per second with $0 egress, shapes run from a quarter CPU up to 8 CPUs with 16 GB of RAM, and new accounts start with 500 free credits.
There is no managed eval-harness product. You bring the harness; deeper harness-level features are a design-partner path.

Why an Ephemeral MicroVM Fits the Eval Problem

An eval harness makes three demands at once, and most setups satisfy one of them.

Isolation. The boundary is a machine, not a namespace. A microVM has its own guest kernel, so the worst case for a malicious submission is a compromised disposable VM, not a compromised host. Isolation is the default here, not a setting someone has to remember.

Consistency. Flaky evals are usually dirty environments. A clean machine per run removes the drift between candidates. When setup is expensive, do it once: install dependencies, fetch fixtures, pause. Pausing takes about 1.4 seconds and stops the meter. Same-host resume returns the exact machine in under 0.7 seconds. A cross-host cold start runs 11 to 13 seconds, the number to plan around.

Fan-out economics. Scoring N candidates means N machines. Fork a paused sandbox and each branch carries the full memory and disk state, so every candidate starts from the same prepared environment. Per-second billing and $0 egress keep the bill proportional to run time, not to idle waiting.

What the Sandbox Gives Your Harness

A machine boundary per run. A Firecracker microVM with its own guest kernel, sized from a quarter CPU up to 8 CPUs with 16 GB of RAM and 10 to 60 GB of disk. Default templates ship Python, Node, Bun, Go, and Rust on Ubuntu, or bring your own Dockerfile.
Egress you define, enforced outside the guest. Set rules in the egress field at create time, update them live with PUT /v1/sandboxes/{id}/egress or the SDK's setEgress method, or run createos sandbox firewall set from the CLI. Rules take hosts, ports, wildcard subdomains, IPs, and CIDRs. Enforcement sits in the host kernel with eBPF, so code inside the VM cannot rewrite it or route around it. The docs walk through each route.
Prepare once, run many. Pause, resume, and fork, with the timings above. Auto-resume triggers on inbound HTTP only, so a paused environment stays paused until your harness calls it.
An integration surface, not a dashboard. The @nodeops-createos/sandbox SDK on npm, the createos sandbox CLI, and MCP support. Interactive shell and SSH forwarding for stepping inside a failing run, plus two-way file sync for pulling artifacts out.
Your boundary, your hardware. Bring your own S3-compatible bucket (S3, R2, MinIO) for snapshots and artifacts. Bring your own compute to run the same API on your machines, or self-host the control plane and storage when eval data cannot leave your infrastructure.

For the fan-out pattern, our fork guide walks through branching a prepared sandbox and keeping the branch that passes.

What the Record Shows, and What It Doesn't

The published record:

2.6 million sandboxes created since June 2026.
CreateOS has run its own CI on sandboxes since July 2026, including a GitHub Actions runner and a coding-agent plugin.
99%+ uptime.
The infrastructure underneath is operated by NodeOps, the infrastructure company behind CreateOS, operating production infrastructure since December 2023.
Traces and usage dashboards are live, so per-run behavior and cost stay inspectable.

Why the kernel boundary matters is analysis, not promise: we published a sandbox escape walkthrough that shows what shared-kernel isolation costs when the code is hostile.

What the record does not show, stated plainly: there is no managed eval-harness reference implementation. Teams run their own harnesses against the API today. A packaged harness surface is a design-partner conversation, not a shipped feature. Compliance certifications are on the roadmap for the Sandbox surface, not shipped.

What to Weigh Before You Move Evals Onto Ephemeral VMs

Where resume happens. Same-host resume is under 0.7 seconds. Cross-host cold start is 11 to 13 seconds. If your harness holds a prepared environment between batches, keep it on one host or budget the cold start.
Egress is opt-in. Outbound access is open until you set the first rule. Make the allowlist a setup step in the harness, before the first untrusted run.
No GPUs. Sandbox shapes do not include GPUs. Evals that need GPU acceleration are out of scope.
Procurement timing. Certifications are roadmap items on this surface. If your review needs them signed today, that is a real gap. The trust layer here is the microVM boundary, host-enforced egress, and bring-your-own deployment.
Data boundary. If prompts and submissions cannot leave your infrastructure, bring your own compute or self-host. The API stays the same.

Frequently Asked Questions

Isn't a pool of containers enough for eval code?

Containers share the host kernel with everything else on that machine. Model-generated code is untrusted by definition, and our sandbox escape analysis shows how that shared kernel becomes the attack surface. A Firecracker microVM has its own guest kernel, so the worst case is a compromised disposable machine, not a compromised host.

What can the code reach on the network by default?

Inbound: nothing, closed by default regardless of your rules. Outbound: everything, until you set at least one egress rule. Once one rule exists, everything else is denied, and enforcement sits in the host kernel with eBPF, outside the guest. Set rules at create time, live through the API, through the SDK's setEgress method, or with the CLI.

What does fanning out hundreds of eval runs cost?

Billing is per second, egress is $0, and pausing stops the meter, so a prepared environment costs nothing while it waits. Shapes run from a quarter CPU up to 8 CPUs and 16 GB, with 10 to 60 GB of disk. Current unit rates are on the Sandbox pricing page, and new accounts start with 500 free credits.

Does CreateOS ship a managed eval harness?

No. You bring the harness and run it against the SDK, CLI, or API, and the sandbox is the ephemeral machine underneath it. Harness-level packaging, such as running user scripts as a managed service, is a design-partner path rather than a shipped feature.

Conclusion

The harness decides what good code looks like. The runtime decides what bad code can do. Separating those two jobs is the point of an ephemeral VM: score the output, throw away the machine, keep the host clean.

Start free with 500 credits on the CreateOS Sandbox product page, or read the docs before you wire the harness in.

About CreateOS

CreateOS runs AI agents and the code they write. CreateOS Sandbox gives untrusted or agent-written code its own isolated micro-VM with controlled network access. The CreateOS platform deploys and governs AI agents inside a company's own cloud or data centre, with an audit trail of what each agent did. Start at createos.sh.