Awesome Testing

Agents and harnesses · reviewed · reviewed Aug 30, 2026 · 3 min

What is an agent sandbox?

A sandbox places execution inside an isolated, resource-limited environment with deliberately scoped files, processes, network, credentials, lifetime, and outputs, while policy outside the sandbox still decides which actions are authorized.

Isolation limits the blast radius of execution; it does not replace permission checks, careful tool design, or a threat model.

Put execution behind a boundary

An agent may generate a shell command, program, build script, browser action, or test. Treat the resulting workload as untrusted even when the user asked for it. The harness prepares an isolated environment, mounts only required inputs, applies limits, executes, captures declared outputs, and then destroys or resets the environment.

flowchart LR
  H[Harness policy] --> P[Prepare isolated workspace]
  I[Explicit inputs] --> P
  P --> X[Untrusted execution]
  X --> F[Scoped filesystem]
  X --> N[Filtered network]
  X --> R[CPU, memory, process, time limits]
  X --> O[Declared outputs and logs]
  O --> V[Validate before release]
  V --> H

The boundary can use operating-system isolation, an application kernel such as gVisor, a microVM such as Firecracker, a separate machine, or layered controls. These options expose different host surfaces and have different compatibility, startup, density, and performance trade-offs. “Runs in a container” is an implementation fact, not a complete security claim.

Scope every channel

Isolation is multidimensional. Define each channel explicitly:

  • Filesystem: a fresh workspace, read-only inputs where possible, no host home directory, and no implicit mounts of sockets or configuration.
  • Processes: an unprivileged identity, minimal capabilities, process limits, and no access to host process namespaces.
  • Network: default-deny egress or a small allowlist, protected metadata endpoints, bounded DNS, and no assumption that outbound traffic is harmless.
  • Credentials: short-lived, task-specific credentials injected only when required and never copied into model context or durable artifacts.
  • Resources: CPU, memory, storage, file descriptors, subprocesses, output bytes, and wall-clock time.
  • Lifecycle: a known image, one task or tenant per isolation unit, explicit cancellation, output extraction, and reliable cleanup.

A writable repository mount may be the intended capability for a coding agent, but it should not accidentally expose SSH keys, package-registry credentials, sibling repositories, or the container runtime socket. Network access for dependency installation should not imply access to every internal service.

Isolation and authorization are different

A sandbox answers “where can this workload have effects?” Authorization answers “may this user perform this exact effect now?” Both are required.

Code can stay perfectly inside a sandbox and still misuse an allowed production API credential. Conversely, a fully authorized task can contain buggy or hostile generated code that should not run on the host. Approval, tenant checks, business policy, and idempotency belong at tool and effect boundaries outside the workload.

Outputs cross the boundary too. Treat generated files, archives, logs, links, and test reports as untrusted. Validate type and size, prevent path traversal and symlink escape during extraction, scan or render risky formats safely, and preserve provenance.

Containment has a boundary

A sandbox is not proof of safety and not a substitute for secure architecture. System-call filtering reduces exposed kernel surface but, by itself, is not a complete sandbox. Virtualization adds a boundary but still depends on host configuration, patching, device exposure, network controls, and a correct lifecycle.

Isolation also does not make secrets safe to expose. If a workload can read a credential and reach an attacker-controlled endpoint, it may exfiltrate it without escaping. Side channels and shared hardware may remain outside the chosen threat model and should be stated rather than silently ignored.

Validate escape and cleanup paths

Turn the threat model into executable probes. Attempt reads and writes through absolute paths, .., symlinks, hard links, procfs, device files, inherited descriptors, environment variables, sockets, and mounted runtime APIs. Probe loopback, DNS, cloud metadata addresses, private networks, redirects, and alternate encodings. Verify that denied access creates no host effect.

Exercise fork bombs, large outputs, disk fill, memory pressure, CPU loops, descriptor exhaustion, nested archives, cancellation races, and processes that ignore signals. Confirm limits apply to descendants and cleanup works after timeout, harness crash, and host restart.

Run parallel tenants and prove that files, process state, caches, network identity, credentials, snapshots, and output channels do not cross. Compare the prepared environment with its declared manifest, patch the isolation stack, and keep escape tests in a dedicated security suite. Finally, verify the product-level contract: only declared outputs leave the sandbox, and an isolated success cannot bypass authorization at the real effect boundary.

Sources and further reading

  1. 01
    Security ModelgVisor · documentation · source checked Aug 30, 2026

    An official threat model and defense-in-depth description for an application-kernel sandbox, including explicit limitations.

  2. 02
    Firecracker DesignFirecracker · documentation · source checked Aug 30, 2026

    The official microVM architecture and threat-containment layers, including process jailing and unfiltered guest egress.

  3. 03
    Seccomp BPFLinux kernel documentation · documentation · source checked Aug 30, 2026

    The kernel interface for reducing system-call surface and its explicit warning that filtering alone is not a sandbox.

  4. 04
    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsXie et al. · research · published Apr 11, 2024 · source checked Aug 30, 2026

    A primary agent benchmark using resettable virtual machines for realistic file, application, and operating-system effects.