paoloap wrote companion code for a Medium article on agent harnesses, including five layers: execution boundary, sandboxing, memory persistence, verification loops, and context pipelines, each with failure and guard scripts running alongside.
Move enforcement out of system prompts to deterministic code; a rule like "never delete without human approval" becomes a pre-execution hook denying the call, not a sentence the model might forget.Every demo runs with no API key; the model is replaced by a ~40-line scripted stand-in that emits a fixed sequence of tool calls, making each failure reproducible and the CI suite able to assert behaviour rather than smoke-test it. A single dependency-free `harness.py` can be dropped into any existing agent loop to add boundary checks, path allowlists, hostname allowlists, a persistent store, read-only review, dry-run, and token-cost distillation.
- `boundary()` detects coroutine functions and awaits them, fixing a silent no-op that made the guard a no-op on the most common (async) agent loops
- `host_allowlist` compares the parsed hostname, so `api.openai.com.evil.com` and `evil.com/?x=api.openai.com` are both refused
- `Denied` subclasses `str` so it drops into the same slot a tool result occupies, keeping existing loops unchanged
- `boundary(rules, max_repeats=3)` detects identical consecutive denials and changes the message to break a deterministic deadlock at full token cost
- The repo is MIT-licensed, Python 3.8+, and has no external dependencies
Move enforcement out of system prompts to deterministic code; a rule like "never delete without human approval" becomes a pre-execution hook denying the call, not a sentence the model might forget.Every demo runs with no API key; the model is replaced by a ~40-line scripted stand-in that emits a fixed sequence of tool calls, making each failure reproducible and the CI suite able to assert behaviour rather than smoke-test it. A single dependency-free `harness.py` can be dropped into any existing agent loop to add boundary checks, path allowlists, hostname allowlists, a persistent store, read-only review, dry-run, and token-cost distillation.
- `boundary()` detects coroutine functions and awaits them, fixing a silent no-op that made the guard a no-op on the most common (async) agent loops
- `host_allowlist` compares the parsed hostname, so `api.openai.com.evil.com` and `evil.com/?x=api.openai.com` are both refused
- `Denied` subclasses `str` so it drops into the same slot a tool result occupies, keeping existing loops unchanged
- `boundary(rules, max_repeats=3)` detects identical consecutive denials and changes the message to break a deterministic deadlock at full token cost
- The repo is MIT-licensed, Python 3.8+, and has no external dependencies
Paolo Perrone states the code "harness" around an LLM agent affects cost more than capability, breaking it into five layers: execution boundary, sandboxing, memory persistence, verification loops, and context pipelines.
The agent loop is six lines of Python; engineering determines what the model does, accesses, retains, checks, and sees, affecting token spend and unsanctioned actions.
A June 2026 preprint measured up to a 40x difference in tokens per solved task across three harnesses on the same model, while pass-rate gaps were 0-8 percentage points with confidence intervals crossing zero.
- The motivating anecdote: a prompt rule forbidding unapproved edits was violated 76 times in one afternoon; rewriting it as a pre-dispatch hook that returns a deny eliminated the failures entirely.
A denylist sandbox is defeated by path traversal (`work/../secrets/api_key`), whereas an allowlist combined with `os.path.normpath` before comparison catches every spelling of the same file.
The author notes six issues: no peer-reviewed harness study by August 2026, harness gains don't generalize to new tasks, a fine-tuning success contradicted its reward signal, harnesses aren't portable across models, the field lacks a definition, and harness effects decrease with better base models.
- All five layers ship as runnable Python scripts in a public repo (github.com/paoloap-py/agent-harness-guide) that use a scripted stand-in for the model, so every failure mode is reproducible without an API key.
The agent loop is six lines of Python; engineering determines what the model does, accesses, retains, checks, and sees, affecting token spend and unsanctioned actions.
A June 2026 preprint measured up to a 40x difference in tokens per solved task across three harnesses on the same model, while pass-rate gaps were 0-8 percentage points with confidence intervals crossing zero.
- The motivating anecdote: a prompt rule forbidding unapproved edits was violated 76 times in one afternoon; rewriting it as a pre-dispatch hook that returns a deny eliminated the failures entirely.
A denylist sandbox is defeated by path traversal (`work/../secrets/api_key`), whereas an allowlist combined with `os.path.normpath` before comparison catches every spelling of the same file.
The author notes six issues: no peer-reviewed harness study by August 2026, harness gains don't generalize to new tasks, a fine-tuning success contradicted its reward signal, harnesses aren't portable across models, the field lacks a definition, and harness effects decrease with better base models.
- All five layers ship as runnable Python scripts in a public repo (github.com/paoloap-py/agent-harness-guide) that use a scripted stand-in for the model, so every failure mode is reproducible without an API key.
The author explains how enabling the built-in sandbox feature in Claude Code has transformed their productivity by allowing for an autonomous workflow. By activating auto-allow mode via the /sandbox command, users can permit AI agents to execute repetitive tasks like running tests or installing dependencies without needing constant manual approval for every individual command, which reduces mental overhead and time spent waiting on prompts.
- Claude Code provides a built-in sandbox that uses macOS Seatbelt on Mac and Bubblewrap on Linux/WSL2.
- The environment restricts file writing primarily to the current project directory and temporary session files.
- Network access is controlled, requiring manual approval when the agent attempts to connect to a new domain.
- For maximum security and complete isolation from a system, using Docker containers or virtual machines remains more robust than the built-in sandbox features.
- Claude Code provides a built-in sandbox that uses macOS Seatbelt on Mac and Bubblewrap on Linux/WSL2.
- The environment restricts file writing primarily to the current project directory and temporary session files.
- Network access is controlled, requiring manual approval when the agent attempts to connect to a new domain.
- For maximum security and complete isolation from a system, using Docker containers or virtual machines remains more robust than the built-in sandbox features.
Simon Willison explores his latest approach to running untrusted Python code safely within applications by utilizing MicroPython inside a WebAssembly (WASM) sandbox. The project addresses the security risks of plugin systems where code normally executes with full privileges, potentially leading to data leaks or system compromise. By leveraging wasmtime and an alpha package called micropython-wasm, Willison demonstrates how to enforce memory and CPU limits while providing controlled access to host functions through a custom thread-based request queue for persistent interpreter state.
Main topics:
- Security challenges in Python plugin systems
- Advantages of WebAssembly as a sandboxing technology
- Building the micropython-wasm alpha package
- Implementation details for persistent state and host functions
- Integration with Datasette Agent to execute code via LLMs
Main topics:
- Security challenges in Python plugin systems
- Advantages of WebAssembly as a sandboxing technology
- Building the micropython-wasm alpha package
- Implementation details for persistent state and host functions
- Integration with Datasette Agent to execute code via LLMs
