@omarsar0 writes on X that the fastest path to genuinely understanding agent harnesses is to build one from scratch in TypeScript or Python, starting with a minimal ReAct implementation prompted from Google's original paper, targeting three clean components—an LLM inference module (multi-model, OpenRouter-backed, with separable system prompt), an MCP tools module for interoperability, and a simple agent loop that ties them together—then logging every input/output at each boundary and iterating against a small set of diverse test tasks so each change is inspectable. The punchline: skip the framework first, because only once you've felt the loop, the tokens, and the tool calls in your own code do the "next steps"—memory, skills, subagents—stop being black boxes you configure and become modules you actually know how to tune.
- LLM module: wraps inference across multiple frontier models via OpenRouter; system prompt either embedded or isolated for context-engineering experiments
- Tools module: implement as MCP (Model Context Protocol) tools for cross-harness interoperability, or as bespoke functions if experienced
- Agent loop: ReAct pattern (alternating reasoning traces and action calls) encapsulating both LLM and tools; exit conditions handled via system-prompt instructions (non-deterministic), code-level checks (deterministic), or both
- Logging strategy: capture loop in/out, every LLM call in/out, and every tool-call in/out; run a fixed diverse task suite after each modification
- Scaling path: keep architecture modular so memory, skills, and subagent orchestration can be bolted on once the core loop is understood
- Shortcut alternatives (if not building from scratch): Pi SDK or LangChain harness tooling
- LLM module: wraps inference across multiple frontier models via OpenRouter; system prompt either embedded or isolated for context-engineering experiments
- Tools module: implement as MCP (Model Context Protocol) tools for cross-harness interoperability, or as bespoke functions if experienced
- Agent loop: ReAct pattern (alternating reasoning traces and action calls) encapsulating both LLM and tools; exit conditions handled via system-prompt instructions (non-deterministic), code-level checks (deterministic), or both
- Logging strategy: capture loop in/out, every LLM call in/out, and every tool-call in/out; run a fixed diverse task suite after each modification
- Scaling path: keep architecture modular so memory, skills, and subagent orchestration can be bolted on once the core loop is understood
- Shortcut alternatives (if not building from scratch): Pi SDK or LangChain harness tooling
mini-swe-agent is a radically simple Python-based agent from the Princeton and Stanford team behind SWE-bench that uses only bash as its tool, maintains a completely linear message history, and executes each action via independent subprocess.run calls. Despite being roughly 100 lines of core agent code, it scores over 74% on SWE-bench verified and is used by organizations including Meta, NVIDIA, IBM, and Anyscale.
- The core design argument is that as language models grow more capable, elaborate tool scaffolds become unnecessary and the LM itself should drive the shell
- Supports sandboxed deployment via docker, podman, singularity, bwrap, and others; installable from PyPI via uvx, pipx, or pip
- The core design argument is that as language models grow more capable, elaborate tool scaffolds become unnecessary and the LM itself should drive the shell
- Supports sandboxed deployment via docker, podman, singularity, bwrap, and others; installable from PyPI via uvx, pipx, or pip
Anurag Singh writes that providing Claude Code with read-only access to a SaaS application's server logs allowed the coding agent to identify and propose fixes for real performance issues. By observing error patterns, traces, and metrics directly within the environment rather than relying on manual bug reports, the agent was able to autonomously trace bugs back to specific lines of code across various files.
- The experiment highlights a shift toward AI agents joining the "on-call" workflow by inspecting live operational telemetry.
- To mitigate security risks, it is recommended using Model Context Protocol (MCP) servers to restrict an agent's tools to read-only actions.
- Major observability companies like Sentry and Datadog are already implementing similar features to automate root cause analysis and pull request generation.
- The experiment highlights a shift toward AI agents joining the "on-call" workflow by inspecting live operational telemetry.
- To mitigate security risks, it is recommended using Model Context Protocol (MCP) servers to restrict an agent's tools to read-only actions.
- Major observability companies like Sentry and Datadog are already implementing similar features to automate root cause analysis and pull request generation.
Yuhao Wu writes about HarnessDev, a benchmark that evaluates LLMs' ability to build and iteratively improve their own agent harness—the model-external execution infrastructure that wraps a model and shapes its task performance. The benchmark has two stages: Creation, where the agent builds a complete execution system from a minimal seed and a few cases, and Evolution, where it revises its own harness using downstream execution feedback. Generated harnesses substantially lag behind mature human-engineered references on code and search/research, while matching or exceeding them on writing and machine-learning experimentation, with large variation in execution cost.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
NPC-Worldwide (primary contributor cagostino) presents npcsh, a composable multi-agent shell that interprets both bash commands and natural language within a single interactive interface. Built primarily in Rust with a Python backend (npcpy) for the LLM inference loop, it lets users delegate tasks to named agents, define custom "Jinxes" (Jinja Execution templates) for tool-use and skills, and works with any model provider LiteLLM supports. A 100-task benchmark suite scores how well various models can drive the shell, with results ranging from 23% (Qwen3.5 0.8b) to 97% (Qwen3.5 35b, Ornith 35b, Kimi K2.7-Code 1t).
- Agent definitions support three interchangeable formats: .npc YAML files, agents.md markdown, and agents/ directories with per-agent .md files
- The Python backend (npcpy) is explicitly temporary and slated for replacement by a Rust-native runner (npcrs)
- The project references an arxiv paper on "ALARA for Agents: Least-Privilege Context Engineering Through Portable Composable Multi-Agent Teams"
- Supports local model runtimes including Ollama, LM Studio, and MLX (Apple Silicon)
- Currently at v2.1.16 with 128 releases, 473 stars, and 854 commits
- Agent definitions support three interchangeable formats: .npc YAML files, agents.md markdown, and agents/ directories with per-agent .md files
- The Python backend (npcpy) is explicitly temporary and slated for replacement by a Rust-native runner (npcrs)
- The project references an arxiv paper on "ALARA for Agents: Least-Privilege Context Engineering Through Portable Composable Multi-Agent Teams"
- Supports local model runtimes including Ollama, LM Studio, and MLX (Apple Silicon)
- Currently at v2.1.16 with 128 releases, 473 stars, and 854 commits
Paolo Perrone states the code "harness" around an LLM agent affects cost more than capability, breaking it into five layers: execution boundary, sandboxing, memory persistence, verification loops, and context pipelines.
The agent loop is six lines of Python; engineering determines what the model does, accesses, retains, checks, and sees, affecting token spend and unsanctioned actions.
A June 2026 preprint measured up to a 40x difference in tokens per solved task across three harnesses on the same model, while pass-rate gaps were 0-8 percentage points with confidence intervals crossing zero.
- The motivating anecdote: a prompt rule forbidding unapproved edits was violated 76 times in one afternoon; rewriting it as a pre-dispatch hook that returns a deny eliminated the failures entirely.
A denylist sandbox is defeated by path traversal (`work/../secrets/api_key`), whereas an allowlist combined with `os.path.normpath` before comparison catches every spelling of the same file.
The author notes six issues: no peer-reviewed harness study by August 2026, harness gains don't generalize to new tasks, a fine-tuning success contradicted its reward signal, harnesses aren't portable across models, the field lacks a definition, and harness effects decrease with better base models.
- All five layers ship as runnable Python scripts in a public repo (github.com/paoloap-py/agent-harness-guide) that use a scripted stand-in for the model, so every failure mode is reproducible without an API key.
The agent loop is six lines of Python; engineering determines what the model does, accesses, retains, checks, and sees, affecting token spend and unsanctioned actions.
A June 2026 preprint measured up to a 40x difference in tokens per solved task across three harnesses on the same model, while pass-rate gaps were 0-8 percentage points with confidence intervals crossing zero.
- The motivating anecdote: a prompt rule forbidding unapproved edits was violated 76 times in one afternoon; rewriting it as a pre-dispatch hook that returns a deny eliminated the failures entirely.
A denylist sandbox is defeated by path traversal (`work/../secrets/api_key`), whereas an allowlist combined with `os.path.normpath` before comparison catches every spelling of the same file.
The author notes six issues: no peer-reviewed harness study by August 2026, harness gains don't generalize to new tasks, a fine-tuning success contradicted its reward signal, harnesses aren't portable across models, the field lacks a definition, and harness effects decrease with better base models.
- All five layers ship as runnable Python scripts in a public repo (github.com/paoloap-py/agent-harness-guide) that use a scripted stand-in for the model, so every failure mode is reproducible without an API key.
Ory Team states traditional IAM frameworks (MFA, SSO, fixed API keys, IP whitelisting) are insufficient for autonomous AI agents entering production, as these agents reason, use tools, and execute multi-step workflows without constant human intervention.
The article describes six identity capabilities for securing these agents: verifiable cryptographically-signed agent identities with delegation chains, just-in-time ephemeral credentials, relationship-based access control bound to task intent, machine-speed automated containment via circuit breakers, in-the-loop runtime policy enforcement with configurable human approvals, and a web-scale identity control plane that handles machine-speed throughput and rapid sub-agent lifecycle governance.
- Sponsored post by Ory; Insight Partners (TNS owner) is an investor in both Ory and TNS.
- Ken Buckler (EMA Research Director) is quoted: "most organizations are woefully unprepared" for the security risks of managing agentic identities.
- A comparison table contrasts agents with humans and service accounts across velocity, decision logic, auth mechanics, and access granularity, highlighting that agents need ephemeral delegation and contextual attestation rather than passkeys or static keys.
- The ReBAC example given: "Agent X may read Document Y only if human user Z is the document owner and the active workflow is 'Data Summarization'."
- PKCE and strict token-binding are called out to prevent credential replay outside the agent's intended runtime context.
The article describes six identity capabilities for securing these agents: verifiable cryptographically-signed agent identities with delegation chains, just-in-time ephemeral credentials, relationship-based access control bound to task intent, machine-speed automated containment via circuit breakers, in-the-loop runtime policy enforcement with configurable human approvals, and a web-scale identity control plane that handles machine-speed throughput and rapid sub-agent lifecycle governance.
- Sponsored post by Ory; Insight Partners (TNS owner) is an investor in both Ory and TNS.
- Ken Buckler (EMA Research Director) is quoted: "most organizations are woefully unprepared" for the security risks of managing agentic identities.
- A comparison table contrasts agents with humans and service accounts across velocity, decision logic, auth mechanics, and access granularity, highlighting that agents need ephemeral delegation and contextual attestation rather than passkeys or static keys.
- The ReBAC example given: "Agent X may read Document Y only if human user Z is the document owner and the active workflow is 'Data Summarization'."
- PKCE and strict token-binding are called out to prevent credential replay outside the agent's intended runtime context.
Borui Kang presents Harness Continual Learning (HCL), where an agent's harness (prompts, memories, tools, skills, routing rules) evolves around a frozen foundation model, unlike updating model parameters. The paper defines "harness-level forgetting" as losing reliable behavior due to harness updates and proposes a guarded evolution mechanism. A Continual Optimizer generates candidate harnesses from feedback, and a Continual Evaluator commits changes only after verifying improvement, retention, and validity. Experiments in textual reasoning, multimodal perception, and open-world interaction show capability accumulation and failure recovery, with over 10% relative gains versus baselines.
- Four execution-facing harness components: Task Interface, Experience Memory, Capability Map, and Adaptive Router.
- Controlled retention sweeps show the stability''-plasticity trade-off can be explicitly adjusted at the harness level.
- The work reframes continual learning away from parameter updates toward externalized, inspectable agent state.
- Four execution-facing harness components: Task Interface, Experience Memory, Capability Map, and Adaptive Router.
- Controlled retention sweeps show the stability''-plasticity trade-off can be explicitly adjusted at the harness level.
- The work reframes continual learning away from parameter updates toward externalized, inspectable agent state.
Anurag Singh replaced five Python scripts (backup, organizer, renamer, cleaner, watchdog) with a local LLM agent, which made errors the scripts didn't (wrong directories, skipped steps, false success reports).Each of the original scripts followed explicit rules through a scheduler; the agent instead added a longer inference chain (inspect, interpret, choose a tool, build a command, execute, review) to tasks that fixed logic already described completely, while also holding a loaded model in memory between runs.
- AutomationBench scores for frontier models remain well under 20%: GPT-5.6 Sol 18.1%, GPT-5.5 12.9%, Claude Opus 4.8 15.5%, Gemini 3.5 Flash 14.5%
- Granting an LLM system-level access creates a prompt-injection vector: a malicious file on disk could carry instructions the agent interprets as commands
- Singh's proposed fix: let the agent classify and route ambiguous requests, then hand off to a validator + fixed script for the actual filesystem action
- The five original scripts covered photo backup, extension-based Downloads sorting, file renaming, app-cache clearing, and a disk-threshold alert
- AutomationBench scores for frontier models remain well under 20%: GPT-5.6 Sol 18.1%, GPT-5.5 12.9%, Claude Opus 4.8 15.5%, Gemini 3.5 Flash 14.5%
- Granting an LLM system-level access creates a prompt-injection vector: a malicious file on disk could carry instructions the agent interprets as commands
- Singh's proposed fix: let the agent classify and route ambiguous requests, then hand off to a validator + fixed script for the actual filesystem action
- The five original scripts covered photo backup, extension-based Downloads sorting, file renaming, app-cache clearing, and a disk-threshold alert
Anurag Singh replaced his home lab cron scripts with Qwen3.5 9B using an agent harness with shell access. He expected contextual reasoning to be superior to rigid automation. The local model succeeded in identifying ballooned directories or judging if a container restart was needed, but it failed more often, sometimes stalling or silently skipping checks.He concluded that deterministic scripts remain the more dependable choice for routine tasks and pointed to n8n as a sensible middle ground when the friction is writing and maintaining code rather than the logic itself.
- A 9-billion-parameter local model needs several GB of RAM just to load weights, which is painful on a home server already running Docker, DNS, and other services.
- Singh's specific hardware ceiling: roughly 14B parameters on a 16 GB MacBook, maybe 32B on an M5 Pro, beyond which you need a dedicated rig.
- His suggested hybrid: let the local model read an error log and draft a short explanation, then have n8n relay that summary without granting the model permission to restart or modify anything.
The model's failure mode was not wrong commands but an inconsistent process—the same prompt and the same system state, yet different execution paths on successive runs.
- A 9-billion-parameter local model needs several GB of RAM just to load weights, which is painful on a home server already running Docker, DNS, and other services.
- Singh's specific hardware ceiling: roughly 14B parameters on a 16 GB MacBook, maybe 32B on an M5 Pro, beyond which you need a dedicated rig.
- His suggested hybrid: let the local model read an error log and draft a short explanation, then have n8n relay that summary without granting the model permission to restart or modify anything.
The model's failure mode was not wrong commands but an inconsistent process—the same prompt and the same system state, yet different execution paths on successive runs.
