Yuhao Wu writes about HarnessDev, a benchmark that evaluates LLMs' ability to build and iteratively improve their own agent harness—the model-external execution infrastructure that wraps a model and shapes its task performance. The benchmark has two stages: Creation, where the agent builds a complete execution system from a minimal seed and a few cases, and Evolution, where it revises its own harness using downstream execution feedback. Generated harnesses substantially lag behind mature human-engineered references on code and search/research, while matching or exceeding them on writing and machine-learning experimentation, with large variation in execution cost.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
Sara A. Metwalli writes that coding agents differ from basic autocomplete or assistants because they can interact with their environment by examining repositories, modifying files, and running tests in a loop. To use them effectively, developers should provide clear constraints and context rather than just long prompts, following a workflow of inspection, planning, implementation, testing, and review. Instead of delegating massive tasks like rewriting entire applications, users should break work into small, testable problems to maintain control over the architectural integrity and quality of the generated code.
- Agents are best suited for exploratory or multi-step tasks such as finding/fixing bugs or investigating failing tests.
- A recommended prompt structure includes a Goal, Context, Constraints, Acceptance Criteria, and Validation steps.
- The developer's role shifts from writing raw code to directing, assessing, and coordinating complex code-generating systems.
A successful agentic request should include five specific elements rather than just long prompts:
```text
Goal: What you want to achieve »
Context: Specific files or architecture to inspect »
Constraints: Rules/boundaries the agent must follow »
Acceptance criteria: The definition of a "finished" task, e.g., passing tests »
Validation: How to verify success, e.g., run specific test commands »
```
- Agents are best suited for exploratory or multi-step tasks such as finding/fixing bugs or investigating failing tests.
- A recommended prompt structure includes a Goal, Context, Constraints, Acceptance Criteria, and Validation steps.
- The developer's role shifts from writing raw code to directing, assessing, and coordinating complex code-generating systems.
A successful agentic request should include five specific elements rather than just long prompts:
```text
Goal: What you want to achieve »
Context: Specific files or architecture to inspect »
Constraints: Rules/boundaries the agent must follow »
Acceptance criteria: The definition of a "finished" task, e.g., passing tests »
Validation: How to verify success, e.g., run specific test commands »
```
This study provides a comprehensive architectural analysis of Claude Code, an agentic coding tool capable of executing shell commands, editing files, and interacting with external services. By examining the TypeScript source code and comparing it to the open-source OpenClaw system, the researchers identify how different deployment contexts influence design choices regarding safety, execution, and capability management.
Key topics include:
- Analysis of five core human values driving agent architecture: decision authority, safety, reliable execution, capability amplification, and contextual adaptability.
- Breakdown of technical components such as permission systems with ML-based classification, context management pipelines, and extensibility mechanisms like MCP and plugins.
- Comparative study between CLI-based agents and gateway-level personal assistant architectures.
- Identification of six future design directions for the evolution of AI agent systems.
Key topics include:
- Analysis of five core human values driving agent architecture: decision authority, safety, reliable execution, capability amplification, and contextual adaptability.
- Breakdown of technical components such as permission systems with ML-based classification, context management pipelines, and extensibility mechanisms like MCP and plugins.
- Comparative study between CLI-based agents and gateway-level personal assistant architectures.
- Identification of six future design directions for the evolution of AI agent systems.
As AI agents evolve from autocomplete tools to active contributors (opening PRs, managing infrastructure), DevOps must adapt. This playbook outlines the shift through these key strategic pillars:
* **Foundational Prerequisites:** Robust CI/CD, automated testing, and Infrastructure as Code are essential for agentic workflows.
* **Evolving Engineering Roles:** Engineers transition from code producers to system designers, agent operators, and quality stewards.
* **Structured Collaboration:** Integration across IDEs, PRs, pipelines, and production environments is required.
* **Repository Design:** Repositories must act as explicit interfaces using skill profiles and instruction files.
* **Development Methodology:** Shift from ephemeral prompt engineering to durable, specification-driven development.
* **Governance & Security:** Implement frameworks for custom agent consistency/auditability and transform CI/CD into active verifiers of semantic intent and security.
* **New Success Metrics:** Move from volume-based productivity counts to outcome-based and trust-boundary measurements.
* **Foundational Prerequisites:** Robust CI/CD, automated testing, and Infrastructure as Code are essential for agentic workflows.
* **Evolving Engineering Roles:** Engineers transition from code producers to system designers, agent operators, and quality stewards.
* **Structured Collaboration:** Integration across IDEs, PRs, pipelines, and production environments is required.
* **Repository Design:** Repositories must act as explicit interfaces using skill profiles and instruction files.
* **Development Methodology:** Shift from ephemeral prompt engineering to durable, specification-driven development.
* **Governance & Security:** Implement frameworks for custom agent consistency/auditability and transform CI/CD into active verifiers of semantic intent and security.
* **New Success Metrics:** Move from volume-based productivity counts to outcome-based and trust-boundary measurements.
The author distinguishes between vibe coding, a reckless approach where developers prompt and accept AI output without review, and agentic engineering, a disciplined professional workflow. While vibe coding is useful for rapid prototyping and MVPs, it lacks the rigor required for scalable or secure systems. Agentic engineering involves orchestrating AI agents under strict human oversight, treating them as fast but unreliable junior developers who require architectural direction and relentless testing.
Key points:
- Distinction between vibe coding (prototyping) and agentic engineering (professional discipline).
- The importance of design docs, rigorous code reviews, and comprehensive test suites in AI workflows.
- How AI-assisted development rewards strong engineering fundamentals rather than replacing them.
- The risk of skill atrophy among junior developers who rely on prompting without understanding underlying principles.
Key points:
- Distinction between vibe coding (prototyping) and agentic engineering (professional discipline).
- The importance of design docs, rigorous code reviews, and comprehensive test suites in AI workflows.
- How AI-assisted development rewards strong engineering fundamentals rather than replacing them.
- The risk of skill atrophy among junior developers who rely on prompting without understanding underlying principles.
A single CLAUDE.md file to improve Claude Code behavior, derived from Andrej Karpathy's observations on LLM coding pitfalls.
This article explores the "Ralph" technique, a method for using Large Language Models (LLMs) to automate software engineering through continuous, autonomous loops. Rather than seeking a perfect prompt, the author advocates for a "monolithic" approach where a single process performs one task per loop, guided by strict specifications and technical standard libraries. The author demonstrates this by using the technique to build "CURSED," a brand-new programming language, even in the absence of training data for that specific language. By managing context windows through subagents and implementing robust backpressure via testing and static analysis, the "Ralph" technique aims to significantly automate greenfield software development projects.
This article by Sebastian Raschka explores the fundamental architecture of coding agents and agent harnesses. Rather than focusing solely on the raw capabilities of Large Language Models, the author delves into the surrounding software layers—the "harness"—that enable effective software engineering tasks. The piece identifies six critical components: providing live repository context, optimizing prompt shapes for cache reuse, implementing structured tool access, managing context bloat through clipping and summarization, maintaining structured session memory, and utilizing bounded subagents for task delegation. By examining these building blocks, the article illustrates how a well-designed system can significantly enhance the practical utility of both standard and reasoning models in complex coding environments.
CAID is a new multi-agent framework for software engineering tasks. It improves accuracy and speed by using a central planner, isolated workspaces for concurrent work, and test-based verification—inspired by human developer collaboration with tools like Git. Evaluations show CAID significantly outperforms single-agent approaches.
A-Evolve, a new framework developed by Amazon researchers, aims to revolutionize the development of agentic AI systems. It addresses the current bottleneck of manual tuning by introducing an automated evolution process. Described as a potential "PyTorch moment" for agentic AI, A-Evolve moves away from hand-tuned prompts towards a scalable system where agents improve their code and logic iteratively.
The framework centers around an ‘Agent Workspace’ with components like manifest files, prompts, skills, tools, and memory. A five-stage loop—Solve, Observe, Evolve, Gate, and Reload—ensures stable improvements. A-Evolve is modular, allowing for "Bring Your Own" approaches to agents, environments, and algorithms, and has demonstrated State-of-the-Art performance on benchmarks like MCP-Atlas and SWE-bench Verified.
The framework centers around an ‘Agent Workspace’ with components like manifest files, prompts, skills, tools, and memory. A five-stage loop—Solve, Observe, Evolve, Gate, and Reload—ensures stable improvements. A-Evolve is modular, allowing for "Bring Your Own" approaches to agents, environments, and algorithms, and has demonstrated State-of-the-Art performance on benchmarks like MCP-Atlas and SWE-bench Verified.
