Yuhao Wu writes about HarnessDev, a benchmark that evaluates LLMs' ability to build and iteratively improve their own agent harness—the model-external execution infrastructure that wraps a model and shapes its task performance. The benchmark has two stages: Creation, where the agent builds a complete execution system from a minimal seed and a few cases, and Evolution, where it revises its own harness using downstream execution feedback. Generated harnesses substantially lag behind mature human-engineered references on code and search/research, while matching or exceeding them on writing and machine-learning experimentation, with large variation in execution cost.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
Liyan Tang and colleagues write about WikiSkill, a framework that co-evolves reusable agent skills with a persistent knowledge base to systematically accumulate LLM agent experience. The system separates raw execution experience, accumulated knowledge, and executable skills, continuously consolidating experience into the wiki so subsequent skill updates can build on prior learning.
- Larger models benefit more from evolved skills, while smaller models equipped with skills can outperform substantially larger ones without them
- Skills evolved by one model can outperform self-evolved skills in another, enabling cross-model transfer
- No official code exists; a community member implemented it faithfully, catching two harmful skills via the validation gate and documenting a negative run where the proposer honestly produced no useful skills
- Ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution
- Larger models benefit more from evolved skills, while smaller models equipped with skills can outperform substantially larger ones without them
- Skills evolved by one model can outperform self-evolved skills in another, enabling cross-model transfer
- No official code exists; a community member implemented it faithfully, catching two harmful skills via the validation gate and documenting a negative run where the proposer honestly produced no useful skills
- Ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution
This project, `autoresearch-opencode`, is an autonomous experiment loop designed for use with OpenCode. It's a port of `pi-autoresearch`, but implemented as a pure skill, eliminating the need for an MCP server and relying solely on instructions the agent follows using its built-in tools. The skill allows users to automate optimization tasks, as demonstrated by the example of optimizing the BogoSort algorithm which achieved a 7,802x speedup by leveraging Python's `bisect` module for sorted-state detection.
The system maintains state using a JSONL file, enabling resume/pause functionality and detailed experiment tracking. It provides a dashboard for monitoring progress and ensures data integrity through atomic writes and validation checks.
The system maintains state using a JSONL file, enabling resume/pause functionality and detailed experiment tracking. It provides a dashboard for monitoring progress and ensures data integrity through atomic writes and validation checks.
In this episode of Rebecca State Of Mind, Rebecca Tsao explores the differences between overachieving and high achieving. She discusses these concepts with Farah, a fellow dharma coach and Harvard graduate, who shares insights into how these identities have shaped her life. The conversation delves into the transition from being an overachiever/high achiever to embracing uncertainty and discovering new, magical experiences.
