This paper introduces Meta-Harness, an innovative outer-loop system designed to automate the optimization of model harnesses for large language model (LLM) applications. While traditional harnesses are largely designed by hand, Meta-Harness employs an agentic proposer that searches over harness code by accessing source code, scores, and execution traces. The researchers demonstrate significant performance gains across multiple domains: improving text classification efficiency, enhancing accuracy in retrieval-augmented math reasoning for IMO-level problems, and surpassing hand-engineered baselines in agentic coding tasks. The results suggest that providing automated systems with richer access to prior experience can successfully enable the automated engineering of complex LLM harnesses.
