Benjamin Marie writes that the effectiveness of an LLM in long-horizon agentic coding tasks depends heavily on the harness used to drive it rather than just the model itself. Through testing Qwen3.8 27B across three different interfaces—Mini-SWE Agent, Claude Code, and Pi—the author found that while specific configurations like "benchmaxxed" Pi can solve the highest number of tasks, other setups like Claude Code achieve better functional coverage (F2P). The study highlights how critical engineering choices, such as preserving reasoning traces or managing output token limits, are essential for successful agentic performance.
- The evaluation used DeepSWE 1.1, a benchmark comprising 113 long-horizon tasks from 91 open-source repositories.
- Performance varies significantly based on whether reasoning traces are preserved between turns and how context budgets are managed.
- Pi at medium effort was found to offer the best balance of efficiency and accuracy.
- Results were influenced by factors like session recovery, patch reliability, and output-token settings. author »
- The evaluation used DeepSWE 1.1, a benchmark comprising 113 long-horizon tasks from 91 open-source repositories.
- Performance varies significantly based on whether reasoning traces are preserved between turns and how context budgets are managed.
- Pi at medium effort was found to offer the best balance of efficiency and accuracy.
- Results were influenced by factors like session recovery, patch reliability, and output-token settings. author »
An experiment exploring whether a local large language model can manage a home server without strict guardrails. By using the Pi agent harness and the Qwen3.6-35B-A3B model, the author successfully enabled an LLM to control a Proxmox VE node. The setup demonstrated impressive capabilities in managing LXC containers, checking system metrics, creating snapshots, and provisioning new virtual machines. Despite these successes, the system encountered difficulties with guest terminal command execution, persistent configuration management, and multi-node scaling.
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
