Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
Abstract
Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark Brel to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed foundation model. Task holdout is commonly used to guard against harness overfitting. It varies semantic tasks but leaves the benchmark protocol fixed, so a bad genius Proposer can produce a cheating harness whose improvement over the initial harness on Brel depends on a benchmark-wide shortcut. We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over valid counterfactual benchmarks. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a held-out confirmation set determines whether the counterfactual enters a finite archive. We formalize an ideal shortcut-neutralized benchmark B0 and establish theoretical guarantees linking finite counterfactual archives to B0 and characterizing sequential Challenger search. We evaluate CHASE on Syn-Ledger and OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol transformations.
Related
- ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on
- No Shortcuts to Infinity: Weil-Petersson Distance in Calabi-Yau Complex Structure Moduli Space
- Pauli-Sparse regularised Counterdiabatic Shortcuts for Linear-Ramp QAOA
- The evolution of antigen specific cancer immunotherapy from adoptive cell therapy to personalized cancer vaccines
- UQ-ACE: Stability-Aware Counterfactual Controller Tuning via Bayesian SINDy
BibTeX
@misc{zhu2026genius,
title = {{Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts}},
author = {Guojun Zhu and Xunheng Huang and Peng Yin and Jiahui Xie and Sanguo Zhang and Doudou Zhou},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.18366},
eprint = {2609.18366},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.18366},
}
RIS
TY - UNPB
TI - Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
AU - Guojun Zhu
AU - Xunheng Huang
AU - Peng Yin
AU - Jiahui Xie
AU - Sanguo Zhang
AU - Doudou Zhou
PY - 2026
DA - 2026/09/24
PB - arXiv
DO - 10.48550/arxiv.2609.18366
UR - https://arxiv.org/abs/2609.18366
AB - Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark B_rel to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed foundation model. Task holdout is commonly used to guard against harness overfitting. It varies semantic tasks but leaves the benchmark protocol fixed, so a bad genius Proposer can produce a cheating harness whose improvement over the initial harness on B_rel depends on a benchmark-wide shortcut. We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over valid counterfactual benchmarks. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a held-out confirmation set determines whether the counterfactual enters a finite archive. We formalize an ideal shortcut-neutralized benchmark B_0 and establish theoretical guarantees linking finite counterfactual archives to B_0 and characterizing sequential Challenger search. We evaluate CHASE on Syn-Ledger and OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol transformations.
ER -