Artificial Intelligence · arXiv:2609.18366 v4 · submitted 16 Sep 2026
Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
The paper says
“An optimized agent harness can improve on held-out tasks by relying on formatting conventions that those tasks share with the rest of the benchmark.”
more of the abstract
We introduce Counterfactual Harness Search and Evolution (CHASE) to test whether harness improvements survive changes to the evaluation protocol. A Proposer edits the harness around a fixed foundation model. A Challenger then searches for a protocol change that preserves the tasks but reduces the updated harness's gain over the initial harness. Executable checks test whether the transformation preserves task semantics. If the gain loss is confirmed on held-out tasks, CHASE adds the transformation to an archive that constrains subsequent harness updates. We give conditions under which this finite archive bounds the gain loss on an ideal benchmark with shortcuts removed. We evaluate CHASE on Syn-Ledger, OfficeQA, FinanceBench, and SWE-Bench Pro V2. On Syn-Ledger, removing the known shortcuts yields 38.30% accuracy for CHASE and 18.27% for HarnessEvolve. On the 21 certification tasks from SWE-Bench Pro V2 HARD-51, changing the issue formatting and tool-output field order reduces HarnessEvolve's gain over the initial harness from 8.57 to 3.81 percentage points. The final CHASE harness retains 10.48 points of its 11.43-point gain under the same change.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 17 Sep 2026 · of version 1 of the paper; version 4 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. CHASE makes a real contribution to agent evaluation by tackling benchmark-wide shortcuts, but its core finite-archive guarantee rests on an unverified composition closure assumption that needs attention. This line is wrong
2 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 4 (before Theorem 1)
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
Theorem 1 assumes that Bval is closed under composition (i.e., that composing two valid transformations yields another valid transformation). The validity firewall (Supplementary B.3) checks each proposed transformation individually but does not verify that compositions of archived counterfactuals remain in Bval.
more of objection 1
Since K(A) is defined as the minimum number of archived transformations whose composition produces B0, the induction step in the proof of Theorem 1 uses intermediate benchmarks bj∘⋯∘b1 and requires each to be in Bval. If composition breaks validity, the bounds on ΔBS and G0 are unsupported. This is a load-bearing gap because the main theoretical contribution is precisely the translation from a finite archive to B0-certificates. Please either (a) provide a mechanism to certify composition closure (e.g., a compositional check in the firewall, or a theoretical condition under which composition preserves validity), or (b) relax the theoretical claims to hold only for archives where the required compositions are explicitly verified, and discuss the limitation openly.
The paper says, at Section 4, Theorem 1
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The parameter ρ = supH,Bb,Bb' [Δb'∘b(H;H0) − Δb(H;H0) − Δb'(H;H0)]+ is introduced but no bound or intuition is given for its magnitude in practice. Since ρ appears additively in the bound multiplied by (K(A)−1), even a small ρ can dominate the guarantee when K(A) is moderate. The paper would benefit from a discussion of plausible values of ρ or an argument that ρ is typically small for the kind of transformations considered.
3 minor objections
The paper says, at Section 3.2
Pith says: minor objection 1 of 3
The definition of the validity firewall as a set of automated checks is described only in Supplementary B.3; the main text would benefit from a brief summary of the check categories (e.g., task/answer preservation, resource limits) to give readers a clearer sense of what constitutes a 'valid' counterfactual.
The paper says, at Section 5.3 (Table 1)
Pith says: minor objection 2 of 3
It is notable that HarnessCompass performs below RawHarness on OfficeQA (bRrel 64.04% vs 67.98%). The paper attributes this to a mismatch between the original HarnessCompass setting (SWE-bench) and OfficeQA. It would be helpful to add a sentence or a reference to Supplementary C.4 explaining whether the generalization gate was re-tuned or left at defaults, to clarify whether this comparison is a fair test of the method.
show 1 more
The paper says, at Supplementary A (proof of Theorem 1)
Pith says: minor objection 3 of 3
The induction argument uses the bound Δb'∘b(H;H0) ≤ ΓA(H;H0) + ρ for each step, but the inequality Δb'∘b ≤ Δb + Δb' + ρ is an upper bound; the induction can be tightened by noting that Δb'∘b ≤ min(ΓA, ...) but that is not essential. The proof is clear otherwise.
What would settle it
Take two valid counterfactuals from the paper: Φb1 (table-context relocation in OfficeQA) and Φb2 (a simple document-permutation that passes the firewall). Apply Φb2 after Φb1 to produce the composed benchmark Bb2◦b1. Run all validity firewall checks (Table B.2) on Bb2◦b1. If any check fails, then the composition is not in Bval, and the closure assumption used in Theorem 1 is violated. If it passes, verify that the composition genuinely preserves task semantics by checking that the correct answer and available information remain unchanged. Report whether the composition is valid; if it is not, the K(A) chain for bounding ∆BS is not necessarily valid, weakening Theorem 1.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 17 Sep 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
I finished the Bad Genius paper. Here's my take.
The paper correctly identifies a genuine failing of task holdout—it leaves benchmark-wide protocol shortcuts intact—and introduces a clever counterfactual-search method (CHASE) that proactively discovers them. The theoretical analysis linking finite archives to an ideal neutralized benchmark B0 is new and provides statistical motivation. Experiments on Syn-Ledger and OfficeQA show promise: CHASE retains released gains while reducing gain destruction under valid protocol changes, and it outperforms HarnessCompass on both benchmarks. The experimental design with disjoint task splits is careful, and the paper is refreshingly honest about its limitations.
The softest spot is the composition closure assumption underlying Theorem 1. The stress-test note is correct: Bval is assumed closed under composition, but the validity firewall only checks individual transformations. Compositions of valid counterfactuals may be invalid, breaking the K(A)-based induction that links finite archives to B0. This is a load-bearing assumption that the paper neither verifies nor relaxes. The authors should either provide a mechanism ensuring composition preserves validity (e.g., checks on compositions) or revise the theorem to require B0 directly in the archive. As written, the central guarantee is partially unsupported.
Other concerns: the thresholds (η_conf=0.075, ε=0.05) are set ad hoc without operationalizing Theorem 2’s guidance that γ > r. The experimental anomaly of HarnessCompass below RawHarness on OfficeQA is acknowledged but not resolved; it likely reflects a misimplementation of its gate. CHASE exceeding Oracle-B0 on Syn-Ledger is surprising and could use more explanation, since Oracle-B0 has privileged access to B0. These are minor relative to the composition issue.
Overall, the paper addresses a real problem with a well-motivated method and has clearly been written by serious researchers. The composition assumption is the main barrier to fully trusting the theoretical story.
The paper is for researchers working on agent evaluation, benchmark design, and harness optimization. I would bring it to a reading group and would cite it with caveats.
Recommendation: accept for peer review conditional on addressing the composition closure issue (either verify it or relax the theorems) and clarifying the experimental anomalies.
The strongest objection
The central finite-archive guarantee (Theorem 1) relies on the assumption that Bval is closed under composition (Section 4, before Theorem 1). The validity firewall (Supplementary B.3, Table B.2) checks each proposed transformation individually but does not verify that compositions of archived counterfactuals remain in Bval. Since B0 is unknown and the chain yielding K(A) may involve composing several archived transformations, an invalid composition would mean that the intermediate benchmarks in the chain are not valid, so the induction step linking ∆_(b_j◦···◦b_1) to the archived Γ_A fails. The paper provides no mechanism or check to ensure that compositions preserve task semantics. If a composition is invalid, the derived bound on ∆BS does not apply, and the guarantee that G0 is near G_min,A is unsupported. This is a load-bearing gap because the main theoretical contribution is precisely the translation from a finite archive to B0-certificates.
The reader identified the validity firewall as the weakest assumption, which is correct but broad. I sharpen this to a specific, undefended gap: the closure under composition assumed in Theorem 1. The paper gives no reason to believe that composing individually valid counterfactuals preserves validity, and the firewall does not check compositions. Without closure, the induction from Γ_A to ∆BS collapses because the intermediate composite benchmarks may be invalid. This does not change the verdict because the paper is already conditional; it strengthens the case that the theoretical guarantees rest on unverified structural properties of Bval. The reader's rationale mentioned strong theoretical assumptions not empirically verified; my concern aligns but is more precise about which assumption matters most for Theorem 1.
The referee's report
If the theoretical guarantees hold, CHASE addresses an important and understudied failure mode in agent evaluation: benchmark-wide shortcuts that survive task holdout. The idea of using an online Challenger to generate validity-preserving counterfactuals as constraints during harness evolution is novel and well-motivated. The careful experimental design with disjoint task splits, multiple rollouts, and a synthetic benchmark that provides ground-truth B0 is a strength. The paper provides explicit concentration bounds and a packing argument that gives a certificate for non-confirmed rounds.
Pith's reading of the paper (a machine's summary, not the authors')
Automated harness optimization can produce a 'bad genius' that scores well on a released benchmark by exploiting protocol-wide statistical correlations rather than genuine capability improvement. Task holdout alone cannot catch such benchmark-wide shortcuts because the protocol itself stays fixed. CHASE introduces a Challenger that, after each Proposer update, searches for a valid protocol transformation that destroys the current harness's gain while keeping task semantics unchanged. Confirmed counterfactuals enter an archive that constrains subsequent evolution. The paper provides finite-archive statistical guarantees linking the archive to an ideal shortcut-neutralized benchmark (Theorem 1), bounds the number of confirmable counterfactuals (Theorem 2), and certifies when further search cannot find a large violation (Theorem 3). On OfficeQA and a synthetic benchmark, CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol changes.
What carries the argument. The gain-destruction contrast $\Delta_b(H;H_0)$ and the archive $\mathcal{A}_t$ of confirmed validity-preserving counterfactual benchmarks. The Challenger searches for transformations $\Phi_b$ that maximize $\Delta_b$ while passing a validity firewall; confirmed entries constrain the Proposer's subsequent optimization. The finite-archive guarantee (Theorem 1) relates $\Delta_{BS}(H^\star;H_0)$ on the unknown ideal benchmark $B_0$ to the empirical maximum over the archive, with composition depth $K(\mathcal{A})$ and subadditivity gap $\rho$ controlling the bound.
Core claim. The paper establishes that alternating a Proposer with a Challenger that searches for validity-preserving benchmark counterfactuals can steer harness evolution away from benchmark-wide shortcuts. The central estimand is gain destruction $\Delta_b(H;H_0) = G_{\text{rel}}(H;H_0) - G_b(H;H_0)$, the difference between the harness's gain on the released benchmark and its gain on a counterfactual benchmark $B_b$. A finite archive of confirmed counterfactuals, each with large $\Delta_b$, becomes a set of constraints: the Proposer must maintain gains within a tolerance $\epsilon$ for every $B_b$ in the archive. Theorems 1\u20133 translate a finite archive into guarantees about the ideal neutralized benchmark $B_0$, limit the number of counterfactuals that can be added, and provide a stopping certificate when no large violation can be found.
The premise it rests on. The validity firewall (Supplementary B.3) correctly determines whether a protocol transformation preserves task semantics, which in practice relies on automated checks that may miss semantically significant changes or wrongly reject harmless ones, potentially excluding valid counterfactuals that would expose shortcuts or including invalid ones that break the guarantee.
If the paper is right (Pith's inferences, not the authors')
Evaluation of automatically optimized harnesses should incorporate protocol counterfactuals, not just held-out tasks, to assess whether observed gains are robust to protocol variation.
Harness-evolution methods can adopt an online Challenger to automatically detect and constrain benchmark-wide shortcuts, reducing reliance on fixed hand-coded generalization gates.
The statistical guarantees enable a bounded-budget certification: after a finite number of confirmed counterfactuals, the process must reach a round where no further large shortcut can be found, certifying near-optimal released gain with small gain destruction.
The validity firewall becomes a critical component; its design and verification directly affect the credibility of the guarantee, and future work must provide rigorous methods for checking semantic equivalence of protocol transformations.
The framework generalizes to any automated system optimization where a fixed evaluation protocol is repeatedly used, such as robot policy evaluation with procedural environments.
The Challenger itself is an AI agent that may invent new shortcuts; the resulting Proposer\u2013Challenger dynamic resembles a game, and one should verify that the Challenger does not introduce bias or become exploitable by a sufficiently sophisticated Proposer.
The theoretical bounds depend on composition depth $K(\mathcal{A})$ and subadditivity gap $\rho$, which are defined relative to an unknown $B_0$ and may be hard to bound in practice; a practical certification would need to make worst-case assumptions that weaken the guarantee.
The approach is only as strong as the Challenger\u2019s search breadth: on large, multi-modal benchmarks an exhaustive search over valid protocol transformations is infeasible, so some benchmark-wide shortcuts may remain undiscovered; combining CHASE with human audit of protocol correlations could strengthen the resulting archive.
The weakest link is the validity firewall: if an automated check incorrectly approves a transformation that subtly changes task semantics, the counterfactual no longer reflects protocol variation alone, and the guarantee decomposes; rigorous formal verification of semantics-preserving transformations remains an open problem.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
An optimized agent harness can improve on held-out tasks by relying on formatting conventions that those tasks share with the rest of the benchmark. We introduce Counterfactual Harness Search and Evolution (CHASE) to test whether harness improvements survive changes to the evaluation protocol. A Proposer edits the harness around a fixed foundation model. A Challenger then searches for a protocol change that preserves the tasks but reduces the updated harness's gain over the initial harness. Executable checks test whether the transformation preserves task semantics. If the gain loss is confirmed on held-out tasks, CHASE adds the transformation to an archive that constrains subsequent harness updates. We give conditions under which this finite archive bounds the gain loss on an ideal benchmark with shortcuts removed. We evaluate CHASE on Syn-Ledger, OfficeQA, FinanceBench, and SWE-Bench Pro V2. On Syn-Ledger, removing the known shortcuts yields 38.30% accuracy for CHASE and 18.27% for HarnessEvolve. On the 21 certification tasks from SWE-Bench Pro V2 HARD-51, changing the issue formatting and tool-output field order reduces HarnessEvolve's gain over the initial harness from 8.57 to 3.81 percentage points. The final CHASE harness retains 10.48 points of its 11.43-point gain under the same change.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“We evaluate CHASE on a synthetic benchmark and on OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol changes.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.26 · human participants 0.04 · controlled comparison 0.26 · randomized assignment 0.02 · states quantitative result 0.11 · compares with prior results 0.44 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark $B_{\mathrm{rel}}$ to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed target agent. | background 1.00 | 0.11 | 0.04 | asserted 1.00 |
| Task holdout varies semantic tasks but leaves the benchmark protocol fixed, so a "bad genius" Proposer can produce a cheating harness whose released-benchmark gain depends on a benchmark-wide shortcut. | background 0.78 | 0.13 | 0.07 | asserted 0.67 |
| We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over validity-preserving benchmark counterfactuals. | contribution 0.88 | 0.88 | 0.09 | asserted 1.00 |
| After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. | method 0.99 | 0.77 | 0.05 | asserted 0.99 |
| A validity firewall checks that task semantics are preserved, while a confirmation set determines whether the counterfactual enters a finite archive. | method 1.00 | 0.75 | 0.04 | asserted 0.99 |
| We formalize an exact shortcut-neutralized benchmark $B_0$ and establish statistical guarantees linking finite counterfactual archives to $B_0$ and characterizing sequential Challenger search. | contribution 0.97 | 0.94 | 0.11 | asserted 1.00 |
| We evaluate CHASE on a synthetic benchmark and on OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol changes. | result 1.00 | 0.96 | 0.69 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.78
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,675 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (2)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment