{"id":"b1347039-5a6e-4744-9d6f-1bd169a7e4a7","arxiv_id":"2607.24054","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A CLEAN/GOLD/SHAM value-substitution audit (AcquaBench) shows agent success often tracks a supplied target value, and that simple exposure detectors can miss this dependence.","lead":"This paper introduces a way to test whether an AI agent's correct answers actually depended on target information it acquired during evaluation, rather than on the intended benchmark context. It matters because modern agent benchmarks let models search and retrieve, so a high score can hide that the model was effectively handed the answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SHAM is not a neutral control: in D0, SHAM accuracy is 6–8 pp below CLEAN for all models, so ΔGS conflates correct-value benefit with decoy interference; a neutral-source condition is needed to bound this.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: SHAM is not a neutral control, and the paper acknowledges the unresolved decomposition. The D0 data actually strengthen this concern: SHAM is numerically below CLEAN, so the decoy is not merely an inactive source; it appears to reduce success. This makes ΔGS an upper bound on the pure benefit of the correct value, not a direct estimate of it. The paper's central contribution—the GOLD/SHAM audit protocol—remains valuable, and the editorial framing is appropriately scoped (the authors explicitly disclaim help/harm separation). However, for the central claim that 'success follows the correct value,' the interpretation of ΔGS as evidence of target-value support should be conditional on bounding the negative SHAM effect. The proposed NEUTRAL condition directly measures the interference term and would settle whether the 19–26 pp effect is primarily benefit, harm, or both. Since this is the same condition the reader used to justify CONDITIONAL, and no new concern supersedes it, the verdict should remain CONDITIONAL until the control validity is resolved. The paper's statistical protocol, frozen populations, and honesty about scope are strengths that prevent a stronger negative conclusion.","tokens_in":23011,"tokens_out":4658,"duration_ms":44562,"concrete_test":"Run a NEUTRAL condition on the frozen 135 D0 qids for at least one model (e.g., Llama-8B, retrieval surface): use the same base source and template but fill the value slot with a clearly non-answer placeholder (e.g., 'VALUE NOT GIVEN') that preserves sentence syntax and length. Compute SHAM−NEUTRAL (interference term) and GOLD−NEUTRAL (pure benefit) with the same qid-cluster percentile bootstrap. If SHAM−NEUTRAL is significantly negative, the decoy actively misleads and the reported ΔGS overstates value benefit; if GOLD−NEUTRAL remains positive (e.g., >10 pp), the provenance claim survives. Repeat on all four action surfaces and a second model to check generality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central estimand ΔGS is defined as GOLD minus SHAM, interpreted as 'the score response tracks target correctness beyond matched source exposure.' But SHAM is not a neutral control: in D0, SHAM accuracy is below CLEAN for every model (Llama-8B 45.0% vs 53.0%, Qwen-14B 60.0% vs 67.0%, Qwen-32B 64.3% vs 72.0%, Table 2). Thus ΔSC = E[Y_S − Y_C] is negative (−8.0, −7.0, −7.7 pp), and the decomposition ΔGS = ΔGC − ΔSC (Supplementary B.2) means the observed ΔGS (25.9, 23.9, 19.1 pp) is inflated by whatever harm the decoy value causes. The paper explicitly concedes 'It does not separately identify help from the correct value and interference from the incorrect value' (Supplementary B.2) and only states that SHAM matches source structure and exposure opportunity, not realized attention or action paths (A.1). Without an independent bound on the interference term, a sizable portion of ΔGS could reflect decoy-induced failures rather than support from the correct value. The D2 result, based on 36 qids and 12–25 positives, inherits the same issue. If a neutral-source condition showed SHAM ≈ NEUTRAL, the concern would resolve; if SHAM < NEUTRAL, the headline effect is not a clean measure of target-value support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines 'success provenance' as the missing evaluation object in agent benchmarks: whether an observed success depended on a target value acquired during the evaluated trajectory, rather than on benchmark-authorized information. It introduces AcquaBench, a within-qid matched CLEAN/GOLD/SHAM value-substitution audit on HotpotQA, across four standardized action surfaces (retrieval, tool, memory, subagent), with qid-clustered percentile-bootstrap inference on frozen artifact-eligible populations. In D0 (direct single-source sufficiency), GOLD exceeds SHAM by 19.1–25.9 percentage points across Llama-3.1-8B, Qwen2.5-14B, and Qwen2.5-32B, and a style-controlled validation reproduces the pattern. In D2 (distributed two-source sufficiency), GOLD exceeds SHAM by 11.8 and 14.6 points, while the single-source coloc score no longer transfers as a high-score marker (AUROC 0.376 and 0.142). A within-Qwen model comparison shows a supported 5.0-point CLEAN gap compresses to -0.6 points under raw GOLD, without establishing rank inversion. The paper is carefully scoped with an explicit claim-boundary matrix, and it reports code, frozen configurations, and provenance checks.","tokens_in":23335,"tokens_out":2725,"duration_ms":26795,"significance":"If the central contrast is valid, the paper offers a practical audit for agent evaluation that goes beyond outcome accuracy and exposure detection, and it provides evidence that target-value dependence can persist beyond a single-source detector's observation unit. The design's strengths are substantial: prespecified contrasts, frozen populations, qid-clustered bootstrap, style controls, per-surface descriptive transparency, and a machine-checkable release protocol. The main numerical claims are internally consistent and the paper does not overclaim; for example, it explicitly avoids asserting rank inversion or a scaling law. The contribution would be of clear interest to the agent-benchmarking community, and the open repository strengthens reproducibility.","major_comments":[{"comment":"The central estimand ΔGS is interpreted as showing that success tracks target correctness beyond matched source exposure. But ΔGS = ΔGC − ΔSC, and in D0 ΔSC is negative for every model (Table 2: SHAM−CLEAN is −8.0, −7.0, −7.7 pp). A negative ΔSC means the decoy harms performance relative to CLEAN, so ΔGS conflates benefit from the correct value with interference from the incorrect value. The manuscript concedes this ('It does not separately identify help from the correct value and interference from the incorrect value') and states in A.1 that SHAM does not assert identical realized attention or action paths. Without a neutral-source condition or an independent bound on the interference term, the headline 'success follows the correct value' is not uniquely supported. This is load-bearing because ΔGS is the paper's main value-dependence statistic.","section":"AcquaBench Design / Supplementary B.2"},{"comment":"The D2 claim that behavioral dependence persists after answer sufficiency exceeds a single-source observation unit rests on the premise that 'max single-source construction score ≤ 0.25' ensures no single source answers the question. The gate is applied to construction artifacts using the supplied target value, whereas the runtime detector uses the model's predicted answer and observed sources (B.5). The paper itself notes these are not the same random variable. With only 36 qids and 12–25 positives, I would like explicit evidence—or a released audit—that no individual Gold-exposed source in the actual observed trajectories was answer-sufficient, and that the two sources must be composed. The two mechanism cases help, but they are only two trajectories.","section":"Supplementary A.5 / Table 3"},{"comment":"The D1 boundary results show a large positive GOLD−SHAM for Llama (+15.1 pp) despite a negative GOLD−CLEAN (−2.0 pp), and a near-zero/negative GOLD−SHAM for Qwen (−2.0 pp) in the same construction. The paper treats D1 as a measurement boundary, which is a reasonable reading, but it also illustrates that the value-substitution effect is not invariant to answer form and scoring. This reinforces the first concern: ΔGS can be driven by surface-level interactions rather than target correctness alone. I would ask the authors to state more explicitly how D1 constrains the generalizability of the D0 'success provenance' interpretation to other scoring conventions.","section":"Results: D1 / Table 3"},{"comment":"The model-comparison claim is appropriately qualified ('does not establish rank inversion'), but the practical reading in the abstract—'a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points'—could be strengthened by reporting the joint distribution of the two gaps rather than two marginal intervals. Since the same qid clusters underlie both quantities, a cluster-bootstrap test of whether the CLEAN gap exceeds the GOLD gap would directly address whether the compression is statistically meaningful. This is a suggestion for additional analysis rather than an objection to the current phrasing.","section":"Table 4 / Model-comparison audit"}],"minor_comments":[{"comment":"The D2 row in Table S1 appears to list '50 44 36' in the parent/template/paired columns; the main text says a separately frozen 50-question parent sample with 36 paired qids. Please reconcile the '44' entry or clarify what it denotes.","section":"Table S1"},{"comment":"The term 'coloc' is defined in the body but not in the abstract; readers of the abstract will not know what AUROC 0.376 and 0.142 refer to. Consider a brief gloss.","section":"Abstract / Introduction"},{"comment":"The prompt template shows a user exemplar with '<ACTION>' placeholder; the exact substitution for each action surface (Search, Lookup, ReadMemory, AskSubagent) is described in text but would be easier to verify with explicit serialized examples in the supplement.","section":"Supplementary E.2"},{"comment":"The sentence 'The D0 behavior also has the structural signature predicted by direct acquisition' could be misunderstood as post hoc; it would help to cite the prespecified nature of the coloc direction earlier (it is mentioned in B.5, but the main text could make this explicit).","section":"Results: D0"}],"recommendation":"major_revision","confidential_remarks":"The paper is careful and well-scoped, with strong reproducibility practices. The central issue is the interpretation of GOLD−SHAM as clean evidence of target-value dependence when SHAM is demonstrably harmful relative to CLEAN. I would not reject: the authors have already disclosed the limitation, and a neutral-source condition or a bounding argument would likely resolve it. The D2 premise also deserves a small, concrete validation (e.g., per-trajectory artifact audit). If these are addressed, the paper would be a solid contribution to agent evaluation methodology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this paper builds a controlled, within-question audit asking whether an agent's score follows the correct target value supplied during evaluation. The D0 result is strong — GOLD beats SHAM by 19.1 to 25.9 points across three models, all 12 per-surface estimates positive, and the qid-clustered bootstrap intervals exclude zero. The design is careful, and the paper is unusually honest about its boundaries.\n\nWhat's new is the matched CLEAN/GOLD/SHAM value-substitution protocol with frozen populations, four standardized action surfaces, and joint qid-clustered analysis. It addresses a real blind spot: exposure detection tells you a source was seen, but not whether success depended on the correct value. D2 adds a clever boundary — distribute the answer across two sources, and behavioral dependence persists even though the single-source coloc signature inverts, with AUROC below 0.5. The model-comparison result is a clean illustration of how raw GOLD scoring can compress a supported CLEAN gap.\n\nThe main weakness is the SHAM control, as the stress-test note says. SHAM accuracy is 6–8 points below CLEAN for every model, so the GOLD-minus-SHAM contrast mixes benefit from the correct value with harm from the decoy. The paper explicitly concedes it doesn't separate those, and I don't think the existence claim — that scores track value correctness — requires that separation. A decoy that interferes is itself evidence of value sensitivity. But if the authors want to claim GOLD-minus-SHAM measures \"support\" rather than \"sensitivity,\" they need a neutral no-information control or at least a bounded analysis of decoy interference. That's a revision point, not a fatal flaw.\n\nD2 rests on only 36 qids and 12–25 positives, with wide CIs; the authors scope it as a boundary result, and the AUROC inversion is descriptive. D1 is rightly treated as a measurement boundary, not positive evidence. The whole thing is limited to HotpotQA and one orchestration setup, which they say plainly.\n\nThis paper deserves a serious referee. The methodology is reusable, the write-up is transparent, and the claim-boundary matrix plus reproduction details go well beyond the norm. I'd send it out; the SHAM-control issue is exactly what a good reviewer should push on, either toward a neutral condition or a sharper wording of what the contrast identifies.","headline":"Careful, honestly scoped controlled experiment that gives agent evaluation a reusable test for whether scores depend on evaluation-time target values; the SHAM control isn't neutral, but the main inference survives.","tokens_in":23844,"tokens_out":3411,"would_cite":true,"duration_ms":35034,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A correct-answer score does not say why an agent succeeded: the paper introduces success provenance and audits it by comparing matched CLEAN, GOLD, and SHAM conditions.","keywords":["success provenance","agent evaluation","value substitution","CLEAN-GOLD-SHAM","benchmark auditing","retrieval agents","model comparison","AcquaBench"],"falsifier":"Run the same frozen D0 qids with four conditions: CLEAN, GOLD, SHAM, and NEUTRAL (a controlled source of matched form that provides no usable information about the question). If GOLD minus SHAM is large while GOLD minus NEUTRAL is near zero, the SHAM decoy itself is interfering and the claimed target-value attribution fails. If GOLD minus SHAM is approximately equal to GOLD minus NEUTRAL, then SHAM interference is negligible and the attribution stands.","tokens_in":22818,"feed_emoji":"🎯","tokens_out":4037,"duration_ms":36916,"temperature":0.7,"pith_summary":"The paper argues that outcome accuracy cannot distinguish intended reasoning from answer acquisition during evaluation. It introduces 'success provenance' as a missing evaluation object and tests it with a matched three-condition intervention: CLEAN keeps authorized information, GOLD adds the correct target value, and SHAM adds a matched incorrect value. Across a multi-hop benchmark and three open-weight models, GOLD exceeds SHAM by 19.1 to 25.9 percentage points in the direct-acquisition regime, showing that success follows the correct value beyond source exposure. Under distributed two-source sufficiency, the behavioral contrast persists (11.8 and 14.6 points) even when a single-source lexical detector fails. If right, benchmarks should report whether the evaluated information state supported success, not just whether the answer was correct.","feed_headline":"Agent success follows planted values, audit shows","feed_subtitle":"Matched GOLD vs SHAM contrasts expose score dependence on acquired targets and how exposed scoring can distort model rankings.","key_machinery":"The central object is the matched CLEAN/GOLD/SHAM value-substitution design. CLEAN exposes only benchmark-authorized evidence; GOLD adds a controlled source containing the correct target value; SHAM preserves source structure and exposure opportunity while substituting a deterministic matched incorrect value. The key identity is the paired contrast GOLD minus SHAM: a positive value attributes the score response to target correctness beyond matched source exposure. Supporting machinery includes the premature-sufficiency construction (D0), the distributed two-source composition (D2), and the coloc score, a lexical overlap of question tokens and predicted-answer tokens in a single observed sour","core_discovery":"The central claim is that success is not self-explanatory: once an agent can change its information state, correctness no longer tells whether the score depended on a target value acquired inside the evaluated trajectory. The paper establishes this by holding the question, model, action channel, and scorer fixed and varying only the supplied target value. In the direct single-source regime, GOLD minus CLEAN ranges from 11.3 to 18.0 percentage points, and the matched correct-versus-incorrect contrast GOLD minus SHAM ranges from 19.1 to 25.9 points, with all model-level intervals excluding zero. Under distributed sufficiency, GOLD still exceeds SHAM by 11.8 and 14.6 points, while the single-so","pith_inferences":["Editorial inference: The GOLD-minus-SHAM contrast conflates benefit from the correct value with interference from the incorrect value; a neutral, uninformative source control could bound the interference term and sharpen the attribution.","Editorial inference: If the method generalizes, benchmark designers could adopt a standard 'provenance report' that separates authorized-context performance from acquired-target dependence, changing how agent capabilities are compared.","Editorial inference: The D2 AUROC below 0.5 suggests the coloc score's orientation is regime-dependent; a detector that is agnostic to the information structure or trained on distributed examples might recover visibility where this fixed rule fails.","Editorial inference: The frozen populations are small and drawn from one benchmark family; extending the audit to larger, multi-benchmark sets would test whether the 19-to-26-point margins and the D2 persistence are stable."],"forward_implications":["If success tracks target correctness in this matched sense, outcome-only leaderboards conflate earned success with acquired success and overstate capability.","Behavioral value dependence can persist even when no single source is sufficient, so single-source exposure detectors can miss real acquisition.","Exposed scoring can compress or reverse point gaps between models without establishing rank inversion, so model comparisons should be audited under different information conditions.","Answer form and scorer sensitivity can change measured effects, marking a measurement boundary that should be reported alongside acquisition results.","Benchmark reports should accompany accuracy with a provenance view: the CLEAN condition and the GOLD-minus-SHAM contrast."],"fun_headline_variants":["Success tracks planted values, audit finds","Agent scores rise with exposed correct answers","Audit: success depends on acquired targets","Planted values drive agent success, audit shows","Benchmark success not self-explanatory, audit proves"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The matched SHAM condition must preserve everything about the GOLD intervention except the correctness of the value; if the wrong value distracts, confuses, or changes retrieval behavior in a way a neutral no-information control would not, then GOLD minus SHAM conflates benefit from the correct value with interference from the incorrect one.","fun_headline_variants_meta":{"raw":{"variants":["Success tracks planted values, audit finds","Agent scores rise with exposed correct answers","Audit: success depends on acquired targets","Planted values drive agent success, audit shows","Benchmark success not self-explanatory, audit proves"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1310,"prompt_tokens":790,"completion_tokens":520,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":452}},"tokens_in":534,"tokens_out":520,"duration_ms":6247,"temperature":1.0,"reasoning_tokens":452,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:07:53.082366+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same frozen D0 qids with four conditions: CLEAN, GOLD, SHAM, and NEUTRAL (a controlled source of matched form that provides no usable information about the question). If GOLD minus SHAM is large while GOLD minus NEUTRAL is near zero, the SHAM decoy itself is interfering and the claimed target-value attribution fails. If GOLD minus SHAM is approximately equal to GOLD minus NEUTRAL, then SHAM interference is negligible and the attribution stands.","supporting_citations":[],"review_version":1}