{"id":"a033e5e1-84b7-4eb7-8ad6-76fc497bfa26","arxiv_id":"2609.00549","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The Retrieval-Invoked Actual-Use Effect (RAE) measures the same-task outcome difference between skill-enabled and skill-disabled executions, and reveals that aggregate retrieval lift frequently masks negative actual skill use across 17 LLMs.","lead":"This paper introduces a new evaluation metric, RAE, that measures whether retrieving external skills actually helps or harms an LLM agent's performance on the specific tasks where retrieval occurred. Unlike conventional aggregate metrics, RAE reveals that many models that appear to benefit from skill retrieval system-wide actually perform worse on the tasks where they used a skill, challenging the reliability of current evaluation practices.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Headline sign-disagreement cells rely on aggregate retrieval lift computed over tiny non-retrieved subsets, with no confidence intervals reported for either side.","rationale":"The reader's weakest assumption concerns the paired skill-disabled execution as a counterfactual control. The paper explicitly acknowledges this limitation in Section 6 ('RAE is not an unbiased causal effect over a pre-treatment task population') and bounds it with diagnostic controls. My concern is different and, I think, more directly load-bearing for the headline claim: the aggregate retrieval lift used in the sign-disagreement comparisons has no reported uncertainty, and in the two signature reversal cells its non-retrieved denominator is extremely small. The paper's own Appendix I shows DeepSeek-V3.2 s42 coverage of 96.2%, meaning the aggregate lift contrasts 77 retrieved tasks against 3 non-retrieved tasks. A single extra correct or incorrect answer in that 3-task group changes the lift by about a third of the reported magnitude. The same issue applies, less severely, to Gemini-2.5-Flash-lite. The reporting filter in Appendix A only checks that both denominators are nonzero, which is a weak condition for a headline quantitative paradox. The paper is otherwise careful: it gives exact McNemar p-values and bootstrap CIs for RAE, reports pooled counts, and includes retrieval-depth sensitivity. But none of that addresses the instability of the comparison metric. Since the abstract and introduction make strong claims that 'multiple models that appear to benefit system-wide actually harm their own performance,' and those claims hinge on cells like DeepSeek-V3.2 +31.6/-9.1, the lack of uncertainty quantification for the aggregate-lift side is a concrete, testable gap. I am not claiming the paradox is false; I am claiming the current evidence does not establish it as robust. A CONDITIONAL verdict asking for the aggregate-lift confidence intervals and n_skip sensitivity would settle whether the paradox survives. This is not an attack on RAE's validity; it is a request to make the contrast it is measured against as well-calibrated as the RAE estimates themselves.","tokens_in":22418,"tokens_out":6056,"duration_ms":61012,"concrete_test":"Recompute every reportable cell in Tables 3, 6, and 7 with (i) 20,000-resample bootstrap confidence intervals for both RAE and aggregate retrieval lift, and (ii) a reporting filter requiring n_skip >= 10, plus a sensitivity table for n_skip >= 5/10/20. Then re-count sign disagreements using only cells where both metrics are defined, both denominators meet the filter, and both 95% intervals exclude zero. If DeepSeek-V3.2 s42 or Gemini-2.5-Flash-lite s42 aggregate-lift intervals include zero, or if the sign-disagreement fraction drops materially below 5/13, the 'pervasive paradox' conclusion should be weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central paradox claim compares RAE against aggregate retrieval lift, but the aggregate lift in the two headline reversal cells is computed over very small non-retrieved task subsets. On MBPP+ seed 42, DeepSeek-V3.2 has 96.2% retrieval coverage (ncond=77 of 80, Appendix I, Table 19), so the non-retrieved denominator is only 3 tasks; moving one task flips aggregate lift by roughly 33 pp. Gemini-2.5-Flash-lite has roughly 80% coverage, leaving about 16 non-retrieved tasks. The reporting filter in Appendix A requires only that both denominators be nonzero; there is no minimum n_skip and no confidence interval for aggregate retrieval lift. Consequently, the reported '+31.6 pp' and '+6.2 pp' may be statistically indistinguishable from zero, and the 5/13 sign-disagreement count may partly reflect noise in a nearly empty denominator rather than a systematic evaluation paradox. The paper reports exact McNemar tests and bootstrap intervals for RAE, but not for the aggregate lift it is contrasted against. This asymmetry is the weakest load-bearing point: the RAE construction itself is sound, but the headline 'pervasive paradox' depends on the reliability of the comparison metric on the other side of the sign disagreement.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces the Retrieval-Invoked Actual-Use Effect (RAE), a same-task paired metric that measures whether an LLM agent's retrieval and use of external skills actually improves its performance on the exact tasks where retrieval occurred. Using an execution design that compares skill-enabled and skill-disabled runs across 17 LLMs on MBPP+, HumanEval+, and Math500, the authors document frequent sign disagreement: positive aggregate retrieval lift but negative RAE. They attribute this to task-selection bias in aggregate metrics and diagnose mechanistic failures (e.g., ignored skills, format errors) through annotation and content-control experiments.","tokens_in":22610,"tokens_out":2039,"duration_ms":18802,"significance":"If the empirical claims hold, RAE provides a principled correction to a blind spot in agent evaluation. The paired-design methodology is technically sound and addresses a genuine confound in retrieved-vs-skipped comparisons. The finding that several models show negative RAE despite positive aggregate lift—across multiple benchmarks and seeds—would be practically important for auditing tool-augmented systems. The diagnostic analyses (skill-content controls, adherence annotation) add mechanistic depth. However, the headline paradox claim depends on the reliability of the aggregate retrieval lift estimates it is contrasted against, and this is where the paper's reporting is incomplete.","major_comments":[{"comment":"The central claim of pervasive sign disagreement relies on comparing RAE against aggregate retrieval lift, yet the aggregate lift values for several headline reversal cells are computed over extremely small non-retrieved subsets (n_skip). For example, DeepSeek-V3.2 on MBPP+ seed 42 has n_skip = 3 (Appendix I, Table 19), and Gemini-2.5-Flash-lite has n_skip ≈ 16. The paper reports no confidence intervals or bootstrap variability for aggregate retrieval lift, and does not impose a minimum n_skip threshold for sign-disagreement counting. As a result, it is unclear whether the '+31.6 pp' and '+6.2 pp' values are statistically distinguishable from zero, and whether the 5/13 sign-disagreement count reflects a systematic paradox or noise from tiny denominators. The authors should (a) report n_skip for every cell used in sign-disagreement counts, (b) compute and report 95% confidence intervals for aggregate retrieval lift (e.g., via bootstrap or exact binomial methods), and (c) show how the sign-disagreement count changes when cells with n_skip below a threshold (e.g., 10 or 15) are excluded. Without this, the paper's strongest claim is insufficiently supported.","section":"§5.1, Table 3, Appendix A"}],"minor_comments":[{"comment":"Several cells report aggregate lift or RAE as '+0.0' or '-0.0'. This suggests rounding; please report the precise values or indicate that the metric is exactly zero (e.g., when b = c in RAE).","section":"Table 3 and Appendix B"},{"comment":"Aggregate retrieval lift is defined as a difference of proportions. For cells with very small N_skip or N_ret, the denominator should be clearly stated. Consider reporting the raw numbers (correct_ret, N_ret, correct_skip, N_skip) in a supplementary table for all reportable cells.","section":"Appendix A, Equation (15)"},{"comment":"The harmful-transition annotation uses GPT-5.5, which is not a publicly documented model at the time of review. The human audit (Appendix G.1) shows moderate agreement (α=0.377). The authors should explicitly note that the annotation is a post-hoc diagnostic and that the fine-grained label taxonomy is interpretive. The current language is sufficiently cautious; no action required.","section":"§5.5, Table 12"},{"comment":"The paper repeatedly uses 'pervasive paradox' (Abstract, §1). Given the concern about small-denominator aggregate lift, consider tempering this language to 'frequent sign disagreement' until the robustness of the aggregate side is established.","section":"General"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The core idea here is solid and overdue. RAE—conditioning the same-task comparison on the actual retrieval-invoked subset—is a clear methodological improvement over aggregate retrieval lift. The paper does a thorough job across 17 models, multiple benchmarks, partitioned seeds, and diagnostic controls. The empirical finding that positive aggregate lift and negative RAE co-occur in several models is genuinely interesting and should make people think twice about how they report tool-use benefits.\n\nThat said, the stress-test concern about the small non-retrieved denominator in the headline reversal cells is real. For DeepSeek-V3.2 on MBPP+ seed 42, aggregate lift is computed over only 3 non-retrieved tasks; moving one task flips the lift by 33 pp. The paper does not report confidence intervals for aggregate lift, so we have no idea whether +31.6 pp is statistically distinguishable from zero or from the negative RAE. This doesn't invalidate the RAE construction, but it does weaken the 'pervasive paradox' narrative—several of the sign-disagreement cells may be noise rather than a systematic effect. The authors do show that the count of sign-disagreement cells is stable under alternative ncond thresholds (Appendix A, Table 5), but that doesn't address the n_skip issue. They should add bootstrap intervals for aggregate lift and explicitly discuss the instability arising from near-empty denominator sets.\n\nThe paper acknowledges its own limitations clearly (Section 6, Limitations), which is good. The fixed BM25 retriever and fixed skill libraries also mean the results are not immediately generalizable. But as a methodological critique, the paper stands up. I would accept it with a request to address the aggregate lift uncertainty and to tone down the 'pervasive' language where it rests on fragile cells.\n\nFor peer review: yes, this deserves a serious referee. The contribution is real, the experiments are carefully done, and the main weakness is fixable. I would bring it to a reading group.","headline":"The paper convincingly shows that aggregate retrieval lift can mislead, and RAE is a useful corrective, but the headline reversal cells rely on very small non-retrieved subsets, which weakens the strength of the central paradox claim.","tokens_in":23092,"tokens_out":1131,"duration_ms":11299},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper claims that aggregate retrieval lift gives a misleading picture of tool use: across 17 LLMs, models frequently show positive benchmark-level gains while scoring worse on the very tasks where they retrieved and injected a skill…","keywords":["Skill Following","Retrieval-Invoked Actual-Use Effect","RAE","paired evaluation design","selection bias","tool-use evaluation","retrieval-augmented LLM agents","evaluation metrics"],"falsifier":"Run the identical paired protocol with retrieval forced on every task, so $S_{\\mathrm{call}} = S_{\\mathrm{all}}$, for DeepSeek-V3.2 and Gemini-2.5-Flash-lite: if RAE turns positive when the model cannot choose when to retrieve, the harm is a byproduct of the model's retrieval policy rather than of skill integration, while a persistent negative RAE under forced retrieval confirms the integration chain as the bottleneck. A second check is to swap the fixed BM25 retriever for an oracle that always returns a genuinely matching skill: if the sign flips, the retriever's content selection, rather than the model's ability to follow skills, is the decisive factor.","tokens_in":24028,"feed_emoji":"📉","tokens_out":11504,"duration_ms":95065,"temperature":0.7,"pith_summary":"The paper claims that standard evaluations of skill-using LLM agents answer the wrong question: they compare the tasks where a model chose to retrieve against the tasks where it did not, and those two sets differ in difficulty, so the comparison cannot say whether the retrieved skill actually helped. It formalizes Skill Following as the whole chain from invoking retrieval to integrating returned content into a correct answer, and proposes the Retrieval-Invoked Actual-Use Effect (RAE): the pass/fail difference between matched skill-enabled and skill-disabled runs of the same task, restricted to tasks where retrieval actually returned content. On 17 LLMs across MBPP+, HumanEval+, and Math500, the two views frequently disagree in sign — Gemini-2.5-Flash-lite and DeepSeek-V3.2, for example, show positive aggregate retrieval lift but negative RAE on MBPP+, and Llama-3.3-70B shows +14.2 pp aggregate lift with a $-39.4$ pp RAE on Math500. If the claim holds, aggregate retrieval lift is not evidence of tool-use proficiency, and evaluations should report same-task paired effects.","feed_headline":"Aggregate retrieval gains hide real per-task harm in LLM agents","feed_subtitle":"The new paired metric RAE shows models often do worse on the tasks where they actually used a retrieved skill.","key_machinery":"The load-bearing object is the paired execution design behind RAE. Every task is run twice: a skill-enabled run has a search_skills tool and a fixed procedural skill library in its environment, while a skill-disabled run uses an identical prompt with the tool definition removed, sharing the same task text, generation seed, and decoding settings. Each task's two outcomes place it in one of four transition cells — concordant pass, helpful flip, harmful flip, concordant fail — and RAE is $(b-c)/n_{\\mathrm{cond}}$ on the retrieval-invoked subset, OAE is $(b-c)/n_{\\mathrm{pairs}}$ on all tasks, and the identity $\\mathrm{OAE} = (n_{\\mathrm{cond}}/n_{\\mathrm{pairs}})\\mathrm{RAE} + (n_{\\mathrm{skip}}/n_{\\mathrm{pairs}})\\Delta_{\\mathrm{skip}}$ connects them. The skill-disabled selection gap $\\Delta_{\\mathrm{select}}$ then measures the baseline difficulty difference between tasks where retrieval fired and tasks where it did not, which is exactly the confound that aggregate retrieval lift overlooks.","core_discovery":"On the paper's own terms, the central discovery is an evaluation paradox: an agent can look more capable with retrieval enabled while the retrieval actually damages the very tasks it is applied to. RAE is computed as the difference between helpful flips (skill-disabled run fails, skill-enabled run passes) and harmful flips (skill-disabled run passes, skill-enabled run fails), divided by the number of retrieval-invoked tasks, with OAE being the same paired quantity over the whole benchmark and the decomposition $\\mathrm{OAE} = \\frac{n_{\\mathrm{cond}}}{n_{\\mathrm{pairs}}}\\mathrm{RAE} + \\frac{n_{\\mathrm{skip}}}{n_{\\mathrm{pairs}}}\\Delta_{\\mathrm{skip}}$ showing how the two relate. Applying this protocol to 17 LLMs with fixed Markdown skill libraries and BM25 retrieval, the paper reports sign disagreement between aggregate retrieval lift and RAE in 5 of 13 reportable cells on MBPP+ seed 42 and in 3 of 13 on HumanEval+, with reversal patterns also on Math500. The paper traces the disagreement to retrieval-induced selection bias, quantified by the skill-disabled selection gap $\\Delta_{\\mathrm{select}}$, and to mechanistic failures — most harmful transitions are cases where the model retrieved a skill and then ignored it, solved independently, or leaked tool traces into the output. Skill-content controls keep RAE negative for the two anchor models even when retrieved text is random, filler, or corrupted, which the paper reads as evidence that the retrieval-to-answer chain itself, not the quality of the skill text, is the bottleneck.","pith_inferences":["Editorial: the pair-and-condition protocol transfers to other tool types — code execution, web search, memory — wherever a parallel no-tool run is feasible; nothing in RAE requires skills to be Markdown documents.","Editorial: splitting RAE into an oracle-retrieval variant (retrieval forced on every task) and the free-choice variant would separate 'knowing when to fetch' from 'knowing how to use,' which the paper itself lists as the ablation needed for finer causal decomposition.","Editorial: given the dominant ignored/independent failure mode, a testable intervention is to require the model to explicitly transform or cite the retrieved skill before answering; if negative RAE softens, the bottleneck is attention to injected context rather than content relevance."],"forward_implications":["Benchmark reports should carry same-task paired metrics like RAE alongside aggregate lift, since the two frequently disagree in sign and aggregate lift alone can endorse a deployment that actively hurts the tasks using the tool.","High retrieval coverage should not be marketed as tool proficiency: the paper shows coverage and RAE decouple, with several high-coverage models showing negative actual-use effects.","Aggregate gains driven by selective retrieval on easy tasks will not survive regimes where retrieval is mandatory or difficulty-matched, so system comparisons based on aggregate lift need revalidation on paired outcomes.","Because RAE stays negative under random, filler, and corrupted skill content for the anchor models, improving the skill library's content alone will not rescue the pipeline; the integration and adherence stage is where the loss is incurred."],"supporting_citations":[{"why":"Supplies MBPP+ and the EvalPlus extended test suite used to score coding-task correctness.","marker":"Liu et al., 2023"},{"why":"Supplies HumanEval+ as the secondary coding benchmark for cross-benchmark consistency checks.","marker":"Chen et al., 2021"},{"why":"Supplies the MATH dataset from which the Math500 cross-domain task set is drawn.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the Math500 partition used for the mathematical-reasoning replication.","marker":"Lightman et al., 2024"},{"why":"Supplies BM25, the fixed retrieval mechanism that decides which skills enter the agent context.","marker":"Robertson and Zaragoza, 2009"},{"why":"SkillsBench, the paired skill-enabled benchmark whose aggregate reporting RAE contrasts with.","marker":"Li et al., 2026"},{"why":"SWE-Skills-Bench, a second paired skill baseline used to position RAE's conditioning on actual retrieval.","marker":"Han et al., 2026"}],"fun_headline_variants":["New metric RAE reveals LLM agents often worse when using retrieved skills","Aggregate retrieval lift masks per-task damage in LLM agents","Evaluation paradox: retrieval helps on average but hurts per-task","Retrieval boosts aggregate scores but harms individual tasks in LLMs","RAE exposes the hidden cost of skill retrieval in LLM agents"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paired comparison assumes that the skill-disabled run is a clean counterfactual for the same task without retrieval — that removing the tool definition from the prompt changes nothing else about how the model behaves, even though the absence of the tool could subtly alter task interpretation or confidence, and even though the tasks being measured are exactly the ones the model itself chose to retrieve on.","fun_headline_variants_meta":{"raw":{"variants":["New metric RAE reveals LLM agents often worse when using retrieved skills","Aggregate retrieval lift masks per-task damage in LLM agents","Evaluation paradox: retrieval helps on average but hurts per-task","Retrieval boosts aggregate scores but harms individual tasks in LLMs","RAE exposes the hidden cost of skill retrieval in LLM agents"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"cost":0.000210168,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000210168,"upstream_inference_prompt_cost":0.000118888,"upstream_inference_completions_cost":9.128e-05},"total_tokens":1380,"prompt_tokens":1054,"completion_tokens":326,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":236}},"tokens_in":1054,"tokens_out":326,"cost_source":"openrouter_usage","duration_ms":4045,"temperature":1.0,"reasoning_tokens":236,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T00:47:54.350730+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the identical paired protocol with retrieval forced on every task, so $S_{\\mathrm{call}} = S_{\\mathrm{all}}$, for DeepSeek-V3.2 and Gemini-2.5-Flash-lite: if RAE turns positive when the model cannot choose when to retrieve, the harm is a byproduct of the model's retrieval policy rather than of skill integration, while a persistent negative RAE under forced retrieval confirms the integration chain as the bottleneck. A second check is to swap the fixed BM25 retriever for an oracle that always returns a genuinely matching skill: if the sign flips, the retriever's content selection, rather than the model's ability to follow skills, is the decisive factor.","supporting_citations":[{"cited_title":"Is Your Code Generated by Chat","cited_arxiv_id":null,"evidence_quote":"Supplies MBPP+ and the EvalPlus extended test suite used to score coding-task correctness."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}