{"id":"45c13e06-0baa-430c-90ac-16d8bc9c842e","arxiv_id":"2608.05886","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An RL-trained 14B file-retrieval agent injected into a frozen coding agent reduces resolved-instance token use by about 19 percent on SWE-Bench Verified with no measured resolve-rate loss.","lead":"CodeGrep is a 14B retrieval agent trained with reinforcement learning to find files for coding agents, cutting rounds by 15 percent and tokens by 19 percent on resolved SWE-Bench issues when injected into a frozen OpenHands agent. The paper also maps a precision threshold that separates retrievers which hurt a coding agent from those which help it, a rule of thumb for anyone building retrieval-augmented agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +1.2pp resolve lift and 15%/19% efficiency gains are selected from three iterations run on the same 500 eval instances; without a held-out split or significance testing, the headline numbers are not unbiased estimates and the central efficiency claim is unverified.","rationale":"Read in good faith, the paper is a transparently written systems contribution. The paired-instance analysis in §5.3 gives credible evidence of per-issue acceleration on the 96 co-resolved instances, the reward-design study is internally consistent, and the worktree-based RL sandbox is a genuine engineering contribution. The concern here is not that the result is false, but that the reported magnitudes are not yet evidenced at the claimed specificity. The reader identified CATM behavioral relevance as the weakest assumption, which is a real threat to the internal retrieval-precision numbers. However, the downstream injection evaluation is the external check that could rescue the central claim even if CATM labels are noisy. The more load-bearing threat is that the downstream evaluation itself was used to select v3 among three trained variants, so the headline numbers are selected maxima with no significance testing. The reader lists this as a mechanical problem in the rationale, but it is not the stated weakest assumption, hence partial agreement. Because the paper already reports a paired-instance analysis that survives part of the worry, the appropriate verdict remains conditional: the efficiency direction is credible, but a held-out evaluation and proper confidence intervals are required before the specific +1.2pp and 15%/19% figures should be accepted as stable estimates.","tokens_in":16439,"tokens_out":5186,"duration_ms":57260,"concrete_test":"Hold out a random 100-instance split of SWE-Bench Verified that plays no role in further model selection (or, if no split remains, run on a fresh external benchmark such as SWE-Bench Lite). Pre-register the frozen OpenHands baseline versus CodeGrep v3, also running v1 and v2 for completeness. Report exact binomial 95% confidence intervals for the resolve-rate difference and paired bootstrap confidence intervals for rounds and tokens on co-resolved instances. If the resolve-rate CI contains zero and the efficiency CIs include no improvement, the +1.2pp and 15%/19% claims should be downgraded; otherwise the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is the evaluation protocol for the headline claim. CodeGrep v3 was chosen after v1, v2, and v3 had all been evaluated on the same 500 SWE-Bench Verified instances (§3.5–§3.6, §4.3), with early stopping at step 897 and the line-range reward removed in v3. No held-out final evaluation is reported. Reporting the best of three configurations on the same data makes the +1.2pp (6/500 instances) and −15%/−19% values selected maxima rather than unbiased point estimates. At n=500, the standard error of the baseline resolve rate is roughly 1.9pp, so +1.2pp is well within one standard error. The internal retrieval results in Table 2 do not break this loop: they are scored against CATM labels, which are also the reward target the policy was trained on, produced by the same GLM judge and w≥0.15 filter (§3.4). The paired-instance analysis in §5.3 is a meaningful correction, but it covers only 96 co-resolved instances from the same selected run, and no confidence intervals are reported anywhere. The case study is illustrative but single-instance. Thus the central claim needs an independent evaluation before the reported magnitudes can be trusted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CodeGrep, a 14B retrieval agent trained with GRPO to perform multi-turn grep/glob/read tool calls and return candidate files to a frozen OpenHands downstream coding agent. On SWE-Bench Verified (500 instances), the authors report that CodeGrep v3 lifts the resolve rate by +1.2pp (25.8% to 27.0%) and reduces rounds by 15% and tokens by 19% on resolved instances. They also characterize a precision threshold governing downstream utility: BM25 (precision 0.375) hurts, Jina (0.445) is neutral, and CodeGrep (0.677) buys efficiency. The method section describes CATM, a trajectory-mining pipeline that produces relevance labels from 67K open-source agent trajectories, and a Git-worktree RL environment. The paper includes a reward-design study comparing reward-layer vs. advantage-layer efficiency scaling across three training iterations (v1, v2, v3).","tokens_in":16520,"tokens_out":4830,"duration_ms":50215,"significance":"If the headline efficiency result survives independent evaluation, CodeGrep would be a useful open contribution: the worktree-based RL sandbox, the CATM label-mining pipeline, and the advantage-layer efficiency-signal study are all of practical value to the agent-retrieval community. The paper also deserves credit for a clean experimental isolation (only the retriever output is injected into a frozen downstream agent), for reporting both pooled and paired-instance efficiency numbers, and for the paired analysis in §5.3 that preserves 60–80% of the pooled effect. However, the current evidence base is not yet sufficient: the headline numbers are selected from three iterations evaluated on the same 500-instance set, no held-out split is reported, and the internal retrieval metric is aligned with the training objective. These issues directly affect the central claims and must be addressed before the reported magnitudes can be trusted.","major_comments":[{"comment":"The headline numbers in Table 3 are selected maxima from three training iterations evaluated on the same 500 SWE-Bench Verified instances. The v3 configuration was chosen after v1, v2, and v3 had all been evaluated on this identical set, and early stopping at step 897 (§3.6) also uses reward/KL curves on the same data. No held-out split is reported. Consequently, the +1.2pp resolve lift (6/500 instances) is within one standard error of the baseline (SE ≈ 1.9pp at n=500), and the −15%/−19% efficiency values are best-of-three rather than unbiased estimates. The claim in §4.3(i) that the resolve lift is 'reproducibly positive' is not supported by repeated runs or significance testing.","section":"§3.5–§3.6, §4.3, Table 3"},{"comment":"The paired-instance analysis is the correct check and preserves 60–80% of the pooled effect, but it still covers only the 96 co-resolved instances from the same selected v3 run and reports no confidence intervals. A bootstrap or paired test over these 96 instances is needed to determine whether the −9% rounds and −15% tokens differences are distinguishable from noise; without such uncertainty quantification, the central efficiency claim remains unverified at the precision stated in the abstract.","section":"§5.3"},{"comment":"The internal retrieval-quality evaluation scores retrievers against CATM labels, which are exactly the reward target of the GRPO training, using the same GLM judge and the same w≥0.15 filter. Any systematic bias in the one-to-many attribution or the conservative RELEVANT default therefore shifts both the training signal and the measured precision values in the same direction. The precision values that anchor the §5.1 threshold (0.375/0.445/0.677) are thus partly measures of alignment with the training objective, not independent retrieval quality. The paper mentions a senior-engineer audit in §4.2, but does not report its size, inter-annotator agreement, or whether the audit re-labeled the evaluation set; this needs to be specified, ideally with a human-annotated held-out subset.","section":"§3.4, §4.2, Table 2"},{"comment":"The three-regime precision-threshold claim is inferred from three retrievers that differ not only in precision but also in source and modality (lexical BM25, dense Jina, RL agent). The monotone association between precision and downstream utility is suggestive, but it is confounded with retriever type. The paper should either add more retrievers at intermediate precision values or explicitly temper the claim from a causal threshold to a correlation observed on three points; the current wording in the abstract ('downstream utility follows a precision threshold') overstates the evidence.","section":"§5.1"}],"minor_comments":[{"comment":"The answer schema still includes line_ranges even though v3 removes the line-range reward component and the downstream editor consumes only file paths. Clarify whether the final model still emits line_ranges and, if so, why the schema was not simplified.","section":"§3.2"},{"comment":"The CATM stage-2 judge thresholds are described as 'reasoning_tokens≥30' in Appendix A.2, while §3.4 and Eq. (1)–(2) emphasize the w≥0.15 filter with global β and μraw. Clarify how the two thresholds interact and whether the token threshold is part of the weight computation.","section":"§3.4"},{"comment":"The 'Turns' column is not defined in the caption for the non-agentic baselines. State explicitly that it is the mean inference-time turn count of the agentic retriever and does not apply to BM25/Jina.","section":"Table 2"},{"comment":"The baseline resolve rate is reported as 25.8%, within one standard deviation of the public 25.2±0.7%. Since the paper later relies on a +1.2pp difference, it would be helpful to state the number of seeds or runs that produced the 25.8% value and whether the baseline itself is a single rollout.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central efficiency claim is promising, but the evaluation protocol currently lacks a held-out split and uncertainty quantification; both are needed before the headline magnitudes can be accepted. The paired-instance analysis and the frozen-downstream-agent setup are genuine strengths. The internal retrieval evaluation's alignment with the training target is a serious but addressable concern. I recommend requiring the authors to add either a held-out evaluation or a principled significance analysis, and to report confidence intervals for the paired efficiency differences."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honestly, the efficiency direction is credible, but the headline numbers as reported are the best of three runs on the same 500 instances, so they aren't unbiased estimates. The +1.2pp resolve lift is 6 out of 500 instances, within roughly one standard error of the cited baseline variance, and the paper itself (§7) says the contribution is concentrated on rollout cost, so nobody should lean on the resolve lift.\n\nWhat is genuinely new and good here is the reward-design ablation. The v1→v2→v3 comparison—reward-layer vs advantage-layer efficiency scaling, then dropping the line-range term that the downstream editor never consumes—is a clean, honest experiment, and the KL drift numbers (0.31 vs 0.09) illustrate a real mechanism. The engineering is also worthwhile: the git-worktree sandbox that turns minutes of Docker setup into milliseconds, and the CATM one-to-many attribution for parallel tool calls. The paper is transparent: it reports both pooled and paired efficiency numbers, concedes the selection effect in §5.3, and gives enough appendix detail to reproduce the baselines.\n\nThe soft spots are real but proportionately located. First, model selection on the test set: v3 was chosen after v1/v2/v3 all ran on the same 500 SWE-Bench instances, and no held-out final evaluation is reported. That makes the pooled magnitudes selected maxima. The paired analysis on the 96 co-resolved instances is the right check, and it preserves 60–80% of the pooled effect, so the efficiency direction probably holds; but the -9% rounds / -15% tokens come with no confidence intervals. Second, the internal retrieval-quality table scores CodeGrep against the same CATM labels it was trained on. The auditor pass helps but doesn't break the circularity. Third, the precision-threshold claim is bracketed, because BM25/Jina/CodeGrep differ in recall and output cardinality as well as precision. None of these is fatal to the central efficiency claim, but they mean the magnitudes should be treated as provisional until there's a held-out run and artifacts.\n\nThis paper is for people building agentic retrieval or RL for tool use. The reward-design lesson alone is worth the read. I'd send it to peer review—a serious referee could push for a proper train/val/test split, bootstrapped CIs, and artifact release. With those, the efficiency claim could be solid. As is, conditional.","headline":"Plausible efficiency result, but the headline numbers are best-of-three selections on the same eval set, and the resolve lift is inside the noise; the reward-design ablation is the most solid piece.","tokens_in":17284,"tokens_out":3438,"would_cite":false,"duration_ms":33095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 14B retrieval agent cuts coding-agent token waste by 19% and nudges resolve rate up 1.2 percentage points.","keywords":["code retrieval","LLM coding agents","reinforcement learning","GRPO","SWE-Bench Verified","precision threshold","trajectory mining","token efficiency"],"falsifier":"Re-run the internal retrieval evaluation with one-to-one attribution, crediting each post-reasoning trace only to the file the agent actually reasoned about, and with gold-patch files as ground truth; if CodeGrep's file precision then falls below roughly 0.45 or the paired-instance efficiency gain (−9% rounds, −15% tokens) disappears, the precision-threshold claim and the headline efficiency lift would not survive.","tokens_in":16021,"feed_emoji":"🔍","tokens_out":5348,"duration_ms":49131,"temperature":0.7,"pith_summary":"The paper claims that much of an LLM coding agent's token budget goes to finding the file to patch, not patching it, and that a dedicated retrieval agent can compress that phase. CodeGrep, a 14B model trained with GRPO to issue parallel grep, glob, and read calls, is injected into a frozen OpenHands agent; on SWE-Bench Verified it preserves and slightly improves resolve rate (25.8% to 27.0%) while cutting rounds by 15% and tokens by 19% on resolved instances. The authors also report a precision threshold: retrieval below roughly 0.45 precision hurts the downstream agent, while CodeGrep's 0.677 precision crosses into net efficiency gains. A reader should care because the result isolates a practical lever that converts directly into rollout cost and because the reward-design study identifies where efficiency signals should enter GRPO training.","feed_headline":"RL retriever cuts coding-agent tokens 19% and lifts fixes 1.2pp","feed_subtitle":"A precision threshold decides when retrieval helps; CodeGrep crosses it on SWE-Bench Verified.","key_machinery":"The load-bearing mechanism is the precision-threshold characterisation of retrieval-agent coupling, measured as file-level precision of injected candidates against a CATM-mined ground-truth set. On one side sits CodeGrep itself: a 14B Qwen3-based retriever trained with GRPO over a Git-worktree sandbox, emitting up to eight parallel grep, glob, and read calls per turn and up to four turns before returning a file list. On the other sits CATM (Code Agent Trajectory Mining), which labels a file relevant if a past OpenHands agent opened it and produced non-trivial post-reasoning, with exponential-saturation weighting and a threshold filter. The efficiency signal enters through the advantage layer via $s(\\bar{c}) = \\sqrt{\\min(\\bar{c}/4,1)}$, which the paper says preserves group reward ranking while shrinking policy drift to roughly one-third of reward-layer scaling.","core_discovery":"On the paper's own terms, the central discovery is that retrieval quality maps to downstream utility through a precision threshold rather than a linear payoff: BM25 (precision 0.375) degrades the frozen agent, Jina (0.445) is neutral, and CodeGrep (0.677) buys efficiency, cutting 15% of rounds and 19% of tokens on resolved SWE-Bench Verified instances while adding +1.2pp resolve rate. The paper claims this is enabled by training a retrieval agent end-to-end with GRPO against behavioural relevance labels mined from 67K open-source agent trajectories, with the efficiency signal applied at the GRPO advantage layer instead of the reward layer to avoid policy drift. The paired-instance analysis narrows the per-issue efficiency effect to -9% rounds and -15% tokens on the 96 instances both configurations resolve.","pith_inferences":["The precision threshold is identified from only three retrievers; a testable extension is to vary CodeGrep's precision continuously, for example by truncating its file lists, to measure the exact crossover point the paper leaves as an open range between 0.45 and 0.68.","The behavioural relevance labels from CATM may carry exploration noise; if the labels were re-audited with one-to-one attribution, CodeGrep's measured precision and the threshold position could shift, so the headline efficiencies should be re-checked against gold-patch-only labels.","Because the downstream agent is frozen, the gains likely compound if the retriever and coder are co-trained end-to-end against a resolve-rate signal; the paper lists this as future work and it is the natural next experiment.","The worktree sandbox makes multi-turn agent RL feasible on a single node, and that infrastructure could plausibly be reused to train other read-only tool policies such as test selection or dependency tracing."],"forward_implications":["If the precision threshold holds, retrieving at precision above roughly 0.45 becomes a reliable way to compress agent rollouts without sacrificing resolve rate.","The +1.2pp resolve lift and efficiency cuts are attributable to retrieved files injected into a frozen agent, so the retriever can be improved or swapped without retraining the downstream coder.","Training efficiency signals at the advantage layer rather than the reward layer should generalise to other multi-turn GRPO tool-use agents, reducing KL drift and length exploitation.","The CATM pipeline turns existing agent trajectories into relevance supervision without human annotation, which could scale retrieval training to new codebases and datasets.","Removing a line-range output that the downstream editor never consumes both stabilised training and improved downstream efficiency; agents whose editors do consume line ranges may behave differently."],"supporting_citations":[{"why":"SWE-Bench Verified supplies the 500 issues, gold patches, and resolve-rate benchmark every downstream comparison uses.","marker":"[Jimenez et al., 2023]"},{"why":"OpenHands is the frozen coding agent into which CodeGrep's files are injected and whose rollout costs are measured.","marker":"[Wang et al., 2024]"},{"why":"This public release provides the 67K OpenHands trajectories mined by CATM and the 25.2% baseline the paper reproduces.","marker":"[Nebius AI, 2025]"},{"why":"GRPO is the RL algorithm whose group-relative advantages the paper modifies to carry the efficiency signal.","marker":"[Shao et al., 2024]"},{"why":"The Jina-1.5B dense retriever is the mid-precision baseline defining the neutral regime of the precision threshold.","marker":"[Jina AI, 2024]"},{"why":"BM25 is the lexical baseline whose low precision defines the 'retrieval hurts' regime.","marker":"[Robertson and Zaragoza, 2009]"},{"why":"LRAT's trajectory-mining template is adapted by CATM to handle parallel reads and per-trajectory aggregation.","marker":"[Zhou et al., 2026]"},{"why":"This concurrent commercial system provides the reward-design contrast that motivates dropping the line-range term in v3.","marker":"[Pan et al., 2025]"}],"fun_headline_variants":["Retrieval precision threshold decides when coding agents save tokens","RL retriever hits precision 0.677, cuts agent tokens 19%","Precision threshold tells when retrieval helps LLM coding agents","CodeGrep: precision 0.677 unlocks 19% token savings","Why 0.677 precision is the tipping point for coding-agent retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training and internal evaluation rest on CATM's behavioural labels: a file is relevant if a past agent opened it and produced judged non-trivial reasoning, and if those agent reads carry exploration noise, both the trained retriever and the precision numbers that anchor the threshold shift.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval precision threshold decides when coding agents save tokens","RL retriever hits precision 0.677, cuts agent tokens 19%","Precision threshold tells when retrieval helps LLM coding agents","CodeGrep: precision 0.677 unlocks 19% token savings","Why 0.677 precision is the tipping point for coding-agent retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000538,"raw_usage":{"total_tokens":2618,"prompt_tokens":1015,"completion_tokens":1603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":1519}},"tokens_in":631,"tokens_out":1603,"duration_ms":12491,"temperature":1.0,"reasoning_tokens":1519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:45:17.904904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the internal retrieval evaluation with one-to-one attribution, crediting each post-reasoning trace only to the file the agent actually reasoned about, and with gold-patch files as ground truth; if CodeGrep's file precision then falls below roughly 0.45 or the paired-instance efficiency gain (−9% rounds, −15% tokens) disappears, the precision-threshold claim and the headline efficiency lift would not survive.","supporting_citations":[],"review_version":1}