Pith. sign in

REVIEW 4 major objections 4 minor 15 references

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 14B retrieval agent cuts coding-agent token waste by 19% and nudges resolve rate up 1.2 percentage points.

desk verdict Plausible efficiency result, but the headline numbers are best-of-three selections on the same eval set, and the resolve lift is inside the noise; the reward-design ablation is the most solid piece. read the letter →

arxiv 2608.05886 v1 pith:ILBQLXKX submitted 2026-08-06 cs.SE cs.AI

Wuya Chen , Yihao yang , Yang Cao , Yue Lin This is my paper · ORCID
classification cs.SEcs.AI
keywords coderetrievalLLMcodingagentsreinforcementlearningGRPOSWE-BenchVerifiedprecisionthresholdtrajectoryminingtokenefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that much of an LLM coding agent's token budget goes to finding the file to patch, not patching it, and that a dedicated retrieval agent can compress that phase. CodeGrep, a 14B model trained with GRPO to issue parallel grep, glob, and read calls, is injected into a frozen OpenHands agent; on SWE-Bench Verified it preserves and slightly improves resolve rate (25.8% to 27.0%) while cutting rounds by 15% and tokens by 19% on resolved instances. The authors also report a precision threshold: retrieval below roughly 0.45 precision hurts the downstream agent, while CodeGrep's 0.677 precision crosses into net efficiency gains. A reader should care because the result isolates a practical lever that converts directly into rollout cost and because the reward-design study identifies where efficiency signals should enter GRPO training.

What carries the argument

The load-bearing mechanism is the precision-threshold characterisation of retrieval-agent coupling, measured as file-level precision of injected candidates against a CATM-mined ground-truth set. On one side sits CodeGrep itself: a 14B Qwen3-based retriever trained with GRPO over a Git-worktree sandbox, emitting up to eight parallel grep, glob, and read calls per turn and up to four turns before returning a file list. On the other sits CATM (Code Agent Trajectory Mining), which labels a file relevant if a past OpenHands agent opened it and produced non-trivial post-reasoning, with exponential-saturation weighting and a threshold filter. The efficiency signal enters through the advantage layer via $s(\bar{c}) = \sqrt{\min(\bar{c}/4,1)}$, which the paper says preserves group reward ranking while shrinking policy drift to roughly one-third of reward-layer scaling.

What would settle it

Re-run the internal retrieval evaluation with one-to-one attribution, crediting each post-reasoning trace only to the file the agent actually reasoned about, and with gold-patch files as ground truth; if CodeGrep's file precision then falls below roughly 0.45 or the paired-instance efficiency gain (−9% rounds, −15% tokens) disappears, the precision-threshold claim and the headline efficiency lift would not survive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that retrieval quality maps to downstream utility through a precision threshold rather than a linear payoff: BM25 (precision 0.375) degrades the frozen agent, Jina (0.445) is neutral, and CodeGrep (0.677) buys efficiency, cutting 15% of rounds and 19% of tokens on resolved SWE-Bench Verified instances while adding +1.2pp resolve rate. The paper claims this is enabled by training a retrieval agent end-to-end with GRPO against behavioural relevance labels mined from 67K open-source agent trajectories, with the efficiency signal applied at the GRPO advantage layer instead of the reward layer to avoid policy drift. The paired-instance analysis narrows the per-issue efficiency effect to -9% rounds and -15% tokens on the 96 instances both configurations resolve.

Load-bearing premise

The whole training and internal evaluation rest on CATM's behavioural labels: a file is relevant if a past agent opened it and produced judged non-trivial reasoning, and if those agent reads carry exploration noise, both the trained retriever and the precision numbers that anchor the threshold shift.

Editorial extensions

If this is right

  • If the precision threshold holds, retrieving at precision above roughly 0.45 becomes a reliable way to compress agent rollouts without sacrificing resolve rate.
  • The +1.2pp resolve lift and efficiency cuts are attributable to retrieved files injected into a frozen agent, so the retriever can be improved or swapped without retraining the downstream coder.
  • Training efficiency signals at the advantage layer rather than the reward layer should generalise to other multi-turn GRPO tool-use agents, reducing KL drift and length exploitation.
  • The CATM pipeline turns existing agent trajectories into relevance supervision without human annotation, which could scale retrieval training to new codebases and datasets.
  • Removing a line-range output that the downstream editor never consumes both stabilised training and improved downstream efficiency; agents whose editors do consume line ranges may behave differently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The precision threshold is identified from only three retrievers; a testable extension is to vary CodeGrep's precision continuously, for example by truncating its file lists, to measure the exact crossover point the paper leaves as an open range between 0.45 and 0.68.
  • The behavioural relevance labels from CATM may carry exploration noise; if the labels were re-audited with one-to-one attribution, CodeGrep's measured precision and the threshold position could shift, so the headline efficiencies should be re-checked against gold-patch-only labels.
  • Because the downstream agent is frozen, the gains likely compound if the retriever and coder are co-trained end-to-end against a resolve-rate signal; the paper lists this as future work and it is the natural next experiment.
  • The worktree sandbox makes multi-turn agent RL feasible on a single node, and that infrastructure could plausibly be reused to train other read-only tool policies such as test selection or dependency tracing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CodeGrep, a 14B retrieval agent trained with GRPO to perform multi-turn grep/glob/read tool calls and return candidate files to a frozen OpenHands downstream coding agent. On SWE-Bench Verified (500 instances), the authors report that CodeGrep v3 lifts the resolve rate by +1.2pp (25.8% to 27.0%) and reduces rounds by 15% and tokens by 19% on resolved instances. They also characterize a precision threshold governing downstream utility: BM25 (precision 0.375) hurts, Jina (0.445) is neutral, and CodeGrep (0.677) buys efficiency. The method section describes CATM, a trajectory-mining pipeline that produces relevance labels from 67K open-source agent trajectories, and a Git-worktree RL environment. The paper includes a reward-design study comparing reward-layer vs. advantage-layer efficiency scaling across three training iterations (v1, v2, v3).

Significance. If the headline efficiency result survives independent evaluation, CodeGrep would be a useful open contribution: the worktree-based RL sandbox, the CATM label-mining pipeline, and the advantage-layer efficiency-signal study are all of practical value to the agent-retrieval community. The paper also deserves credit for a clean experimental isolation (only the retriever output is injected into a frozen downstream agent), for reporting both pooled and paired-instance efficiency numbers, and for the paired analysis in §5.3 that preserves 60–80% of the pooled effect. However, the current evidence base is not yet sufficient: the headline numbers are selected from three iterations evaluated on the same 500-instance set, no held-out split is reported, and the internal retrieval metric is aligned with the training objective. These issues directly affect the central claims and must be addressed before the reported magnitudes can be trusted.

major comments (4)
  1. [§3.5–§3.6, §4.3, Table 3] The headline numbers in Table 3 are selected maxima from three training iterations evaluated on the same 500 SWE-Bench Verified instances. The v3 configuration was chosen after v1, v2, and v3 had all been evaluated on this identical set, and early stopping at step 897 (§3.6) also uses reward/KL curves on the same data. No held-out split is reported. Consequently, the +1.2pp resolve lift (6/500 instances) is within one standard error of the baseline (SE ≈ 1.9pp at n=500), and the −15%/−19% efficiency values are best-of-three rather than unbiased estimates. The claim in §4.3(i) that the resolve lift is 'reproducibly positive' is not supported by repeated runs or significance testing.
  2. [§5.3] The paired-instance analysis is the correct check and preserves 60–80% of the pooled effect, but it still covers only the 96 co-resolved instances from the same selected v3 run and reports no confidence intervals. A bootstrap or paired test over these 96 instances is needed to determine whether the −9% rounds and −15% tokens differences are distinguishable from noise; without such uncertainty quantification, the central efficiency claim remains unverified at the precision stated in the abstract.
  3. [§3.4, §4.2, Table 2] The internal retrieval-quality evaluation scores retrievers against CATM labels, which are exactly the reward target of the GRPO training, using the same GLM judge and the same w≥0.15 filter. Any systematic bias in the one-to-many attribution or the conservative RELEVANT default therefore shifts both the training signal and the measured precision values in the same direction. The precision values that anchor the §5.1 threshold (0.375/0.445/0.677) are thus partly measures of alignment with the training objective, not independent retrieval quality. The paper mentions a senior-engineer audit in §4.2, but does not report its size, inter-annotator agreement, or whether the audit re-labeled the evaluation set; this needs to be specified, ideally with a human-annotated held-out subset.
  4. [§5.1] The three-regime precision-threshold claim is inferred from three retrievers that differ not only in precision but also in source and modality (lexical BM25, dense Jina, RL agent). The monotone association between precision and downstream utility is suggestive, but it is confounded with retriever type. The paper should either add more retrievers at intermediate precision values or explicitly temper the claim from a causal threshold to a correlation observed on three points; the current wording in the abstract ('downstream utility follows a precision threshold') overstates the evidence.
minor comments (4)
  1. [§3.2] The answer schema still includes line_ranges even though v3 removes the line-range reward component and the downstream editor consumes only file paths. Clarify whether the final model still emits line_ranges and, if so, why the schema was not simplified.
  2. [§3.4] The CATM stage-2 judge thresholds are described as 'reasoning_tokens≥30' in Appendix A.2, while §3.4 and Eq. (1)–(2) emphasize the w≥0.15 filter with global β and μraw. Clarify how the two thresholds interact and whether the token threshold is part of the weight computation.
  3. [Table 2] The 'Turns' column is not defined in the caption for the non-agentic baselines. State explicitly that it is the mean inference-time turn count of the agentic retriever and does not apply to BM25/Jina.
  4. [§4.1] The baseline resolve rate is reported as 25.8%, within one standard deviation of the public 25.2±0.7%. Since the paper later relies on a +1.2pp difference, it would be helpful to state the number of seeds or runs that produced the 25.8% value and whether the baseline itself is a single rollout.

Circularity Check

1 steps flagged · score 3.0 of 10

Internal retrieval-quality metrics recycle the training objective; the headline downstream gains are externally measured and not circular, but are selected maxima.

  1. fitted input called prediction [§3.4 (Ground truth structure), §3.5 (Component scores), §4.2 (Retrieval evaluation), Table 2]
    "For each issue x, the reward’s target set is G(x) = Gpatch(x)∪ {f∈ L(x) :w f ≥0.15} ... The internal evaluation set is constructed from held-out swe-rebench instances with CATM-mined labels ... Each retriever is scored with Fβ (β=0.5)."

    CodeGrep's GRPO reward is the same precision-biased Fβ over the same CATM-merged target set G(x) that is used to score retrieval quality in Table 2. Reporting CodeGrep's precision (0.677) and mean Fβ (0.576) on CATM-labeled instances therefore measures how well the policy optimized its own training objective on held-out instances, rather than an independent estimate of retrieval quality. The precision-threshold characterization (§5.1) uses these self-same values as the x-axis that separates 'hurts / neutral / buys efficiency'; because the threshold is placed between values that are partly objective-aligned, this part of the argument is partially circular. The downstream injection results (§4.3) are measured on a frozen agent and are not circular, which limits the severity.

full rationale

The central efficiency and resolve-rate claims are not circular: CodeGrep's output is injected into a frozen OpenHands agent, and resolve rate, rounds, and tokens are measured on the external SWE-Bench Verified benchmark, with the baseline independently reproducing the public 25.2±0.7% number. The only partial circularity is in the internal retrieval-quality evaluation (Table 2): the Fβ metric and the CATM-merged target set G(x) used there are the same objective CodeGrep was trained against, so the precision values that anchor the precision-threshold characterization partly measure reward alignment rather than independent retrieval quality. This does not invalidate the downstream results, but it means the threshold's x-axis is not fully independent. Separately, the headline +1.2pp, −15%, and −19% figures come from v3, selected after v1/v2/v3 were all evaluated on the same 500 instances (§3.6, §4.3), so those magnitudes are selected maxima without a held-out final split or confidence intervals; this is an evaluation-protocol concern rather than circularity. No load-bearing self-citations or imported uniqueness theorems were found; the closely related works cited (LRAT, Cognition SWE-grep) are external and used for comparison or contrast.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The training signal, the internal evaluation, and the headline measurements each rest on a small set of hand-chosen constants and behavioral assumptions. The most important are the CATM labeling assumptions (behavioral relevance, one-to-many attribution, conservative judge default), the chosen Fβ=0.5 precision bias, the 0.15 weight threshold and median-saturation constant in the CATM weight function, and the hand-chosen advantage-scaling saturation point c̄=4. None of these is derived from an external principle; each shifts the measured precision values and therefore the position of the claimed threshold. No invented physical or ontological entities are introduced; the retriever, CATM labels, and worktree sandbox are engineered artifacts rather than postulated entities.

free parameters (5)
  • CATM weight threshold w≥0.15 = 0.15
    Chosen threshold on the saturation weight w(f) (Eq. 2) that decides whether a CATM-mined file enters the training target set G(x) (§3.4). No external criterion justifies this value.
  • CATM saturation median β = global median of post-reasoning token lengths
    Data-derived constant in the exponential saturation weight (Eq. 1) that controls how quickly reasoning length saturates file relevance; estimated on the 67K trajectories and shifts which files survive the 0.15 threshold.
  • Judge minimum reasoning tokens = 30 tokens
    Minimum post-reasoning length for a file to be judged in CATM Stage 2 (Appendix A.2, Table 5); interacts with the saturation weight and the judge's conservative default.
  • Fβ precision bias β=0.5 = 0.5
    Precision-biased Fβ used both as the training reward and the retrieval-quality metric (§4.2, Table 2). Motivated qualitatively by the false-positive cost, never fitted, so its effect on the reported precision ordering is a choice.
  • Advantage scaling saturation point c̄=4 = 4 (half of the 8-call per-turn budget)
    Hand-chosen constant in s(c̄)=sqrt(min(c̄/4,1)) (Eq. 7), the advantage-layer efficiency signal. The functional form and saturation point were selected in the v2 design, not derived.
assumptions (6)
  • domain assumption A file is relevant if a past agent opened it and produced non-trivial post-reasoning grounded in its contents.
    CATM's behavioral definition of relevance (§3.4). If agent reads encode misdirected exploration rather than actual relevance, the training target is biased; the paper itself discards 52.3% of mined candidates as misdirected reads and documentation lookups.
  • domain assumption One assistant message following up to 8 parallel reads is valid evidence of relevance for each read file.
    CATM one-to-many attribution (Stage 1-2, Table 4 vs LRAT). A shared post-reasoning is applied to every file in a parallel batch, so files that did not contribute to the reasoning can receive positive weight.
  • domain assumption Judge ambiguity should default to RELEVANT.
    CATM Stage 2 conservatively treats any non-explicit NOT_RELEVANT output as RELEVANT (§3.4), admitting borderline noise into G(x) to avoid dropping true positives.
  • domain assumption A false positive costs more than a false negative for the downstream agent.
    This justifies the precision-biased Fβ (β=0.5) in both training and evaluation (§4.2). The paper gives a qualitative argument (context pollution vs recoverable miss), not a measured cost ratio.
  • domain assumption Multiplying group-relative GRPO advantages by s(c̄) preserves the optimization signal.
    The v2/v3 advantage-layer reweighting (Eq. 7) modifies per-rollout advantages after group ranking (§3.5); the paper assumes this does not distort the relative policy update, and supports it only with single-run training curves.
  • domain assumption The merged target G(x)=Gpatch(x) ∪ CATM survivors with w≥0.15 is the correct relevant set for evaluation.
    Held-out retrieval evaluation scores against this merged set (§3.4, §4.2). Gold-patch files alone are acknowledged incomplete, so the completeness of the union depends on CATM label quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents." pith.science (2026). https://pith.science/paper/ILBQLXKX

@misc{pith2026260805886,
  author       = {Pith},
  title        = {Pith review of: CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILBQLXKX}},
  note         = {Machine review of arXiv:2608.05886}
}
read the original abstract

Modern LLM coding agents such as Claude Code and OpenHands share a common inefficiency: they spend much of their token budget finding the file to patch, rather than patching it. On SWE-Bench Verified, a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved issue, with many calls spent on grep, glob, and view_file during repository exploration. We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue multi-turn parallel grep, glob, and read tool calls and return candidate files to a frozen downstream coding agent. On all 500 SWE-Bench Verified instances, CodeGrep preserves resolve rate while substantially improving efficiency: 27.0% versus 25.8% for the no-retrieval baseline, with 15% fewer rounds and 19% fewer tokens on resolved instances. Across retrievers, downstream utility follows a precision threshold: BM25 with precision 0.375 degrades the agent, Jina with precision 0.445 is neutral, and CodeGrep with precision 0.677 crosses the threshold at which retrieval begins to reduce rollout cost. To enable this study, we mine supervision from 67K open-source agent trajectories using CATM and build a Git-worktree environment for multi-turn agent RL. In our setting, applying the efficiency signal at the advantage layer rather than the reward layer reduces KL drift and translates cleanly into downstream efficiency. We will release the model, training pipeline, RL environment, and evaluation harnesses.

Figures

Figures reproduced from arXiv: 2608.05886 by the authors.

Figure 1
Figure 1. System overview. Training (blue) produces CodeGrep weights from open-source agent trajectories via [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Training dynamics of the three reward-design iterations. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Three-layer RL environment. Layer 1 extracts unique ( [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training-stability diagnostics over the same [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages

  1. [2]

    classical BM25

    Taken together, the three configurations trace a clean design trade-off: v1 (reward-layer scaling) contains the length problem but at the cost of pol- icy drift (KL climbing to ∼0.31 ; Figure 2b); v2 (advantage-layer scaling) controls policy drift but opens the length channel; only v3 (no line-range) resolves both, producing training that is simulta- neou...

    work page 2000

  2. [3]

    Nebius AI

    ToRL: Scaling tool-integrated RL.arXiv preprint arXiv:2503.23383. Nebius AI

    arXiv

  3. [4]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y

    Rea- sonIR: Training retrievers for reasoning tasks.arXiv preprint arXiv:2504.20595. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo

    arXiv

  4. [6]

    arXiv preprint arXiv:2407.16741

    Openhands: An open plat- form for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741. Zihan Wang, Kangrui Wang, Qineng He, Linjie Zhang, Ye Zheng, Xujie Zhou, Xinyu Deng, Bingxuan Liu, Chi Zhang, Heng Ji, and Manling Li

    arXiv

  5. [7]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang

    RA- GEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang

    arXiv

  6. [8]

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen

    Agentless: Demystifying LLM-based software engineering agents.arXiv preprint arXiv:2407.01489. Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen

    arXiv

  7. [9]

    An Yang et al

    SWE-Fixer: Training open-source LLMs for effective and ef- ficient GitHub issue resolution.arXiv preprint arXiv:2501.05040. An Yang et al

    arXiv

  8. [10]

    John Yang, Carlos E

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. John Yang, Carlos E. Jimenez, Alexander Wettig, Kil- ian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    arXiv

Show all 15 references
  1. [11]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao

    Swe-agent: Agent-computer inter- faces enable automated software engineering.arXiv preprint arXiv:2405.15793. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao

    arXiv

  2. [13]

    assistant

    Moatless tools: A framework for LLM-assisted code editing. https://github.com/ aorwall/moatless-tools. A Infrastructure Details This appendix preserves the complete engineering realization of the RL environment (§3.3) and the CATM pipeline (§3.4). A.1 RL Environment Implementa...

    2026

  3. [14]

    an average of weighted F1 scores over file retrieval and line retrieval tasks

    describes its reward as “an average of weighted F1 scores over file retrieval and line retrieval tasks” with advantages scaled “by the average number of tool calls used per turn.” Their blog does not specify β, the line-range matching rule, the tool- scale functional form, or ...

    2025

  4. [2023]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han

    Swe-bench: Can language mod- els resolve real-world github issues?arXiv preprint arXiv:2310.06770. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han

    arXiv

  5. [2024]

    Xingyao Wang et al

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300. Xingyao Wang et al

    arXiv

  6. [2025]

    Search-R1: Training LLMs to reason and leverage search en- gines with reinforcement learning.arXiv preprint arXiv:2503.09516. Jina AI

    arXiv

  7. [2026]

    9 Albert Örwall

    Learning to retrieve from agent trajectories.arXiv preprint arXiv:2604.04949. 9 Albert Örwall

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.