Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Finding the right repository files before a coding agent edits is a separate, unsolved problem—and no single retrieval method wins it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 20:53 UTC pith:3QVNXUBC

load-bearing objection Solid diagnostic benchmark that cleanly isolates agent context-finding; the “no family wins” story holds, but the 0.004 MRR crown and miss-rate urgency rest on thin ice. the 3 major comments →

arxiv 2607.24882 v1 pith:3QVNXUBC submitted 2026-07-27 cs.IR cs.AIcs.CL

Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Bowen Qin , Yi Xie This is my paper
classification cs.IR cs.AIcs.CL
keywords coding agentsrepository retrievalcode searchcontext acquisitionbenchmarkselective retrievalbudgeted context yieldagentic relevance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Coding agents are usually scored on whether the final patch works, which hides an earlier step: locating the repository files the agent must read. This paper isolates that upstream context-finding problem in a file-level benchmark built from real workflow signals—pull requests, review comments, failure traces, and anchored edits—scored against frozen base-commit snapshots. Relevance means “what the agent needs next,” not text similarity between query and file. Across 345 positive cases, embeddings, lexical methods, and structure-aware repo maps each win on different metrics and tasks; logged agents still never touch any gold file on roughly three in ten samples; and simple score thresholds that look good on wrong-repository controls fail on natural “no local fix” issues. A small seed pilot shows better initial retrieval context raises file F1 with less later exploration than random context, while oracle gold context leaves clear headroom. The sympathetic takeaway is that agent retrieval should be hybrid and task-aware, and should be measured before end-to-end patch success hides it.

Core claim

On Agent Retrieval Bench, no retrieval family dominates agentic repository context finding: among open baselines on 345 positive samples, Qwen3-Embedding-4B leads sample-weighted MRR, Qwen3-Embedding-8B leads Recall@20, and RepoMap leads budgeted context yield at 8K tokens, with task winners shifting sharply (RepoMap on trace2code; embeddings elsewhere). Selective abstention calibrated with counterfactual controls does not improve success on natural no-gold cases, and logged agent trajectories miss every gold file on 27–35% of samples.

What carries the argument

Agentic relevance: a file counts as relevant when reading it would help the agent’s next workflow step (tests for an implementation change, extra context beyond a reviewed file, root-cause source from a failure, or ripple files beyond an anchor), not when it is the text most similar to the query. The benchmark operationalizes this with four positive tasks plus natural and counterfactual no-gold strata, evaluated by ranked-file metrics and budgeted context yield (BCY) under frozen base-commit corpora.

Load-bearing premise

The labeled gold files—backed by PRs, review responses, resolving changes, and ripple evidence—are treated as a good enough stand-in for “files an agent needs next,” even though other useful unlabeled files may exist.

What would settle it

Run a paired seed intervention on executable issue-resolution instances with fixed model, tools, and budgets: if retrieval-derived seeds do not raise official patch resolution rates versus no-seed and random non-gold seeds, or if methods that win on this file-level gold systematically lose when gold is redefined by successful agent trajectories or fuller human labels, the central claim about upstream retrieval quality and method complementarity would not hold as stated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Coding-agent systems should combine semantic embeddings, lexical/path signals, and repository structure rather than rely on one retriever family.
  • Benchmarks and product evals should score context acquisition (rank, token budget, abstention, exploration cost) separately from final patch success.
  • Abstention for externally resolved issues needs better signals than raw top retrieval scores calibrated on wrong-repository mismatches.
  • Better initial retrieval seeds can cut post-seed search cost and raise context quality, but oracle seeds show large remaining headroom after localization.
  • Task-specific winners (e.g., structure on failure traces) imply workflow-aware routing or fusion rather than a single global ranking model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Production agent stacks that only report pass@k on issues may be optimizing edit/test loops while leaving a large, measurable miss rate in first-file discovery unaddressed.
  • The file-versus-span gap (median gold files hundreds of lines; labeled evidence a few percent of the file) suggests the next bottleneck after file retrieval is cheap within-file localization under the same budgets.
  • Fusion gains without training imply many “agent memory” failures are still rank-fusion and seed-design problems, not only model-scale problems.
  • Natural no-gold calibration failure hints that deployment gates need issue-type or maintainer-signal features, not repository-mismatch detectors alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Agent Retrieval Bench (ARB), a file-level benchmark for the context-acquisition stage of coding agents: given a workflow-derived query (PR summary, review comment, failure trace, or anchored edit) and a repository frozen at a pre-resolution base commit, retrievers must rank the files an agent needs to read next, or abstain. The benchmark contains 427 samples across 25 repositories (345 positive, 50 natural no-gold, 32 counterfactual wrong-repository controls) over a corpus of ~392K files / 7.9M chunks. The authors define "agentic relevance" with a four-relation taxonomy, introduce a budgeted metric (BCY) and an exploration-savings proxy (PES), and evaluate lexical, RepoMap, five embedding models, RRF fusion, logged agent trajectories, and a 45-sample closed-tool seed-intervention pilot. Headline findings: no retrieval family dominates (Qwen3-4B leads weighted MRR, Qwen3-8B Recall@20, RepoMap BCY@8k and trace2code); counterfactual-calibrated abstention thresholds fail on natural no-gold cases; logged agents miss all gold files on 27–35% of samples; retrieval-derived seeds beat random seeds on File F1 under a fixed tool budget. The paper is careful about leakage (zero fatal shortcut hits), scoping (no patch-success causality claims), and discloses gold non-exhaustiveness, repository imbalance, and single-run pilot variance.

Significance. If the results hold, this is a useful and well-engineered diagnostic benchmark for an under-measured layer of coding-agent pipelines. Particular strengths worth naming: (i) base-commit corpora with zero detected fatal leakage across four explicit shortcut categories (exact gold paths, raw patches, fix commits, generated review artifacts), documented in Table 3; (ii) evidence-backed, externally grounded gold rather than retriever-derived labels; (iii) unusual methodological candor — non-exhaustive gold, provisional pplx rows, single-run seed pilot, and task/taxonomy confounding are all disclosed in the text rather than buried; (iv) several robust sensitivity analyses (repository-macro re-weighting, BCY tau sweep, file-vs-span granularity, candidate-filter ablation) that preempt the obvious objections; and (v) falsifiable negative results (natural-only abstention failure) reported with the same prominence as positive ones. The central qualitative finding — that semantic, lexical, and structural retrievers are complementary across workflow signals and budgets — is supported by wide-margin task-level results (e.g., Table 20) and does not depend on the thin aggregate margins flagged in

major comments (3)
  1. [§8.1, Table 4; Abstract] The headline comparative claim 'Qwen3-Embedding-4B has the best sample-weighted MRR' rests on a 0.0043 margin (0.2379 vs 0.2336 for Qwen3-8B) over 345 repository-imbalanced samples (Gin alone contributes 88). No paired bootstrap, confidence interval, or significance test is reported anywhere in the paper, even though the comparison is per-sample paired and a bootstrap would be trivial to run on the released artifacts. This margin is further exposed to the gold-incompleteness the paper itself acknowledges (§5.1: labels 'do not imply that every potentially useful file has been exhaustively labeled'; §11: 'alternative useful files may remain unlabeled'). That incompleteness is plausibly not method-neutral: unlabeled useful files are likely lexically/semantically close to the query (adjacent helpers, near-duplicate modules), which would systematically reward embedding/lexical families over s
  2. [§1 and §8.7, Table 13] The '27–35% of samples never touch any gold file' headline inherits the same label-completeness issue in the opposite direction: a sample is counted as a complete miss even if the agent read a genuinely useful but unlabeled file (the §11 caveat applies symmetrically). The 35.2%/27.2–29.3% figures are therefore upper bounds on true context-acquisition failure, not point estimates, and the bias is again potentially family/task-correlated (e.g., trace2code agents reading an unlabeled but adjacent source file). The abstract and §1 present these as clean miss rates. Please either state explicitly that these are upper bounds under evidence-backed-but-non-exhaustive gold, or provide a small human adjudication of a sample of 'both miss' cases (Table 15 reports only 9.4–14.6% both-miss at top-20, which would be a natural adjudication target) to bound the inflation.
  3. [§8.3, Tables 7–8] Table 8 reports best methods within 'primary agentic-relevance' slices, but by construction (§8.3 precedence rule, acknowledged in §11) the causal-indirect slice is exactly trace2code and the structural-indirect slice occurs only in comment2context. The table therefore largely re-labels the task-level results of Tables 6/20 rather than adding an orthogonal mechanism analysis, and rows like 'Causal-indirect: best R@20 = RRF-Q8+RM (0.8795)' are numerically the trace2code fusion result from Table 9. The paper does disclose the confounding, which is good practice, but the presentation invites readers to read Table 8 as independent evidence of complementarity when it is not. Recommendation: either add a within-task secondary-relation contrast (e.g., within code2test, D-primary vs W-primary performance, which is a genuine non-task-confounded slice and is feasible given 94.8% of samples have mu
minor comments (6)
  1. [Appendix B, comment2context example] The released comment2context example lists tokio/src/sync/mpsc/chan.rs in both gold.must_context_files and negative_distractors. A file cannot be both gold and a hard negative; if this reflects a real record in the release rather than a transcription error, it warrants a schema-level consistency check (and a note on how many records are affected), since distractor fields are advertised as part of the benchmark's realistic-distractor design (§5.1).
  2. [§8.1, Table 5] Table 5 shows RepoMap's repository-macro MRR collapsing from 0.2158 (weighted) to 0.1298, a far larger sensitivity than any embedding model shows. Since RepoMap's BCY@8k and trace2code wins are part of the headline, one sentence explaining that its advantage is concentrated in a few repositories (presumably Gin-heavy trace2code) would help readers calibrate the 'structural methods matter' claim.
  3. [Global (typesetting)] The extracted PDF shows spacing artifacts in several places (e.g., 'Benchisdefinedbywhatanagentneedsnext', 'Itsprimarygolddefinition'). If these are in the source rather than the extraction pipeline, they need a copy-editing pass.
  4. [§8.2, Table 6] The pplx-4B row is appropriately marked provisional (tokenizer warning, Appendix C), but Table 6 lists pplx as the edit2ripple MRR winner (0.2877) without the dagger caveat visible in the table itself; add the footnote marker there so the task-winner table is self-consistent with the caveat.
  5. [§8.8, Table 19] All selective-track thresholds, RRF k=60, BCY tau=1, and taxonomy cutoffs (0.10/0.12/0.16/0.30) are stated, which is good. However, the natural-only selective conclusion ('no ranker improves selective success') rests on n=50 no-gold cases; a Wilson interval on the no-gold abstention rates in Table 19 (e.g., Jina 0.94 on 50 cases) would show how much of the mixed-vs-natural gap is estimation noise versus a real calibration gap.
  6. [§8.6, Table 12] Table 12's legacy character-packing line-overlap diagnostic is clearly labeled as non-canonical, but the extremely low line F1 values (max 0.0276) deserve one sentence of interpretation: is this primarily chunk-boundary granularity, or genuine within-file localization failure? This matters for how readers should weight the file-level leaderboard (cf. §11's own caution).

Circularity Check

1 steps flagged

Empirical benchmark with externally evidenced gold; only minor definitional renaming in a secondary taxonomy slice the paper itself flags as confounded.

specific steps
  1. self definitional [§8.3 primary projection; §11 Limitations; Tables 7–8 vs Table 20]
    "The primary projection is deterministic: trace2code selects C ... Among the 287 taxonomy-labeled samples, causal-indirect is exactly the trace2code task, and structural-indirect primary cases occur only in comment2context; relevance-type results are therefore confounded with task composition."

    By construction every trace2code sample is labeled primary causal-indirect, so Table 8’s causal-indirect best-MRR (RepoMap 0.2742) is identical to the trace2code task result, not an independent relevance-type effect. The paper states this confound and treats the slice as descriptive only; it does not underwrite the main leaderboard.

full rationale

Agent Retrieval Bench is a construction-and-measurement paper, not a fitted derivation. Positive gold is taken from workflow/resolving-change evidence (PR-linked tests, post-comment commits, root-cause implementation files, ripple-changed files) that is independent of the evaluated retrievers; selective no-gold labels rest on maintainer evidence or deliberate wrong-repository pairing. Main leaderboard claims (family non-dominance, task winners, BCY curves, selective calibration failure, trajectory miss rates, seed-intervention deltas) are empirical scores against that fixed gold, not parameters fitted then re-reported as predictions. There is no self-citation uniqueness theorem, no ansatz smuggled from the authors’ prior work, and no load-bearing self-definition of the primary metrics. The sole mild circularity is secondary: the primary agentic-relevance projection assigns every trace2code sample to causal-indirect by rule, so the “causal-indirect” method ranking is the trace2code ranking under another name—an issue the paper explicitly acknowledges as task confounding and does not treat as a causal discovery. That does not force the central comparative claims. Score 1.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 4 invented entities

As a benchmark/evaluation paper, load-bearing commitments are definitional and constructional rather than physical axioms: agentic relevance as next-read context, gold from resolving-change evidence, frozen base-commit corpora, and chosen metrics/thresholds. Free parameters are evaluation knobs (RRF k, BCY tau, abstention threshold selection, taxonomy cutoffs) that affect slices more than the existence of complementary method rankings.

free parameters (5)
  • RRF constant k=60 = 60
    Standard reciprocal-rank fusion damping used in post-hoc hybrid ranks; changes fused ordering.
  • BCY minimum-content threshold tau = 1
    Canonical BCY uses tau=1 (any content token exposure); sensitivity checks other taus but main tables depend on this choice.
  • Selective abstention threshold via repo-grouped 5-fold balanced-accuracy max = per-fold argmax balanced accuracy
    Thresholds are selected on calibration folds rather than fixed a priori; protocol choice drives the negative selective-success result.
  • Primary relevance-projection cutoffs (e.g., d_i thresholds 0.10/0.12/0.16/0.30) = task-specific d_i and witness rules in §8.3
    Deterministic taxonomy slicing thresholds; affect descriptive primary-relation tables, not main gold labels.
  • Seed-pilot tool/token budgets (16 calls, 8k post-seed tokens, final 3 files, n=45) = 16 calls / 8k tokens / 3 files / 45 samples
    Hand-chosen closed-tool limits and sample size shape File F1 and exploration comparisons.
axioms (6)
  • ad hoc to paper A file is relevant iff reading it would materially help the agent’s next correct workflow step, even without direct query-file semantic similarity.
    Core evaluation semantics introduced in §3; operationalized via task-specific gold definitions.
  • domain assumption Historical resolving-change / PR / review-response evidence identifies gold next-context files at the pre-fix base commit.
    Construction pipeline in §5; authors note labels are evidence-backed but not exhaustive.
  • domain assumption Evaluating against frozen base-commit snapshots prevents fixed-code leakage and matches pre-resolution agent state.
    Stated throughout §1–6 as the hygiene standard for the benchmark.
  • domain assumption File-level ranking (with optional budgeted packing) is the primary unit for measuring coding-agent context acquisition.
    Problem definition §3; later span analysis shows this understates within-file localization difficulty.
  • domain assumption Standard IR metrics (Recall@k, MRR) plus BCY/PES/selective success are appropriate agent-oriented scores for this layer.
    Metrics section §3 and evaluation suite contribution; BCY packing and tokenizer are paper-specified conventions.
  • standard math Cosine similarity of L2-normalized embeddings and max chunk-to-file aggregation are valid zero-shot embedding baselines without task prompts.
    Appendix C baseline protocol; common IR practice, not optimized per model card.
invented entities (4)
  • Agentic relevance (D/S/W/C relation taxonomy) no independent evidence
    purpose: Formalize indirect workflow-dependent relevance beyond semantic code search and enable descriptive slices.
    Introduced in §3 and operationalized in §8.3; multi-label with deterministic primary projection. Independent handle is limited to released witness fields on this benchmark.
  • Budgeted Context Yield (BCY@B) no independent evidence
    purpose: Measure how much labeled gold context fits under a token budget after ranking and packing.
    Defined formally in §3 with regex_code_tokenizer_v1 and greedy path-header packing; paper-specific metric convention.
  • Potential Exploration Savings (PES@k) no independent evidence
    purpose: Upper-bound proxy for avoidable gold-localization delay when a retriever hits gold early relative to agent trajectories.
    Defined in §3 from joined trajectory and rank artifacts; explicitly not a causal saved-tool-call estimate.
  • Agent Retrieval Bench task suite (code2test, comment2context, trace2code, edit2ripple, selective no-gold) no independent evidence
    purpose: Provide workflow-grounded retrieval targets and abstention controls for coding agents.
    Main artifact of the paper; external validity depends on release adoption and future paired repair studies.

pith-pipeline@v1.2.0-grok45-kimik3 · 32287 in / 4145 out tokens · 85702 ms · 2026-07-31T20:53:15.112382+00:00 · methodology

0 comments
read the original abstract

Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.

Figures

Figures reproduced from arXiv: 2607.24882 by Bowen Qin, Yi Xie.

Figure 1
Figure 1. Figure 1: Agent Retrieval Bench protocol and task-specific agentic relevance. Every method receives a workflow-derived query and the repository frozen at its base commit, then ranks files or abstains. Positive targets represent the context needed for the next useful workflow step; selective cases test whether the system can recognize that no repository-local context should be returned. 3. We define an agent-oriented… view at source ↗
Figure 2
Figure 2. Figure 2: Canonical BCY curves on edit2ripple. Gray lines show the other five baselines. The verified Qwen ordering reverses between fully packed 4k and 8k budgets. † marks the provisional pplx run; 16k/32k points can be lower bounds from stored top-20 rankings. The figure also shows why edit2ripple is not just an easier lexical variant. Lexical matching is competitive with RepoMap by MRR and BCY, but neither domina… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 7 linked inside Pith

  1. [1]

    Aider repository map documentation, 2026

    Aider. Aider repository map documentation, 2026. URLhttps://aider.chat/docs/repomap.html. Accessed 2026-05-28

    2026

  2. [2]

    Akbar and Avinash C

    Shayan A. Akbar and Avinash C. Kak. A large-scale comparative evaluation of IR-based tools for bug localization. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 21–31, 2020. doi: 10.1145/33 79597.3387474

    arXiv 2020

  3. [4]

    Cormack, Charles L

    Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759, 2009. doi: 10.1145/1571941.1572114

    arXiv 2009

  4. [5]

    Diffusion-pretrained dense and contextual embeddings.arXiv preprint arXiv:2602.11151, 2026

    Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Milliken, Bo Wang, and Denis Bykov. Diffusion-pretrained dense and contextual embeddings.arXiv preprint arXiv:2602.11151, 2026. URLhttps://arxiv.org/abs/2602.11151

    arXiv 2026

  5. [6]

    Springer London,

    Orlena Gotel, Jane Cleland-Huang, and Andrea Zisman, editors.Software and Systems Traceability. Springer London,

  6. [7]

    CodeSearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019. URLhttps://arxiv.org/abs/19 09.09436

    Pith/arXiv arXiv 1909

  7. [8]

    Cumulated gain-based evaluation of IR techniques.ACM Transactions on Information Systems, 20(4):422–446, 2002

    Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques.ACM Transactions on Information Systems, 20(4):422–446, 2002. doi: 10.1145/582415.582418

    arXiv 2002

  8. [9]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE- bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations,

  9. [10]

    René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. InProceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437–440, 2014. doi: 10.1145/2610384.2628055

    arXiv 2014

  10. [11]

    Efficient code embeddings from code generation models.arXiv preprint arXiv:2508.21290, 2025

    Daria Kryvosheieva, Saba Sturua, Michael Günther, Scott Martens, and Han Xiao. Efficient code embeddings from code generation models.arXiv preprint arXiv:2508.21290, 2025. URLhttps://arxiv.org/abs/2508.21290

    Pith/arXiv arXiv 2025

  11. [12]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems, volume 33, pages 9459–9474,

  12. [13]

    RepoBench: Benchmarking repository-level code auto-completion systems

    Tianyang Liu, Canwen Xu, and Julian McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. InInternational Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=pPjZIO uQuF

    2024

  13. [14]

    CodeXGLUE: A machine learning benchmark dataset for code understanding and generation

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A machine learning benchmark dataset for code understanding a...

    2021

  14. [15]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. doi: 10.1561/1500000019

    doi:10.1561/1500000019 2009

  15. [16]

    CoRNStack: High-quality contrastive data for better code retrieval and reranking

    Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, and Heng Ji. CoRNStack: High-quality contrastive data for better code retrieval and reranking. InInternational Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=iyJOUELYir

    2025

  16. [17]

    Agentless: Demystifying LLM-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024. URLhttps://arxiv.org/abs/2407.01489

    Pith/arXiv arXiv 2024

  17. [18]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-1601. URLhttps://proceedings.neurips.cc/paper_fil es/paper/2024/hash/5a7c947568...

    doi:10.52202/079017-1601 2024

  18. [19]

    CORE-Bench: A comprehensive benchmark for code retrieval in the era of agentic coding.arXiv preprint arXiv:2606.11864, 2026

    Fuwei Zhang, Yanzhao Zhang, Mingxin Li, Dingkun Long, Lexiang Hu, Pengjun Xie, Zhao Zhang, and Fuzhen Zhuang. CORE-Bench: A comprehensive benchmark for code retrieval in the era of agentic coding.arXiv preprint arXiv:2606.11864, 2026. URLhttps://arxiv.org/abs/2606.11864

    Pith/arXiv arXiv 2026

  19. [20]

    SWE-Explore: Benchmarking how coding agents explore repositories.arXiv preprint arXiv:2606.07297, 2026

    Shaoqiu Zhang, Yuhang Wang, Jialiang Liang, Yuling Shi, Wenhao Zeng, Maoquan Wang, Shilin He, Ningyuan Xu, Siyu Ye, Kai Cai, and Xiaodong Gu. SWE-Explore: Benchmarking how coding agents explore repositories.arXiv preprint arXiv:2606.07297, 2026. URLhttps://arxiv.org/abs/2606.07297

    Pith/arXiv arXiv 2026

  20. [21]

    Qwen3 Embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025. URLhttps://arxiv.org/abs/2506.05176. 29

    Pith/arXiv arXiv 2025

  21. [2012]

    doi: 10.1007/978-1-4471-2239-5

    doi:10.1007/978-1-4471-2239-5

  22. [2020]

    28 Agent Retrieval Bench

    URLhttps://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html. 28 Agent Retrieval Bench

    2020

  23. [2024]

    URLhttps://openreview.net/forum?id=VTF8yNQM66

  24. [2025]

    URLhttps://arxiv.org/abs/2503.09089

    Pith/arXiv arXiv