CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding

Dingkun Long; Fuwei Zhang; Fuzhen Zhuang; Lexiang Hu; Mingxin Li; Pengjun Xie; Yanzhao Zhang; Zhao Zhang

Code retrieval for coding agents collapses from snippet search to requirement-driven repository search, and simple PR fine-tuning only partly closes the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 18:02 UTC pith:KO75DG7N

load-bearing objection Solid, useful benchmark paper: real L1→L2 gap on repository-state retrieval, careful construction, L3 noisier than advertised but still worth using. the 3 major comments →

arxiv 2606.11864 v2 pith:KO75DG7N submitted 2026-06-10 cs.IR

CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding

Fuwei Zhang , Yanzhao Zhang , Mingxin Li , Dingkun Long , Lexiang Hu , Pengjun Xie , Zhao Zhang , Fuzhen Zhuang This is my paper

classification cs.IR

keywords code retrievalagentic codingembedding modelsissue localizationrepository searchbenchmarksupervised fine-tuningbroader context

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that coding agents need a different kind of code retrieval than classic snippet search. Given a bug report or feature request, an agent must search a concrete repository state, find the files and functions that must change, gather supporting context, and ignore similar local distractors. Existing benchmarks mostly test docstring-to-function or decontextualized matching, so they miss that problem. CORE-Bench answers with three levels: traditional code understanding, issue-to-edit localization, and broader context retrieval, built from curated search tasks and software-change instances with temporally grounded repository snapshots. Across strong embedding models, performance falls sharply from Level-1 to Levels 2 and 3. Supervised fine-tuning on pull-request supervision lifts repository-level scores substantially, but a clear gap remains, which the authors read as evidence that agentic code retrieval is still open.

Core claim

Requirement-driven repository search for agentic coding is much harder than traditional code search: representative embedding models that score well on foundational code understanding drop sharply on issue-to-edit localization and broader context retrieval, while simple in-domain fine-tuning on pull-request supervision improves those agentic levels but still leaves a large remaining gap.

What carries the argument

CORE-Bench, a three-level retrieval benchmark: Level-1 code understanding, Level-2 issue-to-edit localization from pre-change repository snapshots and patch-aligned chunks, and Level-3 broader context retrieval labeled via agent browse trajectories, multi-model relevance judging, and allowlist-based usefulness checks.

Load-bearing premise

The paper treats broader-context labels as functionally useful mainly because they come from agent browsing plus LLM votes and because agents resolve more tasks when restricted to those labels than to edit-only labels.

What would settle it

If independent coding agents, given only Level-3 labels versus only Level-2 labels or human-curated context, do not resolve more tasks under the Level-3 set, or if re-judged relevance on held-out trajectories shows high noise, the claim that Level-3 measures agent-useful context would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid, useful benchmark paper: real L1→L2 gap on repository-state retrieval, careful construction, L3 noisier than advertised but still worth using.

read the letter

Punchline: this is a real benchmark resource for code retrieval under agentic coding, not just another docstring-to-function set. The sharp drop from Level-1 to Level-2 is the result that sticks, and the PR-based SFT gains are consistent enough to take seriously.

What is new is the framing and the data construction. They keep hard traditional code-search tasks as Level-1, then build Level-2 from SWE-bench-family instances with pre-PR checkouts, AST chunking, and patch-aligned edit targets. That is the right unit for “where do I edit?” Query filtering for answer leaks and boilerplate is done carefully. Level-3 tries to capture supporting context via agent browse traces, multi-LLM relevance votes, and allowlist re-runs. Scale is large (180K+ queries, 106K L3 labels), and the experiments cover sparse/dense models, languages, difficulty regimes, and rewrite variants. The paper is honest that rewriting often does not help and that SFT closes only part of the gap.

Soft spots, in proportion. Level-2 is the clean core. Level-3 is weaker: labels come from what Mini-SWE-Agent browsed, then LLM-judged, with relative allowlist resolve rates as functional validation. That can over-weight agent-browseable text and partial adjacency, and unresolved trajectories are retained after judging. So L3 NDCG/Recall and SFT gains on L3 should be read as a useful proxy, not as pure “necessary context.” The stress-test overstates circularity—the embedding models under test do not mint the labels—but the proxy is still imperfect. SWE-bench-family and language skew (Python-heavy) and the usual “retrieval ≠ full agent success” caveat are real but expected for this genre.

Math and metrics are standard (InfoNCE SFT, NDCG@10, Recall@100). Citations look appropriate for code search and agentic SE. Who it is for: people building or evaluating retrievers for coding agents, and anyone who still treats CodeSearchNet-style scores as enough. I would bring it to reading group, cite it if I work on code retrieval or agent tooling, and send it to peer review. Engage with the resource; treat L2 as primary and L3 as directional.

Referee Report

3 major / 0 minor

Summary. The paper introduces CORE-Bench, a three-level code-retrieval benchmark aimed at agentic coding rather than isolated snippet matching. Level-1 curates traditional code-understanding retrieval tasks; Level-2 converts SWE-bench-series instances into issue-to-edit localization over pre-PR repository snapshots with patch-aligned chunk labels; Level-3 expands targets to broader supporting context via Mini-SWE-Agent browse traces, multi-LLM relevance voting, and allowlist-based functional checks. Across sparse and dense embedding models, the authors report a large drop from Level-1 to Levels 2/3, and show that simple in-domain SFT on non-SWE-bench pull-request supervision substantially improves Level-2/3 scores while leaving a remaining gap. The resource is large (≈180K queries; 106K Level-3 qrels) and includes rewrite variants, multi-language coverage, and difficulty-regime splits.

Significance. If the construction and evaluation hold, CORE-Bench fills a real and timely gap: existing code-search suites (CodeSearchNet, CoIR, many MTEB code tasks) do not stress requirement-driven search over concrete repository states with local distractors and multi-location targets. Level-2 alone is already a useful, patch-grounded localization benchmark for retrievers used by coding agents. The multi-model comparison, language/difficulty breakdowns, and PR-based SFT results are concrete contributions that can steer embedding work beyond docstring-style matching. Strengths include explicit construction steps (pre-PR checkout, AST/LangChain chunking, patch alignment), repository-level evaluation protocol, and open release of the dataset. The main significance risk is that Level-3’s novelty depends on how faithfully trajectory-derived labels measure agent-useful context rather than agent-browsed text.

major comments (3)

§3.2.3 and Table 11: Level-3 is the paper’s main extension beyond edit localization, but its functional validation is only relative (higher allowlist resolve rate under L3 vs L2) and confounded by a more restricted agent environment than the full Mini-SWE-Agent run. The manuscript also retains unresolved trajectories after LLM judging and includes partial/adjacent context. As written, Level-3 can measure recovery of annotator-agent browse sets more than independently necessary context. Please either (i) add a human-validated sample of Level-3 qrels with inter-annotator agreement and precision estimates, (ii) report stricter necessity-style ablations (e.g., remove random subsets of L3 labels and re-measure resolve rate), or (iii) substantially narrow the claim language so Level-3 is presented as trajectory-supported context coverage with known proxy limitations.
§4.2 / Table 3 and the abstract’s “sharp drop … substantial room for further progress” claim: the Level-1→Level-2 drop is well supported by patch-aligned labels and is load-bearing. The Level-1→Level-3 drop and SFT gains on Level-3 inherit the label-quality uncertainty above, especially given denser multi-target qrels (41.27 Rels/Q) and emphasis on Recall@100. Separate the main quantitative claims: report Level-2 as the primary agentic result, and Level-3 as secondary with explicit sensitivity analysis (high-only vs high+partial labels; edit-only vs expanded targets). Without that separation, the headline gap can over-read a partially circular proxy.
§3.4 / Appendix B: in-domain SFT uses PR descriptions with patch-aligned positives and repository-local hard negatives from 628 repos outside the SWE-bench family. This is a good design, but the paper needs a clearer leakage and distribution check: overlap of organizations/projects with evaluation repos, temporal cutoff relative to SWE-bench instances, and whether SFT gains are driven by identifier/path memorization versus requirement-conditioned ranking. A simple control (e.g., path-shuffled or identifier-ablated evaluation, or training with only lexical BM25 hard negatives vs random) would make the “substantial improvement / remaining gap” conclusion more credible.

Circularity Check

0 steps flagged

No derivation circularity: CORE-Bench is an empirical IR benchmark with patch-aligned Level-2 labels, trajectory-derived Level-3 labels independent of the embedding models under test, and disjoint-repo SFT.

full rationale

CORE-Bench does not claim a first-principles derivation whose conclusion is forced by its inputs. Level-2 qrels are constructed by checking out pre-PR repository states and aligning PR patches to AST/LangChain chunks (Section 3.2.2); those labels are historical edit locations, not quantities fitted from the embedding models being scored. Level-3 qrels come from Mini-SWE-Agent browse traces plus multi-LLM relevance votes and are validated by allowlist resolve-rate comparisons (Section 3.2.3, Table 11); the annotators are generative agents/judges (Qwen3.5-397B-A17B, Claude Sonnet 4.6), not the dual-encoder embedding baselines in Table 3. In-domain SFT uses PR supervision from repositories outside the SWE-bench family (Section 3.4, Appendix B), so train and eval corpora are designed to be disjoint. Reported metrics are standard NDCG@10 and Recall@100. The observed Level-1→Level-2/3 performance drop and SFT gains are empirical comparisons, not predictions that reduce by construction to fitted parameters or self-defined targets. Concerns that Level-3 may over-weight agent-browsed rather than strictly necessary context are measurement-validity issues, not circular reductions of the form 'prediction = input by definition.' No self-definitional loop, fitted-input-as-prediction, load-bearing self-citation uniqueness claim, or renamed known result appears in the load-bearing chain.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claims rest on standard IR evaluation practice plus domain choices about what counts as an edit target and as useful broader context. There are few fitted scientific constants; the main free choices are training/evaluation hyperparameters and annotation thresholds. Invented entities are the benchmark levels and construction pipeline rather than physical objects.

free parameters (4)

InfoNCE temperature τ and 1:8 positive-to-negative training ratio
SFT results depend on these contrastive-training choices; they are standard but not uniquely determined by theory.
64 repository-local hard negatives per query
Hard-negative pool size is a design choice that shapes how well SFT teaches distractor rejection.
Three SFT epochs and chosen Qwen3-Embedding base scales
Reported SFT gains are for these specific training budgets and base models.
LLM relevance vote threshold / ensemble (two Qwen votes + one Claude vote)
Level-3 qrels depend on this judging setup and acceptance of score>0 labels.

axioms (4)

domain assumption Patch-aligned pre-PR chunks are the correct gold targets for issue-to-edit localization.
Level-2 treats historical edited spans as the relevant documents; this is reasonable but ignores alternative valid edit sites.
domain assumption Files browsed by a coding agent during resolution attempts, after LLM relevance filtering, constitute useful broader context for completing the change.
Core Level-3 construction assumption in Section 3.2.3; partially checked by allowlist resolution gains.
domain assumption NDCG@10 and Recall@100 on embedding retrieval are informative proxies for usefulness inside agentic coding workflows.
The paper evaluates standalone retrievers, not full agent end-to-end success under the same retrieval policy.
standard math Standard dual-encoder similarity and InfoNCE-style contrastive learning are appropriate training/evaluation formalisms.
Uses conventional dense-retrieval scoring and multi-positive InfoNCE (Eq. 1).

invented entities (2)

CORE-Bench three-level evaluation hierarchy (L1/L2/L3) no independent evidence
purpose: Separate foundational code understanding from edit localization and broader context retrieval for agentic coding.
The hierarchy is a paper-defined evaluation construct, not an independently measured natural phenomenon.
Automated Level-3 broader-context annotation pipeline with allowlist validation no independent evidence
purpose: Create multi-target context labels that developers rarely record in PR metadata.
The pipeline is a methodological invention whose quality is supported mainly by internal allowlist resolve-rate comparisons.

pith-pipeline@v1.1.0-grok45 · 46711 in / 3214 out tokens · 33913 ms · 2026-07-14T18:02:58.739745+00:00 · methodology

0 comments

read the original abstract

Code retrieval is becoming central to coding agents, but agentic coding requires more than matching a natural-language query to an isolated snippet. Given a user request, a coding agent needs to navigate a concrete repository state, locate relevant files and functions, gather supporting context, and filter similar in-repository distractors. Existing code retrieval benchmarks mainly evaluate docstring-to-function or snippet-level matching, thereby missing this requirement-driven repository search problem. To address this gap, we introduce CORE-Bench, a comprehensive benchmark for code retrieval in the era of agentic coding. CORE-Bench evaluates code retrieval ability at three levels: code understanding, issue-to-edit localization, and broader context retrieval. Built from curated code-search tasks and SWE-bench-series instances, CORE-Bench contains over 180K queries and 106K broader-context relevance labels. Experiments with representative embedding models show a sharp drop from traditional code search to code retrieval in agentic coding settings. Simple supervised fine-tuning of existing embedding models significantly improves performance in this setting, suggesting substantial room for further progress.

Figures

Figures reproduced from arXiv: 2606.11864 by Dingkun Long, Fuwei Zhang, Fuzhen Zhuang, Lexiang Hu, Mingxin Li, Pengjun Xie, Yanzhao Zhang, Zhao Zhang.

**Figure 2.** Figure 2: Overall pipeline of CORE-Bench (LEVEL-2 and LEVEL-3) benchmark construction. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Detailed performance comparison of top mod [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Spearman correlation of NDCG@10 between LEVEL-1 tasks and LEVEL-2/3 averages. 4.3 In-domain SFT Performance The last group of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
cs.IR 2026-07 conditional novelty 6.0

A 427-sample workflow-grounded benchmark finds complementary strengths across lexical, structural, and embedding retrieval for coding-agent context acquisition, with large residual miss rates and a selective-abstentio...

Reference graph

Works this paper leans on

10 extracted references · 4 linked inside Pith · cited by 1 Pith paper

[1]

Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan

Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938. Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan

Pith/arXiv arXiv
[2]

Cosqa: 20,000+ web queries for code search and question answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 5690–5700. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Mil- tiadis Allamanis, and Marc Brocks...

Pith/arXiv arXiv 2019
[3]

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers

Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. InProceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014–203...

Pith/arXiv arXiv 2023
[4]

Stephen Robertson and Hugo Zaragoza

Swerank: Software issue localization with code ranking.arXiv preprint arXiv:2505.07849. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond.Foundations and Trends in Information Re- trieval, 3(4):333–389. Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, and Heng ...

Pith/arXiv arXiv 2009
[5]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859

Opencodeinterpreter: Integrating code gener- ation with execution and refinement. InFindings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859. A Detailed Dataset Statistics Table 8 reports the sub-dataset-level statistics for all three CORE-Bench levels behind the aggregate summary in Table 1. LEVEL-1.LEVEL-1 brings together s...

2024
[6]

They provide a controlled general-embedding family for testing how model scale affects zero- shot transfer to agentic code retrieval before any in-domain adaptation

are public Qwen3-Embedding models. They provide a controlled general-embedding family for testing how model scale affects zero- shot transfer to agentic code retrieval before any in-domain adaptation. Code-oriented models. • CodeRankEmbed(Suresh et al., 2025) is a 137M bi-encoder trained for code retrieval us- ing contrastive data from CoRNStack. It is a ...

2025
[7]

delete()on instances of models without dependencies does not clear PKs

are Jina Code Embeddings checkpoints built from Qwen2.5-Coder backbones. They sup- port multiple code retrieval directions, including natural-language-to-code and code-to-code. • C2LLM-7B(Qin et al., 2025) is a code con- trastive LLM for code retrieval with adaptive cross-attention pooling. • SweRankEmbed-Large(Reddy et al., 2025) is a 7B issue-localizati...

2025
[8]

Fix #1234

FILTER_MEANINGLESS: The query is meaningless, underspecified, non-actionable, or has no substantive problem content. Examples: - Only contains an issue/PR number, such as "Fix #1234" or "Closes #567". - Contains only boilerplate template text with no filled content. - Is extremely short with no actionable information, such as "update", "fix", or "bug". - ...
[9]

Examples: - Explicitly mentions which file(s) to modify

FILTER_ANSWER_LEAK: The query already reveals the solution, making code retrieval trivial. Examples: - Explicitly mentions which file(s) to modify. - Describes exact code changes or diffs. - Provides the fix implementation in the description. - Names specific functions/methods/classes to change AND describes the exact change. Note: Simply mentioning a mod...
[10]

verdict":

KEEP: The query is a valid, meaningful problem description with enough issue-specific signal for retrieval and does not reveal the answer. Respond with ONLY a JSON object in this exact format: {"verdict": "FILTER_MEANINGLESS" | "FILTER_ANSWER_LEAK" | "KEEP", "reason": "<brief reason in English>"} [User prompt template] Query ID: <query_id> Query text: <qu...

[1] [1]

Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan

Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938. Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan

Pith/arXiv arXiv

[2] [2]

Cosqa: 20,000+ web queries for code search and question answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 5690–5700. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Mil- tiadis Allamanis, and Marc Brocks...

Pith/arXiv arXiv 2019

[3] [3]

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers

Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. InProceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014–203...

Pith/arXiv arXiv 2023

[4] [4]

Stephen Robertson and Hugo Zaragoza

Swerank: Software issue localization with code ranking.arXiv preprint arXiv:2505.07849. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond.Foundations and Trends in Information Re- trieval, 3(4):333–389. Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, and Heng ...

Pith/arXiv arXiv 2009

[5] [5]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859

Opencodeinterpreter: Integrating code gener- ation with execution and refinement. InFindings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859. A Detailed Dataset Statistics Table 8 reports the sub-dataset-level statistics for all three CORE-Bench levels behind the aggregate summary in Table 1. LEVEL-1.LEVEL-1 brings together s...

2024

[6] [6]

They provide a controlled general-embedding family for testing how model scale affects zero- shot transfer to agentic code retrieval before any in-domain adaptation

are public Qwen3-Embedding models. They provide a controlled general-embedding family for testing how model scale affects zero- shot transfer to agentic code retrieval before any in-domain adaptation. Code-oriented models. • CodeRankEmbed(Suresh et al., 2025) is a 137M bi-encoder trained for code retrieval us- ing contrastive data from CoRNStack. It is a ...

2025

[7] [7]

delete()on instances of models without dependencies does not clear PKs

are Jina Code Embeddings checkpoints built from Qwen2.5-Coder backbones. They sup- port multiple code retrieval directions, including natural-language-to-code and code-to-code. • C2LLM-7B(Qin et al., 2025) is a code con- trastive LLM for code retrieval with adaptive cross-attention pooling. • SweRankEmbed-Large(Reddy et al., 2025) is a 7B issue-localizati...

2025

[8] [8]

Fix #1234

FILTER_MEANINGLESS: The query is meaningless, underspecified, non-actionable, or has no substantive problem content. Examples: - Only contains an issue/PR number, such as "Fix #1234" or "Closes #567". - Contains only boilerplate template text with no filled content. - Is extremely short with no actionable information, such as "update", "fix", or "bug". - ...

[9] [9]

Examples: - Explicitly mentions which file(s) to modify

FILTER_ANSWER_LEAK: The query already reveals the solution, making code retrieval trivial. Examples: - Explicitly mentions which file(s) to modify. - Describes exact code changes or diffs. - Provides the fix implementation in the description. - Names specific functions/methods/classes to change AND describes the exact change. Note: Simply mentioning a mod...

[10] [10]

verdict":

KEEP: The query is a valid, meaningful problem description with enough issue-specific signal for retrieval and does not reveal the answer. Respond with ONLY a JSON object in this exact format: {"verdict": "FILTER_MEANINGLESS" | "FILTER_ANSWER_LEAK" | "KEEP", "reason": "<brief reason in English>"} [User prompt template] Query ID: <query_id> Query text: <qu...