{"id":"6f634f16-ede8-4a72-8f75-6f5660f52547","arxiv_id":"2412.01007","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CoRNStack, a consistency-filtered 21M-pair contrastive dataset with curriculum hard negatives, yields state-of-the-art code retrievers and the first finetuned LLM listwise code reranker.","lead":"This paper introduces CoRNStack, a 21 million pair training dataset for code search built by filtering noisy docstring-code pairs from The Stack v2 and mining hard negatives. A small retriever and a 7B reranker trained on this data beat much larger prior models on several code retrieval benchmarks and improve function localization on SWE-Bench-Lite.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic overlap between CoRNStack and evaluation benchmarks could inflate the claimed SOTA; exact-match filtering alone is insufficient and is explicitly acknowledged.","rationale":"The reader's weakest assumption is exactly the contamination concern: exact-match removal is insufficient to prevent overlap. The paper's own Limitations admission makes this a real risk, not a hypothetical one. The proposed concrete test is feasible because the dataset, proxy model, and evaluation sets are released, and it would settle whether the central SOTA claim survives semantic deduplication. I agree with the reader's conditional verdict: the evidence is strong and the ablations are informative, but the provenance overlap is unresolved. I do not see a reason to move the verdict to ACCEPT or REJECT without running the test; UNCHANGED preserves the appropriate conditional status. The paper deserves credit for artifact release, clear dataset-construction details, and consistent cross-benchmark improvements, all of which make the question empirically testable rather than a matter of trust.","tokens_in":17516,"tokens_out":2280,"duration_ms":24106,"concrete_test":"Reproduce the CoRNStack mining pipeline and compute the embedding similarity between every evaluation query/gold-code pair (CSN, AdvTest, and the 274 SWE-Bench-Lite examples) and all training pairs using the released proxy encoder. If any evaluation item has a training pair above the paper's own δ=0.7 similarity threshold, retrain the retriever after removing those near-duplicate training pairs and re-run Tables 3 and 6; if the margins over baselines shrink materially, leakage is the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CoRNStack's curated data, rather than any novel architecture, drives state-of-the-art retrieval performance. This claim depends on training/evaluation separation. CoRNStack is built from The Stack v2, which is also the upstream source for CodeSearchNet, AdvTest, and the repositories in SWE-Bench-Lite. The paper's own Limitations section states that heuristic filtering only removed exact matches and that 'additional semantic filtering is needed to catch similar queries and code.' The dual consistency filter (Section 2.2) selects pairs whose text and code are highly similar to each other, which may preferentially retain popular, well-documented snippets that are exactly the kind likely to recur in benchmarks with paraphrased docstrings or slightly modified function bodies. If such near-duplicates persist in the 21M training pairs, the reported zero-shot gains (e.g., 77.9 vs. 74.2 CSN average in Table 3) and the function-localization improvements in Table 6 could partly reflect memorization rather than dataset-driven generalization. This is not an internal inconsistency, but it is a correctness risk that directly bears on the paper's main contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoRNStack, a large-scale (21M pairs) contrastive dataset for code retrieval, built from The Stack v2. Pairs are filtered by a dual consistency criterion using a pretrained embedding model (Jina-Code-v2) and enriched with hard negatives mined via softmax sampling with curriculum temperature decay. The authors finetune a 137M bi-encoder (from Arctic-Embed-M) on this data and a 7B listwise reranker (from Qwen-2.5-Coder) using teacher-ranked examples from Qwen-2.5-32B. They report state-of-the-art results on CodeSearchNet, AdvTest, and CoIR, and improved function localization on 274 SWE-Bench-Lite instances, with the retriever outperforming CodeSage-Large (1.3B) and CodeT5+ despite being smaller.","tokens_in":17700,"tokens_out":5725,"duration_ms":47259,"significance":"The claimed contribution is a public dataset, models, and evidence that data curation by consistency filtering and hard-negative mining can produce a compact retriever that beats much larger models, plus the first LLM-based listwise code reranker that improves function localization. If the results hold, this is a valuable practical contribution for code retrieval and repository-level localization, and the released artifacts (dataset, code, models) are a strength that supports reproducibility. The ablations isolate the benefit of each component, which is useful. However, the central claim that the gains stem from dataset quality is at risk because CoRNStack is derived from the same upstream GitHub corpus as the evaluation benchmarks, and the paper's own Limitations section concedes that only exact-match filtering was used; semantic near-duplicates may remain. This makes the 'state-of-the-art' claims provisional until contamination is quantified or ruled out.","major_comments":[{"comment":"The paper explicitly states that 'Heuristic filtering in CoRNStack removed exact matches with evaluation datasets, but additional semantic filtering is needed to catch similar queries and code.' Since CoRNStack is built from The Stack v2, the same broad GitHub source underlying CodeSearchNet, AdvTest, and the SWE-Bench-Lite repositories, the zero-shot gains in Table 3 (e.g., CSN 77.9 vs. 74.2 for CodeT5+) and the function-localization improvements in Table 6 could be inflated by semantic near-duplicates between training pairs and test items. This is load-bearing for the central claim that dataset quality, rather than architecture or scale, drives performance. Please quantify the residual overlap (e.g., by computing embedding-based similarity between training instances and evaluation queries/code), and ideally retrain or re-evaluate with near-duplicate training pairs removed to show the gains persist.","section":"Limitations"},{"comment":"The dual consistency filter uses Jina-Code-v2 to select pairs that are top-k and above a similarity threshold. This procedure may preferentially retain popular, well-documented functions that also dominate the evaluation benchmarks. The paper should analyze the distribution of the retained pairs (e.g., by repository, star count, or function popularity) and examine whether the measured gains concentrate on functions from repositories that overlap with the test sets. Without such an analysis, the dataset-quality interpretation of the ablation results in Table 4 remains ambiguous.","section":"2.2"},{"comment":"The function-localization experiment compares against a single run of Agentless using released logs. The paper hypothesizes that GPT-4o may have memorized the SWE-Bench-Lite codebases, but the same concern applies to the proposed retriever and reranker, given that CoRNStack is derived from The Stack v2 and the evaluation repositories are public. Please report results on a subset of SWE-Bench-Lite repositories that are verified absent from the training data (or from The Stack v2), or otherwise control for training/evaluation overlap, before claiming that the framework 'significantly improves' localization in real-world settings.","section":"4.3"}],"minor_comments":[{"comment":"The header 'CodeSeachNet' is a typo for 'CodeSearchNet'.","section":"Table 3"},{"comment":"In Equation (1), the negative set H_B ∪ H is not well-defined; for a given query, the positive h_i^+ should be excluded from H. Please clarify the notation to avoid the impression that the current positive is treated as a negative.","section":"3.1 (Eq. 1)"},{"comment":"The first paragraph of Section 3 contains 'CONTRA STACK', which should be 'CoRNStack'; the paper also alternates inconsistently between 'CoRNStack' and 'CORNS TACK'.","section":"3"},{"comment":"The sentence 'Although the code LLM (Qwen-2.5-Text) performs worse in a zero-shot setting' appears to refer to Qwen-2.5-Code; please correct the model name.","section":"4.2.1"},{"comment":"The claim 'We are the first to finetune LLMs as code rerankers' is strong; please either provide a reference to prior code-reranking work or soften the claim.","section":"1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-structured and the released artifacts are a plus. The main concern is the unresolved training/evaluation overlap; the authors should add a leakage analysis. If they can show that the gains persist after removing near-duplicates, the paper would be acceptable. The current version is too strong in its claims given the acknowledged limitation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution. CoRNStack is a 21M-pair contrastive dataset for code retrieval, built by consistency-filtering The Stack v2 with a frozen embedding model, then adding mined hard negatives with a softmax curriculum. The authors also train the first LLM listwise reranker for code (from Qwen-2.5-Coder-7B) and show it improves function localization on SWE-Bench-Lite. All artifacts are released.\n\nWhat's new is the combination and the scale. Consistency filtering, hard-negative mining, and listwise reranking all exist in text retrieval; the paper adapts them cleanly to code, and the ablations (Table 4) show each piece helps. The retriever results are consistent across CSN, AdvTest, and CoIR, and the controlled comparison in Table 10 (2M subset vs full) shows the dataset's scale and quality both matter. The LLM-judged correctness metric in Table 2 is a nice way to show the filtering actually removes noise.\n\nWhere I'd push back: the contamination question is real and unresolved. CoRNStack is built from The Stack v2, which also underlies CodeSearchNet, AdvTest, and the SWE-Bench repos. The authors removed exact matches but admit semantic near-duplicates remain. If those persist, the zero-shot SOTA numbers in Table 3 and the localization gains could be partly memorization. This doesn't invalidate the dataset or the recipe—the ablations compare within the same data—but it does mean the absolute claims over baselines should be read with caution until someone runs a semantic dedup analysis. I'd want to see that before treating the numbers as settled.\n\nMinor issues: no variance is reported for most results; the SWE-Bench-Lite comparison with Agentless is not apples-to-apples because the candidate spaces and pipelines differ; and the filtering inherits Jina-Code-v2's relevance notion, which is a small hereditary bias.\n\nOverall this is a solid empirical paper. The dataset and models are likely to be reused, and the training recipe is clearly described. It deserves serious engagement. I would cite it and recommend the reading group look at it. If it came back to me as a referee, I'd accept it, with the contamination analysis as a requested revision.","headline":"CoRNStack is a solid, artifact-heavy contribution to code retrieval; the main open question is semantic leakage from The Stack into the benchmarks, which the authors themselves flag.","tokens_in":18277,"tokens_out":3082,"would_cite":true,"duration_ms":27382,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a carefully filtered contrastive dataset of text-code pairs, not bigger model capacity, is what makes code retrieval and reranking work.","keywords":["code retrieval","contrastive learning","dataset curation","hard negative mining","listwise reranking","function localization","code embeddings","consistency filtering"],"falsifier":"Look for near-duplicate training pairs: embed every training query and code snippet, embed every evaluation query and code snippet, then remove any training pair whose cosine similarity to an evaluation example exceeds a chosen threshold and retrain the retriever from scratch. If the benchmark scores fall back toward the baselines, the claim that curation quality drives the gains is falsified; if they hold, the quality explanation is supported.","tokens_in":17269,"feed_emoji":"🔍","tokens_out":8485,"duration_ms":71275,"temperature":0.7,"pith_summary":"The paper's central claim is that noisy training data, not model capacity, is the main bottleneck for code retrieval. To test this, the authors build CoRNStack, a 21-million-pair contrastive set of text-code pairs from a large open code corpus, cleaned by keeping only pairs whose proxy-embedding similarity ranks in the top two of the corpus and exceeds a threshold, then enriched with hard negatives. A 137M-parameter retriever trained on these pairs outperforms state-of-the-art code embedding models several times larger on standard text-to-code benchmarks and on a mixed code-retrieval benchmark. The same data trains a 7B-parameter listwise code reranker that further improves ranking, and the retriever-plus-reranker pipeline localizes functions for GitHub issues more accurately than a strong agentic baseline. A sympathetic reader should care because the results suggest that careful data curation can substitute for large model scale in code retrieval, and that reranking, underused in code, can be unlocked from the same data.","feed_headline":"A 21M-pair dataset makes a small code retriever beat 1.3B models","feed_subtitle":"Curated pairs with hard negatives also train code rerankers that improve GitHub-issue function localization.","key_machinery":"The load-bearing mechanism is a three-stage data pipeline. First, dual consistency filtering: using a proxy embedding model over the whole corpus, a (text, code) pair is kept only if the code ranks in the top two most similar snippets for its text and the similarity exceeds 0.7. Second, hard-negative mining: negatives are drawn from a precomputed similarity matrix with softmax-based sampling, and false negatives are removed by a relative threshold; a curriculum anneals the sampling temperature from 0.05 to 0.001 so that negatives get progressively harder. Third, contrastive training with InfoNCE loss for the retriever, and listwise training where a larger teacher LLM orders the positive and top negatives for a 7B student reranker. The same curated triplets carry both training stages, so the mechanism is one dataset reused for retriever and reranker rather than two separate annotation efforts.","core_discovery":"CoRNStack is the discovery: a large-scale contrastive dataset for code where each <query, positive, negatives> triple has been cleaned by dual consistency filtering and augmented with curriculum-mined hard negatives. The paper's experiments claim that training a 137M bi-encoder on this data reaches MRR@1000 77.9 average on CodeSearchNet and 59.5 on AdvTest, surpassing a 1.3B-parameter prior model at 71.2 and 52.7, despite zero-shot evaluation; on CoIR it reaches nDCG@10 60.1. The same dataset yields a listwise reranker that improves MRR@100 from 77.7 to 80.5 on CodeSearchNet and from 56.9 to 69.1 on AdvTest, and the combined retriever-plus-reranker improves function-level localization Top-5 from 50.0% to 67.5% on the retained 274 SWE-Bench-Lite examples. The mechanism is dataset quality: an LLM judge rates CoRNStack pairs 77.1% correct versus 52.9% for the unfiltered source, and ablations show that removing consistency filtering or hard negatives causes a large drop in performance.","pith_inferences":["If the gains are real rather than leakage, the same curation recipe should transfer to private or enterprise codebases, where large language models have not memorized the code during pretraining; this is a natural extension the paper leaves untested.","The paper's limitation note concedes that exact-match removal may leave semantic near-duplicates; an immediate test is to re-run training after similarity-based deduplication against the evaluation corpora.","A testable extension of the reranker result is to wire retriever-plus-reranker into a full repair loop rather than stopping at localization, and to see whether localization gains translate into resolved issues.","Since the data converts text-pretrained encoders into strong code retrievers, the same contrastive recipe could be applied to other structured domains with abundant weakly paired text and objects, such as tables or diagrams."],"forward_implications":["A 137M-parameter retriever trained only on text-to-code pairs surpasses prior models up to ten times larger on standard text-to-code retrieval benchmarks in zero-shot evaluation.","Training a 7B-parameter listwise reranker on the same curated triples improves ranking over the retriever's output on both benchmark sets.","Combining retriever and reranker lifts function localization on a real-world GitHub-issue benchmark from 50.0% to 67.5% at Top-5 on the retained 274 examples.","Existing text encoders can be fine-tuned on the dataset to become competitive code retrievers, so supervised text-ranking pretraining appears to transfer to code."],"supporting_citations":[{"why":"Supplies The Stack v2 corpus from which the bimodal text-code pairs are extracted.","marker":"Lozhkov et al., 2024"},{"why":"Defines CodeSearchNet and its language split, the main evaluation benchmark for the retriever.","marker":"Husain et al., 2019"},{"why":"Provides the prior code embedding baseline and the heuristic filtering choices that CoRNStack follows.","marker":"Zhang et al., 2024"},{"why":"Provides the CodeT5+ baseline and prior contrastive code embedding approach that CoRNStack improves upon.","marker":"Wang et al., 2023b"},{"why":"Supplies the proxy embedding model used in consistency filtering to compute corpus-level similarities.","marker":"Günther et al., 2023"},{"why":"Provides the false-negative filtering and hard-negative mining strategy adapted for the offline stage.","marker":"Moreira et al., 2024"},{"why":"Supplies the teacher-supervised listwise reranking training recipe and window/shuffle augmentation used for the code reranker.","marker":"Pradeep et al., 2023"},{"why":"Provides SWE-Bench-Lite, the real-world GitHub-issue benchmark used to evaluate function localization.","marker":"Jimenez et al., 2024"},{"why":"Contributes the function-localization task formulation and the Agentless baseline that the retriever-plus-reranker is compared against.","marker":"Xia et al., 2024"}],"fun_headline_variants":["Clean contrastive data lets small retriever beat 1.3B models","CoRNStack: curated pairs + hard negatives boost code retrieval","High-quality contrastive data lifts code retrieval and reranking","Better data, not bigger models: 137M retriever tops 1.3B","Consistency-filtered triples train strong code retrievers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the assumption that removing exact string matches keeps evaluation samples out of the training set; if semantic near-duplicates linger, the performance gains could be leakage rather than dataset quality.","fun_headline_variants_meta":{"raw":{"variants":["Clean contrastive data lets small retriever beat 1.3B models","CoRNStack: curated pairs + hard negatives boost code retrieval","High-quality contrastive data lifts code retrieval and reranking","Better data, not bigger models: 137M retriever tops 1.3B","Consistency-filtered triples train strong code retrievers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1355,"prompt_tokens":1045,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":661,"tokens_out":310,"duration_ms":3294,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:45:58.795224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look for near-duplicate training pairs: embed every training query and code snippet, embed every evaluation query and code snippet, then remove any training pair whose cosine similarity to an evaluation example exceeds a chosen threshold and retrain the retriever from scratch. If the benchmark scores fall back toward the baselines, the claim that curation quality drives the gains is falsified; if they hold, the quality explanation is supported.","supporting_citations":[],"review_version":1}