{"id":"d4dd5c45-7015-4293-a200-0a5179c794b7","arxiv_id":"2506.20081","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Code retrievers rely on surface text like identifiers and docstrings rather than code semantics, and a description-based reranking pipeline, SACL, compensates for that bias but its headline gains come mostly from artificially normalized code.","lead":"This paper shows that code retrieval models lean heavily on names, docstrings, and other text rather than what the code actually does, and that they favor well-documented code even when it is irrelevant. It proposes SACL, a pipeline that generates plain-language descriptions of code and combines their similarity scores with the original code scores to improve retrieval and bug localization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval gains are measured only under asymmetric normalization (queries untouched, docs normalized); because SACL's reranking adds text that restores lexical overlap with the original query, the headline improvement may be an artifact rather than evidence against real-world textual bias.","rationale":"The reader's weakest assumption identifies the core confound: only code documents are normalized while queries retain their original lexical content. This asymmetry can explain both Discovery 1's dramatic performance drops and SACL's apparent recovery, since SACL's generated descriptions essentially re-introduce the missing textual overlap. The unnormalized HumanEval result of 0.0 gain and the absence of a real retrieval metric on SWE-Bench-Lite further weaken the abstract's claim of substantial improvement. Table 4's identical scores across two different code generation models also indicate a likely experimental reporting error, though this is secondary to the normalization asymmetry. Because the central effectiveness claim is not supported as stated, the reader's REJECT verdict with moderate confidence remains appropriate. A symmetric-normalization control would settle whether the measured textual bias is genuine or partly an artifact of the evaluation design.","tokens_in":12776,"tokens_out":3072,"duration_ms":34985,"concrete_test":"Re-run the §2.1/§2.2 normalization study with the query normalized in exactly the same way as the documents (identifiers renamed, docstrings removed from the query), and re-run Table 2 under that symmetric setting. If the Recall@1 drop from 98.6% to 9.0% persists and SACL still yields roughly 12.8% gains, the asymmetry concern is refuted. If most of the drop and gain disappears, the headline claims should be restricted to asymmetric synthetic normalization and the paper should be revised accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that retrievers 'heavily rely on surface textual features' and that SACL 'substantially improves code retrieval'—rests on the normalization experiments in §2.1/§2.2 and Table 2. In those experiments only the code documents are normalized; the natural-language query is left in its original form (§2.1 setup). Thus the query still contains original identifiers and docstring words, while the positive document, after full normalization, contains only placeholders. The collapse from 98.6% to 9.0% Recall@1 (and rank 25.92 to 96.98 in Table 1) can therefore be explained by an embedding-space mismatch between a lexically rich query and a stripped document, not necessarily by a learned preference for surface-level textual features over functional semantics. The paper's own Figure 4 supports this reading: SACL's descriptions increase ROUGE-1 overlap between query and positive documents, so the method works largely by restoring the lexical signal that the experiment removed. On unnormalized HumanEval, SACL shows 0.0 gain (Table 2, Docstring row, 98.8 to 98.8), and the only real-world evidence (SWE-Bench-Lite, Table 3) is file-localization accuracy on 300 instances with no significance test. The bias diagnosis may be real, but the headline effectiveness claim is not established by these experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes code retrieval systems by systematically normalizing textual features (docstrings, function/variable names) while preserving functionality, and reports two discoveries: (1) retrievers rely heavily on surface-level textual features rather than functional semantics, and (2) they are biased toward well-documented code even when the documentation is irrelevant. Based on these findings, the authors propose SACL, a framework that augments code retrieval with LLM-generated natural-language descriptions and uses semantic-augmented in-context localization for repository-level fault localization. They report substantial Recall@1 gains on HumanEval/MBPP under normalization settings, file-localization gains on SWE-Bench-Lite, and downstream Pass@1 improvements. The paper includes controlled experiments across embedding retrievers and LLM rerankers, several case studies, and an analysis of why the approach works in terms of lexical overlap.","tokens_in":12989,"tokens_out":5383,"duration_ms":62182,"significance":"If the bias diagnosis and the effectiveness claims were fully established, this work would make a useful contribution to code retrieval research by showing that current embedding-based retrievers and LLM rerankers rely heavily on lexical signals, and by offering a concrete reranking strategy. The controlled normalization framework is a valuable analytical tool, and the case studies illustrate qualitative failures that are easy to understand. However, the significance is weakened by three problems: the headline retrieval gains are measured only under synthetic asymmetric normalization, the unnormalized HumanEval retrieval gain is zero, and the SWE-Bench-Lite result is a file-localization accuracy on 300 instances without significance testing. The paper does not provide machine-checked proofs or released code; its value rests on the empirical claims, which are currently only partially supported. With additional real-world evaluation and proper controls, the core ideas could be a solid contribution; as presented, the evidence is not sufficient for the strength of the abstract claims.","major_comments":[{"comment":"The normalization experiments are asymmetric: only the code documents are normalized, while the natural-language queries are left in their original form. Because the original query retains docstring words and identifier names, normalizing the positive document removes exactly the lexical overlap that the query depends on. The dramatic drops (e.g., 98.6% to 9.0% Recall@1 on MBPP) and the corresponding SACL gains may therefore be artifacts of this artificially created lexical mismatch, not evidence of a learned preference for surface features over semantics. Figure 4 supports this reading: SACL improves retrieval by increasing ROUGE-1 overlap between query and positive descriptions, i.e., by restoring the lexical signal that the experiment removed. To support the bias claim, the authors should include a symmetric control where both queries and documents are normalized, and should additionally evaluate on naturally sparse or poorly documented code rather than only synthetic normalization.","section":"§2.1–2.2, Tables 1–2"},{"comment":"The headline claim that SACL 'substantially improves code retrieval (e.g., by 12.8% / 9.4% / 7.0% Recall@1 on HumanEval / MBPP / SWE-Bench-Lite)' is misleading because the HumanEval and MBPP numbers are obtained under the full-normalization setting, not on original code. On unnormalized HumanEval retrieval, the reported gain is 0.0 (Table 2, Docstring row: 98.8 to 98.8), and no result for original code is reported. The abstract and introduction should clearly separate gains on synthetic normalized corpora from gains on naturally occurring code, and the paper should report the unnormalized baseline and SACL performance explicitly. Without this, a reader cannot judge whether SACL helps in realistic retrieval scenarios.","section":"Abstract and Table 2"},{"comment":"The SWE-Bench-Lite '7.0% Recall@1' improvement is not a retrieval Recall@1 comparable to the HumanEval/MBPP results; it is the file-localization accuracy of the Agentless pipeline (i.e., whether the localized files contain the ground-truth patch locations). This is an important distinction because the localization step is an LLM prompt, not a retrieval metric. Moreover, the result is based on only 300 instances, with no confidence intervals, variance estimates, or significance tests. The difference between 79.0% and 86.0% (or 70.0% to 78.0%) could be within run-to-run variation of the Agentless pipeline. The authors should report multiple runs or a significance test and should clearly label the metric as file-localization accuracy rather than Recall@1.","section":"Table 3 and §4.2"},{"comment":"The hyperparameter α in Equation (1) is selected to maximize Recall@k on the same test benchmarks (HumanEval and MBPP), as shown in Figure 5, and the final results use α=0.7 (Appendix A.1). No validation split or independent tuning procedure is described. Since the gains are strongly dependent on α (with peaks at 0.7 and lower performance for α=0 or α=1), the reported improvements may be optimistically biased by test-set tuning. The paper should describe how α was chosen (e.g., on a held-out validation set) or provide a sensitivity analysis that does not rely on the test set.","section":"§4.4, Figure 5, and §A.1"}],"minor_comments":[{"comment":"Please clarify that the HumanEval/MBPP numbers are under full normalization, and that the SWE-Bench-Lite number is file-localization accuracy, not a retrieval Recall@1 in the same sense.","section":"Abstract"},{"comment":"The row labels 'Docstring', 'Func Name', and 'All' are ambiguous. They should be defined explicitly (e.g., 'remove docstrings', 'remove docstrings + rename functions', 'remove docstrings + rename functions and variables'), and a row for the original unnormalized code should be added.","section":"Table 2"},{"comment":"The order of normalization types is inconsistent between the retrieval panels ('Base Docstr Var Fn Full') and the reranking panels ('Base Docstr Fn Var Full'), which makes it hard to compare the plots. Please use the same order in all panels and add axis labels.","section":"Figure 2"},{"comment":"Equation (1) introduces α but does not say how it should be set in practice; Appendix A.1 states α=0.7, but the paper should note whether this value transfers across datasets or is tuned per benchmark.","section":"Section 3.1"},{"comment":"The text emphasizes the 4.88% Pass@1 gain on HumanEval, but Table 4 shows the gain is zero under the Docstring and Func Name settings and appears only under 'All' normalization. Please state this explicitly so the reader understands the synthetic conditions under which the generation gains are obtained.","section":"Table 4"},{"comment":"The limitations paragraph is helpful, but it does not address the asymmetric normalization confound or the small scale of the SWE-Bench-Lite evaluation; please add those caveats to the limitations section.","section":"Conclusion and Limitations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a valuable analytical framework, but the central effectiveness claim is not established by the current experiments. The asymmetric normalization protocol is a confound that can be addressed with additional controls and real-world evaluation. I recommend major revision rather than outright rejection because the issues are fixable: add symmetric normalization and unnormalized retrieval baselines, report significance testing for SWE-Bench-Lite, and clarify how α is selected. If the authors cannot provide these, the paper would be better suited to rejection, as the headline gains would then be unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the normalization study is a genuinely useful diagnostic, but the paper oversells the fix. The controlled experiments in Section 2 are the real contribution; the SACL effectiveness claims in the abstract are not supported by the experiments as reported, and one table looks wrong.\n\nWhat is new and good: The authors systematically replace docstrings, function names, and variable names with placeholders while preserving code functionality, and measure the effect on both embedding retrievers and LLM rerankers. This is a clean way to quantify textual reliance, and the results are stark: GIST-large drops from 98.6% to 9.0% Recall@1 on MBPP under full normalization, and Table 1 shows that the ground-truth document's rank deteriorates far more when it alone is normalized (e.g., 87.18 to 288.27 for GIST-large) than when all documents are normalized. That is a real, reproducible-style diagnostic of bias toward well-documented code. I buy Discovery 1 and Discovery 2 as empirical findings.\n\nNow the soft spots, in proportion. First, the headline recalls are measured under synthetic normalization. The 12.8% and 9.4% gains in the abstract are from the 'All' rows in Table 2, where the baseline is already collapsed. On the docstring-removed setting (closest to naturally occurring code in this table), HumanEval shows 0.0 gain and MBPP shows 7.8. There is no fully unnormalized retrieval experiment, so 'substantially improves code retrieval' overstates what is shown. Second, the SWE-Bench-Lite file-localization gains (8.0 and 7.0 points) come from 300 instances with no significance testing; those are 24 and 21 instances respectively and could easily be noise. Third, Table 4 reports identical Pass@1 values for Qwen2.5-Coder-7B and Deepseek-coder-7b in every condition. Two different models will not produce identical numbers to two decimals across all settings; this looks like an experimental error and it undermines confidence in the generation tables. Fourth, the hyperparameter alpha is tuned on the same HumanEval/MBPP test sets via Figure 5, which is test-set tuning. Fifth, SACL is essentially document expansion with LLM-generated descriptions, and the paper does not cite or compare with prior document expansion work (e.g., doc2query), which weakens the novelty claim.\n\nOne point where I disagree with the stress-test note: the asymmetric normalization (documents normalized, queries untouched) is a reasonable protocol for isolating document-side textual reliance. The collapse is genuine evidence of lexical dependence, not merely an embedding-space artifact. However, Figure 4 confirms that SACL works by restoring lexical overlap between query and document, so it compensates for the bias rather than removing it.\n\nRecommendation: send to peer review, but with a clear expectation of major revision. The bias analysis deserves an audience, and the method is a reasonable heuristic, but the claims need to be reframed, the table error fixed, and either a held-out alpha or results on unnormalized code reported. This is not ready as-is.","headline":"The normalization analysis is a solid diagnostic, but the SACL effectiveness claims are supported only under synthetic normalization, and Table 4 has a likely error.","tokens_in":13573,"tokens_out":4258,"would_cite":false,"duration_ms":41057,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that code retrievers rely on surface textual cues rather than code semantics, and that augmenting code with LLM-generated descriptions substantially improves retrieval and downstream code generation.","keywords":["code retrieval","textual bias","semantic augmentation","reranking","fault localization","retrieval-augmented code generation","normalization analysis","LLM-generated descriptions"],"falsifier":"Normalize the queries as well as the documents (or evaluate on naturally under-documented code with docstrings and identifier names removed before indexing) and rerun the retrieval experiments; if Recall@1 no longer collapses under full normalization, or if irrelevant well-documented documents are no longer preferred, the claimed bias is an artifact of the asymmetric setup rather than a property of current code retrievers.","tokens_in":12511,"feed_emoji":"🔍","tokens_out":11476,"duration_ms":101448,"temperature":0.7,"pith_summary":"This paper aims to establish that current code retrievers are biased toward surface-level textual features—docstrings and identifier names—rather than code functionality, and that this bias can be partially corrected by augmenting code with semantic descriptions. The authors demonstrate the bias through a normalization analysis that replaces docstrings, function names, and variable names with placeholders while preserving behavior, causing recall to collapse (e.g., GIST-large Recall@1 on MBPP drops from 98.6% to 9.0%). They then propose SACL, which reranks retrieved code using a weighted combination of code–query similarity and similarity between the query and an LLM-generated description of each code snippet, and augments repository tree structures with file descriptions for localization. On HumanEval, MBPP, and SWE-Bench-Lite, SACL reports substantial retrieval and localization gains that also improve code generation Pass@1. The work matters because retrieval quality is a known bottleneck in retrieval-augmented code generation, and the proposed fix requires no retraining of the retriever.","feed_headline":"SACL reranking lifts code retrieval by up to 15.2%","feed_subtitle":"Adding LLM-written descriptions to code and repo trees offsets retrievers' reliance on names and docstrings.","key_machinery":"The load-bearing mechanism is the normalization-based analysis framework combined with semantic augmentation. Normalization systematically strips textual cues (docstrings, function names, variable names) from code while preserving its functionality, which isolates how much retrievers depend on surface text. Semantic augmentation then generates a concise natural-language description of each retrieved code snippet or repository file using an 8B-parameter LLM, and reranks by the weighted score $\\text{Score}_{\\text{final}} = (1-\\alpha)\\,\\text{Score}_{\\text{code}} + \\alpha\\,\\text{Score}_{\\text{desc}}$ with $\\alpha=0.7$, turning text-to-code comparison into text-to-text comparison. For repository-level localization, the same idea is applied to the file tree, where each file is annotated with a generated description of its purpose and relationships, supplying the LLM with semantic context that filenames alone lack.","core_discovery":"The paper's central discovery is that code retrievers, despite being trained on code, predominantly match queries to documents by textual overlap rather than by functional equivalence. Using normalization—replacing docstrings, function names, and variable names with placeholders—the paper finds that Recall@1 degrades sharply for both embedding-based retrievers and LLM-based rerankers, and that the degradation is worst when only the ground-truth document is normalized while the rest of the corpus remains well-documented. That asymmetry shows a bias toward well-documented code even when the documentation is irrelevant. To counter this, the paper proposes semantic-augmented code reranking and semantic-augmented in-context localization: generated descriptions of code and repository files are combined with original retrieval scores ($\\alpha=0.7$ weight on description similarity), and augmented repo trees are fed to an LLM for file localization. The reported improvements are up to 12.8% Recall@1 on HumanEval, 9.4% on MBPP under full normalization, and up to 8.0% file-localization accuracy on SWE-Bench-Lite, with corresponding gains in code generation.","pith_inferences":["Because the normalization experiments leave queries untouched, the measured bias may be partly an artifact of asymmetric normalization; a cleaner test would normalize queries and documents together to see whether the bias claim survives.","Since descriptions are generated by an 8B model, the description-query scores could be distilled into the base retriever, making semantic awareness available at inference time without an LLM call.","The same description-augmentation idea could transfer to other structured inputs, such as API schemas or configuration files, where LLMs must ground decisions in repository context.","The gains are measured under artificial normalization; on naturally documented codebases the practical margin may be smaller, so the real-world impact depends on how under-documented common corpora actually are."],"forward_implications":["If the bias claim is right, retrieval-augmented code generation from real-world, sparsely documented codebases will systematically retrieve the wrong code, and SACL's description-based reranking should recover much of that loss without retraining or changing the retriever.","The finding that benefits are largest on MBPP (low lexical overlap) and under full normalization implies that the method's value scales with how poorly the corpus is documented.","Because SACL improves file localization within the Agentless pipeline, repo-level fault localization should improve even when the underlying LLM is unchanged, raising issue-resolution rates on benchmarks like SWE-Bench-Lite.","The code-generation pass-through (up to 4.88% Pass@1 on HumanEval under full normalization) suggests that fixing retrieval bias directly improves downstream generation quality.","Peak performance at $\\alpha=0.7$ rather than at pure code retrieval ($\\alpha=0$) or pure description retrieval ($\\alpha=1$) indicates that code and description signals carry complementary information, so hybrid reranking is the robust choice."],"supporting_citations":[{"why":"This reference supplies the HumanEval benchmark used for script-level retrieval and generation evaluation.","marker":"Chen et al. 2021"},{"why":"This reference supplies the MBPP benchmark, which the paper uses to show larger bias and larger SACL gains due to lower lexical overlap.","marker":"Austin et al. 2021"},{"why":"This reference supplies SWE-Bench-Lite, the repo-level issue-solving benchmark used for localization and generation experiments.","marker":"Jimenez et al. 2023"},{"why":"This reference provides GIST-large, the embedding retriever used as the base model for the analysis and SACL reranking experiments.","marker":"Solatorio 2024"},{"why":"This reference provides the Agentless pipeline and its file localization step, which SACL augments with file descriptions.","marker":"Xia et al. 2024"},{"why":"This reference provides CodeSearchNet, cited as the source of docstring-function contrastive training pairs that may explain the textual bias.","marker":"Husain et al. 2019"},{"why":"This reference provides CodeRAG-Bench and the Recall@k evaluation protocol used for retrieval performance.","marker":"Wang et al. 2025"},{"why":"This reference provides ReACC, an earlier retrieval-augmented code completion framework whose gains motivate the RACG setting.","marker":"Lu et al. 2022"}],"fun_headline_variants":["Code retrievers are text-biased; SACL adds semantics to fix it","Semantic reranking fixes code retrievers' textual bias (12.8% gain)","SACL: semantic-augmented reranking lifts code retrieval by 12.8%","Code retrieval favours documentation over code; SACL rebalances with semantics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bias analysis assumes that normalizing only the code documents, not the queries, is a fair way to measure what retrievers rely on; if that asymmetry itself creates the drop, the bias claim is overstated.","fun_headline_variants_meta":{"raw":{"variants":["Code retrievers are text-biased; SACL adds semantics to fix it","Semantic reranking fixes code retrievers' textual bias (12.8% gain)","SACL: semantic-augmented reranking lifts code retrieval by 12.8%","Code retrieval favours documentation over code; SACL rebalances with semantics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000869,"raw_usage":{"total_tokens":3778,"prompt_tokens":971,"completion_tokens":2807,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2719}},"tokens_in":587,"tokens_out":2807,"duration_ms":23202,"temperature":1.0,"reasoning_tokens":2719,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:56:51.600471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Normalize the queries as well as the documents (or evaluate on naturally under-documented code with docstrings and identifier names removed before indexing) and rerun the retrieval experiments; if Recall@1 no longer collapses under full normalization, or if irrelevant well-documented documents are no longer preferred, the claimed bias is an artifact of the asymmetric setup rather than a property of current code retrievers.","supporting_citations":[],"review_version":1}