Pith. sign in

REVIEW 2 major objections 2 minor 154 cited by

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

T0 review · 2 major / 2 minor · reviewed 2026-05-12 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Releasing the CodeSearchNet Corpus of 6 million functions and a challenge with 99 annotated queries enables evaluation of semantic code search across six languages.

desk verdict This is a data release paper that gives the field a bigger multi-language code corpus and challenge set, but the lack of validation numbers on the auto-generated queries and expert labels is a real gap. read the letter →

arxiv 1909.09436 v3 pith:4IPOKMJK submitted 2019-09-20 cs.LG cs.IRcs.SEstat.ML

Hamel Husain , Ho-Hsiang Wu , Tiferet Gazit , Miltiadis Allamanis , Marc Brockschmidt This is my paper · ORCID
classification cs.LGcs.IRcs.SEstat.ML
keywords semanticcodesearchretrievalnaturallanguagequeriescorpusbenchmarkprogramminglanguagesinformation
checked against Cost.FunctionalEquationFoundation.HierarchyEmergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a new resource for semantic code search by collecting millions of functions from open-source projects in Go, Java, JavaScript, PHP, Python, and Ruby. It pairs many of those functions with automatically generated natural-language text scraped from documentation and adds a set of expert-labeled queries to create a concrete benchmark. This setup lets researchers train models that retrieve code snippets matching vague natural-language descriptions and compare results on the same test cases. Simple baselines are included to show initial performance and to lower the barrier for new participants. The authors intend the release to support ongoing competitions and future expansion to additional queries and languages.

What carries the argument

The CodeSearchNet Corpus of functions paired with scraped documentation text, plus the expert-annotated query set used to score retrieval relevance.

What would settle it

An independent check finding low agreement among experts on the same query-code pairs or showing that the scraped documentation text rarely matches how developers actually phrase searches would undermine the benchmark.

Watch

Extended reading notes

Core claim

By releasing the CodeSearchNet Corpus containing approximately 6 million functions across six programming languages together with automatically generated natural-language descriptions for two million of them, and by creating the CodeSearchNet Challenge of 99 natural-language queries annotated with roughly four thousand expert relevance judgments, the work supplies a standardized corpus and evaluation set for semantic code search.

Load-bearing premise

Mechanically scraped and preprocessed function documentation yields sufficiently accurate and representative natural-language queries and the expert annotations are consistent and unbiased measures of relevance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims to release the CodeSearchNet Corpus containing about 6 million functions from six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby), with automatically generated query-like natural language descriptions for 2 million functions obtained by mechanically scraping and preprocessing associated documentation. It also introduces the CodeSearchNet Challenge consisting of 99 natural language queries with about 4k expert relevance annotations of likely results from the corpus, describes the methodology for corpus construction and labeling, and provides a number of simple baseline solutions for semantic code search, with the aim of enabling evaluation and tracking of progress via a future competition and leaderboard.

Significance. If the generated queries and expert annotations prove reliable and representative, the release of this large-scale multi-language corpus and annotated challenge would be a significant contribution to semantic code search research. It provides a standardized benchmark at a scale (6M functions, 99 queries, 4k annotations) that could facilitate model development and comparison in bridging natural language and code, similar to other information retrieval benchmarks, and the inclusion of baselines supports immediate usability.

major comments (2)
  1. [Methodology for corpus and query generation] The methodology section describing corpus construction and query generation states that the 2 million query-like descriptions are obtained by mechanically scraping and preprocessing function documentation, but provides no quantitative validation (e.g., accuracy against human queries, diversity metrics, or fidelity checks) to support that these yield sufficiently accurate and representative natural language queries for the challenge.
  2. [Challenge construction and expert annotation process] The section on the CodeSearchNet Challenge and expert labels describes the 99 queries and ~4k annotations but reports no inter-annotator agreement statistics, annotation consistency measures, or bias validation, which is load-bearing for the central claim that the challenge enables reliable evaluation of semantic code search progress.
minor comments (2)
  1. [Abstract and conclusion] The abstract and conclusion mention plans to host a competition and leaderboard but do not specify the platform, timeline, or evaluation protocol details.
  2. [Baseline solutions] The baselines are introduced as 'simple' solutions; adding implementation details or pseudocode would improve reproducibility without altering the core contribution.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review. We appreciate the opportunity to clarify the distinction between the corpus construction and the challenge, and to address the points on validation and annotation reliability.

read point-by-point responses

  1. Referee: [Methodology for corpus and query generation] The methodology section describing corpus construction and query generation states that the 2 million query-like descriptions are obtained by mechanically scraping and preprocessing function documentation, but provides no quantitative validation (e.g., accuracy against human queries, diversity metrics, or fidelity checks) to support that these yield sufficiently accurate and representative natural language queries for the challenge.

    Authors: We thank the referee for raising this. We wish to clarify an important distinction: the 2 million automatically generated descriptions are part of the released CodeSearchNet Corpus and serve as weak supervision for training models; they are obtained by scraping and preprocessing existing function documentation from open-source repositories. These descriptions are not the queries used in the CodeSearchNet Challenge. The challenge instead uses a separate set of 99 expert-written natural language queries, each paired with expert relevance annotations over candidate functions from the corpus. Because the corpus descriptions are mechanically derived from documentation that already exists in the source code, their representativeness is bounded by the quality of that documentation, which we describe in the methodology section. We did not include additional quantitative validation (such as diversity metrics or fidelity checks against human queries) because the primary contribution is the release of the large-scale resource itself rather than a claim that the derived descriptions perfectly match human queries. We can, however, add basic descriptive statistics on the generated descriptions (e.g., length distributions and language-specific characteristics) in a revision to improve transparency. revision: partial

  2. Referee: [Challenge construction and expert annotation process] The section on the CodeSearchNet Challenge and expert labels describes the 99 queries and ~4k annotations but reports no inter-annotator agreement statistics, annotation consistency measures, or bias validation, which is load-bearing for the central claim that the challenge enables reliable evaluation of semantic code search progress.

    Authors: We acknowledge that reporting inter-annotator agreement would strengthen the perceived reliability of the annotations. The 99 queries were authored by the paper authors and a small group of domain experts, and the approximately 4k relevance judgments were performed by the same expert annotators following a written annotation protocol that specified relevance criteria, handling of edge cases, and tie-breaking rules. Due to the expert-only nature of the task and resource limitations, we collected only a single annotation per query–function pair and therefore do not have the data required to compute standard IAA metrics such as Cohen’s kappa or Fleiss’ kappa. We attempted to mitigate bias and inconsistency through careful query curation, pilot annotation rounds, and discussion of difficult cases among annotators. We agree this is a limitation of the current release. In the revised manuscript we will expand the description of the annotation protocol, explicitly note the absence of multiple annotations, and discuss the implications for benchmark reliability. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: dataset release paper with no derivations or self-referential reductions

full rationale

The paper's core contribution is the release of the CodeSearchNet Corpus (6M functions across 6 languages) and Challenge (99 NL queries with ~4k expert annotations), obtained by scraping documentation for 2M functions and expert labeling. No equations, fitted parameters, predictions, or derivation chains appear in the abstract or described methodology. Claims about enabling evaluation of semantic code search progress rest on the data collection process itself rather than any mathematical reduction to prior inputs. No self-citations or ansatzes are invoked as load-bearing steps. This matches the expected non-circular outcome for a pure data/benchmark paper.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is an empirical data-release paper with no mathematical derivations. It introduces no free parameters, axioms, or invented entities; all contributions rest on the described collection and annotation procedures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeSearchNet Challenge: Evaluating the State of Semantic Code Search." pith.science (2026). https://pith.science/paper/4IPOKMJK

@misc{pith2026190909436,
  author       = {Pith},
  title        = {Pith review of: CodeSearchNet Challenge: Evaluating the State of Semantic Code Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4IPOKMJK}},
  note         = {Machine review of arXiv:1909.09436}
}
read the original abstract

Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 154 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 422 citations worldwide. See all 154 Pith citations

  1. Architecture Determines Observability of Transformers

    cs.LG 2026-04 unverdicted novelty 8.0 of 10

    Certain transformer architectures lose internal linear signals for decision quality during training, making observability an architecture-dependent property rather than a universal one.

  2. Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Massive activations in hybrid linear attention LLMs form pre-attention spikes before full attention layers and inter-spike plateaus that grow with full attention density.

  3. SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References

    cs.SE 2026-08 conditional novelty 7.0 of 10

    SONAR evaluates code summaries without reference summaries by regenerating code from the summary, and shows that only correctness and abstraction strongly predict LLM performance on downstream software engineering tasks.

  4. Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

    cs.LG 2026-08 conditional novelty 7.0 of 10

    SALT shows that a subspace-aligned domain centroid plus a rank-1 or rank-2 user residual can recover or exceed high-rank LoRA accuracy while cutting the per-adapter swap footprint by up to 16x.

  5. Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.

  6. RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    RESOURCE2SKILL converts multimodal human resources into a hierarchical Skill Wiki of executable agent skills, reporting +11.9 percentage point average gains over no-skill baselines across seven authoring domains.

  7. Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    Introduces SolidityBench benchmark and SolidityScore metric for repository-level Solidity code generation, finding supervised fine-tuning outperforms prompting, CoT, ICL, and RAG methods on evaluated LLMs.

  8. AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Introduces AMALIA-VL, the first open-source instruction-tuned LVLM for European Portuguese, using a high-resolution vision encoder, pt-PT language model, learned connector, and three-stage training on a custom data mix.

  9. Signature filtering: a lightweight enhancement for statistical watermark detection in large language models

    cs.LG 2026-06 conditional novelty 7.0 of 10

    Signature filtering learns unreliable tokens with MILP and removes them at detection time, raising true positive rates from 8-31% to 78-99% across Kgw, Sweet, Unigram, and Exp watermarks on multiple corpora and LLMs w...

  10. Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?

    cs.IR 2026-06 conditional novelty 7.0 of 10

    On heterogeneous document collections, only query expansion and a newly introduced per-source calibrated corrector (SSCC) deliver reliable gains beyond a strong cross-encoder reranker; other common retrieval enhanceme...

  11. HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    HybridCodeAuthorship is a new benchmark dataset of interleaved human-AI Python code that shows existing detection algorithms reach at most 0.48 F1 at chunk level and 0.56 F1 at line level.

  12. CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding

    cs.IR 2026-06 conditional novelty 7.0 of 10

    Existing embedding models drop sharply from traditional code search to requirement-driven repository retrieval, while in-domain PR fine-tuning substantially improves Level-2/3 performance on CORE-Bench.

  13. SWE-Explore: Benchmarking How Coding Agents Explore Repositories

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    SWE-Explore is a new benchmark evaluating repository exploration by coding agents on 848 issues across 203 repositories, using line-level ground truth from successful agent trajectories and showing agentic methods out...

  14. VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    VISTA is a new benchmark for end-to-end visual spec-to-web-app generation by LLM agents, featuring five prompt conditions, manual UI annotations, multi-metric evaluation, and results on four agent systems showing part...

  15. Test-Time Speculation

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    TTS adapts speculator models online via target model verifications to improve acceptance lengths by up to 72% over prior methods, with gains increasing for longer generations.

  16. Unified Communication Compression Beyond Global Error Bounds for Distributed Nonconvex Optimization

    math.OC 2026-04 unverdicted novelty 7.0 of 10

    A unified compression algorithm for distributed nonconvex optimization achieves O(1/sqrt(T)) convergence for locally-bounded compressors, matching centralized 1-bit methods, with an improved O(1/T^{2/3}) rate after on...

  17. Can You Trust the Vectors in Your Vector Database? Black-Hole Attack from Embedding Space Defects

    cs.CR 2026-04 unverdicted novelty 7.0 of 10

    Injecting a few vectors near the embedding-space centroid can make them appear in top-k results for up to 94.4% of queries via centrality-driven hubness.

  18. Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech

    cs.CL 2026-03 accept novelty 7.0 of 10

    Progressive LLM-initialized training plus teacher-student distillation yields omnilingual sentence embeddings that halve FLORES error and cut BIBLE error 15× while supporting speech and zero-shot Spectrum transfer.

  19. When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Aurora unifies speculative decoder training and serving via asynchronous RL on inference traces, delivering 1.5x day-0 speedup on frontier models and 1.25x adaptation gains on distribution shifts.

  20. From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering

    cs.SE 2025-07 accept novelty 7.0 of 10

    Developers do not feed raw requirements into LLMs; they manually decompose requirements into programming tasks and enrich prompts with design, architecture, and code context before generating code.

  21. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner

    cs.CL 2025-06 conditional novelty 7.0 of 10

    SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.

  22. Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Models keep verbatim code retrieval stable across positions but lose the ability to reason about centrally placed code, and standard benchmarks understate this because they allow pattern matching.

  23. Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding

    cs.SE 2025-05 conditional novelty 7.0 of 10

    An interactive code generation pipeline that uses editable inline comments as a feedback channel improves pass@1 on HumanEval/MBPP and user task success in a small study.

  24. OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research

    cs.SE 2025-04 accept novelty 7.0 of 10

    OpenClassGen supplies 324,843 real-world Python classes with self-contained skeletons and static metrics to support LLM class generation research and evaluation.

  25. SoK: Where to Fuzz? Assessing Target Selection Methods in Directed Fuzzing

    cs.SE 2025-02 accept novelty 7.0 of 10

    Simple code metrics, especially Leopard's vulnerability scores, retrieve crash-relevant functions more accurately than sanitizer heuristics, recently-changed code, or deep learning models on a corpus of 1,621 real crashes.

  26. Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot

    cs.SE 2025-01 conditional novelty 7.0 of 10

    In an empirical study of 437 code snippets, Bing CoPilot usually provided at least one link with code similar to its output, while Gemini rarely did, exposing a real provenance gap.

  27. InCoder: A Generative Model for Code Infilling and Synthesis

    cs.SE 2022-04 unverdicted novelty 7.0 of 10

    InCoder is the first generative model to directly perform zero-shot code infilling via bidirectional context from a masked-then-appended training scheme, matching left-to-right models on synthesis while improving on t...

  28. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

    cs.SE 2020-09 conditional novelty 7.0 of 10

    CodeBLEU improves correlation with human programmer scores on code synthesis tasks by adding syntactic AST matching and semantic data-flow matching to the standard BLEU n-gram approach.

  29. GraphCodeBERT: Pre-training Code Representations with Data Flow

    cs.SE 2020-09 accept novelty 7.0 of 10

    GraphCodeBERT uses data flow graphs in pre-training to capture semantic code structure and reaches state-of-the-art results on code search, clone detection, translation, and refinement.

  30. From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs

    cs.CL 2026-07 conditional novelty 6.5 of 10

    On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.

  31. PatchOptic for Shared-State LLM Workflows with Projected Views and Verified Structured Updates

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Projected reads plus verified structured patches and path footprints reduce leakage and illegal commits in shared-state LLM workflows on a 46-case benchmark.

  32. Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark

    cs.IR 2026-07 accept novelty 6.5 of 10

    A multi-domain corpus of 5,264 scientific GitHub repos plus two IR benchmarks (219 expert queries; 117,950 snippets / 119,720 queries) shows large domain- and documentation-driven gaps in scientific code search.

  33. Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

    cs.IR 2026-07 conditional novelty 6.0 of 10

    A 427-sample workflow-grounded benchmark finds complementary strengths across lexical, structural, and embedding retrieval for coding-agent context acquisition, with large residual miss rates and a selective-abstentio...

  34. Enhancing Code Understanding for Impact Analysis by Combining Transformers and Program Dependence Graphs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Athena beats LSI-based impact analysis by ~10% (mRR/mAP/HIT@10) by propagating transformer code embeddings over call and class-member dependence graphs.

  35. SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A quality-aware code-search reranking framework with LLM-synthesized quality variants and a hierarchical ranking loss beats relevance-only training on a new QPA/MCA benchmark.

  36. Token Reduction Is Not Cost Reduction

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Token reduction of tool output does not reliably lower billed coding-agent cost; prompt-cache traffic dominates, and aggressive compression can also destroy task success.

  37. Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

    cs.SE 2026-07 conditional novelty 6.0 of 10

    QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.

  38. The Decomposition Is the Fingerprint: Per-Component Identity for Agent Skills

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    A per-component SimHash fingerprint supplies structural identity for AI agent skills, recovering family membership under paraphrase and refactoring with AUC 0.974 while localizing changes.

  39. Securing Code Understanding: Detecting Natural Backdoor Vulnerability in Code Language Models

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    Natural backdoors are prevalent in CodeLMs; the authors propose ScanNBT to detect them after analyzing differences from injected backdoors, transferability, and causes.

  40. UniRTL: Unifying Code and Graph for Robust RTL Representation Learning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    UniRTL unifies RTL code and CDFG through mutual masked modeling and hierarchical training with a graph-aware tokenizer, outperforming prior single-modality methods on performance prediction and code retrieval.

  41. Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    Code-QA-Bench uses an answer-first pipeline and three-condition experiments to generate 628 tasks across 10 Python repositories and quantify that code access drives most performance gains while documentation adds only...

  42. Strong Teacher Not Needed? On Distillation in LLM Pretraining

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Even small or undertrained teachers improve larger LLM students via distillation with tuned loss mixing, while stronger teachers can saturate or reverse gains and distillation aids generalization more than in-domain fit.

  43. Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fix

    cs.IR 2026-05 unverdicted novelty 6.0 of 10

    A q-log odds variant of BM25 raises NDCG@10 by 89% relative on CodeSearchNet Go under fixed generic tokenization while recovering standard BM25 at q=1.

  44. XSearch: Explainable Code Search via Concept-to-Code Alignment

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    XSearch achieves explainable code search by breaking queries into functional concepts and matching them directly to code statements, delivering large gains on out-of-distribution benchmarks.

  45. Don't Use a Cannon to Kill a Fly: Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations

    cs.SE 2025-11 conditional novelty 6.0 of 10

    AdaLoRA-L restricts edits to API-specific layers and raises specificity by 33–836% (relative) on a new 3,000+ instance benchmark while staying close to AdaLoRA's effectiveness.

  46. A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?

    cs.SE 2025-11 unverdicted novelty 6.0 of 10

    Student models distilled from code language models often fail to deeply mimic teachers, showing up to 62% behavioral discrepancies and 285% worse drops under attacks that accuracy metrics miss.

  47. Understanding Robustness of Model Editing in Code LLMs

    cs.SE 2025-11 unverdicted novelty 6.0 of 10

    A controlled benchmark on 2040 problems reveals poor generalization and high interference in model editing for API updates in code LLMs, with many successes being workarounds rather than true migrations.

  48. ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new open-source benchmark evaluates LLM-generated end-to-end ML pipelines from Kaggle competition descriptions translated into 13 languages, with 6 private tasks to limit data leakage.

  49. PseudoBridge: Pseudo Code as the Bridge for Better Semantic and Logic Alignment in Code Retrieval

    cs.SE 2025-09 unverdicted novelty 6.0 of 10

    PseudoBridge uses LLM-synthesized pseudo-code to bridge NL semantics and PL logic plus logic-invariant style augmentation to boost robustness and generalization in code retrieval.

  50. Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter

    cs.CY 2025-09 conditional novelty 6.0 of 10

    After ChatGPT's launch, Stack Overflow posts became longer and shifted toward medium-difficulty questions, suggesting users turn to the crowd for more complex problems.

  51. Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation

    cs.IR 2025-09 reject novelty 6.0 of 10

    SolveRank trains a contrastive retriever on LLM-generated logically equivalent problem variants and reports improved retrieval and code generation, though the retrieval evaluation is circular.

  52. Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity

    cs.SE 2025-08 conditional novelty 6.0 of 10

    Across 500k Python/Java samples, AI-generated functions are shorter, simpler, and trigger more security findings, while human functions carry more complexity and maintainability warnings.

  53. LinkAnchor: An Autonomous LLM-Based Agent for Issue-to-Commit Link Recovery

    cs.SE 2025-08 unverdicted novelty 6.0 of 10

    LinkAnchor is an LLM-based autonomous agent with lazy-access architecture for recovering issue-to-commit links in software repositories.

  54. How Quantization Impacts Privacy Risk on LLMs for Code?

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Quantizing code LLMs reduces membership inference effectiveness, with 4-bit compression giving larger privacy and performance drops than 8-bit.

  55. Fine-Tuning Code Language Models to Detect Cross-Language Bugs

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Fine-tuning 13 CodeLMs on a constructed CLB dataset with nine interaction types improves detection, with UniXcoder-base reaching F1 0.7407 and small models outperforming large ones.

  56. Structural Code Search using Natural Language Queries

    cs.SE 2025-07 conditional novelty 6.0 of 10

    An LLM-based system translates English code search questions into Semgrep and GQL queries, reaching 55-70% precision/recall on a new synthetic 400-query Java benchmark.

  57. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval

    cs.AI 2025-06 conditional novelty 6.0 of 10

    jina-embeddings-v4 is a multimodal embedding model built on Qwen2.5-VL with LoRA adapters and dual single/multi-vector outputs, and it introduces the Jina-VDR benchmark for visually rich retrieval.

  58. Retrieval-Augmented Code Review Comment Generation

    cs.SE 2025-06 conditional novelty 6.0 of 10

    Retrieval-augmented conditioning on similar code-review pairs improves review comment generation over generation-only baselines, with larger gains for low-frequency tokens, though improvements over retrieval-only base...

  59. CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A code retrieval benchmark with quality labels for correctness, efficiency, security, and maintainability shows most retrievers fail to prefer high-quality code, and quality-aware fine-tuning improves this without mea...

  60. MGS3: A Multi-Granularity Self-Supervised Code Search Framework

    cs.SE 2025-05 conditional novelty 6.0 of 10

    MGS3 trains code search models on multi-granularity comment-code alignments, improving retrieval across function, block, and statement-level benchmarks.

See all 154 Pith citations

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · cited by 154 Pith papers (see all)

  1. [1]

    Miltiadis Allamanis. 2018. The Adverse Effects of Code Duplication in Machine Learning Models of Code. arXiv preprint arXiv:1812.06469 (2018)

    work page arXiv 2018

  2. [2]

    Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 81

    work page 2018

  3. [3]

    Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In Proceedings of the International Conference on Machine Learning (ICML)

    work page 2016

  4. [4]

    Uri Alon, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018)

    work page Pith review arXiv 2018

  5. [5]

    Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275 (2017)

    work page Pith review arXiv 2017

  6. [6]

    Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra

    work page

  7. [7]

    arXiv preprint arXiv:1905.03813 (2019)

    When Deep Learning Met Code Search. arXiv preprint arXiv:1905.03813 (2019)

    work page arXiv 1905

  8. [8]

    Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Ben- gio. 2014. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. Syntax, Semantics and Structure in Statistical Translation (2014)

    work page 2014

Show all 26 references
  1. [9]

    Within” computes the NDCG only on the functions within the human-annotated examples. “All

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv CodeSearchNet Encoder CodeSearchNet Challenge– NDCG Within CodeSearchNet Challenge– NDCG All Text Code Go Java JS PHP ...

    2018 arXiv

  2. [10]

    Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured Neural Summarization. arXiv preprint arXiv:1811.01824 (2018)

    2018

  3. [11]

    Philip Gage. 1994. A new algorithm for data compression. The C Users Journal 12, 2 (1994), 23–38

    1994

  4. [12]

    Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) . IEEE, 933–944

    2018

  5. [13]

    Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API Learning. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE)

    2016

  6. [14]

    Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems . 10073–10083

    2018

  7. [15]

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Map- ping language to code in programmatic context. arXiv preprint arXiv:1808.09588 (2018)

    2018

  8. [16]

    Yoon Kim. 2014. Convolutional neural networks for sentence classification.arXiv preprint arXiv:1408.5882 (2014)

    2014

  9. [17]

    Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. In International Conference on Language Resources and Evaluation

    2018

  10. [18]

    Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent Predictor Networks for Code Generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2016

  11. [19]

    Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: a map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 84

    2017

  12. [20]

    Manning, Prabhakar Raghavan, and Hinrich Schütze

    Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Intro- duction to Information Retrieval . Cambridge University Press

    2008

  13. [21]

    Bhaskar Mitra, Nick Craswell, et al. 2018. An introduction to neural information retrieval. Foundations and Trends® in Information Retrieval 13, 1 (2018), 1–126

    2018

  14. [22]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2016

  15. [23]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems . 5998–6008

    2017

  16. [24]

    Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. 2019. CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning. (2019)

    2019

  17. [25]

    Ziyu Yao, Daniel S Weld, Wei-Peng Chen, and Huan Sun. 2018. StaQC: A Sys- tematically Mined Question-Code Dataset from Stack Overflow. InProceedings of the 2018 World Wide Web Conference on World Wide Web . International World Wide Web Conferences Steering Committee, 1693–1703

    2018

  18. [26]

    Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General- Purpose Code Generation. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2017

Pith tools

Reviewed May 12, 2026 · model on record in the stance chip above.