Pith. sign in

REVIEW 2 major objections 2 minor 91 cited by

GraphCodeBERT: Pre-training Code Representations with Data Flow

T0 review · 2 major / 2 minor · reviewed 2026-05-15 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GraphCodeBERT improves code understanding by pre-training on data flow edges that track where variable values come from.

desk verdict GraphCodeBERT gets consistent gains on four code tasks by swapping data-flow edges for ASTs plus two new pre-training objectives. read the letter →

arxiv 2009.08366 v4 pith:2XS2L4VX submitted 2020-09-17 cs.SE cs.CL

Daya Guo , Shuo Ren , Shuai Lu , Zhangyin Feng , Duyu Tang , Shujie Liu , Long Zhou , Nan Duan
show 10 more authors
Alexey Svyatkovskiy Shengyu Fu Michele Tufano Shao Kun Deng Colin Clement Dawn Drain Neel Sundaresan Jian Yin Daxin Jiang Ming Zhou
This is my paper · ORCID
classification cs.SEcs.CL
keywords pre-trainedmodelscoderepresentationsdataflowprogramminglanguagessearchclonedetectiontranslationrefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing code models treat programs as token sequences and miss the relations that define their meaning. This work replaces deep syntax trees with a flatter data-flow graph that records definition-use links between variables. Two new pre-training tasks teach the model to predict those links and to align token representations with the graph nodes. The resulting Transformer, equipped with graph-guided attention, reaches state-of-the-art accuracy on code search, clone detection, translation, and refinement.

What carries the argument

The data-flow graph, which links variables by their definition-use relations, together with graph-guided masked attention that lets the Transformer attend along those edges.

What would settle it

A version of the model trained without any data-flow edges would match or exceed the full GraphCodeBERT on the four evaluation tasks.

Watch

Extended reading notes

Core claim

GraphCodeBERT augments masked language modeling with edge prediction on the data-flow graph and cross-modal alignment between source code and the graph. The data-flow graph encodes semantic relations of the form 'where-the-value-comes-from' without the deep nesting of an abstract syntax tree. These structure-aware objectives are realized through an efficient graph-guided masked attention mechanism inside a Transformer, yielding measurable gains on four downstream code tasks.

Load-bearing premise

Data-flow edges supply enough semantic structure to improve code understanding without needing the full syntactic hierarchy of an abstract syntax tree.

Editorial extensions

If this is right

  • Code models can capture semantic relations more efficiently by using flat data-flow graphs rather than deep parse trees.
  • Adding explicit structure-prediction and alignment objectives during pre-training produces measurable gains on search, detection, and repair tasks.
  • The graph-guided attention mechanism allows a standard Transformer to incorporate code structure at modest extra cost.
  • State-of-the-art results on four distinct tasks indicate that semantic structure transfers across code understanding problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same data-flow pre-training could be applied to languages beyond those tested to test whether the semantic edges are language-agnostic.
  • Hybrid models that combine data-flow edges with selected AST subtrees might further improve performance on tasks that require deep syntactic awareness.
  • Downstream tools such as automated program repair or code summarization may benefit from the richer variable-relation representations learned here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces GraphCodeBERT, a Transformer-based pre-trained model for code that replaces syntactic AST structure with semantic data-flow edges encoding 'where-the-value-comes-from' relations between variables. It augments standard masked language modeling with two new structure-aware objectives (edge prediction and code-structure alignment) and implements them via graph-guided masked attention. The model is evaluated on code search, clone detection, code translation, and code refinement, where it reports state-of-the-art results and shows a preference for structure-level attention heads.

Significance. If the empirical gains are reproducible, the work demonstrates that a lightweight semantic graph (data flow) can deliver measurable improvements over token-only baselines and over deeper syntactic hierarchies while remaining computationally efficient. The explicit attention analysis and the two new pre-training tasks provide concrete, falsifiable evidence that structure-aware objectives transfer to downstream code tasks.

major comments (2)
  1. [§4] §4 (Experimental Setup): the paper must report the exact pre-training corpus size, vocabulary construction, and whether all baselines were re-trained on identical data; without these details the SOTA claim on the four tasks cannot be verified as arising from the proposed structure components rather than data differences.
  2. [§3.3] §3.3 (Graph-guided Masked Attention): the description of how data-flow edges are extracted from source code (e.g., via static analysis or heuristic rules) is insufficiently precise; a concrete algorithm or pseudocode is needed to ensure the structure is reproducible and not post-hoc tuned to the downstream tasks.
minor comments (2)
  1. [Abstract] Abstract: the sentence 'can improve GraphCodeBERT and achieves state-of-the-art' is grammatically awkward and should be rephrased to clarify that the added components improve upon prior models.
  2. [§3.1] Figure 1 or §3.1: the visualization of data-flow edges versus AST would benefit from an explicit side-by-side example on the same code snippet to illustrate the claimed reduction in hierarchy depth.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive evaluation and the recommendation of minor revision. The comments are constructive and focus on reproducibility, which we fully support. We address both major comments below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [§4] §4 (Experimental Setup): the paper must report the exact pre-training corpus size, vocabulary construction, and whether all baselines were re-trained on identical data; without these details the SOTA claim on the four tasks cannot be verified as arising from the proposed structure components rather than data differences.

    Authors: We agree that these experimental details are necessary for verifying that gains come from the proposed structure-aware components. The pre-training corpus, vocabulary construction, and baseline training protocols are identical to those in the CodeBERT paper on which GraphCodeBERT is built; we will add an explicit paragraph (or subsection) in §4 that states the exact corpus size, the BPE vocabulary construction procedure, and confirms that all reported baselines were either retrained or evaluated on the identical data splits and test sets. This revision will be made. revision: yes

  2. Referee: [§3.3] §3.3 (Graph-guided Masked Attention): the description of how data-flow edges are extracted from source code (e.g., via static analysis or heuristic rules) is insufficiently precise; a concrete algorithm or pseudocode is needed to ensure the structure is reproducible and not post-hoc tuned to the downstream tasks.

    Authors: We acknowledge that the current description of edge extraction is high-level and should be made fully reproducible. Data-flow edges are obtained via standard static reaching-definitions analysis on variable assignments and uses (not heuristics tuned to downstream tasks). In the revised manuscript we will insert a short algorithm box with pseudocode in §3.3 that outlines the steps: (1) parse the function, (2) identify variable definition and use sites, (3) compute reaching definitions, and (4) emit an edge from each definition to its reachable uses. The same deterministic procedure is applied uniformly during pre-training and downstream evaluation. This addition will be made. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; claims are empirical

full rationale

The paper introduces GraphCodeBERT by replacing ASTs with data-flow edges and adding two pre-training objectives (edge prediction and code-structure alignment) inside a Transformer with graph-guided masked attention. All central claims are supported by downstream empirical results on code search, clone detection, translation, and refinement rather than any derivation, fitted parameter, or self-citation that reduces the result to its inputs by construction. No equations or steps equate a prediction to a fitted input; performance gains are measured against external baselines.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The model inherits the standard Transformer architecture and masked-language-modeling objective; the only added assumptions are that data-flow edges are a useful semantic proxy and that the two new objectives can be optimized jointly with MLM.

assumptions (1)
  • domain assumption Data-flow edges capture the essential semantic relations needed for code understanding tasks.
    Stated in the abstract as the motivation for choosing data flow over AST.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GraphCodeBERT: Pre-training Code Representations with Data Flow." pith.science (2026). https://pith.science/paper/2XS2L4VX

@misc{pith2026200908366,
  author       = {Pith},
  title        = {Pith review of: GraphCodeBERT: Pre-training Code Representations with Data Flow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2XS2L4VX}},
  note         = {Machine review of arXiv:2009.08366}
}
read the original abstract

Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 91 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 91 Pith citations

  1. CODEBLOCK: Learning to Supervise Code at the Right Granularity

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    CodeBlock partitions code responses into syntactically coherent blocks, scores them with generalized cross-entropy and data-flow signals, and applies sparse supervision to achieve higher pass@1 than full SFT using 1.9...

  2. Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Multilingual execution-grounded benchmark finds top open code LLM at 23.64% correctness versus 57.2% human baseline, with compile errors dominating 63% of failures.

  3. BioDefect: The First Dataset for Defect Detection in Bioinformatics Software

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    BioDefect is a new dataset for defect detection in bioinformatics software that improves average F1-scores by 29.61% to 38.04% over existing datasets when evaluated on nine language models.

  4. FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    FML-Bench shows that a simple greedy hill-climber performs nearly as well as complex tree-search agents on ML research tasks, with an adaptive strategy that switches exploration modes outperforming all tested agents.

  5. Evaluating Tool Cloning in Agentic-AI Ecosystems

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Tool cloning is pervasive in agentic AI ecosystems, with 60% of high-Jaccard and 85% of high-ssdeep similar pairs verified as true clones in a study of over 8,800 repositories.

  6. Can You Trust the Vectors in Your Vector Database? Black-Hole Attack from Embedding Space Defects

    cs.CR 2026-04 unverdicted novelty 7.0 of 10

    Injecting a few vectors near the embedding-space centroid can make them appear in top-k results for up to 94.4% of queries via centrality-driven hubness.

  7. How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests

    cs.SE 2026-01 unverdicted novelty 7.0 of 10

    AI coding agents produce pull requests with substantially more commits and slightly higher description-to-diff similarity than human developers, based on analysis of 29,095 merged PRs.

  8. Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detection

    cs.SE 2026-01 unverdicted novelty 7.0 of 10

    Fisher information selects task-relevant parts of graph features to fuse with pretrained code models, improving vulnerability detection F1 by up to 6.3 points on BigVul, Devign, and ReVeal.

  9. AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming

    cs.CL 2025-07 reject novelty 7.0 of 10

    AlgoSimBench tests whether LLMs can identify algorithmically similar programming problems, and Attempted Solution Matching improves accuracy by comparing LLM-generated solution attempts, though the benchmark's design ...

  10. Can Large Language Models Understand Intermediate Representations in Compilers?

    cs.LG 2025-02 conditional novelty 7.0 of 10

    Large language models can parse compiler intermediate representations but consistently fail at instruction-level reasoning such as control flow, loop handling, and exact execution simulation.

  11. Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs

    cs.SE 2023-05 unverdicted novelty 7.0 of 10

    LLMs achieve strong results on syntax parsing tasks but show limited and variable performance on dynamic reasoning, with a clear performance hierarchy across model scales.

  12. SimP: Unifying Syntax- and Semantic-Guided Techniques for Efficient Program Reduction

    cs.PL 2026-08 conditional novelty 6.0 of 10

    SimP combines Perses-style deletion with a long-tail detector and LLM-based semantic and syntactic mutation to reduce compiler bug test cases 1.75x faster than the better of Perses and LPR, while keeping reduced progr...

  13. SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization

    cs.AI 2026-07 conditional novelty 6.0 of 10

    SCOPE evolves LLM-generated auxiliary objective functions and selects a validated portfolio of them to guide fixed combinatorial search engines under strict black-box query budgets.

  14. Enhancing Code Understanding for Impact Analysis by Combining Transformers and Program Dependence Graphs

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Athena beats LSI-based impact analysis by ~10% (mRR/mAP/HIT@10) by propagating transformer code embeddings over call and class-member dependence graphs.

  15. SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A quality-aware code-search reranking framework with LLM-synthesized quality variants and a hierarchical ranking loss beats relevance-only training on a new QPA/MCA benchmark.

  16. XRFormer: Multiscale Tokenization for XRF Representation Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A multiscale convolutional tokenizer plus MSM/PPP pretraining yields more accurate, parameter-efficient transformers for XRF pigment identification and unmixing than ViT, SpectralFormer, or 1D-CNN baselines.

  17. Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Refusal-ablated LLMs outperform aligned models on code-grounded localization and early executable patch generation, while aligned models retain advantages on shallow diagnostic tasks under neutral wording.

  18. Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study

    cs.SE 2026-07 unverdicted novelty 6.0 of 10

    LLM-based methods achieve higher F1-scores than traditional approaches for equivalent mutant detection in Java and C, with fine-tuned code embeddings performing best and showing cross-lingual generalization.

  19. The Decomposition Is the Fingerprint: Per-Component Identity for Agent Skills

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    A per-component SimHash fingerprint supplies structural identity for AI agent skills, recovering family membership under paraphrase and refactoring with AUC 0.974 while localizing changes.

  20. Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    TestHumanizer uses LLMs as refactoring layers on EvoSuite suites to reach 88-98% compilation rates and better readability on 350 classes from Defects4J and SF110 while preserving coverage.

  21. Test Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Replication of TCS strategies on 17 LLM instances across three code tasks shows only partial generalization from vision DNN results, with uncertainty features aiding early failure discovery and representation features...

  22. Semantic Code Clone Detection: Are We There Yet?

    cs.SE 2026-06 conditional novelty 6.0 of 10

    SOTA semantic code clone detectors exhibit substantial performance degradation on distribution-shifted yet semantically equivalent clones, revealing reliance on lexical and structural shortcuts rather than semantic un...

  23. Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

    cs.CR 2026-06 conditional novelty 6.0 of 10

    A curated temporal-split benchmark shows LLMs achieve at most 52.1% vulnerability detection accuracy after fine-tuning because backbone directional biases resist correction and contamination provides no benefit.

  24. PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    PromptAudit evaluates five prompting strategies across five LLMs on 1000 CVEs and finds chain-of-thought prompting yields the strongest overall performance while adaptive chain-of-thought and self-consistency reduce e...

  25. XSearch: Explainable Code Search via Concept-to-Code Alignment

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    XSearch achieves explainable code search by breaking queries into functional concepts and matching them directly to code statements, delivering large gains on out-of-distribution benchmarks.

  26. SemChunk-C: Semantic Segmentation for C Code

    cs.SE 2026-05 unverdicted novelty 6.0 of 10

    SemChunk-C trains lightweight Ettin-based models to detect semantic chunk boundaries and assign functional categories in C-related code, matching larger LLMs on accuracy.

  27. Better Call Grep: Evaluating and Improving Grep-Like Lexical Retrieval for Repository-Level Code Completion

    cs.SE 2026-01 conditional novelty 6.0 of 10

    LLM-generated ripgrep queries plus BM25 re-ranking and line-interval de-duplication outperform graph- and RL-based retrievers for repository-level code completion on CrossCodeEval and RepoEval-Updated.

  28. A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?

    cs.SE 2025-11 unverdicted novelty 6.0 of 10

    Student models distilled from code language models often fail to deeply mimic teachers, showing up to 62% behavioral discrepancies and 285% worse drops under attacks that accuracy metrics miss.

  29. Multi Language Models for On-the-Fly Syntax Highlighting

    cs.SE 2025-10 unverdicted novelty 6.0 of 10

    Unified multi-language deep learning model for on-the-fly syntax highlighting using normalization and few-shot learning to support six languages with lower deployment cost.

  30. PseudoBridge: Pseudo Code as the Bridge for Better Semantic and Logic Alignment in Code Retrieval

    cs.SE 2025-09 unverdicted novelty 6.0 of 10

    PseudoBridge uses LLM-synthesized pseudo-code to bridge NL semantics and PL logic plus logic-invariant style augmentation to boost robustness and generalization in code retrieval.

  31. VISION: Robust and Interpretable Code Vulnerability Detection Leveraging Counterfactual Augmentation

    cs.AI 2025-08 conditional novelty 6.0 of 10

    LLM-generated counterfactual code pairs with flipped vulnerability labels, used to train a GNN, sharply improve CWE-20 detection and attribution on the released CWE-20-CFA benchmark.

  32. Fine-Tuning Code Language Models to Detect Cross-Language Bugs

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Fine-tuning 13 CodeLMs on a constructed CLB dataset with nine interaction types improves detection, with UniXcoder-base reaching F1 0.7407 and small models outperforming large ones.

  33. When Retriever Meets Generator: A Joint Model for Code Comment Generation

    cs.SE 2025-07 conditional novelty 6.0 of 10

    RAGSum couples retriever and generator in one CodeT5 model with contrastive pre-training, joint fine-tuning, and ROUGE-L based self-refinement, outperforming CMR-Sum, JOINTCOM, and Llama-3.1-8B on three code-comment datasets.

  34. Modeling Code: Is Text All You Need?

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A GNN-encoded LLVM IR graph, prepended as soft prompts to a frozen code LLM, improves accuracy on device mapping, algorithm classification, vulnerability detection, and code translation tasks.

  35. I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A fine-tuned encoder-only CodeT5 model attributes LLM-generated C code to its source model with up to 97.56% binary and 95.40% five-class accuracy on a new 32,000-program benchmark.

  36. Retrieval-Augmented Code Review Comment Generation

    cs.SE 2025-06 conditional novelty 6.0 of 10

    Retrieval-augmented conditioning on similar code-review pairs improves review comment generation over generation-only baselines, with larger gains for low-frequency tokens, though improvements over retrieval-only base...

  37. Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A graph-integrated open-source LLM with agentless RAG resolves 43% of SWE-bench Lite issues, best among open-weight models.

  38. UntrustVul: An Automated Approach for Identifying Untrustworthy Alerts in Vulnerability Detection Models

    cs.SE 2025-03 unverdicted novelty 6.0 of 10

    UntrustVul identifies untrustworthy vulnerability predictions by marking lines that neither match historical vulnerability patterns nor influence vulnerable lines through dependencies, reporting AUC 70-88% and F1 82-9...

  39. XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants

    cs.CR 2025-03 unverdicted novelty 6.0 of 10

    XOXO is a cross-origin context poisoning attack on AI coding assistants that uses a Cayley Graph search algorithm (GCGS) to find stealthy perturbations, achieving 75.72% average success rate across five tasks and elev...

  40. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.

  41. ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Corrupted, ambiguous context semantics, not the presence of [MASK] symbols, drive MLM accuracy loss; expanding each [MASK] into multiple modeled states mitigates this.

  42. Enhancing LLM's Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection

    cs.SE 2025-01 conditional novelty 6.0 of 10

    RATester injects gopls-fetched definitions into LLM prompts during unit test generation, achieving 26.25% average line coverage and more killed mutants than baselines.

  43. XSema: A Novel Framework for Semantic Extraction of Cross-chain Transactions

    cs.DC 2024-12 conditional novelty 6.0 of 10

    A cross-chain transaction classifier using motif statistics and event-log embeddings reaches 99.7% accuracy on known bridges and 94.8% accuracy on held-out bridges.

  44. Examining the Use and Impact of an AI Code Assistant on Developer Productivity and Experience in the Enterprise

    cs.HC 2024-12 conditional novelty 6.0 of 10

    A mixed-methods study of IBM's internal watsonx Code Assistant with 669 survey respondents and 15 usability participants finds code understanding is the top use case and perceived productivity gains are small and unev...

  45. CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking

    cs.CL 2024-12 conditional novelty 6.0 of 10

    CoRNStack, a consistency-filtered 21M-pair contrastive dataset with curriculum hard negatives, yields state-of-the-art code retrievers and the first finetuned LLM listwise code reranker.

  46. CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics

    cs.SE 2024-11 conditional novelty 6.0 of 10

    Using GPT-4 scoring and test-code heuristics, VulSifter cleans vulnerability-fixing commits into a dataset with 90.6% label correctness, enabling models that outperform those trained on existing datasets.

  47. ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code

    cs.SE 2024-11 conditional novelty 6.0 of 10

    A prompt-engineered LLM pipeline can generate production code assertions with up to 83.5% compile accuracy and 0.526 ROUGE-L similarity to developer-written assertions.

  48. CodeSAM: Source Code Representation Learning by Infusing Self-Attention with Multi-Code-View Graphs

    cs.SE 2024-11 conditional novelty 6.0 of 10

    Code-view graphs can be infused into a transformer as self-attention masks, yielding small but consistent gains on code search, clone detection, and program classification.

  49. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

    cs.SE 2021-02 unverdicted novelty 6.0 of 10

    CodeXGLUE supplies a standardized collection of 10 code-related tasks, 14 datasets, an evaluation platform, and BERT-, GPT-, and encoder-decoder-style baselines.

  50. Finetuning Lightweight LLMs for Control Flow Graph Generation

    cs.SE 2026-07 conditional novelty 5.5 of 10

    Fine-tuned ~3–7B LLMs generate unified digraph CFGs from incomplete/erroneous code and show partial cross-language transfer to held-out JavaScript.

  51. LLM-Enhanced Hierarchical Heterogeneous Graph Representation Learning for Malicious Python Package Detection

    cs.CR 2026-07 conditional novelty 5.5 of 10

    H2GLM combines LLM-inferred function roles with hierarchical heterogeneous GNN message passing to detect and localize malicious Python packages more accurately than prior ML, graph, and LLM baselines.

  52. HyperFL: Query-Adaptive Representation Learning for Software Fault Localization

    cs.SE 2026-08 reject novelty 5.0 of 10

    HyperFL uses a hypernetwork to generate query-specific LoRA parameters for the query encoder, reporting improved fault localization retrieval, yet the evaluation lacks a clear train/test split and external gains are marginal.

  53. UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning

    cs.SE 2026-06 unverdicted novelty 5.0 of 10

    UNICS pre-trains on a pseudocode dataset for cross-lingual logic then applies multi-task transfer learning with hard-positive mining and dynamic hard-negative sampling to reach claimed SOTA on multilingual code-search...

  54. JupOtter: Cell-Level Bug Detection in Jupyter Notebooks

    cs.SE 2026-06 unverdicted novelty 5.0 of 10

    JupOtter introduces notebook-specific tokenization, cell-level bug prediction, and OtterDataset to achieve higher F1 scores than static analyzers and LLMs on two of three evaluation datasets.

  55. ConcernBERT: Learning Responsibilities Using Class Membership

    cs.SE 2026-06 unverdicted novelty 5.0 of 10

    ConcernBERT is a BERT embedding model trained with triplet loss on class membership to encode concern-level semantics in Java entities, evaluated by recovering original classes from merged unlabeled groups on a new da...

  56. Prompt Optimization for LLM Code Generation via Reinforcement Learning

    cs.SE 2026-05 unverdicted novelty 5.0 of 10

    A PPO agent with hybrid actions and test-driven rewards optimizes prompts for code LLMs, raising strict Pass@1 scores on MBPP+, HumanEval+, and APPS over prior methods.

  57. Social Life of Code: Modeling Evolution through Code Embedding and Opinion Dynamics

    cs.SE 2026-02 unverdicted novelty 5.0 of 10

    Code embeddings combined with the Expressed-Private Opinion model produce trajectories that quantify developer influence and consensus formation across three open-source repositories.

  58. SEER: Spectral Entropy Encoding of Roles for Context-Aware Attention-Based Design Pattern Detection

    cs.SE 2026-01 conditional novelty 5.0 of 10

    SEER adds spectral-entropy role encoding from Laplacian spectra and empirically calibrated time-weighted calling contexts to raise macro-F1 from 92.47% to 93.20% and accuracy from 92.52% to 93.98% on PyDesignNet for 2...

  59. Context-Guided Decompilation: A Step Towards Re-executability

    cs.SE 2025-11 unverdicted novelty 5.0 of 10

    ICL4Decomp applies in-context learning to guide LLMs in generating re-executable decompiled code from binaries, reporting roughly 40% higher re-executability than prior methods across datasets and optimization levels.

  60. Automated Repair of C Programs Using Large Language Models

    cs.SE 2025-09 conditional novelty 5.0 of 10

    An agent that combines spectrum-based fault localization, test feedback, and chain-of-thought prompting repairs 44.93% of 3,902 Codeflaws C bugs, a 3.61-point gain over GPT-4 with CoT.

See all 91 Pith citations

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages · cited by 91 Pith papers (see all)

  1. [1]

    code2seq: Generating Sequences from Structured Representations of Code

    Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400,

    work page Pith review arXiv

  2. [2]

    Structural language models of code

    Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. arXiv, pp. arXiv–1910,

    work page 1910

  3. [3]

    Generative Code Modeling with Graphs

    Marc Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov. Generative code modeling with graphs. arXiv preprint arXiv:1805.08490,

    work page Pith review arXiv

  4. [4]

    Exploring software natural- ness throughneural language models

    Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, et al. Exploring software natural- ness throughneural language models. arXiv preprint arXiv:2006.12641,

    work page arXiv 2006

  5. [5]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    work page Pith review arXiv

  6. [6]

    CodeBERT: A Pre-Trained Model for Programming and Natural Languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155,

    work page Pith review arXiv 2002

  7. [7]

    Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing

    Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Coupling retrieval and meta-learning for context-dependent semantic parsing. ArXiv, abs/1906.07108,

    work page Pith review arXiv 1906

  8. [8]

    CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436,

    work page Pith review arXiv 1909

Show all 35 references
  1. [9]

    Pre-trained contextual embedding of source code

    Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual embedding of source code. arXiv preprint arXiv:2001.00059,

    2001

  2. [10]

    Phrase-based statistical translation of programming languages

    Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, pp. 173–184,

    2014

  3. [11]

    Scelmo: Source code embeddings from language models

    10 Published as a conference paper at ICLR 2021 Rafael-Michael Karampatsis and Charles Sutton. Scelmo: Source code embeddings from language models. arXiv preprint arXiv:2004.13214,

    2021

  4. [12]

    Code prediction by feeding trees to transformers

    Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. arXiv preprint arXiv:2003.13848,

    2003

  5. [13]

    Cross-lingual language model pretraining

    Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291,

    1901 arXiv

  6. [14]

    Code completion with neural attention and pointer networks

    Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. arXiv preprint arXiv:1711.09573,

  7. [15]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

    1907 arXiv

  8. [16]

    Graph-based statistical language model for code

    Anh Tuan Nguyen and Tien N Nguyen. Graph-based statistical language model for code. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1, pp. 858–868. IEEE,

    2015

  9. [17]

    Lexical statistical machine translation for language migration

    Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 651–654,

    2013

  10. [18]

    Divide-and-conquer approach for multi-phase statistical migration for source code (t)

    Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Divide-and-conquer approach for multi-phase statistical migration for source code (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 585–596. IEEE,

    2015

  11. [19]

    Deep contextualized word representations

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365,

    arXiv

  12. [20]

    Abstract syntax networks for code generation and semantic parsing

    Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv:1704.07535,

    arXiv

  13. [21]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683,

    1910 arXiv

  14. [22]

    Towards a big data curated benchmark of inter-project code clones

    Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 476–480. IEEE,

    2014

  15. [23]

    Intellicode compose: Code generation using transformer

    Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. arXiv preprint arXiv:2005.08025,

    2005

  16. [24]

    Detecting code clones with graph neural networkand flow-augmented abstract syntax tree

    11 Published as a conference paper at ICLR 2021 Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural networkand flow-augmented abstract syntax tree. arXiv preprint arXiv:2002.08653,

    2021

  17. [25]

    Deep learning code fragments for code clone detection

    Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning code fragments for code clone detection. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 87–98. IEEE,

    2016

  18. [26]

    Xlnet: Generalized autoregressive pretraining for language understanding

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237,

    1906

  19. [27]

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu

    URL https://arxiv.org/abs/1704.01696. Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural source code representation based on abstract syntax tree. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 783–7...

    2019 arXiv

  20. [28]

    The dataset is the CodeSearchNet dataset 3 (Husain et al., 2019), which includes 2.3M functions with document pairs for six programming languages

    to pretrain our model. The dataset is the CodeSearchNet dataset 3 (Husain et al., 2019), which includes 2.3M functions with document pairs for six programming languages. We train the model on two DGX-2 machines, each having 16 NVIDIA Tesla V100 with 32GB memory. We set the max...

    2019

  21. [29]

    http://

    and follow Husain et al. (2019) to take the first paragraph of the documentation as the query for the corresponding function. However, we observe that some queries contain content unrelated to the code, such as a link “http://...” that refers to external resources. Therefore, w...

    2019

  22. [30]

    (4) Examples whose query is empty or not written in English

    (3) Examples whose query contains special tokens such as “http://”. (4) Examples whose query is empty or not written in English. 3https://github.com/github/CodeSearchNet 12 Published as a conference paper at ICLR 2021 Different from the setting of Husain et al. (2019), the ans...

    2021

  23. [31]

    We also report the results using the same setting of Husain et al

    We use the Adam optimizer to update model parameters and perform early stopping on the development set. We also report the results using the same setting of Husain et al. (2019) in Table

    2019

  24. [32]

    The results show that GraphCodeBERT also achieves the state-of-the-art performance

    In this setting, models are required to retrieve an answer for a query from 1000 candidates. The results show that GraphCodeBERT also achieves the state-of-the-art performance. model Ruby Javascript Go Python Java Php Overall NBow 0.429 0.461 0.641 0.581 0.514 0.484 0.518 CNN ...

    2019

  25. [33]

    Therefore, two codes are semantically similar since they output similar results when given the same input

    In this example, two Java source codes both download content from a given URL and convert the type of the content into string type. Therefore, two codes are semantically similar since they output similar results when given the same input. As we can see, our model gives a high ...

    2021

  26. [34]

    boolean” to “bool

    In this example, the model successfully translates a piece of Java code into its C# version. The differences include the type name (from “boolean” to “bool”) and the usage of getting a string value of a bool variable (from “String.valueOf(b)” to “b.ToString()”). Figure 7: A ca...

    2021

  27. [35]

    http://kmttg.googlecode.com/svn/trunk/version

    The first source code is to return the HTML content from a given URL, while the second source code is to return the last line from a fixed URL “http://kmttg.googlecode.com/svn/trunk/version”. Their semantics are not similar due to their different outputs. Data flow could help Gra...

    2021

Pith tools

Reviewed May 15, 2026 · model on record in the stance chip above.