Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

The paper argues that training code models on a progressive curriculum of obfuscated snippets, with MAML-style meta-updates, restores summarization accuracy on poor-readability code and even improves clean-code accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:36 UTC pith:L2RBTGPT

load-bearing objection A plausible method with a potentially fatal checkpoint-selection asymmetry; worth reading but needs a uniform evaluation protocol. the 3 major comments →

arxiv 2601.05485 v2 pith:L2RBTGPT submitted 2026-01-09 cs.SE

Readability-Robust Code Summarization via Meta Curriculum Learning

Wenhao Zeng , Yitian Chai , Hao Zhou , Fandong Meng , Jie Zhou , Xiaodong Gu This is my paper
classification cs.SE
keywords code summarizationcode readabilityrobustnesscurriculum learningmeta-learningMAMLcode obfuscationfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that poor code readability is a real and under-addressed failure mode for code summarization models, and that it can be fixed without changing the model architecture or pre-training. It first measures that state-of-the-art LLMs lose substantial accuracy when semantic cues such as identifiers and function names are stripped away or misleading dead code is injected, and that prompt engineering and reasoning-oriented variants give only limited relief. It then proposes RoFTCodeSum, a fine-tuning method that builds curricular datasets of increasing obfuscation (function-name erosion, identifier renaming, dead-code injection) and meta-updates gradients, MAML-style, on the harder levels while anchoring on clean code. The central claim is that this meta-curriculum produces robustness to obfuscation and, at the same time, better summaries of the original code, challenging the usual assumption that robustness must trade off against normal performance. A sympathetic reader should care because real code—legacy, decompiled, obfuscated, or hurriedly written—frequently looks more like the curricula than like benchmark data.

Core claim

RoFTCodeSum fine-tunes a language model in two stages per batch: a normal gradient update on original, readable code (learning rate α), then MAML-style local adaptation and a query-gradient meta-update on two curricular levels of obfuscation (β, γ). The curriculum removes semantic cues progressively—function-name erosion first, identifier renaming second—or injects increasing amounts of dead code. On the Python subset of a standard code-summarization benchmark, with DeepSeek-Coder and Qwen2.5-Coder backbones, RoFTCodeSum reports the highest average BLEU and SBERT across original and obfuscated test sets among direct fine-tuning, fine-tuning on all obfuscated data, plain curriculum learning,

What carries the argument

The central machinery is the meta-curriculum: a chain of training sets derived from one base set by controlled readability degradation—FNE renames function names to func_1, IRN renames all local identifiers to var_1, var_2, and DCI inserts 5 then 10 semantically neutral dead statements. Training interleaves a plain fine-tuning step on the base set with MAML's inner-loop support adaptation and outer-loop query meta-update on the harder levels, so global parameters are optimized to adapt quickly and stably to less readable inputs. The paper credits this joint optimization for both the robustness and the clean-code improvement.

Load-bearing premise

The load-bearing premise is that the synthetic ways of making code hard to read—renaming identifiers, eroding function names, injecting dead code—behave like real-world poor readability; if that mapping fails, the demonstrated robustness may not survive contact with naturally messy code.

What would settle it

Hold out a test set of genuinely low-readability code—for instance, decompiled functions with no original identifiers, or a control-flow-flattened version of the benchmark. If RoFTCodeSum's advantage over direct fine-tuning on synthetic obfuscations shrinks to zero on this set, or if both methods collapse equally on flattened control flow, then the claimed readability robustness is an artifact of the synthetic curriculum.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Code summarization can be made meaningfully robust to stripped identifiers, eroded function names, and injected dead code through fine-tuning alone—no new pre-training objectives or architecture changes.
  • Robustness training on a progressive curriculum does not necessarily sacrifice clean-code accuracy; in the reported experiments it improves it, suggesting the trade-off is not inevitable.
  • The recipe is model-agnostic and fits the standard fine-tuning pipeline, making it a practical upgrade for open-weight code models.
  • The same meta-curriculum principle can be carried to other code tasks where naming styles vary, such as defect detection and code translation—applications the paper explicitly names.
  • For real-world deployment, the method points toward more trustworthy summaries on legacy systems, third-party libraries, reverse-engineered code, and security analysis, where poor readability is the norm.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic-to-real mapping holds, the direct testable extension is to evaluate RoFTCodeSum on naturally unreadable code—decompiled functions, stripped binaries, or 'code-smell' repositories; the paper itself notes no such benchmark exists yet.
  • The clean-code improvement hints that the meta-curriculum may act as a regularizer that reduces reliance on surface cues; an interpretability study that perturbs identifier names and tracks attention shifts could make this mechanism observable.
  • The paper's own concession about control-flow flattening suggests a ready stress test: add CFF as a fourth curriculum level and compare models trained with and without it—this would isolate how much of the robustness comes from curriculum breadth versus meta-learning.
  • Since reported performance is more sensitive to the outer rates α/γ than the inner rate β, the benefit may come mostly from the meta-objective's global update rather than from local adaptation; an ablation varying β across a wider range would clarify the division of labor.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how poorly readable/obfuscated code affects code summarization models and proposes a fine-tuning method, RoFTCodeSum, that combines curriculum learning with MAML. The empirical study first benchmarks several LLMs and prompting methods on MLRC and CodeSearchNet under three obfuscation types (DCI, FNE, IRN), reporting performance drops and limited gains from prompt engineering. The proposed method fine-tunes on original code and then meta-updates on progressively obfuscated tasks. Experiments on DeepSeek-Coder and Qwen2.5-Coder report consistent improvements over fine-tuning, fine-tuning on all obfuscated data, curriculum learning, and CLAWSAT across two curricula, while also improving clean-code performance. The paper explicitly acknowledges external-validity limits regarding synthetic obfuscations and unseen complex obfuscations.

Significance. If the reported gains survive a uniform and transparent checkpoint-selection protocol, the contribution is valuable: a simple, architecture-agnostic fine-tuning recipe that improves robustness to specific obfuscation families while preserving or improving original-code accuracy. The empirical study is broad, uses two backbone models, two curricula, and standard metrics, and the paper is candid about its limitations. The main significance is conditional on resolving the checkpoint-selection issue, because as written the central comparison may reflect an uncontrolled protocol choice rather than the method itself.

major comments (3)
  1. [§5.2, Tables 4–7] The checkpoint-selection rule is asymmetric. The text says: for the curriculum-learning baseline, "we select the last checkpoint of the model," while for the final evaluation, "we select the checkpoint that attains the highest average accuracy on the validation sets of all datasets within each curriculum." If the best-checkpoint rule is applied to RoFTCodeSum only, the reported AVG gains (e.g., +3.31 vs +2.44 in Table 4; +1.92 vs +1.38 in Table 6) may be artifacts of oracle selection from a noisy fine-tuning trajectory. If the rule is applied uniformly to all methods, that must be stated explicitly, and the selected epoch/checkpoint for each method and seed must be reported. As written, the central claim in §6.1 and §7.1 that RoFTCodeSum "not only enhances robustness ... but also improves performance on original code" is not fully supported. Please re-run with a uniform rule (e.g., last
  2. [§3.2, Table 1] The text claims "Every tested model ... exhibits consistent reductions" across all poor-readability settings, and concludes "significant and consistent performance degradation." Table 1 contradicts this: Claude-3.5-Sonnet-20241022 on MLRC shows BLEU rising from 8.43 (High) to 8.77 (Low). Table 3 similarly shows QwQ-32B MLRC BLEU increasing from 5.75 to 5.93. The blanket statement is therefore not supported. Please revise the claim to describe a general trend with explicitly noted exceptions, or report error bars / significance tests that show the apparent increases are within noise.
  3. [§8.1, §6.1, §7.1] The method is trained and evaluated on the same obfuscation families (FNE, IRN, DCI), so the reported "strong robustness to semantic obfuscation and interference" is in-distribution robustness with respect to those families. The paper acknowledges in §8.1 that synthetic obfuscation may not capture the full spectrum of naturally occurring low-readability code and in §7.3 that an unseen structural obfuscation such as control-flow flattening would not be fully handled. This is an external-validity limitation on the title-level claim of "readability-robust" behavior. The claims should be scoped to the specific obfuscation families used, or the evaluation should include a held-out obfuscation type or real-world low-readability test set.
minor comments (5)
  1. [§3.1] Typo: "Fuction Name Erosion" should be "Function Name Erosion."
  2. [§5.2] The validation criterion "highest average accuracy" is undefined for a generation task. Please specify the metric (BLEU, SBERT, or both) and exactly which validation sets are averaged. Also report the selected epoch for each method.
  3. [§3.1, §6] No error bars, confidence intervals, or multiple-seed results are reported. The Wilcoxon signed-rank test is mentioned, but the number of paired samples, the test variant, and whether multiple-comparison correction was applied are not given. These details are needed to interpret the statistical claim.
  4. [Abstract] The abstract says "current models and benchmarks are confined to high-readability code," but the paper itself uses CodeSearchNet, which contains code of varied readability and is not restricted to high-readability samples. Please soften this overstatement.
  5. [§3.1] The SBERT metric is described as using "the default implementation and configuration as described in the original paper," but the specific Sentence-BERT model is not named. This hurts reproducibility.

Circularity Check

0 steps flagged

No circular derivation: central claim rests on external metrics and standard baselines; checkpoint-selection asymmetry is a validity concern, not a definitional reduction.

full rationale

The paper's central claim—that RoFTCodeSum improves robustness on obfuscated code while preserving original-code performance—is not obtained by fitting the target quantity or by importing a uniqueness theorem from the authors' own prior work. The training objective (Eq. 1 with MAML updates) is defined independently of the test metrics; evaluation uses external BLEU-4 and SBERT against standard baselines (FT, FTAll, CL, CLAWSAT). Self-citations such as [10] and [46]/[47] are used as background or motivation, not as the load-bearing justification for the reported gains; the paper's own empirical study (Tables 1-3) independently motivates the problem. Reusing FNE/IRN/DCI in both training and test is a generalization limitation, acknowledged in §7.3 and §8.1, rather than circularity: training on these obfuscations does not by construction guarantee test improvement. The checkpoint-selection asymmetry in §5.2 (last checkpoint for the CL baseline vs. best-validation checkpoint for final evaluation) is a potential evaluation-protocol confound that should be checked, but it is not a definitional equivalence between the claim and its inputs. Overall, no significant circularity; score 2 reflects minor non-load-bearing self-citations and the need for a uniform checkpoint rule, not a circular derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central method rests on the choice of synthetic obfuscations as proxies for readability, on metrics as proxies for quality, on MAML's transfer assumptions, and on a small set of tuned learning rates. No new scientific entities are introduced; the method is a combination of existing training techniques applied to a new domain.

free parameters (2)
  • learning rates alpha, beta, gamma = alpha=gamma=beta=5e-5
    Selected by grid search over {1e-4, 5e-5, 2e-5, 1e-5} (RQ6, Figure 5); the reported performance depends on this choice.
  • curriculum difficulty levels
    The progression FNE->IRN and DCI5->DCI10 is chosen by hand to define increasing difficulty; the method's results are tied to these specific curricula.
axioms (4)
  • domain assumption FNE/IRN/DCI obfuscations approximate real-world poor-readability code
    Central motivation and evaluation depend on this; Section 8.1 acknowledges synthetic patterns may not capture natural low-readability code.
  • domain assumption BLEU-4 and SBERT cosine similarity are valid proxies for summarization quality
    Section 8.3 acknowledges they remain imperfect proxies for human judgment.
  • domain assumption MAML local adaptation on support sets transfers to query sets and to the original summarization task
    The meta-update in Algorithm 1 (lines 6-14) assumes inner-loop gradients on obfuscated support batches are useful for the global model; no proof or analysis is provided.
  • domain assumption CodeSearchNet Python subset is representative of code summarization benchmarks
    Experiments use CodeSearchNet with no justification beyond widespread adoption [39,40,48].

pith-pipeline@v1.3.0-alltime-deepseek · 19809 in / 13425 out tokens · 133581 ms · 2026-08-03T11:36:24.632038+00:00 · methodology

0 comments
read the original abstract

Code summarization has emerged as a fundamental technique in the field of program comprehension. While code language models have shown significant advancements, the current models and benchmarks are confined to high-readability code, which contains sufficient semantic cues such as function and variable names. In the real world, however, code is often poorly structured or obfuscated, significantly degrading model performance. In this paper, we first empirically evaluate the robustness of state-of-the-art language models on poor-readability code for the task of code summarization, focusing on (1) their effectiveness, (2) the impact of prompt engineering, and (3) the robustness of different variants. Experimental results reveal that state-of-the-art models-including GPT-4o and DeepSeek-V3 experience a substantial performance drop when faced with poorly readable code, and that prompt engineering and reasoning-enhanced models offer limited improvements. Motivated by these findings, we propose RoFTCodeSum, a novel fine-tuning method that enhances the robustness of code summarization against poorly readable code. RoFTCodeSum marries the concepts of curriculum learning and meta-learning: based on the original dataset for fine-tuning, it creates curricular training sets, e.g., obfuscating function names and identifiers from the code, respectively, that have progressive difficulty in code comprehension. In each training step, the approach meta-updates the gradients using these progressively challenging datasets, thereby optimizing both accuracy and readability robustness simultaneously. Experimental results demonstrate that RoFTCodeSum exhibits increased robustness against semantic perturbation while enhancing performance on the original code.

Figures

Figures reproduced from arXiv: 2601.05485 by Fandong Meng, Hao Zhou, Jie Zhou, Wenhao Zeng, Xiaodong Gu, Yitian Chai.

Figure 1
Figure 1. Figure 1: A motivating example of how poor readability interferes with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of RoFTCodeSum, illustrating the flow from the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An example of two curricular datasets. The semantically obfuscated parts are marked in blue. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the MAML algorithm in our method. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance of RoFTCodeSum (DeepSeek-Coder) under differ [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Two cases of readability robust code summarization by RoFTCodeSum (DeepSeek-Coder). The obfuscated tokens are marked in blue. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction

    cs.CV 2026-04 unverdicted novelty 7.0

    ShredBench shows state-of-the-art MLLMs perform well on intact documents but suffer sharp drops in restoration accuracy as fragmentation increases to 8-16 pieces, indicating insufficient cross-modal semantic reasoning...

  2. CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding

    cs.CL 2026-02 unverdicted novelty 7.0

    Multimodal LLMs process code as images to achieve up to 8x token compression, with visual cues like syntax highlighting aiding tasks and clone detection remaining resilient or even improving under compression.

  3. SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents

    cs.SE 2026-06 unverdicted novelty 5.0

    SWE-MeM introduces adaptive memory management for coding agents via synthesized trajectories and Memory-aware GRPO, reporting 43.4% and 60.2% resolve rates on SWE-Bench Verified for 4B and 30B models while beating bas...

Reference graph

Works this paper leans on

60 extracted references · 15 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Deepseek-coder: When the large language model meets programming – the rise of code intelligence,

    D. Y. Daya Guo, Qihao Zhuet al., “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14196

    Pith/arXiv arXiv 2024

  2. [2]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Danget al., “Qwen2. 5-coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Code needs comments: Enhancing code llms with comment augmentation,

    D. Song, H. Guo, Y. Zhou, S. Xing, Y. Wang, Z. Song, W. Zhang, Q. Guo, H. Yan, X. Qiuet al., “Code needs comments: Enhancing code llms with comment augmentation,” inFindings of the Associa- tion for Computational Linguistics ACL 2024, 2024, pp. 13 640–13 656

    2024

  4. [4]

    Llm-assisted code cleaning for training accurate code generators,

    N. Jain, T. Zhang, W.-L. Chiang, J. E. Gonzalez, K. Sen, and I. Stoica, “Llm-assisted code cleaning for training accurate code generators,”arXiv preprint arXiv:2311.14904, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    How do your code llms perform? empowering code instruction tuning with really good data,

    Y. Wang, K. He, D. Fu, Z. Gongque, H. Xu, Y. Chen, Z. Wang, Y. Fu, G. Dong, M. Diaoet al., “How do your code llms perform? empowering code instruction tuning with really good data,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 14 027–14 043

    2024

  6. [6]

    Do comments follow commenting conventions? a case study in java and python,

    P . Rani, S. Abukar, N. Stulova, A. Bergel, and O. Nierstrasz, “Do comments follow commenting conventions? a case study in java and python,” in2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 2021, pp. 165–169

    2021

  7. [7]

    Are we building on the rock? on the importance of data preprocessing for code summarization,

    L. Shi, F. Mu, X. Chen, S. Wang, J. Wang, Y. Yang, G. Li, X. Xia, and Q. Wang, “Are we building on the rock? on the importance of data preprocessing for code summarization,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 107–119

    2022

  8. [8]

    Adversarial robustness for code,

    P . Bielik and M. Vechev, “Adversarial robustness for code,” in International Conference on Machine Learning. PMLR, 2020, pp. 896–907

    2020

  9. [9]

    Towards generating summaries for lexically confusing code through code erosion

    F. Yan, M. Li, and Z. Zhou, “Towards generating summaries for lexically confusing code through code erosion.” inIJCAI, 2021, pp. 3721–3727

    2021

  10. [10]

    How effectively do code language models understand poor-readability code?

    C. Hu, Y. Chai, H. Zhou, F. Meng, J. Zhou, and X. Gu, “How effectively do code language models understand poor-readability code?” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 795–806

    2024

  11. [11]

    Beyond chat: a framework for llms as human-centered support systems,

    Z. Zhou, “Beyond chat: a framework for llms as human-centered support systems,”arXiv preprint arXiv:2511.03729, 2025

    arXiv 2025

  12. [12]

    Contrastive code representation learning,

    P . Jain, A. Jain, T. Zhang, P . Abbeel, J. E. Gonzalez, and I. Sto- ica, “Contrastive code representation learning,”arXiv preprint arXiv:2007.04973, 2020

    Pith/arXiv arXiv 2007

  13. [13]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,”arXiv preprint arXiv:1909.09436, 2019

    Pith/arXiv arXiv 1909

  14. [14]

    Summarizing source code using a neural attention model,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 2073–2083. [Online]. Available: https://aclanthology.org/P16-1195

    2016

  15. [15]

    Cross-entropy loss functions: Theoretical analysis and applications,

    A. Mao, M. Mohri, and Y. Zhong, “Cross-entropy loss functions: Theoretical analysis and applications,” inInternational conference on Machine learning. pmlr, 2023, pp. 23 803–23 828

    2023

  16. [16]

    Curriculum learning,

    Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProceedings of the 26th annual international conference on machine learning, 2009, pp. 41–48

    2009

  17. [17]

    Evolving culture versus local minima,

    Y. Bengio, “Evolving culture versus local minima,”Growing adap- tive machines: Combining development and learning in artificial neural networks, pp. 109–138, 2014

    2014

  18. [18]

    Why curriculum learn- ing & self-paced learning work in big/noisy data: A theoretical perspective,

    T. Gong, Q. Zhao, D. Meng, and Z. Xu, “Why curriculum learn- ing & self-paced learning work in big/noisy data: A theoretical perspective,”Big Data and Information Analytics, vol. 1, no. 1, pp. 111–127, 2016

    2016

  19. [19]

    Learning a similarity metric discriminatively, with application to face verification,

    S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1. IEEE, 2005, pp. 539–546

    2005

  20. [20]

    Prototypical networks for few- shot learning,

    J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few- shot learning,”Advances in neural information processing systems, vol. 30, 2017

    2017

  21. [21]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P . Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” inInternational conference on machine learning. PMLR, 2017, pp. 1126–1135

    2017

  22. [22]

    On first-order meta- learning algorithms,

    A. Nichol, J. Achiam, and J. Schulman, “On first-order meta- learning algorithms,”arXiv preprint arXiv:1803.02999, 2018

    Pith/arXiv arXiv 2018

  23. [23]

    Learning a metric for code read- ability,

    R. P . Buse and W. R. Weimer, “Learning a metric for code read- ability,”IEEE Transactions on software engineering, vol. 36, no. 4, pp. 546–558, 2009

    2009

  24. [24]

    Improving code readability models with textual features,

    S. Scalabrino, M. Linares-Vasquez, D. Poshyvanyk, and R. Oliveto, “Improving code readability models with textual features,” in2016 IEEE 24th International Conference on Program Comprehension (ICPC). IEEE, 2016, pp. 1–10

    2016

  25. [25]

    Contrastive learning for source code with structural and func- tional properties,

    Y. Ding, L. Buratti, S. Pujar, A. Morari, B. Ray, and S. Chakraborty, “Contrastive learning for source code with structural and func- tional properties,”arXiv preprint arXiv:2110.03868, 2021

    Pith/arXiv arXiv 2021

  26. [26]

    NatGen: generative pre-training by “naturalizing

    S. Chakraborty, T. Ahmed, Y. Ding, P . T. Devanbu, and B. Ray, “NatGen: generative pre-training by “naturalizing” source code,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 18–30

    2022

  27. [27]

    Structured neural summarization,

    P . Fernandes, M. Allamanis, and M. Brockschmidt, “Structured neural summarization,”arXiv preprint arXiv:1811.01824, 2018

    Pith/arXiv arXiv 2018

  28. [28]

    Neural reverse engineering of stripped binaries using augmented control flow graphs,

    Y. David, U. Alon, and E. Yahav, “Neural reverse engineering of stripped binaries using augmented control flow graphs,”Proceed- ings of the ACM on Programming Languages, vol. 4, no. OOPSLA, pp. 1–28, 2020

    2020

  29. [29]

    Debin: Predicting debug information in stripped binaries,

    J. He, P . Ivanov, P . Tsankov, V . Raychev, and M. Vechev, “Debin: Predicting debug information in stripped binaries,” inProceedings of the 2018 ACM SIGSAC Conference on Computer and Communica- tions Security, 2018, pp. 1667–1680

    2018

  30. [30]

    Measur- ing neural efficiency of program comprehension,

    J. Siegmund, N. Peitek, C. Parnin, S. Apel, J. Hofmeister, C. K ¨astner, A. Begel, A. Bethmann, and A. Brechmann, “Measur- ing neural efficiency of program comprehension,” inProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 140–150

    2017

  31. [31]

    Understanding understanding source code with functional magnetic resonance imaging,

    J. Siegmund, C. K ¨astner, S. Apel, C. Parnin, A. Bethmann, T. Le- ich, G. Saake, and A. Brechmann, “Understanding understanding source code with functional magnetic resonance imaging,” in Proceedings of the 36th international conference on software engineering, 2014, pp. 378–389

    2014

  32. [32]

    Contrastive code representation learning,

    P . Jain and A. Jain, “Contrastive code representation learning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

    2021

  33. [33]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P . Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, “Qwen2.5 t...

    Pith/arXiv arXiv 2024

  34. [34]

    Deepseek-v3 technical report,

    DeepSeek-AI, “Deepseek-v3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv 2024

  35. [35]

    Introducing gemini 2.0: our new ai model for the agentic era,

    Google, “Introducing gemini 2.0: our new ai model for the agentic era,” 2024. [Online]. Avail- able: https://blog.google/technology/google-deepmind/google- gemini-ai-update-december-2024/ 14

    2024

  36. [36]

    Claude 3.5 sonnet,

    Anthropic, “Claude 3.5 sonnet,” 2024. [Online]. Available: https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  37. [37]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” 2024. [Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  38. [38]

    Orange: a method for evaluating au- tomatic evaluation metrics for machine translation,

    C.-Y. Lin and F. J. Och, “Orange: a method for evaluating au- tomatic evaluation metrics for machine translation,” inCOLING 2004: Proceedings of the 20th International Conference on Computa- tional Linguistics, 2004, pp. 501–507

    2004

  39. [39]

    CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 8696–8708. [On...

    2021

  40. [40]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tanget al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,”arXiv preprint arXiv:2102.04664, 2021

    Pith/arXiv arXiv 2021

  41. [41]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. [Online]. Available: https://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  42. [42]

    What makes good in-context demonstrations for code in- telligence tasks with llms?

    S. Gao, X.-C. Wen, C. Gao, W. Wang, H. Zhang, and M. R. Lyu, “What makes good in-context demonstrations for code in- telligence tasks with llms?” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 761–773

    2023

  43. [43]

    Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,

    Y. Wang, Z. Zhang, and R. Wang, “Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 8640–8665

    2023

  44. [44]

    Language models can solve computer tasks,

    G. Kim, P . Baldi, and S. McAleer, “Language models can solve computer tasks,”Advances in Neural Information Processing Systems, vol. 36, pp. 39 648–39 677, 2023

    2023

  45. [45]

    Source code summarization in the era of large language models,

    W. Sun, Y. Miao, Y. Li, H. Zhang, C. Fang, Y. Liu, G. Deng, Y. Liu, and Z. Chen, “Source code summarization in the era of large language models,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024, pp. 419–431

    2024

  46. [46]

    Pruning the unsurprising: Efficient code reasoning via first-token surprisal,

    W. Zeng, Y. Wang, C. Hu, Y. Shi, C. Wan, H. Zhang, and X. Gu, “Pruning the unsurprising: Efficient code reasoning via first-token surprisal,”arXiv preprint arXiv:2508.05988, 2025

    arXiv 2025

  47. [47]

    Glimprouter: Efficient collaborative inference by glimpsing one token of thoughts,

    W. Zeng, X. Zhang, Y. Shi, C. Hu, Y. Chen, B. Shen, and X. Gu, “Glimprouter: Efficient collaborative inference by glimpsing one token of thoughts,” 2026. [Online]. Available: https://arxiv.org/abs/2601.05110

    Pith/arXiv arXiv 2026

  48. [48]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” inFindings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics, Nov. 2020, pp. 1536–

    2020

  49. [49]

    Stack overflow developer survey 2022,

    Stack Overflow, “Stack overflow developer survey 2022,” https://survey.stackoverflow.co/2022/, 2022, archived from the original on 27 June 2022. Retrieved 12 August 2022

    2022

  50. [50]

    Tiobe index,

    TIOBE, “Tiobe index,” https://www.tiobe.com/tiobe-index/, 2018, archived from the original on 25 February 2018. Retrieved 3 January 2023. The TIOBE Programming Community index is an indicator of the popularity of programming languages. Updated as required

    2018

  51. [51]

    Bridging the cyber-analysis gap: The democratization of data science,

    J. Healy, L. McInnes, and C. Weir, “Bridging the cyber-analysis gap: The democratization of data science,”The Cyber Defense Review, vol. 2, no. 1, pp. 109–118, 2017

    2017

  52. [52]

    In line with con- text: Repository-level code generation via context inlining,

    C. Hu, W. Zeng, Y. Shi, B. Shen, and X. Gu, “In line with con- text: Repository-level code generation via context inlining,”arXiv preprint arXiv:2601.00376, 2026

    Pith/arXiv arXiv 2026

  53. [53]

    From code to cor- rectness: Closing the last mile of code generation with hierarchical debugging,

    Y. Shi, S. Wang, C. Wan, M. Wang, and X. Gu, “From code to cor- rectness: Closing the last mile of code generation with hierarchical debugging,”arXiv preprint arXiv:2410.01215, 2024

    arXiv 2024

  54. [54]

    Between lines of code: Unraveling the distinct patterns of machine and human program- mers,

    Y. Shi, H. Zhang, C. Wan, and X. Gu, “Between lines of code: Unraveling the distinct patterns of machine and human program- mers,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024, pp. 51–62

    2024

  55. [55]

    Swe-debate: Competitive multi-agent debate for software issue resolution,

    H. Li, Y. Shi, S. Lin, X. Gu, H. Lian, X. Wang, Y. Jia, T. Huang, and Q. Wang, “Swe-debate: Competitive multi-agent debate for software issue resolution,”arXiv preprint arXiv:2507.23348, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    The impact of identifier style on effort and compre- hension,

    D. Binkley, M. Davis, D. Lawrie, J. I. Maletic, C. Morrell, and B. Sharif, “The impact of identifier style on effort and compre- hension,”Empirical software engineering, vol. 18, pp. 219–276, 2013

    2013

  57. [57]

    Clawsat: Towards both robust and accurate code models,

    J. Jia, S. Srikant, T. Mitrovska, C. Gan, S. Chang, S. Liu, and U.- M. O’Reilly, “Clawsat: Towards both robust and accurate code models,” in2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2023, pp. 212–223

    2023

  58. [58]

    Meta-curriculum learning for domain adaptation in neural machine translation,

    R. Zhan, X. Liu, D. F. Wong, and L. S. Chao, “Meta-curriculum learning for domain adaptation in neural machine translation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 16, 2021, pp. 14 310–14 318

    2021

  59. [59]

    On the evaluation of neural code summarization,

    E. Shi, Y. Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” inPro- ceedings of the 44th International Conference on Software Engineering, 2022, pp. 1597–1608

    2022

  60. [1547]

    Available: https://aclanthology.org/2020.findings- emnlp.139

    [Online]. Available: https://aclanthology.org/2020.findings- emnlp.139

    2020