Pith. sign in

REVIEW 4 major objections 5 minor 60 references

This paper claims that Functionality-Oriented Code Self-Evolution, which synthesizes four verified code-variant types from a single snippet, makes embedding models clearly better at code functional consistency, clone detection, and retrieva

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A data synthesis framework generates four syntax/semantics code pair types, and fine-tuning embedding models on the resulting datasets improves code clone detection, functional consistency, and retrieval.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful synthesis framework with a real Type IV gap, but the functional-consistency evaluation is too self-referential to support the strongest claims yet. the 4 major comments →

arxiv 2508.19558 v1 pith:3HEUN5KZ submitted 2025-08-27 cs.SE cs.CLcs.PL

Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking

Zhuohao Li , Wenqing Chen , Jianxing Yu , Zhichao Lu This is my paper
classification cs.SE cs.CLcs.PL
keywords code embeddingsfunctional consistencydata synthesisself-evolutioncode clone detectioncode retrievalLLM benchmarksemantic similarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Embedding models for code are usually judged on clone detection, which mostly rewards syntactic resemblance; this paper argues that the harder question is whether two snippets do the same thing, and that existing benchmarks do not ask it. It introduces Functionality-Oriented Code Self-Evolution, a synthesis framework that takes one code snippet, rewrites it along four evolution directions, and verifies each variant by executing shared test cases and checking a syntax-structure similarity metric. The paper claims that training a code embedding model on these evolved datasets clearly improves functional consistency identification, clone detection, and code retrieval, and that the gains carry over to an independent dataset built from a different source benchmark. A sympathetic reader would care because, if true, this gives a cheap, reusable way to build harder code benchmarks and suggests that embeddings can capture function, not just surface form.

Core claim

The central claim is that functional consistency of code can be made measurable and trainable through generated variants, rather than inferred from syntactic overlap. The framework defines four evolution directions matching four code-pair types: same functionality with similar syntax, same functionality with different syntax, different functionality with different syntax, and different functionality with similar syntax. Each variant is produced by an LLM rewrite, then filtered by executing shared test cases to decide semantic equivalence and by a CodeBLEU threshold to decide syntactic similarity. The paper reports that using the resulting evolved datasets for task alignment improves an embed

What carries the argument

The named mechanism is Functionality-Oriented Code Self-Evolution: from one source snippet, four evolution directions fix the intended syntactic and semantic change to produce variants matching the four code-pair types. The label-carrying step is a two-filter verification loop: running shared test cases decides semantic equivalence (identical outputs make a positive pair; different outputs make a negative pair), and CodeBLEU, a syntax-structure similarity metric, with a threshold decides syntactic similarity. This evolve-and-verify loop turns arbitrary LLM rewrites into labeled training pairs, and the resulting evolved datasets carry the argument for better embeddings.

Load-bearing premise

The framework's labels are only as correct as the generated test cases: a variant counts as functionally equivalent only if it produces identical outputs on those tests, so incomplete or incorrect tests would make the positive and negative labels unreliable.

What would settle it

Run a sample of variants the pipeline labels as functionally equivalent on fresh inputs that were not part of the shared test cases; if a non-trivial share of those pairs produce different outputs, the execution check is too weak to ground the claimed functional consistency, and the benchmark labels are noisy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Embedding-based code comparison can cover functional consistency in one pass per snippet instead of requiring pairwise generative or cross-encoder runs.
  • Existing benchmarks such as POJ-104 underestimate models: fine-tuning on the evolved POJ-Evl data raises functional-consistency F1 from about 0.54 to 0.81 for the evaluated embedding model.
  • The synthesis recipe transfers: applying the same evolution to HumanEval produces a harder benchmark and raises mean F1 from about 0.65 to 0.92 on functional consistency.
  • Evolved datasets also improve clone detection (from about 94.3% to 95.6% MAP@R) and code retrieval (from about 42.3% to 43.4% Acc@50) after task alignment.
  • Because every variant is labeled by execution and CodeBLEU filtering, the pipeline yields reproducible positive and negative pairs across all four type categories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the execution oracle is weak, some positive labels may be false; an audit using held-out inputs or mutation testing of the generated test cases would show whether reported gains reflect true functional understanding or the LLM's rewrite distribution.
  • The four-type synthesis can be repurposed beyond benchmarking: same-function/different-syntax variants are natural refactoring tests, similar-syntax/different-function variants are adversarial inputs for code search, and the paired data can feed contrastive training for code review.
  • The CodeBLEU threshold is tuned per dataset (0.4 for POJ-104, 0.5 for HumanEval); reuse on other languages or programming paradigms likely requires recalibration, and the paper does not quantify sensitivity to that choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Functionality-Oriented Code Self-Evolution, a data synthesis framework that generates code variants across four types defined by syntactic and semantic similarity (Type I–IV). The authors construct two evolved datasets, POJ-Evl and HumanEval-Evl, by prompting LLMs to rewrite or transform code and filtering variants via test-case execution, CodeBLEU thresholds, and limited human review. They evaluate several embedding models on code clone detection, functional consistency identification, and code retrieval, and report that fine-tuning/task alignment on the evolved datasets yields substantial F1 gains, especially on Type III and Type IV negatives, and smaller gains on retrieval and clone detection. The paper includes statistical significance tests and a power analysis.

Significance. If the labels in POJ-Evl and HumanEval-Evl are trustworthy, the paper addresses a real gap: existing code clone datasets are dominated by syntactically similar or trivially different pairs, and the proposed four-type taxonomy is a useful organizing principle for functional consistency evaluation. The empirical results are large and internally consistent for the functional-consistency task, and the authors have released their datasets and implementation, which supports reproducibility. The framework's ability to generate diverse negative samples verified by execution mismatch is a genuine contribution. However, the central claim rests on label correctness and on the absence of confounding from task alignment, and these points currently require additional validation.

major comments (4)
  1. [Section 4.2, Eq. (3)] The labeling rule declares a variant semantically equivalent to the source iff it passes a single unified test suite. Passing finite tests is necessary, not sufficient, for functional equivalence; no held-out inputs or independent reference implementations are used to validate the tests. The 20% human review (Section 3.2.2) is a spot check, not a certification. This unsoundness is concentrated in Type I/II positives, which are already at F1=1.0 in Tables 7 and 9, so the headline Type III/IV gains (which rely on execution mismatches as witnesses of non-equivalence) are more robust. Still, the positive labels and the overall benchmark validity need stronger support, e.g., differential testing against multiple implementations or holdout tests, and an estimate of label noise.
  2. [Tables 9 and 12, Figure 4/5] The functional-consistency comparison is Vanilla Embeddings vs. +POJ-Evl (or +HumanEval-Evl), where the latter includes task alignment on the evolved dataset. No task-alignment-only control is reported for this task. Table 8 includes +Task Alignment on POJ-104 but only for clone detection; Table 13 includes rule-based augmentation but not a pure task-alignment baseline. Consequently, the reported gains could partly reflect the alignment procedure (ICL+instruction) rather than the evolved data content. Please add a control that performs task alignment on the original dataset or on an equally-sized data-augmentation baseline for the functional-consistency task.
  3. [Section 4.4, Tables 11–12] The claimed generalization to HumanEval is not an independent test because HumanEval-Evl is produced by the same pipeline (same LLM prompts, same test-case generation, same CodeBLEU filter). The model may improve on distributional cues of LLM-generated code rather than on functional semantics. The retrieval result on xCodeEval (Table 10) is more independent, but the reported power for that comparison is only 0.34 (Table A.18), so the retrieval evidence is inconclusive. I recommend evaluating on an external, independently annotated functional-consistency benchmark or a human-labeled subset of the evolved data, and reporting the label-agreement rate.
  4. [Section 4.2, Eq. (4) and Section 4.4] The CodeBLEU threshold is a free parameter: θ=0.4 is calibrated on a 100-sample study for POJ-Evl, but θ=0.5 is adopted for HumanEval-Evl without an analogous calibration. The threshold directly controls whether Type IV samples are classified as syntactically similar, which is central to the 'deceptive' negative class. Please provide a sensitivity analysis of the main results with respect to θ, and report the calibration details for HumanEval-Evl.
minor comments (5)
  1. [Table 5] The citation for 'CodeBert + Finetune' is listed as [30], which is CodeLlama; the fine-tuning of CodeBERT should presumably cite [28]. Please correct.
  2. [Table A.18] The power analysis reports Cohen's d and power, but for code retrieval the power is 0.34 and the bootstrap CI is wide. Consider reporting this limitation in the main text and tempering the claim of significant retrieval improvement.
  3. [Section 3.2.2, Eq. (3)] The notation fexec(c, yi) produces a value in {0} or (0,1], but Eq. (3) is described as returning 0 for equivalence and 'otherwise' for non-equivalence; the text around Eq. (3) could be clearer about whether the value is used only as a boolean or as a graded semantic distance.
  4. [Figure 4 vs. Table 9] Figure 4's legend uses 'Task Alignment' for the POJ-Evl condition, while Table 9 calls the same condition '+ POJ-Evl'. This inconsistency could confuse readers; please align the naming.
  5. [Section 4.2] The description of the test-case generation says 'a unified programming example per task' is used as a shared test case. The paper does not specify how the expected outputs for these tests are generated or verified; adding this detail would improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the empirical claims rest on held-out splits and external downstream tasks; shared synthetic-label pipeline is a validity concern, not a circular derivation.

full rationale

I walked the derivation chain. The framework operationally defines semantic equivalence in Eq. 3 as passing LLM-generated test cases, and uses that rule to construct POJ-Evl and HumanEval-Evl. Training and evaluation do share the same synthetic-label pipeline, which raises a legitimate external-validity concern: passing a finite test suite is not sufficient for true functional equivalence, and models may fit distributional cues in LLM-generated code rather than genuine semantics. However, this is not a circularity under the stated patterns. No parameter is fitted to the evaluation labels and then reported as a prediction; downstream functional-consistency results are measured on a held-out 3:1 split of the same dataset, which is standard supervised evaluation. The central claims are additionally supported by externally grounded tasks: code clone detection on POJ-104 (Table 8) and code retrieval on xCodeEval (Table 10), both showing statistically significant gains from the evolved dataset. There are no load-bearing self-citations, no imported uniqueness theorems, no ansatz smuggled in via citation, and no renaming of a known result as a new contribution. The four-type taxonomy is a genuine organizational contribution. The finite-test unsoundness for Type I/II positives is a correctness/benchmark-validity risk, not a claim that reduces by construction to its own inputs; the Limitations section acknowledges instruction sensitivity but not this label-validity issue, which is a completeness gap rather than evidence of circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are introduced. The self-evolution directions are procedural prompts, not postulated entities. The main load is carried by the test-case equivalence assumption and threshold parameters above, which are the true free choices of the framework.

free parameters (2)
  • CodeBLEU threshold theta = 0.4 for POJ-Evl, 0.5 for HumanEval-Evl
    Selected on 100 variants by F1 alignment with two human annotators for POJ-Evl; the same calibration is not performed for HumanEval-Evl, where 0.5 is set without the same evidence. The threshold determines Type I/II vs III/IV syntactic labels.
  • Cosine similarity detection threshold tau = per model (e.g., 0.6 for BGE-M3, 0.4 for CodeBert, 0.3 for UniXcoder)
    Best threshold on BigCloneBench from Table 3 is transferred to POJ-Evl functional-consistency evaluation; results are threshold-dependent, although the paper also reports threshold sweeps.
axioms (5)
  • domain assumption Shared per-task test cases generated by ChatGPT are correct and complete enough to determine functional equivalence.
    Invoked in Section 4.2: 'we prompted the model to summarize the intended functionality of code and generate unified test cases for the entire task.' If the test cases are weak, the labels become noisy.
  • domain assumption Passing identical test outputs implies semantic equivalence, and differing outputs on an executed program implies different functionality.
    Equation 3 defines fexec(c, yi) = 0 as semantic equivalence under test cases. This is a test-based proxy for functional equivalence, not a proof.
  • ad hoc to paper CodeBLEU threshold theta=0.4/0.5 maps to human-perceived syntactic similarity.
    The threshold is fitted on 100 human-labeled variants (F1=0.94) for POJ-Evl and then transferred or reset for HumanEval-Evl; it is not derived from first principles.
  • domain assumption The four-type taxonomy (syntax x semantics) exhaustively and cleanly partitions code pairs.
    Table 1 assumes that every code pair falls into one of Type I-IV based on a binary syntax and semantics distinction; borderline cases are handled by manual filtering.
  • domain assumption Task alignment instructions do not confound the effect of the evolved dataset.
    Downstream evaluations often compare vanilla embeddings to task alignment + evolved data, but the functional consistency tables do not always include a task-alignment-only control, making the isolated dataset effect unclear.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking." pith.science (2026). https://pith.science/paper/3HEUN5KZ

@misc{pith2026250819558,
  author       = {Pith},
  title        = {Pith review of: Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HEUN5KZ}},
  note         = {Machine review of arXiv:2508.19558}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Embedding models have demonstrated strong performance in tasks like clustering, retrieval, and feature extraction while offering computational advantages over generative models and cross-encoders. Benchmarks such as MTEB have shown that text embeddings from large language models (LLMs) capture rich semantic information, but their ability to reflect code-level functional semantics remains unclear. Existing studies largely focus on code clone detection, which emphasizes syntactic similarity and overlooks functional understanding. In this paper, we focus on the functional consistency of LLM code embeddings, which determines if two code snippets perform the same function regardless of syntactic differences. We propose a novel data synthesis framework called Functionality-Oriented Code Self-Evolution to construct diverse and challenging benchmarks. Specifically, we define code examples across four semantic and syntactic categories and find that existing datasets predominantly capture syntactic properties. Our framework generates four unique variations from a single code instance, providing a broader spectrum of code examples that better reflect functional differences. Extensive experiments on three downstream tasks-code clone detection, code functional consistency identification, and code retrieval-demonstrate that embedding models significantly improve their performance when trained on our evolved datasets. These results highlight the effectiveness and generalization of our data synthesis framework, advancing the functional understanding of code.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 39 canonical work pages · 3 internal anchors

  1. [1]

    S ˆ ırbu, G

    A.-G. S ˆ ırbu, G. Czibula, Automatic code generation based on abstract syntax-based encoding. application on malware detec- tion code generation based on mitre att&ck techniques, Ex- pert Systems with Applications 264 (2025) 125821. doi:https: //doi.org/10.1016/j.eswa.2024.125821. URL https://www.sciencedirect.com/science/article/pii/ S0957417424026885

    work page arXiv 2025

  2. [2]

    Sheneamer, S

    A. Sheneamer, S. Roy, J. Kalita, A detection framework for semantic code clones and obfuscated code, Expert Systems with Applications 97 (2018) 405–420. doi:https://doi.org/10.1016/j.eswa.2017.12.040. URL https://www.sciencedirect.com/science/article/pii/ S0957417417308631

    work page doi:10.1016/j.eswa.2017.12.040 2018

  3. [3]

    Martinez-Gil, Source code clone detection using unsupervised sim- ilarity measures, in: International Conference on Software Quality, Springer, 2024, pp

    J. Martinez-Gil, Source code clone detection using unsupervised sim- ilarity measures, in: International Conference on Software Quality, Springer, 2024, pp. 21–37

    work page 2024

  4. [4]

    Sachdev, H

    S. Sachdev, H. Li, S. Luan, S. Kim, K. Sen, S. Chandra, Retrieval on source code: a neural code search, in: Proceedings of the 2nd ACM SIGPLAN international workshop on machine learning and program- ming languages, 2018, pp. 31–41. 24

    work page 2018

  5. [5]

    X. Ling, L. Wu, S. Wang, G. Pan, T. Ma, F. Xu, A. X. Liu, C. Wu, S. Ji, Deep graph matching and searching for semantic code retrieval, ACM Transactions on Knowledge Discovery from Data (TKDD) 15 (5) (2021) 1–21

    work page 2021

  6. [6]

    Sudhamani, L

    M. Sudhamani, L. Rangarajan, Code similarity detection through con- trol statement and program features, Expert Systems with Applications 132 (2019) 63–75. doi:https://doi.org/10.1016/j.eswa.2019.04. 045. URL https://www.sciencedirect.com/science/article/pii/ S0957417419302751

    work page doi:10.1016/j.eswa.2019.04 2019

  7. [7]

    Z. Cui, S. Liu, L. Li, L. Zheng, Seocd: Detecting obsolete code comments by fusing semantic features and expert fea- tures, Expert Systems with Applications 280 (2025) 127470. doi:https://doi.org/10.1016/j.eswa.2025.127470. URL https://www.sciencedirect.com/science/article/pii/ S0957417425010929

    work page arXiv 2025

  8. [8]

    U. Alon, S. Brody, O. Levy, E. Yahav, code2seq: Generating sequences from structured representations of code, in: International Conference on Learning Representations, 2019

    work page 2019

  9. [9]

    U. Alon, M. Zilberstein, O. Levy, E. Yahav, code2vec: Learning dis- tributed representations of code, Proceedings of the ACM on Program- ming Languages 3 (POPL) (2019) 1–29

    work page 2019

  10. [10]

    URL https://openai.com/blog/chatgpt/

    OpenAI, Chatgpt: Optimizing language models for dialogue (2023). URL https://openai.com/blog/chatgpt/

    work page 2023

  11. [11]

    T.-O. Li, W. Zong, Y. Wang, H. Tian, Y. Wang, S.-C. Cheung, J. Kramer, Nuances are the key: Unlocking chatgpt to find failure- inducing tests with differential prompting, in: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, 2023, pp. 14–26

    work page 2023

  12. [12]

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al., Codexglue: A ma- chine learning benchmark dataset for code understanding and genera- tion, arXiv preprint arXiv:2102.04664 (2021). 25

    Pith/arXiv arXiv 2021

  13. [13]

    L. Mou, G. Li, L. Zhang, T. Wang, Z. Jin, Convolutional neural networks over tree structures for programming language processing, in: Proceed- ings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016, pp. 1287–1293

    work page 2016

  14. [14]

    Karimi, L

    A. Karimi, L. Rossi, A. Prati, Aeda: An easier data augmentation tech- nique for text classification, in: Findings of the Association for Compu- tational Linguistics: EMNLP 2021, 2021, pp. 2748–2754

    work page 2021

  15. [15]

    J. Choi, K. Jin, J. Lee, S. Song, Y. Kim, Autoaugment is what you need: Enhancing rule-based augmentation methods in low-resource regimes (2024). arXiv:2402.05584. URL https://arxiv.org/abs/2402.05584

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  16. [16]

    Fowler, Refactoring: improving the design of existing code, Addison- Wesley Professional, 2018

    M. Fowler, Refactoring: improving the design of existing code, Addison- Wesley Professional, 2018

    work page 2018

  17. [17]

    S. Liu, B. Wu, X. Xie, G. Meng, Y. Liu, Contrabert: Enhancing code pre-trained models via contrastive learning, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023, pp. 2476–2487

    work page 2023

  18. [18]

    H. Li, X. Zhou, Z. Shen, Rewriting the code: A simple method for large language model augmented code search, arXiv preprint arXiv:2401.04514 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Cummins, V

    C. Cummins, V. Seeker, J. Armengol-Estap ˜AˇS, A. H. Markosyan, G. Synnaeve, H. Leather, Don’t transform the code, code the trans- forms: Towards precise code rewriting using llms, arXiv preprint arXiv:2410.08806 (2024)

    Pith/arXiv arXiv 2024

  20. [20]

    C. S. Xia, Y. Deng, L. Zhang, Top leaderboard ranking = top cod- ing proficiency, always? evoeval: Evolving coding benchmarks via llm, arXiv preprint (2024)

    work page 2024

  21. [21]

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, D. Jiang, Wizardlm: Empowering large pre-trained language models to follow complex instructions, in: The Twelfth International Conference on Learning Representations, 2023. 26

    work page 2023

  22. [22]

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, D. Jiang, Wizardcoder: Empowering code large language models with evol-instruct, in: The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  23. [23]

    C. Li, M. Qin, S. Xiao, J. Chen, K. Luo, Y. Shao, D. Lian, Z. Liu, Making text embedders few-shot learners (2024). arXiv:2409.15700

    Pith/arXiv arXiv 2024

  24. [24]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka- plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Win- ter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. ...

    Pith/arXiv arXiv 2021

  25. [25]

    DePalma, I

    K. DePalma, I. Miminoshvili, C. Henselder, K. Moss, E. A. AlOmar, Exploring chatgpt’s code refactoring capabilities: An empirical study, Expert Systems with Applications 249 (2024) 123602

    work page 2024

  26. [26]

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, J. M. Zhang, Large language models for software engineering: Survey and open problems, in: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), IEEE, 2023, pp. 31–53

    2023

  27. [27]

    Y. Qu, S. Huang, P. Nie, A review of backdoor attacks and defenses in code large language models: Implications for security measures, Infor- mation and Software Technology (2025) 107707

    work page 2025

  28. [28]

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, CodeBERT: A pre-trained model for pro- gramming and natural languages, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 1536–1547. doi:10.1865...

    doi:10.18653/v1/2020.findings-emnlp.139 2020

  29. [29]

    D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, J. Yin, UniXcoder: Uni- fied cross-modal pre-training for code representation, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. ...

    doi:10.18653/v1/2022.acl-long.499 2022

  30. [30]

    Rozi` ere, J

    B. Rozi` ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. P. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D’efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, G. Synnaeve, Code llama: Open foundation models for code, ArXiv abs/2308.12950 (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al., Qwen2. 5-coder technical report, arXiv preprint arXiv:2409.12186 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Q. Z. Daya Guo, Z. X. Dejian Yang, W. Z. Kai Dong, X. B. Guant- ing Chen, Y. L. Y. Wu, Y. X. Fuli Luo, W. Liang, Deepseek-coder: When the large language model meets programming – the rise of code intelligence (2024)

    work page 2024

  33. [33]

    Zhang, W

    D. Zhang, W. U. Ahmad, M. Tan, H. Ding, R. Nallapati, D. Roth, X. Ma, B. Xiang, CODE REPRESENTATION LEARNING AT SCALE, in: The Twelfth International Conference on Learning Rep- resentations, 2024

    work page 2024

  34. [34]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019)

    Pith/arXiv arXiv 1908

  35. [35]

    Kovaˇ cevi´ c, J

    A. Kovaˇ cevi´ c, J. Slivka, D. Vidakovi´ c, K.-G. Gruji´ c, N. Luburi´ c, S. Proki´ c, G. Sladi´ c, Automatic detection of long method and god class code smells through neural source code embed- dings, Expert Systems with Applications 204 (2022) 117607. doi:https://doi.org/10.1016/j.eswa.2022.117607. URL https://www.sciencedirect.com/science/article/pii/...

    work page arXiv 2022

  36. [36]

    Muennighoff, N

    N. Muennighoff, N. Tazi, L. Magne, N. Reimers, Mteb: Massive text embedding benchmark, arXiv preprint arXiv:2210.07316 (2022). 28

    Pith/arXiv arXiv 2022

  37. [37]

    X. Li, Y. Gong, Y. Shen, X. Qiu, H. Zhang, B. Yao, W. Qi, D. Jiang, W. Chen, N. Duan, CodeRetriever: A large scale contrastive pre- training method for code search, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguis- tics, Abu Dhabi, U...

    doi:10.18653/v1/2022.emnlp-main.187 2022

  38. [38]

    Z. Chen, M. Monperrus, A literature study of embeddings on source code, arXiv preprint arXiv:1904.03061 (2019)

    Pith/arXiv arXiv 1904

  39. [39]

    D. Tian, X. Jia, R. Ma, S. Liu, W. Liu, C. Hu, Bindeep: A deep learning approach to binary code similarity detec- tion, Expert Systems with Applications 168 (2021) 114348. doi:https://doi.org/10.1016/j.eswa.2020.114348. URL https://www.sciencedirect.com/science/article/pii/ S0957417420310332

    work page arXiv 2021

  40. [40]

    W. Gu, Z. Li, C. Gao, C. Wang, H. Zhang, Z. Xu, M. R. Lyu, Cra- dle: Deep code retrieval based on semantic dependency learning, Neural Networks 141 (2021) 385–394

    work page 2021

  41. [41]

    Huang, D

    J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, N. Duan, CoSQA: 20,000+ web queries for code search and question answering, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Lan- guage Processing (Vol...

    doi:10.18653/v1/ 2021

  42. [42]

    M. A. M. Khan, M. S. Bari, X. L. Do, W. Wang, M. R. Parvez, S. Joty, xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval (2023). arXiv:2303.03004

    Pith/arXiv arXiv 2023

  43. [43]

    C. Fang, Z. Liu, Y. Shi, J. Huang, Q. Shi, Functional code clone detec- tion with syntax and semantics fusion learning, in: Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, 2020, pp. 516–527. 29

    work page 2020

  44. [44]

    S. Abid, X. Cai, L. Jiang, Measuring model alignment for code clone detection using causal interpretation, Empirical Software Engineering 30 (2) (2025) 46

    work page 2025

  45. [45]

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, Z. Liu, Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, ArXiv abs/2402.03216 (2024)

    Pith/arXiv arXiv 2024

  46. [46]

    S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, C-pack: Packaged resources to advance general chinese embedding (2023). arXiv:2309.07597

    Pith/arXiv arXiv 2023

  47. [47]

    Code Representation Pre-training with Complements from Program Executions

    J. Huang, J. Zhao, Y. Rong, Y. Guo, Y. He, H. Chen, Code represen- tation pre-training with complements from program executions, arXiv preprint arXiv:2309.09980 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [48]

    Ragkhitwetsagul, J

    C. Ragkhitwetsagul, J. Krinke, D. Clark, A comparison of code similarity analysers, Empirical Software Engineering 23 (2018) 2464–2519

    work page 2018

  49. [49]

    Kartal, E

    Y. Kartal, E. K. Akdeniz, K. ¨Ozkan, Automating modern code review processes with code similarity measurement, Information and Software Technology 173 (2024) 107490

    work page 2024

  50. [50]

    C. K. Roy, J. R. Cordy, A survey on software clone detection research, Queen’s School of computing TR 541 (115) (2007) 64–68

    work page 2007

  51. [51]

    Svajlenko, J

    J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, M. M. Mia, Towards a big data curated benchmark of inter-project code clones, in: 2014 IEEE International Conference on Software Maintenance and Evolution, IEEE, 2014, pp. 476–480

    work page 2014

  52. [52]

    Nashaat, R

    M. Nashaat, R. Amin, A. H. Eid, R. F. Abdel-Kader, An enhanced transformer-based framework for interpretable code clone detection, Journal of Systems and Software 222 (2025) 112347

    work page 2025

  53. [53]

    Mathai, K

    A. Mathai, K. Sedamaki, D. Das, N. S. Mathews, S. Tamilselvam, S. Chi- malakonda, A. Kumar, Codesam: Source code representation learning by infusing self-attention with multi-code-view graphs, arXiv preprint arXiv:2411.14611 (2024)

    Pith/arXiv arXiv 2024

  54. [54]

    Neelakantan, T

    A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. A. Tezak, J. W. Kim, C. Hallacy, J. Heidecke, P. Shyam, 30 B. Power, T. E. Nekoul, G. Sastry, G. Krueger, D. P. Schnurr, F. P. Such, K. S.-K. Hsu, M. Thompson, T. Khan, T. Sherbakov, J. Jang, P. Welinder, L. Weng, Text and code embeddings by contrastive pre- training, ArXiv abs/...

    Pith/arXiv arXiv 2022

  55. [55]

    D. Song, H. Guo, Y. Zhou, S. Xing, Y. Wang, Z. Song, W. Zhang, Q. Guo, H. Yan, X. Qiu, et al., Code needs comments: Enhancing code llms with comment augmentation, in: Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 13640–13656

    work page 2024

  56. [56]

    Evtikhiev, E

    M. Evtikhiev, E. Bogomolov, Y. Sokolov, T. Bryksin, Out of the bleu: how should we assess quality of the code generation models?, Journal of Systems and Software 203 (2023) 111741

    work page 2023

  57. [57]

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V. Ku- mar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, B. Xiang, ReCode: Robustness evaluation of code genera- tion models, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Lo...

    doi:10.18653/v1/2023.acl-long.773 2023

  58. [58]

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, S. Ma, Codebleu: a method for automatic eval- uation of code synthesis (2020). arXiv:2009.10297

    Pith/arXiv arXiv 2020

  59. [59]

    W. Wang, G. Li, B. Ma, X. Xia, Z. Jin, Detecting code clones with graph neural network and flow-augmented abstract syntax tree, in: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2020, pp. 261–271. 31 Appendix A. In this section, we provide additional experimental results and the code evolution ins...

    work page 2020

  60. [60]

    generate a set of test inputs

    generate a more complex code version which performs ab- solutely different functions/intentions from given code and 2. generate a set of test inputs. \r\n You SHOULD make sure the given code and new code have different functionalities/in- tentions. You can generate new code based on the following Code Evolution Instructions, but do not limit yourself to t...

    work page 1900

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.