Pith. sign in

REVIEW 4 major objections 5 minor 60 references

Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that Functionality-Oriented Code Self-Evolution, which synthesizes four verified code-variant types from a single snippet, makes embedding models clearly better at code functional consistency, clone detection, and retrieva

desk verdict A genuinely useful synthesis framework with a real Type IV gap, but the functional-consistency evaluation is too self-referential to support the strongest claims yet. read the letter →

arxiv 2508.19558 v1 pith:3HEUN5KZ submitted 2025-08-27 cs.SE cs.CLcs.PL

Zhuohao Li , Wenqing Chen , Jianxing Yu , Zhichao Lu This is my paper
classification cs.SEcs.CLcs.PL
keywords codeembeddingsfunctionalconsistencydatasynthesisself-evolutionclonedetectionretrievalLLMbenchmarksemanticsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Embedding models for code are usually judged on clone detection, which mostly rewards syntactic resemblance; this paper argues that the harder question is whether two snippets do the same thing, and that existing benchmarks do not ask it. It introduces Functionality-Oriented Code Self-Evolution, a synthesis framework that takes one code snippet, rewrites it along four evolution directions, and verifies each variant by executing shared test cases and checking a syntax-structure similarity metric. The paper claims that training a code embedding model on these evolved datasets clearly improves functional consistency identification, clone detection, and code retrieval, and that the gains carry over to an independent dataset built from a different source benchmark. A sympathetic reader would care because, if true, this gives a cheap, reusable way to build harder code benchmarks and suggests that embeddings can capture function, not just surface form.

What carries the argument

The named mechanism is Functionality-Oriented Code Self-Evolution: from one source snippet, four evolution directions fix the intended syntactic and semantic change to produce variants matching the four code-pair types. The label-carrying step is a two-filter verification loop: running shared test cases decides semantic equivalence (identical outputs make a positive pair; different outputs make a negative pair), and CodeBLEU, a syntax-structure similarity metric, with a threshold decides syntactic similarity. This evolve-and-verify loop turns arbitrary LLM rewrites into labeled training pairs, and the resulting evolved datasets carry the argument for better embeddings.

What would settle it

Run a sample of variants the pipeline labels as functionally equivalent on fresh inputs that were not part of the shared test cases; if a non-trivial share of those pairs produce different outputs, the execution check is too weak to ground the claimed functional consistency, and the benchmark labels are noisy.

Watch

Extended reading notes

Core claim

The central claim is that functional consistency of code can be made measurable and trainable through generated variants, rather than inferred from syntactic overlap. The framework defines four evolution directions matching four code-pair types: same functionality with similar syntax, same functionality with different syntax, different functionality with different syntax, and different functionality with similar syntax. Each variant is produced by an LLM rewrite, then filtered by executing shared test cases to decide semantic equivalence and by a CodeBLEU threshold to decide syntactic similarity. The paper reports that using the resulting evolved datasets for task alignment improves an embed

Load-bearing premise

The framework's labels are only as correct as the generated test cases: a variant counts as functionally equivalent only if it produces identical outputs on those tests, so incomplete or incorrect tests would make the positive and negative labels unreliable.

Editorial extensions

If this is right

  • Embedding-based code comparison can cover functional consistency in one pass per snippet instead of requiring pairwise generative or cross-encoder runs.
  • Existing benchmarks such as POJ-104 underestimate models: fine-tuning on the evolved POJ-Evl data raises functional-consistency F1 from about 0.54 to 0.81 for the evaluated embedding model.
  • The synthesis recipe transfers: applying the same evolution to HumanEval produces a harder benchmark and raises mean F1 from about 0.65 to 0.92 on functional consistency.
  • Evolved datasets also improve clone detection (from about 94.3% to 95.6% MAP@R) and code retrieval (from about 42.3% to 43.4% Acc@50) after task alignment.
  • Because every variant is labeled by execution and CodeBLEU filtering, the pipeline yields reproducible positive and negative pairs across all four type categories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the execution oracle is weak, some positive labels may be false; an audit using held-out inputs or mutation testing of the generated test cases would show whether reported gains reflect true functional understanding or the LLM's rewrite distribution.
  • The four-type synthesis can be repurposed beyond benchmarking: same-function/different-syntax variants are natural refactoring tests, similar-syntax/different-function variants are adversarial inputs for code search, and the paired data can feed contrastive training for code review.
  • The CodeBLEU threshold is tuned per dataset (0.4 for POJ-104, 0.5 for HumanEval); reuse on other languages or programming paradigms likely requires recalibration, and the paper does not quantify sensitivity to that choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Functionality-Oriented Code Self-Evolution, a data synthesis framework that generates code variants across four types defined by syntactic and semantic similarity (Type I–IV). The authors construct two evolved datasets, POJ-Evl and HumanEval-Evl, by prompting LLMs to rewrite or transform code and filtering variants via test-case execution, CodeBLEU thresholds, and limited human review. They evaluate several embedding models on code clone detection, functional consistency identification, and code retrieval, and report that fine-tuning/task alignment on the evolved datasets yields substantial F1 gains, especially on Type III and Type IV negatives, and smaller gains on retrieval and clone detection. The paper includes statistical significance tests and a power analysis.

Significance. If the labels in POJ-Evl and HumanEval-Evl are trustworthy, the paper addresses a real gap: existing code clone datasets are dominated by syntactically similar or trivially different pairs, and the proposed four-type taxonomy is a useful organizing principle for functional consistency evaluation. The empirical results are large and internally consistent for the functional-consistency task, and the authors have released their datasets and implementation, which supports reproducibility. The framework's ability to generate diverse negative samples verified by execution mismatch is a genuine contribution. However, the central claim rests on label correctness and on the absence of confounding from task alignment, and these points currently require additional validation.

major comments (4)
  1. [Section 4.2, Eq. (3)] The labeling rule declares a variant semantically equivalent to the source iff it passes a single unified test suite. Passing finite tests is necessary, not sufficient, for functional equivalence; no held-out inputs or independent reference implementations are used to validate the tests. The 20% human review (Section 3.2.2) is a spot check, not a certification. This unsoundness is concentrated in Type I/II positives, which are already at F1=1.0 in Tables 7 and 9, so the headline Type III/IV gains (which rely on execution mismatches as witnesses of non-equivalence) are more robust. Still, the positive labels and the overall benchmark validity need stronger support, e.g., differential testing against multiple implementations or holdout tests, and an estimate of label noise.
  2. [Tables 9 and 12, Figure 4/5] The functional-consistency comparison is Vanilla Embeddings vs. +POJ-Evl (or +HumanEval-Evl), where the latter includes task alignment on the evolved dataset. No task-alignment-only control is reported for this task. Table 8 includes +Task Alignment on POJ-104 but only for clone detection; Table 13 includes rule-based augmentation but not a pure task-alignment baseline. Consequently, the reported gains could partly reflect the alignment procedure (ICL+instruction) rather than the evolved data content. Please add a control that performs task alignment on the original dataset or on an equally-sized data-augmentation baseline for the functional-consistency task.
  3. [Section 4.4, Tables 11–12] The claimed generalization to HumanEval is not an independent test because HumanEval-Evl is produced by the same pipeline (same LLM prompts, same test-case generation, same CodeBLEU filter). The model may improve on distributional cues of LLM-generated code rather than on functional semantics. The retrieval result on xCodeEval (Table 10) is more independent, but the reported power for that comparison is only 0.34 (Table A.18), so the retrieval evidence is inconclusive. I recommend evaluating on an external, independently annotated functional-consistency benchmark or a human-labeled subset of the evolved data, and reporting the label-agreement rate.
  4. [Section 4.2, Eq. (4) and Section 4.4] The CodeBLEU threshold is a free parameter: θ=0.4 is calibrated on a 100-sample study for POJ-Evl, but θ=0.5 is adopted for HumanEval-Evl without an analogous calibration. The threshold directly controls whether Type IV samples are classified as syntactically similar, which is central to the 'deceptive' negative class. Please provide a sensitivity analysis of the main results with respect to θ, and report the calibration details for HumanEval-Evl.
minor comments (5)
  1. [Table 5] The citation for 'CodeBert + Finetune' is listed as [30], which is CodeLlama; the fine-tuning of CodeBERT should presumably cite [28]. Please correct.
  2. [Table A.18] The power analysis reports Cohen's d and power, but for code retrieval the power is 0.34 and the bootstrap CI is wide. Consider reporting this limitation in the main text and tempering the claim of significant retrieval improvement.
  3. [Section 3.2.2, Eq. (3)] The notation fexec(c, yi) produces a value in {0} or (0,1], but Eq. (3) is described as returning 0 for equivalence and 'otherwise' for non-equivalence; the text around Eq. (3) could be clearer about whether the value is used only as a boolean or as a graded semantic distance.
  4. [Figure 4 vs. Table 9] Figure 4's legend uses 'Task Alignment' for the POJ-Evl condition, while Table 9 calls the same condition '+ POJ-Evl'. This inconsistency could confuse readers; please align the naming.
  5. [Section 4.2] The description of the test-case generation says 'a unified programming example per task' is used as a shared test case. The paper does not specify how the expected outputs for these tests are generated or verified; adding this detail would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims rest on held-out splits and external downstream tasks; shared synthetic-label pipeline is a validity concern, not a circular derivation.

full rationale

I walked the derivation chain. The framework operationally defines semantic equivalence in Eq. 3 as passing LLM-generated test cases, and uses that rule to construct POJ-Evl and HumanEval-Evl. Training and evaluation do share the same synthetic-label pipeline, which raises a legitimate external-validity concern: passing a finite test suite is not sufficient for true functional equivalence, and models may fit distributional cues in LLM-generated code rather than genuine semantics. However, this is not a circularity under the stated patterns. No parameter is fitted to the evaluation labels and then reported as a prediction; downstream functional-consistency results are measured on a held-out 3:1 split of the same dataset, which is standard supervised evaluation. The central claims are additionally supported by externally grounded tasks: code clone detection on POJ-104 (Table 8) and code retrieval on xCodeEval (Table 10), both showing statistically significant gains from the evolved dataset. There are no load-bearing self-citations, no imported uniqueness theorems, no ansatz smuggled in via citation, and no renaming of a known result as a new contribution. The four-type taxonomy is a genuine organizational contribution. The finite-test unsoundness for Type I/II positives is a correctness/benchmark-validity risk, not a claim that reduces by construction to its own inputs; the Limitations section acknowledges instruction sensitivity but not this label-validity issue, which is a completeness gap rather than evidence of circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The self-evolution directions are procedural prompts, not postulated entities. The main load is carried by the test-case equivalence assumption and threshold parameters above, which are the true free choices of the framework.

free parameters (2)
  • CodeBLEU threshold theta = 0.4 for POJ-Evl, 0.5 for HumanEval-Evl
    Selected on 100 variants by F1 alignment with two human annotators for POJ-Evl; the same calibration is not performed for HumanEval-Evl, where 0.5 is set without the same evidence. The threshold determines Type I/II vs III/IV syntactic labels.
  • Cosine similarity detection threshold tau = per model (e.g., 0.6 for BGE-M3, 0.4 for CodeBert, 0.3 for UniXcoder)
    Best threshold on BigCloneBench from Table 3 is transferred to POJ-Evl functional-consistency evaluation; results are threshold-dependent, although the paper also reports threshold sweeps.
assumptions (5)
  • domain assumption Shared per-task test cases generated by ChatGPT are correct and complete enough to determine functional equivalence.
    Invoked in Section 4.2: 'we prompted the model to summarize the intended functionality of code and generate unified test cases for the entire task.' If the test cases are weak, the labels become noisy.
  • domain assumption Passing identical test outputs implies semantic equivalence, and differing outputs on an executed program implies different functionality.
    Equation 3 defines fexec(c, yi) = 0 as semantic equivalence under test cases. This is a test-based proxy for functional equivalence, not a proof.
  • ad hoc to paper CodeBLEU threshold theta=0.4/0.5 maps to human-perceived syntactic similarity.
    The threshold is fitted on 100 human-labeled variants (F1=0.94) for POJ-Evl and then transferred or reset for HumanEval-Evl; it is not derived from first principles.
  • domain assumption The four-type taxonomy (syntax x semantics) exhaustively and cleanly partitions code pairs.
    Table 1 assumes that every code pair falls into one of Type I-IV based on a binary syntax and semantics distinction; borderline cases are handled by manual filtering.
  • domain assumption Task alignment instructions do not confound the effect of the evolved dataset.
    Downstream evaluations often compare vanilla embeddings to task alignment + evolved data, but the functional consistency tables do not always include a task-alignment-only control, making the isolated dataset effect unclear.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking." pith.science (2026). https://pith.science/paper/3HEUN5KZ

@misc{pith2026250819558,
  author       = {Pith},
  title        = {Pith review of: Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HEUN5KZ}},
  note         = {Machine review of arXiv:2508.19558}
}
read the original abstract

Embedding models have demonstrated strong performance in tasks like clustering, retrieval, and feature extraction while offering computational advantages over generative models and cross-encoders. Benchmarks such as MTEB have shown that text embeddings from large language models (LLMs) capture rich semantic information, but their ability to reflect code-level functional semantics remains unclear. Existing studies largely focus on code clone detection, which emphasizes syntactic similarity and overlooks functional understanding. In this paper, we focus on the functional consistency of LLM code embeddings, which determines if two code snippets perform the same function regardless of syntactic differences. We propose a novel data synthesis framework called Functionality-Oriented Code Self-Evolution to construct diverse and challenging benchmarks. Specifically, we define code examples across four semantic and syntactic categories and find that existing datasets predominantly capture syntactic properties. Our framework generates four unique variations from a single code instance, providing a broader spectrum of code examples that better reflect functional differences. Extensive experiments on three downstream tasks-code clone detection, code functional consistency identification, and code retrieval-demonstrate that embedding models significantly improve their performance when trained on our evolved datasets. These results highlight the effectiveness and generalization of our data synthesis framework, advancing the functional understanding of code.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 39 canonical work pages

  1. [1]

    S ˆ ırbu, G

    A.-G. S ˆ ırbu, G. Czibula, Automatic code generation based on abstract syntax-based encoding. application on malware detec- tion code generation based on mitre att&ck techniques, Ex- pert Systems with Applications 264 (2025) 125821. doi:https: //doi.org/10.1016/j.eswa.2024.125821. URL https://www.sciencedirect.com/science/article/pii/ S0957417424026885

    work page arXiv 2025

  2. [2]

    Sheneamer, S

    A. Sheneamer, S. Roy, J. Kalita, A detection framework for semantic code clones and obfuscated code, Expert Systems with Applications 97 (2018) 405–420. doi:https://doi.org/10.1016/j.eswa.2017.12.040. URL https://www.sciencedirect.com/science/article/pii/ S0957417417308631

    work page doi:10.1016/j.eswa.2017.12.040 2018

  3. [3]

    Martinez-Gil, Source code clone detection using unsupervised sim- ilarity measures, in: International Conference on Software Quality, Springer, 2024, pp

    J. Martinez-Gil, Source code clone detection using unsupervised sim- ilarity measures, in: International Conference on Software Quality, Springer, 2024, pp. 21–37

    work page 2024

  4. [4]

    Sachdev, H

    S. Sachdev, H. Li, S. Luan, S. Kim, K. Sen, S. Chandra, Retrieval on source code: a neural code search, in: Proceedings of the 2nd ACM SIGPLAN international workshop on machine learning and program- ming languages, 2018, pp. 31–41. 24

    work page 2018

  5. [5]

    X. Ling, L. Wu, S. Wang, G. Pan, T. Ma, F. Xu, A. X. Liu, C. Wu, S. Ji, Deep graph matching and searching for semantic code retrieval, ACM Transactions on Knowledge Discovery from Data (TKDD) 15 (5) (2021) 1–21

    work page 2021

  6. [6]

    Sudhamani, L

    M. Sudhamani, L. Rangarajan, Code similarity detection through con- trol statement and program features, Expert Systems with Applications 132 (2019) 63–75. doi:https://doi.org/10.1016/j.eswa.2019.04. 045. URL https://www.sciencedirect.com/science/article/pii/ S0957417419302751

    work page doi:10.1016/j.eswa.2019.04 2019

  7. [7]

    Z. Cui, S. Liu, L. Li, L. Zheng, Seocd: Detecting obsolete code comments by fusing semantic features and expert fea- tures, Expert Systems with Applications 280 (2025) 127470. doi:https://doi.org/10.1016/j.eswa.2025.127470. URL https://www.sciencedirect.com/science/article/pii/ S0957417425010929

    work page arXiv 2025

  8. [8]

    U. Alon, S. Brody, O. Levy, E. Yahav, code2seq: Generating sequences from structured representations of code, in: International Conference on Learning Representations, 2019

    work page 2019

Show all 60 references
  1. [9]

    U. Alon, M. Zilberstein, O. Levy, E. Yahav, code2vec: Learning dis- tributed representations of code, Proceedings of the ACM on Program- ming Languages 3 (POPL) (2019) 1–29

    2019

  2. [10]

    URL https://openai.com/blog/chatgpt/

    OpenAI, Chatgpt: Optimizing language models for dialogue (2023). URL https://openai.com/blog/chatgpt/

    2023

  3. [11]

    T.-O. Li, W. Zong, Y. Wang, H. Tian, Y. Wang, S.-C. Cheung, J. Kramer, Nuances are the key: Unlocking chatgpt to find failure- inducing tests with differential prompting, in: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, 2023, pp. 14–26

    2023

  4. [12]

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al., Codexglue: A ma- chine learning benchmark dataset for code understanding and genera- tion, arXiv preprint arXiv:2102.04664 (2021). 25

    2021 arXiv

  5. [13]

    L. Mou, G. Li, L. Zhang, T. Wang, Z. Jin, Convolutional neural networks over tree structures for programming language processing, in: Proceed- ings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016, pp. 1287–1293

    2016

  6. [14]

    Karimi, L

    A. Karimi, L. Rossi, A. Prati, Aeda: An easier data augmentation tech- nique for text classification, in: Findings of the Association for Compu- tational Linguistics: EMNLP 2021, 2021, pp. 2748–2754

    2021

  7. [15]

    J. Choi, K. Jin, J. Lee, S. Song, Y. Kim, Autoaugment is what you need: Enhancing rule-based augmentation methods in low-resource regimes (2024). arXiv:2402.05584. URL https://arxiv.org/abs/2402.05584

    2024 arXiv

  8. [16]

    Fowler, Refactoring: improving the design of existing code, Addison- Wesley Professional, 2018

    M. Fowler, Refactoring: improving the design of existing code, Addison- Wesley Professional, 2018

    2018

  9. [17]

    S. Liu, B. Wu, X. Xie, G. Meng, Y. Liu, Contrabert: Enhancing code pre-trained models via contrastive learning, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023, pp. 2476–2487

    2023

  10. [18]

    H. Li, X. Zhou, Z. Shen, Rewriting the code: A simple method for large language model augmented code search, arXiv preprint arXiv:2401.04514 (2024)

    2024 arXiv

  11. [19]

    Cummins, V

    C. Cummins, V. Seeker, J. Armengol-Estap ˜AˇS, A. H. Markosyan, G. Synnaeve, H. Leather, Don’t transform the code, code the trans- forms: Towards precise code rewriting using llms, arXiv preprint arXiv:2410.08806 (2024)

    2024 arXiv

  12. [20]

    C. S. Xia, Y. Deng, L. Zhang, Top leaderboard ranking = top cod- ing proficiency, always? evoeval: Evolving coding benchmarks via llm, arXiv preprint (2024)

    2024

  13. [21]

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, D. Jiang, Wizardlm: Empowering large pre-trained language models to follow complex instructions, in: The Twelfth International Conference on Learning Representations, 2023. 26

    2023

  14. [22]

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, D. Jiang, Wizardcoder: Empowering code large language models with evol-instruct, in: The Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    C. Li, M. Qin, S. Xiao, J. Chen, K. Luo, Y. Shao, D. Lian, Z. Liu, Making text embedders few-shot learners (2024). arXiv:2409.15700

    2024 arXiv

  16. [24]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka- plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Win- t...

    2021 arXiv

  17. [25]

    DePalma, I

    K. DePalma, I. Miminoshvili, C. Henselder, K. Moss, E. A. AlOmar, Exploring chatgpt’s code refactoring capabilities: An empirical study, Expert Systems with Applications 249 (2024) 123602

    2024

  18. [26]

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, J. M. Zhang, Large language models for software engineering: Survey and open problems, in: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), IEEE, 2023,...

    2023

  19. [27]

    Y. Qu, S. Huang, P. Nie, A review of backdoor attacks and defenses in code large language models: Implications for security measures, Infor- mation and Software Technology (2025) 107707

    2025

  20. [28]

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, CodeBERT: A pre-trained model for pro- gramming and natural languages, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Ass...

    2020 doi

  21. [29]

    D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, J. Yin, UniXcoder: Uni- fied cross-modal pre-training for code representation, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long...

    2022 doi

  22. [30]

    Rozi` ere, J

    B. Rozi` ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. P. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D’efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, G....

    2023 arXiv

  23. [31]

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al., Qwen2. 5-coder technical report, arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  24. [32]

    Q. Z. Daya Guo, Z. X. Dejian Yang, W. Z. Kai Dong, X. B. Guant- ing Chen, Y. L. Y. Wu, Y. X. Fuli Luo, W. Liang, Deepseek-coder: When the large language model meets programming – the rise of code intelligence (2024)

    2024

  25. [33]

    Zhang, W

    D. Zhang, W. U. Ahmad, M. Tan, H. Ding, R. Nallapati, D. Roth, X. Ma, B. Xiang, CODE REPRESENTATION LEARNING AT SCALE, in: The Twelfth International Conference on Learning Rep- resentations, 2024

    2024

  26. [34]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019)

    1908 arXiv

  27. [35]

    Kovaˇ cevi´ c, J

    A. Kovaˇ cevi´ c, J. Slivka, D. Vidakovi´ c, K.-G. Gruji´ c, N. Luburi´ c, S. Proki´ c, G. Sladi´ c, Automatic detection of long method and god class code smells through neural source code embed- dings, Expert Systems with Applications 204 (2022) 117607. doi:https://doi.org/10...

    2022

  28. [36]

    Muennighoff, N

    N. Muennighoff, N. Tazi, L. Magne, N. Reimers, Mteb: Massive text embedding benchmark, arXiv preprint arXiv:2210.07316 (2022). 28

    2022 arXiv

  29. [37]

    X. Li, Y. Gong, Y. Shen, X. Qiu, H. Zhang, B. Yao, W. Qi, D. Jiang, W. Chen, N. Duan, CodeRetriever: A large scale contrastive pre- training method for code search, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natura...

    2022 doi

  30. [38]

    Z. Chen, M. Monperrus, A literature study of embeddings on source code, arXiv preprint arXiv:1904.03061 (2019)

    1904 arXiv

  31. [39]

    D. Tian, X. Jia, R. Ma, S. Liu, W. Liu, C. Hu, Bindeep: A deep learning approach to binary code similarity detec- tion, Expert Systems with Applications 168 (2021) 114348. doi:https://doi.org/10.1016/j.eswa.2020.114348. URL https://www.sciencedirect.com/science/article/pii/ S0...

    2021

  32. [40]

    W. Gu, Z. Li, C. Gao, C. Wang, H. Zhang, Z. Xu, M. R. Lyu, Cra- dle: Deep code retrieval based on semantic dependency learning, Neural Networks 141 (2021) 385–394

    2021

  33. [41]

    Huang, D

    J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, N. Duan, CoSQA: 20,000+ web queries for code search and question answering, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics a...

    2021 doi

  34. [42]

    M. A. M. Khan, M. S. Bari, X. L. Do, W. Wang, M. R. Parvez, S. Joty, xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval (2023). arXiv:2303.03004

    2023 arXiv

  35. [43]

    C. Fang, Z. Liu, Y. Shi, J. Huang, Q. Shi, Functional code clone detec- tion with syntax and semantics fusion learning, in: Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, 2020, pp. 516–527. 29

    2020

  36. [44]

    S. Abid, X. Cai, L. Jiang, Measuring model alignment for code clone detection using causal interpretation, Empirical Software Engineering 30 (2) (2025) 46

    2025

  37. [45]

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, Z. Liu, Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, ArXiv abs/2402.03216 (2024)

    2024 arXiv

  38. [46]

    S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, C-pack: Packaged resources to advance general chinese embedding (2023). arXiv:2309.07597

    2023 arXiv

  39. [47]

    Huang, J

    J. Huang, J. Zhao, Y. Rong, Y. Guo, Y. He, H. Chen, Code represen- tation pre-training with complements from program executions, arXiv preprint arXiv:2309.09980 (2023)

    2023 arXiv

  40. [48]

    Ragkhitwetsagul, J

    C. Ragkhitwetsagul, J. Krinke, D. Clark, A comparison of code similarity analysers, Empirical Software Engineering 23 (2018) 2464–2519

    2018

  41. [49]

    Kartal, E

    Y. Kartal, E. K. Akdeniz, K. ¨Ozkan, Automating modern code review processes with code similarity measurement, Information and Software Technology 173 (2024) 107490

    2024

  42. [50]

    C. K. Roy, J. R. Cordy, A survey on software clone detection research, Queen’s School of computing TR 541 (115) (2007) 64–68

    2007

  43. [51]

    Svajlenko, J

    J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, M. M. Mia, Towards a big data curated benchmark of inter-project code clones, in: 2014 IEEE International Conference on Software Maintenance and Evolution, IEEE, 2014, pp. 476–480

    2014

  44. [52]

    Nashaat, R

    M. Nashaat, R. Amin, A. H. Eid, R. F. Abdel-Kader, An enhanced transformer-based framework for interpretable code clone detection, Journal of Systems and Software 222 (2025) 112347

    2025

  45. [53]

    Mathai, K

    A. Mathai, K. Sedamaki, D. Das, N. S. Mathews, S. Tamilselvam, S. Chi- malakonda, A. Kumar, Codesam: Source code representation learning by infusing self-attention with multi-code-view graphs, arXiv preprint arXiv:2411.14611 (2024)

    2024 arXiv

  46. [54]

    Neelakantan, T

    A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. A. Tezak, J. W. Kim, C. Hallacy, J. Heidecke, P. Shyam, 30 B. Power, T. E. Nekoul, G. Sastry, G. Krueger, D. P. Schnurr, F. P. Such, K. S.-K. Hsu, M. Thompson, T. Khan, T. Sherbakov, J. Jang, P. Weli...

    2022 arXiv

  47. [55]

    D. Song, H. Guo, Y. Zhou, S. Xing, Y. Wang, Z. Song, W. Zhang, Q. Guo, H. Yan, X. Qiu, et al., Code needs comments: Enhancing code llms with comment augmentation, in: Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 13640–13656

    2024

  48. [56]

    Evtikhiev, E

    M. Evtikhiev, E. Bogomolov, Y. Sokolov, T. Bryksin, Out of the bleu: how should we assess quality of the code generation models?, Journal of Systems and Software 203 (2023) 111741

    2023

  49. [57]

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V. Ku- mar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, B. Xiang, ReCode: Robustness evaluation of code genera- tion models, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Pro- ceedings of the 61st...

    2023 doi

  50. [58]

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, S. Ma, Codebleu: a method for automatic eval- uation of code synthesis (2020). arXiv:2009.10297

    2020 arXiv

  51. [59]

    W. Wang, G. Li, B. Ma, X. Xia, Z. Jin, Detecting code clones with graph neural network and flow-augmented abstract syntax tree, in: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2020, pp. 261–271. 31 Appendix A. In thi...

    2020

  52. [60]

    generate a set of test inputs

    generate a more complex code version which performs ab- solutely different functions/intentions from given code and 2. generate a set of test inputs. \r\n You SHOULD make sure the given code and new code have different functionalities/in- tentions. You can generate new code ba...

    1900

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.