Pith. sign in

REVIEW 5 major objections 6 minor 4 cited by

This paper claims that standard embedding-based similarity metrics systematically misclassify semantic opposition, that cosine distance—not the learned vectors—causes much of the code-domain failure, and that LLM judges discriminate semanti

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:28 UTC pith:7CU7ZXWN

load-bearing objection The main claim about Euclidean distance beating cosine on code embeddings is mathematically impossible under the paper's own Table 3, and the category taxonomy is internally contradictory; the benchmark idea is useful but the results as presented are not interpretable. the 5 major comments →

arxiv 2509.09714 v1 pith:7CU7ZXWN submitted 2025-09-08 cs.CL cs.AI

How Small Transformation Expose the Weakness of Semantic Similarity Measures

Serge Lionel Nikiema , Alb\'erick Euraste Djire , Abdoul Aziz Bonkoungou , Micheline B\'en\'edicte Moumoula , Jordan Samhi , Abdoul Kader Kabore , Jacques Klein , Tegawend\'e F. Bissyande This is my paper
classification cs.CL cs.AI
keywords semantic similaritycode clone detectionembedding metricscosine similarityEuclidean distanceLLM-as-a-judgesemantic oppositioncontrolled transformation benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that when embedding-based similarity metrics go wrong, the failure often sits in the distance function, not in the learned vector representation. It builds a controlled benchmark of 5,068 text and code pairs organised into explicit semantic relationships—equivalent, negated, antonymous, unrelated, cross-language—and runs 18 metrics across them. The headline results are that cosine-based embeddings systematically treat semantic opposites and unrelated content as highly similar: CodeBERT reaches a 99.9% false-positive rate and BERTScore rates opposites above synonyms, while switching to Euclidean distance on identical code embeddings improves difference detection by 24-66%, and LLM judges score genuinely different content low (0.00-0.29) where embeddings score it high (0.82-0.99). If these results hold, many software engineering tools that rely on cosine similarity over code embeddings can be made materially more reliable without retraining a model.

Core claim

On the paper's own terms, the central discovery is a systematic, transformation-based diagnosis of where similarity metrics fail. The authors find three regularities. First, embedding models trained on code and text are largely semantic-difference blind: CodeBERT assigns a similarity of 1.00 to code pairs whose meaning has been altered, BERTScore assigns about 0.95 to negated, antonymous, shuffled, and unrelated sentences, and transformer embeddings average 96.2% false positives across difference categories. Second, a large share of the code-domain failure is a measurement artefact: replacing cosine similarity with Euclidean distance on identical CodeBERT vectors cuts the semantic-difference

What carries the argument

The load-bearing device is a controlled transformation protocol: every evaluation pair is generated by a known edit—synonym or antonym substitution, negation insertion, word reordering, translation, semantic-preserving or semantic-altering code mutation, cross-language port—so the true semantic relationship is fixed by construction rather than inferred. That allows the authors to compute false-positive rates per category and to isolate representation quality from distance computation by holding embeddings fixed and changing only the distance metric (cosine, Euclidean, dot product, Jaccard, Pearson, angular). The Euclidean-versus-cosine comparison is the mechanism that produces the paper's ma

Load-bearing premise

The load-bearing premise is that every subset really instantiates the semantic relationship its name says (equivalent, negated, antonymous, unrelated, cross-language), since all false-positive rates and improvement percentages are computed against those labels.

What would settle it

Recompute the central metrics after re-labelling the text subsets with one consistent mapping (the paper currently gives two different definitions for S4 and reads S2/S3 differently in results), and publish the per-pair labels. If BERTScore's ~0.95 false-positive rates on negated and antonymous pairs drop materially, or the 0.82-0.99 versus 0.00-0.29 gap narrows, the semantic-blindness claim fails; if the numbers hold, it is robust.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Existing code-similarity and clone-detection tools that score CodeBERT or GraphCodeBERT with cosine similarity can improve semantic-difference detection by 24-72% simply by using Euclidean distance on the same vectors, with no retraining.
  • High cosine scores should not be read as evidence of semantic equivalence: for embedding models, scores of 0.82-0.99 are routinely assigned to negated, antonymous, shuffled, and unrelated texts, so threshold-based tools will inherit false positives.
  • LLM-based similarity judges, especially with simple prompts, give a credible low range (0.00-0.29) for genuinely different content, so they offer a materially different regime from embedding metrics for semantic-difference detection.
  • Prompting strategy matters for LLM judges: simple prompts maximise difference detection (average 0.19-0.23), while few-shot prompts maximise consistency (0.84-0.89); temperature in 0.0-1.0 has little effect.
  • Lexical and n-gram metrics such as BLEU and TF-IDF are unreliable for opposition because high token overlap makes negated text look similar; they are safer for syntactic variation than for semantic judgment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper measures pairwise similarity thresholds, so the 24-66% Euclidean gain may not transfer automatically to ranking workloads such as code search; a testable extension is to compare cosine versus Euclidean embeddings on retrieval metrics such as recall@k and mean reciprocal rank.
  • Editorial inference: because the paper's own text taxonomy contains inconsistent definitions (S4 is described both as synonym substitution and as antonym substitution, and the discussion reads Table 2 under different category names), the most valuable next artifact would be a release of all labelled pairs under one canonical mapping; the headline percentages should be read as conditional on those
  • Editorial inference: LLM judges' moderate scores (0.62-0.78) on minimal-edit semantic changes could be used as a calibrated uncertainty signal—systems could route middle-band pairs to human review instead of thresholding everything—though the paper does not itself propose this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a diagnostic framework for evaluating semantic similarity measures by applying controlled transformations to natural language and code and testing 18 metrics. The main claims are that embedding-based metrics are 'semantically blind' (e.g., CodeBERT 99.9% false positives, BERTScore rating opposites above synonyms), that changing from cosine to Euclidean distance improves code embedding performance by 24-66% (37-72% for GraphCodeBERT), that LLM-based judges are far better at detecting semantic differences, and that simple prompts and low temperatures give stable LLM evaluations. The framework includes five code and six/seven text semantic categories, automated test-case validation for code, and human annotation for text.

Significance. The intended contribution---a benchmark that isolates failure modes of similarity metrics---is timely and, if the results were sound, would be practically useful for code search, clone detection, and LLM-based evaluation. The paper has credible strengths: it evaluates a wide range of metrics, applies automated test-based validation for code transformations, and reports human annotation for natural-language pairs. However, the load-bearing claims are currently undermined by (i) contradictory category definitions and inconsistent use of category labels in the analysis, and (ii) a distance-metric claim that is mathematically inconsistent with the reported numbers. The 'distance metric breakthrough' and the quantitative false-positive/false-negative claims cannot be accepted as reported.

major comments (5)
  1. [Section 3.2.2 / Section 4.1] The natural-language taxonomy is internally inconsistent and the analysis applies different labels from the definitions. Section 3.2.2 states six subsets but concludes 'seven well-defined semantic subsets'; S4 is defined twice, once as T_synonym and once as T_antonym; and Section 4.1 assigns S2=negation, S3=antonym, S4=shuffle, S6=unrelated, while the formal definitions assign S2=unrelated, S3=negation, S4=antonym, S6=translation. Because Table 2 and Finding 1 are computed under these labels, every text-domain score, accuracy, and false-positive rate reported for these categories is uninterpretable until the mapping is fixed and the evaluation rerun.
  2. [Section 4.2 / Table 3] The headline Euclidean-distance improvement is inconsistent with the table's own numbers. Dot Product and Cosine Similarity are identical for every row (e.g., CodeBERT S2: 0.999/0.999; S3: 0.966/0.966). This identity holds only for L2-normalized embeddings, for which ||u-v||^2 = 2 - 2cos(u,v), so Euclidean distance is a strictly decreasing function of cosine similarity. Any evaluation invariant to monotone score transformations---correlation, AUC, rank-based metrics, or threshold-calibrated classification---must therefore give identical discrimination for the two. The reported 24-66% 'improvement' can only be a scale/threshold artifact. This claim is the basis of the abstract, Finding 2, and RQ-2, so the central result is unsupported as stated.
  3. [Section 4.1 / Finding 1] The paper reports false-positive rates and F1 scores without defining the decision procedure. Table 1 reports F1 and the text reports '99.9% false positive rates' for CodeBERT and '96.2%±1.8%' for transformer embeddings, but no threshold, classifier, or ROC/AUC analysis is specified anywhere in Section 3.4 or 4.1. A high similarity score alone does not constitute a false positive; a threshold or ranking evaluation is required. The headline numbers therefore lack the methodological support needed to sustain the 'catastrophic failure' claim.
  4. [Equation (1) / Tables 3-4] The evaluation metric defined in Eq. (1) is a correlation with the gold standard, yet Tables 3 and 4 report raw similarity scores and percentage 'improvements' relative to cosine. No correlation values, confidence intervals, or threshold-calibration details are given. It is thus unclear whether the reported 'improvement' is in correlation, in some thresholded accuracy, or in raw score magnitude; these are very different claims. The manuscript should report the actual evaluation measure used for each claim.
  5. [Section 3.2.1 / Section 4.1 (code categories)] The code subset S2 is defined as pairs differing by semantic-altering transformations (operator and control-flow mutations), but Section 4.1 repeatedly calls this 'semantic opposition' and treats it as an opposition-detection task. These are not the same phenomenon: altered behavior is not the same as logical opposition. The interpretation of Table 1's S2 column, and the 'opposition' language in Finding 1, rest on this conflation and need to be either justified or corrected.
minor comments (6)
  1. [Abstract] The abstract's phrasing of the main distance-metric result is ambiguous/contradictory: it first says 'switching from Euclidean distance to cosine similarity improved results by 24 to 66 percent,' while the full abstract and Section 1 say Euclidean improves over cosine. Please correct the direction.
  2. [Section 3.2.2] Five transformations are listed (synonym, negation, antonym, reordering, translation), but the text says 'After applying all four transformations.'
  3. [Section 4.2] The text says 'seven distinct distance metrics' but enumerates six: Cosine, Euclidean, Dot Product, Jaccard, Pearson, Angular.
  4. [Section 4.4] The fourth experiment's finding is numbered 'Finding 3', duplicating the numbering in Section 4.3; the fourth finding should be numbered 4.
  5. [Tables 2 and 4] Table 4's headers S2(Antonym), S3(Negation), S4(Shuffle) use the same mislabeled mapping as Section 4.1 and contradict the definitions in Section 3.2.2.
  6. [References] Some references are duplicated (e.g., [5] and [6] are the same Cha 2007 survey) and should be merged.

Circularity Check

1 steps flagged

Euclidean-vs-cosine 'breakthrough' reduces by construction: Table 3 shows Dot=Cosine, so embeddings are normalized and Euclidean distance is a monotone transform of cosine similarity.

specific steps
  1. self definitional [Section 4.2 (Experiment 2), Table 3; Eq. (1); text-domain analysis paragraph]
    "Euclidean distance and dot product yield identical results (as expected for normalized embeddings), providing negligible improvements (0-3%) over cosine similarity. ... Table 3: Cosine Similarity 0.998 0.992 0.999∗ 0.966∗ 0% / Euclidean Distance 0.688 0.595 0.758 0.331 24-66% / Dot Product 0.998 0.992 0.999∗ 0.966∗ 0%"

    Table 3 reports Dot Product and Cosine Similarity as numerically identical for every model and category, which holds only for L2-normalized embeddings. For normalized vectors, ||u-v||^2 = 2 - 2*cos(u,v), so Euclidean distance is a strictly decreasing function of cosine similarity. Any monotonicity-invariant evaluation (Spearman correlation, AUC, or threshold-calibrated accuracy) must rank pairs identically for both metrics. Eq. (1) defines performance as correlation with the gold standard, which cannot change under a monotone transform of the score. The reported 24-66% improvement is therefore a scale artifact: Euclidean distance scores and cosine similarity scores are compared on different scales, not a property of the distance metric. If the embeddings were not normalized, Dot=Cosine wou

full rationale

No fit-to-prediction circularity is present: the evaluated metrics are scored against independently constructed gold labels, and the LLM-versus-embedding comparisons are empirical and would remain meaningful even if the distance-metric claim failed. There is no load-bearing self-citation; the authors' own prior work (e.g., [40]) is cited only as a method reference, not as justification for a central premise. Constructing gold categories via transformation pipelines is standard benchmark practice and is not circular per se. However, the paper's central 'Distance Metric Breakthrough' is an equivalence-by-construction claim: the paper's own Table 3 shows Dot Product = Cosine Similarity, implying normalized embeddings, under which Euclidean distance is a monotone transformation of cosine similarity. Hence the claimed 24-66% improvement cannot be a genuine property of the distance metric; it is a consequence of comparing raw scores on incompatible scales. Additional internal inconsistencies (six vs seven text subsets, contradictory S4 definitions, and swapped S2/S3/S4 labels in Section 4.1) are correctness and validity threats rather than circularity, but they compound the reliability of the headline numbers. Overall score 6: one central result reduces by construction, while the rest of the evaluation is not circular.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The paper introduces no fitted model and no new physical or theoretical entity. Its numerical claims are experimental. The only hand-set constants are interpretation thresholds. The central assumptions are about the validity of test-suite validation and augmentation-based labels, which is where the paper's inconsistencies concentrate.

free parameters (1)
  • Similarity score thresholds = 0.3 (low), 0.7 (catastrophic), 0.8 (high)
    Hand-set in Section 4.1 and Table 1 notes to label score behavior as acceptable or catastrophic. The F1, accuracy, and 'false positive' interpretations depend on these chosen thresholds.
axioms (3)
  • domain assumption Code that passes its associated test suite after a transformation is functionally equivalent to the original; code that fails is semantically altered.
    Sections 3.2.1 and 3.2.3. Test coverage may be incomplete, so pass/fail does not prove the intended semantic relationship for every pair.
  • domain assumption Text augmentations (synonym, negation, antonym, reordering, translation) produce the intended semantic relationships for all pairs.
    Section 3.2.2. Only 163 pairs per subset are human-annotated, so unannotated pairs inherit the augmenter's intended label without individual verification.
  • domain assumption Human annotation of 163 pairs per category at 95% confidence and 10% margin yields a valid gold standard.
    Section 3.2.3. The sample size calculation is not shown, and a sampling margin does not by itself establish label correctness.

pith-pipeline@v1.3.0-alltime-deepseek · 17804 in / 16059 out tokens · 177009 ms · 2026-08-04T23:28:51.082714+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of How Small Transformation Expose the Weakness of Semantic Similarity Measures." pith.science (2026). https://pith.science/paper/7CU7ZXWN

@misc{pith2026250909714,
  author       = {Pith},
  title        = {Pith review of: How Small Transformation Expose the Weakness of Semantic Similarity Measures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CU7ZXWN}},
  note         = {Machine review of arXiv:2509.09714}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This research examines how well different methods measure semantic similarity, which is important for various software engineering applications such as code search, API recommendations, automated code reviews, and refactoring tools. While large language models are increasingly used for these similarity assessments, questions remain about whether they truly understand semantic relationships or merely recognize surface patterns. The study tested 18 different similarity measurement approaches, including word-based methods, embedding techniques, LLM-based systems, and structure-aware algorithms. The researchers created a systematic testing framework that applies controlled changes to text and code to evaluate how well each method handles different types of semantic relationships. The results revealed significant issues with commonly used metrics. Some embedding-based methods incorrectly identified semantic opposites as similar up to 99.9 percent of the time, while certain transformer-based approaches occasionally rated opposite meanings as more similar than synonymous ones. The study found that embedding methods' poor performance often stemmed from how they calculate distances; switching from Euclidean distance to cosine similarity improved results by 24 to 66 percent. LLM-based approaches performed better at distinguishing semantic differences, producing low similarity scores (0.00 to 0.29) for genuinely different meanings, compared to embedding methods that incorrectly assigned high scores (0.82 to 0.99) to dissimilar content.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Detecting Functional Memorization in Code Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    Authors demonstrate functional memorization in code LLMs via counterfactual midtraining comparison on functional equivalence metrics beyond textual overlap.

  2. Residual Risk Analysis in Benign Code: How Far Are We? A Multi-Model Semantic and Structural Similarity Approach

    cs.SE 2026-04 unverdicted novelty 6.0

    Patched functions often remain similar to vulnerable ones, and a new multi-model similarity scoring system identifies residual issues like null pointer dereferences in 61% of high-risk cases from the PrimeVul dataset.

  3. Geometric Configurations of Perturbed Jailbreak Prompts

    cs.CR 2026-07 conditional novelty 5.0

    In six small open-weight LLMs, jailbreak prompts are linearly separable in last-token embeddings by surface form, but not by refusal/compliance behavior.

  4. Can Code Evaluation Metrics Detect Code Plagiarism?

    cs.SE 2026-04 unverdicted novelty 4.0

    Code evaluation metrics like CrystalBLEU perform comparably to dedicated tools such as Dolos and JPlag when ranking plagiarized code pairs across modification levels on open datasets.

Reference graph

Works this paper leans on

57 extracted references · 35 canonical work pages · cited by 4 Pith papers · 4 internal anchors

  1. [1]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    work page 2005

  2. [2]

    Stefan Bellon, Rainer Koschke, Giuliano Antoniol, Jens Krinke, and Ettore Merlo

    work page

  3. [3]

    John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al

    Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al

    work page

  4. [4]

    Stefano Cesare and Yang Xiang. 2010. Control flow graph similarity for malware detection. In2010 IEEE 10th International Conference on Computer and Information Technology. IEEE, 778–783

    work page 2010

  5. [6]

    Sung-Hyuk Cha. 2007. Comprehensive survey on distance/similarity measures between probability density functions.International Journal of Mathematical Models and Methods in Applied Sciences1, 4 (2007), 300–307

    work page 2007

  6. [7]

    Zheng Chen, Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. 2024. Para- phrase Types Elicit Prompt Engineering Capabilities. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 11004–11033. https://aclanthology.org/2024.emnlp-main.617/

    work page 2024

  7. [8]

    James R Cordy and Chanchal K Roy. 2011. The NiCad clone detector. In2011 IEEE 19th International Conference on Program Comprehension. 219–220

    work page 2011

  8. [9]

    DeepSeek. 2024. DeepSeek-V3: Scaling Open-source Language Models with Mixture-of-Experts. https://github.com/deepseek-ai/DeepSeek-V3

    work page 2024

  9. [10]

    Kaustubh Dhole et al. 2021. NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation.arXiv preprint arXiv:2104.08692(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [11]

    Yann Dubois, Rohan Taori, Caglar Gulcehre, Jerry Zhang, Andreas Glaese, Noam Nisan, et al. 2024. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback.arXiv preprint arXiv:2310.01377(2024)

    Pith/arXiv arXiv 2024

  11. [12]

    Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. InPro- ceedings of the 29th ACM/IEEE international conference on Automated software engineering. 313–324

    work page 2014

  12. [13]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Yan, James Feng, and Bing Qin. 2020. CodeBERT: A pre-trained model for programming and natural languages. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1536–1547

    work page 2020

  13. [14]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2020. GraphCodeBERT: Pre- training code representations with data flow.arXiv preprint arXiv:2009.08366 (2020)

    Pith/arXiv arXiv 2020

  14. [15]

    Daya Guo, Shuai Ren, Shuo Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Jian Yao, and Bing Qin. 2021. GraphCodeBERT: Pre-training code representations with data flow. InInternational Conference on Learning Representations

    work page 2021

  15. [16]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Mea- suring coding challenge competence with apps.arXiv preprint arXiv:2105.09938 (2021)

    Pith/arXiv arXiv 2021

  16. [17]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curi- ous Case of Neural Text Degeneration. InInternational Conference on Learning Representations (ICLR). https://arxiv.org/abs/1904.09751

    Pith/arXiv arXiv 2020

  17. [18]

    Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. 2024. The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use.arXiv preprint arXiv:2411.10323(2024). https://arxiv.org/abs/2411.10323

    Pith/arXiv arXiv 2024

  18. [19]

    Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. 2007. DECKARD: Scalable and accurate tree-based detection of code clones. In29th International Conference on Software Engineering (ICSE’07). 96–105

    work page 2007

  19. [20]

    Elmar Juergens, Florian Deissenboeck, Benjamin Hummel, and Stefan Wagner

  20. [21]

    Minghui Kang, Lingming Zhang, and Sarfraz Khurshid. 2019. Assessing the semantic similarity of source code in the presence of syntax errors. InProceed- ings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 310–320

    work page 2019

  21. [22]

    Jens Krinke and Chaiyong Ragkhitwetsagul. 2024. How the Misuse of a Dataset Harmed Semantic Clone Detection.arXiv preprint arXiv:2505.04311(2024)

    Pith/arXiv arXiv 2024

  22. [23]

    Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. 2025. No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding.arXiv preprint arXiv:2503.05061(2025)

    arXiv 2025

  23. [24]

    Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    work page 2004

  24. [25]

    Jorge Martinez-Gil. 2024. Augmenting the Interpretability of GraphCodeBERT for Code Similarity Tasks.arXiv preprint arXiv:2410.05275(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [26]

    George Mathew and Kathryn T Stolee. 2021. Cross-language code search using static and dynamic analyses.arXiv preprint arXiv:2106.09173(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  26. [27]

    Sebastian Nanz and Carlo A Furia. 2015. A comparative study of programming languages in rosetta code. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 778–788

    work page 2015

  27. [28]

    OpenAI. 2023. How should I set the temperature parameter? https://platform.o penai.com/docs/faq/how-should-i-set-the-temperature-parameter Accessed: 2025-07-18

    work page 2023

  28. [29]

    OpenAI. 2024. GPT-4o Technical Report. https://openai.com/index/gpt-4o

    work page 2024

  29. [30]

    OpenAI. 2024. OpenAI text-embedding-3-large model. https://platform.openai. com/docs/guides/embeddings

    work page 2024

  30. [31]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

    2002

  31. [32]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 3982–3992

    work page 2019

  32. [33]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis.arXiv preprint arXiv:2009.10297(2020). 11 NIKIEMA, DJIRE et al

    Pith/arXiv arXiv 2020

  33. [34]

    2009.Detection and analysis of near-miss software clones

    Chanchal Kumar Roy. 2009.Detection and analysis of near-miss software clones. Ph. D. Dissertation. Queen’s University

    work page 2009

  34. [35]

    Chanchal K Roy and James R Cordy. 2007. A survey on software clone detection research. InTechnical Report 541, Queen’s School of Computing

    work page 2007

  35. [36]

    Vaibhav Saini, Jeffrey Svajlenko, Chanchal K Roy, and Cristina V Lopes. 2018. A comparison of code similarity analysers.Empirical Software Engineering23, 4 (2018)

    work page 2018

  36. [37]

    Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval.Information processing & management24, 5 (1988), 513–523

    1988

  37. [38]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation.arXiv preprint arXiv:2004.04696(2020)

    Pith/arXiv arXiv 2020

  38. [39]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. MPNet: Masked and permuted pre-training for language understanding.arXiv preprint arXiv:2004.09297(2020)

    Pith/arXiv arXiv 2020

  39. [40]

    Yewei Song, Cedric Lothritz, Xunzhu Tang, Tegawendé Bissyandé, and Jacques Klein. 2024. Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Comp...

    doi:10.18653/v1/2024.acl-short.3 2024

  40. [41]

    Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. 2024. Is Cosine- Similarity of Embeddings Really About Similarity?. InCompanion Proceedings of the ACM Web Conference 2024 (WWW ’24). https://arxiv.org/abs/2403.05440 arXiv preprint arXiv:2403.05440

    Pith/arXiv arXiv 2024

  41. [42]

    SurveyMonkey. 2024. Sample Size Calculator. https://www.surveymonkey.com /mp/sample-size-calculator/. Accessed: 2025-07-18

    work page 2024

  42. [43]

    Sandeep Tata and Jignesh M Patel. 2007. Estimating the selectivity of tf-idf based cosine similarity predicates.ACM Sigmod Record36, 2 (2007), 7–12

    work page 2007

  43. [44]

    GEM Benchmark Team. 2021. NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation. https://github.com/GEM-benchmark/NL- Augmenter/tree/main. Accessed: 2025-07-18

    work page 2021

  44. [45]

    Patrick von Platen. 2020. How to generate text: using different decoding methods for language generation with Transformers. https://huggingface.co/blog/how- to-generate Accessed: 2025-07-18

    work page 2020

  45. [46]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou

  46. [47]

    Wei Wang, Li Zhang, and Mark Chen. 2025. Exploring the Boundaries Between LLM Code Clone Detection and Code Similarity Assessment on Human and AI-Generated Code.Big Data and Cognitive Computing9, 2 (2025), 41

    work page 2025

  47. [48]

    Yue Wang, Zhiyang Kan, Yao Lin, Shijin Liu, Lidong Li, Long Zhou, Muhao Chen, Ming Zhou, and Duyu Tang. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.arXiv preprint arXiv:2109.00859(2021)

    Pith/arXiv arXiv 2021

  48. [49]

    Morteza Zakeri-Nasrabadi, Saeed Parsa, Mohammad Ramezani, Chanchal Roy, and Masoud Ekhtiarzadeh. 2023. A systematic literature review on source code similarity measurement and clone detection: techniques, applications, and chal- lenges.arXiv preprint arXiv:2306.16171(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  49. [50]

    Tianyi Zhang et al. 2024. Exploring the Impact of Temperature on Large Language Models: Hot or Cold?arXiv preprint arXiv:2506.07295(2024)

    Pith/arXiv arXiv 2024

  50. [52]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi

  51. [53]

    Siyu Zheng, Haotian Liu, Xiang Lin, Yizhong Du, Skyler Zhang, Zhirui Liu, Xiang Li, Xiao Liu, Yizhou Zhang, Yujia Ma, et al. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685(2023). 12

    Pith/arXiv arXiv 2023

  52. [58]

    InInternational Confer- ence on Learning Representations

    BERTScore: Evaluating text generation with BERT. InInternational Confer- ence on Learning Representations

    work page

  53. [2007]

    Comparison and evaluation of clone detection tools.IEEE Transactions on Software Engineering33, 9 (2007), 577–591

    work page 2007

  54. [2015]

    InIEEE International Conference on Software Maintenance and Evolution (ICSME)

    How are functionally similar code clones different?. InIEEE International Conference on Software Maintenance and Evolution (ICSME)

    work page

  55. [2018]

    InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

    Universal Sentence Encoder. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 169– 174

    work page 2018

  56. [2019]

    BERTScore: Evaluating Text Generation with BERT.arXiv preprint arXiv:1904.09675(2019)

    Pith/arXiv arXiv 1904

  57. [2020]

    MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.arXiv preprint arXiv:2002.10957(2020)

    Pith/arXiv arXiv 2002