Pith. sign in

REVIEW 2 major objections 1 minor 3 references

Deep learning models for Java code clone detection show limited generalizability to unseen code, with NiCad performing better.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-10 12:53 UTC

load-bearing objection This replication shows DL Java clone detectors lose performance in zero-shot cross-dataset tests and get beaten by NiCad, but unmeasured dataset differences could explain the gap instead of weak generalizability. the 2 major comments →

arxiv 2604.13783 v1 submitted 2026-04-15 cs.SE cs.PL

Zero-shot Evaluation of Deep Learning for Java Code Clone Detection

Thomas S. Heinze This is my paper
classification cs.SE cs.PL
keywords code clone detectiondeep learningzero-shot evaluationgeneralizabilityJavaNiCadreplication studysoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether deep learning based clone detectors can identify Java code clones in code they were not trained on. It replicates five leading models, including CodeBERT and FA-AST+GMN, and tests them by training or fine-tuning on one set of datasets and functionalities while evaluating on different ones. The experiments find that these models struggle to generalize, performing worse than expected on the unseen code. In the same zero-shot tests, the conventional clone detector NiCad outperforms the deep learning approaches. A sympathetic reader would care because many software engineering tasks assume deep learning tools will handle new code reliably, yet here that does not appear to hold.

Core claim

In zero-shot evaluation scenarios for Java code clone detection, where models are trained or fine-tuned on one collection of datasets and functionalities and then assessed on different ones, the deep learning models demonstrate limited generalizability to unseen code. Analysis of the results shows that the conventional detector NiCad achieves higher performance than the deep learning models under these conditions.

What carries the argument

The zero-shot evaluation scenario that trains or fine-tunes clone detection models on specific Java datasets and functionalities then tests them on different ones to assess generalization.

Load-bearing premise

The chosen datasets and functionalities represent truly unseen code for the models without other differences in complexity, size, or distribution explaining the performance gap.

What would settle it

If a replicated deep learning model achieved high clone detection accuracy on the unseen functionalities that matched or exceeded NiCad, that would falsify the limited generalizability finding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Deep learning clone detectors may require retraining on target code to maintain effectiveness in practice.
  • Conventional tools like NiCad remain competitive for detecting clones across different code distributions.
  • High performance numbers reported for deep learning clone detectors on standard benchmarks may not reflect cross-project utility.
  • Semantic clone detection claims for deep learning may depend on similarity between training and test code.
  • Replications of this kind can identify gaps between in-distribution testing and real deployment conditions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The models may be capturing dataset-specific patterns more than general semantic understanding of functionality.
  • Similar generalization limits could appear in other deep learning applications to code analysis tasks.
  • Hybrid detectors that combine deep learning with conventional methods might offer more robust performance across projects.
  • Future experiments could test whether training on more diverse functionality sets reduces the observed gap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript replicates five state-of-the-art DL-based Java code clone detectors (including CodeBERT and FA-AST+GMN) and evaluates them in a zero-shot scenario, training/fine-tuning on one collection of datasets and functionalities while testing on others. It claims that the models exhibit limited generalizability to unseen code and that the conventional NiCad detector outperforms the DL approaches under these conditions.

Significance. If the zero-shot performance gap is shown to arise from limited generalizability rather than uncontrolled distributional shifts, the result would be significant for software engineering research on clone detection. The work merits credit for its empirical replication of multiple DL architectures, direct comparison against a non-DL baseline, and focus on semantic clones; such benchmarks can guide development of more robust models or hybrid detectors.

major comments (2)
  1. [Abstract and experimental evaluation description] The zero-shot evaluation protocol (described in the abstract and presumably detailed in the experimental section) trains/fine-tunes and evaluates on different datasets and functionalities but provides no evidence of matching or stratifying the sets on covariates such as code size, cyclomatic complexity, token statistics, or clone-type distribution. Without such controls, the observed underperformance relative to NiCad cannot be unambiguously attributed to limited generalizability rather than systematic dataset mismatch, which is load-bearing for the central claim.
  2. [Abstract] The abstract states that 'our experiments demonstrate' limited generalizability and NiCad superiority, yet the provided text gives no quantitative details on the exact datasets, performance metrics (e.g., precision/recall/F1), statistical significance tests, or variance across runs. This absence makes it impossible to assess whether the results support the claim at the level required for a replication study.
minor comments (1)
  1. [Methodology] Clarify the exact list of datasets and functionalities used for training versus evaluation, including any preprocessing steps, to support reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We address each major comment below and outline the revisions we will make to strengthen the presentation of our zero-shot evaluation results.

read point-by-point responses

  1. Referee: [Abstract and experimental evaluation description] The zero-shot evaluation protocol (described in the abstract and presumably detailed in the experimental section) trains/fine-tunes and evaluates on different datasets and functionalities but provides no evidence of matching or stratifying the sets on covariates such as code size, cyclomatic complexity, token statistics, or clone-type distribution. Without such controls, the observed underperformance relative to NiCad cannot be unambiguously attributed to limited generalizability rather than systematic dataset mismatch, which is load-bearing for the central claim.

    Authors: We acknowledge that the original manuscript did not provide explicit covariate matching or stratification statistics. To address this concern, we will add a new subsection in the experimental evaluation that reports comparative statistics on code size, cyclomatic complexity, token counts, and clone-type distributions across the training and test sets. This will allow readers to better gauge the degree of distributional shift. At the same time, we maintain that testing on entirely different datasets and functionalities constitutes a meaningful zero-shot scenario for assessing real-world generalizability, as unseen code in practice would exhibit similar variations. The fact that NiCad, which requires no training on these specific sets, outperforms the DL models remains a substantive finding even after accounting for mismatches. revision: yes

  2. Referee: [Abstract] The abstract states that 'our experiments demonstrate' limited generalizability and NiCad superiority, yet the provided text gives no quantitative details on the exact datasets, performance metrics (e.g., precision/recall/F1), statistical significance tests, or variance across runs. This absence makes it impossible to assess whether the results support the claim at the level required for a replication study.

    Authors: We agree that the abstract should be more self-contained with key quantitative indicators. In the revised version we will expand the abstract to name the primary datasets used for training and testing, report representative F1 scores for the DL models versus NiCad, and note that statistical significance and variance across runs are analyzed in the body. The full experimental section already contains the complete tables with precision, recall, F1, run-to-run variance, and significance tests; the abstract revision will simply surface the most salient numbers to support the claims without lengthening the abstract excessively. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical replication study

full rationale

The paper is an empirical replication and zero-shot evaluation of existing DL clone detectors on Java code. It reports performance metrics from training/fine-tuning on one set of datasets and evaluating on others, with no equations, derivations, fitted parameters renamed as predictions, or self-citation chains that bear the central claim. The generalizability conclusion follows directly from the observed performance gaps in the experiments, which are externally falsifiable via the same datasets and models. No load-bearing step reduces to a self-definition or imported ansatz.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical replication study with no free parameters, axioms, or invented entities; relies on standard assumptions of ML evaluation such as dataset representativeness.

pith-pipeline@v0.9.0 · 5434 in / 1030 out tokens · 47821 ms · 2026-05-10T12:53:01.002727+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Zero-shot Evaluation of Deep Learning for Java Code Clone Detection." pith.science (2026). https://pith.science/paper/2604.13783

@misc{pith2026260413783,
  author       = {Pith},
  title        = {Pith review of: Zero-shot Evaluation of Deep Learning for Java Code Clone Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.13783}},
  note         = {Machine review of arXiv:2604.13783}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Deep Learning (DL) is becoming more and more widespread in clone detection, motivated by achieving near-perfect performance for this task. In particular in case of semantic code clones, which share only limited syntax but implement the same or similar functionality, Deep Learning appears to outperform conventional tools. In this paper, we want to investigate the generalizability of DL-based clone detectors for Java. We therefore replicate and evaluate the performance of five state-of-the-art DL-based clone detectors, including Transformers like CodeBERT and single-task models like FA-AST+GMN, in a zero-shot evaluation scenario, where we train/fine-tune and evaluate on different datasets and functionalities. Our experiments demonstrate that the models' generalizability to unseen code is limited. Further analysis reveals that the conventional clone detector NiCad even outperforms the DL-based clone detectors in such a zero-shot evaluation scenario.

Figures

Figures reproduced from arXiv: 2604.13783 by Thomas S. Heinze.

Figure 1
Figure 1. Figure 1: ROC curve for DL-based clone detectors and con [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: ROC curve for DL-based clone detectors and con [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages

  1. [1]

    K., and Chen, T

    Al-Omari, F., Roy, C. K., and Chen, T. (2020). Semanticclonebench: A semantic code clone benchmark using crowd-source knowledge. In IEEE 14th International Workshop on Software Clones, IWSC 2020, London, ON, Canada, February 18, 2020, pages 57–63. IEEE. Alam, A. I., Roy, P. R., Al-Omari, F., Roy, C. K., Roy, B., and Schneider, K. A. (2023). Gptclonebench:...

    work page 2020

  2. [2]

    2024 , howpublished =

    OpenReview.net. Heinze, T. S., Sch ¨afer, A., and Amme, W. (2026). Stonedetector : Conventional and versatile code clone detection for java.J. Syst. Softw., 236:112799. Higo, Y . (2024). Dataset of functionally equivalent java methods and its application to evaluating clone detection tools.IEICE Trans. Inf. Syst., 107(6):751–760. Kamp, M., Kreutzer, P., a...

    work page arXiv 2026

  3. [3]

    Association for Computational Linguis- tics. Wei, H. and Li, M. (2017). Supervised deep fea- tures for software functional clone detection by exploiting lexical and syntactical information in source code. In Sierra, C., editor,Proceedings of the Twenty-Sixth International Joint Confer- ence on Artificial Intelligence, IJCAI 2017, Mel- bourne, Australia, A...

    work page 2017