{"id":"5812b2d9-c9d8-4420-8bc2-9fa4b6844647","arxiv_id":"2507.04003","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding depth and sibling-index embeddings from abstract syntax trees to CodeBERTa yields small gains on masked language modeling and clone detection, mainly with a weighted-sum integration.","lead":"The authors add tree structure information, such as how deeply a piece of code is nested and its order among its siblings, to a code-understanding AI model called CodeBERTa. The modified model does slightly better on two programming-language tasks, but the gains are small and not always consistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline training protocol is unspecified; if 'Original' is the released CodeBERTa-small checkpoint rather than a same-protocol retrain, all reported gains are confounded.","rationale":"The contribution is empirical: the paper asserts that adding depth and sibling-index embeddings to CodeBERTa yields consistent gains. The load-bearing condition is that the comparison baseline is trained under the same protocol as the variants. Section 4.3 is ambiguous on this point, and Figure 3 explicitly distinguishes an 'original pretrained' model from a 'retrained' model, providing in-paper evidence that the baseline may not share the variants' data or initialization. If that is the case, none of the headline numbers isolate the effect of the proposed embeddings. The reader identified exactly this weakness, and I agree. I would not move the verdict to reject because the method is simple, the Weighted Sum result is plausible, and the necessary control is easy to run; the correct response is to require the authors to specify or rerun the baseline under identical conditions and to report per-seed numbers. Since the reader already assigned CONDITIONAL, my stress-test does not change that verdict.","tokens_in":7483,"tokens_out":6948,"duration_ms":84173,"concrete_test":"Run the 'Original' baseline under exactly the protocol described in Section 4.3 for the Tree-Enhanced variants: same initialization rule, same CodeSearchNet split with the same comment filtering, same three epochs, batch size 32, learning rate 1e-5, and seeds 12345, 550, and 42. Report per-seed MLM loss/accuracy and clone detection accuracy/F1 for this baseline, then recompute the delta to the Weighted Sum variant. If the same-protocol baseline differs from the reported 'Original' row by more than about 0.001 on either task, or if the clone-detection advantage shrinks to within the reported standard deviation, the claimed consistent improvement is not attributable to tree-based positional embeddings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that tree-based positional embeddings improve CodeBERTa. For that claim to hold, the 'Original' row in Tables 1 and 2 must be a clean control: same initialization rule, same data, and same training protocol as the Tree-Enhanced variants. Section 4.3 specifies three epochs, AdamW, learning rate 1e-5, batch size 32, and seeds 12345, 550, and 42, but it never states whether this protocol applies to the baseline or only to the proposed variants. Figure 3's caption compounds the ambiguity by distinguishing an 'original pretrained Transformer (trained on code and comments)' from a 'retrained Transformer (trained exclusively on code without comments)', suggesting that the baseline and the variants may differ in both initialization and training data. If the baseline is simply the released CodeBERTa-small checkpoint, then the reported gaps (MLM loss 0.44388 vs 0.41417; clone detection loss 0.25836 vs 0.21799) mix the effect of tree embeddings with the effect of retraining and data filtering. Additionally, Table 2 already undermines the abstract's 'consistently surpasses': the Sum and Concatenation variants score 0.9159 and 0.9063 versus 0.9173 for the Original on clone detection. The baseline question must be settled before the loss and accuracy improvements can be attributed to depth and sibling-index embeddings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes augmenting CodeBERTa with AST-derived hierarchical positional embeddings, specifically depth and sibling-index embeddings, integrated through three strategies (sum, weighted sum, and concatenation). The authors report masked language modeling (MLM) and clone detection experiments, claiming that the proposed Tree-Enhanced CodeBERTa consistently outperforms the baseline across loss, accuracy, F1, precision, and recall. The key evidence is the weighted-sum configuration achieving MLM loss 0.41417 versus 0.44388 for the baseline, and clone detection loss 0.21799 versus 0.25836.","tokens_in":7783,"tokens_out":2417,"duration_ms":27979,"significance":"If the claims are validated under a clean experimental protocol, the paper would make a modest but useful contribution: a simple, low-overhead way to inject tree structure into a standard Transformer code model, with a clear integration recipe and a small parameter cost (~0.945% overhead). The manuscript is clearly written and the repository link is a positive step for reproducibility. However, the significance is currently limited by the scale of the evaluation (two tasks, one model family, no comparison to prior tree-aware Transformer methods), and the central claim of consistent improvement is not fully supported by the reported numbers, especially in clone detection where two of the three proposed variants underperform the baseline. The conceptual motivation is plausible, but the empirical evidence needs stronger controls and statistical grounding.","major_comments":[{"comment":"The baseline training protocol is not specified. The text states that \"Both Masked Language Modeling (MLM) and clone detection fine-tuning were trained for three epochs\" but does not state whether this applies to the \"Original\" row in Tables 1 and 2 or only to the Tree-Enhanced variants. If the baseline is the released CodeBERTa-small checkpoint, then the reported gaps (MLM loss 0.44388 vs. 0.41417; clone detection loss 0.25836 vs. 0.21799) conflate the effect of the proposed embeddings with the effect of additional training and the switch to code-only data. This is load-bearing: the central claim depends on an apples-to-apples comparison with the same initialization, data, seeds, and training budget. Please specify the exact training protocol for the baseline and, if it was not retrained, provide a same-protocol retrained baseline.","section":"Section 4.3, Tables 1 and 2"},{"comment":"The abstract's claim that Tree-Enhanced CodeBERTa \"consistently surpasses the baseline\" is contradicted by Table 2: the Sum variant (0.9159) and the Concatenation variant (0.9063) both score lower than the Original (0.9173) in clone detection accuracy and F1. Only the Weighted Sum variant improves on the baseline. Please revise the claim to reflect the actual pattern, and discuss why the Sum and Concatenation integration strategies degrade clone detection performance.","section":"Abstract and Section 5.2, Table 2"},{"comment":"The reported differences are not shown to be statistically meaningful. In Table 2, the Weighted Sum accuracy gain over Original is 0.0014, which is within the reported standard deviation bound of 0.002. The statement that standard deviations were \"below 0.002\" is not a substitute for significance testing or per-seed results. Please report per-seed scores, confidence intervals, or a paired significance test (e.g., a paired t-test or Wilcoxon test across the three seeds) for the differences that support the main conclusions.","section":"Section 5, Tables 1 and 2"},{"comment":"The Tree Attention Mask is described as a component of the proposed method, but it is not ablated. The text says the mask was \"selectively applied to special tokens,\" but it is unclear whether the baseline runs also used this mask. If the mask is used in the Tree-Enhanced variants but not in the baseline, then gains attributed to depth and sibling-index embeddings are confounded by the attention mask. Please include an ablation that evaluates the positional embeddings with and without the mask, and specify the mask configuration for every reported model.","section":"Section 4.3 and Section 4.1"}],"minor_comments":[{"comment":"The caption distinguishes an \"original pretrained Transformer (trained on code and comments)\" from a \"retrained Transformer (trained exclusively on code without comments)\", which is ambiguous relative to the tables. Please clarify which of these corresponds to the \"Original\" row in Tables 1 and 2, and whether the retrained model is the same as the proposed model minus the tree embeddings.","section":"Figure 3 caption"},{"comment":"The description of MLM \"pretraining\" on CodeSearchNet for only three epochs is unusual, especially if the starting point is an already pretrained CodeBERTa-small checkpoint. Please clarify whether the model is trained from scratch, continued pretraining, or fine-tuned, and how this relates to the baseline.","section":"Section 4.3"},{"comment":"The Concatenation strategy is said to increase parameter count and computational complexity, but the actual parameter counts for the three variants are not reported. Please provide the parameter counts and, if possible, training time or memory usage for each integration strategy.","section":"Section 4.2 and Table 2"},{"comment":"The notation F(f(x))1 in the recursive definition is confusing because F(f(x)) is a pair and \"1\" is used as a projection. Please use an explicit projection notation, such as depth(parent) or a subscript convention.","section":"Section 3, Equation for F(x)"},{"comment":"Several references have formatting or bibliographic errors: the Husain et al. entry is malformed, the two Tai et al. entries are identical despite being cited with different years (2015a and 2015b), and the PoolC dataset entry lacks author and year information. Please correct these entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the paper is readable, but the experimental section currently does not support the strength of the claims. The baseline ambiguity and the absence of significance testing are fixable within the scope of a revision, so I do not recommend rejection. I would also encourage the editor to ask the authors to make the anonymous repository actually accessible at review time, since the link is central to the reproducibility claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, clear incremental paper. The idea of encoding AST depth and sibling position into transformer positional embeddings is already in the literature the authors cite; the only real twist here is a learnable weighted sum over token, depth, sibling, and positional embeddings, applied to CodeBERTa, plus the claim that it helps on MLM and clone detection. The work is reproducible in principle (anonymized repo, three seeds, hyperparameters), and the limitations section is honest about parser dependence and overhead. Credit where due: the weighted-sum configuration looks like a sensible way to let the model decide how much structure to use, and the t-SNE figure does show cleaner depth-correlated clusters for the tree-enhanced model.\n\nThe soft spots are real and one is load-bearing. The paper never says whether the 'Original' baseline in Tables 1 and 2 is a same-protocol retrain or the released CodeBERTa-small checkpoint. Section 4.3 gives the training protocol for the variants but is silent on the baseline. Figure 3 even distinguishes an 'original pretrained Transformer' from a 'retrained Transformer,' which makes the ambiguity worse. If the baseline was the off-the-shelf checkpoint, then the reported gains mix the effect of tree embeddings with the effect of retraining on filtered data, and the headline numbers mean little. That must be fixed.\n\nSecond, the abstract overclaims. Table 2 shows the Sum and Concatenation variants underperform the baseline on clone detection (0.9159 and 0.9063 vs 0.9173). Only Weighted Sum wins. So 'consistently surpasses' is false as written. The differences are also tiny (<0.01 accuracy) and the paper gives no per-seed numbers or significance tests, only 'sd < 0.002.' That is not enough to distinguish a real effect from noise.\n\nThird, the novelty is thin. Shiv and Quirk, Peng et al., and Oh and Yoo already do tree-based positional encodings; the paper does not compare against any of them. The theoretical section is motivation, not a derivation. None of this is fatal on its own, but together it means the paper is an incremental engineering result with an under-specified evaluation.\n\nVerdict: conditional, leaning useful. The weighted-sum variant might survive a cleaner comparison. This deserves a serious referee, but only with major revision: clarify the baseline, provide per-seed numbers and a paired test, and add at least one comparison to a prior tree-aware transformer. If the baseline turns out to be a checkpoint rather than a retrain, the central claim collapses; if it is a retrain, the paper is a modest but legitimate contribution.","headline":"A modest, clearly-written incremental extension whose weighted-sum variant may help, but the ambiguous baseline and an overclaiming abstract prevent the main result from being trusted as written.","tokens_in":8288,"tokens_out":2283,"would_cite":false,"duration_ms":23594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding AST-derived depth and sibling-index embeddings to CodeBERTa yields consistent improvements over the baseline on masked language modeling and clone detection, with the Weighted Sum configuration best.","keywords":["tree-based positional embeddings","Abstract Syntax Trees","CodeBERTa","code representation learning","masked language modeling","clone detection","hierarchical embeddings","transformer source code models"],"falsifier":"Retrain the plain CodeBERTa-small baseline under the exact Section 4.3 protocol (three epochs, seeds 12345, 550, and 42, batch size 32, learning rate 1e-5) using the same data splits, applying the Tree Attention Mask if it is part of the proposed configuration; if the baseline's MLM loss and clone-detection F1 then match or beat the Weighted Sum results, the reported gains would be artifacts of training budget or the mask rather than the tree embeddings.","tokens_in":7281,"feed_emoji":"🌳","tokens_out":12858,"duration_ms":117950,"temperature":0.7,"pith_summary":"This paper claims that a Transformer model for source code learns better when told where each token sits in the code's Abstract Syntax Tree. The authors add two small learned embedding tables to CodeBERTa, one for token depth in the AST and one for sibling order, and report that the resulting Tree-Enhanced CodeBERTa beats the plain model on masked language modeling and clone detection across loss, accuracy, F1, precision, and recall. If correct, the result means explicit tree structure can be injected into an existing Transformer with under one percent additional parameters and no change to the attention architecture, rather than requiring a specialized tree-based model.","feed_headline":"AST depth and sibling order lift CodeBERTa past baseline on code tasks","feed_subtitle":"Two small AST-derived embedding tables improve MLM and clone-detection scores with under one percent extra parameters.","key_machinery":"The recursive hierarchical position $F(x)=(F(f(x))_1+1, i_x)$ with root $(1,1)$ gives every AST node a depth coordinate and a sibling index. Those two coordinates pass through learned embeddings $h(\\cdot)$ and are combined as $P(x)=\\operatorname{Aggregate}(h(F(x)_1), h(F(x)_2))$, then merged with token, type, and standard positional embeddings by sum, weighted sum, or concatenation with a projection. The Weighted Sum variant, which uses learnable scalar weights over the embedding components, carries the empirical argument because it consistently outperforms the other two integration strategies. A Tree Attention Mask applied to special tokens is also introduced to focus self-attention on structurally relevant tokens.","core_discovery":"The central discovery is that hierarchical positional embeddings derived from ASTs improve both pretraining and fine-tuning of CodeBERTa. On MLM, the Weighted Sum configuration reaches a final loss of 0.41417 versus 0.44388 for the baseline, with accuracy 0.9029 versus 0.8972; on clone detection it reaches a loss of 0.21799 versus 0.25836, with accuracy 0.9187 versus 0.9173. The paper attributes the gains to the model using depth and sibling order as structural cues, with learnable weights that emphasize structure early in training and then let word embeddings dominate. A t-SNE visualization shows the tree-enhanced model's hidden states clustering by AST depth, whereas the baseline clusters mainly by token semantics.","pith_inferences":["Editorial inference: the comparison bundles tree embeddings together with the Tree Attention Mask, so a four-way ablation (mask on/off by embedding on/off) would isolate how much of the gain is structural position versus attention masking.","Editorial inference: transfer to other hierarchical inputs, such as HTML/DOM trees, natural-language parse trees, or mathematical expressions, is the most direct test of whether depth-and-sibling embeddings are a general structural encoding rather than a code-specific trick.","Editorial inference: because all experiments use CodeBERTa-small and three epochs, the open scaling question is whether the gains persist, grow, or shrink on larger models and longer pretraining budgets.","Editorial inference: the paper's own limitations section notes dependence on Tree-Sitter parser quality and extra preprocessing cost, so languages with weaker AST support are the settings where the method is most likely to lose its advantage."],"forward_implications":["Explicit tree structure can be added to an existing transformer with roughly 789,504 extra parameters (about 0.945% of CodeBERTa-small) and no modification to the attention blocks.","Weighted Sum integration is the recommended configuration: it beats Sum and Concatenation on both tasks while keeping the parameter overhead small.","Structural embeddings matter most early in pretraining, after which the learned weights shift emphasis toward word semantics.","On clone detection, tree-enhanced representations better distinguish structurally similar but semantically distinct code pairs, improving accuracy and F1.","If the mechanism generalizes, any transformer over data with a tree parse could receive the same two-table treatment."],"supporting_citations":[{"why":"Supplies the Tree-Sitter parser that produces the ASTs from which depth and sibling indices are extracted.","marker":"tre, 2007"},{"why":"Provides the CodeSearchNet dataset used for masked language modeling pretraining and evaluation.","marker":"Husain et al., 2019"},{"why":"Provides the roughly 600,000 code-pair clone detection dataset used for fine-tuning evaluation.","marker":"PoolC, n.d."},{"why":"Defines the original Transformer positional-embedding mechanism that the paper extends with tree-based position.","marker":"Vaswani et al., 2017"},{"why":"Specifies the RoBERTa-style architecture shared by CodeBERTa-small, the backbone being modified.","marker":"Liu et al., 2019"},{"why":"Defines the masked-language-modeling objective and BERT-style pretraining setup used in the MLM evaluation.","marker":"Devlin et al., 2018"},{"why":"Prior tree positional encoding work that motivates the paper's direct-integration approach and provides the closest comparison.","marker":"Peng et al., 2022"},{"why":"Introduces CSA-Trans, the structure-aware alternative that the paper contrasts with its simpler embedding-layer integration.","marker":"Saeyoon and Shin, 2024"}],"fun_headline_variants":["AST depth and sibling order lift CodeBERTa past baseline","Tree positional embeddings boost CodeBERTa code tasks","Depth and sibling order encode code structure better","CodeBERTa benefits from AST-aware position encoding","Little AST embedding tables give CodeBERTa a boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed advantage assumes the \"Original\" baseline was trained under the same protocol as the tree-enhanced variants, with the same data, three epochs, and the same three seeds, so that the hierarchical embeddings are the only difference.","fun_headline_variants_meta":{"raw":{"variants":["AST depth and sibling order lift CodeBERTa past baseline","Tree positional embeddings boost CodeBERTa code tasks","Depth and sibling order encode code structure better","CodeBERTa benefits from AST-aware position encoding","Little AST embedding tables give CodeBERTa a boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000887,"raw_usage":{"total_tokens":3776,"prompt_tokens":842,"completion_tokens":2934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2861}},"tokens_in":458,"tokens_out":2934,"duration_ms":21815,"temperature":1.0,"reasoning_tokens":2861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:56:54.172974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the plain CodeBERTa-small baseline under the exact Section 4.3 protocol (three epochs, seeds 12345, 550, and 42, batch size 32, learning rate 1e-5) using the same data splits, applying the Tree Attention Mask if it is part of the proposed configuration; if the baseline's MLM loss and clone-detection F1 then match or beat the Weighted Sum results, the reported gains would be artifacts of training budget or the mask rather than the tree embeddings.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior tree positional encoding work that motivates the paper's direct-integration approach and provides the closest comparison."}],"review_version":1}