{"id":"3ea3dd9c-d466-4748-98be-5afae797fa5f","arxiv_id":"2601.20147","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The best token-pruning strategy for code summarization is language-dependent: AST inputs help Java but hurt Python, while function signatures are best for Python.","lead":"This paper tested whether removing pieces of code before training a code-summarization model can cut compute without hurting output, and found the best removal strategy depends on the programming language. The practical upshot: Java models can be fed syntax trees, Python models can be fed function signatures, but the paper's headline numbers are contradicted by one of its own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 9's baseline and AST scores contradict Tables 3/4, so the central cross-language comparison is unsupported; re-running the Java CoderEval evaluation is the decisive check.","rationale":"The reader's central objection is correct and remains the most load-bearing: the inconsistency between Table 9 and Tables 3/4 in the Java CoderEval column directly undermines the abstract's key quantitative claim and the cross-language argument in Section 7. The reader also soundly notes the language-vs-parser confound (JavaLang for Java, Python ast module for Python), which is a genuine, independently important internal-validity threat: even if the tables were reconciled, the claim that 'language semantics' drive the difference would require holding parser behavior constant or documenting what each parser discards. Additional issues—single-run training, entropy as an efficiency proxy, CrystalBLEU leakage risk, swapped table references—are all correctly identified and lower confidence, but the Table 9 contradiction alone justifies a reject/revision verdict. I agree with the reader's assessment and do not see a path to accept the central claim without resolving the numerical inconsistency.","tokens_in":31434,"tokens_out":1429,"duration_ms":29264,"concrete_test":"Take the replication package and re-run the Java CoderEval evaluation for (1) Original FuncomJava baseline and (2) AST-based standalone optimization, using the reported CodeT5+ 220M configuration. Compare the resulting BLEU/ROUGE-L/METEOR to Tables 3, 4, and 9. If the recovered baseline and AST BLEU are 6.32 and 8.67, Table 9 is wrong and the headline 37% claim is valid only relative to 6.32; if they are 12.41 and 9.16, Tables 3/4 are wrong and AST does not improve on Java CoderEval. Also compute the same two conditions on a held-out Java benchmark to confirm which numbers are reproducible.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim—that AST-based optimization yields 37% improvement in Java while Function Signatures excel in Python—depends on cross-language results on the CoderEval benchmark. However, the paper presents inconsistent Java CoderEval numbers: Tables 3 and 4 report Original FuncomJava baseline BLEU 6.32 and AST standalone 8.67, while Table 9 reports baseline 12.41 and AST standalone 9.16 for the same condition. The 37% improvement claimed in the abstract and Section 7/Table 9 cannot both be true if the baseline is 6.32 (8.67 is +37%) or 12.41 (9.16 is -26%). This is not a minor typo: Table 9 is the explicit cross-language comparison and directly supports the headline claim that the optimal strategy is language-dependent. The paper does flag parser choices and proxy metrics as internal-validity threats, but that does not explain the numerical contradiction. Section 5.3.3 also makes claims about AST results on the Mastropaolo benchmark and standalone advantages that rest on the same inconsistent reporting. Until the Java CoderEval numbers are reconciled, the central finding of language-dependent token optimization is not reliably supported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates token-level data optimization for code summarization, comparing AST-based representations, function-signature-based inputs, and CrystalBLEU-guided pruning, optionally cascaded with SIDE-based semantic filtering. Experiments use CodeT5+ (220M) fine-tuned on Funcom Java and Python variants and evaluated on CoderEval, Mastropaolo et al.'s Java benchmark, and a newly introduced PyBench benchmark. The paper claims that the optimal token-reduction strategy is language-dependent: AST gives 37% improvement in Java while degrading in Python, Function Signatures excel in Python but not Java, and CrystalBLEU is robust across languages. It also introduces SIDEpy, a Python-specific semantic alignment metric, and PyBench, a 500-item human-annotated Python benchmark.","tokens_in":31646,"tokens_out":3947,"duration_ms":47404,"significance":"If the central claims held, the paper would make a valuable contribution to data-centric optimization for code summarization, with practical implications for reducing training cost. The paper has notable strengths: it ships a replication package; it validates SIDEpy against human judgments with logistic regression; it applies Holm-Bonferroni corrections and reports effect sizes; and Section 10 explicitly acknowledges multiple internal-validity threats. These strengths, however, are currently undermined by a numerically inconsistent central comparison (Tables 3/4 vs. Table 9) and by unresolved confounds that the paper itself identifies.","major_comments":[{"comment":"The central cross-language claim is unsupported by the reported Java CoderEval numbers. Table 3 gives Original FuncomJava CoderEval BLEU = 6.32 and AST standalone = 8.67, which is a 37.2% improvement and matches the abstract's \"37%\" claim. Table 9, however, lists the same Java baseline as 12.41 and AST standalone as 9.16, which is a 26% degradation. Likewise, Table 4's SIDE baseline is 6.32 and AST SIDE-optimized is 7.04, while Table 9 gives 12.13 and 11.81. ROUGE-L and METEOR also differ (e.g., Java baseline ROUGE-L 34.09 in Table 3 vs. 36.94 in Table 9; METEOR 28.64 vs. 35.35). This is not a minor typo: the 37% improvement claim and the language-dependence conclusion rest on these numbers. The authors must rerun/report the Java CoderEval evaluation consistently and recompute all percentages, effect sizes, and significance tests accordingly.","section":"Table 9 vs. Tables 3/4; abstract; Section 7"},{"comment":"The paper acknowledges that Java ASTs are produced by JavaLang while Python ASTs are produced by Python's ast module, and that parser choices may introduce language-specific biases. This is precisely the confound that threatens the headline \"language-dependent\" conclusion: the observed Java/Python difference could be parser-dependent rather than language-dependent. The acknowledgment alone is insufficient. A concrete control is needed, e.g., using the same parser framework for both languages, or comparing within one language across two parser implementations to estimate parser-induced variation. Without such evidence, the central claim that language semantics, rather than tooling differences, drive the reversal remains a hypothesis.","section":"Section 10 (Internal validity); Sections 5.1 and 7.1"},{"comment":"PyBench is described inconsistently: Section 7.2.1 states the benchmark \"consists of 500 human-annotated instances,\" while the following paragraph says \"This curation process resulted in a final evaluation set of 230 high-quality Python code-summary pairs.\" Since PyBench is a newly introduced benchmark and is used as an evaluation set in Tables 7/8 and in the significance tests of Table 10, the number of instances must be clarified and the reported scores must be tied to the correct dataset. As written, a reader cannot tell which PyBench was evaluated.","section":"Section 7.2.1 and 7.2.2"},{"comment":"The paper states that \"The construction of the CrystalBLEU frequent n-gram list from the training split introduces potential circularity; any leakage from validation/test into that list would bias results.\" This is a load-bearing threat for the CrystalBLEU results, which are used to support the cross-language robustness claim. The manuscript should specify exactly how the frequent n-gram list was computed, what data splits were used, and what leakage checks were performed. Currently the risk is named but not resolved.","section":"Section 10 (Internal validity); Section 5.1"}],"minor_comments":[{"comment":"Typo: \"unoptimized FncomPython dataset\" should be \"FuncomPython.\"","section":"Section 7.1"},{"comment":"In Section 7.2.4 the text refers to \"standalone token-optimization results in Table 8,\" but the standalone Python results are in Table 7; the cascaded results are in Table 8. The table references in the Python section are swapped or otherwise inconsistent.","section":"Section 5.3.3 and 7.2.4"},{"comment":"Table 6's caption says \"Rows with statistically significant predictors (p < 0.05) are shaded\" and the text says \"shaded in black,\" but the in-text description of the shading could be clearer and should be consistent with the actual rendered table.","section":"Section 6.2"},{"comment":"The sentence \"Moreover, by replicating the original experimental setup and releasing our implementation publicly available on GitHub [18],\" is grammatically incomplete and should be finished or merged with the following paragraph.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the Java CoderEval inconsistency between Table 9 and Tables 3/4; this is not a cosmetic discrepancy. If the authors can rerun the Java evaluation and produce consistent numbers, the paper's central claim may become supportable, but as presented the evidence for the headline language-dependence result is unreliable. The paper also leans heavily on the authors' own SIDE/Vitale pipeline and on a human-judgment benchmark co-authored by one of the authors; this is not disqualifying but deserves transparency about independence. I recommend major revision, and I would want to see the corrected Java results and the parser-confound control before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper is not a lightweight submission. The authors ran 16 CodeT5+ fine-tunes across Java and Python, built SIDEpy, released a 500-item human-judged Python benchmark (PyBench), and shipped a replication package. That is real work, and the per-language results are mostly coherent: AST helps in Java, hurts in Python; function signatures reverse that pattern; CrystalBLEU is the robust middle path. If those results survive scrutiny, practitioners get a practical, language-aware compression recipe and some useful tooling.\n\nBut the central cross-language comparison has a load-bearing inconsistency. Table 9 reports Java CoderEval baseline BLEU 12.41 and AST standalone 9.16; Tables 3 and 4 report baseline 6.32 and AST standalone 8.67 for the same condition. Those cannot both be correct. The abstract and Section 7 lean on Table 9 for the 37% AST improvement in Java claim, and 9.16 versus 12.41 is a 26% degradation, not a 37% improvement. This is not a formatting typo; it is the evidence for the paper's main conclusion about language dependence. The authors need to rerun or reconcile the Java CoderEval evaluation.\n\nOther soft spots are real but smaller: single-run training with no seed variance, efficiency measured only through token retention and entropy rather than wall-clock or energy, and a CrystalBLEU pruning list built from the training split with acknowledged leakage risk. There are also swapped table references (Section 7.2.4 points to Table 7 when the cascaded results are in Table 8). The parser-difference threat to the language comparison is mentioned honestly in Section 10, and it is not the main problem here.\n\nOn the strengths side, the per-language tables and the statistical appendix are broadly consistent with the qualitative story. The authors also deserve credit for listing internal-validity threats instead of burying them, and for making code and data available. The reliance on the authors' own SIDE/Vitale pipeline and Crupi benchmark is worth checking, but not disqualifying by itself.\n\nRecommendation: send it to peer review, not desk reject. The artifacts and scope justify referee time. But any acceptance should be conditioned on fixing Table 9 and redoing the Java CoderEval evaluation; without that, the headline cross-language claim is unsupported as written.","headline":"Substantial two-language study with real artifacts, but the central cross-language table contradicts the per-language Java results; fix the Java numbers before anyone leans on the headline claim.","tokens_in":32279,"tokens_out":3657,"would_cite":false,"duration_ms":42595,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the best way to shrink code tokens for summarization depends heavily on the programming language, with ASTs winning in Java, function signatures winning in Python, and which tokens you keep mattering more than how man","keywords":["code summarization","token reduction","abstract syntax tree","function signature","CrystalBLEU","data-centric optimization","SIDE metric","Python benchmark"],"falsifier":"Train the same code summarization models using the same token-reduction pipeline but with a single shared AST extractor applied to both Java and Python (or with AST and signature representations swapped between the two parsers); if the performance reversal disappears, the 'language-dependence' claim is actually parser-dependence.","tokens_in":31218,"feed_emoji":"🔤","tokens_out":1325,"duration_ms":18934,"temperature":0.7,"pith_summary":"The paper tries to establish that token-level data optimization for code summarization is not language-agnostic. It compares three reduction techniques—AST-based representation, function-signature extraction, and CrystalBLEU-guided pruning—on Java and Python, with and without semantic filtering. The central claim is that the optimal strategy flips between languages: ASTs give large performance gains in Java but severe degradation in Python, while function signatures do the opposite. The authors also argue that aggressive token reduction can maintain or even improve summary quality when the right representation is chosen, and that semantic filtering enables even more compression without loss. If correct, practitioners cannot assume a single token-reduction recipe works across languages.","feed_headline":"Token-reduction strategy must follow the language, not the model","feed_subtitle":"ASTs help Java summaries but hurt Python; function signatures win in Python with 83% fewer tokens.","key_machinery":"The paper's central objects are three token-level transformations: abstract syntax tree (AST) token sequences, function signatures (method name, return type, parameters), and CrystalBLEU-guided pruning that removes trivially shared n-grams. These are applied to the training input of a CodeT5+ model, either standalone or cascaded after semantic filtering with the SIDE/SIDEpy coherence metric. The work also introduces SIDEpy, a Python-retrained version of the SIDE semantic alignment metric, and PyBench, a 500-item human-annotated Python evaluation benchmark. The machinery works by showing how each transformation alters the token distribution (measured via Shannon entropy reduction and token re","core_discovery":"The paper's central discovery is a language-dependent reversal in token-reduction effectiveness. In Java, AST-based optimization improves BLEU by about 37% over the baseline while cutting tokens by 56–73%, but the same AST approach in Python degrades performance by up to 49%. Conversely, function-signature extraction performs poorly in Java but is optimal in Python, achieving 83% token reduction with negligible quality loss or even improvement on a harder benchmark. CrystalBLEU-guided pruning provides cross-language robustness, consistently achieving 60–72% token reduction with modest quality trade-offs. The authors interpret these results as evidence that the semantic weight carried by stru","pith_inferences":["A direct extension is that the language-dependence likely extends beyond Java/Python to other languages based on syntax verbosity and naming conventions; for example, C# or Kotlin may behave more like Java, while Ruby or JavaScript (concise, expressive names) may behave more like Python.","The optimal representation may also depend on the downstream task, not just the language: structure-preserving ASTs may favor structural tasks like bug detection, while identifier-preserving signatures may favor intent-inference tasks.","The finding suggests a testable adaptive scheme: dynamically choose the token representation per example based on language and code complexity, rather than applying one strategy corpus-wide.","The parser-choice confounder—Java's AST from java.lang, Python's from the ast module—means the 'language' effect could partly be a parser artifact; replicating with a unified parser would isolate the true source."],"forward_implications":["If correct, code summarization systems should be tuned per language: use ASTs for Java-like verbose languages and function signatures for Python-like concise languages.","Semantic filtering of code-comment pairs before token reduction can enable aggressive compression without quality loss, reducing training compute and energy costs.","Large token reductions (up to 83%) can be applied without hurting summary quality when the retained tokens carry the right semantic information—so data optimization should focus on token selection rather than raw volume.","CrystalBLEU-guided pruning offers a safe, language-robust default when the optimal strategy is unknown.","The SIDEpy metric and PyBench benchmark provide tools for evaluating Python code summarization in a way that better aligns with human judgments."],"fun_headline_variants":["Token pruning: What you drop beats how much you drop","Language flips which code tokens you can safely cut","AST pruning wins in Java, fails in Python","Function signatures trim Python code 83% without loss","Token reduction is language-specific, not model-specific"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The Java-versus-Python difference is caused by language semantics rather than by the particular parsers used to build ASTs and signatures, or by the proxy metrics chosen to measure summary quality.","fun_headline_variants_meta":{"raw":{"variants":["Token pruning: What you drop beats how much you drop","Language flips which code tokens you can safely cut","AST pruning wins in Java, fails in Python","Function signatures trim Python code 83% without loss","Token reduction is language-specific, not model-specific"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1154,"prompt_tokens":743,"completion_tokens":411,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":336}},"tokens_in":487,"tokens_out":411,"duration_ms":4686,"temperature":1.0,"reasoning_tokens":336,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:31:18.559402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same code summarization models using the same token-reduction pipeline but with a single shared AST extractor applied to both Java and Python (or with AST and signature representations swapped between the two parsers); if the performance reversal disappears, the 'language-dependence' claim is actually parser-dependence.","supporting_citations":[],"review_version":1}