{"id":"e3b2c851-077b-4677-9510-1811029866ab","arxiv_id":"2504.12998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A comparison of lightweight ML models for commit message generation finds a cosine-similarity TF-IDF retrieval model reaches BLEU 16.82, below RACE's 25.66, while ChatGPT was preferred in manual evaluation.","lead":"This paper tests several lightweight machine learning models for automatically writing commit messages from code changes, reporting BLEU scores on a standard benchmark. The best result, 16.82, comes from a TF-IDF plus nearest-neighbor retrieval approach, and the authors find that ChatGPT produces better messages than their models in manual reviews, especially for large code diffs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed BLEU superiority over NNGen and other baselines rests on unverified comparability of preprocessing and data split; a 0.4-point margin may not survive exact replication.","rationale":"The reader's weakest assumption is that the experimental setup (split, preprocessing, and BLEU script) matches the prior baselines. This is exactly the most load-bearing issue for the paper's central claim. The paper's own threat-to-validity section does not address this: it discusses data integrity and external validity but never the comparability of BLEU scores across different preprocessing choices. The reliance on a single BLEU table without significance testing is a correctness risk, and the paper itself notes that manual evaluations are subjective. I agree with the reader's conditional verdict: the work is plausible and the code is promised, but the central numeric comparison needs verification. No additional concern beyond the reader's is raised, so the verdict remains CONDITIONAL/UNCHANGED. The proposed concrete test is a direct replication check: rerun both the authors' pipeline and the original baselines on the same split and BLEU script. This would settle whether the 0.4-point margin is real or an artifact of preprocessing mismatch.","tokens_in":9432,"tokens_out":2861,"duration_ms":28641,"concrete_test":"Obtain the authors' Zenodo repository (doi:10.5281/zenodo.10888106) and the original Liu et al. benchmark data and split. Reproduce the reported 16.82 BLEU using the released code. Then, on the identical test split, run the released baseline implementations (or the official NNGen code) with their original preprocessing and BLEU evaluation. If the difference between the TF-IDF model and NNGen is less than 1.0 BLEU point, or if NNGen's score increases above 16.82 when using the original evaluation pipeline, the claimed superiority collapses. Additionally, compute a bootstrap confidence interval over test instances for the 16.82 score; a 95% CI wider than 1.0 would indicate the margin is not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that the TF-IDF cosine-similarity model reaches BLEU 16.82 and thereby surpasses NNGen (16.42), NMT (15.52), and CoDiSum (13.97) on the Liu et al. benchmark. This comparison is only valid if the test set, the candidate/reference tokenization, and the BLEU script are identical to those used for the published baseline scores. The paper says it used the same dataset and the Liu et al. BLEU script, but it also describes a custom preprocessing pipeline: lowercasing, special-character removal, stop-word elimination, lemmatization/stemming, and tokenization before vectorization and evaluation. NNGen, NMT, and CoDiSum each use different tokenization and preprocessing (e.g., NNGen does not lemmatize or remove stop words in the same way; NMT uses subword/word tokenization without stop-word removal). If the authors' preprocessing is applied to both the retrieved messages and the reference commit messages before computing BLEU, the score is not directly comparable to the published baselines, which are computed on the original tokenized references. The margin over NNGen is only 0.40 BLEU points (16.82 vs. 16.42), and the paper provides no error bars, no bootstrap confidence intervals, and no per-instance analysis. Without verification that the preprocessing and split exactly match the baselines' evaluation setup, the headline result that a lightweight model beats several neural approaches is not established. The paper itself does not document the exact split of the Liu et al. dataset or release the specific test indices used, making independent verification difficult.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes lightweight machine learning and NLP models for automated commit message generation, trained on the Java repository dataset used by Liu et al. and Jiang et al. The evaluated approaches are cosine similarity with TF-IDF and nearest neighbors, logistic regression with TF-IDF, word2vec-based similarity and logistic regression, and a PyTorch LSTM. The main quantitative result is a BLEU score of 16.82 for the TF-IDF cosine similarity model, which the authors compare with published scores for RACE, NNGen, NMT, CoDiSum, and other baselines. A small manual evaluation compares the model outputs with ChatGPT-generated messages, with the authors reporting that ChatGPT is best on large diffs. The paper argues that simple, locally runnable models are competitive with more resource-intensive neural approaches.","tokens_in":9835,"tokens_out":4987,"duration_ms":45955,"significance":"If the comparative BLEU result is valid, the paper provides useful evidence that a lightweight TF-IDF retrieval method can be competitive with early neural commit message generators on a public benchmark, at much lower computational cost. The use of a public benchmark, the public code repository, and the use of the Liu et al. BLEU script are concrete strengths that facilitate replication. The significance is tempered, however, by the fact that the winning model is essentially TF-IDF-weighted nearest-neighbor retrieval, which is only a small variant of NNGen, and by the absence of statistical tests, unverified comparability of preprocessing with the published baselines, and a non-blinded manual evaluation. The paper's value is therefore more in the empirical comparison and resource-efficiency argument than in methodological novelty.","major_comments":[{"comment":"The headline comparison with published baselines is not yet established. The paper applies lowercasing, special-character removal, stop-word elimination, lemmatization/stemming, and tokenization before vectorization, but it does not state whether the same normalization is applied to both candidate and reference messages when computing BLEU, nor whether the train/test split is identical to the split used by Liu et al. for NNGen and by Jiang et al. for NMT. Since the margin over NNGen is only 0.40 BLEU points (16.82 vs. 16.42), these uncontrolled choices can easily flip the ranking. The authors should report the exact split, commit the split and preprocessing scripts to the repository, provide bootstrap confidence intervals or other significance measures, and include a sensitivity analysis with and without stop-word removal and lemmatization.","section":"3.3-3.5 and Table 5"},{"comment":"The LSTM result of 0.68 BLEU is so far below the other models that it strongly suggests an implementation flaw rather than a genuine limitation of the architecture. No hyperparameters (embedding size, hidden size, layers, dropout, optimizer, learning rate, epochs, batch size) or decoding procedure are reported, so the result is not reproducible and cannot support the conclusion that LSTM is ineffective. The authors should either repair the pipeline, report full hyperparameters and training details, or remove the LSTM claim from the comparative analysis.","section":"3.5 and Table 4"},{"comment":"The manual evaluation is not blinded, no number of raters is given, no inter-rater agreement is reported, and the sampling procedure is described only as 'a script selected random diffs.' The claims that ChatGPT 'outperformed all ML and NLP models' and that logistic regression matched human understanding rest entirely on this unsubstantiated protocol. The authors should use multiple independent raters, blind the outputs, report agreement statistics, and disclose whether the authors themselves were the evaluators.","section":"3.6 and Section 5 (RQ2/RQ3)"},{"comment":"The messages produced by the cosine similarity model are copied from training-set commit messages, so the 16.82 BLEU result is a retrieval result rather than a generation result. The paper should state this explicitly, discuss the potential for duplicated or near-duplicated diffs between the training and test sets, and clarify the exact technical difference from NNGen (TF-IDF weighting versus bag-of-words) so that readers can judge whether the comparison is meaningful or merely a retuning of the same method.","section":"4.2 and Table 5"}],"minor_comments":[{"comment":"The paper's organization is confusing: the Introduction says Section 4 reviews related work and Section 5 discusses results, but in the actual text Section 5 (Discussion) appears before Section 4 (Related Work). The ordering should be fixed.","section":"1 and 4/5"},{"comment":"Table 3 is a copy-paste of Table 2: the columns still read 'Logistic Regression with TF-IDF' and 'Cosine Similarity with TF-IDF,' even though the table is supposed to compare Word2Vec-based models. The table should be corrected.","section":"Table 3"},{"comment":"The abstract states that the data is split into training, validation, and testing sets, but no validation set is used in the reported results; the authors should clarify whether validation was used for early stopping or hyperparameter selection and how.","section":"Abstract and Section 3.4"},{"comment":"The statement that ChatGPT's BLEU score 'could not be included as it was not released' is unclear, since BLEU can be computed locally from generated outputs. The authors should clarify whether the outputs were not saved or the model was not accessible locally.","section":"Section 5"},{"comment":"The manuscript contains numerous typos and grammatical errors, including 'approch', 'chose', and incomplete sentences in Section 3.3. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The reliability section claims that 'the evaluators are proven to show a significant level of experience and expertise with the Java programming language,' but no evidence or details about the evaluators are provided. This claim should be either substantiated or removed.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The comparability of the BLEU comparison is the central issue, and it is fixable by providing the exact split, preprocessing scripts, and aligned tokenization. The paper appears to be based on a short DMS conference paper, and the copy-paste error in Table 3 and the section ordering suggest a hurried submission; the editor may also wish to verify that the Zenodo repository contains the exact scripts needed to reproduce the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's headline result — BLEU 16.82 for a TF-IDF cosine-similarity nearest-neighbor model — is a hair above NNGen's 16.42, but I would not trust that margin. The authors use a custom preprocessing pipeline (stop-word removal, lemmatization) and never show that their BLEU computation sees the same tokens as the published baselines. Without that verification, the 0.4-point edge is plausibly an artifact. So the main claim that a lightweight model surpasses NNGen is not established.\n\nWhat the paper does well: it's an honest, small empirical study on a standard dataset (Liu et al.'s, originally Jiang et al.'s), with code on Zenodo and a standard BLEU script. That is reproducible, and the paper frames its contribution as efficiency on small hardware, not state-of-art quality. The manual comparison with ChatGPT on large diffs is useful qualitative context, even if it's not a rigorous study.\n\nWhere it's soft: (1) The comparability issue above is real. They say their preprocessing is 'more thorough' than prior work, which is exactly the kind of change that can shift BLEU by more than the observed difference. There are no significance tests or variance estimates. (2) The LSTM score of 0.68 BLEU is implausibly low. That's not a performance limitation; it suggests a broken pipeline (data leakage, short training, or bad tokenization). The authors just call it a limitation, which is not credible. (3) Table 3 is clearly a copy-paste of Table 2 — the header says Word2Vec but the rows say TF-IDF. (4) The manual evaluation is unblinded and likely author-run. They mention it as a threat to validity, but they still use it to draw conclusions about ChatGPT vs. the models.\n\nWho this is for: someone building a baseline for commit message generation, or someone studying how much preprocessing affects retrieval-based methods. It is not a state-of-the-art paper. With heavy revision — fixing the table, computing confidence intervals, clarifying the BLEU preprocessing, and reporting dataset splits — it could be a solid workshop or short-paper contribution. I would accept it for peer review if framed as a replication study, but I would expect the reviewers to demand those fixes.\n\nRegards.","headline":"A thin, reproducible benchmark showing a 0.4 BLEU edge over NNGen that probably doesn't survive matching preprocessing; deserves a workshop-level referee but not a top venue.","tokens_in":10324,"tokens_out":5159,"would_cite":false,"duration_ms":46369,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight TF-IDF cosine-similarity model with nearest-neighbor lookup reaches a BLEU score of 16.82 on the standard commit-message benchmark, beating several earlier neural and retrieval baselines while running on a CPU.","keywords":["commit message generation","code documentation","TF-IDF","cosine similarity","nearest neighbors","BLEU score","lightweight machine learning","large language models"],"falsifier":"Run the released code on the benchmark's unchanged test split with the original preprocessing and the same BLEU script, then recompute NNGen's score under the same conditions; if the cosine model no longer beats NNGen or the 16.82 score does not reproduce, the central quantitative claim fails.","tokens_in":9182,"feed_emoji":"📝","tokens_out":6504,"duration_ms":58361,"temperature":0.7,"pith_summary":"The paper sets out to show that lightweight, locally runnable machine-learning models can generate commit messages from code diffs without the GPU demands of neural translation or retrieval-augmented systems. On the standard Java benchmark, a cosine-similarity model using TF-IDF vectors and a nearest-neighbor lookup achieves a BLEU score of 16.82, the highest among the models the authors built and higher on this metric than NMT, NNGen, CoDiSum, Lucene, and CommitGen. Manual evaluations add a second claim: ChatGPT produces messages that human evaluators prefer over all the ML/NLP models, especially for large diffs, while the lightweight models match human expectations best on small diffs. If correct, the results imply that simple retrieval baselines remain competitive for small changes and that the main advantage of large language models appears where diffs are large.","feed_headline":"Simple cosine model outscores neural nets on commit messages","feed_subtitle":"Lightweight TF-IDF nearest-neighbor model scores 16.82 BLEU, beating NMT and NNGen but not RACE.","key_machinery":"The load-bearing machinery is the pairing of TF-IDF vectorization with cosine similarity and a nearest-neighbor search over training diffs. TF-IDF turns each diff into a weighted bag-of-words vector; the model then finds the most similar training diff and returns the corresponding human-written commit message. No training phase is needed, which is why the method runs in seconds on a CPU. This same retrieval pattern is what lets a simple model exceed several generative approaches on BLEU, since retrieved real messages are well-formed and grounded in actual developer writing.","core_discovery":"The central empirical claim is that on the benchmark dataset of Java code changes, the Cosine Similarity with TF-IDF and Nearest Neighbors model reaches BLEU 16.82, which the authors report as surpassing NNGen, NMT, CoDiSum, Lucene, and CommitGen on the same benchmark while remaining below RACE's 25.66. A second claim is that manual evaluation reverses the ranking for quality: ChatGPT was rated best across diff sizes and clearly outclassed all proposed models on large diffs, whereas models like Logistic Regression with TF-IDF often produced messages closer to the actual ones on small diffs. The paper also claims that these results support using lightweight models in resource-constrained and privacy-sensitive settings, since they run in seconds or minutes on a CPU or a laptop chip.","pith_inferences":["Controlling for preprocessing would test whether the 16.82 score comes from the model itself or from extra stop-word removal and lemmatization; the paper does not isolate this.","A hybrid system that routes small diffs to the lightweight model and large diffs to an LLM would be a natural and testable extension of the paper's own evidence.","The same pipeline could in principle transfer to other languages, but the Java-only dataset means such transfer is unverified and would need new data and preprocessing.","Because the nearest-neighbor method inherits the vocabulary and phrasing of the training set, its ceiling may be tied to how repetitive commit messages are in the repository; the paper does not measure that ceiling."],"forward_implications":["A retrieval-based baseline can outperform several neural generators on BLEU without training, so future commit-message work would benefit from reporting such a baseline for calibrating gains.","The best BLEU score need not correspond to the best human-rated quality; evaluation should combine automatic metrics with manual review of small, medium, and large diffs.","Lightweight models make automatic commit-message generation feasible on laptops and in settings where sending code to external language models is undesirable.","Large diffs remain the territory where LLMs show clear advantage; for small diffs, simpler models are competitive or even preferred.","Model rankings depend on diff size, so a single aggregate BLEU number hides an important per-size pattern."],"supporting_citations":[{"why":"Supplies the benchmark dataset of Java diffs and commit messages, the NNGen baseline, and the BLEU evaluation script the paper reuses.","marker":"[12]"},{"why":"Original source of the dataset and the NMT baseline the paper compares against.","marker":"[9]"},{"why":"Provides the RACE baseline with the highest BLEU score (25.66) in the comparison table.","marker":"[16]"},{"why":"Defines BLEU, the metric used for all quantitative evaluations.","marker":"[15]"},{"why":"Provides the CoDiSum baseline, a tree-based generator the paper's retrieval model outscores.","marker":"[19]"},{"why":"Provides the CommitGen baseline listed in the BLEU comparison.","marker":"[13]"},{"why":"Provides the Lucene retrieval baseline in the BLEU comparison.","marker":"[1]"}],"fun_headline_variants":["Simple TF-IDF model beats neural nets on commit messages","Cosine similarity outscores deep learning for commit messages","Commit messages: lightweight model excels over LSTM and NMT","Old-school TF-IDF surpasses neural baselines on commit messages","For commit messages, simple TF-IDF beats neural networks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison stands on the assumption that the authors' data split, preprocessing, and BLEU calculation match the benchmark used to produce the earlier published scores; if the extra stop-word removal or lemmatization changes the test distribution, the headline ranking could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Simple TF-IDF model beats neural nets on commit messages","Cosine similarity outscores deep learning for commit messages","Commit messages: lightweight model excels over LSTM and NMT","Old-school TF-IDF surpasses neural baselines on commit messages","For commit messages, simple TF-IDF beats neural networks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1808,"prompt_tokens":971,"completion_tokens":837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":755}},"tokens_in":587,"tokens_out":837,"duration_ms":8564,"temperature":1.0,"reasoning_tokens":755,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:16:23.947856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released code on the benchmark's unchanged test split with the original preprocessing and the same BLEU script, then recompute NNGen's score under the same conditions; if the cosine model no longer beats NNGen or the 16.82 score does not reproduce, the central quantitative claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark dataset of Java diffs and commit messages, the NNGen baseline, and the BLEU evaluation script the paper reuses."},{"cited_title":"Jiang, A","cited_arxiv_id":null,"evidence_quote":"Original source of the dataset and the NMT baseline the paper compares against."},{"cited_title":"Papineni, S","cited_arxiv_id":null,"evidence_quote":"Defines BLEU, the metric used for all quantitative evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CoDiSum baseline, a tree-based generator the paper's retrieval model outscores."},{"cited_title":"A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes","cited_arxiv_id":"1704.04856","evidence_quote":"Provides the CommitGen baseline listed in the BLEU comparison."},{"cited_title":"https://lucene.apache.org/","cited_arxiv_id":null,"evidence_quote":"Provides the Lucene retrieval baseline in the BLEU comparison."}],"review_version":1}