{"id":"dd398469-5eec-4380-b5bc-70642f274064","arxiv_id":"2412.17094","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A comparative benchmark of four open-source LLMs for code summarization finds Mistral and Phi-3 leading, but the reported score tables contain implausible identical values.","lead":"This thesis compares four open-source language models, Llama-3, Phi-3, Mistral, and Gemma, on code summarization using BLEU and ROUGE-L scores across six programming languages. It claims Mistral and Phi-3 are the top performers, but the reported results contain patterns that raise data integrity concerns.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported BLEU/ROUGE-L scores are not credible: exact duplicates across distinct models and an inconsistent data split undermine the Mistral-best ranking.","rationale":"I read the paper as a bachelor's thesis attempting a comparative evaluation of four open-weight LLMs on the CodeXGLUE code summarization benchmark. The central claim is a ranking: Mistral performs best overall, with Phi-3 second. For that ranking to be credible, the numbers in Section 4.2 must be genuine measurements from the described experiments. The identical scores across different models and languages are not plausible as natural experimental outcomes; two independently fine-tuned models of different architectures producing exactly the same BLEU and ROUGE-L values to two decimals on the same dataset would be a remarkable coincidence, especially when it happens twice in the same table set. The split inconsistency is an additional objective problem: Section 4.1.2 describes a 70-20-10 split, while Section 3.2 reports official CodeXGLUE dataset sizes that do not match such a split. One could charitably treat either issue as a typo, but together, and in the absence of released code, data, error bars, or significance testing, they make the empirical core of the paper unreliable. This is not a matter of disagreement with the broader research consensus; it is a matter of internal credibility. The reader's weakest-assumption analysis identified the same load-bearing premise: the reported BLEU and ROUGE-L scores are trustworthy measurements. I agree with that assessment. The central conclusion cannot be accepted on the evidence provided. If the authors later release artifacts and the numbers reproduce, a conditional acceptance could be reconsidered, but as submitted the rejection stands.","tokens_in":18343,"tokens_out":4645,"duration_ms":41274,"concrete_test":"Obtain the raw prediction files (or rerun the described fine-tuning protocol with the stated hyperparameters on the Python CodeXGLUE subset) and recompute BLEU and ROUGE-L for Gemma-7b and Llama-3-8b. If the two models do not reproduce the exact tie reported in Table 4.3 (BLEU 7.23 and ROUGE-L 18.95), the reported values are artifacts or errors, and the central ranking is unsupported. Also verify whether the split actually used matches the official CodeXGLUE counts or the 70-20-10 split described in Section 4.1.2; if neither is reconciled, the experimental description is internally inconsistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical core of the paper is Section 4.2's score tables, which drive the claim that Mistral is the best overall performer for code summarization. Those tables contain exact duplicate scores across different models and languages: Table 4.3 gives Gemma-7b and Llama-3-8b identical BLEU 7.23 and ROUGE-L 18.95 on Python, and Table 4.5 gives Phi-3-medium and Llama-3-8b identical BLEU 1.37 and ROUGE-L 7.69 on Go. After independent fine-tuning of different architectures on the same data, exact equality to two decimals on both metrics for different models is implausibly coincidental. This suggests copy-pasted values, aggregate estimates, or non-genuine measurements. Additionally, Section 4.1.2 states a 70-20-10 train-test-development split was used, but Section 3.2 reports the official CodeXGLUE counts (e.g., Python 251,820 training, 13,914 dev, 14,918 test), which are incompatible with a 70/20/10 partition of that corpus. No code, data, error bars, or significance tests are provided. Since the ranking conclusion rests entirely on these suspect measurements, the central claim lacks a trustworthy empirical foundation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a comparative evaluation of four open-source large language models (Gemma-7b, Phi-3-medium, Llama-3-8b, and Mistral-7b) for code summarization, using BLEU and ROUGE-L scores on the CodeXGLUE benchmark across six programming languages. The central claim, stated in Section 4.4 and the conclusion, is that Mistral-7b is the best overall performer, with Phi-3-medium also strong, and that practitioners should therefore prefer these models for code summarization. The paper includes background on RNN-, GNN-, and Transformer-based methods, a description of the fine-tuning setup (LoRA, SFTTrainer, specific hyperparameters), and per-language result tables. No code, data, or trained models are released, and no statistical significance testing is reported.","tokens_in":18711,"tokens_out":3500,"duration_ms":28802,"significance":"If the reported measurements were trustworthy, the comparison would be of practical interest to software engineers selecting open-source LLMs for code summarization, and the CodeXGLUE-based evaluation framework could serve as a useful reference. The paper's strengths are its timely research question and the sensible choice of well-known open models and a standard benchmark. However, the empirical core is not reproducible: no artifacts are provided, no error bars or significance tests are given, and the reported tables contain exact duplicate scores across distinct models. The central ranking claim is therefore not supported by the evidence presented as it stands.","major_comments":[{"comment":"The central ranking claim rests entirely on the BLEU and ROUGE-L scores in these tables, but the tables contain exact duplicate values across different models and languages. For example, Table 4.3 lists Gemma-7b and Llama-3-8b both at BLEU 7.23 and ROUGE-L 18.95, and Table 4.5 lists Phi-3-medium and Llama-3-8b both at BLEU 1.37 and ROUGE-L 7.69. After independent fine-tuning of different architectures, exact equality on both metrics is implausibly coincidental, and the paper provides no error bars, confidence intervals, or significance tests to address this. These duplicates undermine the trustworthiness of the measurements on which the paper's conclusion depends.","section":"Section 4.2, Tables 4.3–4.8"},{"comment":"The data split is described inconsistently. Section 4.1.2 states that a 70-20-10 train-test-development split was adopted, while Section 3.2 reports the official CodeXGLUE counts (for Python, 251,820 training, 13,914 dev, and 14,918 test examples), which are incompatible with a 70/20/10 partition of that corpus. Because the exact training and test sets determine the validity of all reported scores, this inconsistency makes the experimental setup irreproducible and prevents an independent check of the results.","section":"Sections 3.2 and 4.1.2"},{"comment":"The 'Final Verdict' asserts that Mistral emerges as the best overall performer, but the presented tables do not support an unambiguous overall ranking. In Table 4.8 (Ruby), Mistral has the lowest BLEU (2.52) and ROUGE-L (7.74) of the four models, and in Table 4.4 (Java) it is not first on BLEU; the method by which per-language scores are aggregated (e.g., unweighted average, specific weighting, or a qualitative judgment) is never defined. Without such a definition, the central conclusion is not derivable from the reported results.","section":"Section 4.4 and Table 4.8"},{"comment":"The paper claims to fine-tune 7B–8B parameter models with LoRA on CodeXGLUE-scale data using the free tier of Google Colab with NVIDIA Tesla K80 GPUs, but provides no runtime, memory, or throughput measurements, and no code or data are released. This description makes the experiments difficult to verify or replicate, especially given the scale of the datasets and models; the absence of any implementation artifact is a load-bearing reproducibility gap for an empirical paper.","section":"Section 4.1.1 and Section 4.1.4"}],"minor_comments":[{"comment":"The hyperparameter table does not include the LoRA rank or alpha, even though Section 4.1.4 states that a LoRA adapter is added; the optimizer is described as 'Adam' in Section 4.1.4 but listed as 'adamw_8bit' in Table 4.2.","section":"Section 4.1.3 and Table 4.2"},{"comment":"The counts in Table 3.1 are rounded (e.g., PHP 241k, 14k, 13k) and use 'Validation' while Section 4.1.2 uses 'development'; the text also gives precise counts (251,820, 13,914, 14,918) that differ from the rounded table entries, which is confusing for readers trying to reconstruct the data.","section":"Section 3.2, Table 3.1"},{"comment":"The abstract mentions 'BLEU 3.1' and 'ROUGE 3.2' and Section 1 lists F1, precision, and accuracy as metrics, but the experiments report only BLEU and ROUGE-L; this inconsistency between the stated and actual evaluation metrics should be resolved.","section":"Abstract and Section 1"},{"comment":"The Limitation section acknowledges that models may require individualized training strategies and that data quality matters, but it does not mention the lack of code/data release or the absence of statistical tests, which are the most pressing limitations for this study.","section":"Section 4.5"},{"comment":"Several references are incomplete or contain formatting errors, e.g., reference [2] lacks a year and reference [29] lists the author as 'G.Team'; these should be corrected for consistency.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an undergraduate thesis posted to arXiv. The exact duplicate scores in the result tables (e.g., identical BLEU and ROUGE-L for two different models in Tables 4.3 and 4.5) and the inconsistent train/test split are serious integrity and reproducibility concerns that cannot be fixed by minor edits. Even with a full rewrite, the empirical basis for the claimed ranking would need to be re-established from scratch, including code and data release and statistical validation. I recommend rejection in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a bachelor's thesis turned arXiv preprint, comparing Llama-3-8b, Phi-3-medium, Mistral-7b, and Gemma-7b on CodeXGLUE code summarization. The model set is not covered by the papers it cites, and running a standard benchmark across six languages is a legitimate thing to do. The fine-tuning description (LoRA, SFTTrainer, one epoch, Colab K80) is specific enough to be reproduced in principle.\n\nThat is where the good news ends. The empirical core does not hold up as reported. In Table 4.3, Gemma-7b and Llama-3-8b post identical BLEU 7.23 and ROUGE-L 18.95 on Python; in Table 4.5, Phi-3-medium and Llama-3-8b both score BLEU 1.37 and ROUGE-L 7.69 on Go. Different architectures, independently fine-tuned, producing exactly the same numbers to two decimals on both metrics is not plausible. Either the values were copied, aggregated, or something else happened that the paper does not explain. No error bars, no significance tests, and no code or data are provided, so there is no way to check.\n\nThe split description is also internally inconsistent. Section 3.2 gives the official CodeXGLUE Python counts (251,820 train / 13,914 dev / 14,918 test), while Section 4.1.2 claims a 70-20-10 train-test-development split. Those are incompatible partitions of the same corpus. This is exactly the kind of detail that makes a reader doubt the rest of the pipeline.\n\nThere are smaller problems too. The literature review is broad but sloppy: CodeT5 is attributed to Nijkamp et al. (that paper is CodeGen), GraphCodeBERT gets the wrong year, and some claims in the related work do not match the cited sources. The prose is full of typos and the thesis formatting is obvious. None of that is fatal on its own, but it does not help.\n\nWho gets value from this? A practitioner who just wants a quick hint about which open model to try might scan the rankings, but I would not trust the numbers without the artifacts. As a research preprint, the central claim—Mistral is best overall—rests entirely on measurements I cannot believe as reported.\n\nMy recommendation: do not send this to peer review in its current form. Desk reject, or if you are feeling generous, return it with a request for the actual generated summaries, the code, and corrected tables. The comparison could be a modest data point for the field, but only after the evidence is made real.","headline":"A routine benchmark comparison whose central ranking is not supported by the reported numbers: duplicate scores and an inconsistent split make the evidence impossible to trust.","tokens_in":19120,"tokens_out":4320,"would_cite":false,"duration_ms":34899,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mistral-7b and Phi-3-medium lead open LLMs at code summarization, the paper claims, with Mistral best overall across six programming languages.","keywords":["Code summarization","Large language models","Comparative analysis","Mistral-7b","Phi-3-medium","BLEU","ROUGE-L","CodeXGLUE"],"falsifier":"Re-run the four fine-tunings on the CodeXGLUE splits with the paper's hyperparameters and recompute BLEU and ROUGE-L for each language. If the new tables do not reproduce the reported ranking — in particular, if another model equals or beats Mistral on JavaScript and PHP, or if identical scores reappear across models — the \"Mistral best overall\" conclusion does not survive.","tokens_in":18127,"feed_emoji":"🤖","tokens_out":5781,"duration_ms":43975,"temperature":0.7,"pith_summary":"The paper sets out to compare four open-source large language models — Llama-3-8b, Phi-3-medium, Mistral-7b, and Gemma-7b — on the task of generating natural-language summaries of source code. After fine-tuning each model on the CodeXGLUE benchmark across six programming languages and scoring the outputs with BLEU and ROUGE-L, the paper claims that Mistral-7b is the best overall performer and that Phi-3-medium is a close and consistent alternative, with Llama-3 and Gemma trailing on most datasets. The practical point is that developers building code summarization or documentation tools can make an informed open-source model choice rather than relying on general-purpose assumptions.","feed_headline":"Mistral-7b tops four open LLMs at code summarization","feed_subtitle":"Fine-tuned on CodeXGLUE, Mistral and Phi-3 beat Llama-3 and Gemma on BLEU and ROUGE-L scores.","key_machinery":"The comparative setup itself is the load-bearing mechanism: a fixed public benchmark (CodeXGLUE, a filtered subset of CodeSearchNet), the same supervised fine-tuning pipeline applied to all four models, and two standard n-gram-based metrics (BLEU and ROUGE-L) computed on the same test splits. This lets the paper turn \"which model is better at code summarization\" into a direct score comparison across models and programming languages. BLEU measures n-gram overlap with reference summaries, while ROUGE-L measures the longest common subsequence and thus rewards fluency and contextual relevance.","core_discovery":"The central discovery, stated in the paper's own terms, is that fine-tuning changes the ranking: among the four open LLMs tested, Mistral-7b produces summaries with the highest combined BLEU and ROUGE-L scores across most programming languages, with especially strong results on JavaScript and PHP, while Phi-3-medium is the best on Ruby and Go. The paper concludes that Mistral and Phi-3 are the top performers and that Llama-3-8b and Gemma-7b generally lag behind. Section 4.4 states this verdict explicitly: \"Mistral emerges as the best overall performer.\"","pith_inferences":["A reader who accepts the ranking should still treat the per-language margins carefully: BLEU and ROUGE-L measure surface overlap, so a model that paraphrases accurately can score lower than one that copies reference phrasing.","The same comparison could be extended to code generation, code repair, or human judgment of summary usefulness; the paper's conclusion is specifically about n-gram match on this benchmark.","Several reported scores are identical across different models on the same dataset (for example, Phi-3-medium and Llama-3-8b share exact Go scores), so independent re-runs with error bars would confirm whether the ranking's fine distinctions are real.","A natural next experiment is to test whether the Mistral advantage persists on project-specific few-shot prompts, which related work suggests can beat fine-tuning for code summarization."],"forward_implications":["For automatic code documentation, Mistral-7b is the paper's recommended default open-source model, with Phi-3-medium as the leading alternative.","Mistral's margin is largest on JavaScript and PHP; Phi-3-medium is the model to choose for Ruby and Go.","Llama-3-8b and Gemma-7b are not the paper's first choice for summarization, even where their individual scores are respectable.","Fine-tuning with a small batch size and a single epoch on CodeXGLUE is sufficient to separate the models, suggesting the ranking can be reproduced on modest hardware."],"supporting_citations":[{"why":"Supplies the CodeXGLUE benchmark, the dataset of code-comment pairs for six languages that all models are fine-tuned and tested on.","marker":"[23]"},{"why":"CodeSearchNet is the source corpus from which CodeXGLUE is filtered, grounding the dataset's provenance.","marker":"[17]"},{"why":"Defines the Llama-3-8b model that the paper fine-tunes and evaluates.","marker":"[4]"},{"why":"Defines the Phi-3-medium model and its training and architecture claims.","marker":"[1]"},{"why":"Defines the Mistral-7b model that the paper identifies as best overall.","marker":"[19]"},{"why":"Defines the Gemma-7b model and its intended lightweight deployment context.","marker":"[30]"}],"fun_headline_variants":["Mistral-7b tops open LLMs in code summarization","Fine-tuned Mistral beats Llama-3, Gemma on code summaries","Mistral and Phi-3 lead open LLMs in code summarization","Mistral-7b wins code summarization test among four LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported BLEU and ROUGE-L tables are accurate measurements from the fine-tuning runs described, with no duplicated or rounded-away scores, so the ranking they produce is a true reflection of model performance.","fun_headline_variants_meta":{"raw":{"variants":["Mistral-7b tops open LLMs in code summarization","Fine-tuned Mistral beats Llama-3, Gemma on code summaries","Mistral and Phi-3 lead open LLMs in code summarization","Mistral-7b wins code summarization test among four LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1299,"prompt_tokens":858,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":474,"tokens_out":441,"duration_ms":3401,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:47:24.122611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four fine-tunings on the CodeXGLUE splits with the paper's hyperparameters and recompute BLEU and ROUGE-L for each language. If the new tables do not reproduce the reported ranking — in particular, if another model equals or beats Mistral on JavaScript and PHP, or if identical scores reappear across models — the \"Mistral best overall\" conclusion does not survive.","supporting_citations":[{"cited_title":"Llama3modelcard,","cited_arxiv_id":null,"evidence_quote":"Defines the Llama-3-8b model that the paper fine-tunes and evaluates."}],"review_version":1}