{"id":"117ed7ee-f08d-43e0-ba37-46461f11d1a8","arxiv_id":"2606.06197","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning RoBERTa-base on SQuAD1.1 gave the best ROUGE-L/BLEU/BERTScore among twelve models, a result consistent with known capabilities.","lead":"Researchers fine-tuned twelve pre-trained language models on the SQuAD question-answering benchmark and report RoBERTa-base as the best performer, with a ROUGE-L of 86.84%. This is a routine fine-tuning comparison with no new method and sparse experimental detail.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Conclusion (Sec. VI) names Albert-base/Bert-base as highest-performing, contradicting the Abstract and Table III where Roberta-base leads; central claim is internally inconsistent.","rationale":"The reader's weakest assumption is hyperparameter fairness across twelve models. That is a legitimate reproducibility concern, but it is speculative: the paper claims an identical setup, and without evidence of an advantage we cannot say it fails. The conclusion contradiction, by contrast, is a concrete, locatable inconsistency in the paper's own reporting of the central claim. The reviewing rule asks us to treat all manuscript passages as evidence; Section VI directly contradicts the Abstract and Table III. The most load-bearing requirement for the central claim is that the paper consistently and correctly identifies which model performed best. That condition fails. My concrete check would settle whether it is a typo or a deeper error. The verdict remains CONDITIONAL because the claim may be salvageable after correcting the conclusion and providing the missing experimental details; a typo alone would not invalidate the reported table. I therefore do not propose changing the reader's conditional verdict.","tokens_in":9220,"tokens_out":6319,"duration_ms":65895,"concrete_test":"Write a script that parses the numeric tables and compares the max of each metric to the model names in the Abstract and Conclusion. If the max row is Roberta-base, the Conclusion is erroneous and the central claim survives pending clarification; if the max is Albert-base or Bert-base (or if other rows exceed Roberta), the Abstract and Table III are wrong and the central claim collapses. Additionally, verify whether Section IV-E's 'five LLMs' matches the twelve models in Tables II/III; if not, ask the authors to clarify which models were fine-tuned.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that fine-tuned Roberta-base achieves the highest ROUGE-L/BLEU/BERTScore—is contradicted by the paper's own Conclusion. Section VI states: 'In particular, models such as Albert-base and Bert-base achieved the highest performance,' while the Abstract claims Roberta-base, and Table III shows Roberta-base with ROUGE-L 86.84%, BLEU 28.24%, BERTScore 95.38%, all above Albert-base (82.27/26.56/93.85) and Bert-base (78.00/26.24/92.38). This is not a minor stylistic slip: it means the manuscript does not consistently support its headline result. If the Conclusion is authoritative, then the Abstract and Table III are wrong; if Table III is authoritative, the Conclusion is a factual error. A reader cannot determine which. This internal inconsistency casts doubt on the precision of all reported numbers and the care taken in analysis. It also interacts with the missing experimental details (no hyperparameters, epochs, or evaluation-split description), so neither the ranking nor the numbers can be trusted as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a context-based QA system that fine-tunes twelve transformer models of varying sizes (4.4M to 1.7B parameters) on SQuAD1.1, formulated as extractive span prediction with a start/end cross-entropy loss. The authors evaluate baselines and fine-tuned models using ROUGE-L, BLEU, and BERTScore. The Abstract and Table III report that fine-tuned Roberta-base achieves the highest scores (ROUGE-L 86.84%, BLEU 28.24%, BERTScore 95.38%), while the Conclusion states that Albert-base and Bert-base achieved the highest performance. The paper claims that fine-tuning substantially improves all models and that model capacity/architecture plays a major role.","tokens_in":9515,"tokens_out":3533,"duration_ms":34746,"significance":"If the empirical result is reproducible, the paper provides a useful, though incremental, benchmark comparison of fine-tuned transformer models on an extractive QA task. The main value would be as a practical guide to model selection for similar tasks. The paper does not introduce a new architecture, training method, or theoretical insight. It also does not provide code, data splits, or training hyperparameters, which limits its immediate utility and makes the central ranking difficult to verify. The internal inconsistency between the Conclusion and the Abstract/Table III is a serious correctness issue that must be resolved before the paper can be considered reliable.","major_comments":[{"comment":"The Abstract states that the fine-tuned Roberta-base model achieves the highest performance with ROUGE-L 86.84%, BLEU 28.24%, and BERTScore 95.38%, and Table III confirms these numbers are higher than Albert-base (82.27/26.56/93.85) and Bert-base (78.00/26.24/92.38). Section VI, however, says \"models such as Albert-base and Bert-base achieved the highest performance.\" This is a direct contradiction about the paper's headline result. The reader cannot tell which statement is authoritative. Please correct the Conclusion or explain the discrepancy; if Table III is correct, the Conclusion is a factual error. This issue is load-bearing because the central claim is the model ranking.","section":"Section VI (Conclusion) vs. Abstract and Table III"},{"comment":"The fine-tuning procedure is described only as \"the same training dataset and identical preprocessing steps\" for all models. No learning rate, batch size, number of epochs, optimizer, warm-up, sequence length, or per-model hyperparameter search is reported. The models range from 4.4M (Bert-tiny) to 1.7B (Bloom), so a single fixed setup is not obviously fair across all architectures; the reported ranking (Roberta > Albert > Bert) could be an artifact of hyperparameter choice. Additionally, the train/validation/test split of SQuAD1.1 is not described, and it is unclear whether the reported numbers are on the official dev set or some other split. Without these details, the experiments are not reproducible and the ranking is not fully supported.","section":"Section IV-E and Section V-B"},{"comment":"All results are single-point estimates with no error bars, confidence intervals, or statistical significance tests. The differences between Roberta-base and Albert-base are small in relative terms (ROUGE-L 4.57 points, BLEU 1.68 points, BERTScore 1.53 points), and could fall within run-to-run variance. The paper claims \"substantial improvements\" and \"best overall performance\" without any variance analysis. Furthermore, SQuAD is conventionally evaluated with exact-match (EM) and F1; ROUGE-L and BLEU are not standard for extractive span QA, and BLEU is particularly unreliable for short answer spans. Please report EM/F1 or justify the chosen metrics and provide variance estimates.","section":"Section V (Results and Discussion)"},{"comment":"The text says \"Each of the five LLMs is fine-tuned independently\" but Tables II and III list twelve models. This discrepancy should be clarified: either the five LLMs are a subset and the remaining models are trained differently, or the sentence is a typo. As written, it is unclear whether all twelve models were fine-tuned under the same protocol.","section":"Section IV-E, \"five LLMs\""}],"minor_comments":[{"comment":"The text refers to \"SQuAD1.1 2.0\" and \"SQuAD1.1 1.1\"; these should be \"SQuAD2.0\" and \"SQuAD1.1\". SQuAD1.1 contains only answerable questions; SQuAD2.0 adds unanswerable questions.","section":"Section III"},{"comment":"The text references Figure 1 as an overview of the system, but the figure is not included in the manuscript. Please ensure the figure is present or remove the reference.","section":"Figure 1"},{"comment":"The table heading \"THESTANFORDQUESTIONANSWERINGDATASET\" lacks spaces; should be \"The Stanford Question Answering Dataset (SQuAD1.1).\"","section":"Table I"},{"comment":"The sentence \"Overall, Despite advances in LLM-based QA systems\" has inconsistent capitalization; \"Despite\" should be lowercase. Minor typographical polish throughout would help.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"The internal contradiction between the Conclusion and the Abstract/Table III is the most urgent issue and must be fixed. The paper is quite thin methodologically: no hyperparameters, no evaluation split, no statistical significance testing, and no reproducibility artifacts. This is not beyond repair, but the empirical ranking cannot be considered established until these points are addressed. I would not recommend rejection solely on novelty grounds, but the authors need to supply the missing experimental details and correct the inconsistent claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper’s abstract and Table III name Roberta-base as the best fine-tuned model, but the Conclusion says Albert-base and Bert-base achieved the highest performance. That is not a minor wording slip; it is a direct contradiction in the central claim. A reader cannot tell which result the authors believe.\n\nWhat is actually new: not much. The paper fine-tunes twelve transformer models on SQuAD1.1 and evaluates with ROUGE-L, BLEU, and BERTScore. That is a routine exercise. The clearest useful element is the baseline versus fine-tuned comparison, which does show consistent gains and an interesting shift from generative baselines to encoder models after fine-tuning. The writing is straightforward and the dataset description is fine.\n\nThe soft spots are real. The internal contradiction is the biggest one. On top of that, the methodology section says all models are fine-tuned with the same setup but never reports learning rate, batch size, epochs, or any search per model. That matters, because comparing a 4M-parameter BERT-tiny to a 1.1B TinyLlama under one fixed setup can easily produce an artifact. The evaluation uses ROUGE-L, BLEU, and BERTScore on an extractive task where EM and F1 are the community standard; that is not automatically wrong, but it makes the numbers hard to compare with prior work. There are no error bars or significance tests, so the ranking among close models is not supported. No code or data are provided.\n\nI also notice the stress-test note is accurate. The conclusion is inconsistent with the abstract, and that undermines confidence in the care taken with all numbers. I agree with the reader’s conditional verdict, but I would go slightly further: the paper needs a substantive revision before it can be taken seriously. The central finding—that fine-tuning improves QA—is true and well known, and the paper adds no new method or insight. The model zoo is not a scientific contribution absent reproducible setup.\n\nWho is this for? A practitioner wanting a rough sanity check on which encoder models work for extractive QA might skim it, but they would be better served by the existing literature. It does not deserve a serious referee in its current form. If the authors fix the conclusion, add full hyperparameters, report EM/F1, and release code, it could become a minor but acceptable empirical study. As is, I would desk reject it.","headline":"Paper's abstract and Table III say Roberta-base is best; the Conclusion says Albert-base and Bert-base are best—an internal contradiction that poisons the central claim.","tokens_in":9958,"tokens_out":3174,"would_cite":false,"duration_ms":33421,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning a transformer for span prediction on SQuAD1.1 turns weak zero-shot models into reliable answer extractors, with Roberta-base reaching 86.84% ROUGE-L.","keywords":["Question Answering","Answer Extraction","Fine-tuning","SQuAD1.1","ROUGE-L","BERTScore","Span Prediction","Transformer Encoders"],"falsifier":"Re-run all twelve models on SQuAD1.1 with a per-model hyperparameter search (learning rate, batch size, epochs) under the same data and metrics. If any model matches or exceeds Roberta-base's ROUGE-L 86.84%, BLEU 28.24%, and BERTScore 95.38% under tuned settings, the paper's identical-setup comparison is not reliable evidence that Roberta-base is the best extractive QA model.","tokens_in":9139,"feed_emoji":"🎯","tokens_out":5327,"duration_ms":50251,"temperature":0.7,"pith_summary":"The paper claims that task-specific fine-tuning on a benchmark extractive QA dataset is what makes language models reliable at pulling answers out of a passage, and that one shared span-prediction setup can reveal which architectures are best suited. It reports that fine-tuned Roberta-base outperforms eleven other models, including larger decoder-style LLMs, achieving ROUGE-L 86.84%, BLEU 28.24%, and BERTScore 95.38%. The broader point is that pre-training strength alone does not predict QA performance; adaptability to the task matters. If true, this gives practitioners a simple recipe—concatenate context and question, add a start/end span head, fine-tune—for building accurate extractive QA systems.","feed_headline":"86.84 ROUGE-L: fine-tuned Roberta-base wins extractive QA test","feed_subtitle":"Span-prediction fine-tuning lifts every model and outdoes larger general-purpose LLMs on SQuAD1.1.","key_machinery":"The load-bearing piece is the answer-span head: the model assigns probability distributions over start and end token positions, trained by the sum of two cross-entropy losses, L = L_start + L_end. At inference, the span between the highest-probability start and end indices is extracted from the context, so answers remain grounded in the passage. The paper applies this objective to transformer encoders, using the concatenated context and question as input. This span-prediction setup is what converts the models from weak zero-shot generators into extractive QA systems without needing retrieval or external knowledge.","core_discovery":"On SQuAD1.1, fine-tuning a transformer with a span-extraction objective (predicting start and end positions of the answer within the context) substantially improves every model tested. After fine-tuning, Roberta-base reaches the top scores on all three metrics, followed closely by Albert-base-v2 and Bert-base. The paper also finds that the ranking of models flips after fine-tuning: StableLM-2 and Qwen2.5 are the best baselines, but encoder-based masked-language models overtake them once adapted. The conclusion is that selecting a verbatim span from context is best done with a supervised span-prediction objective, and that such fine-tuning is essential for reliable and consistent outputs.","pith_inferences":["Because every SQuAD1.1 answer is a contiguous span, the same span-loss recipe should transfer directly to other extractive reading-comprehension benchmarks with minimal changes—an implication the paper leaves untested.","The consistent gap between high BERTScore and lower ROUGE-L/BLEU hints that answers are often semantically right but lexically off; using BERTScore to rerank candidate spans at inference could be a cheap accuracy boost the paper does not explore.","The paper does not report hyperparameter settings, so a per-model learning-rate, batch-size, or epoch sweep is the natural next experiment to test whether the reported leader board is stable or an artifact of the single shared recipe."],"forward_implications":["Fine-tuning on SQuAD1.1 dramatically lifts extractive QA performance, taking Roberta-base from 3.34% to 86.84% ROUGE-L.","Task-specific adaptation matters more than raw pre-training scale: smaller encoder models surpass larger general-purpose LLMs after fine-tuning.","Model capacity still constrains gains, as Bert-tiny lags far behind larger models even after the same fine-tuning.","The combination of ROUGE-L, BLEU, and BERTScore provides complementary signals: high BERTScore with lower lexical scores indicates semantically relevant but not verbatim answers.","The unified fine-tuning setup offers a practical template for comparing architectures on extractive QA and selecting a strong default model."],"fun_headline_variants":["Fine-tuned Roberta-base dominates SQuAD1.1 with span extraction","Span-prediction fine-tuning boosts every QA model tested","Roberta-base beats larger LLMs after extractive fine-tuning","SQuAD fine-tuning flips model rankings: Roberta-base tops","For extractive QA, supervised span-prediction beats prompting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that one identical fine-tuning recipe is a fair test for twelve models of very different sizes and families; if the recipe happens to favor some architectures, the reported ranking could change.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Roberta-base dominates SQuAD1.1 with span extraction","Span-prediction fine-tuning boosts every QA model tested","Roberta-base beats larger LLMs after extractive fine-tuning","SQuAD fine-tuning flips model rankings: Roberta-base tops","For extractive QA, supervised span-prediction beats prompting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1344,"prompt_tokens":803,"completion_tokens":541,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":547,"tokens_out":541,"duration_ms":5916,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:15:04.766707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run all twelve models on SQuAD1.1 with a per-model hyperparameter search (learning rate, batch size, epochs) under the same data and metrics. If any model matches or exceeds Roberta-base's ROUGE-L 86.84%, BLEU 28.24%, and BERTScore 95.38% under tuned settings, the paper's identical-setup comparison is not reliable evidence that Roberta-base is the best extractive QA model.","supporting_citations":[],"review_version":3}