{"id":"b52d1c85-ed58-4ea3-a86c-b489e09d6299","arxiv_id":"2412.12893","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On SQuAD2, fine-tuned RoBERTa beats all tested LLMs, but LLaMA-3.1-70B outperforms the fine-tuned models on 3 of 5 out-of-distribution QA datasets.","lead":"Fine-tuned question-answering models still beat out-of-the-box large language models on the SQuAD2 benchmark, but a 70-billion-parameter LLaMA model outperformed the fine-tuned models on three of five out-of-distribution QA datasets. The paper also proposes a single-inference prompt for handling unanswerable questions, which could halve inference cost if it works as claimed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD 3-of-5 claim rests on five unnamed datasets and an unspecified 1000-question subsample; answer-format compatibility is unverified.","rationale":"The reader's weakest-assumption identification is exactly the load-bearing issue: the OOD 3-of-5 result cannot be independently evaluated because the five datasets are unnamed and the subsampling procedure is unspecified. My stress-test adds a sharper reason why naming alone may not suffice: the comparison is only meaningful if all five datasets use extractive, span-based answers, since the FT-LMs are architecturally limited to span extraction. If a dataset uses free-form or yes/no answers, the FT-LM scores are structurally capped and the headline comparison is biased. The hand-written post-processing in Sec. 4.3 and the absence of error bars further weaken the OOD claim, but these are secondary to the missing dataset identity and answer-format verification. The in-distribution SQuAD2 result in Table 1 is robust and clearly supports the fine-tuned-over-LLM conclusion; that portion of the paper does not need revision. The paper's own Sec. 7 limitations are consistent with my reading. Since the reader already assigned CONDITIONAL and my concern does not move that verdict, I recommend keeping the verdict unchanged: the paper should be accepted only conditionally, pending release of the OOD dataset details and a demonstration that the reported 3-of-5 advantage is not an artifact of answer-format mismatch or sampling noise.","tokens_in":11871,"tokens_out":4358,"duration_ms":40013,"concrete_test":"Contact the authors for the exact names, versions, and sampling code for the five OOD datasets used in Sec. 5. Independently rerun LLaMA-3.1-70B and the three FT-LMs on the full (not subsampled) validation/test splits of those datasets, using the same prompts and post-processing, and compute per-dataset EM/F1 with 95% bootstrap CIs over the 1000-question subsamples. If the 3-of-5 advantage reverses or the margins become non-significant on any dataset whose answers are non-extractable spans, the generalization claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 5 reports that LLaMA-3.1-70B outperforms the fine-tuned models on 3 of 5 OOD QA datasets (Figs. 3-4), but never names the datasets nor specifies how the 1000-question subsample per dataset was drawn. This is the load-bearing condition for the paper's central generalization claim, and it is currently unverifiable. The concern is not merely missing detail: the fine-tuned models are extractive QA systems (RoBERTa, DistilBERT, Flan-T5 fine-tuned on SQuAD2) that can only output spans from the provided context. If any of the five OOD datasets contains abstractive, yes/no, or otherwise non-extractable answers, those models are structurally handicapped, and the 3-of-5 result would reflect answer-format compatibility rather than LLM generalization. Further, Sec. 4.3 says LLM outputs were cleaned with hand-written regex rules whose effect is not quantified, and Figs. 3-4 report no variance or confidence intervals despite the 1000-question subsampling. The paper's own Sec. 7 concedes results may not transfer to other tasks and a better prompt may exist. The SQuAD2 in-distribution result (Table 1) is not threatened by these issues, but the OOD 3-of-5 claim is.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an empirical comparison of three small fine-tuned models (Flan-T5, DistilBERT, RoBERTa) and six out-of-the-box instruction-tuned LLMs (GPT-4 Turbo and several LLaMA variants) on the SQuAD2 validation set using a newly proposed single-inference prompt designed to handle both answerable and unanswerable questions. The authors report standard EM/F1 scores, a no-answer EM score, a breakdown by interrogative pronoun, and a Levenshtein-distance-based analysis. They also evaluate the best-performing LLM (LLaMA-3.1-70B) and the FT-LMs on 1000 questions from each of five additional QA datasets to test out-of-distribution generalization, claiming that the LLM outperforms the FT-LMs on 3 of 5 datasets. The main SQuAD2 result is that RoBERTa achieves the best F1 (82.43) and EM (79.97), while LLaMA-3.1-70B is the best LLM (F1 73.68).","tokens_in":12248,"tokens_out":6325,"duration_ms":54403,"significance":"If supported, the SQuAD2 comparison provides a useful data point that fine-tuned small extractive models still lead on in-distribution QA, while a large prompted LLM can close much of the gap and may generalize better to new distributions, at higher inference cost. The paper is transparent about its SQuAD2 evaluation setup (prompt in the appendix, standard metrics, per-question-type breakdowns) and explicitly states its limitations. However, the OOD generalization claim, which is the main novel contribution, currently rests on unnamed datasets, unspecified subsampling, unquantified LLM post-processing, and no variance estimates; the claim cannot be accepted as stated.","major_comments":[{"comment":"The five OOD datasets are never named and the procedure for selecting 1000 questions per dataset is not described, so the central claim that LLaMA-3.1-70B outperforms the FT-LMs on 3 of 5 datasets is not independently verifiable or reproducible. Please specify the dataset names and versions, the answer-format characteristics of each, and the exact sampling procedure (e.g., random seed, stratification), or provide the sampled question IDs.","section":"5"},{"comment":"The FT-LMs are extractive QA systems that can only return spans from the context, while the LLM is free-form. If any of the five OOD datasets contains abstractive, yes/no, or otherwise non-span answers, the FT-LMs are structurally handicapped and the 3-of-5 result may reflect format incompatibility rather than generalization. Please report the answer type of each OOD dataset and confirm that all ground-truth answers are extractable spans; if not, the comparison needs to be adjusted or the limitation acknowledged.","section":"5"},{"comment":"The hand-written regex post-processing applied to LLM predictions is described only qualitatively ('greatly improved all calculated scores') and no before/after numbers are given. Since this post-processing is applied only to LLM outputs and is model-specific, it is a potential source of bias in the comparison, including the OOD results. Please provide the exact post-processing rules, the scores before and after application for every LLM and dataset, and a sensitivity check showing that the reported rankings are stable under reasonable variations of the rules.","section":"4.3"},{"comment":"The 1000-question subsample estimates in Figs. 3-4 are reported without variance or confidence intervals, despite the sampling step and the stochastic LLM decoding (temperature 0.1). The 3-of-5 superiority claim is a point estimate; please report per-dataset bootstrap intervals or standard errors and the number of independent runs, so the robustness of the ranking can be assessed.","section":"5"}],"minor_comments":[{"comment":"There are typographical errors throughout (e.g., 'Additonally' in Section 1, 'perfomance' in Section 2, 'Skript' in the Table 3 caption, 'T able' in several captions) that should be corrected.","section":"1"},{"comment":"The NAP threshold is chosen as the mean of the best-EM and best-F1 thresholds on the same validation split; this is a tuned parameter, and the sensitivity of the reported FT-LM scores to this choice should be stated, at least for the no-answer rows.","section":"3.3"},{"comment":"The claim that the proposed single-inference prompt elicits the same unanswerable-question handling as the double-inference method is not empirically tested in this paper; consider adding a direct comparison or rephrasing this as a design goal rather than an equivalence.","section":"3.2"},{"comment":"The statement 'The evaluation in [4] utilizes previous questions and answers on a document as additional context' cites the LLaMA 3 model card; please cite the original work that introduced this prompting practice, as [4] may not be the primary source.","section":"4.1"},{"comment":"Figure 2 would be clearer if the y-axis label and the normalization used for the Levenshtein allowance were defined in the caption.","section":"4.2"},{"comment":"Some references lack arXiv identifiers or URLs (e.g., references 3, 6, 8, 20, 23, 24, 25, 29); please complete the reference list for reproducibility.","section":"2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical study, but the OOD section is too under-specified for the central claim. The authors should also consider releasing the sampled question IDs and post-processing code. No concerns about novelty or attribution beyond the loose citation in Section 4.1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the SQuAD2 numbers are solid and the single-inference prompt is a real, small contribution. The out-of-distribution 3-of-5 claim is the headline, and right now it is under-evidenced because the five datasets are never named and the 1000-question subsample is not described.\n\nWhat the paper does well: a clean comparison of six LLMs against three fine-tuned models on SQuAD2, with sensible hyperparameters and an honest prompt that handles unanswerable questions in one forward pass. LLaMA-3.1-70B's noAns EM of 81.56 shows the prompt works. The Levenshtein-distance analysis is a useful way to surface close-misses, and Fig. 2 nicely separates GPT-4's strict-format adherence from LLaMA's fuzzier answers. The authors also state their own limitations in Sec. 7, which is more than many empirical papers do.\n\nThe soft spot is exactly where the reader and stress-test point: the OOD section. The five datasets are unnamed, the sampling procedure is unspecified, and the fine-tuned models are extractive by construction. If any of those datasets requires abstractive, yes/no, or non-span answers, the FT-LMs are structurally handicapped and the 3-of-5 result would reflect format compatibility, not generalization. The regex post-processing is unquantified and model-specific, and there are no error bars anywhere, even though the 1000-question subsample introduces sampling variance. The NAP threshold averaging is a minor concern; it is post-hoc but at least bounded by the official EM and F1 thresholds, so I would not call it a fatal flaw. None of this threatens the in-distribution Table 1.\n\nThis paper is for QA practitioners weighing fine-tuning against prompting. The in-distribution comparison is a solid replication, the prompt is worth trying, and the OOD direction is promising. I would send it to a serious referee, with the expectation that the OOD claim gets tightened or cut.","headline":"Useful SQuAD2 comparison with a genuinely new single-inference prompt and a promising-but-under-evidenced OOD claim; deserves review with revisions.","tokens_in":12678,"tokens_out":1378,"would_cite":false,"duration_ms":12765,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that fine-tuned small models outperform prompt-only LLMs on in-distribution question answering, while a 70B LLaMA wins on most out-of-distribution sets.","keywords":["question answering","SQuAD2","large language models","fine-tuned models","out-of-distribution generalization","prompting strategy","Levenshtein distance","chain-of-thought"],"falsifier":"Rerun the out-of-distribution evaluation on the five named datasets using the full validation sets (or a documented stratified sample) and the same prompts; if LLaMA-3.1-70B no longer beats RoBERTa on three of the five datasets in EM and F1, the paper's central generalization claim fails.","tokens_in":11635,"feed_emoji":"🤖","tokens_out":8367,"duration_ms":62541,"temperature":0.7,"pith_summary":"This paper asks whether out-of-the-box instruction-following large language models can replace smaller, task-specialized models in question answering. It compares six LLMs with three fine-tuned models on SQuAD2 and finds that the fine-tuned models still win in-distribution: RoBERTa reaches 82.43 F1 versus 73.68 for the best LLM, LLaMA-3.1-70B. On five additional QA datasets used without fine-tuning, however, the same LLaMA model outperforms all fine-tuned models on three of them, suggesting that large prompted models generalize better to new distributions at higher inference cost. The paper also contributes a single-inference prompt that handles answerable and unanswerable questions in one forward pass, and a Levenshtein-distance analysis of near-miss answers.","feed_headline":"Fine-tuned models top LLMs on SQuAD2; LLaMA-70B wins 3 OOD sets","feed_subtitle":"A single-inference prompt lets a 70B LLM match specialized extractive QA models on new data, at higher compute cost.","key_machinery":"The carrying mechanism is the single-inference prompting strategy: a persona prompt that tells the LLM it is a reading-comprehension assistant, asks it to 'think step-by-step' (which elicits chain-of-thought reasoning), and instructs it to reply with the exact quote from the context or the literal string 'unanswerable'. This replaces the previous double-inference approach, where the model first decides answerability and then answers in a second pass, halving the forward passes. The comparison is completed by a standard SQuAD2 evaluation pipeline plus a Levenshtein-distance extension that measures how many character edits separate a prediction from the best ground truth, and a breakdown by interrogative pronoun. For the fine-tuned models, answerability is decided by a no-answer probability threshold chosen as the mean of the best EM and best F1 thresholds.","core_discovery":"The central empirical discovery is the two-part ranking. On the SQuAD2 validation set, the fine-tuned extractive models dominate: RoBERTa scores 79.97 EM and 82.43 F1, DistilBERT 67.89 EM and 70.17 F1, and Flan-T5 64.55 EM and 66.02 F1, while every tested LLM scores lower in EM, with LLaMA-3.1-70B the strongest LLM at 57.13 EM and 73.68 F1. When the same models are evaluated on 1000-question subsamples of five other QA datasets without any fine-tuning, LLaMA-3.1-70B beats all fine-tuned models on three of the five datasets in both EM and F1, while RoBERTa remains best on the other two. The paper interprets this as evidence that sufficiently large prompted LLMs can rival or exceed specialized models when the test distribution differs from the training distribution, at the price of much higher inference cost and a tendency toward longer, less exact answers.","pith_inferences":["Because the five out-of-distribution datasets are never named and the 1000-question sampling procedure is not described, the 3-of-5 result should be treated as provisional until it is reproduced on identifiable public benchmarks; the paper itself provides no identifiers to check representativeness.","The Levenshtein-distance analysis suggests a cheap testable extension: apply more aggressive model-specific post-processing to LLM outputs (e.g., stripping courtesy phrases) and measure whether the sharp EM jump at 8-10 character changes for LLaMA-3.1-70B disappears, which would indicate recoverable score.","A hybrid pipeline combining a large prompted LLM for answerability judgment and a small extractive model for answer spans could combine the OOD robustness of the former with the exactness of the latter; the paper does not test this.","The single-inference prompt might be evaluated on long-context or multi-hop QA, where the 'unanswerable' instruction could behave differently because the context itself contains partial answers that invite hallucination."],"forward_implications":["A single-inference prompt can halve the compute cost of handling unanswerable questions in QA, matching the capability that previously required two separate forward passes.","For in-distribution question answering, a small fine-tuned model such as RoBERTa remains the better choice in accuracy per parameter, since it outperforms all tested LLMs despite being roughly 500 times smaller than LLaMA-3.1-70B.","For out-of-distribution or rapidly changing domains where fine-tuning data is unavailable, a sufficiently large instruction-following LLM can be deployed zero-shot; LLaMA-3.1-70B beats every fine-tuned model on three of the five test datasets.","The Levenshtein-distance results show that allowing even a few character edits barely changes EM for all models, but a 20-character allowance lifts several LLMs substantially, so string-based metrics underestimate the semantic correctness of LLM answers.","All models perform worst on 'why' questions, indicating a common bottleneck in causal reading comprehension that is not specific to LLMs."],"supporting_citations":[{"why":"Introduced the double-inference technique for answerability filtering that the paper's single-inference prompt is designed to replace.","marker":"[1]"},{"why":"Supplies the LLaMA-3 family (8B, 70B, and 3.2 variants) used as the out-of-the-box LLMs in the comparison.","marker":"[4]"},{"why":"Establishes that 'think step-by-step' elicits zero-shot chain-of-thought reasoning, grounding the prompt's reasoning instruction.","marker":"[10]"},{"why":"Provides RoBERTa, the best-performing fine-tuned model and the in-distribution baseline the LLMs are compared against.","marker":"[11]"},{"why":"Defines SQuAD2 with its unanswerable questions, the in-distribution benchmark and the source of the fine-tuning data.","marker":"[16]"}],"fun_headline_variants":["LLaMA-70B tops fine-tuned QA on 3 OOD sets; RoBERTa wins SQuAD2","Single-inference prompt lets 70B LLM match extractive QA on new data","Fine-tuned wins SQuAD2, but LLaMA-70B generalizes better on 3/5 tests","No double inference: LLaMA-70B beats fine-tuned models on 3 OOD datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that LLaMA-3.1-70B outperforms fine-tuned models on 3 of 5 out-of-distribution datasets rests on the assumption that those five datasets are genuine QA benchmarks comparable in format to SQuAD2 and that the 1000-question subsample drawn from each is representative; the paper names neither the datasets nor the sampling method.","fun_headline_variants_meta":{"raw":{"variants":["LLaMA-70B tops fine-tuned QA on 3 OOD sets; RoBERTa wins SQuAD2","Single-inference prompt lets 70B LLM match extractive QA on new data","Fine-tuned wins SQuAD2, but LLaMA-70B generalizes better on 3/5 tests","No double inference: LLaMA-70B beats fine-tuned models on 3 OOD datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1730,"prompt_tokens":1025,"completion_tokens":705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":641,"tokens_out":705,"duration_ms":6126,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:37:10.838265+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the out-of-distribution evaluation on the five named datasets using the full validation sets (or a documented stratified sample) and the same prompts; if LLaMA-3.1-70B no longer beats RoBERTa on three of the five datasets in EM and F1, the paper's central generalization claim fails.","supporting_citations":[{"cited_title":"Evaluating chatgpt as a question answering system: A comprehensive analysis and comparison with existing models, 2023","cited_arxiv_id":null,"evidence_quote":"Introduced the double-inference technique for answerability filtering that the paper's single-inference prompt is designed to replace."},{"cited_title":"Roberta: A robustly optimized bert pretraining approach, 2019","cited_arxiv_id":null,"evidence_quote":"Provides RoBERTa, the best-performing fine-tuned model and the in-distribution baseline the LLMs are compared against."},{"cited_title":"Know what you don’t know: Unanswer- able questions for squad, 2018","cited_arxiv_id":null,"evidence_quote":"Defines SQuAD2 with its unanswerable questions, the in-distribution benchmark and the source of the fine-tuning data."}],"review_version":1}