{"id":"36d38d0e-df17-484c-86e5-53e3a47cb17d","arxiv_id":"2608.10806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Across SQuAD and QuAC, RoBERTa and DistilBERT are generally more stable under Monte Carlo dropout, while ALBERT and BERT-Base are less consistent, and paraphrase perturbation changes the ranking.","lead":"This paper measures how consistently four BERT-based question answering models answer when the model is randomly perturbed or the question is rephrased. It finds that reliability ranks differ by dataset and that accuracy alone misses large stability differences.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5 correlations are computed between unperturbed scores and per-input averages over N=50 MCD samples. Averaging over samples inflates correlation, so high values do not establish that individual stochastic predictions are stable.","rationale":"The most load-bearing concern is that the central claim—MCD is a valid, non-disruptive reliability metric—rests on Table 5 correlations that are inflated by averaging over N=50 stochastic samples per instance. This is a technical flaw internal to the argument, and it undermines the headline claim regardless of the cosine-similarity calibration issue. The reader's weakest_assumption focused on whether cosine similarity at a 0.95 threshold tracks human correctness; that is a valid secondary concern, but it is not what makes the central validation fail. I therefore disagree with the reader's identification of the weakest assumption. The paper is reproducible (code provided) and the descriptive statistics are useful, but the central claim as stated in the abstract and Section 4.2 is not supported by the reported analysis. A conditional acceptance requiring a per-sample reanalysis would be appropriate; if the per-sample correlations turn out to be much lower, the MCD-validity claim would need to be withdrawn or substantially weakened. Since the reader already returned CONDITIONAL, the verdict remains unchanged, but the condition should be revised to include this statistical reanalysis.","tokens_in":18988,"tokens_out":5422,"duration_ms":49171,"concrete_test":"Recompute Table 5 without averaging: for each of the N=50 MCD samples, compute cosine similarity and F1 against ground truth, and correlate these per-sample scores with the unperturbed model's per-instance scores (pooling all samples, or reporting the mean/min per-sample correlation). Also compare correlations at N=10 vs N=50: if the correlation rises with N, it is an averaging artifact. If the per-sample correlations are materially lower than the averaged values in Table 5 (e.g., >0.1 drop), the claim that MCD does not disrupt inference is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 reports correlation coefficients (Table 5) between the unperturbed model's score per instance and the MCD-perturbed score, where 'for the perturbed model, evaluation metrics are averaged across stochastic samples to obtain a single score per input.' This averaging is the core evidence for the abstract's claim that 'enabling MCD during prediction does not disrupt inference dynamics.' The problem is statistical: if each stochastic sample y_ij = x_i + e_ij with zero-mean noise, then the average over N=50 samples converges to x_i, so the correlation between x_i and the averaged \\bar{y}_i approaches 1 as N grows, regardless of the magnitude of the individual fluctuations e_ij. The high coefficients (0.86–0.95) therefore do not demonstrate that dropout leaves predictions stable; they may simply reflect the law of large numbers. The paper even acknowledges that variance bands widen at 10% dropout (Section 4.1), but that variance is not used in the correlation analysis. Without per-sample correlations or a direct measure of answer-span consistency under MCD, the central validation of MCD as a non-disruptive reliability metric is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a framework for assessing the reliability of four BERT-based extractive QA models (BERT-Base, RoBERTa, DistilBERT, ALBERT) on SQuAD 2.0 and QuAC. Reliability is operationalized in two ways: stability under Monte Carlo dropout (MCD), quantified by the mean and standard deviation of per-instance cosine-similarity and F1 scores over N=50 stochastic forward passes, and stability under input paraphrasing filtered to an S-BERT cosine interval of 0.75–0.98. The paper defines a semantic exact-match threshold of 0.95, computes correlations between unperturbed scores and MCD-averaged scores, runs Welch t-tests and Bonferroni-corrected Wilcoxon tests, reports a human evaluation with Fleiss' kappa, and provides qualitative error tables. The central claims are that enabling MCD during prediction does not disrupt inference dynamics, thereby validating MCD as a reliability metric, and that RoBERTa maintains higher reliability while ALBERT and DistilBERT exhibit significant inconsistencies.","tokens_in":19241,"tokens_out":11235,"duration_ms":109235,"significance":"The study addresses a genuine gap: accuracy metrics alone do not capture the consistency of QA models under stochastic inference and input reformulation, and the paper ships reproducible code, a clear algorithmic protocol (Algorithms 1 and 2), a systematic dropout-rate sweep, Bonferroni-corrected pairwise tests, and inter-annotator agreement analysis. These are substantive strengths. Nevertheless, the key statistical evidence for the non-disruptive claim (Table 5) is an averaging artifact, the headline reliability ranking contradicts the paper's own Tables 3–5 and its Welch test, and the 0.95 accuracy threshold is uncalibrated. The limitations acknowledged in Section 6 (encoder-only models, QA-only scope, two datasets, one perturbation approach per type) are reasonable and do not by themselves block the claims; the blocking issues are statistical and internal-consistency problems that can in principle be fixed within the manuscript's scope.","major_comments":[{"comment":"The correlations used to validate MCD are computed between the unperturbed per-instance score and the per-instance score averaged over N=50 stochastic samples, as the text states: 'for the perturbed model, evaluation metrics are averaged across stochastic samples to obtain a single score per input.' If the per-sample metric for input i satisfies y_ij = x_i + e_ij with zero-mean fluctuations, the average converges to x_i as N grows, so the reported coefficients (0.857–0.946) approach 1 regardless of the magnitude of the individual fluctuations e_ij. The per-instance standard deviations in Tables 3–4 (e.g., 0.0906–0.2449 on SQuAD) show that the individual samples vary substantially, and Section 4.1 itself states that the variance bands widen at 10% dropout. The abstract's claim that 'enabling MCD during prediction does not disrupt inference dynamics' therefore does not follow from Table 5. Please report correlations between unperturbed scores and individual stochastic samples (not averages), or use the per-instance standard deviation directly as the stability measure, or provide a span-level agreement analysis across the 50 samples.","section":"Section 4.2, Table 5"},{"comment":"The headline claim that 'RoBERTa maintains higher reliability, whereas AlBERT and DistilBERT exhibit significant inconsistencies' is contradicted by the manuscript's own results. Table 3 (Total rows) shows DistilBERT with lower standard deviations than RoBERTa on SQuAD (cosine 0.1131 vs 0.1543; F1 0.1285 vs 0.1772), and Table 4 shows the same pattern on QuAC (cosine 0.1470 vs 0.1761; F1 0.1360 vs 0.1761). Table 5 shows RoBERTa with the lowest unperturbed/perturbed correlations of the four models on both datasets and both metrics, and the Welch test on QuAC yields p = 0.4547, i.e., no significant difference between RoBERTa and DistilBERT. Section 7 even concludes that 'DistilBERT is more stable in handling internal configuration variations.' The abstract and the Section 4.2 narrative must be rewritten so that the reliability ranking is consistent with the reported standard deviations, correlations, and significance tests.","section":"Abstract and Section 4.2"},{"comment":"Accuracy is defined by a semantic similarity threshold of 0.95 between predicted and ground-truth answer embeddings, but no evidence links this threshold to human judgments of answer correctness. The human evaluation described in Section 4.3 explicitly re-labels only responses 'automatically classified as incorrect' and covers just two of the four models, so it cannot calibrate the 0.95 cutoff or validate the positive classifications that underlie the Acc. and MCD Acc. columns of Tables 3 and 4. Because the near-equality of Acc. and MCD Acc. is offered as additional evidence that stochastic inference does not degrade performance, this unvalidated cutoff is load-bearing. The fix is either a calibration analysis (e.g., a threshold sweep against human judgments, or against lexical exact match) or removal of the semantic-exact-match accuracy framing in favor of the score distributions.","section":"Section 4.2, MCD Accuracy; Section 4.3"},{"comment":"The manuscript does not state which fine-tuned checkpoints or fine-tuning datasets were used for each model, nor which evaluation subsets produced Tables 3–6. This matters because the QuAC results (e.g., RoBERTa total accuracy of about 21% in Table 4) are consistent with evaluating a SQuAD-fine-tuned model on an out-of-domain dataset, which would change the interpretation of the 'low accuracy' dataset comparison that motivates Objective 3 in Section 1. Please specify the exact model identifiers used (e.g., Hugging Face checkpoint names), the fine-tuning data for each model, and the number of evaluation instances per dataset, so that the accuracy–reliability comparison is reproducible and interpretable.","section":"Sections 3.2 and 4, Tables 3–6"}],"minor_comments":[{"comment":"The sentence 'This paper is organized into five sections' is immediately followed by a list that runs from Section 2 through Section 7; the count should be corrected.","section":"Section 1"},{"comment":"The model name is spelled inconsistently ('AlBERT' in the abstract, 'ALBERT' elsewhere, and 'alBERT' in some passages); use a single spelling throughout.","section":"Abstract and throughout"},{"comment":"The rationale for choosing N=50 ('experiments... demonstrated that there is no variation in the model's performance on increasing the number of samples') is asserted without supporting evidence; please include the saturation curve or a table showing mean and variance as a function of N.","section":"Section 3.5"},{"comment":"The Welch test p-value is reported as '0.0'; report it as p < 0.0001 (or an inequality with a specific bound) rather than an exact zero, which is an artifact of floating-point precision.","section":"Section 4.2"},{"comment":"The layout of the error-analysis tables makes it difficult to map each question to the answer columns and to see which output is being compared to which ground truth; please add column headings per model, bold the divergent answers, and state the criterion used to label an output as a hallucination.","section":"Section 5, Tables 7 and 8"},{"comment":"The description of QuAC mentions context-dependent conversational questions but does not state how the dialogue history was encoded (or whether it was discarded) when the model was applied; a sentence on this is needed for reproducibility, since it can materially affect QuAC accuracy.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"Please verify the anonymity of the code repository cited in Section 3.5 and the Data Availability section: the GitHub URL contains the repository path 'uncertainity-quantification/reliability-estimation-qa1', which may identify the authors under a double-blind policy. My main recommendation to the editor is that the paper's central claim about MCD being non-disruptive is salvageable, but only with a per-sample re-analysis of stability; averaged correlations will not settle the question. The abstract must also be brought into agreement with Tables 3–5. If the authors cannot supply per-sample evidence of stability, the validation claim should be substantially weakened or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I read the paper closely. My take: it's a solid, reproducible empirical benchmark that overclaims its headline result. The data in their own Table 3 contradict the abstract's claim that RoBERTa is reliably better and DistilBERT is inconsistent—on SQuAD, DistilBERT has lower standard deviation than RoBERTa on both cosine and F1, and on QuAC the Welch test between them gives p=0.45. The text in Section 4.2 partly admits this, but the abstract doesn't.\n\nWhat's genuinely useful: the systematic dropout-rate sweep (0–35%) with a reasoned choice of 10%, the pairwise Wilcoxon tests with Bonferroni correction for paraphrase robustness, the human evaluation with Fleiss' kappa on a stratified 10% subset, and the released code. The error-analysis tables are vivid and helpful. If the goal is a practical reliability comparison, the framework is a reasonable starting point.\n\nThe soft spots are real. The biggest one is Table 5. They compute the correlation between unperturbed scores and scores averaged over N=50 MCD samples. That averaged quantity converges to the no-dropout score by the law of large numbers, so high correlations are nearly automatic and don't demonstrate that individual stochastic predictions are stable. Their own Tables 3–4 standard deviations are more direct evidence, but the abstract's 'MCD does not disrupt inference dynamics' leans on the weak correlation result. They should either report per-sample correlations or drop that claim to something like 'averaged predictions remain close to deterministic predictions.'\n\nSecond, the semantic-similarity accuracy (cosine >= 0.95) is never calibrated to human judgments. The human evaluation re-labels responses already deemed incorrect by that metric, so it doesn't validate the threshold. They should compute agreement between the 0.95 threshold and human labels on the annotated subset.\n\nThird, the reliability ranking isn't statistically tested across all model pairs for the MCD condition; only RoBERTa vs DistilBERT is tested. The paraphrase robustness does get full pairwise tests, which is good.\n\nMinor: the Tiny Llama appendix is tangential and some related-work paragraphs are filler. The citations of Miok et al. are appropriate for the method.\n\nVerdict: This deserves peer review—an editor should send it out, and I'd expect major revision to align the abstract and conclusions with the data, and to reframe or replace the correlation evidence. It's a useful benchmark for people working on QA reliability, but not a definitive one.","headline":"A reproducible empirical benchmark whose headline rankings don't match its own tables, and whose key correlation evidence is inflated by averaging over MCD samples.","tokens_in":19783,"tokens_out":4326,"would_cite":false,"duration_ms":66362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Monte Carlo dropout gives BERT QA models a usable reliability signal: enabling it at 10% during inference leaves scores statistically unchanged while exposing per-sample variability, and paraphrasing shows reliability rankings depend on…","keywords":["reliability estimation","Monte Carlo dropout","question answering","BERT variants","prediction stability","paraphrasing perturbation","SQuAD","QuAC"],"falsifier":"A calibration study would settle it: take a stratified sample of SQuAD and QuAC predictions, have annotators label each answer as correct, partially correct, or incorrect, and compare those labels with the cosine-similarity-to-gold scores and the 0.95 threshold. If many predictions above the threshold are judged wrong by humans, or many below it are judged right, the reliability rankings do not measure what they claim. A second check is to rerun the Monte Carlo dropout protocol at several dropout rates and random seeds and see whether the model-level ranking by variance is stable across runs.","tokens_in":18763,"feed_emoji":"🤖","tokens_out":9286,"duration_ms":91063,"temperature":0.7,"pith_summary":"This paper tries to establish that the reliability of BERT-family question-answering models can be measured by perturbing the model and the input rather than by accuracy alone. Reliability is defined as the stability of answers when Monte Carlo dropout is enabled at inference and when questions are paraphrased, with cosine similarity and $\\mathrm{F_1}$ against the gold answer as the scoring functions. On SQuAD and QuAC, the authors report that enabling Monte Carlo dropout leaves accuracy and prediction rankings statistically intact, which they read as validating dropout as a reliability metric. They also report that reliability and accuracy come apart: RoBERTa is generally the most consistent, DistilBERT is stable under internal variation, ALBERT is competitive under paraphrasing but highly variable on answerable questions, and BERT-Base is the weakest overall. If the argument holds, model selection for real QA systems should weigh stability under perturbation as heavily as benchmark accuracy.","feed_headline":"Stability under dropout ranks BERT QA models by reliability","feed_subtitle":"RoBERTa and DistilBERT lead; ALBERT's variance on answerable questions shows accuracy is not enough.","key_machinery":"The load-bearing mechanism is repeated stochastic inference: each question is passed through the model 50 times with a 10% dropout rate enabled, and the mean and standard deviation of cosine similarity and $\\mathrm{F_1}$ scores against the gold answer summarize the answer distribution. For input perturbation, a pre-trained paraphrasing model rewrites each question and a cosine filter keeps paraphrases within a 0.75–0.98 similarity band before the paraphrased answer is scored against the gold answer. The operating point was chosen after a sweep: below 5% dropout yields no stochastic spread, while at or above 15% models start emitting blank responses on answerable questions. The paper then uses the correlation between baseline and dropout-era scores as evidence that dropout does not disrupt inference dynamics, and uses Welch's t-test and Bonferroni-corrected Wilcoxon signed-rank tests to decide which model differences are statistically meaningful.","core_discovery":"The central claim is that a single hyperparameter change—keeping dropout active during prediction—turns a QA model into a distribution over answers whose spread is a usable reliability signal. With a 10% dropout rate and 50 stochastic samples per question, the paper finds high per-question correlation between baseline and Monte-Carlo-dropout scores (coefficients from about 0.857 to 0.946 across models and datasets), while the standard deviation of cosine similarity across samples exposes which models are stable. On this measure RoBERTa and DistilBERT come out most consistent, ALBERT shows large variance on answerable questions, and BERT-Base is middling; paraphrasing the input changes the ranking, with ALBERT leading on QuAC and RoBERTa leading on SQuAD. The paper takes these results to show both that accuracy and reliability are partly independent and that Monte Carlo dropout is a valid, non-disruptive way to estimate reliability for QA.","pith_inferences":["Not in the paper: the same protocol could be used as a retraining-free stress test during model development, inserting a stability screen before deployment to reject models whose answer distributions widen under dropout.","Not in the paper: if cosine similarity to the gold answer is accepted as the correctness signal, the framework is a cheap proxy for uncertainty estimation, and a natural extension is to test whether the 10% dropout protocol transfers to generative QA models, where the paper's appendix suggests accuracy may be too low for the metric to be informative.","Not in the paper: the 0.75–0.98 paraphrase filter is a tunable knob, and reliability rankings should be checked across several similarity bands before treating them as fixed model properties."],"forward_implications":["Enabling Monte Carlo dropout at 10% during QA inference can serve as a reliability check without retraining or materially changing test-time accuracy.","RoBERTa and DistilBERT are the safer choices among the four when answer consistency matters on standard QA benchmarks.","Accuracy and reliability are partly decoupled: a high-accuracy model such as RoBERTa still shows detectable variance, while ALBERT's low answerable accuracy on QuAC coexists with competitive paraphrase stability.","Reliability rankings are dataset-specific; on QuAC, where answerable accuracy is much lower, model differences under paraphrasing reverse the SQuAD ordering."],"supporting_citations":[{"why":"Establishes the MCD-based reliability estimation method for BERT classifiers that this paper adapts to answer extraction.","marker":"[17]"},{"why":"Supplies the SQuAD benchmark with answerable and unanswerable questions used to measure high-accuracy reliability.","marker":"[24]"},{"why":"Supplies the QuAC conversational QA benchmark used to test reliability when model accuracy is low.","marker":"[5]"},{"why":"Supplies dropout regularization theory that justifies stochastic sampling during prediction.","marker":"[3]"},{"why":"Supplies the BERT architecture that all four evaluated models are variants of.","marker":"[6]"},{"why":"Supplies the RoBERTa variant and its SQuAD results that the paper extends with reliability analysis.","marker":"[14]"},{"why":"Supplies the ALBERT variant with parameter sharing that the paper evaluates.","marker":"[10]"},{"why":"Supplies Fleiss' kappa for measuring inter-annotator agreement in the human evaluation.","marker":"[11]"}],"fun_headline_variants":["Monte Carlo dropout reveals which BERT QA models you can trust","Dropout stability ranks BERT QA reliability: RoBERTa tops, ALBERT lags","Keep dropout on to measure QA model reliability","RoBERTa and DistilBERT steady, ALBERT shaky under dropout","One hyperparameter change exposes QA model trustworthiness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that cosine similarity between the vector representation of the predicted answer and the vector representation of the gold answer measures whether an answer is correct and stable; the 0.95 threshold used to define accuracy is never calibrated against human judgments, and the human evaluation only re-examines automatically incorrect responses.","fun_headline_variants_meta":{"raw":{"variants":["Monte Carlo dropout reveals which BERT QA models you can trust","Dropout stability ranks BERT QA reliability: RoBERTa tops, ALBERT lags","Keep dropout on to measure QA model reliability","RoBERTa and DistilBERT steady, ALBERT shaky under dropout","One hyperparameter change exposes QA model trustworthiness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":3010,"prompt_tokens":969,"completion_tokens":2041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1953}},"tokens_in":585,"tokens_out":2041,"duration_ms":14558,"temperature":1.0,"reasoning_tokens":1953,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:59:59.805755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A calibration study would settle it: take a stratified sample of SQuAD and QuAC predictions, have annotators label each answer as correct, partially correct, or incorrect, and compare those labels with the cosine-similarity-to-gold scores and the 0.95 threshold. If many predictions above the threshold are judged wrong by humans, or many below it are judged right, the reliability rankings do not measure what they claim. A second check is to rerun the Monte Carlo dropout protocol at several dropout rates and random seeds and see whether the model-level ranking by variance is stable across runs.","supporting_citations":[{"cited_title":"To BAN or not to BAN: Bayesian attention networks for reliable hate speech detection.Cognitive Computation, 14(1):353–371, 2022","cited_arxiv_id":null,"evidence_quote":"Establishes the MCD-based reliability estimation method for BERT classifiers that this paper adapts to answer extraction."},{"cited_title":"QuAC: Question answering in context","cited_arxiv_id":null,"evidence_quote":"Supplies the QuAC conversational QA benchmark used to test reliability when model accuracy is low."},{"cited_title":"Understanding dropout.Advances in neural information processing systems, 26, 2013","cited_arxiv_id":null,"evidence_quote":"Supplies dropout regularization theory that justifies stochastic sampling during prediction."},{"cited_title":"BERT: Pre-training of deep bidirectional transformers for language un- derstanding","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT architecture that all four evaluated models are variants of."},{"cited_title":"AlBERT: A lite BERT for self-supervised learning of language representations.International Conference on Learning Representations., 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the ALBERT variant with parameter sharing that the paper evaluates."},{"cited_title":"The measurement of observer agree- ment for categorical data.biometrics, pages 159–174, 1977","cited_arxiv_id":null,"evidence_quote":"Supplies Fleiss' kappa for measuring inter-annotator agreement in the human evaluation."}],"review_version":1}