{"id":"50b61db1-104d-4e1a-b12a-f01eb2ae1e97","arxiv_id":"1908.09720","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A class-weighted voting ensemble of BiDAF, QANet, and Mnemonic Reader reports F1 81.96 and EM 73.77 on the SQuAD dev set, marginally above Mnemonic Reader's 81.57 and 73.25.","lead":"This paper builds a voting ensemble from three existing reading-comprehension models and gives each model extra weight on question types where it performs well. The best ensemble scores slightly above the strongest single model on SQuAD, though the gain is small and lacks error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.39 F1 edge over Mnemonic Reader rests on a single run with no variance estimate; given the ensemble changes only a minority of predictions, the gain may be noise.","rationale":"The reader's weakest assumption is that class-specific F1 weights computed on a 5% pre-evaluation split remain valid for the final full-data models and for the evaluation set. My concern is adjacent but broader: even if the holdout were perfectly representative, the paper provides only a single run of each model and selects the final variant using the evaluation set itself, so the 0.39 F1 gain over Mnemonic Reader has no quantified reliability. The paper's own agreement tables show the base models agree on the majority of questions, making the ensemble's edge depend on a small, noise-sensitive subset of decisions. The class-aware versus non-class-aware difference (0.1 F1, 0.05 EM) is especially fragile. This does not invalidate the paper, but it does mean the central claim should be treated as preliminary; the reader's CONDITIONAL verdict already reflects that. The diversity analysis and class-wise breakdowns are useful supporting evidence, but they do not close the variance question. A multi-seed replication or paired bootstrap confidence interval would settle whether the reported improvement is real. Therefore I recommend no change to the reader's verdict.","tokens_in":8795,"tokens_out":3993,"duration_ms":45183,"concrete_test":"Replicate the full pipeline with at least 10 random seeds and 10 different 5% holdout splits, training all three base models from scratch each time and computing class-aware ensemble plus Mnemonic Reader F1/EM on the same evaluation set. Then compute a paired bootstrap 95% confidence interval for the ensemble-minus-Mnemonic F1 difference across the 10 runs. If the interval includes 0, or if fewer than 8 of 10 runs favor the ensemble, the reported 0.39-point improvement is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the class-aware ensemble's 0.39 F1 / 0.52 EM improvement over Mnemonic Reader in Table 11. The paper reports one run with one random 5% holdout split and gives no error bars, significance test, or repeated-seed analysis. This is load-bearing because the ensemble mechanism rarely overrides the best model: Tables 5-10 show Mnemonic and QANet produce equal F1 on 74.3% of evaluation questions (81.0% equal EM), and Mnemonic vs. BiDAF agree on 69.9% F1 / 77.9% EM. Algorithm 2 also falls back to the globally best model for undefined classes and whenever no duplicate answers occur, so the ensemble output is often just Mnemonic's answer. The 0.39-point gain therefore depends on a small number of answer substitutions, which are precisely the cases most sensitive to training noise and to the accuracy of the class weights estimated from the 5% split. The class-aware ensemble's advantage over the non-class-aware ensemble is only 0.1 F1 and 0.05 EM, near the level of a single question on the 10,570-item set. Finally, several variants (undefined-class handling, EM vs. F1 weights, length-based classes) were tested on the same evaluation set and the best one reported, so the headline number carries selection bias. Without repeated runs or a confidence interval, the reported improvement is statistically indistinguishable from noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a class-aware weighted voting ensemble for SQuAD question answering. It uses three published architectures (BiDAF, QANet, Mnemonic Reader), defines 14 question classes, splits the SQuAD training set 95/5, trains the base models on the 95% subset to measure per-class average F1, and uses these as voting weights in Algorithm 2. Answers are combined by summing weights of duplicate candidate answers and otherwise falling back to the globally best model. On the SQuAD dev set, the class-aware ensemble scores F1 81.96 and EM 73.77 versus 81.57/73.25 for the best single model (Mnemonic Reader); the non-class-aware ensemble scores 81.86/73.72. The authors also experiment with variants (sum vs. max weights, undefined-class handling, EM-based weights, length-based classification).","tokens_in":9099,"tokens_out":4803,"duration_ms":45609,"significance":"If the reported gains are real, the paper demonstrates a simple, inexpensive way to push SQuAD accuracy beyond a strong single model, and its per-class error analysis is a useful diagnostic. The main strengths are that the voting weights are estimated on a held-out portion of the training set (a legitimate design) and that Algorithms 1–4 are specified explicitly, making the method easy to implement. However, the measured improvement is very small (0.39 F1, 0.52 EM), comes from a single run with no significance testing, and the final configuration was chosen after trying several variants on the same evaluation set. As it stands, the central 'outperforms' claim is plausible but not statistically established.","major_comments":[{"comment":"The central claim rests on a single evaluation run. No error bars, confidence intervals, or significance tests are reported, and the improvements over Mnemonic Reader (0.39 F1, 0.52 EM) and over the non-class ensemble (0.1 F1, 0.05 EM) are small relative to the size of the evaluation set. Since training is stochastic and the 5% pre-evaluation split is random, the reported gain could be within run-to-run variation. Please provide repeated runs with different seeds and split draws, and report paired significance tests (or bootstrap confidence intervals) over the 10,570 evaluation questions.","section":"§5, Table 11"},{"comment":"Several alternative design choices (adding vs. taking the max of duplicate weights, handling undefined questions with voting, using EM rather than F1 weights, and length-based classification) were evaluated directly on the SQuAD evaluation set, and the best-performing variant was then reported as the headline result. This is a form of selection on the test set; the reported advantage may overstate the true improvement of the chosen configuration. The paper should either report results for all variants with error bars, or select the configuration on a separate validation split and then apply it once to the evaluation set.","section":"§5, variant testing paragraphs"},{"comment":"The class-specific weights are estimated from a single random 5% split of the training set, where several classes have very few examples (how big/size: 4, how old: 6, what time: 11, whom: 20). These noisy estimates are then used to weight the final models trained on the full training set, but the paper does not check whether the weights are stable across different 5% splits or across the 95%-vs-100% training regimes. A k-fold or bootstrap procedure over the training set would show whether the measured substitutions that produce the 0.39 F1 gain are robust.","section":"§3, Table 3 and §4, Algorithms 1–2"}],"minor_comments":[{"comment":"QANet is evaluated on 10,482 questions rather than 10,570; please explain the discrepancy (e.g., missing predictions) and how Algorithm 2 handles questions without a QANet candidate.","section":"Tables 5–10"},{"comment":"The figures lack descriptive captions and axis labels; this makes it difficult to interpret the per-class behavior that is central to the paper. Please add proper captions and label the axes.","section":"Figures 1–6"},{"comment":"The manuscript has many typos and grammatical errors, such as 'Each training contains of 30 epochs', 'Figures 5 ans 6', and 'how has the SQuAD data set been splitted'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The claim that merging which and what classes 'has no influence on the final results' is unsupported; please provide evidence (e.g., comparative results) or remove the claim.","section":"§3, merging which into what"},{"comment":"The description of the 'how old' example is confusing: if BiDAF has the highest voting weight, it is not clear why adding the weights of Mnemonic and QANet would override it unless their summed weight exceeds BiDAF's weight. Please clarify the exact voting arithmetic in this example.","section":"§5, how old example"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline empirical paper. The method is simple and transparent, but the claimed gain is small and is supported by a single test-set comparison with selection over variants. I would not reject outright because the issues are addressable with additional experiments, but the revision needs to be evaluated carefully. The paper does not mention releasing code or data, which further weakens reproducibility; I would encourage the editor to ask for that."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, clearly written ensemble study. The new piece is the per-question-class weighting of three off-the-shelf QA models, with class weights computed on a 5% holdout of SQuAD training data. That idea is legitimate, and the paper is honest about its exploration: it tells you that EM-based weights, length-based classes, and voting on undefined questions all made things worse. That is useful negative information. The per-class agreement tables are the most valuable part—they show that the three models disagree in systematic ways, which is a genuinely useful observation for anyone building QA ensembles.\n\nThe problem is the load-bearing claim. The 0.39 F1 and 0.52 EM gains over Mnemonic Reader come from a single evaluation run. No error bars, no repeated seeds, no significance test. The paper also tells you it picked the final configuration by testing alternatives on the evaluation set (Section 5: F1 vs EM weights, add vs max, undefined handling, length classification). That means the headline number carries selection bias, and the paper gives you no way to correct for it. The class-aware ensemble is only 0.1 F1 and 0.05 EM better than the plain weighted ensemble; on a 10,570-question set that is a very small number of question-level changes. Given the ensemble falls back to Mnemonic Reader whenever there is no duplicate answer or the class is undefined, the improvement rests on a small number of answer swaps, which are precisely the cases most sensitive to training noise. The paper's own similarity tables (74.3% equal F1 between Mnemonic and QANet) confirm that the ensemble barely changes most predictions.\n\nSo the central empirical claim is plausible but not established. The diagnostic insight about class-level complementarity is real and worth reporting, but the 0.4-point edge should not be taken as a robust result.\n\nWho is this for? Practitioners who want a concrete recipe for combining QA models, and anyone interested in the failure modes of small-dev-set evaluation. It could be a decent workshop paper or technical note if the authors add multi-seed runs and an honest validation split for configuration selection.\n\nI would not desk-reject it—the method is clear and the negative results are honest—but I would send it back for a major revision requiring variance estimates and a configuration-selection protocol that does not use the evaluation set.","headline":"A clearly written but statistically weak ensemble study: the class-aware weighting idea is real, yet the reported 0.4 F1 gain over the best single model could easily be within run-to-run noise.","tokens_in":9619,"tokens_out":2739,"would_cite":false,"duration_ms":29484,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A class-aware weighted-voting ensemble of three attention-based reading models reaches F1 81.96 and EM 73.77 on SQuAD, beating its best single model.","keywords":["natural language processing","machine comprehension","deep learning","question answering","ensemble learning","weighted voting","SQuAD","attention mechanism"],"falsifier":"Repeat the experiment with several different random 5% splits of the SQuAD training set, retraining all three base models from scratch for each split. If the class-aware ensemble fails to beat Mnemonic Reader in the majority of splits, or if the 0.39-point F1 advantage falls inside the split-to-split variance, the claimed improvement is not stable.","tokens_in":8577,"feed_emoji":"📚","tokens_out":9196,"duration_ms":84096,"temperature":0.7,"pith_summary":"This paper asks whether combining three strong neural reading-comprehension models can beat the best single model on the SQuAD benchmark. It answers yes: an ensemble that groups questions by type and weights each model's vote by its per-class F1 measured on a 5% holdout reaches F1 81.96 and exact-match 73.77, ahead of the best member, Mnemonic Reader (F1 81.57, EM 73.25). The gain is small but appears on both metrics, and a version without question classes also beats the best single model, though by slightly less. If the result holds, it shows that question-type specialization among diverse architectures can be exploited by voting without designing a new network.","feed_headline":"Class-weighted voting beats best single QA model on SQuAD","feed_subtitle":"Three attention-based readers vote per question type, gaining 0.39 F1 over the strongest one.","key_machinery":"The central mechanism is question-class-aware weighted voting. Fourteen question classes are defined by the interrogative phrase opening each SQuAD question: date, during, how are, how big/size, how m/m, how old, what, what time, when, where, who, whom, why, and undefined. For each class, each model's voting weight is the average F1 it scored on that class in a 5% holdout of the training set, using models trained on the remaining 95%. The voting rule then aggregates candidate answers by summing the weights of identical answers, chooses the highest total, and falls back to the globally best model for non-duplicated or undefined cases. The load-bearing observation is that the three base models disagree on most questions, leaving room for voting to correct individual errors.","core_discovery":"The paper claims that a class-specific weighted-voting ensemble of BiDAF, QANet, and Mnemonic Reader outperforms every one of them on SQuAD. The voting weights come from a 5% pre-evaluation split: models are trained on the other 95% of the training set, scored separately on each of fourteen question classes derived from interrogative phrases, and each model receives the average F1 it achieves in each class as its voting weight. At test time, models trained on the full training set each propose an answer; if two or more models give the same answer, their class weights are summed, and the answer with the highest total is returned. When no answer is duplicated, or the question falls outside the defined classes, the globally best model's answer is returned. This procedure reaches F1 81.96 and EM 73.77 on the SQuAD evaluation set, exceeding the best standalone model, Mnemonic Reader, by 0.39 and 0.52 points, and exceeding a class-blind version of the same voting by 0.10 and 0.05 points.","pith_inferences":["The reported margins are small enough that a skeptic would want a significance test across several random 5% splits; the paper gives no variance or repeated-run numbers, so the stability of the 0.39-point F1 edge is open.","Since roughly 53% of SQuAD questions belong to the what class, even a modest improvement concentrated in that dominant class could explain most of the headline gain; testing on a class-balanced evaluation set would show whether the mechanism helps rare question types.","The same recipe should transfer to other reading-comprehension datasets whose questions can be typed by surface cues, but the 5% holdout may be too small on less skewed collections, where cross-validated or online weight estimation would be a natural extension.","The method is a form of mixture-of-experts gated by question type; replacing the hand-built interrogative-phrase taxonomy with a learned question embedding could automate grouping and might generalize beyond the fourteen classes used here."],"forward_implications":["An ensemble built from existing attention-based reading models improves over the strongest member on SQuAD without any architectural change, so model diversity across question types is a usable resource.","Because the class-blind weighted ensemble already beats the best single model, part of the gain comes from voting itself, and the class-aware weighting adds a further 0.10 F1 and 0.05 EM.","Per-class weights estimated from only 5% of the training set are enough, in this setup, to guide voting on the full evaluation set; the paper reports that splitting factors 0.01 and 0.1 gave worse overall results.","The ensemble is not better on every question class: for how-old questions it scores below BiDAF because adding the weights of two equal wrong answers can outweigh one correct answer, so the method's value is global rather than per-class."],"supporting_citations":[{"why":"Provides BiDAF, one of the three base architectures and the source of its context-to-query and query-to-context attention variants.","marker":"[2]"},{"why":"Provides QANet, the second base model, whose class-specific strengths the ensemble exploits for date and how-old questions.","marker":"[23]"},{"why":"Provides Mnemonic Reader, the strongest standalone model and the baseline the ensemble must beat.","marker":"[24]"},{"why":"Supplies the general ensemble-methods background that motivates combining separately trained models by weighted predictions.","marker":"[27]"},{"why":"Supplies bagging, the classic voting-ensemble rationale the paper's weighting scheme builds on.","marker":"[28]"},{"why":"Provides the open-source BiDAF implementation the experiments are based on.","marker":"[29]"},{"why":"Provides the open-source Mnemonic Reader implementation the experiments are based on.","marker":"[30]"}],"fun_headline_variants":["Voting ensemble of QA models edges past Mnemonic Reader on SQuAD","Class-aware voting lifts SQuAD F1 by 0.39 over top single model","Weighted voter among three readers wins SQuAD by a hair","Ensemble vote: three QA models beat best single on SQuAD","SQuAD: class-weighted voting beats best reader"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The class-specific F1 weights measured on a 5% holdout of the SQuAD training set, using models trained on the remaining 95%, are assumed to stay representative when the final models are trained on the full dataset and evaluated on the official evaluation set.","fun_headline_variants_meta":{"raw":{"variants":["Voting ensemble of QA models edges past Mnemonic Reader on SQuAD","Class-aware voting lifts SQuAD F1 by 0.39 over top single model","Weighted voter among three readers wins SQuAD by a hair","Ensemble vote: three QA models beat best single on SQuAD","SQuAD: class-weighted voting beats best reader"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":2963,"prompt_tokens":873,"completion_tokens":2090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1993}},"tokens_in":489,"tokens_out":2090,"duration_ms":13536,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:03:36.533737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the experiment with several different random 5% splits of the SQuAD training set, retraining all three base models from scratch for each split. If the class-aware ensemble fails to beat Mnemonic Reader in the majority of splits, or if the 0.39-point F1 advantage falls inside the split-to-split variance, the claimed improvement is not stable.","supporting_citations":[{"cited_title":"Bidirectional Attention Flow for Machine Comprehension","cited_arxiv_id":null,"evidence_quote":"Provides BiDAF, one of the three base architectures and the source of its context-to-query and query-to-context attention variants."},{"cited_title":"Journal of Artificial Intelligence Research vol.11 pp.169-198, 1999","cited_arxiv_id":null,"evidence_quote":"Supplies the general ensemble-methods background that motivates combining separately trained models by weighted predictions."},{"cited_title":"Bagging Predictors","cited_arxiv_id":null,"evidence_quote":"Supplies bagging, the classic voting-ensemble rationale the paper's weighting scheme builds on."},{"cited_title":"Re-implementation of BiDAF in PyTorch","cited_arxiv_id":null,"evidence_quote":"Provides the open-source BiDAF implementation the experiments are based on."},{"cited_title":"A PyTorch implementation of Mnemonic Reader for the Machine Comprehension task, https://github.com/HK , UST-KnowComp/MnemonicRea der","cited_arxiv_id":null,"evidence_quote":"Provides the open-source Mnemonic Reader implementation the experiments are based on."}],"review_version":1}