{"id":"fec835ef-9d83-4868-8096-cc34cae9aba0","arxiv_id":"1909.00215","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A mutual information maximization regularizer (QAInfomax) improves BERT's robustness on Adversarial-SQuAD, achieving state-of-the-art F1 on ADDSENT and ADDONESENT.","lead":"QAInfomax adds a mutual information regularizer to question answering models, improving their resistance to adversarial distractor sentences. The paper reports state-of-the-art results on Adversarial-SQuAD with BERT, showing the regularizer helps without hurting original accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The observed robustness gain is attributed to MI maximization, but Table 2 ablates the entire auxiliary objective rather than the MI signal; without a same-capacity control critic trained on non-MI labels, the causal claim is underdetermined.","rationale":"The reader's weakest assumption points at exactly the same gap: no control for a similarly parameterized auxiliary classifier. I agree that is the most load-bearing issue. The paper does provide a useful ablation (LC vs GC) and releases code, so the empirical recipe is checkable; the issue is not reproducibility but interpretation. If the MI signal is interchangeable with any auxiliary critic objective, then the theoretical motivation in Sections 2–3 is not doing the explanatory work claimed in the abstract and conclusion. The suggested control directly tests that. The existing CONDITIONAL verdict is appropriate: the result is promising but the central mechanism is not yet pinned down. No verdict change is needed.","tokens_in":7441,"tokens_out":5006,"duration_ms":51025,"concrete_test":"Retrain BERT-S under four conditions with identical data, optimizer, epoch count, and at least five seeds: (a) vanilla Lspan; (b) QAInfomax exactly as in §3.3; (c) same LC/GC critics and α,β,γ, but the positive/negative labels for the BCE objective are randomly permuted each step, so g receives no genuine MI signal; (d) same critics but with the BCE objective negated (anti-MI). Report mean±std F1 on ADDSENT and ADDONESENT. If (c) or (d) is within noise of (b), the MI objective is not the driver; if (b) significantly exceeds both, the attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that adding the QAInfomax loss (Eq. 5: L = Lspan + γLinfo) improves adversarial robustness because it maximizes mutual information among passage, question, and answer. Section 3.3 defines Linfo via the LC and GC objectives, and Table 2 shows +LC (53.6/64.2), +GC (52.2/63.7), +LC+GC (54.5/64.9) versus vanilla BERT (51.0/63.4). But every ablation row that adds an auxiliary term also adds a parameterized bilinear critic g (Eq. 6) with an extra loss and extra training signal. The design never holds the critic architecture and loss scale fixed while changing the MI content of the target. A similarly parameterized auxiliary classifier trained on shuffled labels, or even an anti-MI objective, could plausibly yield comparable gains through generic regularization, modified optimization dynamics, or implicit noise. If that happened, the title/abstract claim that MI maximization drives robustness would not be supported, even though the recipe might still be useful. The reported 'best of at least three runs' numbers without variance further limit how confidently the 3.5-point ADDSENT gap can be attributed to the specific objective. This is an attribution gap, not an allegation of fabrication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QAInfomax, a regularizer for extractive question answering that augments a BERT-based QA system with two contrastive mutual-information objectives: a local constraint (LC) that encourages each answer word representation to share information with the rest of the answer span and its local context, and a global constraint (GC) that encourages a summarized answer representation to share information with all question and passage representations. The final training loss is L = Lspan + γLinfo, where Linfo combines LC and GC via a bilinear discriminator. The method is evaluated on Adversarial-SQuAD, and the reported results show BERT-S + QAInfomax achieving F1 scores of 54.5 on ADDSENT and 64.9 on ADDONESENT, improving over vanilla BERT (51.0/63.4) and surpassing prior published results without additional training data. The ablation in Table 2 indicates that both LC and GC contribute, and Table 3 compares three summarization functions for GC. The paper claims state-of-the-art robustness on Adversarial-SQuAD and attributes the gains to mutual information maximization.","tokens_in":7766,"tokens_out":3906,"duration_ms":40548,"significance":"If the central claim holds, the contribution is useful and timely: it introduces a DIM-style mutual information regularizer to an NLP task, shows that it can be combined with strong pretrained models, and reports robustness gains on a challenging adversarial benchmark without extra training data. The paper is reproducible in spirit: the code is publicly linked, the hyperparameters are fully specified, and the base model is a standard PyTorch reimplementation of BERT. The ablation and the comparison with prior systems are also clearly presented. However, the experiments do not currently isolate the causal role of mutual information maximization from the generic effects of adding an auxiliary contrastive loss, and the statistical support for the headline improvement is thin. These gaps need to be addressed before the central claim can be considered established.","major_comments":[{"comment":"The experiments do not separate the effect of the MI objective from the generic effect of an auxiliary contrastive loss with a bilinear critic. Every ablation row that includes LC or GC also adds the parameterized discriminator g from Eq. (6) and its own training signal, so the observed gains may come from extra capacity, changed optimization dynamics, or implicit regularization rather than from maximizing a specific MI quantity. I would like to see a control experiment that trains the same critic and loss scale with shuffled or anti-informative pairings (e.g., permuted labels or deliberately negative MI targets) while keeping the architecture and γ fixed. If such a control yields comparable improvements, the claim that MI maximization drives robustness would not be supported by the current evidence.","section":"Section 3.3, Table 2"},{"comment":"The statistical claims are not supported by the reported information. The text says that the dagger in Table 1 indicates a significant improvement over baselines with p-value < 0.05, but it does not specify the test, the number of runs, the variance, or the effect size. Furthermore, Section 4.1 states that the reported numbers are the best across at least three runs, which systematically favors noisy runs and makes the 3.5-point ADDSENT gap difficult to interpret. Please report mean ± standard deviation over the runs and specify the paired significance procedure (e.g., paired bootstrap or a paired test on per-example scores).","section":"Section 4.1, Tables 1 and 2"},{"comment":"Equation (2) modifies the DIM/JS lower bound by summing both directions of negative sampling with coefficient 1/2, but no derivation or citation is given to show that this symmetric objective still lower-bounds mutual information. Since the method is called QAInfomax and the title claims mutual information maximization, the paper should either provide a short justification for Eq. (2) as a MI bound or explicitly label it as a heuristic contrastive objective that is only inspired by MI estimation. This clarification is needed because the central attribution to MI maximization depends on the validity of this estimator.","section":"Section 2, Eq. (2)"}],"minor_comments":[{"comment":"The conclusion contains a typo: \"Adversirial\" should be \"Adversarial\".","section":"Section 5"},{"comment":"In the first example, the QAInfomax prediction is spelled \"Issac Newton\" instead of \"Isaac Newton\"; the spelling should be corrected. Also, the third example shows that QAInfomax still fails on an adversarial distractor (\"Rosetta laws\"), which is worth acknowledging in the main text rather than only in an appendix example.","section":"Appendix"},{"comment":"The hyperparameters C, α, β, and γ are fixed to 5, 1, 0.5, and 0.3 without any sensitivity analysis or discussion of how they were selected. A small sensitivity table would help readers understand how stable the reported gains are to these choices.","section":"Section 4.1"},{"comment":"The table text contains rendering artifacts such as \"A DDSENT\" and \"ADDONESENT\" with unusual spacing in several places; these should be cleaned up in the final version.","section":"Tables and text"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the empirical results are encouraging, but the attribution to mutual information maximization is currently underdetermined by the reported experiments. A single control condition (same critic and loss scale, non-MI targets) and proper variance reporting would likely settle the issue. The paper is otherwise clearly written and well-scoped for a workshop or conference submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, useful empirical paper whose recipe probably works, but the title claim—that MI maximization is what drives the robustness gain—is not actually established by the experiments. The contribution is real: first application of Deep Infomax-style MI estimation as an auxiliary objective for extractive QA, with two sensible local/global constraints. The gains on Adversarial-SQuAD are respectable: +3.5 F1 ADDSENT and +1.5 ADDONESENT over BERT-S, with original SQuAD performance essentially unchanged. The ablation shows both constraints matter. That is worth a referee.\n\nWhat the paper does well: the objective is clearly specified, the negative sampling scheme is standard, and the authors are transparent about training speed cost and the fact that they do not compare with Wang and Bansal because those used modified training data. The appendix includes a case where QAInfomax still gets fooled, which is at least not cherry-picked.\n\nThe soft spots are real but not fatal. First, the attribution gap the stress-test flags: Table 2 compares vanilla BERT vs BERT plus the full auxiliary objective. Every auxiliary row adds a bilinear critic and an extra training signal. There is no control that holds the critic architecture and loss scale fixed while changing the MI content (e.g., shuffled labels or an anti-MI target). So we know the recipe helps, but not that maximizing MI is the active ingredient. Second, numbers are best-of-at-least-three runs without variance or error bars, and the p<0.05 claim has no test details. With a 3.5-point gap, that matters. Third, no hyperparameter sensitivity beyond the fixed C, alpha, beta, gamma; the speed hit for GC (28%) is acknowledged but not addressed.\n\nThe circularity burden is fine; no fitted-value issue. The MI estimator is borrowed and the gains are empirical. Citation pattern is appropriate—they cite DIM, MINE, CPC and the QA baselines.\n\nWho this is for: people working on robust reading comprehension or contrastive regularizers. I would bring it to reading group and cite it. I would send it to peer review; it deserves a serious referee. But I would ask for the control experiment, variance estimates, and more careful wording of the SOTA claim.","headline":"Useful regularizer with a real gain on Adversarial-SQuAD, but the causal claim that MI maximization drives robustness is underdetermined without a control critic; still deserves peer review.","tokens_in":8239,"tokens_out":2175,"would_cite":true,"duration_ms":21123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QAInfomax, a mutual-information regularizer, makes reading-comprehension models resistant to adversarial distractor sentences, achieving state-of-the-art F1 on Adversarial-SQuAD.","keywords":["mutual information","question answering","adversarial robustness","deep infomax","distractor sentences","reading comprehension","regularization","Adversarial-SQuAD"],"falsifier":"Train BERT-S with the QAInfomax setup but replace the MI discriminator's labels with a non-MI target, e.g., predicting whether the answer span is longer than five words or predicting the example index in the batch, keeping all other hyperparameters identical. If the resulting ADDSENT F1 reaches or exceeds 54.5, the robustness improvement is not specific to mutual information.","tokens_in":7264,"feed_emoji":"🛡️","tokens_out":9333,"duration_ms":70645,"temperature":0.7,"pith_summary":"This paper tries to establish that the fragility of reading-comprehension systems to adversarial distractor sentences comes from training that rewards predictive correlations instead of genuine understanding. It proposes QAInfomax, a regularizer that maximizes mutual information among passage, question, and answer representations by contrasting real pairs with shuffled negative pairs. Adding QAInfomax to a BERT-based question-answering system improves F1 on Adversarial-SQuAD from 51.0 to 54.5 on ADDSENT and from 63.4 to 64.9 on ADDONESENT, using only the original SQuAD train set. If the attribution is right, the approach provides a training signal that forces models to make answer choices grounded in the full context, not in coincidental word matches.","feed_headline":"QAInfomax regularizer beats adversarial distractor sentences","feed_subtitle":"Distractor sentences stop fooling QA systems when the model is trained to maximize mutual information.","key_machinery":"The load-bearing object is the deep-infomax style mutual information estimator, adapted to text. The paper computes a lower bound on MI via the Jensen-Shannon divergence, implemented as a binary cross-entropy discriminator score $g(x,y)=x^T W y$, and uses it to build two constraints. The Local Constraint pairs each answer word with surrounding context and other answer words; the Global Constraint pairs the sigmoid-mean summarized answer vector with every passage and question word. These constraints form $L_{infomax}$, added to the span loss as $L = L_{span} + \\gamma L_{infomax}$. The discriminator's ability to separate real from shuffled pairs is what drives the representation learning.","core_discovery":"The central claim is that explicitly maximizing mutual information between the answer representation and its context—both locally around the answer span and globally across the whole passage and question—makes a QA system less susceptible to superficially plausible distractor sentences. QAInfomax estimates mutual information through a Jensen-Shannon divergence lower bound, realized as a binary classifier that scores positive (answer, context) pairs against negative pairs from a shuffled batch. The local constraint acts on each answer word; the global constraint acts on a summarized answer vector and compares it with every word of the question and passage. Combined with the span-prediction loss, this regularizer yields state-of-the-art F1 scores on Adversarial-SQuAD (54.5 ADDSENT, 64.9 ADDONESENT for BERT-S), without any extra training data and without degrading performance on the original SQuAD.","pith_inferences":["If the MI attribution is to be trusted, a necessary control is to replace the discriminator's objective with a non-MI auxiliary task of equal capacity; without that control, the robustness gain could come from auxiliary training signal rather than mutual information maximization specifically.","The local constraint is effectively contrastive learning on token spans; the same regularizer may transfer to span-extraction tasks such as named-entity recognition or temporal extraction, where distractor spans also hurt.","Because the global constraint is trained on full-passage representations, it may be approximated by subsampling words to cut the 28% speed penalty while preserving the robustness signal.","The paper does not test transfer of robustness to unseen adversary types (e.g., paraphrased distractors or word-level perturbations); a positive result there would strengthen the claim that MI encourages human-like reasoning."],"forward_implications":["QAInfomax with both constraints improves worst-case adversarial F1 (ADDSENT) by 3.5 points and random adversarial F1 (ADDONESENT) by 1.5 points over the BERT-S baseline.","The regularizer is effective without access to adversarial examples at training time, using only the original SQuAD training data.","The two constraints are complementary: LC alone gives 53.6/64.2, GC alone gives 52.2/63.7, and both together give 54.5/64.9.","The summarization function for the global constraint is not highly sensitive: mean, max, and random sampling of answer-word representations all give competitive F1 scores.","Training speed drops by 28% when both constraints are used, a cost the paper reports as a limitation."],"supporting_citations":[{"why":"Creates the Adversarial-SQuAD benchmark and documents the fragility of reading-comprehension systems to distractor sentences.","marker":"(Jia and Liang, 2017)"},{"why":"Introduces deep infomax, the mutual-information estimation and maximization framework that QAInfomax adapts.","marker":"(Hjelm et al., 2018)"},{"why":"Provides MINE, the neural mutual-information estimator whose lower bound the discriminator implements.","marker":"(Belghazi et al., 2018)"},{"why":"Establishes the generative QA baseline and the argument that discriminative losses cause overfitting to superficial biases.","marker":"(Lewis and Fan, 2018)"},{"why":"Supplies BERT, the base QA model that QAInfomax is applied to.","marker":"(Devlin et al., 2018)"},{"why":"Contributes the contrastive predictive coding objective and the bilinear discriminator used for scoring pairs.","marker":"(Oord et al., 2018)"},{"why":"Motivates the symmetric negative-sampling modification of the MI objective in equation (2).","marker":"(Tian et al., 2019)"}],"fun_headline_variants":["Mutual information regularizer foils adversarial QA distractors","QAInfomax: Mutual information stops distractor sentences","Maximize mutual info to beat QA distractor attacks","Robust QA: mutual information regularizer defeats distractors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim is that the robustness gain on Adversarial-SQuAD comes from maximizing mutual information, but it never runs a control where the same discriminator architecture is trained with a non-MI auxiliary objective; if that control produced the same F1, the gain would not be attributable to mutual information maximization.","fun_headline_variants_meta":{"raw":{"variants":["Mutual information regularizer foils adversarial QA distractors","QAInfomax: Mutual information stops distractor sentences","Maximize mutual info to beat QA distractor attacks","Robust QA: mutual information regularizer defeats distractors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2498,"prompt_tokens":821,"completion_tokens":1677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":1611}},"tokens_in":437,"tokens_out":1677,"duration_ms":46694,"temperature":1.0,"reasoning_tokens":1611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:57:40.448866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train BERT-S with the QAInfomax setup but replace the MI discriminator's labels with a non-MI target, e.g., predicting whether the answer span is longer than five words or predicting the example index in the batch, keeping all other hyperparameters identical. If the resulting ADDSENT F1 reaches or exceeds 54.5, the robustness improvement is not specific to mutual information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the generative QA baseline and the argument that discriminative losses cause overfitting to superficial biases."}],"review_version":1}