REVIEW 3 major objections 4 minor 21 references
QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read QAInfomax, a mutual-information regularizer, makes reading-comprehension models resistant to adversarial distractor sentences, achieving state-of-the-art F1 on Adversarial-SQuAD.
desk verdict Useful regularizer with a real gain on Adversarial-SQuAD, but the causal claim that MI maximization drives robustness is underdetermined without a control critic; still deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the deep-infomax style mutual information estimator, adapted to text. The paper computes a lower bound on MI via the Jensen-Shannon divergence, implemented as a binary cross-entropy discriminator score $g(x,y)=x^T W y$, and uses it to build two constraints. The Local Constraint pairs each answer word with surrounding context and other answer words; the Global Constraint pairs the sigmoid-mean summarized answer vector with every passage and question word. These constraints form $L_{infomax}$, added to the span loss as $L = L_{span} + \gamma L_{infomax}$. The discriminator's ability to separate real from shuffled pairs is what drives the representation learning.
What would settle it
Train BERT-S with the QAInfomax setup but replace the MI discriminator's labels with a non-MI target, e.g., predicting whether the answer span is longer than five words or predicting the example index in the batch, keeping all other hyperparameters identical. If the resulting ADDSENT F1 reaches or exceeds 54.5, the robustness improvement is not specific to mutual information.
Extended reading notes
Core claim
The central claim is that explicitly maximizing mutual information between the answer representation and its context—both locally around the answer span and globally across the whole passage and question—makes a QA system less susceptible to superficially plausible distractor sentences. QAInfomax estimates mutual information through a Jensen-Shannon divergence lower bound, realized as a binary classifier that scores positive (answer, context) pairs against negative pairs from a shuffled batch. The local constraint acts on each answer word; the global constraint acts on a summarized answer vector and compares it with every word of the question and passage. Combined with the span-prediction loss, this regularizer yields state-of-the-art F1 scores on Adversarial-SQuAD (54.5 ADDSENT, 64.9 ADDONESENT for BERT-S), without any extra training data and without degrading performance on the original SQuAD.
Load-bearing premise
The paper's central claim is that the robustness gain on Adversarial-SQuAD comes from maximizing mutual information, but it never runs a control where the same discriminator architecture is trained with a non-MI auxiliary objective; if that control produced the same F1, the gain would not be attributable to mutual information maximization.
Editorial extensions
If this is right
- QAInfomax with both constraints improves worst-case adversarial F1 (ADDSENT) by 3.5 points and random adversarial F1 (ADDONESENT) by 1.5 points over the BERT-S baseline.
- The regularizer is effective without access to adversarial examples at training time, using only the original SQuAD training data.
- The two constraints are complementary: LC alone gives 53.6/64.2, GC alone gives 52.2/63.7, and both together give 54.5/64.9.
- The summarization function for the global constraint is not highly sensitive: mean, max, and random sampling of answer-word representations all give competitive F1 scores.
- Training speed drops by 28% when both constraints are used, a cost the paper reports as a limitation.
Reading between the lines
- If the MI attribution is to be trusted, a necessary control is to replace the discriminator's objective with a non-MI auxiliary task of equal capacity; without that control, the robustness gain could come from auxiliary training signal rather than mutual information maximization specifically.
- The local constraint is effectively contrastive learning on token spans; the same regularizer may transfer to span-extraction tasks such as named-entity recognition or temporal extraction, where distractor spans also hurt.
- Because the global constraint is trained on full-passage representations, it may be approximated by subsampling words to cut the 28% speed penalty while preserving the robustness signal.
- The paper does not test transfer of robustness to unseen adversary types (e.g., paraphrased distractors or word-level perturbations); a positive result there would strengthen the claim that MI encourages human-like reasoning.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes QAInfomax, a regularizer for extractive question answering that augments a BERT-based QA system with two contrastive mutual-information objectives: a local constraint (LC) that encourages each answer word representation to share information with the rest of the answer span and its local context, and a global constraint (GC) that encourages a summarized answer representation to share information with all question and passage representations. The final training loss is L = Lspan + γLinfo, where Linfo combines LC and GC via a bilinear discriminator. The method is evaluated on Adversarial-SQuAD, and the reported results show BERT-S + QAInfomax achieving F1 scores of 54.5 on ADDSENT and 64.9 on ADDONESENT, improving over vanilla BERT (51.0/63.4) and surpassing prior published results without additional training data. The ablation in Table 2 indicates that both LC and GC contribute, and Table 3 compares three summarization functions for GC. The paper claims state-of-the-art robustness on Adversarial-SQuAD and attributes the gains to mutual information maximization.
Significance. If the central claim holds, the contribution is useful and timely: it introduces a DIM-style mutual information regularizer to an NLP task, shows that it can be combined with strong pretrained models, and reports robustness gains on a challenging adversarial benchmark without extra training data. The paper is reproducible in spirit: the code is publicly linked, the hyperparameters are fully specified, and the base model is a standard PyTorch reimplementation of BERT. The ablation and the comparison with prior systems are also clearly presented. However, the experiments do not currently isolate the causal role of mutual information maximization from the generic effects of adding an auxiliary contrastive loss, and the statistical support for the headline improvement is thin. These gaps need to be addressed before the central claim can be considered established.
major comments (3)
- [Section 3.3, Table 2] The experiments do not separate the effect of the MI objective from the generic effect of an auxiliary contrastive loss with a bilinear critic. Every ablation row that includes LC or GC also adds the parameterized discriminator g from Eq. (6) and its own training signal, so the observed gains may come from extra capacity, changed optimization dynamics, or implicit regularization rather than from maximizing a specific MI quantity. I would like to see a control experiment that trains the same critic and loss scale with shuffled or anti-informative pairings (e.g., permuted labels or deliberately negative MI targets) while keeping the architecture and γ fixed. If such a control yields comparable improvements, the claim that MI maximization drives robustness would not be supported by the current evidence.
- [Section 4.1, Tables 1 and 2] The statistical claims are not supported by the reported information. The text says that the dagger in Table 1 indicates a significant improvement over baselines with p-value < 0.05, but it does not specify the test, the number of runs, the variance, or the effect size. Furthermore, Section 4.1 states that the reported numbers are the best across at least three runs, which systematically favors noisy runs and makes the 3.5-point ADDSENT gap difficult to interpret. Please report mean ± standard deviation over the runs and specify the paired significance procedure (e.g., paired bootstrap or a paired test on per-example scores).
- [Section 2, Eq. (2)] Equation (2) modifies the DIM/JS lower bound by summing both directions of negative sampling with coefficient 1/2, but no derivation or citation is given to show that this symmetric objective still lower-bounds mutual information. Since the method is called QAInfomax and the title claims mutual information maximization, the paper should either provide a short justification for Eq. (2) as a MI bound or explicitly label it as a heuristic contrastive objective that is only inspired by MI estimation. This clarification is needed because the central attribution to MI maximization depends on the validity of this estimator.
minor comments (4)
- [Section 5] The conclusion contains a typo: "Adversirial" should be "Adversarial".
- [Appendix] In the first example, the QAInfomax prediction is spelled "Issac Newton" instead of "Isaac Newton"; the spelling should be corrected. Also, the third example shows that QAInfomax still fails on an adversarial distractor ("Rosetta laws"), which is worth acknowledging in the main text rather than only in an appendix example.
- [Section 4.1] The hyperparameters C, α, β, and γ are fixed to 5, 1, 0.5, and 0.3 without any sensitivity analysis or discussion of how they were selected. A small sensitivity table would help readers understand how stable the reported gains are to these choices.
- [Tables and text] The table text contains rendering artifacts such as "A DDSENT" and "ADDONESENT" with unusual spacing in several places; these should be cleaned up in the final version.
Circularity Check
No circularity: the MI regularizer is an independently motivated objective, and the reported robustness gains are empirical results rather than quantities recovered from the model's own definitions.
full rationale
The paper's derivation chain is not circular in the sense of the review rubric. The MI estimator is imported from prior work (MINE/DIM) as a lower bound, and QAInfomax simply combines this estimator with the span-prediction loss in Eq. (5), L = Lspan + gamma*Linfo. No parameter is fitted to the Adversarial-SQuAD F1 values; the reported scores are measured after training on original SQuAD and are not algebraically implied by the objective. The LC and GC losses maximize mutual-information lower bounds, but they do not encode the benchmark metric or the distractor sentences. The absence of a control experiment that holds the discriminator architecture fixed while changing the MI target is a real attribution limitation, and the fixed hyperparameters without a reported tuning procedure raise model-selection concerns, but neither is a definitional reduction: an ablation row that removes the entire auxiliary loss does not make the improvement equal to the loss by construction. There is no load-bearing self-citation, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The central claim rests on an empirical comparison, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Context window size C =
5
- Weight α for LC =
1
- Weight β for GC =
0.5
- Regularizer strength γ =
0.3
assumptions (4)
- standard math The JS-divergence-based mutual information estimator (Eq. 1) is a valid lower bound on MI.
- ad hoc to paper The modified estimator in Eq. (2), which sums both directions of negative sampling, is a valid objective for MI maximization.
- domain assumption In-batch negative samples approximate draws from the product of marginals.
- domain assumption The span extraction loss Lspan remains the primary task and QAInfomax only regularizes it.
Cite this review
Pith. "Pith review of QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization." pith.science (2026). https://pith.science/paper/ZPR7O2KT
@misc{pith2026190900215,
author = {Pith},
title = {Pith review of: QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZPR7O2KT}},
note = {Machine review of arXiv:1909.00215}
}
read the original abstract
Standard accuracy metrics indicate that modern reading comprehension systems have achieved strong performance in many question answering datasets. However, the extent these systems truly understand language remains unknown, and existing systems are not good at distinguishing distractor sentences, which look related but do not actually answer the question. To address this problem, we propose QAInfomax as a regularizer in reading comprehension systems by maximizing mutual information among passages, a question, and its answer. QAInfomax helps regularize the model to not simply learn the superficial correlation for answering questions. The experiments show that our proposed QAInfomax achieves the state-of-the-art performance on the benchmark Adversarial-SQuAD dataset.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. 2018. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062
arXiv 2018
-
[4]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[5]
Monroe D Donsker and SR Srinivasa Varadhan. 1983. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183--212
work page 1983
-
[6]
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670
arXiv 2018
-
[7]
Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2017. Reinforced mnemonic reader for machine reading comprehension. arXiv preprint arXiv:1705.02798
arXiv 2017
-
[8]
Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. 2017. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. arXiv preprint arXiv:1711.07341
arXiv 2017
Show all 21 references
-
[9]
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328
2017 arXiv
-
[10]
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551
2017 arXiv
-
[11]
Mike Lewis and Angela Fan. 2018. Generative question answering: Learning to answer the whole question
2018
-
[12]
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748
2018 arXiv
-
[13]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch
2017
-
[14]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250
2016 arXiv
-
[15]
Mirco Ravanelli and Yoshua Bengio. 2018. Learning speaker representations with mutual information. arXiv preprint arXiv:1812.00271
2018 arXiv
-
[16]
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. http://arxiv.org/abs/1611.01603 Bidirectional attention flow for machine comprehension . CoRR, abs/1611.01603
2016 arXiv
-
[17]
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047--1055. ACM
2017
-
[18]
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive multiview coding. arXiv preprint arXiv:1906.05849
2019 arXiv
-
[19]
Petar Veli c kovi \'c , William Fedus, William L Hamilton, Pietro Li \`o , Yoshua Bengio, and R Devon Hjelm. 2018. Deep graph infomax. arXiv preprint arXiv:1809.10341
2018 arXiv
-
[20]
Yicheng Wang and Mohit Bansal. 2018. Robust machine comprehension models via adversarial training. arXiv preprint arXiv:1804.06473
2018 arXiv
-
[21]
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.