REVIEW 4 major objections 5 minor 10 references
Exploring Neural Net Augmentation to BERT for Question Answering on SQUAD 2.0
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A context-aware convolutional head on BERT outperforms the paper's CNN, LSTM, and fully connected heads on SQuAD 2.0 question answering.
desk verdict Honest small-scale BERT-output-head comparison, but training-effort confounds keep the contextualized-CNN ranking from being conclusive. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the contextualized CNN, a context-aware convolutional layer whose filters are generated from the input itself. A small set of shared filter-generator weights produces a filter-feature map unique to each question-passage sequence, and that generated map is then convolved over the input to produce the final feature map; the paper follows the cited context-aware filter work in requiring $H$ filter generators for every output channel, where $H$ is the hidden size of the input sequence. This mechanism does the work of adapting the convolution kernels to each example during decoding, letting the model keep relevant context while filtering out distractors, and it is why the paper sees a balance between unanswerable and answerable questions that the basic CNN and LSTM heads lack.
What would settle it
Retrain the fully connected, basic CNN, LSTM, and contextualized CNN heads on SQuAD 2.0 with identical hyperparameter search budgets, identical early-stopping rules, and identical wall-clock time, then compare dev F1 after null-threshold adjustment; if the contextualized CNN no longer leads, the paper's central claim fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a contextualized CNN head equipped with generative filter maps outperforms the other output heads explored when attached to bert-base-uncased and fine-tuned on SQuAD 2.0. After adjusting the null-answer threshold, this model scores 70.09 overall F1 on the development set, with 75.32 F1 on unanswerable questions and 64.85 F1 on answerable ones; the basic CNN reaches only 17.48 overall F1 and the LSTM head 50.10 overall F1, while the fully connected head matches LSTM at 50.07. The paper interprets this as evidence that learned, input-dependent convolution kernels preserve the contextual signal in BERT embeddings better than fixed filters, which lose context, and better than an LSTM's fixed recurrence, which collapses onto the no-answer class; it is explicit that training-parity problems prevent a conclusive architectural verdict and that the result does not beat the BERT code base's run_squad.py baseline.
Load-bearing premise
The load-bearing premise is that each output head was trained fairly enough for the scores to reflect the architecture itself; if the fully connected head had been given the same tuning diligence and training time as the contextualized CNN, the ranking could change.
Editorial extensions
If this is right
- Fine-tuning BERT's weights improved every output head in the comparison, so the augmented decoder should be trained end-to-end rather than on fixed BERT features.
- The contextualized CNN is the only tested head that produces meaningful F1 on both SQuAD 2.0 sub-tasks: 75.32 no-answer and 64.85 has-answer; the basic CNN collapses on unanswerable questions and the LSTM collapses on answerable ones.
- Null-threshold adjustment is a necessary part of the evaluation pipeline: the paper reports substantial F1 gains after re-running predictions with an adjusted threshold for predicting no answer.
- Using extracted BERT features can serve as a fast screening stage before expensive fine-tuning, because the in-group ranking of heads was similar under both training modes.
- Augmenting BERT with a contextualized CNN alone did not beat the BERT code base's run_squad.py baseline of 77.0 F1; restoring training parity is a precondition for any stronger claim.
Reading between the lines
- Beyond the paper: the reported gap between the contextualized CNN (70.09 F1) and the LSTM (50.10 F1) is likely inflated by unequal training effort; a fair A/B test with matched budgets could shrink or reverse it.
- Beyond the paper: input-dependent generative filters behave like a per-example attention mechanism, so the same head may transfer to other span-extraction or reading-comprehension datasets where the model must learn to ignore distractors.
- Beyond the paper: null-threshold adjustment is a post-hoc lever that interacts with head choice; calibrating it per architecture rather than globally might change the relative standings as much as the architectures themselves.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares three output heads (basic CNN, contextualized CNN, and LSTM) placed on top of BERT for SQuAD 2.0 question answering, under both fixed-feature and fine-tuning settings. The authors report that the contextualized CNN is the best head, achieving 70.09 overall F1 on the dev set after adjusting the null-answer threshold, with no-answer F1 of 75.32 and has-answer F1 of 64.85. They also document that fine-tuning improves all models and that they did not beat the BERT repository's run_squad.py baseline. The manuscript candidly acknowledges limited hyperparameter search, premature termination of some training jobs, and loss of parity with the reference implementation.
Significance. The question of how to decode BERT representations for extractive QA is relevant, and the comparison of three architecturally distinct heads is a reasonable experimental design. The paper gives credit for reporting threshold-adjusted metrics and for explicitly noting resource constraints and the loss of parity with run_squad.py. However, the central claim that the contextualized CNN is the best output head is not yet conclusively supported because the training effort across models was not controlled. If the authors re-run the comparison with matched training budgets and rigorous threshold selection, the result could be a useful incremental contribution, but the current manuscript is not a controlled architecture benchmark.
major comments (4)
- [Sections 6 and 8] The comparison in Table 2 is confounded by unequal training effort. Section 6 states that training was stopped at a loss inflection that 'occurred at different step size values for different neural architectures,' and Section 8 admits that 'Due to resource constraints, we were forced to kill some training jobs prematurely' and that the authors 'lost performance parity' with the BERT repository's run_squad.py result. Since the fully connected baseline in Table 2 is effectively an always-abstain predictor (99.95 no-answer EM, 0.05 has-answer EM), the fine-tuning pipeline itself was not at parity. The 70.09 F1 of the contextualized CNN versus 17.48 F1 for the basic CNN may therefore reflect optimization budget rather than architectural merit. Please retrain all models with matched step counts, matched stopping criteria, and comparable hyperparameter search, and report the number of steps and compute used for each model.
- [Section 7.1, Table 1] The reported post-threshold F1 values are obtained by adjusting the null-answer threshold and rerunning predictions on the same dev set used for the final numbers. The null threshold is thus a free parameter fitted to the evaluation set, which optimistically biases the headline 70.09 F1. Use a held-out validation split for threshold selection, or report the threshold-selection procedure and present the unadjusted numbers as the primary result.
- [Sections 5.1-5.3 and 6] The manuscript does not provide the final hyperparameter values for any of the three output heads: kernel sizes, number of filters, number of layers, LSTM hidden size, learning rate, number of warmup steps, or the actual stopping step for each run. It also says the basic CNN hyperparameter search was 'limited' (Section 5.1). Without these details, the comparison cannot be reproduced or audited. Please add a hyperparameter table and specify the full training protocol for each final model.
- [Section 7 and Table 2] Each reported number comes from a single run with no error bars or multiple seeds. Given the variability in training time and hardware mentioned in Section 8, some of the reported gaps between architectures may not be stable. Please provide at least three seeds per configuration and report the mean and standard deviation for the main comparison.
minor comments (5)
- [Section 7.1] The statement that the contextualized CNN 'reaches the performance of original Squad implementation by Google' is overstated relative to the 77.0 F1 cited in Section 2; the reported 70.09 F1 is clearly below that.
- [Tables] The second table is also captioned 'Table1' and should be relabeled Table 2; the in-text references to Tables 1 and 2 should be reconciled.
- [Section 6] The phrase 'inflexion point' should be 'inflection point', and the equations in Figure 5 are garbled and should be typeset properly.
- [Section 5.3] 'Shen el al' should be 'Shen et al.', and Reference 9 misspells 'Christopher'.
- [Section 9] The sentence 'in-group rank between fine tuning and extracted features approaches were similar' is unclear; please specify exactly which ranking is being compared and what 'similar' means quantitatively.
Circularity Check
Empirical benchmark against external SQuAD 2.0; no derivation reduces to its own inputs, so no circularity.
full rationale
This paper is an empirical architecture comparison. The central claim — that the contextualized CNN output head achieves the best F1 among the implemented heads on SQuAD 2.0 — is a measured result on an external benchmark, not a prediction derived from fitted parameters or from a self-citation chain. The models are trained and evaluated on the standard SQuAD 2.0 dev set, and the reported scores are direct evaluation metrics (Tables 1 and 2). No equation or theorem in the paper defines one quantity in terms of another in a way that would make the outcome true by construction. The contextualized CNN architecture is taken from Shen et al. [10], an external reference, and the authors explicitly note their divergence from it; this is an implementation choice, not a load-bearing self-citation. The null-threshold adjustment is a standard evaluation practice and constitutes fitting a threshold to the dev set, which is a validity concern about dev-set reuse but not circularity of a derivation. The paper's own limitations in Section 8 (prematurely killed training jobs and lost performance parity with run_squad.py) weaken the internal validity of the architecture ranking, but a training-effort confound is a quality-of-evidence problem, not a circularity problem: the ranking is not equivalent to its inputs by construction. No self-citation, uniqueness theorem, ansatz smuggling, or renaming of a known result appears. The paper is self-contained against external benchmarks, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- null answer threshold =
not reported
- training duration / early stopping step =
varies per model
- CNN and LSTM hyperparameters =
not fully reported
assumptions (2)
- domain assumption BERT base-uncased embeddings are a suitable representation for the SQuAD 2.0 QA task.
- domain assumption The 10% held-out split of the SQuAD training set is representative for early stopping and model selection.
Cite this review
Pith. "Pith review of Exploring Neural Net Augmentation to BERT for Question Answering on SQUAD 2.0." pith.science (2026). https://pith.science/paper/VZN6RC3E
@misc{pith2026190801767,
author = {Pith},
title = {Pith review of: Exploring Neural Net Augmentation to BERT for Question Answering on SQUAD 2.0},
year = {2026},
howpublished = {\url{https://pith.science/paper/VZN6RC3E}},
note = {Machine review of arXiv:1908.01767}
}
read the original abstract
Enhancing machine capabilities to answer questions has been a topic of considerable focus in recent years of NLP research. Language models like Embeddings from Language Models (ELMo)[1] and Bidirectional Encoder Representations from Transformers (BERT) [2] have been very successful in developing general purpose language models that can be optimized for a large number of downstream language tasks. In this work, we focused on augmenting the pre-trained BERT language model with different output neural net architectures and compared their performance on question answering task posed by the Stanford Question Answering Dataset 2.0 (SQUAD 2.0) [3]. Additionally, we also fine-tuned the pre-trained BERT model parameters to demonstrate its effectiveness in adapting to specialized language tasks. Our best output network, is the contextualized CNN that performs on both the unanswerable and answerable question answering tasks with F1 scores of 75.32 and 64.85 respectively.
Reference graph
Works this paper leans on
-
[2]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. arXiv:1810.04805
-
[3]
Unanswerable Questions for SQuAD
Know What You Don't Know: “Unanswerable Questions for SQuAD”. arXiv:1806.03822
-
[7]
Comparative Study of CNN and RNN for Natural Language Processing
“Comparative Study of CNN and RNN for Natural Language Processing”. arXiv:1702.01923
-
[8]
Tf.nn.rnn_cell.LSTMCell : TensorFlow Core r1.14 : TensorFlow
Google, Google. “Tf.nn.rnn_cell.LSTMCell : TensorFlow Core r1.14 : TensorFlow.” TensorFlow , 2019, www.tensorflow.org/api_docs/python/tf/nn/rnn_cell/LSTMCell
work page 2019
-
[9]
Olah, Cristohpher. “Understanding LSTM Networks.” Understanding LSTM Networks -- Colah's Blog , 27 Aug. 2017, colah.github.io/posts/2015-08-Understanding-LSTMs
work page 2017
-
[10]
Learning Context-Sensitive Convolutional Filters for Text Processing
“Learning Context-Aware Convolutional Filters for Text Processing”. arXiv:1709.08294 6
-
[2014]
Convolutional Neural Networks for Sentence Classification
“Convolutional Neural Networks for Sentence Classification”. arXiv:1408.5882
-
[2015]
“A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification”. arXiv:1510.03820
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.