{"id":"f6e22e9e-c06c-40c1-94c5-ffd4579f180d","arxiv_id":"1908.01767","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A contextualized CNN readout on BERT outperforms basic CNN and LSTM readouts on SQuAD 2.0, yet still underperforms vanilla BERT.","lead":"Researchers tested three neural network readout layers on top of the BERT language model for the SQuAD 2.0 question answering benchmark. The best readout, a contextualized CNN, beat the other two tested heads but still scored below the standard BERT baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-effort confound: Section 8 concedes premature job kills and lost parity with run_squad.py, so the contextualized CNN's 70.09 F1 lead may reflect optimization budget, not architecture. Equal-budget retraining is needed.","rationale":"The reader's weakest assumption is that the observed ranking reflects architectural merit rather than unequal training effort, and this is exactly the most load-bearing concern for the paper's central claim. The paper itself provides direct evidence for the concern: Section 8 states that some training jobs were killed prematurely and that parity with the BERT repository baseline was lost, and Section 6 describes per-architecture early stopping at different step counts. The fully connected baseline's collapse to a degenerate no-answer predictor (Table 2: 99.95 no-answer EM, 0.05 has-answer EM) underscores that the training pipeline was not validated against a known result. The score gaps are large, so the contextualized CNN may well be genuinely better, but without equal training effort the comparison does not establish that. A controlled equal-budget retraining is the direct test: if the gap persists, the ranking is credible; if not, the conclusion is an artifact. The reader's CONDITIONAL verdict remains appropriate, so no verdict change is needed. The attack is not an external-consensus disagreement; it is an internal control problem acknowledged in the manuscript, and it can be resolved by a concrete experiment.","tokens_in":7859,"tokens_out":9632,"duration_ms":99258,"concrete_test":"Run a controlled comparison in which the basic CNN, LSTM, and contextualized CNN heads are each trained from the same BERT checkpoint with the same number of optimizer steps, same batch size, same learning-rate schedule, and the same early-stopping rule, repeated over at least 3 seeds, and select each null threshold on a validation split disjoint from the reported dev set. If the LSTM and basic CNN still trail the contextualized CNN by more than 5 F1 points, the ranking is robust; if either closes the gap, the original conclusion is confounded by unequal training effort.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the contextualized CNN is the best BERT output head for SQuAD 2.0, based on Table 2 (70.09 F1 vs 17.48 for basic CNN and 50.10 for LSTM). The load-bearing assumption is that the comparison reflects architecture rather than training effort. That assumption is explicitly insecure. Section 6 says training was stopped when eval loss hit an inflection point, which occurred at different step sizes for different architectures; Section 8 admits 'Due to resource constraints, we were forced to kill some training jobs prematurely' and that the authors 'lost performance parity' between the BERT repository's run_squad.py result and their own fine-tuned fully connected network. The fully connected baseline in Table 2 scores 50.07 F1 after threshold adjustment, with 99.95 no-answer EM and 0.05 has-answer EM, a degenerate always-abstain predictor. Thus the optimization pipeline was not held constant, and the basic CNN was given only a 'limited' hyperparameter search per Section 5.1. Under equal step counts, schedules, and early stopping, the ranking could change materially. Section 9 itself calls for an additional attempt that more closely maintains training parity in order to conclusively assess the benefit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three output heads (basic CNN, contextualized CNN, and LSTM) placed on top of BERT for SQuAD 2.0 question answering, under both fixed-feature and fine-tuning settings. The authors report that the contextualized CNN is the best head, achieving 70.09 overall F1 on the dev set after adjusting the null-answer threshold, with no-answer F1 of 75.32 and has-answer F1 of 64.85. They also document that fine-tuning improves all models and that they did not beat the BERT repository's run_squad.py baseline. The manuscript candidly acknowledges limited hyperparameter search, premature termination of some training jobs, and loss of parity with the reference implementation.","tokens_in":8058,"tokens_out":4478,"duration_ms":43207,"significance":"The question of how to decode BERT representations for extractive QA is relevant, and the comparison of three architecturally distinct heads is a reasonable experimental design. The paper gives credit for reporting threshold-adjusted metrics and for explicitly noting resource constraints and the loss of parity with run_squad.py. However, the central claim that the contextualized CNN is the best output head is not yet conclusively supported because the training effort across models was not controlled. If the authors re-run the comparison with matched training budgets and rigorous threshold selection, the result could be a useful incremental contribution, but the current manuscript is not a controlled architecture benchmark.","major_comments":[{"comment":"The comparison in Table 2 is confounded by unequal training effort. Section 6 states that training was stopped at a loss inflection that 'occurred at different step size values for different neural architectures,' and Section 8 admits that 'Due to resource constraints, we were forced to kill some training jobs prematurely' and that the authors 'lost performance parity' with the BERT repository's run_squad.py result. Since the fully connected baseline in Table 2 is effectively an always-abstain predictor (99.95 no-answer EM, 0.05 has-answer EM), the fine-tuning pipeline itself was not at parity. The 70.09 F1 of the contextualized CNN versus 17.48 F1 for the basic CNN may therefore reflect optimization budget rather than architectural merit. Please retrain all models with matched step counts, matched stopping criteria, and comparable hyperparameter search, and report the number of steps and compute used for each model.","section":"Sections 6 and 8"},{"comment":"The reported post-threshold F1 values are obtained by adjusting the null-answer threshold and rerunning predictions on the same dev set used for the final numbers. The null threshold is thus a free parameter fitted to the evaluation set, which optimistically biases the headline 70.09 F1. Use a held-out validation split for threshold selection, or report the threshold-selection procedure and present the unadjusted numbers as the primary result.","section":"Section 7.1, Table 1"},{"comment":"The manuscript does not provide the final hyperparameter values for any of the three output heads: kernel sizes, number of filters, number of layers, LSTM hidden size, learning rate, number of warmup steps, or the actual stopping step for each run. It also says the basic CNN hyperparameter search was 'limited' (Section 5.1). Without these details, the comparison cannot be reproduced or audited. Please add a hyperparameter table and specify the full training protocol for each final model.","section":"Sections 5.1-5.3 and 6"},{"comment":"Each reported number comes from a single run with no error bars or multiple seeds. Given the variability in training time and hardware mentioned in Section 8, some of the reported gaps between architectures may not be stable. Please provide at least three seeds per configuration and report the mean and standard deviation for the main comparison.","section":"Section 7 and Table 2"}],"minor_comments":[{"comment":"The statement that the contextualized CNN 'reaches the performance of original Squad implementation by Google' is overstated relative to the 77.0 F1 cited in Section 2; the reported 70.09 F1 is clearly below that.","section":"Section 7.1"},{"comment":"The second table is also captioned 'Table1' and should be relabeled Table 2; the in-text references to Tables 1 and 2 should be reconciled.","section":"Tables"},{"comment":"The phrase 'inflexion point' should be 'inflection point', and the equations in Figure 5 are garbled and should be typeset properly.","section":"Section 6"},{"comment":"'Shen el al' should be 'Shen et al.', and Reference 9 misspells 'Christopher'.","section":"Section 5.3"},{"comment":"The sentence 'in-group rank between fine tuning and extracted features approaches were similar' is unclear; please specify exactly which ranking is being compared and what 'similar' means quantitatively.","section":"Section 9"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style empirical report. The authors' candor about premature job kills and loss of parity is commendable, but the central claim needs a controlled retraining with equal budgets and threshold selection on a held-out set. I would not reject the paper outright because the confound can, in principle, be fixed within the scope of the manuscript, but the revision must be substantial rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick take on arXiv:1908.01767. It's a small empirical study comparing three output heads on top of BERT for SQuAD 2.0: basic CNN, LSTM, and contextualized CNN (from Shen et al.). The new result is the within-paper comparison: the contextualized CNN gets 70.09 F1 overall, with 75.32 no-answer F1 and 64.85 has-answer, beating the other two heads by a lot. That much is credible — those gaps are large.\n\nWhat the paper does well: it's honest. Section 8 openly says some training jobs were killed due to compute limits and that their fine-tuned fully connected network lost parity with the BERT repo's run_squad.py result. Section 9 says the conclusion is only about the explored architectures and calls for an additional attempt with training parity. That candor is welcome, and the within-paper ranking is probably real at the level of 'this setup, these hyperparameters.' The contextualized CNN is the only one that learns both no-answer and has-answer behavior; the LSTM degenerates to always predicting no-answer (99.95 EM on no-answer, 0.05 on has-answer), and the basic CNN underfits. The failure modes are clearly described.\n\nWhere it gets soft: the central comparison is confounded by training effort. Training was stopped at loss inflection, which happened at different step counts for different architectures; batch sizes differed (4 with fine-tuning vs 32 without); and some jobs were killed. The fully connected baseline with fine-tuning scores 50.07 F1, which is an always-abstain predictor — that's not a working baseline, it's a degenerate solution. So the claim 'contextualized CNN is best' is really 'contextualized CNN is best among models that didn't collapse under these training conditions.' Since the paper's own numbers show the LSTM collapsed, the ranking might change under equal training budgets, and the paper itself concedes that.\n\nThere are no error bars or multiple seeds, and the null threshold is tuned on the dev set, which inflates the reported numbers. These are standard small-study problems but they matter because the whole contribution is a set of point estimates.\n\nWho is this for? Someone doing a quick survey of possible BERT decoders for extractive QA might read it as a data point, not a definitive result. It's not a paper I'd cite for the architecture comparison without also citing the confound. It deserves a referee if the venue wants an honest negative/conditional result — a good reviewer would ask for parity retraining, error bars, and a non-degenerate baseline. That's the right path. My verdict: not a desk reject in a workshop context, but the claim needs to be narrowed until the training parity issue is addressed. If you're deciding whether to spend time on it, it's a 10-minute read that tells you what not to do when comparing decoders.","headline":"Honest small-scale BERT-output-head comparison, but training-effort confounds keep the contextualized-CNN ranking from being conclusive.","tokens_in":8629,"tokens_out":2626,"would_cite":false,"duration_ms":25945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A context-aware convolutional head on BERT outperforms the paper's CNN, LSTM, and fully connected heads on SQuAD 2.0 question answering.","keywords":["BERT","SQuAD 2.0","question answering","contextualized CNN","generative filter maps","LSTM","fine-tuning","unanswerable questions"],"falsifier":"Retrain the fully connected, basic CNN, LSTM, and contextualized CNN heads on SQuAD 2.0 with identical hyperparameter search budgets, identical early-stopping rules, and identical wall-clock time, then compare dev F1 after null-threshold adjustment; if the contextualized CNN no longer leads, the paper's central claim fails.","tokens_in":7613,"feed_emoji":"🧠","tokens_out":9474,"duration_ms":87544,"temperature":0.7,"pith_summary":"The paper tries to establish that the way a pre-trained BERT model is decoded matters as much as the language model itself for SQuAD 2.0 question answering. Among the output heads it compares, a contextualized CNN with input-dependent generative filters is the only one that handles both unanswerable and answerable questions well, reaching an overall F1 of 70.09 on the development set after fine-tuning and null-threshold adjustment. The paper also claims fine-tuning BERT's weights consistently beats using BERT as a fixed feature extractor, and that the contextualized CNN's balance—no-answer F1 of 75.32 and has-answer F1 of 64.85—is what the SQuAD 2.0 task demands. A fair reader would care because the result points to a cheap, trainable output-layer improvement over BERT's standard single-layer output head, with the caveat that the paper did not beat the BERT code base's own run_squad.py score of 77.0 F1.","feed_headline":"Context-aware CNN beats plain CNN and LSTM on SQuAD 2.0","feed_subtitle":"Its generated filters balance unanswerable and answerable questions, the core SQuAD 2.0 challenge.","key_machinery":"The central object is the contextualized CNN, a context-aware convolutional layer whose filters are generated from the input itself. A small set of shared filter-generator weights produces a filter-feature map unique to each question-passage sequence, and that generated map is then convolved over the input to produce the final feature map; the paper follows the cited context-aware filter work in requiring $H$ filter generators for every output channel, where $H$ is the hidden size of the input sequence. This mechanism does the work of adapting the convolution kernels to each example during decoding, letting the model keep relevant context while filtering out distractors, and it is why the paper sees a balance between unanswerable and answerable questions that the basic CNN and LSTM heads lack.","core_discovery":"On the paper's own terms, the central discovery is that a contextualized CNN head equipped with generative filter maps outperforms the other output heads explored when attached to bert-base-uncased and fine-tuned on SQuAD 2.0. After adjusting the null-answer threshold, this model scores 70.09 overall F1 on the development set, with 75.32 F1 on unanswerable questions and 64.85 F1 on answerable ones; the basic CNN reaches only 17.48 overall F1 and the LSTM head 50.10 overall F1, while the fully connected head matches LSTM at 50.07. The paper interprets this as evidence that learned, input-dependent convolution kernels preserve the contextual signal in BERT embeddings better than fixed filters, which lose context, and better than an LSTM's fixed recurrence, which collapses onto the no-answer class; it is explicit that training-parity problems prevent a conclusive architectural verdict and that the result does not beat the BERT code base's run_squad.py baseline.","pith_inferences":["Beyond the paper: the reported gap between the contextualized CNN (70.09 F1) and the LSTM (50.10 F1) is likely inflated by unequal training effort; a fair A/B test with matched budgets could shrink or reverse it.","Beyond the paper: input-dependent generative filters behave like a per-example attention mechanism, so the same head may transfer to other span-extraction or reading-comprehension datasets where the model must learn to ignore distractors.","Beyond the paper: null-threshold adjustment is a post-hoc lever that interacts with head choice; calibrating it per architecture rather than globally might change the relative standings as much as the architectures themselves."],"forward_implications":["Fine-tuning BERT's weights improved every output head in the comparison, so the augmented decoder should be trained end-to-end rather than on fixed BERT features.","The contextualized CNN is the only tested head that produces meaningful F1 on both SQuAD 2.0 sub-tasks: 75.32 no-answer and 64.85 has-answer; the basic CNN collapses on unanswerable questions and the LSTM collapses on answerable ones.","Null-threshold adjustment is a necessary part of the evaluation pipeline: the paper reports substantial F1 gains after re-running predictions with an adjusted threshold for predicting no answer.","Using extracted BERT features can serve as a fast screening stage before expensive fine-tuning, because the in-group ranking of heads was similar under both training modes.","Augmenting BERT with a contextualized CNN alone did not beat the BERT code base's run_squad.py baseline of 77.0 F1; restoring training parity is a precondition for any stronger claim."],"supporting_citations":[{"why":"Supplies the pre-trained BERT model, the fine-tuning procedure, and the run_squad.py baseline all experiments build on.","marker":"[2]"},{"why":"Defines SQuAD 2.0 with its unanswerable-question challenge; all reported dev-set scores come from this benchmark.","marker":"[3]"},{"why":"Supplies the context-aware filter-generator architecture that the paper adapts into its best-performing head.","marker":"[10]"},{"why":"Guides the basic CNN baseline's hyperparameter choices through a sensitivity analysis of convolutional text classifiers.","marker":"[5]"},{"why":"Motivates the multi-channel 1-D convolution baseline as a standard convolutional text classifier.","marker":"[6]"},{"why":"Provides the comparative motivation for including an LSTM decoder against convolutional decoders.","marker":"[7]"},{"why":"Supplies the LSTM cell implementation used to build the recurrent head.","marker":"[8]"}],"fun_headline_variants":["Context-aware CNN head beats LSTM and plain CNN on SQuAD 2.0","Learned convolution filters in CNN head best fixed ones for QA","Generative filter maps give CNN head edge on SQuAD 2.0","Contextualized CNN gains over LSTM and dense heads on SQuAD 2.0","Adaptive CNN output head boosts SQuAD 2.0 F1 to 70.09"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each output head was trained fairly enough for the scores to reflect the architecture itself; if the fully connected head had been given the same tuning diligence and training time as the contextualized CNN, the ranking could change.","fun_headline_variants_meta":{"raw":{"variants":["Context-aware CNN head beats LSTM and plain CNN on SQuAD 2.0","Learned convolution filters in CNN head best fixed ones for QA","Generative filter maps give CNN head edge on SQuAD 2.0","Contextualized CNN gains over LSTM and dense heads on SQuAD 2.0","Adaptive CNN output head boosts SQuAD 2.0 F1 to 70.09"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1551,"prompt_tokens":932,"completion_tokens":619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":508}},"tokens_in":548,"tokens_out":619,"duration_ms":6516,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:14:26.753038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the fully connected, basic CNN, LSTM, and contextualized CNN heads on SQuAD 2.0 with identical hyperparameter search budgets, identical early-stopping rules, and identical wall-clock time, then compare dev F1 after null-threshold adjustment; if the contextualized CNN no longer leads, the paper's central claim fails.","supporting_citations":[{"cited_title":"Learning Context-Sensitive Convolutional Filters for Text Processing","cited_arxiv_id":"1709.08294","evidence_quote":"Supplies the context-aware filter-generator architecture that the paper adapts into its best-performing head."},{"cited_title":"Tf.nn.rnn_cell.LSTMCell : TensorFlow Core r1.14 : TensorFlow","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM cell implementation used to build the recurrent head."}],"review_version":1}