REVIEW 3 major objections 5 minor 39 references
Attentive History Selection for Conversational Question Answering
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that learned, position-aware attention over conversation history—not a fixed recency heuristic—is what improves conversational question answering, reaching $65.7$ F1 on QuAC.
desk verdict Solid incremental ConvQA paper: the history attention idea is real and well supported, but the MTL claim in the abstract is contradicted by the paper's own ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the history attention module sitting on top of a BERT encoder, together with the positional history answer embedding (PosHAE) that feeds it. PosHAE assigns each passage token a learned embedding indexed by the relative distance $(k-i)$ between the current question and a history turn whose answer covers that token, so BERT sees which tokens were answers and how far back they occurred. Each history turn is encoded in its own BERT sequence, and the attention module computes weights over those sequences; the fine-grained variant computes a separate weight for every token position, then aggregates token representations. These weights carry the argument: they make history selection soft, explainable, and learnable, and they are trained end-to-end with the answer span and dialog act losses.
What would settle it
Train HAM and HAM 'w/o Dialog Act Prediction' on QuAC under the same hyperparameters and evaluate on the hidden test set with a paired significance test: if the version without the dialog act loss matches or exceeds the full model on F1, the multi-task benefit claim for answer span prediction fails; the validation numbers in Table 6 already point in that direction.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that a BERT-based conversational QA model improves by (1) embedding each history answer's position in the dialog, so the encoder sees how distant a history turn is from the current question, and (2) explicitly learning soft attention weights over all history turns, computed at the token level, and using the weighted sum as the representation for answer span and dialog act prediction. The complete model, HAM with BERT-base, obtains $65.7$ F1 on QuAC validation, significantly above the BERT + PosHAE baseline at $64.7$ ($p<0.05$); with BERT-Large it reaches $66.7$ F1 validation and $65.4$ test. Ablations show that replacing the fine-grained attention with sequence-level attention lowers F1 to $64.9$, removing attention entirely lowers it to $61.1$, and replacing PosHAE with the earlier HAE lowers it to $64.2$. The paper also claims that jointly predicting dialog acts and answer spans lets one model handle both tasks, and that answer span supervision improves dialog act prediction, while the reverse direction shows no significant answer-side gain in the reported numbers.
Load-bearing premise
The assumption that predicting dialog acts alongside answers improves answer quality is load-bearing for the paper's multi-task claim, yet the paper's own ablation shows that removing dialog act prediction raises answer F1 from $65.7$ to $65.9$.
Editorial extensions
If this is right
- Conversational QA systems can replace the 'always use the immediately preceding turns' heuristic with learned attention, including attending to remote turns in topic-return cases.
- Encoding history position is cheap and effective: the PosHAE variant outperforms the plain HAE by $0.8$ F1 validation ($64.7$ vs $63.9$), and ablating PosHAE from HAM drops F1 by $1.5$.
- Token-level attention weights give a built-in explanation channel: visualized weights align with drill-down, topic-shift, and topic-return patterns, which could support debugging and user-facing explanations.
- A single multi-task model can serve both answer span and dialog act prediction, with answer span supervision contributing to dialog act accuracy (removing it drops Yes/No from $88.3$ to $86.2$ and Follow up from $62.3$ to $59.7$).
- The gains compose with model scale: BERT-Large raises validation F1 to $66.7$ and test F1 to $65.4$, so the history attention design does not depend on a small encoder.
Reading between the lines
- Because the encoder processes each history turn separately before attention, the same architecture could be lifted to other conversational tasks where history selection matters, such as response ranking or conversational recommendation, replacing fixed recency features with learned soft selection.
- The ablation numbers suggest an asymmetry: answer span supervision helps dialog act prediction, but dialog act loss does not help answer spans (removing it slightly raises F1 to $65.9$); a practical follow-up would drop the dialog act loss when answer quality is the only goal.
- The attention weights could serve as pseudo-labels for training a cheaper history selector, distilling which turns the full BERT model would rely on; this distillation direction is a testable extension the paper does not pursue.
- The gains may concentrate on dialogs with topic shift or topic return; one could test this by slicing QuAC validation by dialog-behavior labels and comparing HAM against the recency heuristic baseline on those subsets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses conversational question answering (ConvQA) on the QuAC benchmark and proposes three main components: a history attention mechanism (HAM) that performs soft selection over conversation history turns, a positional history answer embedding (PosHAE) that augments BERT embeddings with relative position information of history answers, and a multi-task learning (MTL) setup that jointly trains answer span prediction and dialog act prediction. Each history turn is encoded separately with BERT and PosHAE, then token-level attention aggregates the turn representations before span prediction. The full HAM model with BERT-base achieves 65.7 F1 on QuAC validation and 64.4 on the official test set, with a statistically significant (p<0.05) validation F1 improvement over the strongest baseline, BERT+PosHAE (64.7). Ablations show that removing the fine-grained attention, removing history attention, and replacing PosHAE with HAE all lower F1, and attention visualizations illustrate drill-down, topic-shift, and topic-return behaviors.
Significance. If the design conclusions hold, the paper makes a useful empirical contribution to conversational QA: it demonstrates that adding position information to history answer embeddings and learning to re-weight history turns can improve BERT-based ConvQA on a public leaderboard. The paper's strengths include an open-source implementation, comparison against published QuAC baselines, a paired significance test on the main validation F1 metric, and a fairly complete set of ablations for the proposed modules. The main caveats are that one of the three claimed contributions, multi-task learning, is not supported by the paper's own ablation for the primary answer-span task, and the history-attention comparison is not fully isolated from the change in input architecture. These issues are fixable but currently make the abstract and contribution statements stronger than the evidence.
major comments (3)
- [Abstract and §1 contribution (3); §4.4 Table 6, observation (4)] The abstract and contribution (3) claim that multi-task learning improves the performance of ConvQA, but the paper's own ablation contradicts this for the primary task: HAM achieves 65.7 F1, while HAM w/o Dialog Act Prediction achieves 65.9 F1, and observation (4) in Section 4.4 admits that the change is a 'slight and insignificant increase.' The demonstrated benefit of MTL is confined to the secondary dialog act task, where removing answer-span prediction drops Yes/No accuracy from 88.3 to 86.2 and Follow-up accuracy from 62.3 to 59.7. The abstract and contribution (3) should be revised to state that MTL improves dialog act prediction and provides a uniform architecture, or the claim that it improves ConvQA answer-span performance should be removed.
- [§4.4 Table 6; §4.2.1] The evidence that learned history attention outperforms the heuristic selection used by prior work is indirect. HAM differs from BERT+PosHAE both in the attention mechanism and in the input architecture: HAM encodes each history turn separately and aggregates the resulting representations, while BERT+PosHAE packs the selected history turns into a single input sequence and relies on the encoder. The 'HAM w/o History Attention' ablation uses equal weights across separately encoded turns, which is not the 'immediate previous turns' heuristic used by the baselines, as the authors themselves acknowledge in Section 4.4. To support the specific claim that learned 'soft selection' is better than heuristic selection, the paper should add an ablation in which the HAM architecture is run with fixed recency-based weights or hard selection of the last k turns, rather than only with uniform weights.
- [§4.3 Table 4, observation (3)] The paper claims 'substantially higher performance on dialog act prediction compared to baseline methods,' but no significance test or error bar is reported for the dialog act accuracies, and only one baseline (BiDAF++ w/ 2-Context) has dialog act numbers in Table 4. The observed improvements (Yes/No 88.3 vs 86.6; Follow-up 62.3 vs 61.6) may be reliable, but the claim as stated is stronger than the reported evidence. The statement should be qualified or supplemented with significance testing on dialog act predictions. The same applies to the HEQ-Q and HEQ-D differences, which are reported without significance testing.
minor comments (5)
- [Table 1] In the last row of Table 1, the utterance ID is listed as 'A6' but should be 'A7' for consistency with the dialog turn numbering.
- [§4.3 vs Table 6 caption] Section 4.3 observation (3) states 'We can only do significance test on F1,' while the caption of Table 6 states 'We can only do significance test on F1 and dialog act accuracy.' Please reconcile these statements.
- [Table 4] BERT+PosHAE has no test-set F1 in Table 4, so the test-set comparison for the strongest baseline is missing; the significance claim should be clearly limited to the validation set, and the test-set gains over FlowQA (64.4 vs 64.1) should be discussed with appropriate caution.
- [§3.7.2 and §4.4] The full HAM model uses λ=0.1 and µ=0.8 in Equation (7), while the MTL ablation 'HAM w/o Answer Span Prediction' tunes λ in (0.2, 0.4, 0.6, 0.8) and adopts λ=0.2; please clarify whether the different λ values affect the comparison.
- [§3.4 and §1] There are minor language issues, e.g., 'pass it though the history attention network' in Section 3.4 and 'We show that these benefits have contributions to the model performance' in Section 1; these should be corrected in revision.
Circularity Check
No significant circularity: the core HAM/PosHAE results are empirical and evaluated against external baselines; the MTL claim is internally overstated but does not reduce to its own inputs.
full rationale
The paper's central derivation is an end-to-end learning pipeline, not an analytic derivation. Equations (1)-(7) define a standard attention-weighted aggregation and loss combination; none of them defines the target metric (F1/HEQ) in terms of a fitted parameter that is then reported as a prediction. PosHAE extends the authors' earlier HAE [24] by adding a learned relative-position embedding, and its contribution is tested by ablating it (Table 6, 'w/o PosHAE'), while HAM's attention contribution is tested by ablating to equal weights ('w/o History Attention') and by comparing against external baselines (BiDAF++, FlowQA, BERT+HAE). The self-citations to [24] supply a baseline model and a history-representation idea, but they are not used as an unexamined uniqueness theorem or as a substitute for evaluation. The one notable weakness is not circularity: the abstract claims MTL 'improve[s] the performance of ConvQA,' but the paper's own Table 6 shows HAM w/o Dialog Act Prediction at 65.9 F1 versus 65.7 for HAM, and observation (4) acknowledges the increase is 'slight and insignificant.' That is an overclaim or correctness issue for a reviewer to flag, not a derivation that reduces to its inputs. Since the main results are measured on held-out QuAC data against independent baselines, no load-bearing circular step is present.
Assumptions & free parameters
free parameters (4)
- max_history_turns =
6
- lambda (dialog act loss weight) =
0.1 (HAM), 0.2 (HAM w/o Answer Span Prediction)
- mu (answer span loss weight) =
0.8 (HAM)
- pooling method =
max pooling (HAM), average pooling (HAM w/o Answer Span Prediction)
assumptions (4)
- domain assumption BERT provides suitable contextualized token and sequence representations for the ConvQA task.
- domain assumption History questions contribute little and can be ignored; only history answers matter.
- domain assumption The Student's paired t-test is a valid significance test for the F1 differences on QuAC.
- domain assumption QuAC is a representative benchmark for ConvQA and the leaderboard results are reliable.
Cite this review
Pith. "Pith review of Attentive History Selection for Conversational Question Answering." pith.science (2026). https://pith.science/paper/2RWILHIC
@misc{pith2026190809456,
author = {Pith},
title = {Pith review of: Attentive History Selection for Conversational Question Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/2RWILHIC}},
note = {Machine review of arXiv:1908.09456}
}
read the original abstract
Conversational question answering (ConvQA) is a simplified but concrete setting of conversational search. One of its major challenges is to leverage the conversation history to understand and answer the current question. In this work, we propose a novel solution for ConvQA that involves three aspects. First, we propose a positional history answer embedding method to encode conversation history with position information using BERT in a natural way. BERT is a powerful technique for text representation. Second, we design a history attention mechanism (HAM) to conduct a "soft selection" for conversation histories. This method attends to history turns with different weights based on how helpful they are on answering the current question. Third, in addition to handling conversation history, we take advantage of multi-task learning (MTL) to do answer prediction along with another essential conversation task (dialog act prediction) using a uniform model architecture. MTL is able to learn more expressive and generic representations to improve the performance of ConvQA. We demonstrate the effectiveness of our model with extensive experimental evaluations on QuAC, a large-scale ConvQA dataset. We show that position information plays an important role in conversation history modeling. We also visualize the history attention and provide new insights into conversation history understanding.
Figures
Reference graph
Works this paper leans on
-
[1]
N. J. Belkin, C. Cool, A. S., and U. Thiel. Cases , Scripts , and Information-Seeking Strategies : On the Design of Interactive Information Retrieval Systems. 1994
work page 1994
-
[2]
E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. S. Zettlemoyer. QuAC: Question Answering in Context. In EMNLP, 2018
work page 2018
-
[3]
A. Chuklin, A. Severyn, J. R. Trippas, E. Alfonseca, H. Silén, and D. Spina. Prosody Modifications for Question-Answering in Voice-Only Settings. CoRR, 2018
work page 2018
-
[4]
C. Clark and M. Gardner. Simple and Effective Multi-Paragraph Reading Com- prehension. In ACL, 2018
work page 2018
-
[5]
W. B. Croft and R. H. Thompson. I3R: A new approach to the design of document retrieval systems. JASIS, 38:389–404, 1987
work page 1987
-
[6]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, 2018
work page 2018
-
[7]
J. Gao, M. Galley, and L. Li. Neural Approaches to Conversational AI. In SIGIR, 2018
work page 2018
-
[8]
J. Guo, Y. Fan, L. Pang, L. Yang, Q. Ai, H. Zamani, C. Wu, W. B. Croft, and X. Cheng. A Deep Look into Neural Ranking Models for Information Retrieval. CoRR, abs/1903.06902, 2019
arXiv 1903
Show all 39 references
-
[9]
M. Hu, Y. Peng, Z. Huang, X. Qiu, F. Wei, and M. Zhou. Reinforced Mnemonic Reader for Machine Reading Comprehension. In IJCAI, 2018
2018
-
[10]
Huang, C
H.-Y. Huang, C. Zhu, Y. Shen, and W. Chen. FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension. CoRR, abs/1711.07341, 2017
2017 arXiv
-
[11]
Huang, E
H.-Y. Huang, E. Choi, and W. Yih. FlowQA: Grasping Flow in History for Con- versational Machine Comprehension. CoRR, 2018
2018
-
[12]
M. S. Joshi, E. Choi, D. S. Weld, and L. S. Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In ACL, 2017
2017
-
[13]
Kotov and C
A. Kotov and C. Zhai. Towards natural question guided search. In WWW, 2010
2010
-
[14]
Kwiatkowski, J
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Ep- stein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural Questions: a Benchmark for Question Answering Res...
2019
-
[15]
F.-L. Li, M. Qiu, H. Chen, X. Wang, X. Gao, J. Huang, J. Ren, Z. Zhao, W. Zhao, L. Wang, G. Jin, and W. Chu. AliMe Assist : An Intelligent Assistant for Creating an Innovative E-commerce Experience. In CIKM, 2017
2017
-
[16]
X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y.-Y. Wang. Representation Learn- ing Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval. In HLT-NAACL, 2015
2015
-
[17]
X. Liu, P. He, W. Chen, and J. Gao. Multi-Task Deep Neural Networks for Natural Language Understanding. CoRR, abs/1901.11504, 2019
1901 arXiv
-
[18]
Nguyen, M
T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoRR, abs/1611.09268, 2016
2016 arXiv
-
[19]
R. N. Oddy. Information Retrieval through Man-Machine Dialogue. 1977
1977
-
[20]
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018
2018
-
[21]
C. Qu, L. Yang, W. B. Croft, J. R. Trippas, Y. Zhang, and M. Qiu. Analyzing and Characterizing User Intent in Information-seeking Conversations. In SIGIR, 2018
2018
-
[22]
C. Qu, L. Yang, W. B. Croft, F. Scholer, and Y. Zhang. Answer Interaction in Non-factoid Question Answering Systems. In CHIIR, 2019
2019
-
[23]
C. Qu, L. Yang, W. B. Croft, Y. Zhang, J. R. Trippas, and M. Qiu. User Intent Prediction in Information-seeking Conversations. In CHIIR, 2019
2019
-
[24]
C. Qu, L. Yang, M. Qiu, W. B. Croft, Y. Zhang, and M. Iyyer. BERT with History An- swer Embedding for Conversational Question Answering. CoRR, abs/1905.05412, 2019
1905 arXiv
-
[25]
Rajpurkar, J
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In EMNLP, 2016
2016
-
[26]
Rajpurkar, R
P. Rajpurkar, R. Jia, and P. Liang. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL, 2018
2018
-
[27]
Reddy, D
S. Reddy, D. Chen, and C. D. Manning. CoQA: A Conversational Question Answering Challenge. CoRR, abs/1808.07042, 2018
2018 arXiv
-
[28]
M. J. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional Attention Flow for Machine Comprehension. CoRR, abs/1611.01603, 2016
2016 arXiv
-
[29]
Thomas, D
P. Thomas, D. McDuff, M. Czerwinski, and N. Craswell. MISC: A data set of information-seeking conversations. In SIGIR (CAIR’17), 2017
2017
-
[30]
J. R. Trippas, D. Spina, L. Cavedon, H. Joho, and M. Sanderson. Informing the Design of Spoken Conversational Search: Perspective Paper. In CHIIR, 2018
2018
-
[31]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In NIPS, 2017
2017
-
[32]
W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou. Gated Self-Matching Networks for Reading Comprehension and Question Answering. In ACL, 2017
2017
-
[33]
Y. Xu, X. Liu, Y. Shen, J. Liu, and J. Gao. Multi-Task Learning for Machine Reading Comprehension. CoRR, abs/1809.06963, 2018
2018 arXiv
-
[34]
L. Yang, H. Zamani, Y. Zhang, J. Guo, and W. B. Croft. Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation. CoRR, 2017
2017
-
[35]
L. Yang, M. Qiu, C. Qu, J. Guo, Y. Zhang, W. B. Croft, J. Huang, and H. Chen. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. In SIGIR, 2018
2018
-
[36]
M. Yatskar. A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC. CoRR, abs/1809.10735, 2018
2018 arXiv
-
[37]
Zhang and Q
Y. Zhang and Q. Yang. A Survey on MultiTask Learning. 2018
2018
-
[38]
Zhang, X
Y. Zhang, X. Chen, Q. Ai, L. Yang, and W. B. Croft. Towards Conversational Search and Recommendation: System Ask, User Respond. In CIKM, 2018
2018
-
[39]
C. Zhu, M. Zeng, and X. Huang. SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering. CoRR, 2018
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.