Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Attentive History Selection for Conversational Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that learned, position-aware attention over conversation history—not a fixed recency heuristic—is what improves conversational question answering, reaching $65.7$ F1 on QuAC.

desk verdict Solid incremental ConvQA paper: the history attention idea is real and well supported, but the MTL claim in the abstract is contradicted by the paper's own ablation. read the letter →

arxiv 1908.09456 v1 pith:2RWILHIC submitted 2019-08-26 cs.IR

classification cs.IR
keywords ConversationalQuestionAnsweringMulti-turnConversationHistoryAttentionPositionalAnswerEmbeddingBERTDialogActPredictionMulti-taskLearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conversational question answering requires deciding which earlier turns are relevant to the current question, since questions can shift topics or return to old ones. This paper tries to show that the decision can be learned rather than assumed: it encodes each history turn separately with BERT, marks the passage tokens that appeared in history answers with position-aware embeddings, and uses a token-level attention mechanism to aggregate history representations. On QuAC, the resulting model reaches $65.7$ F1 on validation, a statistically significant gain over the strongest baseline (BERT + PosHAE, $64.7$), and $66.7$ with BERT-Large. Ablations indicate that history attention and the position signal each contribute, while the auxiliary dialog act prediction task mainly helps dialog act accuracy rather than answer span performance. If right, the paper establishes that history selection can be a learned, explainable component of conversational QA rather than a fixed recency heuristic.

What carries the argument

The load-bearing object is the history attention module sitting on top of a BERT encoder, together with the positional history answer embedding (PosHAE) that feeds it. PosHAE assigns each passage token a learned embedding indexed by the relative distance $(k-i)$ between the current question and a history turn whose answer covers that token, so BERT sees which tokens were answers and how far back they occurred. Each history turn is encoded in its own BERT sequence, and the attention module computes weights over those sequences; the fine-grained variant computes a separate weight for every token position, then aggregates token representations. These weights carry the argument: they make history selection soft, explainable, and learnable, and they are trained end-to-end with the answer span and dialog act losses.

What would settle it

Train HAM and HAM 'w/o Dialog Act Prediction' on QuAC under the same hyperparameters and evaluate on the hidden test set with a paired significance test: if the version without the dialog act loss matches or exceeds the full model on F1, the multi-task benefit claim for answer span prediction fails; the validation numbers in Table 6 already point in that direction.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a BERT-based conversational QA model improves by (1) embedding each history answer's position in the dialog, so the encoder sees how distant a history turn is from the current question, and (2) explicitly learning soft attention weights over all history turns, computed at the token level, and using the weighted sum as the representation for answer span and dialog act prediction. The complete model, HAM with BERT-base, obtains $65.7$ F1 on QuAC validation, significantly above the BERT + PosHAE baseline at $64.7$ ($p<0.05$); with BERT-Large it reaches $66.7$ F1 validation and $65.4$ test. Ablations show that replacing the fine-grained attention with sequence-level attention lowers F1 to $64.9$, removing attention entirely lowers it to $61.1$, and replacing PosHAE with the earlier HAE lowers it to $64.2$. The paper also claims that jointly predicting dialog acts and answer spans lets one model handle both tasks, and that answer span supervision improves dialog act prediction, while the reverse direction shows no significant answer-side gain in the reported numbers.

Load-bearing premise

The assumption that predicting dialog acts alongside answers improves answer quality is load-bearing for the paper's multi-task claim, yet the paper's own ablation shows that removing dialog act prediction raises answer F1 from $65.7$ to $65.9$.

Editorial extensions

If this is right

  • Conversational QA systems can replace the 'always use the immediately preceding turns' heuristic with learned attention, including attending to remote turns in topic-return cases.
  • Encoding history position is cheap and effective: the PosHAE variant outperforms the plain HAE by $0.8$ F1 validation ($64.7$ vs $63.9$), and ablating PosHAE from HAM drops F1 by $1.5$.
  • Token-level attention weights give a built-in explanation channel: visualized weights align with drill-down, topic-shift, and topic-return patterns, which could support debugging and user-facing explanations.
  • A single multi-task model can serve both answer span and dialog act prediction, with answer span supervision contributing to dialog act accuracy (removing it drops Yes/No from $88.3$ to $86.2$ and Follow up from $62.3$ to $59.7$).
  • The gains compose with model scale: BERT-Large raises validation F1 to $66.7$ and test F1 to $65.4$, so the history attention design does not depend on a small encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the encoder processes each history turn separately before attention, the same architecture could be lifted to other conversational tasks where history selection matters, such as response ranking or conversational recommendation, replacing fixed recency features with learned soft selection.
  • The ablation numbers suggest an asymmetry: answer span supervision helps dialog act prediction, but dialog act loss does not help answer spans (removing it slightly raises F1 to $65.9$); a practical follow-up would drop the dialog act loss when answer quality is the only goal.
  • The attention weights could serve as pseudo-labels for training a cheaper history selector, distilling which turns the full BERT model would rely on; this distillation direction is a testable extension the paper does not pursue.
  • The gains may concentrate on dialogs with topic shift or topic return; one could test this by slicing QuAC validation by dialog-behavior labels and comparing HAM against the recency heuristic baseline on those subsets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses conversational question answering (ConvQA) on the QuAC benchmark and proposes three main components: a history attention mechanism (HAM) that performs soft selection over conversation history turns, a positional history answer embedding (PosHAE) that augments BERT embeddings with relative position information of history answers, and a multi-task learning (MTL) setup that jointly trains answer span prediction and dialog act prediction. Each history turn is encoded separately with BERT and PosHAE, then token-level attention aggregates the turn representations before span prediction. The full HAM model with BERT-base achieves 65.7 F1 on QuAC validation and 64.4 on the official test set, with a statistically significant (p<0.05) validation F1 improvement over the strongest baseline, BERT+PosHAE (64.7). Ablations show that removing the fine-grained attention, removing history attention, and replacing PosHAE with HAE all lower F1, and attention visualizations illustrate drill-down, topic-shift, and topic-return behaviors.

Significance. If the design conclusions hold, the paper makes a useful empirical contribution to conversational QA: it demonstrates that adding position information to history answer embeddings and learning to re-weight history turns can improve BERT-based ConvQA on a public leaderboard. The paper's strengths include an open-source implementation, comparison against published QuAC baselines, a paired significance test on the main validation F1 metric, and a fairly complete set of ablations for the proposed modules. The main caveats are that one of the three claimed contributions, multi-task learning, is not supported by the paper's own ablation for the primary answer-span task, and the history-attention comparison is not fully isolated from the change in input architecture. These issues are fixable but currently make the abstract and contribution statements stronger than the evidence.

major comments (3)
  1. [Abstract and §1 contribution (3); §4.4 Table 6, observation (4)] The abstract and contribution (3) claim that multi-task learning improves the performance of ConvQA, but the paper's own ablation contradicts this for the primary task: HAM achieves 65.7 F1, while HAM w/o Dialog Act Prediction achieves 65.9 F1, and observation (4) in Section 4.4 admits that the change is a 'slight and insignificant increase.' The demonstrated benefit of MTL is confined to the secondary dialog act task, where removing answer-span prediction drops Yes/No accuracy from 88.3 to 86.2 and Follow-up accuracy from 62.3 to 59.7. The abstract and contribution (3) should be revised to state that MTL improves dialog act prediction and provides a uniform architecture, or the claim that it improves ConvQA answer-span performance should be removed.
  2. [§4.4 Table 6; §4.2.1] The evidence that learned history attention outperforms the heuristic selection used by prior work is indirect. HAM differs from BERT+PosHAE both in the attention mechanism and in the input architecture: HAM encodes each history turn separately and aggregates the resulting representations, while BERT+PosHAE packs the selected history turns into a single input sequence and relies on the encoder. The 'HAM w/o History Attention' ablation uses equal weights across separately encoded turns, which is not the 'immediate previous turns' heuristic used by the baselines, as the authors themselves acknowledge in Section 4.4. To support the specific claim that learned 'soft selection' is better than heuristic selection, the paper should add an ablation in which the HAM architecture is run with fixed recency-based weights or hard selection of the last k turns, rather than only with uniform weights.
  3. [§4.3 Table 4, observation (3)] The paper claims 'substantially higher performance on dialog act prediction compared to baseline methods,' but no significance test or error bar is reported for the dialog act accuracies, and only one baseline (BiDAF++ w/ 2-Context) has dialog act numbers in Table 4. The observed improvements (Yes/No 88.3 vs 86.6; Follow-up 62.3 vs 61.6) may be reliable, but the claim as stated is stronger than the reported evidence. The statement should be qualified or supplemented with significance testing on dialog act predictions. The same applies to the HEQ-Q and HEQ-D differences, which are reported without significance testing.
minor comments (5)
  1. [Table 1] In the last row of Table 1, the utterance ID is listed as 'A6' but should be 'A7' for consistency with the dialog turn numbering.
  2. [§4.3 vs Table 6 caption] Section 4.3 observation (3) states 'We can only do significance test on F1,' while the caption of Table 6 states 'We can only do significance test on F1 and dialog act accuracy.' Please reconcile these statements.
  3. [Table 4] BERT+PosHAE has no test-set F1 in Table 4, so the test-set comparison for the strongest baseline is missing; the significance claim should be clearly limited to the validation set, and the test-set gains over FlowQA (64.4 vs 64.1) should be discussed with appropriate caution.
  4. [§3.7.2 and §4.4] The full HAM model uses λ=0.1 and µ=0.8 in Equation (7), while the MTL ablation 'HAM w/o Answer Span Prediction' tunes λ in (0.2, 0.4, 0.6, 0.8) and adopts λ=0.2; please clarify whether the different λ values affect the comparison.
  5. [§3.4 and §1] There are minor language issues, e.g., 'pass it though the history attention network' in Section 3.4 and 'We show that these benefits have contributions to the model performance' in Section 1; these should be corrected in revision.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the core HAM/PosHAE results are empirical and evaluated against external baselines; the MTL claim is internally overstated but does not reduce to its own inputs.

full rationale

The paper's central derivation is an end-to-end learning pipeline, not an analytic derivation. Equations (1)-(7) define a standard attention-weighted aggregation and loss combination; none of them defines the target metric (F1/HEQ) in terms of a fitted parameter that is then reported as a prediction. PosHAE extends the authors' earlier HAE [24] by adding a learned relative-position embedding, and its contribution is tested by ablating it (Table 6, 'w/o PosHAE'), while HAM's attention contribution is tested by ablating to equal weights ('w/o History Attention') and by comparing against external baselines (BiDAF++, FlowQA, BERT+HAE). The self-citations to [24] supply a baseline model and a history-representation idea, but they are not used as an unexamined uniqueness theorem or as a substitute for evaluation. The one notable weakness is not circularity: the abstract claims MTL 'improve[s] the performance of ConvQA,' but the paper's own Table 6 shows HAM w/o Dialog Act Prediction at 65.9 F1 versus 65.7 for HAM, and observation (4) acknowledges the increase is 'slight and insignificant.' That is an overclaim or correctness issue for a reviewer to flag, not a derivation that reduces to its inputs. Since the main results are measured on held-out QuAC data against independent baselines, no load-bearing circular step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central performance claims rest on the choice of BERT as the encoder, the decision to ignore history questions, several tuned hyperparameters (max history turns, loss weights, pooling), and the validity of the significance test. None of these are externally verified in the paper; the hyperparameters are tuned on the validation set.

free parameters (4)
  • max_history_turns = 6
    Set to 6 in BERT+PosHAE because 'it gives the best performance under this setting' (§4.2.1), a validation-set choice.
  • lambda (dialog act loss weight) = 0.1 (HAM), 0.2 (HAM w/o Answer Span Prediction)
    λ=0.1 for HAM from §4.2.3; in the ablation, λ tuned in {0.2,0.4,0.6,0.8} and 0.2 chosen for best performance (§4.4).
  • mu (answer span loss weight) = 0.8 (HAM)
    Set to 0.8 in §4.2.3, and set to 1 or 0 in ablations.
  • pooling method = max pooling (HAM), average pooling (HAM w/o Answer Span Prediction)
    HAM uses max pooling for sequence representations (§4.3 observation 3), while the ablation adopts average pooling 'since they give the best performance' (§4.4).
assumptions (4)
  • domain assumption BERT provides suitable contextualized token and sequence representations for the ConvQA task.
    The entire encoder is a pretrained BERT model (§3.3.1), and the paper relies on its representations without fine-tuning the pretraining objective.
  • domain assumption History questions contribute little and can be ignored; only history answers matter.
    The paper states 'previous works [2, 24] show that history questions contribute little to the performance' (§3.3.2), so PosHAE only embeds history answers.
  • domain assumption The Student's paired t-test is a valid significance test for the F1 differences on QuAC.
    The paper reports p<0.05 from a paired t-test on validation F1 (§4.3), assuming independence and normality of the paired differences across the validation set.
  • domain assumption QuAC is a representative benchmark for ConvQA and the leaderboard results are reliable.
    The paper uses QuAC for all experiments and compares test results to the leaderboard at the CIKM deadline (§4.2.1), which may change over time.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attentive History Selection for Conversational Question Answering." pith.science (2026). https://pith.science/paper/2RWILHIC

@misc{pith2026190809456,
  author       = {Pith},
  title        = {Pith review of: Attentive History Selection for Conversational Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RWILHIC}},
  note         = {Machine review of arXiv:1908.09456}
}
read the original abstract

Conversational question answering (ConvQA) is a simplified but concrete setting of conversational search. One of its major challenges is to leverage the conversation history to understand and answer the current question. In this work, we propose a novel solution for ConvQA that involves three aspects. First, we propose a positional history answer embedding method to encode conversation history with position information using BERT in a natural way. BERT is a powerful technique for text representation. Second, we design a history attention mechanism (HAM) to conduct a "soft selection" for conversation histories. This method attends to history turns with different weights based on how helpful they are on answering the current question. Third, in addition to handling conversation history, we take advantage of multi-task learning (MTL) to do answer prediction along with another essential conversation task (dialog act prediction) using a uniform model architecture. MTL is able to learn more expressive and generic representations to improve the performance of ConvQA. We demonstrate the effectiveness of our model with extensive experimental evaluations on QuAC, a large-scale ConvQA dataset. We show that position information plays an important role in conversation history modeling. We also visualize the history attention and provide new insights into conversation history understanding.

Figures

Figures reproduced from arXiv: 1908.09456 by the authors.

Figure 1
Figure 1. Our model consists of an encoder, a history attention module, an answer span predictor, and a dialog act predic [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The encoder with PosHAE. It zooms in to the en [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Attention visualization for different dialog behaviors. Brighter spots mean higher attention weights. Token ID refers [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages

  1. [1]

    N. J. Belkin, C. Cool, A. S., and U. Thiel. Cases , Scripts , and Information-Seeking Strategies : On the Design of Interactive Information Retrieval Systems. 1994

  2. [2]

    E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. S. Zettlemoyer. QuAC: Question Answering in Context. In EMNLP, 2018

  3. [3]

    Chuklin, A

    A. Chuklin, A. Severyn, J. R. Trippas, E. Alfonseca, H. Silén, and D. Spina. Prosody Modifications for Question-Answering in Voice-Only Settings. CoRR, 2018

  4. [4]

    Clark and M

    C. Clark and M. Gardner. Simple and Effective Multi-Paragraph Reading Com- prehension. In ACL, 2018

  5. [5]

    W. B. Croft and R. H. Thompson. I3R: A new approach to the design of document retrieval systems. JASIS, 38:389–404, 1987

  6. [6]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, 2018

  7. [7]

    J. Gao, M. Galley, and L. Li. Neural Approaches to Conversational AI. In SIGIR, 2018

  8. [8]

    J. Guo, Y. Fan, L. Pang, L. Yang, Q. Ai, H. Zamani, C. Wu, W. B. Croft, and X. Cheng. A Deep Look into Neural Ranking Models for Information Retrieval. CoRR, abs/1903.06902, 2019

Show all 39 references
  1. [9]

    M. Hu, Y. Peng, Z. Huang, X. Qiu, F. Wei, and M. Zhou. Reinforced Mnemonic Reader for Machine Reading Comprehension. In IJCAI, 2018

  2. [10]

    Huang, C

    H.-Y. Huang, C. Zhu, Y. Shen, and W. Chen. FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension. CoRR, abs/1711.07341, 2017

  3. [11]

    Huang, E

    H.-Y. Huang, E. Choi, and W. Yih. FlowQA: Grasping Flow in History for Con- versational Machine Comprehension. CoRR, 2018

  4. [12]

    M. S. Joshi, E. Choi, D. S. Weld, and L. S. Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In ACL, 2017

  5. [13]

    Kotov and C

    A. Kotov and C. Zhai. Towards natural question guided search. In WWW, 2010

  6. [14]

    Kwiatkowski, J

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Ep- stein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural Questions: a Benchmark for Question Answering Res...

  7. [15]

    F.-L. Li, M. Qiu, H. Chen, X. Wang, X. Gao, J. Huang, J. Ren, Z. Zhao, W. Zhao, L. Wang, G. Jin, and W. Chu. AliMe Assist : An Intelligent Assistant for Creating an Innovative E-commerce Experience. In CIKM, 2017

  8. [16]

    X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y.-Y. Wang. Representation Learn- ing Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval. In HLT-NAACL, 2015

  9. [17]

    X. Liu, P. He, W. Chen, and J. Gao. Multi-Task Deep Neural Networks for Natural Language Understanding. CoRR, abs/1901.11504, 2019

  10. [18]

    Nguyen, M

    T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoRR, abs/1611.09268, 2016

  11. [19]

    R. N. Oddy. Information Retrieval through Man-Machine Dialogue. 1977

  12. [20]

    M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018

  13. [21]

    C. Qu, L. Yang, W. B. Croft, J. R. Trippas, Y. Zhang, and M. Qiu. Analyzing and Characterizing User Intent in Information-seeking Conversations. In SIGIR, 2018

  14. [22]

    C. Qu, L. Yang, W. B. Croft, F. Scholer, and Y. Zhang. Answer Interaction in Non-factoid Question Answering Systems. In CHIIR, 2019

  15. [23]

    C. Qu, L. Yang, W. B. Croft, Y. Zhang, J. R. Trippas, and M. Qiu. User Intent Prediction in Information-seeking Conversations. In CHIIR, 2019

  16. [24]

    C. Qu, L. Yang, M. Qiu, W. B. Croft, Y. Zhang, and M. Iyyer. BERT with History An- swer Embedding for Conversational Question Answering. CoRR, abs/1905.05412, 2019

  17. [25]

    Rajpurkar, J

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In EMNLP, 2016

  18. [26]

    Rajpurkar, R

    P. Rajpurkar, R. Jia, and P. Liang. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL, 2018

  19. [27]

    Reddy, D

    S. Reddy, D. Chen, and C. D. Manning. CoQA: A Conversational Question Answering Challenge. CoRR, abs/1808.07042, 2018

  20. [28]

    M. J. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional Attention Flow for Machine Comprehension. CoRR, abs/1611.01603, 2016

  21. [29]

    Thomas, D

    P. Thomas, D. McDuff, M. Czerwinski, and N. Craswell. MISC: A data set of information-seeking conversations. In SIGIR (CAIR’17), 2017

  22. [30]

    J. R. Trippas, D. Spina, L. Cavedon, H. Joho, and M. Sanderson. Informing the Design of Spoken Conversational Search: Perspective Paper. In CHIIR, 2018

  23. [31]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In NIPS, 2017

  24. [32]

    W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou. Gated Self-Matching Networks for Reading Comprehension and Question Answering. In ACL, 2017

  25. [33]

    Y. Xu, X. Liu, Y. Shen, J. Liu, and J. Gao. Multi-Task Learning for Machine Reading Comprehension. CoRR, abs/1809.06963, 2018

  26. [34]

    L. Yang, H. Zamani, Y. Zhang, J. Guo, and W. B. Croft. Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation. CoRR, 2017

  27. [35]

    L. Yang, M. Qiu, C. Qu, J. Guo, Y. Zhang, W. B. Croft, J. Huang, and H. Chen. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. In SIGIR, 2018

  28. [36]

    M. Yatskar. A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC. CoRR, abs/1809.10735, 2018

  29. [37]

    Zhang and Q

    Y. Zhang and Q. Yang. A Survey on MultiTask Learning. 2018

  30. [38]

    Zhang, X

    Y. Zhang, X. Chen, Q. Ai, L. Yang, and W. B. Croft. Towards Conversational Search and Recommendation: System Ask, User Respond. In CIKM, 2018

  31. [39]

    C. Zhu, M. Zeng, and X. Huang. SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering. CoRR, 2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.