{"id":"95642712-6d9a-4170-bd2d-2da8cedb1fb3","arxiv_id":"1908.09456","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A BERT-based model with position-aware history answer embeddings and a learned history attention mechanism improves QuAC F1 by about one point over strong baselines, but multi-task learning with dialog acts does not help answer span prediction.","lead":"This paper builds a conversational question answering model that uses attention weights to select helpful earlier dialogue turns and a position-aware embedding of past answers. It reports a small, statistically significant F1 gain on QuAC, but its multi-task learning contribution is not supported by its own ablation results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MTL contribution overclaimed: Table 6 shows removing dialog act prediction gives F1 65.9 vs 65.7 for HAM, contradicting the abstract's claim that MTL improves ConvQA.","rationale":"The central F1 claim is otherwise reasonably supported: Table 6 shows significant drops when removing fine-grained attention (64.9), history attention (61.1), and PosHAE (64.2), so the attribution to history selection and positional embedding is internally consistent. The weakest point is the MTL contribution, which the reader correctly identified. The paper itself flags the null result in §4.4 observation 4, but the abstract and contribution list still assert that MTL improves ConvQA. This is an overclaim about one of the paper's three stated contributions, not an attack on the main architecture's usefulness. Because the issue is addressable by revising the claim or providing a proper significance test, the existing CONDITIONAL verdict remains appropriate; no change in verdict is needed.","tokens_in":16583,"tokens_out":11302,"duration_ms":123286,"concrete_test":"Run the §4.4 HAM vs HAM w/o Dialog Act Prediction comparison five times with different random seeds using the released code, and perform a paired bootstrap or paired t-test on QuAC validation F1. Report the p-value and 95% CI for the 0.2 F1 gap. If the gap is not significant, or the ablation remains numerically better, the paper should revise contribution (3) and the abstract to state that MTL improves dialog act prediction but not answer span prediction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's abstract and contribution (3) claim that multi-task learning 'improve[s] the performance of ConvQA.' The ablation in §4.4, Table 6, contradicts this for the primary metric: HAM obtains 65.7 F1, while HAM w/o Dialog Act Prediction obtains 65.9 F1, and the authors state the difference is a 'slight and insignificant increase' (observation 4). Thus the assumption that dialog act supervision benefits answer span prediction is not supported by the paper's own numbers. The observed MTL benefit runs in the opposite direction: removing answer span prediction significantly hurts dialog act accuracy (Yes/No from 88.3 to 86.2; Follow up from 62.3 to 59.7). So MTL helps the secondary task, but the blanket claim that MTL improves ConvQA is not established for the task that defines the headline result. This is load-bearing because one of the paper's three listed contributions is directly contradicted by its own experiments; a reader relying on the abstract would take away an unsupported performance claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses conversational question answering (ConvQA) on the QuAC benchmark and proposes three main components: a history attention mechanism (HAM) that performs soft selection over conversation history turns, a positional history answer embedding (PosHAE) that augments BERT embeddings with relative position information of history answers, and a multi-task learning (MTL) setup that jointly trains answer span prediction and dialog act prediction. Each history turn is encoded separately with BERT and PosHAE, then token-level attention aggregates the turn representations before span prediction. The full HAM model with BERT-base achieves 65.7 F1 on QuAC validation and 64.4 on the official test set, with a statistically significant (p<0.05) validation F1 improvement over the strongest baseline, BERT+PosHAE (64.7). Ablations show that removing the fine-grained attention, removing history attention, and replacing PosHAE with HAE all lower F1, and attention visualizations illustrate drill-down, topic-shift, and topic-return behaviors.","tokens_in":16819,"tokens_out":7893,"duration_ms":75286,"significance":"If the design conclusions hold, the paper makes a useful empirical contribution to conversational QA: it demonstrates that adding position information to history answer embeddings and learning to re-weight history turns can improve BERT-based ConvQA on a public leaderboard. The paper's strengths include an open-source implementation, comparison against published QuAC baselines, a paired significance test on the main validation F1 metric, and a fairly complete set of ablations for the proposed modules. The main caveats are that one of the three claimed contributions, multi-task learning, is not supported by the paper's own ablation for the primary answer-span task, and the history-attention comparison is not fully isolated from the change in input architecture. These issues are fixable but currently make the abstract and contribution statements stronger than the evidence.","major_comments":[{"comment":"The abstract and contribution (3) claim that multi-task learning improves the performance of ConvQA, but the paper's own ablation contradicts this for the primary task: HAM achieves 65.7 F1, while HAM w/o Dialog Act Prediction achieves 65.9 F1, and observation (4) in Section 4.4 admits that the change is a 'slight and insignificant increase.' The demonstrated benefit of MTL is confined to the secondary dialog act task, where removing answer-span prediction drops Yes/No accuracy from 88.3 to 86.2 and Follow-up accuracy from 62.3 to 59.7. The abstract and contribution (3) should be revised to state that MTL improves dialog act prediction and provides a uniform architecture, or the claim that it improves ConvQA answer-span performance should be removed.","section":"Abstract and §1 contribution (3); §4.4 Table 6, observation (4)"},{"comment":"The evidence that learned history attention outperforms the heuristic selection used by prior work is indirect. HAM differs from BERT+PosHAE both in the attention mechanism and in the input architecture: HAM encodes each history turn separately and aggregates the resulting representations, while BERT+PosHAE packs the selected history turns into a single input sequence and relies on the encoder. The 'HAM w/o History Attention' ablation uses equal weights across separately encoded turns, which is not the 'immediate previous turns' heuristic used by the baselines, as the authors themselves acknowledge in Section 4.4. To support the specific claim that learned 'soft selection' is better than heuristic selection, the paper should add an ablation in which the HAM architecture is run with fixed recency-based weights or hard selection of the last k turns, rather than only with uniform weights.","section":"§4.4 Table 6; §4.2.1"},{"comment":"The paper claims 'substantially higher performance on dialog act prediction compared to baseline methods,' but no significance test or error bar is reported for the dialog act accuracies, and only one baseline (BiDAF++ w/ 2-Context) has dialog act numbers in Table 4. The observed improvements (Yes/No 88.3 vs 86.6; Follow-up 62.3 vs 61.6) may be reliable, but the claim as stated is stronger than the reported evidence. The statement should be qualified or supplemented with significance testing on dialog act predictions. The same applies to the HEQ-Q and HEQ-D differences, which are reported without significance testing.","section":"§4.3 Table 4, observation (3)"}],"minor_comments":[{"comment":"In the last row of Table 1, the utterance ID is listed as 'A6' but should be 'A7' for consistency with the dialog turn numbering.","section":"Table 1"},{"comment":"Section 4.3 observation (3) states 'We can only do significance test on F1,' while the caption of Table 6 states 'We can only do significance test on F1 and dialog act accuracy.' Please reconcile these statements.","section":"§4.3 vs Table 6 caption"},{"comment":"BERT+PosHAE has no test-set F1 in Table 4, so the test-set comparison for the strongest baseline is missing; the significance claim should be clearly limited to the validation set, and the test-set gains over FlowQA (64.4 vs 64.1) should be discussed with appropriate caution.","section":"Table 4"},{"comment":"The full HAM model uses λ=0.1 and µ=0.8 in Equation (7), while the MTL ablation 'HAM w/o Answer Span Prediction' tunes λ in (0.2, 0.4, 0.6, 0.8) and adopts λ=0.2; please clarify whether the different λ values affect the comparison.","section":"§3.7.2 and §4.4"},{"comment":"There are minor language issues, e.g., 'pass it though the history attention network' in Section 3.4 and 'We show that these benefits have contributions to the model performance' in Section 1; these should be corrected in revision.","section":"§3.4 and §1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for CIKM and the code release is a positive feature. The central results on PosHAE and history attention are plausible, but the MTL overclaim in the abstract and contributions must be corrected, and the history-attention comparison should be strengthened with an ablation that directly contrasts learned attention with heuristic selection. I do not see grounds for rejection if these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the history attention mechanism is the real contribution and the paper's evaluation supports it. The multi-task learning contribution, as advertised in the abstract, is not supported by the paper's own ablation—removing dialog act prediction gives F1 65.9 vs the full model's 65.7. That is a mismatch between the abstract and the evidence, but it is an overclaim about one of three contributions, not a flaw in the central result.\n\nWhat's actually new: the paper extends the authors' own HAE with position information (PosHAE) and, more importantly, introduces a learned history attention over single-turn BERT encodings, including a token-level variant. The design is clean: encode each history turn separately, aggregate with attention, then predict spans and dialog acts. On QuAC, HAM gets 65.7 F1 on validation, which is a significant (p<0.05) gain over the strongest baseline, BERT+PosHAE at 64.7. The ablation shows the attention module matters: removing it drops F1 to 61.1, and replacing token-level attention with sequence-level attention costs about 0.8 F1. The code is open-sourced, and the attention visualizations for drill-down, topic shift, and topic return are a nice touch—they make the learned weights interpretable.\n\nWhere it gets softer: the MTL claim. The paper's third contribution says MTL 'improve[s] the performance of ConvQA,' but Table 6 shows the opposite for answer span prediction: HAM w/o Dialog Act Prediction is 65.9, slightly above HAM's 65.7, and the authors themselves call the difference 'slight and insignificant.' The MTL does help the dialog act task (removing answer span prediction drops Yes/No from 88.3 to 86.2 and Follow up from 62.3 to 59.7), and the body of the paper is honest about that. But the abstract and the contributions list frame MTL as a performance booster for ConvQA, and that framing is not supported. This is fixable with a revised abstract, but as written it's an overclaim.\n\nTwo minor points. Significance testing is only reported for F1; the HEQ and dialog act differences are not tested, which is worth a footnote. And they modified the BERT+HAE baseline by changing max answer length from 30 to 40; it's disclosed in a footnote, so it's fine, but a referee should ask whether all baselines were run under the same conditions.\n\nBottom line: this is a solid, incremental systems paper. The central mechanism works, the evaluation is mostly sound, and the weaknesses are correctable. It deserves a serious referee; I'd push the authors to align the abstract and contribution claims with the actual ablation results.","headline":"Solid incremental ConvQA paper: the history attention idea is real and well supported, but the MTL claim in the abstract is contradicted by the paper's own ablation.","tokens_in":17361,"tokens_out":3000,"would_cite":true,"duration_ms":27091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that learned, position-aware attention over conversation history—not a fixed recency heuristic—is what improves conversational question answering, reaching $65.7$ F1 on QuAC.","keywords":["Conversational Question Answering","Multi-turn Question Answering","Conversation History","History Attention","Positional History Answer Embedding","BERT","Dialog Act Prediction","Multi-task Learning"],"falsifier":"Train HAM and HAM 'w/o Dialog Act Prediction' on QuAC under the same hyperparameters and evaluate on the hidden test set with a paired significance test: if the version without the dialog act loss matches or exceeds the full model on F1, the multi-task benefit claim for answer span prediction fails; the validation numbers in Table 6 already point in that direction.","tokens_in":16395,"feed_emoji":"💬","tokens_out":9142,"duration_ms":79202,"temperature":0.7,"pith_summary":"Conversational question answering requires deciding which earlier turns are relevant to the current question, since questions can shift topics or return to old ones. This paper tries to show that the decision can be learned rather than assumed: it encodes each history turn separately with BERT, marks the passage tokens that appeared in history answers with position-aware embeddings, and uses a token-level attention mechanism to aggregate history representations. On QuAC, the resulting model reaches $65.7$ F1 on validation, a statistically significant gain over the strongest baseline (BERT + PosHAE, $64.7$), and $66.7$ with BERT-Large. Ablations indicate that history attention and the position signal each contribute, while the auxiliary dialog act prediction task mainly helps dialog act accuracy rather than answer span performance. If right, the paper establishes that history selection can be a learned, explainable component of conversational QA rather than a fixed recency heuristic.","feed_headline":"Attention over conversation history lifts QuAC answers to 65.7 F1","feed_subtitle":"Position-aware history embeddings and token-level attention beat the strongest baseline by 1.0 F1.","key_machinery":"The load-bearing object is the history attention module sitting on top of a BERT encoder, together with the positional history answer embedding (PosHAE) that feeds it. PosHAE assigns each passage token a learned embedding indexed by the relative distance $(k-i)$ between the current question and a history turn whose answer covers that token, so BERT sees which tokens were answers and how far back they occurred. Each history turn is encoded in its own BERT sequence, and the attention module computes weights over those sequences; the fine-grained variant computes a separate weight for every token position, then aggregates token representations. These weights carry the argument: they make history selection soft, explainable, and learnable, and they are trained end-to-end with the answer span and dialog act losses.","core_discovery":"On its own terms, the paper's central claim is that a BERT-based conversational QA model improves by (1) embedding each history answer's position in the dialog, so the encoder sees how distant a history turn is from the current question, and (2) explicitly learning soft attention weights over all history turns, computed at the token level, and using the weighted sum as the representation for answer span and dialog act prediction. The complete model, HAM with BERT-base, obtains $65.7$ F1 on QuAC validation, significantly above the BERT + PosHAE baseline at $64.7$ ($p<0.05$); with BERT-Large it reaches $66.7$ F1 validation and $65.4$ test. Ablations show that replacing the fine-grained attention with sequence-level attention lowers F1 to $64.9$, removing attention entirely lowers it to $61.1$, and replacing PosHAE with the earlier HAE lowers it to $64.2$. The paper also claims that jointly predicting dialog acts and answer spans lets one model handle both tasks, and that answer span supervision improves dialog act prediction, while the reverse direction shows no significant answer-side gain in the reported numbers.","pith_inferences":["Because the encoder processes each history turn separately before attention, the same architecture could be lifted to other conversational tasks where history selection matters, such as response ranking or conversational recommendation, replacing fixed recency features with learned soft selection.","The ablation numbers suggest an asymmetry: answer span supervision helps dialog act prediction, but dialog act loss does not help answer spans (removing it slightly raises F1 to $65.9$); a practical follow-up would drop the dialog act loss when answer quality is the only goal.","The attention weights could serve as pseudo-labels for training a cheaper history selector, distilling which turns the full BERT model would rely on; this distillation direction is a testable extension the paper does not pursue.","The gains may concentrate on dialogs with topic shift or topic return; one could test this by slicing QuAC validation by dialog-behavior labels and comparing HAM against the recency heuristic baseline on those subsets."],"forward_implications":["Conversational QA systems can replace the 'always use the immediately preceding turns' heuristic with learned attention, including attending to remote turns in topic-return cases.","Encoding history position is cheap and effective: the PosHAE variant outperforms the plain HAE by $0.8$ F1 validation ($64.7$ vs $63.9$), and ablating PosHAE from HAM drops F1 by $1.5$.","Token-level attention weights give a built-in explanation channel: visualized weights align with drill-down, topic-shift, and topic-return patterns, which could support debugging and user-facing explanations.","A single multi-task model can serve both answer span and dialog act prediction, with answer span supervision contributing to dialog act accuracy (removing it drops Yes/No from $88.3$ to $86.2$ and Follow up from $62.3$ to $59.7$).","The gains compose with model scale: BERT-Large raises validation F1 to $66.7$ and test F1 to $65.4$, so the history attention design does not depend on a small encoder."],"supporting_citations":[{"why":"It supplies the history answer embedding (HAE) scheme and the history-selection/history-modeling framework that the paper extends, and it provides the BERT+HAE baseline of $63.9$ F1.","marker":"[24]"},{"why":"It releases QuAC, defines the ConvQA task, dialog acts, and evaluation metrics, and provides the BiDAF++ baselines.","marker":"[2]"},{"why":"It provides the BERT encoder used as the backbone across all model variants.","marker":"[6]"},{"why":"It is the FlowQA baseline that models conversation history with recurrent structures and is compared against in the main results.","marker":"[11]"},{"why":"It is the qualitative analysis of QuAC documenting topic shift and topic return, which motivates learned history selection.","marker":"[36]"},{"why":"It supplies the multi-task learning rationale and shared-encoder paradigm for combining answer span and dialog act prediction.","marker":"[17]"}],"fun_headline_variants":["Position and attention sharpen conversational QA to 65.7 F1","History attention lifts QuAC to 65.7 F1, beating baseline by 1.0","Multi-task learning with position-aware history hits 65.7 F1 on QuAC","Attentive history selection improves conversational QA to 65.7 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that predicting dialog acts alongside answers improves answer quality is load-bearing for the paper's multi-task claim, yet the paper's own ablation shows that removing dialog act prediction raises answer F1 from $65.7$ to $65.9$.","fun_headline_variants_meta":{"raw":{"variants":["Position and attention sharpen conversational QA to 65.7 F1","History attention lifts QuAC to 65.7 F1, beating baseline by 1.0","Multi-task learning with position-aware history hits 65.7 F1 on QuAC","Attentive history selection improves conversational QA to 65.7 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000926,"raw_usage":{"total_tokens":3997,"prompt_tokens":1005,"completion_tokens":2992,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":2904}},"tokens_in":621,"tokens_out":2992,"duration_ms":21045,"temperature":1.0,"reasoning_tokens":2904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:10:18.115616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HAM and HAM 'w/o Dialog Act Prediction' on QuAC under the same hyperparameters and evaluate on the hidden test set with a paired significance test: if the version without the dialog act loss matches or exceeds the full model on F1, the multi-task benefit claim for answer span prediction fails; the validation numbers in Table 6 already point in that direction.","supporting_citations":[{"cited_title":"BERT with History Answer Embedding for Conversational Question Answering","cited_arxiv_id":"1905.05412","evidence_quote":"It supplies the history answer embedding (HAE) scheme and the history-selection/history-modeling framework that the paper extends, and it provides the BERT+HAE baseline of $63.9$ F1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It releases QuAC, defines the ConvQA task, dialog acts, and evaluation metrics, and provides the BiDAF++ baselines."},{"cited_title":"Devlin, M.-W","cited_arxiv_id":null,"evidence_quote":"It provides the BERT encoder used as the backbone across all model variants."},{"cited_title":"Huang, E","cited_arxiv_id":null,"evidence_quote":"It is the FlowQA baseline that models conversation history with recurrent structures and is compared against in the main results."},{"cited_title":"A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC","cited_arxiv_id":"1809.10735","evidence_quote":"It is the qualitative analysis of QuAC documenting topic shift and topic return, which motivates learned history selection."}],"review_version":1}