Pith. sign in

REVIEW 3 major objections 5 minor 20 references

FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A one-line change to the flow update—feeding in the difference between consecutive reasoning states—improves conversational machine comprehension, with state-of-the-art results on QuAC and on the Scene and Tangrams domains of SCONE.

desk verdict A simple, honest extension of FlowQA with consistent but weakly isolated gains; worth reviewing, and the ablation needs a parameter-matched control. read the letter →

arxiv 1908.05117 v3 pith:ZRT3KEHN submitted 2019-08-14 cs.CL cs.AI

Yi-Ting Yeh , Yun-Nung Chen This is my paper · ORCID
classification cs.CLcs.AI
keywords conversationalmachinecomprehensiondialogueflowinformationgainDeltaQABERTintegrationQuACSCONE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that conversational question answering improves when a model is told, not just implicitly shown, what changed in its reasoning between dialogue turns. Its FlowDelta mechanism feeds the difference between the two previous flow-state vectors into each GRU update, so the network can key in on context spans whose representation shifted as the topic changed. On the QuAC and CoQA dialogue datasets and the SCONE instruction-following task, this simple extra signal raises performance over the base FlowQA model and, combined with BERT, gives state-of-the-art published numbers on QuAC and on two of SCONE's three domains. The point is that reasoning history can be used more effectively by making the information gain explicit rather than relying on the hidden state to encode it.

What carries the argument

The central object is the FlowDelta operation, a one-line modification of the FLOW recurrence: $h_{k,j} = \mathrm{GRU}([c_{k,j}; h_{k-1,j} - h_{k-2,j}], h_{k-1,j})$. Here $j$ indexes context positions and $k$ indexes dialogue turns, so the input concatenates the current context-word representation with the change in the hidden state from two turns ago to the last turn. That difference is the information-gain estimate: spans whose representations moved a lot between turns are the ones the current question likely needs. It is inserted in two places in the BERT variant: before the final span-prediction layer and inside the last BERT layer. The mechanism carries the argument because all reported improvements come from adding this delta, with only a small increase in GRU input dimension.

What would settle it

Train the same BERT-FlowDelta model with the delta term replaced by a random vector of identical shape, or by the previous hidden state itself, while keeping all other hyperparameters fixed; the claim that the difference encodes information gain predicts a clear drop in QuAC and CoQA F1, and if performance is unchanged, the gains are not caused by the delta as an information-gain signal.

Watch

Extended reading notes

Core claim

The central claim is that, in multi-turn reasoning over a passage, the vector difference $h_{k-1,j} - h_{k-2,j}$ between the flow states of consecutive turns is a usable signal of information gain. The paper modifies the FLOW operation so that the GRU at turn $k$ receives $[c_{k,j}; h_{k-1,j} - h_{k-2,j}]$ as input instead of only the context word $c_{k,j}$. The intended effect is that when a question shifts topic or revisits an earlier span, the changed hidden states mark the parts of the context relevant to the current answer. The claim is supported by consistent gains over FlowQA on CoQA and QuAC, by new state-of-the-art results on QuAC and on the Scene and Tangrams domains of SCONE, and by ablations showing both the pre-prediction and inside-BERT placements of the delta contribute.

Load-bearing premise

The load-bearing premise is that the difference between two consecutive flow hidden states is a faithful measure of information gain, so the GRU learns to focus on changed context rather than on some artifact of the recurrence; the paper motivates this with a single illustrative example and selects the difference empirically over alternatives on the CoQA dev set.

Editorial extensions

If this is right

  • Explicitly modeling the turn-to-turn delta improves conversational QA over relying on the implicit FLOW state alone; FlowDeltaQA gains about 0.9 F1 over FlowQA on both CoQA and QuAC.
  • The delta can be attached to different base readers: it helps both the FlowQA architecture and BERT, suggesting it is a portable feature rather than a fix for one model.
  • On QuAC, where topics shift frequently, BERT-FlowDelta reaches higher dialogue-level HEQ-D than the compared attention-based history model, implying better whole-dialogue tracking.
  • Longer or multiple deltas (skipping two turns, or stacking two deltas) do not beat the single consecutive delta, so one step of change is the useful signal.
  • The mechanism transfers from dialogue QA to sequential instruction understanding, with state-of-the-art on two of SCONE's three domains, supporting generalization beyond span prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the delta is most valuable at topic-shift boundaries; one could weight the delta by a learned turn-level gate and see whether the benefit concentrates where the topic changes.
  • The same difference-of-states idea could be applied to any recurrent or layered reasoning model fed a sequence of related queries, such as multi-step arithmetic or interactive task execution, even outside question answering.
  • A stronger control experiment would compare the delta against concatenating the previous hidden state itself or a learned constant vector of the same shape; if those match the delta's gain, the improvement may come from extra input capacity rather than from the information-gain semantics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FlowDelta, a modification of the FLOW operation introduced in FlowQA. Instead of the original update h_{k,j} = GRU(c_{k,j}, h_{k-1,j}), Eq. (2) uses h_{k,j} = GRU([c_{k,j}; h_{k-1,j} - h_{k-2,j}], h_{k-1,j}), arguing that the difference of consecutive hidden states encodes the information gain in dialogue reasoning. The authors also build BERT-FlowDelta, which inserts FlowDelta both inside the last BERT layer (inFlowDelta) and before the answer-span prediction (exFlowDelta). The models are evaluated on CoQA, QuAC, and the SCONE sequential instruction understanding dataset. The paper reports consistent improvements over FlowQA and BERT-Flow, state-of-the-art results on QuAC and on two SCONE domains, and includes an ablation of FlowDelta variants on CoQA dev as well as a qualitative example in Appendix D.

Significance. If the reported improvements are indeed caused by the delta signal, the contribution is valuable: it is a simple, general, and lightweight mechanism that can be plugged into different machine comprehension models, and the authors release their code. The paper also extends the FLOW idea to BERT, which is a useful direction. However, the empirical evidence as presented is not strong enough to establish the central attribution claim. The gains are small, come from single runs, are partly selected on CoQA dev, and crucially the main comparison changes the input dimensionality and therefore the parameter count. The information-gain interpretation rests on an intuitive example and is not tested against parameter-matched alternatives. With additional experiments, the paper could be a solid contribution, but in its current form the core mechanistic claim is under-supported.

major comments (3)
  1. [Section 3.1, Eq. (2), and Table 2] The central comparison changes the GRU input from c_{k,j} (dimension d) to [c_{k,j}; h_{k-1,j} - h_{k-2,j}] (dimension 2d), so FlowDeltaQA and BERT-FlowDelta have strictly more parameters in the input projection than FlowQA and BERT-Flow. The reported gains (CoQA 76.7 to 77.6 in Table 1; QuAC 64.3 to 66.1 in Table 2) are therefore not uniquely attributable to the delta signal. A parameter-matched control is needed: for example, concatenating h_{k-1,j} itself (which has the same dimension as the delta) or a fixed random vector, while keeping the total parameter count equal. Without such a control, the claim that the improvement comes from the information-gain semantics of the difference is not established.
  2. [Appendix C and Table 1] The variant selection is performed on CoQA dev among FlowQA and four FlowDelta variants (SkipDelta, DoubleDelta, Hadamard product, and the proposed delta), with FlowDelta at 77.6 versus Hadamard product at 77.2 and SkipDelta at 76.9. All numbers appear to come from single runs, and no significance tests or variance estimates are reported anywhere in the paper. A 0.4 F1 margin over the Hadamard product is within typical run-to-run noise for this class of models, so the conclusion that delta is the best information-gain encoding is not supported. The authors should report multiple seeds with means and standard deviations, and should clarify whether the QuAC and SCONE results use the same dev-selected configuration or independent tuning.
  3. [Table 3 and Appendix B] The SCONE experiments tune the hidden size per domain (50, 60, 70 for Scene, Alchemy, and Tangrams) for FlowDeltaQA, while no equivalent per-domain tuning is reported for the FlowQA baseline; this makes the comparison difficult to interpret. Moreover, the Alchemy result drops from 76.4 (FlowQA) to 76.1 (FlowDeltaQA), so the paper's claim that FlowDelta "consistently improves" performance on SCONE is not supported by the table. The explanation in Section 4.2 that the Alchemy domain relies less on dialogue history may be plausible, but it is post hoc. Please specify the exact tuning protocol for both models, report error bars, and present the Alchemy decline as a qualification rather than explaining it away.
minor comments (5)
  1. [Section 3.1, Eq. (2)] Equation (2) uses h_{k-2,j}, but the initial conditions for k=1 and k=2 are not defined; the paper should specify h_0 and h_{-1} or state that zero vectors are used.
  2. [Throughout] There are numerous typos that should be corrected: "Figuire" (Section 2), "benefitial" and "Transfomer" (Section 3.2), "instrurctions" and "Quesition" (Appendix A), and "predicitons" (Appendix D).
  3. [Section 4.2] The statement that FlowDelta "introduced few additional parameters" should be quantified with actual parameter counts for FlowQA versus FlowDeltaQA and for BERT-Flow versus BERT-FlowDelta.
  4. [Section 4.2] The claim of outperforming published models on the QuAC leaderboard is tied to "Apr 24, 2019"; because leaderboards change, the snapshot date should appear in the main text or a footnote, not only implicitly in the prose.
  5. [Appendix C] The sentence "Hadamard product outperforms SkipDelta and DoubleDelta and proves its effectiveness" uses "proves" too strongly for a single ablation; a weaker formulation such as "suggests" would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FlowDelta mechanism is an architectural input modification evaluated on external benchmarks, not a fitted parameter or self-citation-derived prediction.

full rationale

The paper's central claim is that explicitly concatenating h_{k-1,j} - h_{k-2,j} into the FLOW GRU update (Eq. 2) improves conversational machine comprehension. This is an architectural modification: the delta term is a fixed function of the model's own previous hidden states, not a parameter fitted to the target datasets. The mechanism is evaluated on external benchmarks (CoQA, QuAC, SCONE) against the prior FlowQA and BERT-Flow baselines, with ablations in Table 2 and variant comparisons in Appendix C. The choice of subtraction over Hadamard product, SkipDelta, and DoubleDelta is made empirically on CoQA dev, which is standard model selection rather than circularity; the selected variant is then applied to held-out tasks. The paper does not derive its improvement from its own prior results, invoke a self-authored uniqueness theorem, or rename a known empirical pattern as a new contribution. The assumption that hidden-state differences indicate information gain is heuristic and could be questioned on grounds of parameter-matched controls or generalizability, but that is a correctness or experimental-design concern, not a circularity concern. There is no equation in which the predicted output reduces by construction to a fitted input, and no load-bearing self-citation chain. Therefore the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The ledger shows the main fitted values (SCONE hidden sizes) and the implicit assumptions about baseline reproduction and the semantic value of hidden-state deltas. No new entities are introduced.

free parameters (3)
  • SCONE hidden size for Scene = 50
    Tuned on the small SCONE dataset (Appendix B) to prevent overfitting.
  • SCONE hidden size for Alchemy = 60
    Tuned per domain, not shared.
  • SCONE hidden size for Tangrams = 70
    Tuned per domain.
assumptions (3)
  • domain assumption The FlowQA baseline numbers used for comparison are accurate and reproduced from the released code (except SCONE).
    The paper relies on these numbers to demonstrate improvement; Appendix B states SCONE numbers are not from the official released code.
  • domain assumption The reduction of SCONE to machine comprehension used in FlowQA is valid and correctly implemented.
    The paper follows Huang et al.'s encoding of world states as integers and logical forms as output spans (Appendix A).
  • ad hoc to paper The hidden-state difference h_{k-1,j} - h_{k-2,j} is a useful signal for information gain in dialogue reasoning.
    This is the core heuristic of Eq. 2, justified only by intuition and one qualitative example.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension." pith.science (2026). https://pith.science/paper/ZRT3KEHN

@misc{pith2026190805117,
  author       = {Pith},
  title        = {Pith review of: FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRT3KEHN}},
  note         = {Machine review of arXiv:1908.05117}
}
read the original abstract

Conversational machine comprehension requires deep understanding of the dialogue flow, and the prior work proposed FlowQA to implicitly model the context representations in reasoning for better understanding. This paper proposes to explicitly model the information gain through dialogue reasoning in order to allow the model to focus on more informative cues. The proposed model achieves state-of-the-art performance in a conversational QA dataset QuAC and sequential instruction understanding dataset SCONE, which shows the effectiveness of the proposed mechanism and demonstrates its capability of generalization to different QA models and tasks.

Figures

Figures reproduced from arXiv: 1908.05117 by the authors.

Figure 1
Figure 1. Illustration of the flow information gain mod [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed FlowDelta models. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Example of the SCONE dataset and its re [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 3 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450

    arXiv 2016

  4. [4]

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174--2184

    work page 2018

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  6. [6]

    Daniel Fried, Jacob Andreas, and Dan Klein. 2017. Unified pragmatic models for generating and following instructions. arXiv preprint arXiv:1711.04987

    arXiv 2017

  7. [7]

    Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. arXiv preprint arXiv:1704.07926

    arXiv 2017

  8. [8]

    Hsin-Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2018. Flowqa: Grasping flow in history for conversational machine comprehension. arXiv preprint arXiv:1810.06683

    arXiv 2018

Show all 20 references
  1. [9]

    Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. 2017. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. arXiv preprint arXiv:1711.07341

    2017 arXiv

  2. [10]

    Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. arXiv preprint arXiv:1606.05378

    2016 arXiv

  3. [11]

    Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W Bruce Croft, and Mohit Iyyer. 2019. Attentive history selection for conversational question answering. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1391--1400

    2019

  4. [12]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822

    2018 arXiv

  5. [13]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

    2016 arXiv

  6. [14]

    Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042

    2018 arXiv

  7. [15]

    Asa Cooper Stickland and Iain Murray. 2019. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. arXiv preprint arXiv:1902.02671

    2019 arXiv

  8. [16]

    Alane Suhr and Yoav Artzi. 2018. Situated mapping of sequential instructions to actions with single-step reward observation. arXiv preprint arXiv:1805.10209

    2018 arXiv

  9. [17]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  10. [18]

    Mark Yatskar. 2018. A qualitative comparison of coqa, squad 2.0 and quac. arXiv preprint arXiv:1809.10735

    2018 arXiv

  11. [19]

    Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541

    2018 arXiv

  12. [20]

    Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.