{"id":"445bb04f-468d-4c9e-872f-f62489bf4529","arxiv_id":"1908.05117","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Modeling the difference between consecutive reasoning states, called FlowDelta, improves conversational machine comprehension accuracy across FlowQA and BERT on CoQA, QuAC, and SCONE.","lead":"This paper adds a simple mechanism to conversational question-answering models that measures how the model's internal reasoning changes from one question to the next, and uses that change to focus on important context. It reports small but consistent accuracy gains on three benchmarks, including a state-of-the-art result at the time on the QuAC dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may reflect added parameters or dev-set variant selection rather than the delta signal itself; Eq. 2 lacks a parameter-matched control.","rationale":"The paper's contribution rests on the causal claim that the delta vector is the operative ingredient. The experiments compare against models with smaller GRU inputs, conflating the new signal with a change in model capacity. The variant study in Appendix C does select among alternatives, but on a single dev set and with differences of a few tenths of F1. This leaves open the alternative explanation that any extra input (or simply more parameters) would produce similar gains. The concern is not that the method is wrong, but that the evidence as presented does not isolate the proposed mechanism. A matched-parameter control with multiple seeds would settle this cheaply. If the control also improves, then the specific 'information gain' interpretation is unsupported, though the method might still be a useful engineering trick. Thus the reader's CONDITIONAL verdict is appropriate; no change in verdict is needed, but the required condition should include a parameter-matched baseline.","tokens_in":7471,"tokens_out":4946,"duration_ms":50980,"concrete_test":"Run on CoQA and QuAC a matched-parameter control: replace the concatenated vector in Eq. 2 with h_{k-1,j} (same concatenated dimension) instead of h_{k-1,j} - h_{k-2,j}, keeping all hyperparameters identical. Train both the delta variant and this control with at least 5 random seeds each, and report mean F1 with 95% confidence intervals. If the delta variant does not exceed the control by more than the confidence interval width, the specific claim that the difference encodes information gain is unsupported, and the gains are explained by extra capacity or generic history features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that explicitly feeding h_{k-1,j} - h_{k-2,j} into the FLOW GRU (Eq. 2) captures information gain and causes the observed improvements. The evidence is an ablation against FlowQA and BERT-Flow, but those baselines have smaller GRU input dimensions and therefore fewer parameters. The input changes from c_{k,j} (dim d) to [c_{k,j}; h_{k-1,j} - h_{k-2,j}] (dim 2d), which alters the input projection and adds parameters. Without a control that adds the same number of parameters using a nonsemantic or alternative vector (e.g., h_{k-1,j} itself or a random vector), the improvement cannot be attributed to the delta's information-gain semantics. This is compounded by the variant selection in Appendix C: delta was chosen as best among five on CoQA dev (range 76.7-77.6), then reported on QuAC and SCONE; the margin over the Hadamard product (77.2) is 0.4 F1, a small single-run difference. Thus the load-bearing assumption is not only that hidden-state differences are informative, but that they are more informative than equally expressive alternatives; this is untested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlowDelta, a modification of the FLOW operation introduced in FlowQA. Instead of the original update h_{k,j} = GRU(c_{k,j}, h_{k-1,j}), Eq. (2) uses h_{k,j} = GRU([c_{k,j}; h_{k-1,j} - h_{k-2,j}], h_{k-1,j}), arguing that the difference of consecutive hidden states encodes the information gain in dialogue reasoning. The authors also build BERT-FlowDelta, which inserts FlowDelta both inside the last BERT layer (inFlowDelta) and before the answer-span prediction (exFlowDelta). The models are evaluated on CoQA, QuAC, and the SCONE sequential instruction understanding dataset. The paper reports consistent improvements over FlowQA and BERT-Flow, state-of-the-art results on QuAC and on two SCONE domains, and includes an ablation of FlowDelta variants on CoQA dev as well as a qualitative example in Appendix D.","tokens_in":7743,"tokens_out":4475,"duration_ms":42106,"significance":"If the reported improvements are indeed caused by the delta signal, the contribution is valuable: it is a simple, general, and lightweight mechanism that can be plugged into different machine comprehension models, and the authors release their code. The paper also extends the FLOW idea to BERT, which is a useful direction. However, the empirical evidence as presented is not strong enough to establish the central attribution claim. The gains are small, come from single runs, are partly selected on CoQA dev, and crucially the main comparison changes the input dimensionality and therefore the parameter count. The information-gain interpretation rests on an intuitive example and is not tested against parameter-matched alternatives. With additional experiments, the paper could be a solid contribution, but in its current form the core mechanistic claim is under-supported.","major_comments":[{"comment":"The central comparison changes the GRU input from c_{k,j} (dimension d) to [c_{k,j}; h_{k-1,j} - h_{k-2,j}] (dimension 2d), so FlowDeltaQA and BERT-FlowDelta have strictly more parameters in the input projection than FlowQA and BERT-Flow. The reported gains (CoQA 76.7 to 77.6 in Table 1; QuAC 64.3 to 66.1 in Table 2) are therefore not uniquely attributable to the delta signal. A parameter-matched control is needed: for example, concatenating h_{k-1,j} itself (which has the same dimension as the delta) or a fixed random vector, while keeping the total parameter count equal. Without such a control, the claim that the improvement comes from the information-gain semantics of the difference is not established.","section":"Section 3.1, Eq. (2), and Table 2"},{"comment":"The variant selection is performed on CoQA dev among FlowQA and four FlowDelta variants (SkipDelta, DoubleDelta, Hadamard product, and the proposed delta), with FlowDelta at 77.6 versus Hadamard product at 77.2 and SkipDelta at 76.9. All numbers appear to come from single runs, and no significance tests or variance estimates are reported anywhere in the paper. A 0.4 F1 margin over the Hadamard product is within typical run-to-run noise for this class of models, so the conclusion that delta is the best information-gain encoding is not supported. The authors should report multiple seeds with means and standard deviations, and should clarify whether the QuAC and SCONE results use the same dev-selected configuration or independent tuning.","section":"Appendix C and Table 1"},{"comment":"The SCONE experiments tune the hidden size per domain (50, 60, 70 for Scene, Alchemy, and Tangrams) for FlowDeltaQA, while no equivalent per-domain tuning is reported for the FlowQA baseline; this makes the comparison difficult to interpret. Moreover, the Alchemy result drops from 76.4 (FlowQA) to 76.1 (FlowDeltaQA), so the paper's claim that FlowDelta \"consistently improves\" performance on SCONE is not supported by the table. The explanation in Section 4.2 that the Alchemy domain relies less on dialogue history may be plausible, but it is post hoc. Please specify the exact tuning protocol for both models, report error bars, and present the Alchemy decline as a qualification rather than explaining it away.","section":"Table 3 and Appendix B"}],"minor_comments":[{"comment":"Equation (2) uses h_{k-2,j}, but the initial conditions for k=1 and k=2 are not defined; the paper should specify h_0 and h_{-1} or state that zero vectors are used.","section":"Section 3.1, Eq. (2)"},{"comment":"There are numerous typos that should be corrected: \"Figuire\" (Section 2), \"benefitial\" and \"Transfomer\" (Section 3.2), \"instrurctions\" and \"Quesition\" (Appendix A), and \"predicitons\" (Appendix D).","section":"Throughout"},{"comment":"The statement that FlowDelta \"introduced few additional parameters\" should be quantified with actual parameter counts for FlowQA versus FlowDeltaQA and for BERT-Flow versus BERT-FlowDelta.","section":"Section 4.2"},{"comment":"The claim of outperforming published models on the QuAC leaderboard is tied to \"Apr 24, 2019\"; because leaderboards change, the snapshot date should appear in the main text or a footnote, not only implicitly in the prose.","section":"Section 4.2"},{"comment":"The sentence \"Hadamard product outperforms SkipDelta and DoubleDelta and proves its effectiveness\" uses \"proves\" too strongly for a single ablation; a weaker formulation such as \"suggests\" would be more appropriate.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward extension of FlowQA, and the core idea is plausible. The main issue is that the current experiments do not yet isolate the delta mechanism from added parameters, and the small reported gains are not accompanied by significance tests or multiple runs. The SCONE tuning asymmetry and the Alchemy drop further weaken the consistency claim. I see no circularity or unsupported invented entities; the paper is an honest empirical study. If the authors provide parameter-matched controls, seed variance, and a clarified SCONE protocol, I would be willing to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FlowDelta is a modest extension of FlowQA that feeds the previous hidden-state difference into the FLOW GRU. It works about as well as claimed on three datasets, but the evidence does not isolate the delta semantics from added parameters and dev-set selection. I would send it to review; the mechanism is simple enough to be useful regardless.\n\nWhat is new: Eq. 2 is not in FlowQA. Concatenating h_{k-1,j} - h_{k-2,j} to the GRU input is a small idea, and the paper does not oversell it. It releases code, evaluates on CoQA, QuAC, and SCONE, and compares against FlowQA and a BERT-Flow baseline. The gains are consistent: +0.9 F1 on CoQA and QuAC for FlowDeltaQA over FlowQA, and BERT-FlowDelta improves from 79.2 to 79.4 on CoQA and from 64.3 to 66.1 on QuAC. On SCONE, two of three domains improve. That is a decent empirical contribution.\n\nWhere it is soft: the stress-test concern is fair. The baseline GRU input is c_{k,j} (dim d); Eq. 2 uses [c_{k,j}; h_{k-1,j}-h_{k-2,j}] (dim 2d). That changes the input projection and adds parameters. Without a control that adds the same number of parameters using a nonsemantic vector (say h_{k-1,j} itself or a random vector), the improvement cannot be attributed to the information-gain semantics. Appendix C also selects delta among five variants on CoQA dev; the margin over the Hadamard product is 0.4 F1. The chosen variant is then reported on QuAC and SCONE with no significance tests or multiple runs. Per-domain hidden size tuning on SCONE adds another degree of freedom. None of this kills the paper; it means the headline number is fragile and the mechanism's edge is plausible but unproven.\n\nThe qualitative example in Appendix D is a single example, fine for illustration only. The claim that the difference indicates information gain is intuitive, not derived; I do not view that as a flaw, just a limit.\n\nBottom line: this is a useful plug-in for people building multi-turn QA models. It deserves a serious referee; the experiments should be strengthened with parameter-matched ablations and variance estimates. I would not cite it as evidence for the information-gain story, but I might cite Eq. 2 as an empirically useful trick.","headline":"A simple, honest extension of FlowQA with consistent but weakly isolated gains; worth reviewing, and the ablation needs a parameter-matched control.","tokens_in":8197,"tokens_out":1909,"would_cite":false,"duration_ms":18174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A one-line change to the flow update—feeding in the difference between consecutive reasoning states—improves conversational machine comprehension, with state-of-the-art results on QuAC and on the Scene and Tangrams domains of SCONE.","keywords":["conversational machine comprehension","dialogue flow","information gain","FlowDelta","FlowQA","BERT integration","QuAC","SCONE"],"falsifier":"Train the same BERT-FlowDelta model with the delta term replaced by a random vector of identical shape, or by the previous hidden state itself, while keeping all other hyperparameters fixed; the claim that the difference encodes information gain predicts a clear drop in QuAC and CoQA F1, and if performance is unchanged, the gains are not caused by the delta as an information-gain signal.","tokens_in":7264,"feed_emoji":"💬","tokens_out":6645,"duration_ms":60084,"temperature":0.7,"pith_summary":"The paper tries to show that conversational question answering improves when a model is told, not just implicitly shown, what changed in its reasoning between dialogue turns. Its FlowDelta mechanism feeds the difference between the two previous flow-state vectors into each GRU update, so the network can key in on context spans whose representation shifted as the topic changed. On the QuAC and CoQA dialogue datasets and the SCONE instruction-following task, this simple extra signal raises performance over the base FlowQA model and, combined with BERT, gives state-of-the-art published numbers on QuAC and on two of SCONE's three domains. The point is that reasoning history can be used more effectively by making the information gain explicit rather than relying on the hidden state to encode it.","feed_headline":"Tracking the change between reasoning states lifts conversational QA","feed_subtitle":"Concatenating the change in reasoning states helps models follow conversation flow and beat prior published results.","key_machinery":"The central object is the FlowDelta operation, a one-line modification of the FLOW recurrence: $h_{k,j} = \\mathrm{GRU}([c_{k,j}; h_{k-1,j} - h_{k-2,j}], h_{k-1,j})$. Here $j$ indexes context positions and $k$ indexes dialogue turns, so the input concatenates the current context-word representation with the change in the hidden state from two turns ago to the last turn. That difference is the information-gain estimate: spans whose representations moved a lot between turns are the ones the current question likely needs. It is inserted in two places in the BERT variant: before the final span-prediction layer and inside the last BERT layer. The mechanism carries the argument because all reported improvements come from adding this delta, with only a small increase in GRU input dimension.","core_discovery":"The central claim is that, in multi-turn reasoning over a passage, the vector difference $h_{k-1,j} - h_{k-2,j}$ between the flow states of consecutive turns is a usable signal of information gain. The paper modifies the FLOW operation so that the GRU at turn $k$ receives $[c_{k,j}; h_{k-1,j} - h_{k-2,j}]$ as input instead of only the context word $c_{k,j}$. The intended effect is that when a question shifts topic or revisits an earlier span, the changed hidden states mark the parts of the context relevant to the current answer. The claim is supported by consistent gains over FlowQA on CoQA and QuAC, by new state-of-the-art results on QuAC and on the Scene and Tangrams domains of SCONE, and by ablations showing both the pre-prediction and inside-BERT placements of the delta contribute.","pith_inferences":["The paper does not test whether the delta is most valuable at topic-shift boundaries; one could weight the delta by a learned turn-level gate and see whether the benefit concentrates where the topic changes.","The same difference-of-states idea could be applied to any recurrent or layered reasoning model fed a sequence of related queries, such as multi-step arithmetic or interactive task execution, even outside question answering.","A stronger control experiment would compare the delta against concatenating the previous hidden state itself or a learned constant vector of the same shape; if those match the delta's gain, the improvement may come from extra input capacity rather than from the information-gain semantics."],"forward_implications":["Explicitly modeling the turn-to-turn delta improves conversational QA over relying on the implicit FLOW state alone; FlowDeltaQA gains about 0.9 F1 over FlowQA on both CoQA and QuAC.","The delta can be attached to different base readers: it helps both the FlowQA architecture and BERT, suggesting it is a portable feature rather than a fix for one model.","On QuAC, where topics shift frequently, BERT-FlowDelta reaches higher dialogue-level HEQ-D than the compared attention-based history model, implying better whole-dialogue tracking.","Longer or multiple deltas (skipping two turns, or stacking two deltas) do not beat the single consecutive delta, so one step of change is the useful signal.","The mechanism transfers from dialogue QA to sequential instruction understanding, with state-of-the-art on two of SCONE's three domains, supporting generalization beyond span prediction."],"supporting_citations":[{"why":"Supplies the FLOW operation and the FlowQA baseline that FlowDelta modifies, and the reproduction numbers it must beat.","marker":"(Huang et al., 2018)"},{"why":"Provides the pretrained BERT model whose layers the FlowDelta mechanism is inserted into for the BERT variant.","marker":"(Devlin et al., 2018)"},{"why":"Defines the QuAC dataset and its HEQ evaluation metrics used to measure the state-of-the-art claim.","marker":"(Choi et al., 2018)"},{"why":"Defines the CoQA dataset used for the main F1 comparisons and the variant ablation.","marker":"(Reddy et al., 2018)"},{"why":"Introduces the SCONE sequential-instruction tasks and supplies the reduction to machine comprehension that FlowDeltaQA is evaluated on.","marker":"(Long et al., 2016)"},{"why":"Provides the attention-based history-selection model that is the closest published comparison on QuAC and is outperformed on HEQ-D.","marker":"(Qu et al., 2019)"}],"fun_headline_variants":["FlowDelta: explicit reasoning-delta signal for conversational QA","Turn-to-turn state difference lifts conversational comprehension","Modeling information gain in dialogue reasoning sets new SOTA","Attention to flow change improves QA on QuAC and SCONE","Delta between reasoning states sharpens multi-turn QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the difference between two consecutive flow hidden states is a faithful measure of information gain, so the GRU learns to focus on changed context rather than on some artifact of the recurrence; the paper motivates this with a single illustrative example and selects the difference empirically over alternatives on the CoQA dev set.","fun_headline_variants_meta":{"raw":{"variants":["FlowDelta: explicit reasoning-delta signal for conversational QA","Turn-to-turn state difference lifts conversational comprehension","Modeling information gain in dialogue reasoning sets new SOTA","Attention to flow change improves QA on QuAC and SCONE","Delta between reasoning states sharpens multi-turn QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2753,"prompt_tokens":803,"completion_tokens":1950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":1873}},"tokens_in":419,"tokens_out":1950,"duration_ms":15315,"temperature":1.0,"reasoning_tokens":1873,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:21:56.132028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same BERT-FlowDelta model with the delta term replaced by a random vector of identical shape, or by the previous hidden state itself, while keeping all other hyperparameters fixed; the claim that the difference encodes information gain predicts a clear drop in QuAC and CoQA F1, and if performance is unchanged, the gains are not caused by the delta as an information-gain signal.","supporting_citations":[],"review_version":1}