{"id":"170228e3-4418-40dd-a566-4f36db823302","arxiv_id":"1908.11787","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"By encoding tables as graphs and selecting answer cells directly with a pointer network, a transformer-style GNN reaches 55.1% accuracy on the SQA conversational table QA benchmark, beating the prior 45.6% state of the art.","lead":"This paper describes a graph-based Transformer model that answers follow-up questions about tables by pointing directly at table cells, skipping logical forms. On the SQA benchmark it raises question accuracy from 45.6% to 55.1%, with the largest gains on follow-up questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conversational context mechanism only marks previous answers, not the earlier question itself; follow-ups that need non-answer content from q1 have no representational path, and the 8% context-error figure rests on a 100-example manual sample.","rationale":"The paper's strongest claim is the empirical SOTA improvement, and the reported numbers (55.1 vs 45.6 ALL) are large. The architecture is described in enough detail to reimplement, and the ablations (no-context 45.1, context 55.1, oracle 61.7) are internally consistent. However, the contribution is framed as a general way to handle conversational references without logical forms, and that framing depends on the context encoding in Section 2.1. The explicit statement that q1's content is not encoded makes the answer-only premise the central architectural assumption. The error analysis is too small to establish that this premise holds broadly, and the architecture provides no mechanism for a follow-up that needs the previous question's wording or the excluded part of a condition. This does not invalidate the SQA numbers, but it does make the general conversational claim conditional. The reader's CONDITIONAL verdict captures this; I would not move to ACCEPT without either code/variance or a direct test of the context premise.","tokens_in":9071,"tokens_out":9232,"duration_ms":88147,"concrete_test":"Build a diagnostic subset from SQA consisting of follow-up questions whose gold answer cannot be derived from the previous answer cells alone—e.g., the CONTEXT example in Appendix Table 4 and similar anaphoric cases—and evaluate the trained model on it, comparing against a matched control subset of answer-set-dependent follow-ups. If accuracy on the diagnostic subset is at or near chance while the control accuracy is high, the answer-only premise fails. A complementary check is to add a node encoding the full previous question text and measure the gain on the same subset; a large gain would confirm the missing representation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the model 'handles conversational context effectively' (Section 6) rests on the context mechanism in Section 2.1: after each turn, the graph marks answer rows, columns, and cells with nominal features. The paper states explicitly: 'the content of q1 is not encoded in the graph, only its answers.' This is sufficient only if every relevant follow-up can be resolved from the previous answer set plus the current question. SQA may mostly satisfy this, but the architecture has no representational fallback for cases such as the appendix's CONTEXT example: 'when was thaddeus bell born?, when was klaus jurgen schneider born?, which is older? The correct birthday is selected, but not the person' (Table 4). Here 'which' refers to entities from the earlier questions, and the model cannot bind the anaphor to the intended persons because only the answer dates are marked, not the question semantics. The paper estimates only 8% of follow-up errors are context-related, but this is from 100 manually annotated examples (Table 3b), so the estimate is noisy and there is no analysis of whether those 8 examples are exactly the ones the architecture cannot represent. The ablation OURS* vs OURS† (45.1 vs 55.1) shows the context features help, but does not show that answer-only features are sufficient; it only shows they are better than no context. If the premise is false outside a narrow subset of SQA, the claim of a general approach to conversational QA without logical forms is overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a neural model for answering sequential questions over tables without producing an intermediate logical form. Tables are encoded as graphs whose nodes are columns, rows, and cells, with edge labels for structure, lexical alignments, and numerical comparisons. A Transformer-based graph encoder computes node representations, and a pointer-network decoder selects answer cells directly. Conversation context is handled by marking the rows, columns, and cells that were the answers to the previous question. On the SQA dataset, the model is reported to improve state-of-the-art question accuracy from 45.6% (CAMP) to 55.1%, with a particularly large gain on follow-up questions (POS3 46.8% vs. 24.8%). The paper includes ablations for the numerical relations and the context mechanism, an oracle experiment quantifying error propagation, and a manual error analysis on 200 examples.","tokens_in":9404,"tokens_out":6801,"duration_ms":61616,"significance":"The paper's central empirical claim is strong and important if it holds: it suggests that conversational table QA can be solved by direct answer-cell selection with graph-structured inputs, bypassing logical forms and their supervision costs. The architecture is clearly described and the ablation of the context features (45.1 vs. 55.1) as well as the oracle-answer experiment (RA 61.7) are valuable for interpreting where the gains come from. The treatment of numeric features (rank, inverse rank, comparison edges) is a useful component and is supported by a targeted ablation. The manual error analysis, while small, gives a tangible picture of residual failure modes. The main risks are the scope of the conversational-context claim given that only prior answers (not prior questions) are encoded, and the absence of any variance reporting for the headline numbers.","major_comments":[{"comment":"The context mechanism encodes only the answers to the previous question and not the question itself, as stated in Section 2.1: 'the content of q1 is not encoded in the graph, only its answers.' This is a representational limitation for follow-up questions that need information from the earlier question beyond the answer set, such as anaphoric references to entities mentioned in q1. The CONTEXT example in Table 4 illustrates exactly this failure: for 'when was thaddeus bell born?, when was klaus jurgen schneider born?, which is older?' the model selects the correct birthday but not the person, because the persons are not in the previous answer set. The claim in Section 6 that the model 'particularly handles conversational context effectively' is therefore supported only for a restricted class of context dependence, and the 8% CONTEXT error rate in Table 3b rests on 100 manually annotated follow-ups, giving a wide confidence interval and no analysis of whether those errors are precisely the ones this architecture cannot represent. Please either provide an estimate of how many SQA follow-ups require non-answer content from previous questions, or qualify the conversational-context claim to reflect the answer-only context assumption.","section":"Section 2.1 (Context paragraph), Table 4, Section 6"},{"comment":"No variance, confidence intervals, or per-run values are reported for any of the model's numbers, despite the statement that all model numbers are averaged over 5 independent runs. The central empirical claim—an improvement in ALL from 45.6% (CAMP) to 55.1% and in SEQ from 13.2% to 28.1%—is a comparison of point estimates without a measure of dispersion or a significance test. This makes it impossible to assess whether the reported gains are robust to random initialization, which matters particularly for the smaller differences (e.g., POS1: 67.2 vs. DYNSP† 70.4). Please report per-run results or confidence intervals, or at least the standard deviation over the 5 runs, so the reader can evaluate the reliability of the state-of-the-art claim.","section":"Section 5, Table 1; Section 4, 'All numbers given for our model are averaged over 5 independent runs'"}],"minor_comments":[{"comment":"The paper states that hyperparameters are tuned with Google Vizier but does not describe the validation split used for model selection or how many times the test set was accessed. Please clarify the validation procedure so the reader can rule out test-set overfitting.","section":"Section 4 (Experimental Setup)"},{"comment":"The manual error analysis is based on 100 initial and 100 follow-up questions with no reported inter-annotator agreement. A brief description of the annotation protocol and a measure of agreement would increase confidence in the error-category proportions, especially the 8% CONTEXT figure.","section":"Appendix B (Error Analysis and Table 3)"},{"comment":"Cells with identical text within a column are collapsed into a single node. This design choice may lose the distinction between multiple rows with the same value, which appears to be related to the ANSWER SET error type reported in Table 3. It would be helpful to discuss how the model or post-processing recovers the intended answer set in such cases.","section":"Section 2.1 (Graph Formulation)"},{"comment":"The paper says the model is 'not sensitive to table size changes' based on Figure 3, but the figure only shows a scatter plot without a quantitative correlation or trend line. Adding a simple correlation coefficient or a comparison of accuracy on the largest vs. smallest tables would make the claim more precise.","section":"Section 5 (Results)"},{"comment":"The reference list contains duplicate entries for Neelakantan et al. (2016a and 2016b) with identical titles; one is presumably the arXiv version and one the published version, but they should be distinguished properly.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its core approach and the reported empirical gains are plausible, but the conversational-context claim is broader than the architecture supports, and the lack of variance reporting is a reproducibility concern for a state-of-the-art claim. These issues are addressable in revision. I would also suggest the editor consider the journal's policy on code/artifact release, since no code is provided and some experimental details (validation split) are omitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a clear architectural idea—skip logical forms and point at table cells via a graph encoder—and it backs that idea with a large gain on SQA (55.1 vs. 45.6). The main caveat is that the conversational context mechanism is narrower than the title implies: it only marks previous answers, not the earlier questions, and the paper says so explicitly. That is an honest limitation, but it means the 'conversational' claim is mostly about follow-ups that can be resolved from the previous answer set.\n\nWhat's new: encoding the table as a graph with row, column, and cell nodes, using a Transformer-style GNN, and decoding with a pointer network that selects cells directly. The numeric relations (rank, comparisons) are a practical addition, and the ablation shows they help (51.5 without vs. 55.1 with). Context marking gives a bigger jump, from 45.1 to 55.1. The evaluation is on one benchmark, SQA, but that is the standard one for this task, and the comparison set (CAMP, DYNSP, NP, FP) is the right one.\n\nSoft spots: no code, no error bars despite averaging over five runs, and the manual error analysis covers only 200 examples. The stress-test point about anaphora is fair: in the 'when was X born... which is older?' case, the model cannot bind 'which' to the persons because only the dates are marked, not the entities from the earlier question. The paper's own 8% context-error figure comes from 100 follow-up questions, so treat it as a rough estimate. These issues are real but not disqualifying. The architecture is simple, the paper is honest about what it does not encode, and the gain over strong baselines is large enough that it would likely survive added variance.\n\nFor whom: anyone working on table QA or conversational semantic parsing who wants a practical baseline that avoids logical-form supervision. It deserves a serious referee; the main requests would be code/checkpoints, variance, and a larger error analysis that specifically tests context failures.\n\nI would engage with it. It is not a field revolution, but it is a useful result and an unusually direct approach.","headline":"A solid, honest table-QA paper: direct cell selection beats logical-form baselines on SQA, but the conversational context mechanism is answer-only, which is a real limitation.","tokens_in":9914,"tokens_out":1841,"would_cite":true,"duration_ms":17017,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer graph encoder turns a table into nodes for rows, columns and cells, and a pointer network answers a conversation by selecting cells directly, beating prior systems on the Sequential Question Answering benchmark without ever…","keywords":["conversational question answering","table question answering","graph neural network","pointer network","logical form free","sequential question answering","SQA","Transformer"],"falsifier":"On the SQA test set, isolate follow-up questions whose correct answer requires a number, entity, or condition that appears only in the first question's text and not in its answer cells; if a human can answer them and the model's accuracy on such a subset is near chance, the answer-marking context mechanism is insufficient.","tokens_in":8892,"feed_emoji":"📊","tokens_out":4547,"duration_ms":41451,"temperature":0.7,"pith_summary":"The paper argues that conversational questions over tables can be answered directly by selecting cells from an encoded graph, with no logical-form intermediate. It encodes the table as a graph of column, row, and cell nodes, lets a Transformer-style graph network read the graph together with the current question, and uses a pointer network to pick answer cells; conversational context is supplied by marking the previous answer's rows, columns and cells. On the Sequential Question Answering (SQA) dataset the model reaches 55.1% overall question accuracy, up from the previous state of the art of 45.6%, and it roughly doubles the accuracy on third-turn follow-up questions. If this stands, it makes conversational table QA trainable directly from question-answer pairs, without semantic parsers, manually defined operations, or reinforcement learning.","feed_headline":"Pointer network beats logical-form systems on table conversations","feed_subtitle":"On SQA, accuracy rises from 45.6% to 55.1%, with the largest gains on follow-up turns.","key_machinery":"The central object is a table-to-graph encoding: columns, rows, cells, question nodes, and question tokens are all graph nodes, with directed edges between columns/rows and cells, edit-distance alignments between question n-grams and table text, and comparison relations between numeric expressions in the question and numeric cells. Numerical cells carry rank and inverse-rank features within their column, and the previous answer is encoded by adding ANSWER ROW, ANSWER COLUMN, and ANSWER CELL features to the relevant nodes. A Transformer-based graph neural network encoder, using Shaw et al.'s additive edge-vector self-attention, turns this graph into contextualized node representations, and a pointer-network copy mechanism in the decoder selects answer values directly from table-cell nodes.","core_discovery":"The central claim is that sequential questions about a table can be answered by direct cell selection from a graph-encoded table, and that this outperforms existing logical-form-based approaches on SQA. The model improves the state of the art from 45.6% (CAMP) to 55.1% in overall question accuracy, a relative error reduction of 18%, and its largest gains are on follow-up questions: third-turn accuracy reaches 46.8%, compared with 25.9% for the previous best contextual parser. Providing the previous reference answer at test time pushes overall accuracy to 61.7%, indicating a 6.6-point loss from error propagation, while the no-context variant reaches only 45.1%, showing that the answer-marking mechanism contributes about 10 points.","pith_inferences":["Inference beyond the paper: if answer-marking is the only context channel, the model should fail on follow-ups whose referent is a term in the earlier question's wording rather than in its answer set; constructing a synthetic set of such questions would expose this boundary empirically.","Inference beyond the paper: because first-question accuracy trails the best prior parser by about 3.7 points, combining the graph encoder with pretrained language representations -- the paper names BERT as future work -- should close that gap while preserving the logical-form-free design.","Inference beyond the paper: the same graph-plus-pointer scheme maps naturally to knowledge graphs by treating entities and relations as nodes, which would extend the answer-marking context mechanism beyond tables.","Inference beyond the paper: the 8% context-error estimate comes from only 100 manually annotated follow-up questions, so the true ceiling of the answer-marking mechanism is not tightly measured; a larger error study would give a more reliable bound."],"forward_implications":["Conversational QA over tables can be trained from question-answer pairs alone, removing the need for annotated logical forms or SQL programs.","Follow-up question accuracy, the hardest part of conversational QA, improves substantially -- 46.8% versus 25.9% on third-turn questions -- suggesting that answer-marking is often sufficient conversational context.","Numeric comparisons and superlatives are handled without explicit programs: removing the numeric relations and rank features drops accuracy from 55.1% to 51.5%.","The non-autoregressive context mechanism is simple enough to be combined with other answer-selection architectures, since it only adds nominal features to nodes.","Error propagation is bounded: with oracle previous answers, accuracy rises to 61.7%, so roughly 6.6 points of the remaining error is due to propagating mistakes from earlier turns."],"supporting_citations":[{"why":"Supplies the SQA dataset and the DynSP baseline that searches over latent logical forms; the central benchmark and the weakest-supervision setting the paper adopts.","marker":"Iyyer et al., 2017"},{"why":"Provides the Transformer self-attention formulation with additive edge-vector representations that the graph encoder extends.","marker":"Shaw et al., 2019"},{"why":"Supplies the base Transformer architecture, including relative position representations and the learning-rate schedule, underlying the encoder and decoder.","marker":"Vaswani et al., 2017"},{"why":"Pointer networks are the mechanism used to select answer cells directly from graph nodes, replacing a fixed output vocabulary.","marker":"Vinyals et al., 2015"},{"why":"CAMP is the previous state-of-the-art contextual parser at 45.6% overall accuracy, the main baseline the paper improves upon.","marker":"Sun et al., 2018b"},{"why":"Float Parser is one of the non-contextual baselines and represents the compositional semantic-parsing approach that the paper compares against.","marker":"Pasupat and Liang, 2015"}],"fun_headline_variants":["Pointer network answers table conversations without logical forms","No logical forms needed for conversational table QA","Graph-based pointer network beats logical-form baselines on SQA","Pointer network cuts error by 18% on conversational table QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that everything a follow-up question needs from the earlier turn is contained in the previous answer's marked rows, columns and cells, because the earlier question's wording is not encoded anywhere in the graph.","fun_headline_variants_meta":{"raw":{"variants":["Pointer network answers table conversations without logical forms","No logical forms needed for conversational table QA","Graph-based pointer network beats logical-form baselines on SQA","Pointer network cuts error by 18% on conversational table QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000877,"raw_usage":{"total_tokens":3719,"prompt_tokens":796,"completion_tokens":2923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":2860}},"tokens_in":412,"tokens_out":2923,"duration_ms":19265,"temperature":1.0,"reasoning_tokens":2860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:06:34.342252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the SQA test set, isolate follow-up questions whose correct answer requires a number, entity, or condition that appears only in the first question's text and not in its answer cells; if a human can answer them and the model's accuracy on such a subset is near chance, the answer-marking context mechanism is insufficient.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Transformer self-attention formulation with additive edge-vector representations that the graph encoder extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pointer networks are the mechanism used to select answer cells directly from graph nodes, replacing a fixed output vocabulary."}],"review_version":1}