Pith. sign in

REVIEW 2 major objections 5 minor 33 references

Answering Conversational Questions on Structured Data without Logical Forms

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A transformer graph encoder turns a table into nodes for rows, columns and cells, and a pointer network answers a conversation by selecting cells directly, beating prior systems on the Sequential Question Answering benchmark without ever…

desk verdict A solid, honest table-QA paper: direct cell selection beats logical-form baselines on SQA, but the conversational context mechanism is answer-only, which is a real limitation. read the letter →

arxiv 1908.11787 v1 pith:KGTQJZDM submitted 2019-08-30 cs.CL

classification cs.CL
keywords conversationalquestionansweringtablegraphneuralnetworkpointerlogicalformfreesequentialSQATransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that conversational questions over tables can be answered directly by selecting cells from an encoded graph, with no logical-form intermediate. It encodes the table as a graph of column, row, and cell nodes, lets a Transformer-style graph network read the graph together with the current question, and uses a pointer network to pick answer cells; conversational context is supplied by marking the previous answer's rows, columns and cells. On the Sequential Question Answering (SQA) dataset the model reaches 55.1% overall question accuracy, up from the previous state of the art of 45.6%, and it roughly doubles the accuracy on third-turn follow-up questions. If this stands, it makes conversational table QA trainable directly from question-answer pairs, without semantic parsers, manually defined operations, or reinforcement learning.

What carries the argument

The central object is a table-to-graph encoding: columns, rows, cells, question nodes, and question tokens are all graph nodes, with directed edges between columns/rows and cells, edit-distance alignments between question n-grams and table text, and comparison relations between numeric expressions in the question and numeric cells. Numerical cells carry rank and inverse-rank features within their column, and the previous answer is encoded by adding ANSWER ROW, ANSWER COLUMN, and ANSWER CELL features to the relevant nodes. A Transformer-based graph neural network encoder, using Shaw et al.'s additive edge-vector self-attention, turns this graph into contextualized node representations, and a pointer-network copy mechanism in the decoder selects answer values directly from table-cell nodes.

What would settle it

On the SQA test set, isolate follow-up questions whose correct answer requires a number, entity, or condition that appears only in the first question's text and not in its answer cells; if a human can answer them and the model's accuracy on such a subset is near chance, the answer-marking context mechanism is insufficient.

Watch

Extended reading notes

Core claim

The central claim is that sequential questions about a table can be answered by direct cell selection from a graph-encoded table, and that this outperforms existing logical-form-based approaches on SQA. The model improves the state of the art from 45.6% (CAMP) to 55.1% in overall question accuracy, a relative error reduction of 18%, and its largest gains are on follow-up questions: third-turn accuracy reaches 46.8%, compared with 25.9% for the previous best contextual parser. Providing the previous reference answer at test time pushes overall accuracy to 61.7%, indicating a 6.6-point loss from error propagation, while the no-context variant reaches only 45.1%, showing that the answer-marking mechanism contributes about 10 points.

Load-bearing premise

The design assumes that everything a follow-up question needs from the earlier turn is contained in the previous answer's marked rows, columns and cells, because the earlier question's wording is not encoded anywhere in the graph.

Editorial extensions

If this is right

  • Conversational QA over tables can be trained from question-answer pairs alone, removing the need for annotated logical forms or SQL programs.
  • Follow-up question accuracy, the hardest part of conversational QA, improves substantially -- 46.8% versus 25.9% on third-turn questions -- suggesting that answer-marking is often sufficient conversational context.
  • Numeric comparisons and superlatives are handled without explicit programs: removing the numeric relations and rank features drops accuracy from 55.1% to 51.5%.
  • The non-autoregressive context mechanism is simple enough to be combined with other answer-selection architectures, since it only adds nominal features to nodes.
  • Error propagation is bounded: with oracle previous answers, accuracy rises to 61.7%, so roughly 6.6 points of the remaining error is due to propagating mistakes from earlier turns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference beyond the paper: if answer-marking is the only context channel, the model should fail on follow-ups whose referent is a term in the earlier question's wording rather than in its answer set; constructing a synthetic set of such questions would expose this boundary empirically.
  • Inference beyond the paper: because first-question accuracy trails the best prior parser by about 3.7 points, combining the graph encoder with pretrained language representations -- the paper names BERT as future work -- should close that gap while preserving the logical-form-free design.
  • Inference beyond the paper: the same graph-plus-pointer scheme maps naturally to knowledge graphs by treating entities and relations as nodes, which would extend the answer-marking context mechanism beyond tables.
  • Inference beyond the paper: the 8% context-error estimate comes from only 100 manually annotated follow-up questions, so the true ceiling of the answer-marking mechanism is not tightly measured; a larger error study would give a more reliable bound.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a neural model for answering sequential questions over tables without producing an intermediate logical form. Tables are encoded as graphs whose nodes are columns, rows, and cells, with edge labels for structure, lexical alignments, and numerical comparisons. A Transformer-based graph encoder computes node representations, and a pointer-network decoder selects answer cells directly. Conversation context is handled by marking the rows, columns, and cells that were the answers to the previous question. On the SQA dataset, the model is reported to improve state-of-the-art question accuracy from 45.6% (CAMP) to 55.1%, with a particularly large gain on follow-up questions (POS3 46.8% vs. 24.8%). The paper includes ablations for the numerical relations and the context mechanism, an oracle experiment quantifying error propagation, and a manual error analysis on 200 examples.

Significance. The paper's central empirical claim is strong and important if it holds: it suggests that conversational table QA can be solved by direct answer-cell selection with graph-structured inputs, bypassing logical forms and their supervision costs. The architecture is clearly described and the ablation of the context features (45.1 vs. 55.1) as well as the oracle-answer experiment (RA 61.7) are valuable for interpreting where the gains come from. The treatment of numeric features (rank, inverse rank, comparison edges) is a useful component and is supported by a targeted ablation. The manual error analysis, while small, gives a tangible picture of residual failure modes. The main risks are the scope of the conversational-context claim given that only prior answers (not prior questions) are encoded, and the absence of any variance reporting for the headline numbers.

major comments (2)
  1. [Section 2.1 (Context paragraph), Table 4, Section 6] The context mechanism encodes only the answers to the previous question and not the question itself, as stated in Section 2.1: 'the content of q1 is not encoded in the graph, only its answers.' This is a representational limitation for follow-up questions that need information from the earlier question beyond the answer set, such as anaphoric references to entities mentioned in q1. The CONTEXT example in Table 4 illustrates exactly this failure: for 'when was thaddeus bell born?, when was klaus jurgen schneider born?, which is older?' the model selects the correct birthday but not the person, because the persons are not in the previous answer set. The claim in Section 6 that the model 'particularly handles conversational context effectively' is therefore supported only for a restricted class of context dependence, and the 8% CONTEXT error rate in Table 3b rests on 100 manually annotated follow-ups, giving a wide confidence interval and no analysis of whether those errors are precisely the ones this architecture cannot represent. Please either provide an estimate of how many SQA follow-ups require non-answer content from previous questions, or qualify the conversational-context claim to reflect the answer-only context assumption.
  2. [Section 5, Table 1; Section 4, 'All numbers given for our model are averaged over 5 independent runs'] No variance, confidence intervals, or per-run values are reported for any of the model's numbers, despite the statement that all model numbers are averaged over 5 independent runs. The central empirical claim—an improvement in ALL from 45.6% (CAMP) to 55.1% and in SEQ from 13.2% to 28.1%—is a comparison of point estimates without a measure of dispersion or a significance test. This makes it impossible to assess whether the reported gains are robust to random initialization, which matters particularly for the smaller differences (e.g., POS1: 67.2 vs. DYNSP† 70.4). Please report per-run results or confidence intervals, or at least the standard deviation over the 5 runs, so the reader can evaluate the reliability of the state-of-the-art claim.
minor comments (5)
  1. [Section 4 (Experimental Setup)] The paper states that hyperparameters are tuned with Google Vizier but does not describe the validation split used for model selection or how many times the test set was accessed. Please clarify the validation procedure so the reader can rule out test-set overfitting.
  2. [Appendix B (Error Analysis and Table 3)] The manual error analysis is based on 100 initial and 100 follow-up questions with no reported inter-annotator agreement. A brief description of the annotation protocol and a measure of agreement would increase confidence in the error-category proportions, especially the 8% CONTEXT figure.
  3. [Section 2.1 (Graph Formulation)] Cells with identical text within a column are collapsed into a single node. This design choice may lose the distinction between multiple rows with the same value, which appears to be related to the ANSWER SET error type reported in Table 3. It would be helpful to discuss how the model or post-processing recovers the intended answer set in such cases.
  4. [Section 5 (Results)] The paper says the model is 'not sensitive to table size changes' based on Figure 3, but the figure only shows a scatter plot without a quantitative correlation or trend line. Adding a simple correlation coefficient or a comparison of accuracy on the largest vs. smallest tables would make the claim more precise.
  5. [Throughout] The reference list contains duplicate entries for Neelakantan et al. (2016a and 2016b) with identical titles; one is presumably the arXiv version and one the published version, but they should be distinguished properly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SQA gains are measured on a held-out test split against external baselines; the only overlapping-author citation (Shaw et al., 2019) supplies a graph-encoder component, not the central claim.

full rationale

The paper's central claim is empirical: a graph-encoded table with a pointer-based answer selector reaches 55.1% ALL accuracy on SQA's held-out test set, versus CAMP's 45.6% (Table 1). This number is not derived from any fitted equation or from the model's training objective in a way that would make the test result true by construction; the model is trained on the SQA training split and evaluated on the test split. The context mechanism (Section 2.1) marks previous answer rows, columns, and cells with nominal features, but this is a design choice rather than a circular reduction: the features are extracted from the supervision signal, and the paper explicitly measures the effect of removing them (OURS* vs OURS†, 45.1 vs 55.1). The oracle variant OURS† (RA) is clearly labeled as using the previous reference answer and is not presented as the system's own prediction. The only self-citation with overlapping authors is Shaw et al. (2019) for the additive edge-vector self-attention formulation; this is a reusable component and does not carry the paper's central claim. The acknowledged limitation that only the previous answers, not the content of q1, are encoded (Section 2.1, 'the content of q1 is not encoded in the graph, only its answers') is a representational limitation, not a circularity. No step in the paper reduces by definition to its inputs, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper contributes an architecture and an empirical evaluation. It introduces no derivation constants; the only fitted values are the network weights and hyperparameters tuned on validation. The central claim rests on several domain assumptions about the sufficiency of the graph representation, edit-distance alignment, answer-only context, and numeric features, each stated or implicit in Section 2.

free parameters (1)
  • Hyperparameter configuration (layers, hidden dim, dropout, heads, batch size) = layers 3 or 6; hidden dim 128, 256, or 512; dropout 0.2 to 0.5; heads 4 to 16; batch size 32 or 64
    Selected by Google Vizier tuning on validation data; the reported SQA test scores depend on these choices and are not reported with variance.
assumptions (4)
  • domain assumption The graph encoding of tables (columns, rows, cells, and question words as nodes) can represent all needed table semantics for SQA.
    The method's success relies on this representation; SQA was designed so answers are table cells, which makes it plausible, but the paper does not prove it.
  • domain assumption Levenshtein edit-distance alignment between question n-grams and table text is a sufficient lexical linker.
    Section 2.1 uses normalized edit distance above 0.5 to connect question tokens to table cells; the error analysis shows match errors when this linking fails.
  • domain assumption Marking previous answers with nominal features conveys enough conversational context.
    Section 2.1 Context explicitly states that the content of the previous question is not encoded, only its answers; this is also the paper's own stated limitation.
  • domain assumption Rank and inverse-rank features plus comparison edges capture numeric reasoning in table questions.
    Section 2.1 Numeric Operations adds these features; an ablation gives a 3.6 point overall gain, but complex numeric reasoning remains a major error source.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Answering Conversational Questions on Structured Data without Logical Forms." pith.science (2026). https://pith.science/paper/KGTQJZDM

@misc{pith2026190811787,
  author       = {Pith},
  title        = {Pith review of: Answering Conversational Questions on Structured Data without Logical Forms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGTQJZDM}},
  note         = {Machine review of arXiv:1908.11787}
}
read the original abstract

We present a novel approach to answering sequential questions based on structured objects such as knowledge bases or tables without using a logical form as an intermediate representation. We encode tables as graphs using a graph neural network model based on the Transformer architecture. The answers are then selected from the encoded graph using a pointer network. This model is appropriate for processing conversations around structured data, where the attention mechanism that selects the answers to a question can also be used to resolve conversational references. We demonstrate the validity of this approach with competitive results on the Sequential Question Answering (SQA) task (Iyyer et al., 2017).

Figures

Figures reproduced from arXiv: 1908.11787 by the authors.

Figure 1
Figure 1. Example of a table encoded as a graph in relation to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The model input is a graph, with nodes corre [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Scatter-plot of accuracy and table size. Each [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 18 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. arXiv preprint arXiv:1902.07198

  4. [4]

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. https://www.aclweb.org/anthology/D18-1241 Q u AC : Question answering in context . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174--2184, Brussels, Belgium. Association for Computational Linguistics

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  6. [6]

    Li Dong and Mirella Lapata. 2018. https://www.aclweb.org/anthology/P18-1068 Coarse-to-fine decoding for neural semantic parsing . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731--742, Melbourne, Australia. Association for Computational Linguistics

  7. [7]

    Sculley, editors

    Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. http://www.kdd.org/kdd2017/papers/view/google-vizier-a-service-for-black-box-optimization Google Vizier: A Service for Black-Box Optimization

  8. [8]

    Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. http://dl.acm.org/citation.cfm?id=3327144.3327217 Dialog-to-action: Conversational question answering over a large-scale knowledge base . In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS'18, pages 2946--2955, USA. Curran Associates Inc

Show all 33 references
  1. [9]

    Izzeddin Gur, Semih Yavuz, Yu Su, and Xifeng Yan. 2018. https://www.aclweb.org/anthology/P18-1124 D ial SQL : Dialogue based structured query generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

  2. [10]

    Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. https://doi.org/10.18653/v1/P17-1167 Search-based neural structured learning for sequential question answering . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

  3. [11]

    Robin Jia and Percy Liang. 2016. https://doi.org/10.18653/v1/P16-1002 Data recombination for neural semantic parsing . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12--22, Berlin, Germany. Association...

  4. [12]

    Diederik P Kingma and Jimmy Ba. 2014. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . CoRR, abs/1412.6980

  5. [13]

    Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. https://doi.org/10.18653/v1/D17-1160 Neural semantic parsing with type constraints for semi-structured tables . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516--...

  6. [14]

    Le, Kenneth D

    Chen Liang, Jonathan Berant, Quoc V. Le, Kenneth D. Forbus, and Ni Lao. 2016. http://arxiv.org/abs/1611.00020 Neural symbolic machines: Learning semantic parsers on freebase with weak supervision . CoRR, abs/1611.00020

  7. [15]

    Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. http://papers.nips.cc/paper/8204-memory-augmented-policy-optimization-for-program-synthesis-and-semantic-parsing.pdf Memory augmented policy optimization for program synthesis and semantic parsing . In...

  8. [16]

    Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tom \'a s Ko c isk \'y , Fumin Wang, and Andrew Senior. 2016. https://doi.org/10.18653/v1/P16-1057 Latent predictor networks for code generation . In Proceedings of the 54th Annual Meeting of the Association fo...

  9. [17]

    Reginald Long, Panupong Pasupat, and Percy Liang. 2016. https://doi.org/10.18653/v1/P16-1138 Simpler context-dependent logical forms via model projections . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

  10. [19]

    Le, Mart \' n Abadi, Andrew McCallum, and Dario Amodei

    Arvind Neelakantan, Quoc V. Le, Mart \' n Abadi, Andrew McCallum, and Dario Amodei. 2016 b . http://arxiv.org/abs/1611.08945 Learning a natural language interface with neural programmer . CoRR, abs/1611.08945

  11. [20]

    Panupong Pasupat and Percy Liang. 2015. https://doi.org/10.3115/v1/P15-1142 Compositional semantic parsing on semi-structured tables . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur...

  12. [21]

    Haoruo Peng, Ming-Wei Chang, and Wen-tau Yih. 2017. https://doi.org/10.18653/v1/D17-1252 Maximum margin reward networks for learning from explicit and implicit supervision . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2368--...

  13. [22]

    Bojan Petrovski, Ignacio Aguado, Andreea Hossmann, Michael Baeriswyl, and Claudiu Musat. 2018. https://www.aclweb.org/anthology/W18-5710 Embedding individual table columns for resilient SQL chatbots . In Proceedings of the 2018 EMNLP Workshop SCAI : The 2nd International Works...

  14. [23]

    Khapra, Karthik Sankaranarayanan, and Sarath Chandar

    Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17181 Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowl...

  15. [24]

    Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. 2019. https://www.aclweb.org/anthology/P19-1010 Generating logical forms from graph representations of text and entities . In Proceedings of the 57th Annual Meeting of the Association for Computat...

  16. [25]

    Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. https://doi.org/10.18653/v1/N18-1203 Learning to map context-dependent sentences to executable formal queries . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguist...

  17. [26]

    Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018 a . https://www.aclweb.org/anthology/D18-1455 Open domain question answering using early fusion of knowledge bases and text . In Proceedings of the 2018 Conference on Em...

  18. [27]

    Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, Xiaocheng Feng, and Bing Qin. 2018 b . http://arxiv.org/abs/1809.04271 Knowledge-aware conversational semantic parsing over web tables . CoRR, abs/1809.04271

  19. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fe...

  20. [29]

    Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. http://papers.nips.cc/paper/5866-pointer-networks.pdf Pointer networks . In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692--2...

  21. [30]

    Chenglong Wang, Po-Sen Huang, Alex Polozov, Marc Brockschmidt, and Rishabh Singh . 2018. https://www.microsoft.com/en-us/research/publication/execution-guided-neural-program-decoding/ Execution-guided neural program decoding . In ICML Neural Abstract Machines and Program Induc...

  22. [31]

    Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436

  23. [32]

    Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. https://doi.org/10.3115/v1/P15-1128 Semantic parsing via staged query graph generation: Question answering with knowledge base . In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin...

  24. [33]

    Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018. https://doi.org/10.18653/v1/N18-2093 T ype SQL : Knowledge-based type-aware neural text-to- SQL generation . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computat...

  25. [34]

    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.