REVIEW 2 major objections 5 minor 33 references
Answering Conversational Questions on Structured Data without Logical Forms
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A transformer graph encoder turns a table into nodes for rows, columns and cells, and a pointer network answers a conversation by selecting cells directly, beating prior systems on the Sequential Question Answering benchmark without ever…
desk verdict A solid, honest table-QA paper: direct cell selection beats logical-form baselines on SQA, but the conversational context mechanism is answer-only, which is a real limitation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a table-to-graph encoding: columns, rows, cells, question nodes, and question tokens are all graph nodes, with directed edges between columns/rows and cells, edit-distance alignments between question n-grams and table text, and comparison relations between numeric expressions in the question and numeric cells. Numerical cells carry rank and inverse-rank features within their column, and the previous answer is encoded by adding ANSWER ROW, ANSWER COLUMN, and ANSWER CELL features to the relevant nodes. A Transformer-based graph neural network encoder, using Shaw et al.'s additive edge-vector self-attention, turns this graph into contextualized node representations, and a pointer-network copy mechanism in the decoder selects answer values directly from table-cell nodes.
What would settle it
On the SQA test set, isolate follow-up questions whose correct answer requires a number, entity, or condition that appears only in the first question's text and not in its answer cells; if a human can answer them and the model's accuracy on such a subset is near chance, the answer-marking context mechanism is insufficient.
Extended reading notes
Core claim
The central claim is that sequential questions about a table can be answered by direct cell selection from a graph-encoded table, and that this outperforms existing logical-form-based approaches on SQA. The model improves the state of the art from 45.6% (CAMP) to 55.1% in overall question accuracy, a relative error reduction of 18%, and its largest gains are on follow-up questions: third-turn accuracy reaches 46.8%, compared with 25.9% for the previous best contextual parser. Providing the previous reference answer at test time pushes overall accuracy to 61.7%, indicating a 6.6-point loss from error propagation, while the no-context variant reaches only 45.1%, showing that the answer-marking mechanism contributes about 10 points.
Load-bearing premise
The design assumes that everything a follow-up question needs from the earlier turn is contained in the previous answer's marked rows, columns and cells, because the earlier question's wording is not encoded anywhere in the graph.
Editorial extensions
If this is right
- Conversational QA over tables can be trained from question-answer pairs alone, removing the need for annotated logical forms or SQL programs.
- Follow-up question accuracy, the hardest part of conversational QA, improves substantially -- 46.8% versus 25.9% on third-turn questions -- suggesting that answer-marking is often sufficient conversational context.
- Numeric comparisons and superlatives are handled without explicit programs: removing the numeric relations and rank features drops accuracy from 55.1% to 51.5%.
- The non-autoregressive context mechanism is simple enough to be combined with other answer-selection architectures, since it only adds nominal features to nodes.
- Error propagation is bounded: with oracle previous answers, accuracy rises to 61.7%, so roughly 6.6 points of the remaining error is due to propagating mistakes from earlier turns.
Reading between the lines
- Inference beyond the paper: if answer-marking is the only context channel, the model should fail on follow-ups whose referent is a term in the earlier question's wording rather than in its answer set; constructing a synthetic set of such questions would expose this boundary empirically.
- Inference beyond the paper: because first-question accuracy trails the best prior parser by about 3.7 points, combining the graph encoder with pretrained language representations -- the paper names BERT as future work -- should close that gap while preserving the logical-form-free design.
- Inference beyond the paper: the same graph-plus-pointer scheme maps naturally to knowledge graphs by treating entities and relations as nodes, which would extend the answer-marking context mechanism beyond tables.
- Inference beyond the paper: the 8% context-error estimate comes from only 100 manually annotated follow-up questions, so the true ceiling of the answer-marking mechanism is not tightly measured; a larger error study would give a more reliable bound.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a neural model for answering sequential questions over tables without producing an intermediate logical form. Tables are encoded as graphs whose nodes are columns, rows, and cells, with edge labels for structure, lexical alignments, and numerical comparisons. A Transformer-based graph encoder computes node representations, and a pointer-network decoder selects answer cells directly. Conversation context is handled by marking the rows, columns, and cells that were the answers to the previous question. On the SQA dataset, the model is reported to improve state-of-the-art question accuracy from 45.6% (CAMP) to 55.1%, with a particularly large gain on follow-up questions (POS3 46.8% vs. 24.8%). The paper includes ablations for the numerical relations and the context mechanism, an oracle experiment quantifying error propagation, and a manual error analysis on 200 examples.
Significance. The paper's central empirical claim is strong and important if it holds: it suggests that conversational table QA can be solved by direct answer-cell selection with graph-structured inputs, bypassing logical forms and their supervision costs. The architecture is clearly described and the ablation of the context features (45.1 vs. 55.1) as well as the oracle-answer experiment (RA 61.7) are valuable for interpreting where the gains come from. The treatment of numeric features (rank, inverse rank, comparison edges) is a useful component and is supported by a targeted ablation. The manual error analysis, while small, gives a tangible picture of residual failure modes. The main risks are the scope of the conversational-context claim given that only prior answers (not prior questions) are encoded, and the absence of any variance reporting for the headline numbers.
major comments (2)
- [Section 2.1 (Context paragraph), Table 4, Section 6] The context mechanism encodes only the answers to the previous question and not the question itself, as stated in Section 2.1: 'the content of q1 is not encoded in the graph, only its answers.' This is a representational limitation for follow-up questions that need information from the earlier question beyond the answer set, such as anaphoric references to entities mentioned in q1. The CONTEXT example in Table 4 illustrates exactly this failure: for 'when was thaddeus bell born?, when was klaus jurgen schneider born?, which is older?' the model selects the correct birthday but not the person, because the persons are not in the previous answer set. The claim in Section 6 that the model 'particularly handles conversational context effectively' is therefore supported only for a restricted class of context dependence, and the 8% CONTEXT error rate in Table 3b rests on 100 manually annotated follow-ups, giving a wide confidence interval and no analysis of whether those errors are precisely the ones this architecture cannot represent. Please either provide an estimate of how many SQA follow-ups require non-answer content from previous questions, or qualify the conversational-context claim to reflect the answer-only context assumption.
- [Section 5, Table 1; Section 4, 'All numbers given for our model are averaged over 5 independent runs'] No variance, confidence intervals, or per-run values are reported for any of the model's numbers, despite the statement that all model numbers are averaged over 5 independent runs. The central empirical claim—an improvement in ALL from 45.6% (CAMP) to 55.1% and in SEQ from 13.2% to 28.1%—is a comparison of point estimates without a measure of dispersion or a significance test. This makes it impossible to assess whether the reported gains are robust to random initialization, which matters particularly for the smaller differences (e.g., POS1: 67.2 vs. DYNSP† 70.4). Please report per-run results or confidence intervals, or at least the standard deviation over the 5 runs, so the reader can evaluate the reliability of the state-of-the-art claim.
minor comments (5)
- [Section 4 (Experimental Setup)] The paper states that hyperparameters are tuned with Google Vizier but does not describe the validation split used for model selection or how many times the test set was accessed. Please clarify the validation procedure so the reader can rule out test-set overfitting.
- [Appendix B (Error Analysis and Table 3)] The manual error analysis is based on 100 initial and 100 follow-up questions with no reported inter-annotator agreement. A brief description of the annotation protocol and a measure of agreement would increase confidence in the error-category proportions, especially the 8% CONTEXT figure.
- [Section 2.1 (Graph Formulation)] Cells with identical text within a column are collapsed into a single node. This design choice may lose the distinction between multiple rows with the same value, which appears to be related to the ANSWER SET error type reported in Table 3. It would be helpful to discuss how the model or post-processing recovers the intended answer set in such cases.
- [Section 5 (Results)] The paper says the model is 'not sensitive to table size changes' based on Figure 3, but the figure only shows a scatter plot without a quantitative correlation or trend line. Adding a simple correlation coefficient or a comparison of accuracy on the largest vs. smallest tables would make the claim more precise.
- [Throughout] The reference list contains duplicate entries for Neelakantan et al. (2016a and 2016b) with identical titles; one is presumably the arXiv version and one the published version, but they should be distinguished properly.
Circularity Check
No circularity: the SQA gains are measured on a held-out test split against external baselines; the only overlapping-author citation (Shaw et al., 2019) supplies a graph-encoder component, not the central claim.
full rationale
The paper's central claim is empirical: a graph-encoded table with a pointer-based answer selector reaches 55.1% ALL accuracy on SQA's held-out test set, versus CAMP's 45.6% (Table 1). This number is not derived from any fitted equation or from the model's training objective in a way that would make the test result true by construction; the model is trained on the SQA training split and evaluated on the test split. The context mechanism (Section 2.1) marks previous answer rows, columns, and cells with nominal features, but this is a design choice rather than a circular reduction: the features are extracted from the supervision signal, and the paper explicitly measures the effect of removing them (OURS* vs OURS†, 45.1 vs 55.1). The oracle variant OURS† (RA) is clearly labeled as using the previous reference answer and is not presented as the system's own prediction. The only self-citation with overlapping authors is Shaw et al. (2019) for the additive edge-vector self-attention formulation; this is a reusable component and does not carry the paper's central claim. The acknowledged limitation that only the previous answers, not the content of q1, are encoded (Section 2.1, 'the content of q1 is not encoded in the graph, only its answers') is a representational limitation, not a circularity. No step in the paper reduces by definition to its inputs, so the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Hyperparameter configuration (layers, hidden dim, dropout, heads, batch size) =
layers 3 or 6; hidden dim 128, 256, or 512; dropout 0.2 to 0.5; heads 4 to 16; batch size 32 or 64
assumptions (4)
- domain assumption The graph encoding of tables (columns, rows, cells, and question words as nodes) can represent all needed table semantics for SQA.
- domain assumption Levenshtein edit-distance alignment between question n-grams and table text is a sufficient lexical linker.
- domain assumption Marking previous answers with nominal features conveys enough conversational context.
- domain assumption Rank and inverse-rank features plus comparison edges capture numeric reasoning in table questions.
Cite this review
Pith. "Pith review of Answering Conversational Questions on Structured Data without Logical Forms." pith.science (2026). https://pith.science/paper/KGTQJZDM
@misc{pith2026190811787,
author = {Pith},
title = {Pith review of: Answering Conversational Questions on Structured Data without Logical Forms},
year = {2026},
howpublished = {\url{https://pith.science/paper/KGTQJZDM}},
note = {Machine review of arXiv:1908.11787}
}
read the original abstract
We present a novel approach to answering sequential questions based on structured objects such as knowledge bases or tables without using a logical form as an intermediate representation. We encode tables as graphs using a graph neural network model based on the Transformer architecture. The answers are then selected from the encoded graph using a pointer network. This model is appropriate for processing conversations around structured data, where the attention mechanism that selects the answers to a question can also be used to resolve conversational references. We demonstrate the validity of this approach with competitive results on the Sequential Question Answering (SQA) task (Iyyer et al., 2017).
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. arXiv preprint arXiv:1902.07198
work page Pith review arXiv 2019
-
[4]
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. https://www.aclweb.org/anthology/D18-1241 Q u AC : Question answering in context . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174--2184, Brussels, Belgium. Association for Computational Linguistics
work page 2018
-
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[6]
Li Dong and Mirella Lapata. 2018. https://www.aclweb.org/anthology/P18-1068 Coarse-to-fine decoding for neural semantic parsing . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731--742, Melbourne, Australia. Association for Computational Linguistics
work page 2018
-
[7]
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. 2017. http://www.kdd.org/kdd2017/papers/view/google-vizier-a-service-for-black-box-optimization Google Vizier: A Service for Black-Box Optimization
work page 2017
-
[8]
Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2018. http://dl.acm.org/citation.cfm?id=3327144.3327217 Dialog-to-action: Conversational question answering over a large-scale knowledge base . In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS'18, pages 2946--2955, USA. Curran Associates Inc
arXiv 2018
Show all 33 references
-
[9]
Izzeddin Gur, Semih Yavuz, Yu Su, and Xifeng Yan. 2018. https://www.aclweb.org/anthology/P18-1124 D ial SQL : Dialogue based structured query generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...
2018
-
[10]
Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. https://doi.org/10.18653/v1/P17-1167 Search-based neural structured learning for sequential question answering . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...
2017 doi
-
[11]
Robin Jia and Percy Liang. 2016. https://doi.org/10.18653/v1/P16-1002 Data recombination for neural semantic parsing . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12--22, Berlin, Germany. Association...
2016 doi
-
[12]
Diederik P Kingma and Jimmy Ba. 2014. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . CoRR, abs/1412.6980
2014 arXiv
-
[13]
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. https://doi.org/10.18653/v1/D17-1160 Neural semantic parsing with type constraints for semi-structured tables . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516--...
2017 doi
-
[14]
Le, Kenneth D
Chen Liang, Jonathan Berant, Quoc V. Le, Kenneth D. Forbus, and Ni Lao. 2016. http://arxiv.org/abs/1611.00020 Neural symbolic machines: Learning semantic parsers on freebase with weak supervision . CoRR, abs/1611.00020
2016 arXiv
-
[15]
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. http://papers.nips.cc/paper/8204-memory-augmented-policy-optimization-for-program-synthesis-and-semantic-parsing.pdf Memory augmented policy optimization for program synthesis and semantic parsing . In...
2018
-
[16]
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tom \'a s Ko c isk \'y , Fumin Wang, and Andrew Senior. 2016. https://doi.org/10.18653/v1/P16-1057 Latent predictor networks for code generation . In Proceedings of the 54th Annual Meeting of the Association fo...
2016 doi
-
[17]
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. https://doi.org/10.18653/v1/P16-1138 Simpler context-dependent logical forms via model projections . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...
2016 doi
-
[19]
Le, Mart \' n Abadi, Andrew McCallum, and Dario Amodei
Arvind Neelakantan, Quoc V. Le, Mart \' n Abadi, Andrew McCallum, and Dario Amodei. 2016 b . http://arxiv.org/abs/1611.08945 Learning a natural language interface with neural programmer . CoRR, abs/1611.08945
2016 arXiv
-
[20]
Panupong Pasupat and Percy Liang. 2015. https://doi.org/10.3115/v1/P15-1142 Compositional semantic parsing on semi-structured tables . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur...
2015 doi
-
[21]
Haoruo Peng, Ming-Wei Chang, and Wen-tau Yih. 2017. https://doi.org/10.18653/v1/D17-1252 Maximum margin reward networks for learning from explicit and implicit supervision . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2368--...
2017 doi
-
[22]
Bojan Petrovski, Ignacio Aguado, Andreea Hossmann, Michael Baeriswyl, and Claudiu Musat. 2018. https://www.aclweb.org/anthology/W18-5710 Embedding individual table columns for resilient SQL chatbots . In Proceedings of the 2018 EMNLP Workshop SCAI : The 2nd International Works...
2018
-
[23]
Khapra, Karthik Sankaranarayanan, and Sarath Chandar
Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17181 Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowl...
2018
-
[24]
Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. 2019. https://www.aclweb.org/anthology/P19-1010 Generating logical forms from graph representations of text and entities . In Proceedings of the 57th Annual Meeting of the Association for Computat...
2019
-
[25]
Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. https://doi.org/10.18653/v1/N18-1203 Learning to map context-dependent sentences to executable formal queries . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguist...
2018 doi
-
[26]
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018 a . https://www.aclweb.org/anthology/D18-1455 Open domain question answering using early fusion of knowledge bases and text . In Proceedings of the 2018 Conference on Em...
2018
-
[27]
Yibo Sun, Duyu Tang, Nan Duan, Jingjing Xu, Xiaocheng Feng, and Bing Qin. 2018 b . http://arxiv.org/abs/1809.04271 Knowledge-aware conversational semantic parsing over web tables . CoRR, abs/1809.04271
2018 arXiv
-
[28]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fe...
2017
-
[29]
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. http://papers.nips.cc/paper/5866-pointer-networks.pdf Pointer networks . In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692--2...
2015
-
[30]
Chenglong Wang, Po-Sen Huang, Alex Polozov, Marc Brockschmidt, and Rishabh Singh . 2018. https://www.microsoft.com/en-us/research/publication/execution-guided-neural-program-decoding/ Execution-guided neural program decoding . In ICML Neural Abstract Machines and Program Induc...
2018
-
[31]
Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436
2017 arXiv
-
[32]
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. https://doi.org/10.3115/v1/P15-1128 Semantic parsing via staged query graph generation: Question answering with knowledge base . In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin...
2015 doi
-
[33]
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018. https://doi.org/10.18653/v1/N18-2093 T ype SQL : Knowledge-based type-aware neural text-to- SQL generation . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computat...
2018 doi
-
[34]
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103
2017 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.