REVIEW 4 major objections 8 minor 1 cited by
RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector
T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Intent and slot labels, added to queries as SLU annotations, improve open-domain long-table tabular question answering by guiding retrieval and answer generation.
desk verdict Useful new dataset, but SLUTQA's central claim is confounded by demonstration selection; worth sending to review with a request for fixes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the SLU label pair attached to each query: an intent category drawn from 16 types and slot tags in Inside-Outside-Beginning format covering city, district, development name, company name, year, and month. The SR module puts these labels into few-shot prompts that rewrite the query into a table-caption-style summary, which is then matched against table captions with BM25; the SFA module uses the same labels to prune irrelevant rows and columns of retrieved long tables or to constrain SQL generation. Labels come either from a fine-tuned BERT model or from in-context learning with a handful of examples.
What would settle it
Run SLUTQA against a vanilla pipeline on a set of naturally collected, non-template real estate queries with human-annotated SLU labels, measuring retrieval F1 and answer exact match; if the SLU-based gains shrink to near the vanilla level, the claim that SLU labels drive the improvement would be refuted. A simpler check is whether an SR summary generated without SLU labels retrieves tables as well as the SLU-labeled version.
Extended reading notes
Core claim
On the paper's own terms, the central empirical discovery is that intent and slot labels, treated as extra query annotations and injected into in-context learning prompts, materially improve both table retrieval and answer quality for open-domain long-table TQA. The paper reports that using SLU labels in the retrieval module raises table-retrieval F1 over the vanilla query-summary approach, and that using them in the filtering and SQL-generation module improves Table EM by about four percentage points in markdown format and ECR and pass@1 by more than 20 points in SQL format across Qwen2-7B, Qwen2-72B, and GLM4-9B. The paper frames RETQA as the first TQA dataset to include SLU labels, and SLUTQA as the first framework to combine SLU tasks with TQA tasks, with the claimed benefit coming without fine-tuning the LLM.
Load-bearing premise
The benchmark's queries, built from templates and LLM rewrites, have slot labels that align exactly with table captions and column headers, so real-world user queries that lack this clean alignment may not show the same gains.
Editorial extensions
If this is right
- RETQA provides a benchmark where open-domain retrieval, long tables averaging 252.9 rows, multi-table queries, and multi-intent queries can be evaluated together in Chinese.
- SLUTQA improves table-retrieval F1 and markdown and SQL answer accuracy over a vanilla pipeline on Qwen2-7B, Qwen2-72B, and GLM4-9B without fine-tuning the LLM.
- Fine-tuning BERT for SLU prediction yields further gains, showing that better label quality translates into better TQA performance.
- SLU-guided row and column filtering reduces the long-table bottleneck for LLMs with limited context windows.
- Multi-table and multi-intent queries are explicitly labeled, enabling separate study of retrieval, filtering, and reasoning in open-domain table QA.
Reading between the lines
- The reported gains may partly rely on RETQA's construction, where table captions were designed to match the variables in the query templates, making the SLU-to-caption mapping unusually clean; real-world deployment would need to test transfer to naturally collected queries.
- SLU labels could be plugged into dense retrieval or embedding-based table search rather than only BM25, since the caption-style summary produced by the SR module is a compact query representation.
- The same intent-plus-slot annotation scheme could transfer to other vertical domains with structured tables, such as healthcare or logistics, where long tables and open-domain retrieval also arise.
- A testable extension is to measure how sensitive SLUTQA is to SLU prediction errors, since the framework's benefit depends on label quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RETQA, a Chinese open-domain tabular QA dataset for the real estate domain, with 4,932 tables and 20,762 QA pairs, and proposes SLUTQA, a framework that uses spoken language understanding (SLU) labels (intent and slot) to improve retrieval and answer generation with large language models via in-context learning. The dataset is constructed from public real estate data using 90 templates, with reverse-engineered SLU labels and LLM-rewritten queries, and answers are provided in Markdown, SQL, and natural language. Experiments on Qwen2-7b, Qwen2-72b, and GLM4-9b report that SLUTQA improves Markdown table exact match, SQL execution ratio/pass@1, and table retrieval F1 over a vanilla baseline. The paper includes ablations of the retrieval (SR) and answer (SFA) modules, but these ablations do not fully isolate the effect of SLU labels from the demonstration-selection policy, and the paper reports no variance, repeated runs, or significance tests.
Significance. If the empirical claims hold, RETQA would be a substantial new resource: it is the first large-scale open-domain Chinese TQA dataset for real estate, with long tables and multi-table queries, and it is the first TQA dataset to include SLU labels. The dataset construction is described transparently, the data sizes are large, and the code and data are promised for release. The idea of using SLU labels to guide table retrieval and filtering is intuitive and could be useful beyond this domain. However, the central contribution is compromised by the experimental design: the reported gains of SLUTQA over 'Vanilla' change two variables at once (SLU labels and demonstration selection), and the only condition that holds the selection policy fixed is limited to Markdown answers with gold labels. Without a control that isolates the SLU-label signal, the paper's headline claim that SLU labels 'significantly improve' performance is not established.
major comments (4)
- [Experiments, 'Result and Analysis' (Tables 2-4, 6)] The main experiments compare a Vanilla baseline that uses 'five randomly selected examples' for ICL with SLUTQA, whose SR and SFA modules select demonstrations 'sharing the same intent' and inject intent/slot tokens into the prompt. This changes two factors at once: the presence of SLU labels and the demonstration-selection policy. Table 5's 'Simplified' condition is the only one that holds the selection policy fixed (intent-matched, no labels), but it is reported only for Markdown answers and uses gold SLU labels rather than predicted ones; no such control is provided for SQL generation (Table 6) or for query-summary-based retrieval (Table 4). A reader therefore cannot tell whether the improvements come from the SLU labels themselves or from better-matched demonstrations. Please add a no-label intent-matched baseline for SQL and retrieval experiments, and if possible a predicted-label condition for the Table 5 ablation.
- [Appendix F and Tables 2-3] Appendix F states that no multiple runs, mean/variance statistics, or significance tests were performed. This is a load-bearing issue because several reported differences are small (e.g., Table EM improvements of 1.6-4.4 points absolute in Table 2, and retrieval F1 differences of 0.7-2.6 points in Table 4 for ICL vs Vanilla). Without variance estimates or significance tests, the word 'significantly' in the abstract and conclusions is unsupported, and the strength of the central claim cannot be evaluated. At minimum, report results over multiple runs with standard deviations and a paired test (e.g., bootstrap or approximate randomization) for the key comparisons.
- [Dataset Construction, 'Intent and Slot Annotation' and 'QA Pair Generation'] The SLU labels are reverse-engineered from the 90 templates, and the table captions are themselves standardized from the same templates (e.g., 'Land Auction Information Table for {city district}'). The SR module generates a query summary from intent/slot values that were derived from the templates, and those slots align exactly with caption entities. This creates a closed loop: the labels are designed to match the retrieval targets, so high retrieval performance with SLU labels may reflect this artificial alignment rather than a general property of SLU-aware retrieval. The paper should either provide evidence that the template-derived labels transfer to naturally occurring user queries (e.g., a small human-written query set with manually annotated SLU labels) or moderate the claim that SLU labels improve open-domain TQA in general, as opposed to on this specific template-aligned benchmark.
- [Experiments, 'Baselines' and Table 4] The description of the Vanilla baseline is inconsistent. In 'Baselines' the paper says the vanilla approach 'generated a query summary for BM25 retrieval and then produced the final answers using ICL with five randomly selected examples.' But in the SR ablation (Table 4), 'Vanilla' also uses a generated query summary for BM25 retrieval. This means the SR module is being compared against a baseline that already uses the same idea (summary-based retrieval), and the only difference is again the demonstration-selection policy and the presence of SLU labels. Please specify exactly what the Vanilla summary is, how it is generated, and whether the same example-selection policy is used; otherwise the retrieval comparison is not interpretable as an ablation of SLU labels.
minor comments (8)
- [Method, 'SLU Module'] There is a typo: 'within the SLUQTA framework' should be 'SLUTQA framework'.
- [Dataset Construction, 'Intent and Slot Annotation'] The slot type text contains an odd typo: '“month”,’' has an extra quote mark. Please fix.
- [Method, 'SFA Module'] In the SQL-format description, 'the SLA module for SQL formats' should be 'the SFA module'.
- [Abstract and Code availability] The abstract says the code is at github.com/jensen-w/RETQA, but the paper's code line says github.com/jensenw1/RETQA; please make these consistent.
- [Table 1] The legend for Table 1 only explains '–', but the table uses '%' and '!' symbols; please define all symbols in the caption or legend.
- [Appendix D and Method, 'SLU Module'] The method section says '22 examples' for few-shot SLU prediction, while Appendix D says 'a sufficient number of samples'; please reconcile these numbers or specify the exact setting.
- [Method, 'SR Module'] For multi-intent queries, the selection of 'five examples from the training set that share the same intent' is under-specified: please clarify whether the example must match all predicted intents or any one of them, and how ties are broken.
- [Related Works, 'TQA Datasets'] The dataset comparison in Table 1 would be more informative if it included a Chinese TQA dataset or another open-domain long-table benchmark; currently the comparison is mostly against English datasets, which makes the 'first Chinese' claim harder to evaluate.
Circularity Check
RETQA's slot-to-caption alignment makes SLU-label retrieval gains partly self-definitional; main Vanilla/SLUTQA comparison also confounds SLU labels with intent-matched ICL selection.
-
self definitional
[Dataset Construction, 'Intent and Slot Annotation'; Method, 'SR Module'; Table 4]
""For slot labels, we categorize the entities into six types: "city", "district", "development name", "company name", "year", and "month", based on the entity types in the query templates and corresponding table headers." ... "we propose the SR module, which utilizes ICL with SLU labels to enable the large LLMs to generate a query summary in the form of a table caption.""
Slot categories are defined from the same query templates and table headers that determine the standardized captions; on RETQA a caption is a fixed pattern over exactly those slot values (e.g., city, district, year, month). The SR module's query summary is an LLM re-emission of the caption pattern from the slots, so retrieving the gold caption via BM25 is close to inverting the dataset's caption template. The retrieval gain in Table 4 therefore reflects the constructional alignment between slots and captions rather than a general property of SLU labels for arbitrary open-domain tables.
full rationale
The strongest circular element is the construction loop shared by the dataset and the SR module: the six slot types are extracted from the query templates and table headers, and the same headers/templates determine the standardized captions that serve as retrieval targets. An SR prompt that is asked to turn slots into a 'query summary in the form of a table caption' is therefore being asked to re-generate the caption template from its own ingredients, so the Table 4 retrieval improvements are partly a self-fulfilling benchmark property. This does not collapse the whole paper: the SFA markdown ablation (Table 5) compares Simplified vs SFA under gold tables/labels and holds demonstration selection closer to fixed, giving an independent test of SLU labels in answer generation. However, the headline Vanilla-vs-SLUTQA comparisons in Tables 2, 3, 4, and 6 change two variables at once (SLU labels and ICL demonstration selection: Vanilla uses five randomly selected examples, while SR/SFA select examples sharing the predicted intent), so the size of the claimed 'significant improvement' is not attributable to SLU labels alone; only Table 5 provides an appropriate control, and only for markdown answers. Appendix F candidly states that no repeated runs or significance tests were completed, consistent with the absence of variance reporting. The self-citations (Cheng/Yang/Jia 2023; Zhu et al. 2024) are used for background on SLU tasks and are not load-bearing. Overall, partial circularity via the caption/slot construction loop, plus an important confound, yields a score of 6.
Assumptions & free parameters
free parameters (5)
- SLU few-shot example count =
22
- SR module example count =
5
- SFA module example count =
2-5
- Query template count =
90
- Intent category count =
16
assumptions (4)
- domain assumption LLM-rewritten queries preserve the slot and intent labels assigned to the original template queries.
- domain assumption Standardized table captions are sufficient retrieval keys, and the query summary format matches them.
- domain assumption Executable SQL generated from templates is a correct proxy for the answer.
- domain assumption DeepSeek-V1's naturalness scores are a valid measure of query quality.
Cite this review
Pith. "Pith review of RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector." pith.science (2026). https://pith.science/paper/3EB3MSEN
@misc{pith2026241210104,
author = {Pith},
title = {Pith review of: RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector},
year = {2026},
howpublished = {\url{https://pith.science/paper/3EB3MSEN}},
note = {Machine review of arXiv:2412.10104}
}
read the original abstract
The real estate market relies heavily on structured data, such as property details, market trends, and price fluctuations. However, the lack of specialized Tabular Question Answering datasets in this domain limits the development of automated question-answering systems. To fill this gap, we introduce RETQA, the first large-scale open-domain Chinese Tabular Question Answering dataset for Real Estate. RETQA comprises 4,932 tables and 20,762 question-answer pairs across 16 sub-fields within three major domains: property information, real estate company finance information and land auction information. Compared with existing tabular question answering datasets, RETQA poses greater challenges due to three key factors: long-table structures, open-domain retrieval, and multi-domain queries. To tackle these challenges, we propose the SLUTQA framework, which integrates large language models with spoken language understanding tasks to enhance retrieval and answering accuracy. Extensive experiments demonstrate that SLUTQA significantly improves the performance of large language models on RETQA by in-context learning. RETQA and SLUTQA provide essential resources for advancing tabular question answering research in the real estate domain, addressing critical challenges in open-domain and long-table question-answering. The dataset and code are publicly available at \url{https://github.com/jensen-w/RETQA}.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
TLCCSP: A Scalable Framework for Enhancing Time Series Forecasting with Time-Lagged Cross-Correlations
A framework that retrieves time-lagged correlated series via a shifted dynamic time warping distance, or a learned approximation of it, and uses them as auxiliary inputs to improve time series forecasting.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Badaro, G.; Saeed, M.; and Papotti, P. 2023. Transformers for Tabular Data Representation: A Survey of Models and Applications. Transactions of the Association for Computational Linguistics, 11: 227--249
work page 2023
-
[4]
Chen, W. 2023. Large Language Models are few (1)-shot Table Reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, 1120--1130
work page 2023
-
[5]
Cheng, L.; Yang, W.; and Jia, W. 2023. A scope sensitive and result attentive model for multi-intent spoken language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, 12691--12699. AAAI Press
work page 2023
-
[6]
DeepSeek-AI. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv preprint arXiv:2401.02954
arXiv 2024
-
[7]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171--4186
2019
-
[8]
Fang, X.; Xu, W.; Tan, F. A.; Zhang, J.; Hu, Z.; Qi, Y. J.; Nickleach, S.; Socolinsky, D.; Sengamedu, S.; Faloutsos, C.; et al. 2024. Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey. arXiv preprint arXiv:2402.17944
arXiv 2024
Show all 28 references
-
[9]
GLM, T.; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Rojas, D.; Feng, G.; Zhao, H.; Lai, H.; et al. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv preprint arXiv:2406.12793
2024 arXiv
-
[10]
He, X.; Zhou, M.; Xu, X.; Ma, X.; Ding, R.; Du, L.; Gao, Y.; Jia, R.; Chen, X.; Han, S.; et al. 2024. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries. In Proceedings of the AAAI Conference on Artificial Intelligence, 18206...
2024
-
[11]
Herzig, J.; Mueller, T.; Krichene, S.; and Eisenschlos, J. 2021. Open Domain Question Answering over Tables via Dense Retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 512--519
2021
-
[12]
X.; and Wen, J.-R
Jiang, J.; Zhou, K.; Dong, Z.; Ye, K.; Zhao, W. X.; and Wen, J.-R. 2023. StructGPT: A General Framework for Large Language Model to Reason over Structured Data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 9237--9251
2023
-
[13]
Kong, K.; Zhang, J.; Shen, Z.; Srinivasan, B.; Lei, C.; Faloutsos, C.; Rangwala, H.; and Karypis, G. 2024. OpenTab: Advancing Large Language Models as Open-domain Table Reasoners. CoRR, abs/2402.14361
2024 arXiv
-
[14]
Kweon, S.; Kwon, Y.; Cho, S.; Jo, Y.; and Choi, E. 2023. Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table. In Findings of the Association for Computational Linguistics: ACL 2023, 8285--8297
2023
-
[15]
Pal, V.; Yates, A.; Kanoulas, E.; and de Rijke, M. 2023. MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6322--6334
2023
-
[16]
Pasupat, P.; and Liang, P. 2015. Compositional Semantic Parsing on Semi-Structured Tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Paper...
2015
-
[17]
Qin, L.; Wei, F.; Chen, Q.; Zhou, J.; Huang, S.; Si, J.; Lu, W.; and Che, W. 2024. CroPrompt: Cross-task Interactive Prompting for Zero-shot Spoken Language Understanding. arXiv preprint arXiv:2406.10505
2024 arXiv
-
[18]
Qin, L.; Wei, F.; Ni, M.; Zhang, Y.; Che, W.; Li, Y.; and Liu, T. 2022. Multi-domain spoken language understanding using domain-and task-aware parameterization. Transactions on Asian and Low-Resource Language Information Processing, 21(4): 1--17
2022
-
[19]
Qin, L.; Xie, T.; Che, W.; and Liu, T. 2021. A Survey on Spoken Language Understanding: Recent Advances and New Frontiers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021...
2021
-
[20]
A.; and Marcus, M
Ramshaw, L. A.; and Marcus, M. P. 1999. Text chunking using transformation-based learning. In Natural language processing using very large corpora, 157--176. Springer
1999
-
[21]
Robertson, S.; and Zaragoza, H. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval , 3(4): 333--389
2009
-
[22]
E.; and Walker, S
Robertson, S. E.; and Walker, S. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, ...
1994
-
[23]
Shwartz-Ziv, R.; and Armon, A. 2022. Tabular data: Deep learning is not all you need. Information Fusion, 81: 84--90
2022
-
[24]
Sui, Y.; Zou, J.; Zhou, M.; He, X.; Du, L.; Han, S.; and Zhang, D. 2023. Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning. arXiv preprint arXiv:2312.09039
2023 arXiv
-
[25]
Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671
2024 arXiv
-
[26]
Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; et al. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Meth...
2018
-
[27]
Zhong, V.; Xiong, C.; and Socher, R. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103
2017 arXiv
-
[28]
Zhu, Z.; Cheng, X.; An, H.; Wang, Z.; Chen, D.; and Huang, Z. 2024. Zero-Shot Spoken Language Understanding via Large Language Models: A Preliminary Study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation...
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.