{"id":"1d6f1db4-2bb9-4d78-8d8a-476fbd893d11","arxiv_id":"2412.10104","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RETQA is a new Chinese real estate tabular QA dataset, and SLUTQA, which uses spoken language understanding labels, improves LLM retrieval and answer accuracy on it.","lead":"This paper introduces RETQA, a Chinese question-answering dataset for real estate tables with 20,762 questions over 4,932 tables. It also proposes SLUTQA, a framework that adds intent and slot labels to help large language models retrieve and answer table questions more accurately.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SLUTQA gains are confounded: Vanilla uses random ICL examples while SR/SFA use intent-matched demonstrations, so the reported effect is not isolated to SLU labels.","rationale":"The reader's weakest assumption concerns dataset representativeness; my concern is internal to the evaluation. The two are related but distinct: even if RETQA perfectly mirrors real user queries, the reported experiments would not establish that SLU labels are the cause of the improvements, because the Vanilla baseline and SLUTQA differ in demonstration selection as well as in label availability. Table 5 partially controls for this in the markdown SFA module, but the SQL and retrieval results remain confounded, and Appendix F explicitly concedes the absence of variance or significance testing. The dataset construction is a genuine contribution, and the pipeline may well be effective, but the central attribution to SLU labels should be verified with an intent-matched no-label condition. This does not move the verdict: a conditional acceptance remains appropriate, so I mark UNCHANGED relative to the reader's CONDITIONAL verdict.","tokens_in":671,"tokens_out":855,"duration_ms":107086,"concrete_test":"Run a three-arm controlled comparison on Tables 2-4 and 6 with at least three random seeds and report confidence intervals: (A) Vanilla: random demonstrations, no SLU labels; (B) No-Label Intent-Matched: demonstrations selected by the same predicted intents as SLUTQA but with all intent/slot tokens removed from the SR and SFA prompts, keeping every other pipeline step identical; (C) SLUTQA as reported. If B and C are statistically indistinguishable, the gains come from intent-conditioned demonstration selection rather than from SLU labels; if C significantly exceeds B, the label-attribution survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SLU labels drive SLUTQA's improvements. But the main experiments (Tables 2-4 and 6) compare a Vanilla baseline described as using five randomly selected ICL examples with SLUTQA, whose SR and SFA modules select demonstrations 'sharing the same intent' and place intent/slot tokens in the prompt. This changes two variables at once: the presence of SLU labels and the demonstration-selection policy. Only Table 5's 'Simplified' condition holds the selection policy fixed, and it covers markdown answers only; no such control is reported for SQL generation (Table 6) or for query-summary-based retrieval (Table 4). Appendix F confirms that no repeated runs or significance tests were performed. If a no-label, intent-matched baseline performs as well as SLUTQA, the paper's strongest claim—that SLU labels themselves improve open-domain TQA—is not established, even though the dataset may still be a useful resource.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RETQA, a Chinese open-domain tabular QA dataset for the real estate domain, with 4,932 tables and 20,762 QA pairs, and proposes SLUTQA, a framework that uses spoken language understanding (SLU) labels (intent and slot) to improve retrieval and answer generation with large language models via in-context learning. The dataset is constructed from public real estate data using 90 templates, with reverse-engineered SLU labels and LLM-rewritten queries, and answers are provided in Markdown, SQL, and natural language. Experiments on Qwen2-7b, Qwen2-72b, and GLM4-9b report that SLUTQA improves Markdown table exact match, SQL execution ratio/pass@1, and table retrieval F1 over a vanilla baseline. The paper includes ablations of the retrieval (SR) and answer (SFA) modules, but these ablations do not fully isolate the effect of SLU labels from the demonstration-selection policy, and the paper reports no variance, repeated runs, or significance tests.","tokens_in":14696,"tokens_out":2918,"duration_ms":31594,"significance":"If the empirical claims hold, RETQA would be a substantial new resource: it is the first large-scale open-domain Chinese TQA dataset for real estate, with long tables and multi-table queries, and it is the first TQA dataset to include SLU labels. The dataset construction is described transparently, the data sizes are large, and the code and data are promised for release. The idea of using SLU labels to guide table retrieval and filtering is intuitive and could be useful beyond this domain. However, the central contribution is compromised by the experimental design: the reported gains of SLUTQA over 'Vanilla' change two variables at once (SLU labels and demonstration selection), and the only condition that holds the selection policy fixed is limited to Markdown answers with gold labels. Without a control that isolates the SLU-label signal, the paper's headline claim that SLU labels 'significantly improve' performance is not established.","major_comments":[{"comment":"The main experiments compare a Vanilla baseline that uses 'five randomly selected examples' for ICL with SLUTQA, whose SR and SFA modules select demonstrations 'sharing the same intent' and inject intent/slot tokens into the prompt. This changes two factors at once: the presence of SLU labels and the demonstration-selection policy. Table 5's 'Simplified' condition is the only one that holds the selection policy fixed (intent-matched, no labels), but it is reported only for Markdown answers and uses gold SLU labels rather than predicted ones; no such control is provided for SQL generation (Table 6) or for query-summary-based retrieval (Table 4). A reader therefore cannot tell whether the improvements come from the SLU labels themselves or from better-matched demonstrations. Please add a no-label intent-matched baseline for SQL and retrieval experiments, and if possible a predicted-label condition for the Table 5 ablation.","section":"Experiments, 'Result and Analysis' (Tables 2-4, 6)"},{"comment":"Appendix F states that no multiple runs, mean/variance statistics, or significance tests were performed. This is a load-bearing issue because several reported differences are small (e.g., Table EM improvements of 1.6-4.4 points absolute in Table 2, and retrieval F1 differences of 0.7-2.6 points in Table 4 for ICL vs Vanilla). Without variance estimates or significance tests, the word 'significantly' in the abstract and conclusions is unsupported, and the strength of the central claim cannot be evaluated. At minimum, report results over multiple runs with standard deviations and a paired test (e.g., bootstrap or approximate randomization) for the key comparisons.","section":"Appendix F and Tables 2-3"},{"comment":"The SLU labels are reverse-engineered from the 90 templates, and the table captions are themselves standardized from the same templates (e.g., 'Land Auction Information Table for {city district}'). The SR module generates a query summary from intent/slot values that were derived from the templates, and those slots align exactly with caption entities. This creates a closed loop: the labels are designed to match the retrieval targets, so high retrieval performance with SLU labels may reflect this artificial alignment rather than a general property of SLU-aware retrieval. The paper should either provide evidence that the template-derived labels transfer to naturally occurring user queries (e.g., a small human-written query set with manually annotated SLU labels) or moderate the claim that SLU labels improve open-domain TQA in general, as opposed to on this specific template-aligned benchmark.","section":"Dataset Construction, 'Intent and Slot Annotation' and 'QA Pair Generation'"},{"comment":"The description of the Vanilla baseline is inconsistent. In 'Baselines' the paper says the vanilla approach 'generated a query summary for BM25 retrieval and then produced the final answers using ICL with five randomly selected examples.' But in the SR ablation (Table 4), 'Vanilla' also uses a generated query summary for BM25 retrieval. This means the SR module is being compared against a baseline that already uses the same idea (summary-based retrieval), and the only difference is again the demonstration-selection policy and the presence of SLU labels. Please specify exactly what the Vanilla summary is, how it is generated, and whether the same example-selection policy is used; otherwise the retrieval comparison is not interpretable as an ablation of SLU labels.","section":"Experiments, 'Baselines' and Table 4"}],"minor_comments":[{"comment":"There is a typo: 'within the SLUQTA framework' should be 'SLUTQA framework'.","section":"Method, 'SLU Module'"},{"comment":"The slot type text contains an odd typo: '“month”,’' has an extra quote mark. Please fix.","section":"Dataset Construction, 'Intent and Slot Annotation'"},{"comment":"In the SQL-format description, 'the SLA module for SQL formats' should be 'the SFA module'.","section":"Method, 'SFA Module'"},{"comment":"The abstract says the code is at github.com/jensen-w/RETQA, but the paper's code line says github.com/jensenw1/RETQA; please make these consistent.","section":"Abstract and Code availability"},{"comment":"The legend for Table 1 only explains '–', but the table uses '%' and '!' symbols; please define all symbols in the caption or legend.","section":"Table 1"},{"comment":"The method section says '22 examples' for few-shot SLU prediction, while Appendix D says 'a sufficient number of samples'; please reconcile these numbers or specify the exact setting.","section":"Appendix D and Method, 'SLU Module'"},{"comment":"For multi-intent queries, the selection of 'five examples from the training set that share the same intent' is under-specified: please clarify whether the example must match all predicted intents or any one of them, and how ties are broken.","section":"Method, 'SR Module'"},{"comment":"The dataset comparison in Table 1 would be more informative if it included a Chinese TQA dataset or another open-domain long-table benchmark; currently the comparison is mostly against English datasets, which makes the 'first Chinese' claim harder to evaluate.","section":"Related Works, 'TQA Datasets'"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest in Appendix F about the missing statistical rigor, which is commendable, but the headline claim of 'significantly improves' is not supportable without variance reporting or significance tests. The larger issue is the confound between SLU labels and demonstration selection; this is fixable with additional experiments. The dataset contribution may stand regardless, but the paper should be revised to either add the missing controls or substantially soften the claims about SLU labels being the cause of the improvements. If the authors can run the no-label intent-matched control and report variance, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. RETQA gives the field 20,762 Chinese real estate QA pairs over long, multi-table, open-domain tables, with SLU labels and captions. That is a legitimate new resource, and the paper is honest about its construction and about the fact that no variance or significance tests were run (Appendix F). I think the dataset deserves a serious referee.\n\nWhat I am less convinced by is the method claim. The main experiments compare a Vanilla baseline that uses five randomly selected ICL examples with SLUTQA, whose SR and SFA modules select demonstrations sharing the same intent. That changes two variables at once: presence of SLU labels and the demonstration-selection policy. The stress-test note is right. The only control that holds the selection policy fixed is Table 5's \"Simplified\" condition, and there Vanilla actually beats Simplified on Qwen2-7B, while on the other two models simplified but label-free prompting already captures a large part of the gain. No such matched control exists for SQL generation (Table 6) or for retrieval (Table 4). So the claim that SLU labels themselves drive the improvement is not established. The effect could largely be better demonstration selection.\n\nThere is also a circularity issue, as the reader flagged. The SLU labels are reverse-engineered from the 90 templates, and those templates determine the standardized table captions. The SR module generates a query summary from slots, which is essentially reconstructing the caption from components that came from the caption. That alignment likely inflates the retrieval gains. Real user queries with less clean label-caption correspondence may not benefit the same way.\n\nThe paper does some things well. The dataset statistics are thorough, the three answer formats are useful, and the authors are candid about the missing repeated runs. The BERT-based SLU results in Appendix D look strong, which is plausible. But the current experimental design does not isolate the value of SLU labels.\n\nIf I were the editor, I would send this to review but with a clear request: add an intent-matched, label-free baseline for all answer formats and retrieval, report variance and significance, and ideally include a small set of non-template queries to test the label-caption alignment issue. The dataset itself is probably worth publishing regardless of how the method ablations turn out. The paper is not a desk reject, but the central empirical claim needs rework before it can be accepted as stated.","headline":"Useful new dataset, but SLUTQA's central claim is confounded by demonstration selection; worth sending to review with a request for fixes.","tokens_in":745,"tokens_out":2065,"would_cite":false,"duration_ms":36019,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Intent and slot labels, added to queries as SLU annotations, improve open-domain long-table tabular question answering by guiding retrieval and answer generation.","keywords":["RETQA","tabular question answering","open-domain QA","spoken language understanding","long-table QA","real estate data","in-context learning","BM25 retrieval"],"falsifier":"Run SLUTQA against a vanilla pipeline on a set of naturally collected, non-template real estate queries with human-annotated SLU labels, measuring retrieval F1 and answer exact match; if the SLU-based gains shrink to near the vanilla level, the claim that SLU labels drive the improvement would be refuted. A simpler check is whether an SR summary generated without SLU labels retrieves tables as well as the SLU-labeled version.","tokens_in":14326,"feed_emoji":"🏘️","tokens_out":4053,"duration_ms":41138,"temperature":0.7,"pith_summary":"This paper introduces RETQA, a large-scale open-domain Chinese tabular question-answering dataset for real estate, containing 4,932 tables and 20,762 question-answer pairs across property information, real estate company finance, and land auction domains. The authors argue that existing TQA datasets do not capture the real-world combination of long tables, open-domain retrieval, and multi-table queries, and they propose that adding spoken language understanding labels, namely intent and slot annotations, to each query helps large language models retrieve the right tables and produce accurate answers. To support this, the paper presents the SLUTQA framework, which uses SLU labels to generate table-caption-style query summaries for BM25 retrieval and to filter long tables or generate SQL before answering. Experiments show consistent gains over a vanilla pipeline across three LLMs in both markdown and SQL answer formats. If the claim holds, SLU labels are a cheap, effective annotation signal for open-domain long-table TQA.","feed_headline":"SLU labels lift table QA by 20+ points","feed_subtitle":"New RETQA dataset and SLUTQA framework show intent and slot tags improve retrieval and answers on long real-estate tables.","key_machinery":"The load-bearing object is the SLU label pair attached to each query: an intent category drawn from 16 types and slot tags in Inside-Outside-Beginning format covering city, district, development name, company name, year, and month. The SR module puts these labels into few-shot prompts that rewrite the query into a table-caption-style summary, which is then matched against table captions with BM25; the SFA module uses the same labels to prune irrelevant rows and columns of retrieved long tables or to constrain SQL generation. Labels come either from a fine-tuned BERT model or from in-context learning with a handful of examples.","core_discovery":"On the paper's own terms, the central empirical discovery is that intent and slot labels, treated as extra query annotations and injected into in-context learning prompts, materially improve both table retrieval and answer quality for open-domain long-table TQA. The paper reports that using SLU labels in the retrieval module raises table-retrieval F1 over the vanilla query-summary approach, and that using them in the filtering and SQL-generation module improves Table EM by about four percentage points in markdown format and ECR and pass@1 by more than 20 points in SQL format across Qwen2-7B, Qwen2-72B, and GLM4-9B. The paper frames RETQA as the first TQA dataset to include SLU labels, and SLUTQA as the first framework to combine SLU tasks with TQA tasks, with the claimed benefit coming without fine-tuning the LLM.","pith_inferences":["The reported gains may partly rely on RETQA's construction, where table captions were designed to match the variables in the query templates, making the SLU-to-caption mapping unusually clean; real-world deployment would need to test transfer to naturally collected queries.","SLU labels could be plugged into dense retrieval or embedding-based table search rather than only BM25, since the caption-style summary produced by the SR module is a compact query representation.","The same intent-plus-slot annotation scheme could transfer to other vertical domains with structured tables, such as healthcare or logistics, where long tables and open-domain retrieval also arise.","A testable extension is to measure how sensitive SLUTQA is to SLU prediction errors, since the framework's benefit depends on label quality."],"forward_implications":["RETQA provides a benchmark where open-domain retrieval, long tables averaging 252.9 rows, multi-table queries, and multi-intent queries can be evaluated together in Chinese.","SLUTQA improves table-retrieval F1 and markdown and SQL answer accuracy over a vanilla pipeline on Qwen2-7B, Qwen2-72B, and GLM4-9B without fine-tuning the LLM.","Fine-tuning BERT for SLU prediction yields further gains, showing that better label quality translates into better TQA performance.","SLU-guided row and column filtering reduces the long-table bottleneck for LLMs with limited context windows.","Multi-table and multi-intent queries are explicitly labeled, enabling separate study of retrieval, filtering, and reasoning in open-domain table QA."],"supporting_citations":[{"why":"Supplies the BERT encoder used in the fine-tuning variant of the SLU module for intent and slot prediction.","marker":"(Devlin et al. 2019)"},{"why":"Provides the BM25 probabilistic retrieval framework that the SR module and vanilla baseline use to match query summaries against table captions.","marker":"(Robertson and Zaragoza 2009)"},{"why":"OpenTab is the open-domain table reasoning baseline that motivates the BM25 retrieval setup and the SQL-as-intermediate-step approach.","marker":"(Kong et al. 2024)"},{"why":"Establishes that LLMs can reason over tables via in-context learning, the basis for SLUTQA's few-shot prompting strategy.","marker":"(Chen 2023)"},{"why":"MultiTabQA supplies the multi-table answer generation framework and the row, column, and cell exact-match metrics used for markdown answers.","marker":"(Pal et al. 2023)"},{"why":"Open-WikiTable is the open-domain TQA dataset used as a comparison point, highlighting RETQA's longer tables and SLU labels.","marker":"(Kweon et al. 2023)"},{"why":"NQ-TABLES is the earlier open-domain table QA dataset that frames the retrieval-over-tables problem RETQA extends.","marker":"(Herzig et al. 2021)"},{"why":"Supplies the Qwen2 models used both as the evaluated LLMs and as the query rewriter in dataset construction.","marker":"(Yang et al. 2024)"}],"fun_headline_variants":["SLU labels lift real-estate table QA by 20+ points","RETQA: 20K open-domain QA pairs for real estate","SLUTQA: SLU tags boost table retrieval and answers","First large-scale Chinese table QA for real estate","SLU injection sharpens LLM answers on long tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's queries, built from templates and LLM rewrites, have slot labels that align exactly with table captions and column headers, so real-world user queries that lack this clean alignment may not show the same gains.","fun_headline_variants_meta":{"raw":{"variants":["SLU labels lift real-estate table QA by 20+ points","RETQA: 20K open-domain QA pairs for real estate","SLUTQA: SLU tags boost table retrieval and answers","First large-scale Chinese table QA for real estate","SLU injection sharpens LLM answers on long tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1445,"prompt_tokens":965,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":581,"tokens_out":480,"duration_ms":5435,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:20:31.851719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SLUTQA against a vanilla pipeline on a set of naturally collected, non-template real estate queries with human-annotated SLU labels, measuring retrieval F1 and answer exact match; if the SLU-based gains shrink to near the vanilla level, the claim that SLU labels drive the improvement would be refuted. A simpler check is whether an SR summary generated without SLU labels retrieves tables as well as the SLU-labeled version.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BM25 probabilistic retrieval framework that the SR module and vanilla baseline use to match query summaries against table captions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MultiTabQA supplies the multi-table answer generation framework and the row, column, and cell exact-match metrics used for markdown answers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Open-WikiTable is the open-domain TQA dataset used as a comparison point, highlighting RETQA's longer tables and SLU labels."}],"review_version":1}