{"id":"c770d12c-fb67-48c3-ac04-ff94183980ac","arxiv_id":"2505.24377","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of methods that enhance LLM inference by integrating external structured knowledge from tables and knowledge graphs.","lead":"This survey reviews how large language models can be improved by pulling in external data, especially tables and knowledge graphs. It organizes existing methods into categories and compares their strengths and weaknesses.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 is not a unified benchmark: self-consistency boosts BINDER/DATER, so the 'hybrid is most robust' conclusion is confounded and unsupported by the presented evidence.","rationale":"The reader identified the comparability of Table 1 as the weakest assumption; my stress-test confirms and sharpens it: the self-consistency confound is concrete and measurable. I did not find an additional independent failure of the paper's core taxonomy or review. The taxonomy is reasonable, and the qualitative KG discussion is clearly labeled as such. The paper's own limitations section acknowledges missing controls (table size, multi-hop complexity, table-vs-KG comparison), which further weakens the generality of 'most robust across diverse question types and datasets.' However, because this is a survey and the comparative claim can be corrected by removing the 'unified experimental setting' assertion, adding explicit heterogeneity caveats, or softening the conclusion, the reader's CONDITIONAL verdict remains appropriate. I therefore set verdict_should_be to UNCHANGED.","tokens_in":12921,"tokens_out":5745,"duration_ms":62278,"concrete_test":"Re-run a representative subset of Table 1 (e.g., Text-to-SQL, NormTab, CoT, Chain-of-Table, BINDER, DATER, TabSQLify, H-STAR) on WikiTQ and TabFact under identical conditions: same GPT-3.5-turbo snapshot, same prompt template, same table serialization, and the same decoding budget (e.g., greedy; then a second pass with self-consistency n=20 for every method). If hybrid methods no longer occupy the top ranks in both settings, or if the ordering within hybrid changes when self-consistency is removed, the Section 4.1 conclusion is not robust. A cheaper alternative: check the original BINDER and DATER papers for non-self-consistency numbers and recompute Table 1 with those.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'hybrid reasoning currently delivers the most robust performance across diverse question types and datasets' (Section 4.1) rests entirely on Table 1. That table is not the 'unified experimental setting' promised in Section 4. The footnote explicitly states that BINDER and DATER were reproduced with self-consistency (50 and 20 inference iterations, respectively, on TabFact/WikiTQ), while other methods are reported from their original papers or from Abhyankar et al. (2025), which do not use this procedure. Self-consistency is known to materially improve accuracy for GPT-3.5-class models. Consequently, the hybrid-vs-neural-vs-symbolic gaps in Table 1 are confounded with decoding procedure: BINDER and DATER may have received a boost that other methods — including the neural baseline Chain-of-Table (59.9/80.2) and the hybrid H-STAR (69.6/85.0) — did not receive. The comparison also mixes methods that use greedy decoding, few-shot prompts, and specialized pipelines without controlling for prompt format, table serialization, or sampling budget. The claim of a unified experimental setting is therefore contradicted by the table's own notes, and the conclusion does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey examines methods that integrate external knowledge into LLM inference, with a focus on structured sources: tables and knowledge graphs. It introduces taxonomies (for tables: symbolic, neural, and hybrid reasoning; for KGs: loose and tight coupling), reviews representative methods, and presents comparative tables: Table 1 for table reasoning on WikiTQ and TabFact under GPT-3.5-Turbo, and Table 2 for KG-based methods. The paper concludes that hybrid table reasoning is currently the most robust and that tight coupling in KG reasoning yields more accurate and robust behavior. It also discusses limitations and future directions, including error propagation, input size constraints, efficiency, multimodal knowledge, and real-time reasoning.","tokens_in":13109,"tokens_out":4552,"duration_ms":57987,"significance":"The paper's taxonomy is internally consistent and provides a useful organizational frame for a rapidly growing literature on inference-time knowledge integration. Its coverage of recent table- and KG-based methods, together with the benchmark summary in Appendix A, is a helpful starting point for practitioners. The authors are transparent about several limitations in the final section. However, the paper's comparative claims rest on evidence that is not fully controlled: Table 1 is not the 'unified experimental setting' it claims to be, and Table 2 relies on qualitative, self-reported characterizations. The survey's value is primarily descriptive; its strongest conclusions currently outrun its evidence.","major_comments":[{"comment":"The claim in Section 4 (first paragraph) that methods are benchmarked 'under a unified experimental setting' is contradicted by the table's own footnotes. BINDER and DATER are reported with self-consistency (50 and 20 inference iterations, respectively), while other methods in the same table, including Chain-of-Table, TabSQLify, and H-STAR, are reported from their original papers or from Abhyankar et al. (2025) without that procedure. Since self-consistency is known to improve accuracy for GPT-3.5-class models, the performance gaps underlying the conclusion that 'hybrid reasoning currently delivers the most robust performance' (Section 4.1) are confounded with decoding procedure. The comparison also mixes different prompt formats, table serializations, and sampling budgets. Please either provide a matched-setting comparison, or explicitly restrict the conclusion to the reported settings and discuss the direction and possible size of the confound.","section":"Section 4, Table 1"},{"comment":"The statement that 'tight coupling typically yields more accurate and robust reasoning' is not supported by the evidence in Table 2. The 'Performance' column contains qualitative, largely self-reported characterizations such as 'Better than ToG' and 'Achieves SOTA in multiple datasets,' with several entries marked 'Not mentioned.' No unified benchmark or quantitative comparison is provided for KG-based methods, and the paper's own Limitations section acknowledges that no direct comparative evaluation was performed between table-based and KG-based methods. This comparative claim should be rephrased as a qualitative trend inferred from the cited literature, or supported by a controlled comparison.","section":"Section 3.2.2, Table 2"},{"comment":"The narrative illustration of TabSQLify versus H-STAR in Figure 4 is a useful qualitative example, but it is not evidence that hybrid reasoning 'currently delivers the most robust performance across diverse question types and datasets.' The phrase 'across diverse question types and datasets' is too strong given that Table 1 covers only two datasets and the comparative setting is not controlled. Please temper the conclusion to reflect the scope of the presented evidence.","section":"Section 4.1, Figure 4"}],"minor_comments":[{"comment":"The headings 'KG L LLM' and 'KG N LLM' use nonstandard symbols that are not defined in the text; please replace them with clearer notation such as 'loose coupling (KG + LLM)' and 'tight coupling (KG ⊗ LLM)' or define the symbols at first use.","section":"Section 3.2"},{"comment":"The caption says 'GPT-3.5-Turbo results' but the footnote shows that some numbers are reproduced from other papers that may use different model versions or decoding settings; please state the original source for each row and clarify which numbers are directly reproduced versus newly computed.","section":"Table 1 caption"},{"comment":"The phrase 'our analysis reveals a critical issue about error propagation' is supported by a single qualitative example; please soften it to 'a potential issue' or support it with additional references or a broader analysis.","section":"Section 5, first paragraph"},{"comment":"There is a typographical error in the figure: 'Hybird' should be 'Hybrid'.","section":"Figure 2"},{"comment":"The sentence 'converting their structured relational data into neural representations suitable for LLM' uses a singular 'LLM'; please use 'LLMs' for consistency.","section":"Section 2.2.2"}],"recommendation":"major_revision","confidential_remarks":"This is a useful and honest survey, but the comparative claims need substantial reworking. The main issue is that Table 1 is not a controlled comparison, and the KG comparison in Table 2 is qualitative. The authors could either weaken the claims to match the descriptive nature of the survey or include a genuinely controlled meta-analysis. I would not reject the paper; the taxonomy and benchmark summary are valuable contributions. The paper's self-acknowledged limitations are a positive sign, but they should be reflected more strongly in the abstract and conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serviceable survey of inference-time external-knowledge integration for LLMs, with a clear taxonomy and a wide literature net. But the one conclusion that would make it actionable—hybrid table reasoning is the most robust—doesn't survive its own Table 1.\n\nWhat it does well: the structured/unstructured split, the symbolic/neural/hybrid classification for tables, loose/tight coupling for KGs, and the method-by-method summaries in Section 3 are all readable and mostly accurate. The limitations section is genuinely honest about what the survey doesn't compare (table size, multi-hop complexity, table-vs-KG). That level of candor is rare and worth respecting. There's also a GitHub repo, which helps.\n\nThe soft spot is the comparative analysis in Section 4.1. The text promises a 'unified experimental setting' for Table 1, but the table's own footnote states that BINDER and DATER were reproduced with self-consistency—50 and 20 inference iterations, respectively—while the other entries were taken from their original papers, which did not use that decoding scheme. Self-consistency gives a known boost to GPT-3.5-class models, so the hybrid-vs-neural gap is confounded with inference budget. The claim that hybrid reasoning 'currently delivers the most robust performance' is therefore not supported by the presented evidence. It may be true, but this table doesn't prove it. This is fixable: relabel the table as a compilation of published results, delete 'unified experimental setting,' and soften the conclusion. Better yet, rerun methods under one decoding protocol. The selection of methods also lacks stated inclusion criteria; calling it 'representative' would be more honest than implying completeness.\n\nOn novelty: the taxonomy overlaps substantially with the table-reasoning survey and the KG-LLM roadmap that the paper itself cites. Surveys don't need to invent new categories, and the consolidation is still useful, but the contribution is organizational, not conceptual.\n\nWho this is for: newcomers who want a map of the field and a starting list of methods. It's a solid introduction, not a definitive reference. If this came to me for peer review, I'd send it out—the authors did the cataloguing work, and a referee can insist on fixing the comparative claims. I wouldn't cite its benchmark numbers in my own work until they're regenerated under controlled conditions.","headline":"A serviceable survey of inference-time knowledge integration for LLMs whose practical 'hybrid is best' conclusion is not supported by its own non-comparable benchmark table.","tokens_in":13687,"tokens_out":2931,"would_cite":false,"duration_ms":33101,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that hybrid symbolic-plus-neural reasoning currently delivers the most robust LLM performance on table question answering, and that tightly coupled LLM–knowledge-graph loops yield more accurate and robust reasoning than…","keywords":["large language models","external knowledge","table reasoning","knowledge graphs","hybrid reasoning","loose and tight coupling","retrieval-augmented generation","taxonomy"],"falsifier":"Re-run the GPT-3.5-Turbo comparisons on WikiTQ and TabFact with every method using the same number of inference samples (e.g., greedy decoding for all, or self-consistency for all), and check whether hybrid methods still occupy the top of both leaderboards; if neural or symbolic methods overtake them under matched settings, the paper's central ordering is an artefact of decoding budgets.","tokens_in":12707,"feed_emoji":"📊","tokens_out":4632,"duration_ms":51348,"temperature":0.7,"pith_summary":"The paper offers a systematic map of how external knowledge can be plugged into large language models at inference time, focusing on two structured sources: tables and knowledge graphs. It sorts table methods into symbolic (SQL generation), neural (direct LLM reasoning), and hybrid pipelines, and sorts KG methods by how tightly the LLM and the graph interact. On the basis of benchmark numbers compiled from prior work, it argues that hybrid table methods such as TabSQLify and H-STAR are currently the most robust across question types and datasets, and that tight coupling such as Think-on-Graph tends to beat loose coupling. The survey's practical payoff is a rough design rule: prefer hybrid or tightly coupled integration when accuracy matters, at the cost of higher complexity and latency.","feed_headline":"Hybrid reasoning wins for table-powered LLM QA, survey finds","feed_subtitle":"Review of 13 methods on WikiTQ and TabFact says combining SQL with neural inference beats either alone.","key_machinery":"The load-bearing machinery is the taxonomy itself plus a comparative benchmark table. The survey partitions external knowledge into unstructured and structured data, then further partitions structured integration: for tables, symbolic vs neural vs hybrid reasoning; for knowledge graphs, loose vs tight coupling. These two axes organize the entire method review, and Table 1 (GPT-3.5-Turbo results on WikiTQ and TabFact) and Table 2 (qualitative comparison of KG methods) supply the evidence for the performance ordering. The taxonomy does the argumentative work by turning a scattered literature into a set of design choices with measurable trade-offs.","core_discovery":"The central claim is that the design space of knowledge-enhanced LLM inference can be organized by two axes—structuredness of the knowledge source and coupling/reasoning style—and that along these axes a clear performance ordering emerges. For tables, methods that combine symbolic execution (e.g., SQL) with neural reasoning outperform purely symbolic and purely neural approaches: the best scores in the compiled GPT-3.5-Turbo comparison (H-STAR at 69.6 on WikiTQ and 85.0 on TabFact) come from hybrid designs, and hybrid methods occupy the top of the table. For knowledge graphs, methods that let the LLM iteratively explore and reason over the graph (tight coupling, exemplified by Think-on-Graph and its extensions) yield more accurate and robust reasoning than one-shot retrieval-and-prompting (loose coupling). The paper does not claim hybrid and tight integration dominate unconditionally: it notes the advantage depends on coordination between components and warns that symbolic errors can propagate into the neural stage.","pith_inferences":["The compiled benchmark in Table 1 mixes experiments with and without self-consistency (DATER and BINDER used 20 or 50 inference iterations), so the hybrid-vs-neural ordering may in part reflect decoding budget rather than architecture; rerunning all methods with matched inference settings is a testable check.","The taxonomy suggests a natural extension to multimodal structured data: the same loose/tight coupling distinction could categorize methods that align images or audio with graph entities, a direction the paper lists as open.","The paper's table-centred finding that hybrid methods are most robust on both fact verification (TabFact) and QA (WikiTQ) hints that hybrid coordination may generalize to other symbolic executors, such as Python or SPARQL, beyond SQL.","A direct head-to-head comparison of table vs KG enhancements on the same task suite would clarify when each structured source earns its integration cost."],"forward_implications":["If the paper's comparative conclusion holds, practitioners building table QA systems should start from a hybrid design (SQL-based subtable extraction plus LLM reasoning) rather than pure text-to-SQL or pure end-to-end prompting.","For knowledge-graph QA, investing in an iterative LLM-as-agent loop over the graph should beat a single retrieval-and-prompt pass, and extensions with self-correction (Plan-on-Graph) or internally generated missing facts (Generate-on-Graph) push accuracy further.","The error-propagation finding implies future hybrid systems need fault-tolerant coordination, e.g., letting the neural stage detect and repair bad symbolic intermediate results.","Tightly coupled KG methods can lift smaller models to near the level of larger ones (ToG-2.0 claim), so coupling strategy is a cost lever independent of model size."],"supporting_citations":[{"why":"Defines the symbolic text-to-SQL baseline against which all other table methods are compared.","marker":"(Rajkumar et al., 2022)"},{"why":"Supplies NormTab, a normalized symbolic baseline, and reports the Text-to-SQL scores reproduced in Table 1.","marker":"(Nahid and Rafiei, 2024a)"},{"why":"Supplies Chain-of-Table, the representative neural reasoning method in the table comparison.","marker":"(Wang et al., 2024)"},{"why":"Supplies TabSQLify, a hybrid method central to the comparison, and is the source of the BINDER and DATER results in Table 1.","marker":"(Nahid and Rafiei, 2024b)"},{"why":"Supplies H-STAR, the top-scoring hybrid method, and reports the End-to-End QA, Few-shot QA, and CoT neural baselines.","marker":"(Abhyankar et al., 2025)"},{"why":"Supplies Think-on-Graph, the exemplar of tight coupling that the KG comparison builds on.","marker":"(Sun et al., 2024)"},{"why":"Supplies ToG-2.0, the tight-coupling extension that combines structured and unstructured knowledge and achieves SOTA on 6 of 7 datasets.","marker":"(Ma et al., 2025)"},{"why":"Supplies KAPING, the representative loose-coupling method whose zero-shot gains anchor the loose-vs-tight comparison.","marker":"(Baek et al., 2023)"},{"why":"Supplies Chain-of-Knowledge, the loose-coupling method that verifies intermediate reasoning steps, used to show the trade-off between generality and resource use.","marker":"(Li et al., 2024b)"},{"why":"Supplies Plan-on-Graph, the tight-coupling extension with self-correction that motivates the discussion of adaptive breadth and error recovery.","marker":"(Chen et al., 2024)"}],"fun_headline_variants":["Hybrid beats pure for table-powered LLM QA, survey shows","Tight KG coupling beats one-shot prompting in LLM survey","External knowledge: hybrid and tight integration lead LLMs","Survey: Structured knowledge integration enhances LLM reasoning","LLM table QA: combine SQL with neural for best performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark comparison assumes that the accuracy numbers taken from different papers were produced under the same inference conditions, even though some methods used a self-consistency decoding that reran the model many times and others did not.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid beats pure for table-powered LLM QA, survey shows","Tight KG coupling beats one-shot prompting in LLM survey","External knowledge: hybrid and tight integration lead LLMs","Survey: Structured knowledge integration enhances LLM reasoning","LLM table QA: combine SQL with neural for best performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2312,"prompt_tokens":871,"completion_tokens":1441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":487,"tokens_out":1441,"duration_ms":18969,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:23:04.494311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the GPT-3.5-Turbo comparisons on WikiTQ and TabFact with every method using the same number of inference samples (e.g., greedy decoding for all, or self-consistency for all), and check whether hybrid methods still occupy the top of both leaderboards; if neural or symbolic methods overtake them under matched settings, the paper's central ordering is an artefact of decoding budgets.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies H-STAR, the top-scoring hybrid method, and reports the End-to-End QA, Few-shot QA, and CoT neural baselines."}],"review_version":1}