REVIEW 4 major objections 3 minor
ST-Raptor claims a tree representation lets LLMs answer questions on complex semi-structured tables, beating nine baselines by up to 20% in accuracy.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
ST-Raptor introduces a tree-based representation and LLM operation pipelines with verification to improve question answering on semi-structured tables.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A plausible engineering contribution to a real bottleneck, but the headline accuracy claim is under-specified and the full evidence isn't available from the abstract alone. the 4 major comments →
ST-Raptor: LLM-Powered Semi-Structured Table Question Answering
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that a tree representation specifically designed for semi-structured tables, called the Hierarchical Orthogonal Tree, enables LLMs to answer questions over tables with complex layouts—hierarchical headers, merged cells, and non-rectangular content—more accurately than existing alternatives. The paper introduces a construction algorithm for this tree, a small set of primitive tree operations that can be composed into pipelines, and a verification mechanism with forward validation (checking each execution step) and backward validation (reconstructing the question from the predicted answer to gauge reliability). The reported result is a 20% accuracy improvement over nine ba
What carries the argument
The Hierarchical Orthogonal Tree (HO-Tree) is the central mechanism: a structural model that encodes a table's layout by separating vertical and horizontal header hierarchies and capturing merged-cell relationships as tree nodes. Its work is to let LLMs reason about a table as a structured object rather than as raw pixels or flattened rows, with a defined set of tree operations (navigate, filter, aggregate) that can be chained into executable pipelines. The two-stage verification—forward and backward—then checks that the pipeline was executed sensibly and that the final answer is plausible by reverse-constructing the query.
Load-bearing premise
The load-bearing premise is that the Hierarchical Orthogonal Tree representation completely captures the layout and semantic information of a semi-structured table that is needed to answer natural language questions—if any layout detail (like a merged cell or a nuanced header relationship) is lost during the tree transformation, all downstream operations and the final answers will be wrong.
What would settle it
A concrete check would be to take a semi-structured table with a complex layout (e.g., a two-level hierarchical header where a column spans multiple subcategories) and a question that specifically depends on that layout, construct the HO-Tree, and see whether the system answers correctly. If a human can answer a question that the tree representation cannot even encode—such as 'what is the total of all cells whose row header is under group A and column header is under group B' in a table where the merged-cell relationship is non-orthogonal or crosses both axes—then the tree construction has los
If this is right
- If the claimed accuracy holds, organizations relying on human analysts to interpret semi-structured tables for financial reports, medical records, or transaction logs could automate a large fraction of routine question answering.
- The HO-Tree representation could be adapted beyond question answering to table summarization, data cleaning, or table-to-text generation, since it preserves layout information that flattened representations discard.
- The two-stage verification approach—especially backward validation by reconstructing the question—could be reused in other LLM pipelines that need to audit whether a computed answer is trustworthy.
- The SSTQA benchmark provides a concrete testbed for future work on semi-structured table QA, giving the community a common set of real tables and questions to measure against.
Where Pith is reading between the lines
- The paper's success depends heavily on the HO-Tree construction being lossless: if there exist real tables whose layout relationships cannot be captured by the tree (for instance, tables with diagonal headers or nested merged cells of unusual shapes), the method would likely fail exactly on those cases, which the 102-table benchmark may not cover.
- The claimed 20% margin could shrink on tables where the layout is simple (e.g., standard relational tables), since the advantage of a specialized tree representation is most visible when other methods break.
- One testable extension is to apply ST-Raptor to tables with merged cells that span multiple rows and columns simultaneously, or to tables where the same value appears in multiple semantically distinct positions, to see whether the tree operations resolve ambiguity as intended.
- Backward validation reconstructs the question from the answer; when reconstruction succeeds for the wrong reason, the mechanism could give false confidence. A stress test would deliberately inject an incorrect tree operation and check whether backward validation always catches it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ST-Raptor, an LLM-based framework for question answering over semi-structured tables (e.g., financial reports, medical records). It introduces a Hierarchical Orthogonal Tree (HO-Tree) to represent complex table layouts, a set of tree operations for QA tasks, and a two-stage verification mechanism. The authors present a new benchmark, SSTQA, with 764 questions over 102 real-world tables, and claim that ST-Raptor outperforms nine baselines by up to 20% in answer accuracy. The code is made publicly available. I was provided only the abstract; the full text was not available, so this report is necessarily provisional and cannot assess the technical or experimental details.
Significance. Semi-structured table QA is practically important, and the proposed HO-Tree representation plus targeted verification is a plausible direction. If the claimed gains are confirmed, the work would be a useful addition to table QA research. The public code link is a strength, and the introduction of a new benchmark addresses a genuine gap. However, the abstract alone does not demonstrate the central empirical claim, and the absence of experimental details prevents a soundness assessment.
major comments (4)
- [Abstract] The central claim, 'outperforms nine baselines by up to 20% in answer accuracy,' is reported as a maximum, not an average or a representative gain. No per-baseline numbers, standard deviations, or statistical significance tests are given. With only 764 questions over 102 tables, the observed differences could fall within sampling noise. The authors should report mean/median gains, per-baseline accuracy tables, and significance tests (or at least confidence intervals) to support the headline claim.
- [Abstract (SSTQA)] The new benchmark SSTQA is introduced without details on data collection, annotation guidelines, question types, or train/test splits. Because the method is evaluated on its own new benchmark, the risk of overfitting to the evaluation set is nontrivial. The paper should describe how the 764 questions were created, whether annotators were independent of the method developers, inter-annotator agreement, and how baselines were configured and tuned.
- [Abstract (HO-Tree)] The load-bearing premise is that the HO-Tree captures complex semi-structured table layouts completely enough for QA. However, the abstract gives no definition of the tree construction algorithm, the set of basic tree operations, or the alignment procedure. No formal characterization or ablation is offered to show that no layout information (e.g., merged cells, hierarchical header relationships) is lost. This premise needs support, either through formal properties of the representation or through experiments that systematically vary layout complexity.
- [Abstract (verification)] The two-stage verification mechanism—forward validation and backward validation—is only named, not described. Since it is a novel component, the paper should specify what each stage checks, how errors are detected, and provide examples of cases where verification catches or misses an error. Without this, the reader cannot judge whether the mechanism contributes meaningfully to accuracy.
minor comments (3)
- [Abstract] The phrase 'up to 20%' is ambiguous: it could mean the maximum improvement over a single baseline on a single subset, or a representative gain. Please report the range and the average improvement explicitly.
- [Abstract] The sentence 'conducts operation-table alignment for accurate pipeline execution' is vague. A compact definition or an example pipeline would clarify the proposed mechanism.
- [General] The full text was not available in the materials provided for review. Please ensure the complete manuscript, including all experimental tables and ablation studies, is supplied for a proper evaluation.
Circularity Check
No circularity identified from the available abstract; the claim is an empirical benchmark result with no self-referential derivation.
full rationale
The available evidence (abstract only) presents an empirical systems paper. ST-Raptor's components—HO-Tree construction, tree operations, two-stage verification, and the SSTQA benchmark—are described as new methods and data, not as quantities fitted to the evaluation outcome. The central claim, 'outperforms nine baselines by up to 20% in answer accuracy,' is an empirical comparison on 764 questions over 102 real-world tables. There is no equation, definition, or cited prior result in the abstract that makes the prediction equivalent to an input by construction. The phrase 'up to 20%' is ambiguous and could reflect a selected best case, but ambiguity about effect size is a reporting/significance concern, not circularity. No self-citation is visible in the abstract, and no claim is justified solely by the authors' prior work. Therefore, no circular step can be quoted or exhibited, and the honest finding is 'no significant circularity' with score 0.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Semi-structured table layouts can be faithfully represented as a Hierarchical Orthogonal Tree without loss of information needed for QA.
- domain assumption Large language models can reliably decompose questions, generate tree operation pipelines, and execute the operations after alignment.
Cite this review
Pith. "Pith review of ST-Raptor: LLM-Powered Semi-Structured Table Question Answering." pith.science (2026). https://pith.science/paper/NCYU2QNE
@misc{pith2026250818190,
author = {Pith},
title = {Pith review of: ST-Raptor: LLM-Powered Semi-Structured Table Question Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/NCYU2QNE}},
note = {Machine review of arXiv:2508.18190}
}
read the original abstract
Semi-structured tables, widely used in real-world applications (e.g., financial reports, medical records, transactional orders), often involve flexible and complex layouts (e.g., hierarchical headers and merged cells). These tables generally rely on human analysts to interpret table layouts and answer relevant natural language questions, which is costly and inefficient. To automate the procedure, existing methods face significant challenges. First, methods like NL2SQL require converting semi-structured tables into structured ones, which often causes substantial information loss. Second, methods like NL2Code and multi-modal LLM QA struggle to understand the complex layouts of semi-structured tables and cannot accurately answer corresponding questions. To this end, we propose ST-Raptor, a tree-based framework for semi-structured table question answering using large language models. First, we introduce the Hierarchical Orthogonal Tree (HO-Tree), a structural model that captures complex semi-structured table layouts, along with an effective algorithm for constructing the tree. Second, we define a set of basic tree operations to guide LLMs in executing common QA tasks. Given a user question, ST-Raptor decomposes it into simpler sub-questions, generates corresponding tree operation pipelines, and conducts operation-table alignment for accurate pipeline execution. Third, we incorporate a two-stage verification mechanism: forward validation checks the correctness of execution steps, while backward validation evaluates answer reliability by reconstructing queries from predicted answers. To benchmark the performance, we present SSTQA, a dataset of 764 questions over 102 real-world semi-structured tables. Experiments show that ST-Raptor outperforms nine baselines by up to 20% in answer accuracy. The code is available at https://github.com/weAIDB/ST-Raptor.
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.