{"id":"47bd283e-5dd5-4033-b25a-66352d2de879","arxiv_id":"2508.18190","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"ST-Raptor introduces a tree-based representation and LLM operation pipelines with verification to improve question answering on semi-structured tables.","lead":"ST-Raptor is a new LLM-based system that answers questions about messy, semi-structured tables by turning each table into a tree and running small operation steps. The authors report it beats nine existing methods by up to 20% accuracy and release a new benchmark and code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central accuracy claim rests on an unreported 'up to 20%' comparison; if this is a selected best case, the headline overstates typical gains.","rationale":"The Pith Reader focused on whether the HO-Tree representation completely captures table layout, which is a theoretical assumption about the method's ceiling. While that is relevant, the paper's central claim is empirical: it outperforms nine baselines. The most load-bearing concern is whether that empirical claim is supported by fair, statistically meaningful comparisons. The abstract gives only 'up to 20%' with no per-baseline detail, no error bars, and no significance tests, and the dataset is new and relatively small. The reader's own verdict is UNVERDICTED because the full evidence is unavailable; my concern aligns with that but targets the specific ambiguous 'up to 20%' statement rather than the tree representation. Thus the verdict remains UNVERDICTED/UNCHANGED, pending the concrete reproduction test.","tokens_in":803,"tokens_out":2008,"duration_ms":26379,"concrete_test":"Run the released ST-Raptor code on SSTQA and reproduce the full per-baseline accuracy table. Then compute the mean improvement over all nine baselines, not just the maximum, and apply a paired significance test (e.g., McNemar's test or bootstrap) to each comparison. If the average improvement is near zero or the 'up to 20%' figure is confined to one favorable baseline/question subset, the central claim is overstated; if all nine baseline comparisons show consistent significant gains, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that ST-Raptor 'outperforms nine baselines by up to 20% in answer accuracy' on SSTQA. The phrase 'up to 20%' is ambiguous: it could be the best improvement over a single baseline on a single subset, or it could be a representative mean gain. No per-baseline numbers, standard deviations, or significance tests are given in the abstract. Since the dataset comprises only 764 questions over 102 tables, the observed differences could easily be within sampling noise, especially if the 20% figure is the maximum across many comparisons. The paper introduces its own benchmark (SSTQA), so the risk of the method being tuned to the evaluation set is real unless the questions are independently verified and the baselines are given strong, fair configurations. The load-bearing premise is therefore not the HO-Tree's theoretical completeness but the empirical validity of the reported superiority. If the 20% is a cherry-picked best case and the average improvement over the nine baselines is small or not significant, the paper's central claim fails even if the tree representation is sound. This concern cannot be resolved from the abstract alone; it requires the full experimental table and ideally a reproduction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ST-Raptor, an LLM-based framework for question answering over semi-structured tables (e.g., financial reports, medical records). It introduces a Hierarchical Orthogonal Tree (HO-Tree) to represent complex table layouts, a set of tree operations for QA tasks, and a two-stage verification mechanism. The authors present a new benchmark, SSTQA, with 764 questions over 102 real-world tables, and claim that ST-Raptor outperforms nine baselines by up to 20% in answer accuracy. The code is made publicly available. I was provided only the abstract; the full text was not available, so this report is necessarily provisional and cannot assess the technical or experimental details.","tokens_in":1101,"tokens_out":2571,"duration_ms":32930,"significance":"Semi-structured table QA is practically important, and the proposed HO-Tree representation plus targeted verification is a plausible direction. If the claimed gains are confirmed, the work would be a useful addition to table QA research. The public code link is a strength, and the introduction of a new benchmark addresses a genuine gap. However, the abstract alone does not demonstrate the central empirical claim, and the absence of experimental details prevents a soundness assessment.","major_comments":[{"comment":"The central claim, 'outperforms nine baselines by up to 20% in answer accuracy,' is reported as a maximum, not an average or a representative gain. No per-baseline numbers, standard deviations, or statistical significance tests are given. With only 764 questions over 102 tables, the observed differences could fall within sampling noise. The authors should report mean/median gains, per-baseline accuracy tables, and significance tests (or at least confidence intervals) to support the headline claim.","section":"Abstract"},{"comment":"The new benchmark SSTQA is introduced without details on data collection, annotation guidelines, question types, or train/test splits. Because the method is evaluated on its own new benchmark, the risk of overfitting to the evaluation set is nontrivial. The paper should describe how the 764 questions were created, whether annotators were independent of the method developers, inter-annotator agreement, and how baselines were configured and tuned.","section":"Abstract (SSTQA)"},{"comment":"The load-bearing premise is that the HO-Tree captures complex semi-structured table layouts completely enough for QA. However, the abstract gives no definition of the tree construction algorithm, the set of basic tree operations, or the alignment procedure. No formal characterization or ablation is offered to show that no layout information (e.g., merged cells, hierarchical header relationships) is lost. This premise needs support, either through formal properties of the representation or through experiments that systematically vary layout complexity.","section":"Abstract (HO-Tree)"},{"comment":"The two-stage verification mechanism—forward validation and backward validation—is only named, not described. Since it is a novel component, the paper should specify what each stage checks, how errors are detected, and provide examples of cases where verification catches or misses an error. Without this, the reader cannot judge whether the mechanism contributes meaningfully to accuracy.","section":"Abstract (verification)"}],"minor_comments":[{"comment":"The phrase 'up to 20%' is ambiguous: it could mean the maximum improvement over a single baseline on a single subset, or a representative gain. Please report the range and the average improvement explicitly.","section":"Abstract"},{"comment":"The sentence 'conducts operation-table alignment for accurate pipeline execution' is vague. A compact definition or an example pipeline would clarify the proposed mechanism.","section":"Abstract"},{"comment":"The full text was not available in the materials provided for review. Please ensure the complete manuscript, including all experimental tables and ablation studies, is supplied for a proper evaluation.","section":"General"}],"recommendation":"uncertain","confidential_remarks":"Only the abstract was supplied; the full text appears empty in the review materials. I could not conduct a substantive review of the methods or experiments. A proper assessment requires the complete manuscript, including the dataset description, baseline configurations, and full experimental results. The reported 'up to 20%' claim is not independently verifiable from the abstract alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a straightforward systems paper that takes a known pain point—semi-structured tables with hierarchical headers and merged cells—and offers a concrete LLM-based pipeline. The abstract introduces two things that look genuinely new: the Hierarchical Orthogonal Tree (HO-Tree) representation and a two-stage verification mechanism. That's a reasonable architectural sketch. The SSTQA benchmark, with 764 questions over 102 real tables, is also useful, and the code link suggests a real artifact. I can see a group working on table QA or LLM agents getting value from this, especially if the code is clean and the dataset is released.\n\nWhere I hesitate: the central claim is \"outperforms nine baselines by up to 20%.\" \"Up to\" is doing a lot of work. That could be the best improvement over the weakest baseline, or a subset-specific gain, or nothing near the average. With 764 questions, the differences could be within noise for many baselines. The abstract gives no per-baseline numbers, no confidence intervals, no significance tests, and no description of how baselines were configured. I'd also like the authors to tell us where HO-Tree construction fails—what layout elements break the representation. The novelty claim isn't placed against earlier tree-based table methods or LLM agent frameworks, so it's hard to tell what the actual delta is.\n\nNone of this is fatal. The paper seems well-motivated, the design is coherent, and there's a genuine empirical question that deserved a careful answer. If the full paper provides solid ablations and honest average gains, this is a solid contribution. If the 20% is a cherry-picked best case, the headline misleads.\n\nRecommendation: send it to peer review. The editor should press the authors to clarify the \"up to\" claim, report all baseline numbers, and make the code and dataset truly usable. I'd rather have a referee check the numbers than desk-reject something with this much surface plausibility. For my own work, I wouldn't cite it yet; I need to see the experiments first. But I'd bring it to a reading group to see how the method compares.","headline":"A plausible engineering contribution to a real bottleneck, but the headline accuracy claim is under-specified and the full evidence isn't available from the abstract alone.","tokens_in":717,"tokens_out":682,"would_cite":false,"duration_ms":24741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ST-Raptor claims a tree representation lets LLMs answer questions on complex semi-structured tables, beating nine baselines by up to 20% in accuracy.","keywords":["semi-structured tables","question answering","large language models","hierarchical tree","HO-Tree","table layout","verification","SSTQA"],"falsifier":"A concrete check would be to take a semi-structured table with a complex layout (e.g., a two-level hierarchical header where a column spans multiple subcategories) and a question that specifically depends on that layout, construct the HO-Tree, and see whether the system answers correctly. If a human can answer a question that the tree representation cannot even encode—such as 'what is the total of all cells whose row header is under group A and column header is under group B' in a table where the merged-cell relationship is non-orthogonal or crosses both axes—then the tree construction has los","tokens_in":774,"feed_emoji":"📊","tokens_out":1583,"duration_ms":19530,"temperature":0.7,"pith_summary":"This paper argues that the reason natural-language question answering over real-world semi-structured tables is hard is that existing methods either flatten the table into structured rows (losing layout) or rely on multimodal models that misread flexible layouts. The proposed solution, ST-Raptor, represents a semi-structured table as a Hierarchical Orthogonal Tree (HO-Tree) that preserves header hierarchies and merged-cell structure. Given a question, the system decomposes it into sub-questions, maps each to a sequence of basic tree operations, and executes that pipeline with a two-stage verification step. The authors claim this approach outperforms nine baseline methods by up to 20% in answer accuracy on a new benchmark of 764 questions over 102 real tables.","feed_headline":"Tree representation lifts table question answering by up to 20%","feed_subtitle":"ST-Raptor converts messy semi-structured tables into a hierarchical tree, then verifies answers by reconstructing the question.","key_machinery":"The Hierarchical Orthogonal Tree (HO-Tree) is the central mechanism: a structural model that encodes a table's layout by separating vertical and horizontal header hierarchies and capturing merged-cell relationships as tree nodes. Its work is to let LLMs reason about a table as a structured object rather than as raw pixels or flattened rows, with a defined set of tree operations (navigate, filter, aggregate) that can be chained into executable pipelines. The two-stage verification—forward and backward—then checks that the pipeline was executed sensibly and that the final answer is plausible by reverse-constructing the query.","core_discovery":"The central claim is that a tree representation specifically designed for semi-structured tables, called the Hierarchical Orthogonal Tree, enables LLMs to answer questions over tables with complex layouts—hierarchical headers, merged cells, and non-rectangular content—more accurately than existing alternatives. The paper introduces a construction algorithm for this tree, a small set of primitive tree operations that can be composed into pipelines, and a verification mechanism with forward validation (checking each execution step) and backward validation (reconstructing the question from the predicted answer to gauge reliability). The reported result is a 20% accuracy improvement over nine ba","pith_inferences":["The paper's success depends heavily on the HO-Tree construction being lossless: if there exist real tables whose layout relationships cannot be captured by the tree (for instance, tables with diagonal headers or nested merged cells of unusual shapes), the method would likely fail exactly on those cases, which the 102-table benchmark may not cover.","The claimed 20% margin could shrink on tables where the layout is simple (e.g., standard relational tables), since the advantage of a specialized tree representation is most visible when other methods break.","One testable extension is to apply ST-Raptor to tables with merged cells that span multiple rows and columns simultaneously, or to tables where the same value appears in multiple semantically distinct positions, to see whether the tree operations resolve ambiguity as intended.","Backward validation reconstructs the question from the answer; when reconstruction succeeds for the wrong reason, the mechanism could give false confidence. A stress test would deliberately inject an incorrect tree operation and check whether backward validation always catches it."],"forward_implications":["If the claimed accuracy holds, organizations relying on human analysts to interpret semi-structured tables for financial reports, medical records, or transaction logs could automate a large fraction of routine question answering.","The HO-Tree representation could be adapted beyond question answering to table summarization, data cleaning, or table-to-text generation, since it preserves layout information that flattened representations discard.","The two-stage verification approach—especially backward validation by reconstructing the question—could be reused in other LLM pipelines that need to audit whether a computed answer is trustworthy.","The SSTQA benchmark provides a concrete testbed for future work on semi-structured table QA, giving the community a common set of real tables and questions to measure against."],"supporting_citations":[],"fun_headline_variants":["Tree structure answers messy tables 20% better","LLM tree model boosts table QA accuracy by 20%","Semi-structured table QA gets 20% lift with tree-based LLM","ST-Raptor: tree pipeline for complex tables, 20% accuracy gain","Tree representation lifts table QA by 20%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the Hierarchical Orthogonal Tree representation completely captures the layout and semantic information of a semi-structured table that is needed to answer natural language questions—if any layout detail (like a merged cell or a nuanced header relationship) is lost during the tree transformation, all downstream operations and the final answers will be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Tree structure answers messy tables 20% better","LLM tree model boosts table QA accuracy by 20%","Semi-structured table QA gets 20% lift with tree-based LLM","ST-Raptor: tree pipeline for complex tables, 20% accuracy gain","Tree representation lifts table QA by 20%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2692,"prompt_tokens":830,"completion_tokens":1862,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1775}},"tokens_in":574,"tokens_out":1862,"duration_ms":14772,"temperature":1.0,"reasoning_tokens":1775,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:30:04.032890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to take a semi-structured table with a complex layout (e.g., a two-level hierarchical header where a column spans multiple subcategories) and a question that specifically depends on that layout, construct the HO-Tree, and see whether the system answers correctly. If a human can answer a question that the tree representation cannot even encode—such as 'what is the total of all cells whose row header is under group A and column header is under group B' in a table where the merged-cell relationship is non-orthogonal or crosses both axes—then the tree construction has los","supporting_citations":[],"review_version":1}