{"id":"6402befe-6e46-415d-ad71-30cfefc51a6f","arxiv_id":"2412.08970","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"QueryTableSummarizer++ claims state-of-the-art multi-table query-focused summarization using an LLM trained with table-aware pre-training, fine-tuning, and RL, but the reported results are not reproducible from the paper.","lead":"A new LLM-based framework for query-focused summarization over multiple tables is proposed, combining table-aware pre-training, query-aligned fine-tuning, and reinforcement learning. The paper reports large gains over three baselines on a new benchmark, but provides no code, data, or experimental details needed to verify the claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SOTA rests on an unreleased benchmark and uncited baselines; Table 1's small margins cannot be verified or attributed to the method.","rationale":"The reader's weakest assumption identifies exactly the load-bearing condition: the benchmark must be fair and the baselines must be real, well-configured comparators. My reading agrees. The paper provides no artifacts, no citations for the three baselines, and no dataset description, so the central empirical claim cannot be independently verified. I also note the §3.3 serialization contradiction, which adds correctness risk to the stated methodology but is secondary to the unverifiable comparison. Since the reader already recommends REJECT with moderate confidence, my analysis does not change that verdict. It would move to CONDITIONAL only if the authors release data, code, baseline configurations, and significance tests; until then, the evidence is insufficient to accept the SOTA claim.","tokens_in":7831,"tokens_out":2680,"duration_ms":29359,"concrete_test":"Request the dataset and code, or independently rerun the reported pipeline on the released QFMTS benchmark (Zhang et al., ECAI 2024) using the original implementations of TabFact+T5, MultiTable-BERT, and QuerySummarizer with identical train/validation/test splits and metrics. If QueryTableSummarizer++ does not beat those baselines by a statistically significant margin (e.g., bootstrap confidence intervals on BLEU-4 and ROUGE-L), the Table 1 claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—QueryTableSummarizer++ significantly outperforms state-of-the-art baselines—depends entirely on the empirical comparison in §4.2 and Table 1. For that claim to hold, the new benchmark must be a fair, representative testbed and the three baselines must be strong, optimally configured comparators. Neither condition is currently checkable. The dataset is unreleased, unnamed, and its construction is not described: there is no source data, no size, no train/test split, no annotation protocol, and no inter-annotator agreement. The baselines 'TabFact + T5', 'MultiTable-BERT', and 'QuerySummarizer' are given without citations, model versions, hyperparameters, training configurations, or decoding settings, so there is no way to know whether they were run at their best or even match the named methods. Table 1 also reports no variance, confidence intervals, or significance tests, so the 3.3-point BLEU-4 gap over the best baseline (47.9 vs. 51.2) cannot be distinguished from evaluation noise or benchmark bias. A secondary correctness risk is the internal contradiction between the abstract's claim of 'no intermediate serialization steps' and §3.3, where fine-tuning uses 'a serialized representation of the query Q and the tables T'; this undermines the stated mechanism but does not itself refute the empirical numbers. The empirical claim fails because the evidence needed to verify it is absent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QueryTableSummarizer++, an LLM-based generative framework for query-focused summarization over multiple tables. The method combines a generative likelihood objective (Eq. 1), table-aware pre-training with row-column masking and inter-table relationship prediction (Eqs. 2-3), query-aligned fine-tuning with a contrastive loss (Eq. 4), and a reinforcement-learning objective with a weighted reward (Eqs. 5-7), all combined into a single objective in Eq. 8. The experimental section reports BLEU-4, ROUGE-L, and F1 scores on a self-constructed benchmark, an ablation study, a human evaluation, and analyses of domain generalization, query complexity, and table-count scalability. The central claim is that QueryTableSummarizer++ significantly outperforms state-of-the-art baselines while eliminating intermediate serialization. The stress-test concern that the empirical comparison cannot be verified is confirmed by the manuscript: the benchmark is not described or released, the baselines are not cited or configured, no variance or significance information is given, and the RL reward terms are unspecified.","tokens_in":8252,"tokens_out":10100,"duration_ms":99601,"significance":"If the empirical results were fully documented and reproducible, the paper would address a timely task, and the three-stage training recipe (table-aware pre-training, query-aligned fine-tuning, and RL with feedback) would be a plausible, potentially useful contribution to structured-data summarization. The high-level formulation of the training objectives is clear, and the task itself is relevant. However, the paper as submitted provides no checkable evidence for its main claim: no data, code, or baseline implementations are released; the benchmark is only vaguely described; and the evaluation tables report point estimates without error bars or significance tests. The RL reward functions are not defined, leaving open the possibility that the training objective directly optimizes the evaluation metrics. Because the central SOTA claim rests entirely on Table 1, the significance of the contribution cannot currently be assessed; the paper does not ship reproducible code, data, machine-checked proofs, or parameter-free derivations that would otherwise substantiate the results.","major_comments":[{"comment":"The central claim that QueryTableSummarizer++ significantly outperforms state-of-the-art baselines is not supported by the reported evidence. The benchmark used in §4.1 is described only as consisting of multi-table queries over healthcare, finance, and sports with 2–6 tables; no source data, dataset size, train/test split, annotation protocol, or inter-annotator agreement is given, and no data availability statement is provided. The three baselines (TabFact + T5, MultiTable-BERT, QuerySummarizer) are named without citations, model versions, hyperparameters, or decoding settings, so their strength and configuration cannot be checked. Table 1 reports single point estimates with no error bars, confidence intervals, or significance tests; the 3.3-point BLEU-4 gap over the best baseline (51.2 vs 47.9) is therefore indistinguishable from evaluation noise or dataset bias. Because Table 1 is the only evidence for the SOTA claim in the abstract, the empirical core of the paper is currently unverifiable.","section":"§4.1–4.2, Table 1"},{"comment":"The abstract and §1 say the method eliminates the need for intermediate serialization steps and eliminates the reliance on intermediate preprocessing modules, but §3.3 says the fine-tuning input comprises a serialized representation of the query Q and the tables T. If serialized input is part of the method, the novelty claim must be restated; as written, the method's defining distinction from prior serialization-based approaches is internally inconsistent.","section":"§1 vs. §3.3"},{"comment":"The RL reward R(S) in Eq. (5) is defined as a weighted combination of Relevance(S), Coherence(S), and Brevity(S), but the three functions are not defined and the weights α, β, γ are not reported. If Relevance or Brevity are computed with BLEU, ROUGE, or F1, then the RL stage would be directly optimizing the same metrics used for evaluation in Table 1, making the comparison circular. The paper must specify the reward implementations, the numerical weights, and whether the evaluation metrics were excluded from the reward computation.","section":"§3.4, Eq. (5)"},{"comment":"The human evaluation is reported without the protocol needed to interpret it: the paper does not state how the 20 participants were recruited, whether they rated all systems on the same items, whether annotations were independent, or what statistical test supports the claimed superiority. Table 7 reports only preference percentages (72.4% vs 27.6%) with no test against chance and no inter-annotator agreement for Table 3, so these results do not add independent confirmation of the automated-metric comparisons.","section":"§4.4, Tables 3 and 7"}],"minor_comments":[{"comment":"The claim of improvements of up to 10% is not supported by Table 1, where the largest relative improvement is about 8.3% (ROUGE-L, 49.8 vs 46.0); the numbers in the introduction should be reconciled with the experimental tables.","section":"§1"},{"comment":"The backbone LLM is never identified (e.g., T5, LLaMA, or another architecture); without this information the method is not reproducible and the comparison is hard to interpret.","section":"§4.1"},{"comment":"The similarity function f(Q,T), the negative-sample construction T′, and the temperature τ are not defined or specified, so the contrastive loss in Eq. (4) is not fully specified.","section":"Eq. (4)"},{"comment":"The text says the removed components significantly impact performance, but no significance tests are reported for the ablation results; this wording should be qualified or supported with statistics.","section":"§4.3, Table 2"}],"recommendation":"reject","confidential_remarks":"To the editor: I would not send this manuscript back for a standard revision cycle without first requiring a complete data and software release and a full description of the baseline implementations. The reference list also contains a large number of self-citations whose relevance to multi-table summarization is not clear; it would be worth asking the authors to trim or justify them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible pipeline paper whose central empirical claim cannot be verified. The task is real, the architecture is sensible, but the paper withholds the evidence needed to take the numbers seriously.\n\nWhat it does well: the problem is well motivated, and the method is a clean combination of established techniques—masked cell prediction, inter-table relation classification, contrastive learning, and RL. The equations are standard and the ablations show each component contributes. The human evaluation, though small, is a reasonable extra signal. I also appreciate that the paper cites QFMTS and QTSumm, the prior work that defines the task.\n\nThe problems are in the evidence. The benchmark is unreleased, unnamed, and described in three sentences: no size, no annotation protocol, no splits, no inter-annotator agreement. The three baselines—TabFact + T5, MultiTable-BERT, QuerySummarizer—are given without citations, versions, hyperparameters, or decoding settings. There are no error bars or significance tests, so the 3.3-point BLEU-4 lead over the best baseline could be evaluation noise. More damning: the abstract claims the method 'eliminates the need for intermediate serialization steps,' but §3.3 says the fine-tuning input is 'a serialized representation of the query Q and the tables T.' That is a direct contradiction, and it undercuts a core selling point. The RL reward weights (α, β, γ) are unspecified, so if the reward terms overlap with BLEU/ROUGE/F1, the RL stage could be optimizing the test metric directly. The paper never addresses that circularity.\n\nI agree with the stress-test: the empirical claim fails because the evidence for it is absent. This isn't a flawed-but-checkable experiment; it's a case where the authors ask the reader to accept the numbers on faith. The method itself might be fine—the pieces are known to work in other contexts—but without the data, baseline configurations, and a resolution of the serialization contradiction, there is nothing to referee.\n\nWho is this for? As it stands, no one. It could be a cautionary example in a discussion of evaluation practices, but that's not the intended audience. A serious editor should desk reject this. If the authors release the dataset, properly configure and name the baselines, and fix the serialization claim, it could become a modest incremental contribution worth a full review.","headline":"Plausible pipeline paper whose central empirical claim is unverifiable: unreleased benchmark, unnamed baselines, and an abstract that contradicts its own method section.","tokens_in":8619,"tokens_out":3064,"would_cite":false,"duration_ms":28040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An end-to-end LLM trained with table-aware pre-training, query-aligned fine-tuning, and reinforcement learning can generate query-relevant multi-table summaries, outperforming three baselines on a new benchmark.","keywords":["query-focused summarization","multi-table data","large language models","table-aware pre-training","reinforcement learning with feedback","table reasoning","generative summarization","benchmark dataset"],"falsifier":"Evaluate the same training recipe on an existing public multi-table summarization benchmark, such as QTSumm or QFMTS, against the original published baseline systems; if the roughly three-point BLEU-4 margin over the best baseline shrinks or reverses, the claim of state-of-the-art superiority would collapse.","tokens_in":7603,"feed_emoji":"📊","tokens_out":10061,"duration_ms":91371,"temperature":0.7,"pith_summary":"Query-focused summarization over multiple tables is usually tackled as a pipeline of table serialization, retrieval, and reasoning modules. This paper argues that a single large language model can learn the whole task end-to-end if it is first trained to understand table structure and inter-table relationships, then fine-tuned on query-aligned summaries, then refined with reinforcement learning that rewards relevant, coherent, and brief outputs. The proposed system, QueryTableSummarizer++, reports BLEU-4 of 51.2%, ROUGE-L of 49.8%, and F1 of 48.5% on a new benchmark spanning healthcare, finance, and sports, ahead of three baselines. If the results are right, multi-table question answering and analytics reporting become a single generation problem rather than a chain of specialized components.","feed_headline":"End-to-end LLM pipeline tops multi-table summarization baselines","feed_subtitle":"QueryTableSummarizer++ posts top BLEU, ROUGE, and F1 scores on a new healthcare, finance, and sports benchmark.","key_machinery":"The mechanism that carries the argument is the composite training objective. It combines a generative likelihood, a table-aware pre-training loss comprising row-column cell masking and binary inter-table relationship prediction, a contrastive loss that aligns the query with the correct table set and repels a negative sample, and a policy-gradient reinforcement learning term whose reward is a weighted sum of relevance, coherence, and brevity scores. This joint objective is what lets a generic LLM reason across tables, and the ablation study attributes the metric gains to the pre-training and RL components.","core_discovery":"The paper's central claim, stated in its own terms, is that an end-to-end generative framework built on a large language model and enhanced with table-aware pre-training, query-aligned fine-tuning, and reinforcement learning with feedback achieves state-of-the-art performance for query-focused summarization over multi-table data. The model takes the query and the tables as input and directly generates the summary, removing the intermediate serialization and reasoning modules used by earlier systems. Across a benchmark of multi-table queries involving 2-6 tables, the full model beats TabFact+T5, MultiTable-BERT, and QuerySummarizer on BLEU-4, ROUGE-L, and F1, with particularly clear advantages on complex queries and as the number of tables grows; ablations show that removing either the table-aware pre-training stage or the reinforcement learning stage lowers all three metrics.","pith_inferences":["If the gains replicate on public benchmarks, table-aware pre-training with inter-table relationship prediction could become a default auxiliary objective for fine-tuning LLMs on tabular data, well beyond summarization.","The paper does not ablate the relationship-prediction loss separately from cell masking, so an untested question is whether inter-table relationship modeling is the true driver of the complex-query gains; a targeted ablation would settle that.","Since the RL reward is built from the same automated metrics used for evaluation, part of the reported improvement may come from tuning toward BLEU, ROUGE, and F1; a testable extension is to swap in faithfulness or factuality rewards and re-measure.","The 'up to 10%' improvement mixes different baseline comparisons; a fairer public comparison would report per-baseline margins with variance across multiple dataset splits."],"forward_implications":["Multi-table summarization can be treated as a pure text generation problem, so practitioners can drop hand-built serialization and reasoning pipelines in favor of a single trained LLM.","The four-term training objective is portable: the same masking, relationship-prediction, contrastive, and RL setup could be applied to other structured-data-to-text tasks such as database question answering and automated report generation.","Because the model's relative advantage grows on complex queries and on six-table inputs, the approach is pointed at realistic analytics settings where users ask questions across many tables.","Human preference results, with 72.4% of pairwise judgments favoring the new model over the best baseline, suggest the output quality advantage is visible to readers, not only in automated scores."],"supporting_citations":[{"why":"Defines the query-focused multi-table summarization task and supplies a prior method, QFMTS, that this paper positions against.","marker":"Zhang et al., 2024"},{"why":"Introduces QTSumm, a query-focused table summarization dataset that motivates the benchmark design here.","marker":"Zhao et al., 2023b"},{"why":"Provides a reasoning-aware table-to-text baseline, ReTAG, that this paper contrasts with its end-to-end approach.","marker":"Ghosal et al., 2023"},{"why":"Presents a question-then-pinpoint reasoner for table summarization, representing an alternative to the paper's direct generation strategy.","marker":"Seo et al., 2024"},{"why":"Supplies a prior dataset and method for long-text and multi-table summarization, supporting the paper's scalability claims.","marker":"Liu et al., 2023"}],"fun_headline_variants":["LLM skips serialization, tops query-table summarization","LLM with RL feedback dominates multi-table query summarization","End-to-end LLM framework sets new SOTA for multi-table summaries","Query-aware LLM summarizer beats prior art on multi-table queries","LLM summarization without serialization beats multi-table baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the newly constructed benchmark is representative and that the three named-but-uncited baselines were configured at their best; if the dataset is biased or the baselines are weaker than the published state of the art, the reported BLEU, ROUGE, and F1 gains would not reflect genuine superiority.","fun_headline_variants_meta":{"raw":{"variants":["LLM skips serialization, tops query-table summarization","LLM with RL feedback dominates multi-table query summarization","End-to-end LLM framework sets new SOTA for multi-table summaries","Query-aware LLM summarizer beats prior art on multi-table queries","LLM summarization without serialization beats multi-table baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3551,"prompt_tokens":894,"completion_tokens":2657,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2569}},"tokens_in":510,"tokens_out":2657,"duration_ms":20296,"temperature":1.0,"reasoning_tokens":2569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:49.806346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same training recipe on an existing public multi-table summarization benchmark, such as QTSumm or QFMTS, against the original published baseline systems; if the roughly three-point BLEU-4 margin over the best baseline shrinks or reverses, the claim of state-of-the-art superiority would collapse.","supporting_citations":[{"cited_title":"Retag: Reasoning aware table to analytic text generation","cited_arxiv_id":null,"evidence_quote":"Provides a reasoning-aware table-to-text baseline, ReTAG, that this paper contrasts with its end-to-end approach."},{"cited_title":"Unveiling implicit table knowledge with question-then-pinpoint reasoner for insightful table summarization","cited_arxiv_id":null,"evidence_quote":"Presents a question-then-pinpoint reasoner for table summarization, representing an alternative to the paper's direct generation strategy."}],"review_version":1}