{"id":"5f397010-ddc2-474e-868a-06b61d4cc35f","arxiv_id":"2412.09884","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"TableQuest evaluates LLMs on table tasks embedded in real 10-K reports and finds that current models handle extraction but struggle with multi-step reasoning, though the evaluation has a self-judging bias.","lead":"This paper introduces TableQuest, a benchmark of 240 question-answer pairs built from S&P 500 annual reports, testing language models on extracting facts, doing calculations, and writing financial analyses. On the tested models, accuracy is modest and drops as questions become harder, but the evaluation is weakened by using GPT-4-Turbo as question generator, judge, and top-scoring model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's central ranking and hard-vs-easy claim rely on GPT-4-Turbo judging answers against gold answers it generated itself, with no independent human check on the eval subset; this circularity leaves the main result unverified.","rationale":"The reader's weakest assumption identifies exactly the load-bearing condition: the synthesized gold answers must be correct and the GPT-4-Turbo judge must be impartial. Both are required for the abstract's claim that models locate facts reasonably but stumble on multi-step reasoning and calculations. Neither condition is established. The only validation studies are off-domain token recall in Appendix F and a small prompt-tuning annotation exercise in Appendix B; neither verifies the final 240-question eval subset or checks judge bias. The Table 3 exact-match accuracy figures actually fail to show a clear easy-versus-hard gap, which makes the ELO table the sole quantitative support for the headline claim, and that ELO table is produced by the circular judge. A concrete human-adjudication experiment on a stratified sample would settle whether the concern lands: if independent human scores reproduce the ranking and difficulty gradient, the central conclusion could be rehabilitated; if not, the paper's main result is an artifact of self-judging. This does not change the reader's reject verdict, but it identifies the precise evidence needed to either redeem or falsify the benchmark's core claim.","tokens_in":9819,"tokens_out":6367,"duration_ms":77265,"concrete_test":"Take a stratified random sample of 80 of the 240 evaluation questions (27 per difficulty), and have three finance-literate annotators independently (a) verify each GPT-4-Turbo gold answer against the original 10-K HTML table, marking it correct, incorrect, or unverifiable, and (b) rank the seven anonymized model responses generated under the paper's exact inference prompt, with model identities hidden, using the same accuracy/clarity/logic rubric. Convert the human rankings into pairwise wins and compute a human-judged ELO and a human-validated accuracy for each model; compare these with Tables 2 and 3. If gpt-4-turbo no longer ranks first, or if the easy-versus-hard gap does not reproduce, the central claim fails as reported; if the ranking and gap survive, the circularity concern is not the decisive issue.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is supported almost entirely by an evaluation loop in which the same model writes the gold answers and judges the responses. The 240 evaluation questions and their gold answers are synthesized by GPT-4-Turbo (Section 2.1, Appendix A), and the judging prompt in Appendix A asks gpt-4-turbo to compare two model predictions against the gold answer using accuracy, clarity, and logicality. For the open-ended analytics questions there is no objective correctness criterion, and the judge has a direct incentive to prefer outputs that resemble its own generated gold answers. GPT-4-Turbo is also itself the top-ranked model in Table 2, so self-preference can inflate its ELO and deflate the other models. Appendix F does not address this: it validates synthesis on 2,100 academic-paper questions using token recall, reports only 0.736 average recall, and never tests judge impartiality. Appendix B human annotation covers only 50 questions per difficulty and was used to refine prompts, not to independently label the final 240-question eval subset. Table 3 further undermines the 'often falter on hard tasks' claim: gpt-4-turbo's hard accuracy (65.625%) is not below its easy accuracy (63.75%), and Meta-Llama-3.1-70B-Instruct has identical accuracy (48.75%) on both, so the difficulty gradient rests on the circular ELO numbers. Finally, despite the abstract's claim that evaluation data are public, the paper provides no link or code, preventing any independent check of the benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TableQuest, a benchmark for evaluating holistic table comprehension in real-world financial reports (S&P 500 10-K filings). The benchmark consists of 240 QA pairs at three difficulty levels (extraction, calculation, analytics), synthesized by GPT-4-Turbo with human-in-the-loop filtering. The authors evaluate seven text-based LLMs and two multimodal variants, reporting ELO ratings computed by a GPT-4-Turbo judge and, in the appendix, accuracy numbers. The central empirical claim is that current LLMs locate facts with reasonable accuracy but often falter on sophisticated reasoning or multi-step calculations. The paper also includes a qualitative failure-mode analysis and a self-supervised validation study on academic papers.","tokens_in":10137,"tokens_out":5079,"duration_ms":51725,"significance":"If the evaluation were trustworthy, TableQuest would fill a real gap: existing table QA benchmarks largely use isolated tables without surrounding context, whereas this benchmark uses full table-text documents. The three-level skill taxonomy and the use of pairwise ELO judging for open-ended analytics are interesting methodological ideas. The paper also ships a qualitative case study and an attempted self-supervised validation of the synthesis pipeline. However, the central empirical results (model ranking and the hard-vs-easy difficulty gradient) rest on a self-evaluation loop in which the same model generates the gold answers, serves as the judge, and tops the leaderboard. The provided validation does not break this loop, and the accuracy/ELO results are internally inconsistent. As a benchmark paper, it also fails to provide the promised public data or code. These issues collectively undermine the paper's main claims as presented.","major_comments":[{"comment":"The evaluation is circular: GPT-4-Turbo synthesizes the questions and reference answers (Section 2.1, Appendix A), serves as the judging model that compares model predictions against those reference answers (Section 3, Appendix A), and is the top-ranked model in Table 2. The judging prompt explicitly asks for 'semantic consistency with the gold answer' and even prefers 'verbatim' matches, creating an incentive for the judge to favor outputs resembling its own generated gold answers. Appendix B's human annotation covered only 50 questions per difficulty and was used to refine prompts, not to independently label the final 240-question evaluation subset. Appendix F validates synthesis on 2,100 academic-paper questions using token recall, but does not test judge impartiality or the correctness of the financial gold answers. The authors must provide an independent human evaluation on a sample of the final eval set, or a bias-controlled study with a blind judge, before the ranking and difficulty-gap claims can be accepted.","section":"§2.1, §3, Appendix A, Table 2"},{"comment":"The reported ELO ratings and accuracy percentages are inconsistent in ways that are not explained. For example, gpt-4o-vision has the highest overall accuracy (70.0%) but a lower overall ELO (892.46) than gpt-4-turbo (ELO 1164.35, accuracy 65.625%). Similarly, Meta-Llama-3.1-70B-Instruct has identical accuracy (48.75%) across overall, easy, and medium columns but its ELO varies substantially across difficulties (880.19, 1008.80, 703.98). If ELO and accuracy are meant to measure related constructs, these discrepancies require an explanation; if they measure different things, the relationship should be stated explicitly. The paper should report per-difficulty accuracy and the exact ELO computation details, including the number of pairwise comparisons per model pair and the handling of ties.","section":"Table 3"},{"comment":"No error bars, confidence intervals, or significance tests are provided anywhere in the paper. With only 80 questions per difficulty level, differences such as the 1.46-point ELO gap between Qwen2-72B-Instruct (867.81) and Meta-Llama-3.1-70B-Instruct (869.27) are almost certainly within noise. The authors should report bootstrap confidence intervals or another uncertainty quantification for both ELO and accuracy, and avoid drawing strong conclusions from small differences.","section":"§3, Table 3, Results"},{"comment":"The self-supervised validation in Appendix F does not validate the central claims. It achieves an average token-recall of only 0.736 on a different domain (academic papers), which is a weak semantic metric and does not establish that the financial gold answers are correct. Moreover, the validation does not assess the impartiality of the GPT-4-Turbo judge. Appendix B's human annotation is limited to 50 questions per difficulty and is explicitly described as used for prompt refinement, not as an independent label of the final 240-question eval set. These gaps leave the correctness of the gold answers and the judge's behavior unverified.","section":"Appendix F and Appendix B"},{"comment":"The abstract and conclusion state that 'the evaluation data, judging procedure and results of this study' are made publicly available, but the paper provides no link, repository, or contact for obtaining them. For a benchmark paper, the data and code are the main artifact; without them, the benchmark cannot be used or independently checked. The authors must provide a working URL or a clear statement of how to access the data and evaluation scripts.","section":"Abstract, §5, Availability"}],"minor_comments":[{"comment":"There is a grammatical error in the sentence 'We observe that among open-source model, However, Meta-Llama-3.1-70B-Instruct...' — the word 'However' is misplaced and the sentence is incomplete.","section":"§4"},{"comment":"The column headers in Table 3 are ambiguous: 'ELO Rating Accuracy (%) Overall Hard Overall Easy Medium' does not clearly map which columns are ELO and which are accuracy. Please label them explicitly, e.g., 'ELO (Overall)', 'ELO (Hard)', 'Accuracy (Overall)', 'Accuracy (Easy)', 'Accuracy (Medium)'.","section":"Table 3"},{"comment":"Reference [20] (MultiTabQA) is missing publication venue and year details, and appears incomplete.","section":"References"},{"comment":"The temperature and other decoding parameters for the GPT-4-Turbo judge are not specified; the authors only state temperature 0.05 for the evaluated models. This should be reported for reproducibility.","section":"Appendix A"},{"comment":"No inter-annotator agreement or quantitative acceptance/rejection rates are reported for the human annotation step, despite the claim of maintaining 'a high standard of accuracy.' Provide numbers such as Cohen's kappa or the fraction of questions edited/rejected.","section":"Appendix B"},{"comment":"The figure caption mentions a sample image rendered from HTML, but the figure itself is not described in the text; please refer to it in Section 1 or 2 and ensure it is legible in the final version.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The paper is a benchmark contribution with a promising task design, but the current manuscript has a fundamental self-evaluation problem that touches every empirical claim, plus incomplete availability information. The lack of a data link is especially problematic for a benchmark paper. If the authors rerun the evaluation with an independent judge and human validation on the final eval set, provide error bars, and release the data, the work could potentially be suitable for a major revision at another venue. In its current form, I cannot recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the benchmark design: real 10-K reports, a three-skill hierarchy (extraction, calculation, analytics), and a progressive multi-round synthesis protocol that builds each question on previous ones. That is a genuinely sensible way to create diverse, context-grounded QA pairs, and the authors earn credit for trying to validate the pipeline with human annotation and a self-supervised recall check on academic papers. If the dataset were released, it would likely be a modest but usable resource for long-context financial table QA. That said, the evaluation that carries the paper's main claim is not reliable. The same model, GPT-4-Turbo, writes the gold answers, serves as the judge for the ELO ranking, and is also the top-ranked model in Table 2. That is a direct self-preference loop, and Appendix F does not break it: the recall of 0.736 on academic-paper questions is weak, and it never tests judge impartiality. The human annotation in Appendix B covered only 50 questions per difficulty and was used to refine prompts, not to independently label the final 240-question eval set. The inconsistencies in Table 3 make things worse: gpt-4o-vision has the highest accuracy (70.0%) but a lower ELO than gpt-4-turbo, whose hard accuracy (65.625%) is actually above its easy accuracy (63.75%). The claim that models \"falter\" on hard tasks rests mainly on ELO numbers that come from the circular judge. There are also no error bars, no human baseline, and despite the abstract saying the data are public, no link or code appears in the paper. These are load-bearing problems for a benchmark paper whose contribution is a set of evaluation numbers. The synthesis protocol itself is novel enough that a serious editor should send it to review, but the evaluation section needs major work: independent human grading or at least a judge that did not generate the gold answers, reconciliation of accuracy with ELO, and the actual data made available. As it stands, I would not cite the ranking results, but I would keep an eye on the dataset if it ever ships.","headline":"A creative benchmark construction undermined by an evaluation loop where the same model writes the gold answers, judges the outputs, and tops the ranking.","tokens_in":10672,"tokens_out":1360,"would_cite":false,"duration_ms":15469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that, in real table-rich financial reports, large language models can locate facts with reasonable accuracy but consistently falter on multi-step calculations and analytical reasoning, and introduces a benchmark…","keywords":["table question answering","LLM evaluation","financial reports","10-K filings","long-context models","table comprehension","benchmark","numerical reasoning"],"falsifier":"Take the 240 evaluation questions and have a panel of financial analysts with the table context produce their own gold answers and rate each model's response, then compare their ratings to the GPT-4-Turbo judge's ELO scores; a substantial mismatch, or a case where the judge prefers a response that contradicts the stated table numbers, would falsify the claim that the ranking measures table comprehension in the wild.","tokens_in":9603,"feed_emoji":"📊","tokens_out":4490,"duration_ms":43347,"temperature":0.7,"pith_summary":"The paper introduces TableQuest, a benchmark for testing whether large language models can read tables the way a human analyst does: not one isolated skill but extraction, calculation, and analytical reasoning together, inside the natural context of real financial filings. The authors compiled question–answer pairs from S&P 500 companies' 10-K reports, organised into easy, medium, and hard tiers, and probed seven state-of-the-art models. Their central finding is that the models do reasonably well at locating facts in a table but often fail at multi-step calculations and at synthesising what the numbers mean. If the benchmark is sound, it gives a more realistic measure of table comprehension than earlier single-skill benchmarks, and it suggests where future work on tabular reasoning should concentrate.","feed_headline":"LLMs find facts in tables but miss the math","feed_subtitle":"TableQuest tests seven models on real 10-K filings and exposes a sharp gap on hard analytical questions.","key_machinery":"The central object is TableQuest itself: question–answer pairs built from SEC EDGAR 10-K HTML tables, generated by GPT-4-Turbo through a progressive multi-round chat that moves from cell extraction (easy) to calculation (medium) to analytical synthesis (hard), then filtered by a hybrid human–machine pipeline that includes finance-expert guidelines and recall-based self-validation. The benchmark carries the argument because it places every question inside the document context where the skill must be exercised, rather than in an isolated table; the three difficulty levels are what let the paper separate fact-location from multi-step reasoning.","core_discovery":"On TableQuest, a dataset of 240 questions drawn from authentic 10-K reports, current LLMs display a split profile: they can extract a cell value or retrieve a fact with reasonable accuracy, but their performance drops when a question demands chained numerical operations or an analytical conclusion. The paper reports ELO scores with GPT-4o as a fixed baseline and GPT-4-Turbo as judge, and the resulting ranking places GPT-4-Turbo and Claude-3-5-Sonnet clearly above the open-source models, with the gap widest on hard questions. The authors interpret the pattern as evidence that holistic table comprehension, where a reader must combine retrieval, arithmetic, and domain awareness in one pass, remains a largely unsolved capability for current models.","pith_inferences":["Because the same model family (GPT-4-Turbo) generated the gold answers and judged the responses, the absolute ELO ordering may partly reflect a model's stylistic affinity to its own outputs; a neutral human re-judging of a sample would test this.","The benchmark's design could be pushed further by forcing models to combine several tables or to challenge an incorrect premise, which would test whether the hard-question deficit is about arithmetic or about cross-table integration.","The reported ELO metric is relative to a GPT-4o baseline with a base score of 1000; swapping the baseline model would change the numbers but, if the paper is right, should preserve the qualitative ordering on hard tasks."],"forward_implications":["Proprietary models such as GPT-4-Turbo and Claude-3-5-Sonnet lead the ELO ranking, and the gap over open-source models is largest on hard analytical questions.","Models such as Gemini-1.5-pro and Meta-Llama-3.1-70B score well on easy and medium questions but lose ground on hard questions, so overall ELO hides weaknesses that the difficulty split exposes.","Retrieving a fact from a table is not the bottleneck; multi-step calculations and query-focused analytical summaries are where current models fail.","Truncating inputs beyond the context limit affected fewer than 2% of samples, so the hard-question gap cannot be blamed on context-length limits.","The self-validation experiment, where the synthesis pipeline was re-run on academic papers and checked by recall against evidence sentences, supports the transferability of the data-generation procedure."],"supporting_citations":[{"why":"Financial numerical-reasoning dataset used to contrast the earlier isolated-table design with TableQuest's in-context benchmark.","marker":"[3]"},{"why":"Hybrid tabular and textual QA benchmark in finance, a narrower predecessor that TableQuest aims to extend.","marker":"[10]"},{"why":"Financial QA benchmark that motivates evaluating models on realistic financial documents.","marker":"[2]"},{"why":"Long-context retrieval format that TableQuest adapts to retrieving information from table cells in real documents.","marker":"[14]"},{"why":"Serving backend used to run the open-weight models included in the evaluation.","marker":"[21]"},{"why":"Long-context dataset used as the source of academic papers in the self-supervised validation of the QA synthesis pipeline.","marker":"[26]"}],"fun_headline_variants":["TableQuest: LLMs lag on multi-step table math","LLMs ace table lookup, flunk table analysis","Benchmark finds LLMs weak on analytical table tasks","LLMs retrieve facts but fail on table reasoning","TableQuest shows LLMs miss the math on real tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic question–answer pairs generated by GPT-4-Turbo are factually correct and that GPT-4-Turbo's judgments of model responses are fair; if either fails, the ranking and the claim about stumbling on hard questions would rest on unstable ground.","fun_headline_variants_meta":{"raw":{"variants":["TableQuest: LLMs lag on multi-step table math","LLMs ace table lookup, flunk table analysis","Benchmark finds LLMs weak on analytical table tasks","LLMs retrieve facts but fail on table reasoning","TableQuest shows LLMs miss the math on real tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2350,"prompt_tokens":930,"completion_tokens":1420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1343}},"tokens_in":546,"tokens_out":1420,"duration_ms":10262,"temperature":1.0,"reasoning_tokens":1343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:36:45.186197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 240 evaluation questions and have a panel of financial analysts with the table context produce their own gold answers and rate each model's response, then compare their ratings to the GPT-4-Turbo judge's ELO scores; a substantial mismatch, or a case where the judge prefers a response that contradicts the stated table numbers, would falsify the claim that the ranking measures table comprehension in the wild.","supporting_citations":[{"cited_title":"FinQA: A Dataset of Numerical Reasoning over Financial Data","cited_arxiv_id":null,"evidence_quote":"Financial numerical-reasoning dataset used to contrast the earlier isolated-table design with TableQuest's in-context benchmark."},{"cited_title":"TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance","cited_arxiv_id":null,"evidence_quote":"Hybrid tabular and textual QA benchmark in finance, a narrower predecessor that TableQuest aims to extend."},{"cited_title":"Needle in a haystack","cited_arxiv_id":null,"evidence_quote":"Long-context retrieval format that TableQuest adapts to retrieving information from table cells in real documents."},{"cited_title":"steps\"), then provide a succinct answer (marked by","cited_arxiv_id":null,"evidence_quote":"Long-context dataset used as the source of academic papers in the self-supervised validation of the QA synthesis pipeline."}],"review_version":1}