{"id":"d574dfd7-d2fe-408c-8d84-58254e01d7ff","arxiv_id":"2506.09657","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"A hybrid open-source LLM pipeline for table QA achieves 80% accuracy on SemEval-2025 Task 8, ranking 13th among open-source systems.","lead":"This paper describes a question-answering system that combines SQL and Python code generation, retrieval, self-correction, and an orchestrator to answer questions over tables. It reports 80% accuracy at SemEval-2025 Task 8, showing that open-source models can compete with proprietary ones on this benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed superiority over GPT-4o rests on an unmatched baseline; GPT-4o with the same pipeline hits 87%, so the open-source advantage is unestablished.","rationale":"The reader's identified weakness is the representativeness of the 100-question dev set, which mainly affects ablations and qualitative conclusions. The official test-set score of 80% is externally validated and does not depend on the dev set. However, the paper's headline claim of 'surpassing GPT-4o' is a dev-set comparison, and it is more directly threatened by the unmatched configurations: 88% for the full open-source pipeline versus 74% for what appears to be bare GPT-4o, despite the paper's own report that GPT-4o with the same pipeline reaches 87%. This is a stronger and more specific concern than representativeness alone. It is load-bearing because the only direct evidence of 'rivaling proprietary models' in the abstract/introduction is this comparison; the global ranking actually shows the system behind several proprietary teams. The concern does not amount to fraud or misreporting of the official result—the authors disclose the 87% GPT-4o-with-pipeline figure—but the abstract's framing overstates the open-source advantage. A conditional acceptance requiring a matched comparison (or a rewording of the claim) would preserve the paper's genuine contribution: a competitive open-source pipeline with a solid official score and honest error analysis. I credit the paper for reporting negative results (reformulation, self-correction) and for making the official evaluation externally verifiable; those elements support acceptance, but the central comparative claim should be corrected.","tokens_in":13797,"tokens_out":5601,"duration_ms":60502,"concrete_test":"Run GPT-4o through the complete pipeline described in Sec. 3, including the Llama-3.3 orchestrator, on the same 100-question dev set used for Fig. 2. If GPT-4o achieves ≥88%, the claim that the open-source system 'surpasses GPT-4o' is an artifact of unequal configurations. Also report GPT-4o with and without each pipeline component so the 74% baseline configuration is explicit.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central assertion that open-source models can rival proprietary models is anchored on the dev-set result '88% accuracy, surpassing GPT-4o’s 74%' (abstract, Sec. 6). But the 88% is the full orchestrated pipeline—two code generators, E2E, RAG, column selection, and a Llama-3.3 orchestrator—whereas the 74% appears to be plain GPT-4o without these components. The paper itself states in Sec. 6: 'Our pipeline applied to GPT4o (w/o orchestrator) also performs well (87%)'. Thus, when GPT-4o is given the same pipeline minus the orchestrator, it nearly matches the open-source 88%. The missing condition—GPT-4o with the full pipeline including the orchestrator—is exactly the experiment needed to test 'surpassing GPT-4o'. Without it, the 14-point gap is attributable mostly to the pipeline, not to the open-source model family. The official test (80%, 20th globally, while the global winner reaches 95%) further tempers the 'rivaling proprietary' claim. This does not invalidate the competition result, but it undermines the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a table QA system developed for SemEval 2025 Task 8 (DataBench). The pipeline combines text-to-SQL and text-to-Pandas code generation, retrieval-augmented generation, an end-to-end answering module, a self-correction mechanism, and a Llama-3.3 orchestrator that selects among candidate answers. On the official test set the system reports 80.08% accuracy, ranking 13th among 38 open-source teams and 20th among 53 teams overall. On a 100-question development set, the full open-source pipeline reaches 88% accuracy, and the paper claims this surpasses GPT-4o's 74%. Ablations examine question reformulation, column selection, column renaming, and the orchestrator's decision behavior, and the paper includes an error analysis of code-based failures.","tokens_in":13973,"tokens_out":10298,"duration_ms":100952,"significance":"If the claims were fully supported, this would be a useful system-description contribution to table QA: it shows an orchestrated open-source pipeline can be competitive in a shared task, it provides a detailed component analysis, and it makes code available. The official competition result is externally validated, and the publicly verifiable test score is a genuine strength. However, the headline comparison against GPT-4o is not a controlled experiment, and the ablation conclusions rest on only 100 development questions without error bars or significance testing. These issues do not invalidate the competition result, but they materially weaken the paper's central 'bridging the gap' claim and require revision before the paper can be accepted.","major_comments":[{"comment":"The headline claim that open-source models surpass GPT-4o (88% vs. 74%) is not supported by a controlled comparison. The 88% is the full open-source pipeline (two code generators, E2E, RAG, column selection, and a Llama-3.3 orchestrator), whereas the 74% appears to be GPT-4o without the pipeline. The paper itself reports in Sec. 6 that 'Our pipeline applied to GPT4o (w/o orchestrator) also performs well (87%)', reducing the gap to one point. The missing condition—GPT-4o with the full pipeline including the orchestrator—is exactly what is needed to test the superiority claim. Without it, the open-source advantage is unestablished; at most, the results support 'comparable' performance under an uncontrolled comparison. Since this comparison appears in the abstract, introduction, and conclusion, it is load-bearing and requires either a matched experiment or a reformulated claim.","section":"Abstract; Sec. 6"},{"comment":"All ablations and qualitative conclusions are based on a development set of only the first 100 QA pairs, with no confidence intervals or significance testing. With n=100, differences of a few percentage points (e.g., the 88% vs. 87% full-pipeline comparison, or the reformulation contrasts such as 68% vs. 84%) are within sampling noise. The official test set provides only the final system's accuracy, so the internal claims about which components help or hurt are not externally validated. The authors should report interval estimates, validate the main ablation conclusions on a larger sample from the benchmark, or explicitly restrict the conclusions to the observed dev subset.","section":"Sec. 4; Sec. 5 (Fig. 2)"},{"comment":"The paper makes contradictory statements about self-correction: Sec. 5.1 says it 'improved accuracy by refining failed queries', while Sec. 8.3 says it was 'largely ineffective' because at least one Python and one SQL agent usually produce a runnable solution. These statements need reconciliation with an ablation that isolates self-correction. Without such an experiment, the contribution of this component is unclear, and the reader cannot tell whether the self-correction mechanism should be retained in the described pipeline.","section":"Sec. 5.1; Sec. 8.3"}],"minor_comments":[{"comment":"There are repeated typos, including 'Codetral' for 'Codestral' (e.g., Sec. 5 and Fig. 2) and 'accuarcy' in Sec. 6; a proofreading pass is needed.","section":"Throughout"},{"comment":"Cross-reference errors: Sec. 3 says 'as shown in Fig. 3' but the system overview is Fig. 1, while the orchestrator decision distribution is Fig. 3; all figure references should be checked.","section":"Sec. 3; Sec. 8.1"},{"comment":"Table 4 reports 'Unique datasets 49' but Sec. 4 says the dataset comprises 65 tables; since 65 × 20 = 1300 QA pairs, the table entry appears to be a typo and should be corrected.","section":"Sec. 4; Table 4"},{"comment":"The abstract and conclusion mention a GitHub repository but no URL is provided in the manuscript; include the repository link for reproducibility.","section":"Abstract; Sec. 11"},{"comment":"The abstract says 'top-13 ranking among the 38 participating teams' but Sec. 7 makes clear this is the open-source-only section; in the global ranking the system is 20th of 53 teams, so the abstract should specify 'open-source' to avoid ambiguity.","section":"Sec. 7"},{"comment":"The paper says the system achieved 80% on both the DataBench evaluation and on the lite part, but no separate lite results are reported; specify the score or remove the claim.","section":"Sec. 7"},{"comment":"The retrieval component does not specify the embedding model or the exact retrieval implementation, which is needed for reproducibility.","section":"Sec. 3.4"},{"comment":"The orchestrator decision analysis is based on 41 questions, but the paper does not explain how these 41 questions were selected from the 100-question development set.","section":"Sec. 8.1"},{"comment":"In the SQL generation prompt, Example 2 uses square brackets around 'salary' while the stated rule says to use double quotes; the example should match the rule.","section":"Fig. 10"}],"recommendation":"major_revision","confidential_remarks":"The competition result is solid, and the paper is a standard system-description contribution. The main barrier to acceptance is the uncontrolled GPT-4o comparison in Sec. 6, which the skeptic's stress-test correctly identified, together with the very small development set for all ablations. Both issues are fixable: the authors can run GPT-4o with the full pipeline including the orchestrator, or they can reframe the abstract and conclusion from 'surpassing' to 'comparable'. The official test score and the public code repository are strengths that should be preserved. I see no circularity or novelty disclosure concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a competent shared-task system paper, and the official result is real: 80% accuracy on DataBench, 13th out of 38 open-source teams. The ablation study is genuinely useful, especially the negative findings — question reformulation hurt, self-correction rarely mattered because multiple code generators usually produced a runnable candidate, and the orchestrator mostly confirmed consensus but occasionally filtered bad outputs. The error analysis is concrete, and the limitations section is honest about cases where all candidates are wrong.\n\nThe main problem is the headline comparison. The abstract and Section 6 claim open-source models 'surpassing GPT-4o' (88% vs 74%), but that compares the full orchestrated open-source pipeline against bare GPT-4o. The paper itself reports GPT-4o with the same pipeline (minus orchestrator) at 87%, so the 14-point gap is mostly pipeline, not model family. The missing condition — GPT-4o with the full pipeline including the orchestrator — is exactly what would test the 'surpassing' claim. As written, the open-source advantage is marginal, not the advertised gap. This is a significant overstatement, but it doesn't invalidate the engineering result: a fully open-source stack, well-orchestrated, gets within a point of GPT-4o on identical scaffolding. That is a worthwhile practical finding.\n\nOther soft spots are minor for a competition report: the dev set is only 100 questions, all ablations and qualitative conclusions rest on it with no error bars or significance testing, and the code URL is promised but not given. The citation pattern looks normal, with relevant prior work and some self-citations that are not problematic.\n\nThis deserves a serious referee. I would send it to review with the expectation that the authors fix the comparison and soften the 'surpassing' language. The competition result is externally validated, and the negative results about reformulation and self-correction are worth recording. For a proceedings audience, this is a useful participant report.","headline":"Solid SemEval system report with an externally validated official score, but the headline 'surpassing GPT-4o' claim rests on an uncontrolled comparison and should be toned down.","tokens_in":14534,"tokens_out":3079,"would_cite":false,"duration_ms":33761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source pipeline with an LLM orchestrator reaches 88% on table QA dev set, beating GPT-4o's 74%, and 80% official.","keywords":["table question answering","large language models","text-to-SQL","text-to-code generation","retrieval-augmented generation","LLM orchestrator","open-source LLMs","DataBench"],"falsifier":"Run the full pipeline on the 400 held-out DataBench QA pairs not used for development: if accuracy falls well below 80% or below a comparably run GPT-4o pipeline, the claim that orchestrated open-source models rival proprietary models would be weakened.","tokens_in":1662,"feed_emoji":"📊","tokens_out":4083,"duration_ms":86576,"temperature":0.7,"pith_summary":"The paper reports a fully open-source system for answering questions over tabular data in SemEval-2025 Task 8. It combines several independent answer generators - SQL queries, Pandas code, and a direct end-to-end model - along with retrieval and a final orchestrator LLM that chooses the most plausible successful answer. On the authors' 100-question development set this setup reaches 88% accuracy, above the 74% they measured for GPT-4o in the same pipeline, and on the official test set it scores 80%, ranking 13th among 38 open-source teams. The authors take this as evidence that orchestrated open-source LLMs can rival proprietary models on structured-data QA.","feed_headline":"Matched GPT-4o on table QA with open-source models","feed_subtitle":"An LLM orchestrator combines SQL, Python, and direct answers to score 80% on DataBench.","key_machinery":"The load-bearing component is the orchestrator, a Llama-3.3-70B instruction-tuned model that receives every successfully executed candidate answer - including the generated code and a truncated result - and must return the number of the most plausible one. It supports the ensemble by resolving conflicts between divergent outputs, rejecting mismatched formats, and confirming consensus, while all other components (retrieval, column selection, code generation, self-correction) feed it candidates.","core_discovery":"The central claim is that an ensemble of open-source LLMs, coordinated by a selection oracle, can match proprietary models on table QA. The system generates candidate answers via text-to-SQL and Pandas code, augments prompts with retrieved rows and predicted relevant columns, includes an end-to-end model for questions that need semantic reading, and then asks a Llama-3.3-70B model to pick the best successful candidate. On the development set the full pipeline reaches 88%, exceeding the 74% achieved by GPT-4o running in the same pipeline, and the official test set score is 80%. The paper argues this narrows the practical gap between open-source and proprietary solutions in this task.","pith_inferences":["Beyond the paper: the 8-point drop from dev (88%) to test (80%) suggests the first-100 dev subset is not fully representative, so readers should treat the dev-set comparisons as diagnostic rather than predictive.","Beyond the paper: the pattern of independent solvers plus a strong selector should transfer to other structured-data QA domains, including database question answering over enterprise tables.","Beyond the paper: replacing the orchestrator with a learned router that predicts whether a question needs SQL, Python, or direct semantic answering is the natural next step, especially because the orchestrator actively adjudicated almost 37% of dev questions."],"forward_implications":["If the claim holds, competitive table QA is achievable without proprietary APIs by combining several open-source generators with an orchestrator.","The ablation results indicate that combining multiple code generators and an end-to-end model improves over any single model, while question reformulation hurts accuracy.","The orchestrator's success depends on at least one correct candidate; when all candidates are wrong, the system cannot recover, as the paper acknowledges.","The official score of 80% versus the 26% stable-code baseline shows a large improvement, though the gap to the top open-source team's 95% remains sizeable."],"supporting_citations":[{"why":"Defines DataBench, the benchmark the system is built for.","marker":"(Osés Grijalba et al., 2024)"},{"why":"Is the SemEval-2025 Task 8 reference that supplies the official test set and leaderboard ranking used for the 80% result.","marker":"(Osés-Grijalba et al., 2025)"},{"why":"Is the GPT-4o system the paper compares against on the development set.","marker":"(OpenAI et al., 2024)"},{"why":"Provides the Llama 3.3-70B orchestrator model.","marker":"(Grattafiori et al., 2024)"},{"why":"Provides Codestral 25.01, one of the code-generation models.","marker":"(Mistral AI Team, 2025)"},{"why":"Provides Qwen2.5-Coder-32B, the other code-generation model.","marker":"(Hui et al., 2024)"},{"why":"Provides MiniMax-01, the end-to-end semantic answerer.","marker":"(MiniMax et al., 2025)"},{"why":"Motivates the retrieval-augmented generation step used to inject relevant rows into prompts.","marker":"(Gao et al., 2024)"}],"fun_headline_variants":["Open-source LLMs match GPT-4o on table QA","Ensemble of open-source models ties GPT-4o in table QA","Orchestrated open-source LLMs reach 80% on table QA","Llama-3.3 orchestrator lifts open-source table QA to 80%","Open-source stack matches proprietary on table QA"],"cache_read_input_tokens":16640,"weakest_assumption_plain":"The load-bearing premise is that the first 100 QA pairs used as the development set are representative of the full DataBench benchmark; if they are not, the reported 88% and the qualitative comparisons against GPT-4o may not hold on the official test distribution.","fun_headline_variants_meta":{"raw":{"variants":["Open-source LLMs match GPT-4o on table QA","Ensemble of open-source models ties GPT-4o in table QA","Orchestrated open-source LLMs reach 80% on table QA","Llama-3.3 orchestrator lifts open-source table QA to 80%","Open-source stack matches proprietary on table QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1255,"prompt_tokens":857,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":307}},"tokens_in":473,"tokens_out":398,"duration_ms":4145,"temperature":1.0,"reasoning_tokens":307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:41:46.215112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on the 400 held-out DataBench QA pairs not used for development: if accuracy falls well below 80% or below a comparably run GPT-4o pipeline, the claim that orchestrated open-source models rival proprietary models would be weakened.","supporting_citations":[],"review_version":1}