Pith. sign in

REVIEW 4 major objections 5 minor 19 references

MRT at SemEval-2025 Task 8: Maximizing Recovery from Tables with Multiple Steps

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-step explain-then-code pipeline reaches 70.5% accuracy on tabular QA.

desk verdict Solid shared-task system paper with a real reproducibility asset, but the headline 70.50% is not backed by the paper's own test table, and the stated use of the test split for development undermines the central number as reported. read the letter →

arxiv 2505.22264 v1 pith:G5MPRI2R submitted 2025-05-28 cs.CL cs.AIcs.IR

Maximiliano Hormazábal Lagos , Álvaro Bueno Saez , Héctor Cerezo-Costas , Pedro Alonso Doval , Jorge Alcalde Vesteiro This is my paper
classification cs.CLcs.AIcs.IR
keywords tabularquestionansweringSemEval-2025Task8LLMcodegenerationPandasmulti-steppipelineDatabenchanswertypenormalizationerrorrecovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that tabular question answering is better done as a sequence of auditable modules than as one end-to-end LLM call: profile the table, write the reasoning steps in natural language, translate those steps into Python/Pandas code, execute the code, and recover from exceptions by sending the error back to the code generator. Implemented with open-source LLMs (Qwen2.5, Qwen2.5-Coder, Llama-3, Phi-4) and fine-grained prompts per step, the pipeline is reported to reach 70.50% accuracy on Subtask 1 of SemEval-2025 Task 8. The design matters because it makes the link between question and answer inspectable at every stage, and because each module can be tuned or replaced independently.

What carries the argument

The load-bearing mechanism is the Coder–Runner retry loop around a fixed function contract: parse_dataframe(df: pd.DataFrame) -> str. The Explainer's numbered natural-language instructions are injected as the prompt for a code LLM, which must return exactly this function; the Runner executes it and, on exception, appends the traceback to the prompt so the same model can repair the code, with at most three execution retries and up to four LLM parser-assisted syntax fixes before execution. Supporting modules make the loop effective: the Column Descriptor caches per-table statistical and semantic profiles, the Interpreter asks an LLM to fit the output to one of the task's five expected types, and the rule-based Formatter applies final casts such as 2.0 to 2.

What would settle it

Run the released v1.0.0 system end-to-end on the SemEval-2025 Task 8 subtask 1 test split with the same Qwen2.5-14B/Llama-3-8B/Phi-4-14B checkpoints and apply the official scorer; if the result lands near 66.7% instead of 70.50%, the headline accuracy is not reproducible from the documented configuration.

Watch

Extended reading notes

Core claim

MRT's central claim is that decomposing table QA into five stages—column profiling, instruction generation with self-review, code generation, execution with retries, and answer-type normalization—raises the factual accuracy of LLM answers over real-world tables. For each question, the system produces a single Pandas function parse_dataframe(df) and runs it; when the code throws an exception, the Coder module receives the error text and regenerates the function, up to three times. On the Databench-based SemEval-2025 Task 8, the paper reports 70.50% accuracy on subtask 1, and the error analysis attributes the largest share of failures (37.66%) to wrong natural-language instructions, with wrong value filtering adding 14.29%.

Load-bearing premise

The entire headline result rests on the premise that 70.50% came from a valid official evaluation of the system as described; the paper's own tables show at most 66.7% for the ensemble on the test split, and the test split was used during development, so that premise is not established by the written account.

Editorial extensions

If this is right

  • The best documented single configuration (Qwen2.5-14B Explainer, Qwen2.5-Coder-14B) reaches 66.5% test accuracy before formatting; majority voting across Llama-3-8B, Phi-4-14B, and Qwen2.5-14B explainers adds only 0.2 points.
  • List answers are the weak spot: ListNum and ListCat accuracies on the test split are around 0.626 and 0.556, versus 0.829 for Boolean answers, so future work should target list construction and ordering rather than simple lookups.
  • The system's own analysis says most errors come from instruction quality, not code execution, so improving the Explainer and its self-review step should pay off more than improving the Coder.
  • Discouraging groupby in the Coder prompt prevents a major class of runtime exceptions, an example of how prompt-level constraints can substitute for additional training.
  • Formatting heuristics can backfire: on the dev split, accuracy after the Formatter is lower than after the Runner for every configuration, meaning post-processing needs to be evaluated as part of the pipeline, not assumed to help.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never reconciles the headline 70.50% with the 66.7% ensemble accuracy shown in its own Table 5; the most economical reading is that the headline comes from an official or later run not fully documented here, so a reader should treat it as unverified until the released system reproduces it under the official scorer.
  • A value-linking preprocessing module—mapping question mentions such as 'Biden' to table values such as 'Joe Biden' before code generation—could directly attack the 14.29% wrong-value-filtering error class and likely add more accuracy than further model swaps.
  • Because the modules are separately callable, the pipeline doubles as a diagnostic harness: with the Coder fixed, replacing the Explainer across Llama-3-8B, Phi-4-14B, and Qwen2.5-14B moves dev accuracy from 0.606 to 0.766, isolating instruction planning as the largest controllable lever.
  • The same explain-generate-execute-recover scaffold should transfer to SQL-based table QA and multi-table joins, but the retry loop would need an explicit timeout or semantic validator to avoid spinning on questions that are genuinely ambiguous.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes MRT, a multi-step system for SemEval-2025 Task 8 (question answering over tabular data) that combines column description, natural-language instruction generation, Python code generation, execution, and answer formatting using open-source LLMs. The authors report a subtask-1 accuracy of 70.50% on the DataBench test set, and they release the code at a public repository. The paper includes ablation-style results across three LLM configurations on validation and test splits, along with a manual error analysis.

Significance. If the reported 70.50% figure is reliable, the paper documents a competitive open-source LLM pipeline for tabular QA with a clear modular design and public code. The system description is detailed, the module-level breakdown is useful, and the code release is a genuine contribution to reproducibility. However, the central numerical claim is not currently traceable to the described evaluation, which is a correctness issue that must be resolved before the paper's main result can be accepted.

major comments (4)
  1. [Abstract; §1; §6; §5.1, Table 5] The headline claim of 70.50% accuracy on subtask 1 is not supported by the paper's own reported test-split results. Table 5 gives the best ensemble accuracy as 66.7% and the best single model (Qwen2.5 14B) as 66.5%, and no table or sentence reports a run that yields 70.50%. Please identify the exact run, configuration (including tie-breaking and any post-processing), and evaluation source for the 70.50% figure, and reconcile it with Table 5. Without this, the central result is unverifiable.
  2. [§4.1] The statement 'Train, dev and test splits have been used for the development of the modules' indicates that the reported test-split results were used for configuration selection. If the test split was used for development, then the 66.7% figure in Table 5 is not an unbiased estimate of generalization, and the 70.50% claim cannot be interpreted as a held-out performance. Please clarify which results, if any, were obtained on data never used during development, or explicitly state that the reported numbers include test-set-based tuning, with the corresponding caveat in the abstract and conclusion.
  3. [§5.1, Table 5 vs. §4.1] The paper's evaluation protocol is ambiguous about which split is the official evaluation split. Section 4.1 lists train/dev/test splits with 988/320/522 questions, and Section 5.1 presents 'test split' results in Table 5, but no official evaluation or leaderboard results are described. The 70.50% might plausibly come from SemEval's official scorer, yet this is never stated. Please clearly distinguish the locally computed test-split accuracy from the official subtask-1 score, and report both with exact definitions.
  4. [§5.2; Table 6] The manual error analysis reports percentages that do not sum to 100% (14.29 + 37.66 + 14.29 + 6.49 + 3.90 + 23.38 = 100.01, so rounding is likely), and the 'Others' category is large (23.38%). Since 'Others' is described as containing ambiguous questions and possibly incorrect ground truth, this category makes the error attribution less informative. Please state how the manual analysis was conducted (e.g., number of errors examined, annotation procedure, inter-annotator agreement) and provide the raw counts behind the percentages.
minor comments (5)
  1. [§4.2] The sentence 'Llama 32, Phi3 and Qwen4 models of different sizes have been used' contains malformed model names and should be corrected to 'Llama 3.2, Phi-3, and Qwen2.5' for consistency with the rest of the text.
  2. [§3.3] The library 'lib_23' is mentioned without a citation or URL, and the description 'parse python 2 code into python 3' is likely a typo for the `lib2to3` module; please clarify the exact tool used.
  3. [§5.1] The phrase 'After performing the same analysis in the test split (5)' should refer to 'Table 5' explicitly, and similarly the development-split table should be labeled 'Table 3' where it is first mentioned.
  4. [Table 3 and Table 5] The tables would benefit from a column indicating the Confidence or variance across runs, since LLM-based code generation is stochastic and the reported differences between configurations (e.g., 0.665 vs 0.667) may be within run-to-run variation.
  5. [§4.3] Table 2 is described as summarizing configurations, but it omits the Column Descriptor module and the Interpreter details for some experiments; please either add the missing configuration or state explicitly that Table 2 only lists the modules that were varied.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical pipeline evaluated against an external benchmark, and no claim reduces to its own inputs.

full rationale

MRT is described as a modular pipeline of LLM prompts, heuristics, code generation, execution, and formatting modules, and all reported results are accuracy figures on SemEval-2025 Task 8 splits. There is no theoretical derivation, no fitted parameter that is later renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via self-citation. References to prior work (TAPAS, Seq2SQL, TableRAG, Chain-of-Table, DataBench) are standard background and are not load-bearing self-citations. The only internal inconsistency is numerical: the abstract and conclusion claim 70.50% accuracy, while Table 5 reports 0.667 for the best ensemble on the test split and Section 4.1 states that train, dev, and test splits were used for development. That is a reproducibility and reporting-consistency concern, not a circularity concern, because the claimed score is not obtained by construction from the described system or by definitional equivalence to any input. The pipeline's outputs are measured against an external, independently defined benchmark, so the central claim has independent content even if the exact provenance of the 70.50% figure is unclear. Accordingly, no circular step can be exhibited and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The system is empirical; no theoretical derivation. The central claim depends on the test set, the evaluation, and the models. There are no fitted parameters in the scientific sense, but several hand-chosen thresholds and retry limits. No invented entities.

free parameters (4)
  • max_coder_retries = 3
    Maximum number of times the Coder regenerates code after an exception, chosen by hand in Section 3.3.
  • max_revision_attempts = 4
    Maximum attempts to correct syntax via LLM revision in Section 3.3.
  • unique_values_threshold = 7
    Threshold for listing all categorical values in the Column Descriptor prompt, set in Section 3.2.
  • ensemble_tie_break_order = Qwen > Phi-4 > Llama
    Priority given to Qwen2.5 answers when the ensemble vote ties, set in Section 5.1.
assumptions (3)
  • domain assumption LLMs generate useful Python code for table operations given the prompts.
    The whole pipeline assumes the selected open-source models can translate instructions to correct Pandas code, validated empirically in Sections 4.2 and 5.
  • domain assumption The DataBench test set and its gold labels are reliable measures of question-answering performance.
    The reported accuracy is computed against these labels; Section 5.2 even flags some ground truth errors, but the score is taken as valid.
  • domain assumption Python code execution is a sufficient oracle for tabular reasoning.
    The system answers by running generated code, so any question that cannot be expressed as Pandas operations fails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MRT at SemEval-2025 Task 8: Maximizing Recovery from Tables with Multiple Steps." pith.science (2026). https://pith.science/paper/G5MPRI2R

@misc{pith2026250522264,
  author       = {Pith},
  title        = {Pith review of: MRT at SemEval-2025 Task 8: Maximizing Recovery from Tables with Multiple Steps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5MPRI2R}},
  note         = {Machine review of arXiv:2505.22264}
}
abstract

In this paper we expose our approach to solve the \textit{SemEval 2025 Task 8: Question-Answering over Tabular Data} challenge. Our strategy leverages Python code generation with LLMs to interact with the table and get the answer to the questions. The process is composed of multiple steps: understanding the content of the table, generating natural language instructions in the form of steps to follow in order to get the answer, translating these instructions to code, running it and handling potential errors or exceptions. These steps use open source LLMs and fine grained optimized prompts for each task (step). With this approach, we achieved a score of $70.50\%$ for subtask 1.

Figures

Figures reproduced from arXiv: 2505.22264 by the authors.

Figure 1
Figure 1. Diagram of the system showing all the steps [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Examples of outputs of the explainer Input Expected Output "False" Boolean False True Boolean True (unchanged) 1, 21, 14 List of numbers [1, 21, 14] Water, Normal List of strings ["Water", "Normal"] ["16.0", "1.0"] List of numbers [16.0, 1.0] 0.2748 Number 0.2748 (unchanged) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Examples of outputs of the Column Descrip [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 3 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905

    arXiv 2024

  4. [4]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  5. [5]

    Si-An Chen, Lesly Miculicich, Julian Martin Eisenschlos, Zifeng Wang, Zilong Wang, Yanfei Chen, Yasuhisa Fujii, Hsuan-Tien Lin, Chen-Yu Lee, and Tomas Pfister. 2024. http://arxiv.org/abs/2410.04739 Tablerag: Million-token table understanding with language models

    arXiv 2024

  6. [6]

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. http://arxiv.org/abs/1909.02164 Tabfact: A large-scale dataset for table-based fact verification

    arXiv 2020

  7. [7]

    Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. https://doi.org/10.18653/v1/2020.acl-main.398 Tapas: Weakly supervised table parsing via pre-training . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.398 2020

  8. [8]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    arXiv 2024

Show all 19 references
  1. [9]

    Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019. http://arxiv.org/abs/1902.01069 A comprehensive exploration on wikisql with table-aware word contextualization

    2019 arXiv

  2. [10]

    Suvarna Kadam and Vinay Vaidya. 2020. Review and analysis of zero, one and few shot learning approaches. In Intelligent Systems Design and Applications: 18th International Conference on Intelligent Systems Design and Applications (ISDA 2018) held in Vellore, India, December 6-...

    2020

  3. [11]

    Tianyang Liu, Fei Wang, and Muhao Chen. 2024. https://doi.org/10.18653/v1/2024.naacl-long.26 Rethinking tabular data understanding with large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:...

    2024 doi

  4. [12]

    Jorge Os \'e s Grijalba, Luis Alfonso Ure n a-L \'o pez, Eugenio Mart 'i nez C \'a mara, and Jose Camacho-Collados. 2025. S em E val-2025 task 8: Question answering over tabular data. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vien...

    2025

  5. [13]

    Alfonso Ure \ n a-L \'o pez, Eugenio Mart \'i nez C \'a mara, and Jose Camacho-Collados

    Jorge Os \'e s Grijalba, L. Alfonso Ure \ n a-L \'o pez, Eugenio Mart \'i nez C \'a mara, and Jose Camacho-Collados. 2024. https://aclanthology.org/2024.lrec-main.1179/ Question answering over tabular data with D ata B ench: A large-scale empirical evaluation of LLM s . In Pro...

    2024

  6. [14]

    Yucheng Ruan, Xiang Lan, Jingying Ma, Yizhi Dong, Kai He, and Mengling Feng. 2024. http://arxiv.org/abs/2408.10548 Language modeling on tabular data: A survey of foundations, techniques and evolution

    2024 arXiv

  7. [15]

    Yuxiang Wang, Jianzhong Qi, and Junhao Gan. 2025. http://arxiv.org/abs/2410.12846 Accurate and regret-aware numerical problem solver for tabular question answering

    2025 arXiv

  8. [16]

    Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. http://arxiv.org/abs/2401.04398 Chain-of-table: Evolving tables in the reasoning chain for table ...

    2024 arXiv

  9. [17]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  10. [18]

    Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2024 b . Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.05210

    2024 arXiv

  11. [19]

    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. http://arxiv.org/abs/1709.00103 Seq2sql: Generating structured queries from natural language using reinforcement learning

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.