Pith. sign in

REVIEW 3 major objections 5 minor 21 references

MRT at IberLEF-2025 PRESTA Task: Maximizing Recovery from Tables with Multiple Steps

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A modular pipeline generates Python code to answer 85% of Spanish table questions.

desk verdict Honest shared-task system description; the 85% score is a real benchmark result, but the table overlap and weak ablation mean it is not evidence that the new modules generalize. read the letter →

arxiv 2507.12981 v1 pith:UXMTVAG5 submitted 2025-07-17 cs.CL cs.AI

Maximiliano Hormazábal Lagos , Álvaro Bueno Sáez , Héctor Cerezo-Costas , Pedro Alonso Doval , Jorge Alcalde Vesteiro This is my paper
classification cs.CLcs.AI
keywords TableQuestionAnsweringLargeLanguageModelsCodeGenerationSpanishPRESTAColumnSelectionFuzzyMatchingMulti-steppipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes a modular system for answering questions over Spanish survey tables. The system decomposes each question into a sequence of steps—selecting relevant columns, writing natural-language instructions, generating and executing Python code, and formatting the answer—using open-source 14B-parameter language models. The authors report an accuracy of 85% on the PRESTA test set of the IberLEF 2025 shared task, a setting where tables average 174 columns. The central claim is that this divide-and-conquer, code-generation approach can handle very wide and ambiguously labeled tables while remaining explainable, because every intermediate instruction and code fragment can be inspected.

What carries the argument

The object that carries the argument is the MRT pipeline itself: a multi-step 'maximizing recovery from tables' system in which each module has one narrow task. A column selector asks an LLM, in batches of 25 columns, which columns might matter for the question; the explainer then produces a JSON of natural-language instructions, listed column names, and filter values, with Levenshtein-based correction of misspelled column names and added clarification lines for value mismatches; the coder, prompted with a library of pre-coded generic functions, writes Python that the runner executes, retrying on exceptions; and the interpreter/formatter converts the output to the expected type. The custom function library—ranging from counting occurrences and filtering by numeric thresholds to fuzzy 'contains' matching—is what lets the coder focus on orchestration instead of low-level pandas logic, and it is derived from the training tables.

What would settle it

Run the same pipeline, with the hand-built column-removal rules disabled, on a newly collected set of Spanish survey tables that were not available during development; if accuracy drops well below the reported 85%, the headline number was inflated by the train/test table overlap.

Watch

Extended reading notes

Core claim

The central claim is that MRT, a sequential pipeline that compiles natural-language instructions into executable Python, achieves 85% accuracy on Spanish table question answering despite tables that average 174.1 columns. The paper attributes this result to three new components over its predecessor: an LLM-based column selector that prunes the prompt to groups of 25 columns, a library of pre-coded generic Python functions that steer the coder away from error-prone pandas usage, and clarification instructions plus fuzzy matching that reconcile how values appear in the question with how they are stored in the table. The authors also report that the pipeline runs about three times faster with column selection, and that a majority-vote ensemble over eight runs stabilises the output. They frame the main benefit as explainability: errors can be traced to specific instructions or code, and the system needs no fine-tuning, only prompt engineering around open-weight models.

Load-bearing premise

The system's 85% accuracy is measured on a test set whose tables are the same tables used for training and development, and the hand-written column-removal rules and pre-coded functions were built from those tables, so the score may not hold on tables the system has never seen.

Editorial extensions

If this is right

  • Wide tables, around 174 columns on average, can be processed with small prompts because the column selector cuts the table before any instruction generation; the whole pipeline also runs about three times faster.
  • The main remaining error sources become reachable and debuggable: wrong natural-language instructions and removal of a relevant column account for most failures, while code exceptions and wrong cell filtering drop sharply.
  • An ensemble with majority voting over five to eight runs gives stable answers and is treated as sufficient for the system's use.
  • Open-source 14B models, one of them a code-specialised variant, can match the task without any fine-tuning, which lowers the barrier to applying the approach elsewhere.
  • The gap between validation (71%) and test (85%) suggests the test questions are simpler, so the system's headline score should be read together with the validation number.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the PRESTA test tables are the same tables used for training and development, the 85% figure likely overstates how the system would perform on tables it has never seen; the durable contribution is probably the pipeline design rather than the score.
  • The hand-built column-removal rules and the semi-automated function library were created from the training tables, so on genuinely new table schemas those modules would need to be rebuilt or revalidated; fuzzy matching is the component most likely to transfer as-is.
  • The trick of presenting pre-coded Python functions as tool-calling options to the code model is a cheap way to inject domain knowledge into an LLM pipeline and could be tested on other structured-data tasks such as SQL generation or spreadsheet automation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents MRT, a modular table question-answering system submitted to the IberLEF 2025 PRESTA task. The pipeline uses open-source LLMs to select columns, generate natural-language instructions, translate these into Python code, execute the code, and format the answer. The authors add new components to an earlier SemEval system: a column selector, rules for removing uninformative columns, clarification instructions, custom code-generation functions, and fuzzy matching for categorical values. On the official PRESTA test set the system achieves 85% accuracy, placing third. The paper also reports a dev-set ablation study and a manual error analysis, and it explicitly acknowledges that the dataset is too small and the ablation differences too slight to confirm the benefit of the new modules.

Significance. If the 85% test score is taken as evidence of a generalizable system, the paper is a useful contribution to LLM-based table QA in Spanish, particularly for wide tables with ambiguous column names. The system is reproducible: the code is publicly available, the models are open 14B-parameter checkpoints, and the experimental protocol (8 repetitions, majority voting) is clearly described. The modular design is a strength for explainability and debugging. However, the evidential value of the headline result is substantially weakened by the overlap between test tables and development tables, and the ablation study does not demonstrate that the new components improve accuracy. The paper is honest about these limitations, but the central claim of the title and abstract—that the new features enable 'maximizing recovery'—is not firmly supported by the evidence presented.

major comments (3)
  1. [§5.1, Table 2; §4.2.2; §4.2.4] The test set does not contain unseen tables: Section 5.1 states that 'The tables used in the test consist of the same tables of the train and the dev splits,' and Table 2 confirms that the 10 test tables are the union of the 6 training and 4 dev tables. Because the handcrafted column-removal rules in §4.2.2 (e.g., dropping 'N_R_' and 'Ns_Nc_*' columns) and the custom function pool in §4.2.4 were derived from the training split and expanded during a 'testing and debugging phase,' these components can be applied to the exact table schemas and quirks that appear in the test set. The 85% score therefore reflects performance on tables that were available during system development, not generalization to unseen tables. I request a leave-one-table-out evaluation or an evaluation on genuinely held-out tables to support the claim that the design choices, rather than table-specific tuning, drive the result.
  2. [§6.4, Table 6] The ablation study does not confirm that the new modules improve accuracy. Removing the column selector raises the dev score from 0.71 to 0.74, and removing custom functions gives 0.72, both equal to or better than the full system; only removing fuzzy substitution decreases the score (0.69). With only 100 dev questions and 4 tables, these differences are within random variation, a point the authors themselves make in the text. Consequently, the paper's emphasis on the new features as the source of the test-set success is not supported by the reported data. The authors should either provide additional evidence (e.g., statistical significance testing, a larger dev set, or a per-table breakdown) or explicitly reframe the contribution as a systems description rather than a validated improvement.
  3. [§6.3] The sentence 'we can prove the benefits of some of our new features' overstates what the manual error analysis shows. The analysis is a qualitative attribution of error categories on the dev set; it does not compare against a counterfactual without the new features. This statement is directly contradicted by the ablation results in Table 6, which show no consistent benefit. Please replace 'prove' with a more cautious formulation, such as 'provide qualitative evidence consistent with,' and reconcile this paragraph with the ablation discussion in §6.4.
minor comments (5)
  1. [§7] The conclusion states 'a 0.85% of accuracy'; this should read '85% accuracy' to avoid confusion with 0.85 percent.
  2. [§4.1] The phrase 'the PRESTA dataset patented new limitations' appears to be a typo; 'patented' should likely be 'presented' or 'posed.'
  3. [Table 6] The column headers contain a duplicated 'List' entry ('List List'); the header row should be cleaned to match the answer types listed in Table 3.
  4. [§6.2] The sentence 'It matches our intuition that is that the test set questions are in average simpler than the validation set questions' is grammatically awkward and should be revised.
  5. [Appendix B] The function `_round_was_useful` is called in the fuzzy matching code but is not defined in the appendix; please either include its definition or add a comment explaining its intended behavior.

Circularity Check

2 steps flagged · score 4.0 of 10

Test-set tables are identical to train/dev tables, so hand-built column-removal rules and custom functions transfer to the test set; the reported 85% is partly a table-specific fit, though test questions are unseen.

  1. fitted input called prediction [Sections 4.2.4, 5.1, 6.2]
    "Section 5.1: "The tables used in the test consist of the same tables of the train and the dev splits." Section 4.2.4: "These functions were obtained from the training split." Section 6.2: "They achieve scores of 85%, 85%, and 83% respectively.""

    The paper reports test accuracy on tables that were available during development. The custom function pool was explicitly derived from the training split, which contains the same tables as the test split, so the functions encode table-specific quirks and can be applied verbatim at test time. The 85% figure is therefore not an unbiased estimate of performance on unseen table schemas; part of the claimed success reduces to fitting module behavior to the exact tables in the test split. Because the test questions are new, the reduction is partial rather than total.

  2. fitted input called prediction [Sections 4.2.2, 5.1, 6.2]
    "Section 4.2.2: "We identified exceptional columns that led to significant and recurrent errors, that we consider are not needed in any of the questions seen... This is the case of 'N_R_'..." and "there is one table that has columns identified as 'Ns_Nc_0', 'Ns_Nc_1', 'Ns_Nc_2', etc.""

    The rules for removing uninformative columns were hard-coded by inspecting the training tables, including column names specific to particular tables (e.g., 'Ns_Nc_0'). Because Section 5.1 states that test tables are the same as train/dev tables, these removal rules apply unchanged to the test split. The improvement attributed to the system is thus partly due to developer knowledge of the exact test-table schemas rather than to a generalizable mechanism, inflating the evidentiary value of the claimed accuracy for unseen tables.

full rationale

The core reported result (85% test accuracy) is measured by the official PRESTA platform and the test questions are distinct from the train/dev questions, so the result is not fabricated. However, the test tables are the same tables used for development, as stated in Section 5.1. The custom functions in Section 4.2.4 were obtained from the training split and expanded during a testing/debugging phase, and the column-removal rules in Section 4.2.2 were manually identified from specific tables (e.g., columns named 'Ns_Nc_0', 'N_R_'). These components transfer directly to the test split because of the table overlap. The ablation study in Section 6.4, run on the dev split (which is also in the test split), shows no consistent gain from the new modules (removing custom functions raises dev score from 0.71 to 0.72), and the paper itself concedes that 'the dataset lacks the size and diversity in order to be statistically relevant.' Thus the 85% figure is partly a table-specific fit rather than evidence of generalizable design, but because the official evaluation and unseen questions provide independent grounding, the circularity is partial (score 4) rather than total.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim of 85% accuracy rests on the fairness of the PRESTA evaluation and the reliability of open-source LLMs. The system's design additionally depends on training-derived helper functions, manually chosen fuzzy thresholds, and an arbitrary ensemble size, none of which are shown to be essential.

free parameters (2)
  • fuzzy_match_threshold = 90 (in _best_fuzzy_match), 75 (in filter_rows_that_contain_column_value)
    Hand-chosen similarity thresholds for fuzzy string matching; no sensitivity analysis is provided.
  • number_of_repetitions = 8
    Ensemble size for majority voting; the paper notes 5 is enough, making 8 an arbitrary conservative choice.
assumptions (3)
  • domain assumption The PRESTA test set and official scoring metric provide a valid measure of table QA performance.
    The reported accuracy is measured against this benchmark; the overlap between test and train tables weakens this assumption.
  • domain assumption The open-source Qwen2.5-14B and Qwen2.5-Coder-14B models behave reliably as instructed.
    The system treats these LLMs as fixed reasoning and code-generation components; no fine-tuning or model variability analysis is performed.
  • ad hoc to paper The custom helper functions derived from the PRESTA training split are generic enough to be useful on the test questions.
    The functions were created by analyzing the training tables, which are the same tables used in the test split, so their generality is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MRT at IberLEF-2025 PRESTA Task: Maximizing Recovery from Tables with Multiple Steps." pith.science (2026). https://pith.science/paper/UXMTVAG5

@misc{pith2026250712981,
  author       = {Pith},
  title        = {Pith review of: MRT at IberLEF-2025 PRESTA Task: Maximizing Recovery from Tables with Multiple Steps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXMTVAG5}},
  note         = {Machine review of arXiv:2507.12981}
}
read the original abstract

This paper presents our approach for the IberLEF 2025 Task PRESTA: Preguntas y Respuestas sobre Tablas en Espa\~nol (Questions and Answers about Tables in Spanish). Our solution obtains answers to the questions by implementing Python code generation with LLMs that is used to filter and process the table. This solution evolves from the MRT implementation for the Semeval 2025 related task. The process consists of multiple steps: analyzing and understanding the content of the table, selecting the useful columns, generating instructions in natural language, translating these instructions to code, running it, and handling potential errors or exceptions. These steps use open-source LLMs and fine-grained optimized prompts for each step. With this approach, we achieved an accuracy score of 85\% in the task.

Figures

Figures reproduced from arXiv: 2507.12981 by the authors.

Figure 1
Figure 1. Diagram of the system showing all the steps involved in the generation of the response. 4.1. New challenges addressed in this task Compared with the SemEval task, the PRESTA dataset patented new limitations that our former MRT system had: • A larger number of columns in each table implied enormous prompts in the explainer module. Sometimes, this leads to exceptions running LLMs. Having more columns increases the cha… view at source ↗
Figure 2
Figure 2. Error repetition frequency in all the 7 experiments in the validation set LLM exceptions due to context size and improving the speed of the system at the same time. We used middle-size pretrained LMs with 14𝐵 parameters achieving a third place in the task with a 0.85% of accuracy. One clear benefit of our approach is its explainability as the user can very easily understand what is the source of the errors by seeing… view at source ↗
Figure 3
Figure 3. Accuracy score using an ensemble with majority voting Declaration on Generative AI During the preparation of this work, the authors used Grammarly in order to: Grammar and spelling check and GPT-4 in order to: suggestions for academic writing style. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content [PITH_FULL_IMAGE:figures/full_… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages

  1. [1]

    T. Liu, F. Wang, M. Chen, Rethinking Tabular Data Understanding with Large Language Models, in: K. Duh, H. Gomez, S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics, Mexico City, ...

    doi:10.18653/v1/2024.naacl-long.26 2024

  2. [2]

    Y. Ruan, X. Lan, J. Ma, Y. Dong, K. He, M. Feng, Language modeling on tabular data: A survey of foundations, techniques and evolution, 2024. URL: https://arxiv.org/abs/2408.10548. arXiv:2408.10548

    arXiv 2024

  3. [3]

    Hormazabal-Lagos, A

    M. Hormazabal-Lagos, A. B. Saez, H. Cerezo-Costas, P. A. Doval, J. A. Vesteiro, MRT at SemEval- 2025 Task 8: Maximizing Recovery from Tables with Multiple Steps, in: Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Association for Computational Linguistics, Vienna, Austria, 2025

    work page 2025

  4. [4]

    J. A. González-Barba, L. Chiruzzo, S. M. Jiménez-Zafra, Overview of IberLEF 2025: Natural Language Processing Challenges for Spanish and other Iberian Languages, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2025), co-located with the 41st Conference of the Spanish Society for Natural Language Processing (SEPLN 2025), CEUR-WS. org, 2025

    work page 2025

  5. [5]

    Osés-Grijalba, L

    J. Osés-Grijalba, L. A. Ureña-López, E. M. Cámara, J. Camacho-Collados, Overview of PRESTA at IberLEF 2025: Question Answering Over Tabular Data In Spanish, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2025), co-located with the 41st Conference of the Spanish Society for Natural Language Processing (SEPLN 2025), CEUR-WS. org, 2025

    work page 2025

  6. [6]

    Herzig, P

    J. Herzig, P. K. Nowak, T. Müller, F. Piccinno, J. Eisenschlos, TaPas: Weakly Supervised Table Parsing via Pre-training, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020. URL: http://dx.doi. org/10.18653/v1/2020.acl-main.398. doi:10.18653/v1/2020.acl-main.398

    doi:10.18653/v1/2020.acl-main.398 2020

  7. [7]

    Q. Liu, B. Chen, J. Guo, M. Ziyadi, Z. Lin, W. Chen, J.-G. Lou, Tapex: Table Pre-Training via Learning a Neural SQL Executor, arXiv preprint arXiv:2107.07653 (2021)

    arXiv 2021

  8. [8]

    OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering

    Z. Jiang, Y. Mao, P. He, G. Neubig, W. Chen, OmniTab: Pretraining with Natural and Synthetic Data for Few-Shot Table-based Question Answering, arXiv preprint arXiv:2207.03637 (2022)

    work page Pith review arXiv 2022

Show all 21 references
  1. [9]

    X. Liu, S. Shen, B. Li, P. Ma, R. Jiang, Y. Zhang, J. Fan, G. Li, N. Tang, Y. Luo, A Survey of NL2SQL with Large Language Models: Where are we, and Where are we Going?, arXiv preprint arXiv:2408.05109 (2024)

    2024

  2. [10]

    A. Su, A. Wang, C. Ye, C. Zhou, G. Zhang, G. Chen, G. Zhu, H. Wang, H. Xu, H. Chen, et al., TableGPT2: A Large Multimodal Model with Tabular Data Integration, arXiv preprint arXiv:2411.02059 (2024)

    2024 arXiv

  3. [11]

    Zhong, C

    V. Zhong, C. Xiong, R. Socher, Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning, 2017. URL: https://arxiv.org/abs/1709.00103. arXiv:1709.00103

    2017 arXiv

  4. [12]

    W. Chen, H. Wang, J. Chen, Y. Zhang, H. Wang, S. Li, X. Zhou, W. Y. Wang, TabFact: A Large-scale Dataset for Table-based Fact Verification, 2020. URL: https://arxiv.org/abs/1909.02164. arXiv:1909.02164

    2020 arXiv

  5. [13]

    Hwang, J

    W. Hwang, J. Yim, S. Park, M. Seo, A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization, 2019. URL: https://arxiv.org/abs/1902.01069. arXiv:1902.01069

    2019 arXiv

  6. [14]

    Osés Grijalba, L

    J. Osés Grijalba, L. A. Ureña-López, E. Martínez Cámara, J. Camacho-Collados, Question Answering over Tabular Data with DataBench: A Large-Scale Empirical Evaluation of LLMs, in: N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (Eds.), Proceedings of the 2024 Join...

    2024

  7. [15]

    S.-A. Chen, L. Miculicich, J. M. Eisenschlos, Z. Wang, Z. Wang, Y. Chen, Y. Fujii, H.-T. Lin, C.-Y. Lee, T. Pfister, TableRAG: Million-Token Table Understanding with Language Models, 2024. URL: https://arxiv.org/abs/2410.04739. arXiv:2410.04739

    2024 arXiv

  8. [16]

    Z. Wang, H. Zhang, C.-L. Li, J. M. Eisenschlos, V. Perot, Z. Wang, L. Miculicich, Y. Fujii, J. Shang, C.- Y. Lee, T. Pfister, Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding,

  9. [17]

    Osés Grijalba, L

    J. Osés Grijalba, L. A. Ureña-López, E. Martínez Cámara, J. Camacho-Collados, SemEval-2025 Task 8: Question Answering over Tabular Data, in: Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Association for Computational Linguistics, Vienna,...

    2025

  10. [18]

    F. Zhou, M. Hu, H. Dong, Z. Cheng, S. Han, D. Zhang, Tacube: Pre-computing data cubes for answering numerical-reasoning questions over tabular data, 2022. URL: https://arxiv.org/abs/2205. 12682. arXiv:2205.12682

    2022 arXiv

  11. [19]

    He, Achieving tool calling functionality in llms using only prompt engineering without fine- tuning, 2024

    S. He, Achieving tool calling functionality in llms using only prompt engineering without fine- tuning, 2024. URL: https://arxiv.org/abs/2407.04997. arXiv:2407.04997

    2024 arXiv

  12. [20]

    S. Sheu, A. Chang, W. Huang, Fast similarity search in string databases, in: 19th International Conference on Advanced Information Networking and Applications (AINA’05) Volume 1 (AINA papers), volume 1, 2005, pp. 617–622 vol.1. doi:10.1109/AINA.2005.185. A. Appendix I: Custom ...

    2005 doi

  13. [2024]

    arXiv:2401.04398

    URL: https://arxiv.org/abs/2401.04398. arXiv:2401.04398

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.