{"id":"533dbf4a-3740-4321-83ef-df5c3382cd89","arxiv_id":"2507.12981","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MRT, an LLM code-generation pipeline for Spanish table QA, achieves 85% accuracy on the IberLEF 2025 PRESTA test set.","lead":"A team describes a multi-step pipeline that answers Spanish questions about tables by generating and running Python code, reaching 85% accuracy on the PRESTA benchmark. The approach combines LLM-based column selection, fuzzy matching, and pre-coded helper functions to cope with wide, messy survey data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 85% test score may be inflated because test tables are the same as train/dev, so handcrafted column-removal rules (§4.2.2) and custom functions (§4.2.4) built from those tables can transfer directly to test; a leave-one-table-out evaluation would settle this.","rationale":"I read the paper in good faith: it is a shared-task system description that honestly reports official scores, makes code public, and explicitly cautions that the dataset is too small for statistically meaningful ablations. The strongest claim—85% on the PRESTA test—is externally verified, but its interpretation as evidence for the general value of the design choices rests on the test set being independent of development. Section 5.1 shows it is not: test tables are the same 10 tables as train+dev. The manual rules and functions were developed with full access to those exact tables. That is a genuine leak, not an artifact, and it is the same concern the reader identified. The paper's own ablation data (Table 6) weaken the claim that individual modules help, so the 85% is best read as an upper bound for this specific benchmark setup. I would keep the reader's CONDITIONAL verdict: the paper should be accepted as a system description only if the authors acknowledge and, if possible, quantify the table-overlap limitation, ideally with a leave-one-table-out evaluation or an evaluation on unseen tables. I found no independent reason to move the verdict to reject or to accept unconditionally.","tokens_in":10348,"tokens_out":6355,"duration_ms":76038,"concrete_test":"Hold out one PRESTA table at a time and rebuild the column-removal rules and custom functions using only the remaining tables (and development questions), then score the held-out table's test questions; average this leave-one-table-out accuracy. If it falls well below 85% (e.g., toward the 71% validation score), the reported score depends on table-specific tuning. As a simpler alternative, run the final system unchanged on a new set of Spanish tables not present in the PRESTA development splits (e.g., from DataBench-style data) and report accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the modular MRT system, with column selection and fuzzy matching, achieves 85% on PRESTA test. The official scoring is real, but the evidential value of this number for the paper's design choices depends on the test set being an unbiased sample of unseen table-question pairs. Section 5.1 states 'The tables used in the test consist of the same tables of the train and the dev splits,' and Table 2 confirms test has 10 tables, exactly the union of train (6) and dev (4). This means every test table was available during development. Consequently, the hardcoded column-removal rules (§4.2.2, e.g., dropping 'N_R_' and 'Ns_Nc_*' columns) and the custom function pool (§4.2.4, derived from the training split and expanded during a 'testing and debugging phase') are table-specific adaptations that can be applied verbatim at test time. These are not learned parameters, but they are a form of tuning to the exact table schemas and quirks in the test set. The ablation study (Table 6) does not resolve this: it is run on the 4 dev tables, which are also part of the test set, and shows no consistent gain from the new modules (removing custom functions actually raises dev score from 0.71 to 0.72). Thus the 85% may substantially overstate the system's performance on genuinely unseen tables. The reader's concern is valid and load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MRT, a modular table question-answering system submitted to the IberLEF 2025 PRESTA task. The pipeline uses open-source LLMs to select columns, generate natural-language instructions, translate these into Python code, execute the code, and format the answer. The authors add new components to an earlier SemEval system: a column selector, rules for removing uninformative columns, clarification instructions, custom code-generation functions, and fuzzy matching for categorical values. On the official PRESTA test set the system achieves 85% accuracy, placing third. The paper also reports a dev-set ablation study and a manual error analysis, and it explicitly acknowledges that the dataset is too small and the ablation differences too slight to confirm the benefit of the new modules.","tokens_in":10668,"tokens_out":2484,"duration_ms":26246,"significance":"If the 85% test score is taken as evidence of a generalizable system, the paper is a useful contribution to LLM-based table QA in Spanish, particularly for wide tables with ambiguous column names. The system is reproducible: the code is publicly available, the models are open 14B-parameter checkpoints, and the experimental protocol (8 repetitions, majority voting) is clearly described. The modular design is a strength for explainability and debugging. However, the evidential value of the headline result is substantially weakened by the overlap between test tables and development tables, and the ablation study does not demonstrate that the new components improve accuracy. The paper is honest about these limitations, but the central claim of the title and abstract—that the new features enable 'maximizing recovery'—is not firmly supported by the evidence presented.","major_comments":[{"comment":"The test set does not contain unseen tables: Section 5.1 states that 'The tables used in the test consist of the same tables of the train and the dev splits,' and Table 2 confirms that the 10 test tables are the union of the 6 training and 4 dev tables. Because the handcrafted column-removal rules in §4.2.2 (e.g., dropping 'N_R_' and 'Ns_Nc_*' columns) and the custom function pool in §4.2.4 were derived from the training split and expanded during a 'testing and debugging phase,' these components can be applied to the exact table schemas and quirks that appear in the test set. The 85% score therefore reflects performance on tables that were available during system development, not generalization to unseen tables. I request a leave-one-table-out evaluation or an evaluation on genuinely held-out tables to support the claim that the design choices, rather than table-specific tuning, drive the result.","section":"§5.1, Table 2; §4.2.2; §4.2.4"},{"comment":"The ablation study does not confirm that the new modules improve accuracy. Removing the column selector raises the dev score from 0.71 to 0.74, and removing custom functions gives 0.72, both equal to or better than the full system; only removing fuzzy substitution decreases the score (0.69). With only 100 dev questions and 4 tables, these differences are within random variation, a point the authors themselves make in the text. Consequently, the paper's emphasis on the new features as the source of the test-set success is not supported by the reported data. The authors should either provide additional evidence (e.g., statistical significance testing, a larger dev set, or a per-table breakdown) or explicitly reframe the contribution as a systems description rather than a validated improvement.","section":"§6.4, Table 6"},{"comment":"The sentence 'we can prove the benefits of some of our new features' overstates what the manual error analysis shows. The analysis is a qualitative attribution of error categories on the dev set; it does not compare against a counterfactual without the new features. This statement is directly contradicted by the ablation results in Table 6, which show no consistent benefit. Please replace 'prove' with a more cautious formulation, such as 'provide qualitative evidence consistent with,' and reconcile this paragraph with the ablation discussion in §6.4.","section":"§6.3"}],"minor_comments":[{"comment":"The conclusion states 'a 0.85% of accuracy'; this should read '85% accuracy' to avoid confusion with 0.85 percent.","section":"§7"},{"comment":"The phrase 'the PRESTA dataset patented new limitations' appears to be a typo; 'patented' should likely be 'presented' or 'posed.'","section":"§4.1"},{"comment":"The column headers contain a duplicated 'List' entry ('List List'); the header row should be cleaned to match the answer types listed in Table 3.","section":"Table 6"},{"comment":"The sentence 'It matches our intuition that is that the test set questions are in average simpler than the validation set questions' is grammatically awkward and should be revised.","section":"§6.2"},{"comment":"The function `_round_was_useful` is called in the fuzzy matching code but is not defined in the appendix; please either include its definition or add a comment explaining its intended behavior.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a shared-task system description with a real official score, which is a legitimate contribution for this venue. However, the overlap between test and development tables is a serious threat to the paper's central claim, and the ablation data actually weaken the case for the new modules. I would urge the editors to require the authors to add either a leave-one-table-out experiment or an explicit, prominent disclaimer that the reported accuracy is not a measure of generalization to unseen tables. Without such a change, the paper's contribution is mostly archival rather than evidential."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid shared-task system description, not a fundamental advance. The 85% official score on PRESTA is real, but the paper itself does the honest work of telling you why it should not be read as strong evidence for the new modules. The stress-test concern about train/test table overlap is valid, and I would go a bit further: because test tables are the same as train/dev, the handcrafted column-removal rules and custom functions are table-specific adaptations that transfer directly to evaluation. That does not make the score fake, but it does mean the central empirical claim is a benchmark result, not a generalizable finding.\n\nWhat is actually new: the column selector that handles very wide tables (174 columns average), the clarification-instruction mechanism for fixing value/name mismatches, and the pre-coded generic functions with fuzzy matching. These are pragmatic engineering contributions. The paper is readable, the code is public, and the authors include a manual error analysis plus an ablation study. Credit where due: they explicitly state the ablation shows no consistent gain, and they note the dataset is too small to draw statistical conclusions. That is unusually candid for a shared-task paper.\n\nSoft spots: the ablation is underpowered (4 dev tables, 100 questions), and the test score being 15 points higher than dev is explained by \"simpler questions\" with no direct evidence. The fuzzy matching threshold and repetition count are free parameters, and there is no variance estimate. The biggest issue is the table overlap, which the stress-test correctly identifies. I do not think it is fatal—this is a system description, and the authors disclose the overlap—but anyone citing the 85% as evidence for the design should be warned.\n\nWho it is for: people working on table QA with LLMs, especially in Spanish-language or wide-table settings, and shared-task organizers looking for baseline descriptions. It deserves a serious referee because it is a competent, honest system paper with reproducible code. I would accept it with minor revisions, mainly asking for a clearer statement that the score is not evidence of generalizable improvement.\n\nRecommendation: send to peer review. It is a useful record of a competitive system, and the authors' own limitations section does most of the referee's work.","headline":"Honest shared-task system description; the 85% score is a real benchmark result, but the table overlap and weak ablation mean it is not evidence that the new modules generalize.","tokens_in":11205,"tokens_out":2508,"would_cite":false,"duration_ms":26723,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular pipeline generates Python code to answer 85% of Spanish table questions.","keywords":["Table Question Answering","Large Language Models","Code Generation","Spanish","PRESTA","Column Selection","Fuzzy Matching","Multi-step pipeline"],"falsifier":"Run the same pipeline, with the hand-built column-removal rules disabled, on a newly collected set of Spanish survey tables that were not available during development; if accuracy drops well below the reported 85%, the headline number was inflated by the train/test table overlap.","tokens_in":10161,"feed_emoji":"📊","tokens_out":5573,"duration_ms":57825,"temperature":0.7,"pith_summary":"This paper describes a modular system for answering questions over Spanish survey tables. The system decomposes each question into a sequence of steps—selecting relevant columns, writing natural-language instructions, generating and executing Python code, and formatting the answer—using open-source 14B-parameter language models. The authors report an accuracy of 85% on the PRESTA test set of the IberLEF 2025 shared task, a setting where tables average 174 columns. The central claim is that this divide-and-conquer, code-generation approach can handle very wide and ambiguously labeled tables while remaining explainable, because every intermediate instruction and code fragment can be inspected.","feed_headline":"Python-writing LLM pipeline answers 85% of Spanish table queries","feed_subtitle":"The system splits the task into column selection, instruction generation, and code execution, using open 14B models.","key_machinery":"The object that carries the argument is the MRT pipeline itself: a multi-step 'maximizing recovery from tables' system in which each module has one narrow task. A column selector asks an LLM, in batches of 25 columns, which columns might matter for the question; the explainer then produces a JSON of natural-language instructions, listed column names, and filter values, with Levenshtein-based correction of misspelled column names and added clarification lines for value mismatches; the coder, prompted with a library of pre-coded generic functions, writes Python that the runner executes, retrying on exceptions; and the interpreter/formatter converts the output to the expected type. The custom function library—ranging from counting occurrences and filtering by numeric thresholds to fuzzy 'contains' matching—is what lets the coder focus on orchestration instead of low-level pandas logic, and it is derived from the training tables.","core_discovery":"The central claim is that MRT, a sequential pipeline that compiles natural-language instructions into executable Python, achieves 85% accuracy on Spanish table question answering despite tables that average 174.1 columns. The paper attributes this result to three new components over its predecessor: an LLM-based column selector that prunes the prompt to groups of 25 columns, a library of pre-coded generic Python functions that steer the coder away from error-prone pandas usage, and clarification instructions plus fuzzy matching that reconcile how values appear in the question with how they are stored in the table. The authors also report that the pipeline runs about three times faster with column selection, and that a majority-vote ensemble over eight runs stabilises the output. They frame the main benefit as explainability: errors can be traced to specific instructions or code, and the system needs no fine-tuning, only prompt engineering around open-weight models.","pith_inferences":["Because the PRESTA test tables are the same tables used for training and development, the 85% figure likely overstates how the system would perform on tables it has never seen; the durable contribution is probably the pipeline design rather than the score.","The hand-built column-removal rules and the semi-automated function library were created from the training tables, so on genuinely new table schemas those modules would need to be rebuilt or revalidated; fuzzy matching is the component most likely to transfer as-is.","The trick of presenting pre-coded Python functions as tool-calling options to the code model is a cheap way to inject domain knowledge into an LLM pipeline and could be tested on other structured-data tasks such as SQL generation or spreadsheet automation."],"forward_implications":["Wide tables, around 174 columns on average, can be processed with small prompts because the column selector cuts the table before any instruction generation; the whole pipeline also runs about three times faster.","The main remaining error sources become reachable and debuggable: wrong natural-language instructions and removal of a relevant column account for most failures, while code exceptions and wrong cell filtering drop sharply.","An ensemble with majority voting over five to eight runs gives stable answers and is treated as sufficient for the system's use.","Open-source 14B models, one of them a code-specialised variant, can match the task without any fine-tuning, which lowers the barrier to applying the approach elsewhere.","The gap between validation (71%) and test (85%) suggests the test questions are simpler, so the system's headline score should be read together with the validation number."],"supporting_citations":[{"why":"The previous MRT system described by the same authors at SemEval-2025 Task 8, which this work extends and uses as the baseline for new features.","marker":"[3]"},{"why":"The PRESTA task overview defines the dataset splits, evaluation metric, and Spanish tabular QA setting in which the 85% score is reported.","marker":"[5]"},{"why":"The analogous English-language SemEval-2025 Task 8, whose tables averaged 24.8 columns, serves as the comparison that motivates the wide-table challenges.","marker":"[17]"},{"why":"The Tacube idea of pre-computing data cubes inspired the paper's pre-coded generic Python function library, though the authors do not precompute cubes.","marker":"[18]"},{"why":"Supports the claim that presenting functions as tool calls can be achieved through prompt engineering alone, grounding the custom-function design.","marker":"[19]"},{"why":"Supplies the fuzzy string similarity methodology used to match categorical values that appear differently in questions versus table cells.","marker":"[20]"}],"fun_headline_variants":["LLM writes Python to answer 85% of Spanish table questions","Coding LLM hits 85% on Spanish table QA","Multi-step LLM pipeline scores 85% on Spanish tables","Python-generating LLM answers 85% of Spanish table queries","Column-pruning LLM boosts Spanish table QA to 85%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's 85% accuracy is measured on a test set whose tables are the same tables used for training and development, and the hand-written column-removal rules and pre-coded functions were built from those tables, so the score may not hold on tables the system has never seen.","fun_headline_variants_meta":{"raw":{"variants":["LLM writes Python to answer 85% of Spanish table questions","Coding LLM hits 85% on Spanish table QA","Multi-step LLM pipeline scores 85% on Spanish tables","Python-generating LLM answers 85% of Spanish table queries","Column-pruning LLM boosts Spanish table QA to 85%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2423,"prompt_tokens":865,"completion_tokens":1558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1469}},"tokens_in":481,"tokens_out":1558,"duration_ms":12790,"temperature":1.0,"reasoning_tokens":1469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:32:58.038660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline, with the hand-built column-removal rules disabled, on a newly collected set of Spanish survey tables that were not available during development; if accuracy drops well below the reported 85%, the headline number was inflated by the train/test table overlap.","supporting_citations":[{"cited_title":"Hormazabal-Lagos, A","cited_arxiv_id":null,"evidence_quote":"The previous MRT system described by the same authors at SemEval-2025 Task 8, which this work extends and uses as the baseline for new features."},{"cited_title":"Osés-Grijalba, L","cited_arxiv_id":null,"evidence_quote":"The PRESTA task overview defines the dataset splits, evaluation metric, and Spanish tabular QA setting in which the 85% score is reported."},{"cited_title":"Osés Grijalba, L","cited_arxiv_id":null,"evidence_quote":"The analogous English-language SemEval-2025 Task 8, whose tables averaged 24.8 columns, serves as the comparison that motivates the wide-table challenges."},{"cited_title":"TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Data","cited_arxiv_id":"2205.12682","evidence_quote":"The Tacube idea of pre-computing data cubes inspired the paper's pre-coded generic Python function library, though the authors do not precompute cubes."},{"cited_title":"Achieving Tool Calling Functionality in LLMs Using Only Prompt Engineering Without Fine-Tuning","cited_arxiv_id":"2407.04997","evidence_quote":"Supports the claim that presenting functions as tool calls can be achieved through prompt engineering alone, grounding the custom-function design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fuzzy string similarity methodology used to match categorical values that appear differently in questions versus table cells."}],"review_version":1}