REVIEW 4 major objections 4 minor 25 references
MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper introduces MTabVQA, a benchmark of 3,745 question-answer pairs that require reading and combining two to five table images, and shows that current VLMs struggle while a LoRA-tuned 7B model reaches 43.4% exact match, exceeding…
desk verdict MTabVQA fills a real gap in multi-table visual reasoning benchmarks, but its core validity claim rests on verification that is thinner than the claims require. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the MTabVQA benchmark itself, defined by three design choices that carry the argument: (1) a graph-based relational sampling algorithm (Algorithm 1) that starts from a seed table and walks foreign-key links to sample up to Nmax=50 rows per table while preserving rows needed for multi-table joins; (2) a visual rendering pipeline that turns the sampled relational data into table images with ten randomized styling themes; and (3) a two-stage verification pipeline in which three LLM agents score each QA pair and a human annotator makes the final call. The benchmark's claim to measure multi-hop visual reasoning depends on all three working together: the sampling must keep the answer rows in the rendered tables, the rendering must force models to read pixels rather than text, and the verification must keep out single-table or incorrect questions.
What would settle it
Re-answer a random sample of MTabVQA-Eval questions with full, untruncated tables rendered as images and with a second, independent human re-verification of the same QA pairs; if scores jump or a noticeable share of pairs turn out single-table-answerable, the benchmark's difficulty and validity claims are weakened.
Extended reading notes
Core claim
MTabVQA is constructed by selecting multi-table join queries from five source datasets (Spider, QFMTS, ATIS, MiMoTable, and BIRD), sampling linked rows from related tables using a graph-based relational sampling algorithm that caps each table at 50 rows, rendering the sampled tables as images with ten random visual themes, and generating QA pairs either by converting SQL queries into natural-language questions or through taxonomy-guided LLM generation. A verification stage uses three LLM agents plus one human annotator to filter out questions that are wrong, single-table-answerable, or lacking multi-hop reasoning. On the resulting 3,745-pair eval set, the paper finds that zero-shot VLMs struggle, with GPT-4.1 at 37.0% EM and the strongest open-source model (Gemma-3-12B) at 11.8% EM; fine-tuning on the full MTabVQA-Instruct set with LoRA yields TableVision at 43.4% EM and 68.2% F1, the best overall scores, though still far from solved. The paper also reports that SFT outperforms GRPO and CoT prompting on a 2,395-pair Spider subset, and that fine-tuning data source matters more than sheer scale, since the 10,990-pair MultiTabQA subset underperforms the 2,395-pair Spider subset.
Load-bearing premise
The benchmark's measured scores reflect true multi-table visual reasoning only if Algorithm 1's row sampling keeps every row needed to answer each question after truncating tables to 50 rows, and only if the three-LLM-plus-one-human verification lets through no wrong, single-table-answerable, or leakage-prone questions.
Editorial extensions
If this is right
- Current proprietary and open-source VLMs are far from reliable on multi-table visual reasoning, with the best zero-shot model (GPT-4.1) below 40% EM, so this capability should be treated as an open problem rather than solved.
- Supervised fine-tuning on a relatively small, targeted instruction set can lift a 7B open-source VLM above a much larger proprietary model on this task, suggesting parameter-efficient tuning is a viable route for document and web agents.
- Data composition matters: fine-tuning on a large but misaligned source (MultiTabQA) hurts performance compared with a smaller, aligned source (Spider), implying that instruction data for visual tables should be chosen for task relevance and diversity, not just volume.
- The benchmark's 14 reasoning categories and 2-to-5-table questions provide a reusable test bed for measuring progress on visual multi-hop reasoning, including comparisons of SFT, GRPO, and CoT.
- The gap between TableVision (43.4% EM) and any near-solved benchmark indicates substantial headroom for better visual parsing, layout understanding, and cross-table aggregation.
Reading between the lines
- Not tested in the paper: because questions come from LLM generation and verification, some eval pairs may be answerable from a single table or from text extracted by OCR; a human double-annotation study on a sample of MTabVQA-Eval would test how much of the reported gap is true multi-table reasoning versus visual parsing difficulty.
- The Nmax=50 truncation makes the benchmark easier than real-world tables and may inflate fine-tuning gains; extending evaluation to untruncated or larger tables would test whether TableVision's advantage persists when relevant rows are not all visible.
- The benchmark could be turned into a diagnostic by ablating the visual dimensions (rendering style, table count, row count) to measure which factors drive model failures, an analysis the paper does not run.
- The SFT-over-GRPO result may be specific to the simple EM/F1 reward; a reward that credits partial multi-hop steps could change the ranking, which the paper itself hints at as a hypothesis.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MTabVQA, a benchmark of 3,745 multi-hop visual question-answering pairs over multiple rendered table images, constructed by sampling relational tables from existing text-to-SQL datasets (Spider, BIRD, ATIS, MiMoTable, QFMTS, MultiTabQA), rendering them as images, and generating QA pairs via SQL-to-question conversion and taxonomy-guided generation with LLM and human verification. The authors benchmark several open-source and proprietary VLMs, report that current models perform poorly (e.g., GPT-4.1 achieves 37.0% EM overall), and show that fine-tuning a Qwen2.5-VL-7B model with LoRA on their released MTabVQA-Instruct dataset yields a model (TableVision) that achieves 43.4% EM and 68.2% F1 overall, surpassing all baselines. They also compare SFT, CoT, and GRPO post-training strategies and analyze how data scale and source composition affect fine-tuning performance.
Significance. If the benchmark's multi-table validity holds, MTabVQA fills a genuine gap: existing visual table benchmarks are single-table, and existing multi-table benchmarks are text-based. The public release of the dataset and code, the extensive baseline evaluation, and the positive fine-tuning results are useful contributions to the community. The finding that a 7B open-source VLM fine-tuned on MTabVQA-Instruct can outperform GPT-4.1 on this task is potentially important. However, the central validity claim that every question requires reasoning across at least two table images is not established by the paper's verification pipeline, and the empirical comparisons lack statistical grounding. These issues must be addressed before the benchmark's headline results can be interpreted as measuring multi-table visual reasoning.
major comments (4)
- [Section 3.2 (Algorithm 1) and Section 3.5 (Verification)] The claim that every MTabVQA QA pair 'necessitate[s] multi-hop reasoning across several visually rendered table images' (Abstract) is not supported by the construction and verification procedures. Algorithm 1 samples each child table exclusively from rows linked via foreign keys to already-sampled parent rows (Algorithm 1, lines 9-10). For any SQL query whose join predicate is the only cross-table operation, the sampled child table contains only rows that already satisfy the join with the parent sample, so the join's filtering effect is materialized in the data presented. A model may then answer many such questions by reading only the parent table, because the child table adds no further discriminating information. The verification prompt (Appendix C, Figure 5) asks LLM agents to judge whether 'at least two tables were necessary,' but the agents are shown the generated reasoning steps, which typically assert a join, rather than an ablation with one table removed; the single human annotator (acknowledged in Limitations) cannot systematically test all subset-ablation conditions. This is load-bearing: if a substantial fraction of MTabVQA-Eval is de facto single-table, the benchmark does not measure multi-table visual reasoning and the headline comparison (Table 3: TableVision 43.4% EM vs. GPT-4.1 37.0% EM) may reflect single-table parsing. The paper should quantify the multi-table necessity by running an oracle ablation: for a random sample of eval questions, provide a strong model (or a text-based oracle with access to only one table's content) with each individual table image and measure how often the correct answer can be derived from a single table; report the fraction of questions that genuinely require a second table. If that fraction is not close to 100%, the benchmark's validity claim and the comparison need to be revised accordingly.
- [Section 4.1 (Benchmarking)] The evaluation protocol uses a single generation at temperature 1.0 and top-P 1.0 (Section 4.1) with no repeated sampling, no confidence intervals, and no significance tests. At temperature 1.0, VLM outputs are highly variable, and the differences that drive the paper's central claims are modest: TableVision's overall EM advantage over GPT-4.1 is 6.4 points (43.4 vs. 37.0), and on the ATIS split the best model achieves only 33.0 EM. Without multiple sampling runs and variance estimates, it is not possible to determine whether these gaps are stable or within generation noise. The paper should report results over at least three to five sampling runs with mean and standard deviation, or use deterministic (temperature 0) decoding for the headline comparison, and perform a significance test (e.g., paired bootstrap) for the key contrasts.
- [Section 4.3 and Table 4] The generalization claim for TableVision is weakened by source-database overlap between the training and evaluation splits. MTabVQA-Instruct and MTabVQA-Eval both draw from Spider, ATIS, and MiMoTable (Table 2), and the fine-tuning subsets and evaluation sub-splits are derived from the same source databases: for example, the Spider Subset is trained on Spider-derived QA pairs and evaluated on MTabVQA-Spider, and the full MTabVQA-Instruct includes ATIS and MiMoTable data while MTabVQA-Eval also contains ATIS and MiMo splits. Although QA pairs are disjoint, this is domain overlap rather than exact leakage, and it inflates the apparent effectiveness of fine-tuning on same-source splits. The paper should include a fully disjoint-source evaluation (e.g., train only on MultiTabQA- and BIRD-derived data and evaluate on the QFMTS- and Spider-derived eval splits) or explicitly discuss this overlap as a limitation that tempers the generalization claim for TableVision.
- [Appendix B.1 (Spider Dataset)] The filtering criteria for selecting the 1,048 Spider eval questions from the 1,270 join questions are stated only as 'criteria ensuring clear multi-hop reasoning paths, unambiguous answers from sampled data, and visual representability.' No operational definitions, worked examples, or exclusion statistics are provided. Because the same vague criteria are used for the QFMTS, ATIS, and MiMoTable filters (Appendices B.2, B.5, B.6), the selection pipeline is not reproducible and could introduce an unmeasured bias toward certain question shapes or difficulty levels. The authors should release the exact filtering rules and code and report, for each source dataset, the number of instances excluded at each filtering step and the reasons for exclusion.
minor comments (4)
- [Limitations] The Limitations section honestly acknowledges the single-annotator verification, but the paper should also report inter-annotator agreement on a random subset if a second annotator is added, or at least describe how the single annotator's decisions were audited.
- [Section 3.4 (Multi-Hop QA Pair Generation)] The text says the SQL query was 'executed it on sampled table subsets (SA, SB) for a ground-truth answer'; please clarify that the ground-truth answer is computed on the sampled data, not on the full original database, so that the benchmark measures reasoning over the presented sampled tables.
- [Abstract and Section 2.1] There are minor typos and formatting issues: 'a instruction-tuning dataset' should be 'an instruction-tuning dataset', and 'LLaV A' appears with an unusual space in several places (e.g., Sections 2.2 and 4.1).
- [Table 1] The column heading '# Tables/Databases' for MTabVQA lists 8,499, but the relationship between the 3,745 QA pairs and 8,499 tables is not explained in the text; please add a sentence clarifying whether this counts unique table images or table instances across all QA pairs.
Circularity Check
No significant circularity: MTabVQA is an externally grounded benchmark construction, and the fine-tuning improvement is measured on disjoint held-out QA pairs; noted same-source overlap and single-annotator verification are validity limitations, not circular reductions.
full rationale
MTabVQA is a dataset-construction paper, not a first-principles derivation, and its central claims are empirical measurements rather than predictions derived from fitted parameters. The benchmark tables, questions, and answers come from external sources (Spider, BIRD, ATIS, MiMoTable, QFMTS, MultiTabQA) with an independent rendering and verification pipeline. The fine-tuning result (TableVision, Section 4.1) is trained on MTabVQA-Instruct and evaluated on MTabVQA-Eval; although both splits draw on some of the same source databases (Section 4.3, Table 2), the QA pairs are explicitly disjoint, and the eval questions are not contained in the training set, so the improvement is not forced by construction. The same-source overlap is a legitimate concern about generalization to new domains, but it is not a circularity: no fitted parameter is renamed as a prediction. The LLM-based verification in Section 3.5 and the single-human-annotator limitation (Limitations section) are quality risks, and Algorithm 1's foreign-key sampling could in principle allow some questions to be answered from fewer tables than intended; however, the paper does not derive its headline numbers from those checks, and the benchmark's external grounding means the measured model failures remain informative. The use of Gemini-2.0-Flash for generation/verification and also as an evaluated baseline is self-referential in a data-creation sense, but its score (34.1% EM) is far from perfect, so its evaluation is not predetermined. No load-bearing claim reduces to a self-citation, an imported uniqueness theorem, or a definitional equivalence.
Assumptions & free parameters
free parameters (3)
- Nmax =
50
- Verification score threshold =
>=7.0
- Eval selection thresholds =
unspecified
assumptions (5)
- domain assumption The rendered table images faithfully represent the underlying relational data.
- domain assumption The sampled table subsets contain all rows needed to answer each query.
- domain assumption The source SQL queries and their ground-truth executions are correct.
- domain assumption LLM verification agents and the single human annotator accurately judge QA validity.
- domain assumption Multi-hop reasoning is genuinely required for each selected question.
Cite this review
Pith. "Pith review of MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space." pith.science (2026). https://pith.science/paper/UHMZMEWO
@misc{pith2026250611684,
author = {Pith},
title = {Pith review of: MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space},
year = {2026},
howpublished = {\url{https://pith.science/paper/UHMZMEWO}},
note = {Machine review of arXiv:2506.11684}
}
read the original abstract
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in interpreting visual layouts and text. However, a significant challenge remains in their ability to interpret robustly and reason over multi-tabular data presented as images, a common occurrence in real-world scenarios like web pages and digital documents. Existing benchmarks typically address single tables or non-visual data (text/structured). This leaves a critical gap: they don't assess the ability to parse diverse table images, correlate information across them, and perform multi-hop reasoning on the combined visual data. We introduce MTabVQA, a novel benchmark specifically designed for multi-tabular visual question answering to bridge that gap. MTabVQA comprises 3,745 complex question-answer pairs that necessitate multi-hop reasoning across several visually rendered table images. We provide extensive benchmark results for state-of-the-art VLMs on MTabVQA, revealing significant performance limitations. We further investigate post-training techniques to enhance these reasoning abilities and release MTabVQA-Instruct, a large-scale instruction-tuning dataset. Our experiments show that fine-tuning VLMs with MTabVQA-Instruct substantially improves their performance on visual multi-tabular reasoning. Code and dataset (https://huggingface.co/datasets/mtabvqa/MTabVQA-Eval) are available online (https://anonymous.4open.science/r/MTabVQA-EMNLP-B16E).
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Single-table questions are invalid
Check if the question is well-formed and genuinely requires multi-hop reasoning across MULTIPLE provided tables. Single-table questions are invalid
-
[2]
If the answer is incorrect,'is_valid'must be'false'
Verify if the answer is accurate based only on the information present in the given tables. If the answer is incorrect,'is_valid'must be'false'
-
[3]
Check if the'tables_used'field correctly lists relevant tables and if at least two tables were necessary
-
[4]
Validate if the reasoning steps are logical, coherent, and correctly lead from the table data to the answer. Respond with ONLY a valid JSON object (no introductory text, markdown formatting, or code blocks outside the JSON structure) containing the following keys: {{ "is_valid": true/false, "verification_comments": "Your detailed verification comments exp...
-
[7]
If the question ask to get specific columns, return only mentioned columns
Be specific and follow the instructions in the question. If the question ask to get specific columns, return only mentioned columns
-
[8]
We utilized the EasyR1 framework7 for these experiments, training for a total of 270 steps
experiments discussed in Section 4.2 for fine-tuning the Qwen2.5-VL-3B model. We utilized the EasyR1 framework7 for these experiments, training for a total of 270 steps. The training was conducted on a subset of MTabVQA-Instruct derived from the Spider dataset (2,395 examples). (a) Reward: Accuracy (EM) (b) Reward: F1 Score (c) Reward: Structural Format (...
-
[16]
If the question is unanswerable based on the provided tables, state "The question cannot be answered based on the provided data
-
[17]
However, take the time to think and reason before giving your answer
Please provide only the answer which has been asked, without any additional text (try to use few tokens). However, take the time to think and reason before giving your answer. Also, try to provide an answer even if you are unsure
Show all 25 references
-
[18]
type": "json_schema
Provide the answer in JSON format with given response schema as given [['ans1','ans2'],['ans3','ans4']]. Respond only with valid JSON format. Take your time to understand the question. Break it down into smaller steps. Come up with an answer and examine your reasoning. Finally...
-
[19]
Carefully examine the provided table(s): Pay close attention to the column headers, the data types within each column, and the relationships between tables if multiple tables are given
-
[20]
Understand the question being asked: Identify the specific information being requested and determine which table(s) and columns are relevant to answering the question
-
[21]
This reasoning is crucial and MUST be included in the final output
Reason step-by-step (Chain of Thought): Before generating the final answer, formulate a clear chain of thought outlining how you identified the relevant data, performed necessary operations (filtering, joining, aggregation, calculations), and arrived at the result. This reason...
-
[22]
Extract the necessary information from the table(s): Perform any required filtering, joining, aggregation, or calculations on the data based on your chain of thought to arrive at the answer
-
[23]
You can use SQL logic internally during your reasoning (Chain of Thought), but the final output should not contain raw SQL code
Do not include any SQL queries in the final answer JSON. You can use SQL logic internally during your reasoning (Chain of Thought), but the final output should not contain raw SQL code
-
[24]
type": "object
Be accurate and avoid hallucinations: Your answer must be completely based on the data in the provided table(s). . Provide the output strictly in the specified JSON format: The output must be a single JSON object containing two keys:`chain_of_thought`(a string detailing your r...
-
[25]
Carefully examine the provided table(s) Pay close attention to the column headers, the data types within each column, and the relationships between tables if multiple tables are given
-
[26]
Identify the specific information being requested and determine which table(s) and columns are relevant to answering the question
Understand the question being asked. Identify the specific information being requested and determine which table(s) and columns are relevant to answering the question
-
[27]
Perform any required filtering, joining, aggregation, or calculations on the data to arrive at the answer
Extract the necessary information from the table(s). Perform any required filtering, joining, aggregation, or calculations on the data to arrive at the answer
-
[28]
The answer should be directly responsive to the question and presented in a human-readable format
Formulate a clear and concise answer in natural language. The answer should be directly responsive to the question and presented in a human-readable format. It may involve listing data, presenting a single value, or explaining a derived insight
-
[29]
But you can use it internally, to come up with answer
Do not include any SQL queries in the answer. But you can use it internally, to come up with answer
-
[30]
Your answer should be completely based on the data in the provided table(s)
Be accurate and avoid hallucinations. Your answer should be completely based on the data in the provided table(s). Do not introduce any external information or make assumptions not supported by the data
-
[31]
Strictly, Give answer in this format, using the example below as reference: You FIRST think about the reasoning process as an internal monologue and then provide the final answer
Provide the answer in JSON format with given response schema as given [['ans1','ans2'],['ans3','ans4']].Respond only with valid JSON format, as shown in the example above. Strictly, Give answer in this format, using the example below as reference: You FIRST think about the rea...
-
[2017]
CoRR, abs/1709.00103
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR, abs/1709.00103. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on a hyb...
2021 arXiv
-
[2023]
Weijia Zhang, Vaishali Pal, Jia-Hong Huang, E
TableLlama: Towards Open Large Generalist Models for Tables.CoRR, abs/2311.09206. Weijia Zhang, Vaishali Pal, Jia-Hong Huang, E. Kanoulas, and Maarten de Rijke. 2024b. QFMTS: Generating Query-Focused Summaries over Multi- Table Inputs. InEuropean Conference on Artificial Intel...
2024 arXiv
-
[2024]
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu
Tables as Images? Exploring the Strengths and Limitations of LLMs on Multimodal Representations of Tabular Data.CoRR, abs/2402.12424. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebV oyager: Building an end-to- en...
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.