TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning

Jia Wang; Lanrui Wang; Mingyu Zheng; Weiping Wang; Yang Hao; Zheng Lin; Zhifan Feng

REVIEW 3 major objections 5 minor 1 cited by

TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TableDreamer claims that weakness-guided, from-scratch synthesis of 27K table tuning samples raises a target LLM's tabular accuracy from 49.07% to 60.69% and beats larger synthetic datasets.

desk verdict Well-executed data synthesis paper with a genuinely new from-scratch table generation plus weakness-guided iteration; the evaluation format confound for JSON parsing needs to be addressed before the headline gains are taken at face value. read the letter →

arxiv 2506.08646 v1 pith:P7WHC5IS submitted 2025-06-10 cs.CL cs.AIcs.LG

Mingyu Zheng , Zhifan Feng , Jia Wang , Lanrui Wang , Zheng Lin , Yang Hao , Weiping Wang This is my paper

classification cs.CLcs.AIcs.LG

keywords tableinstructiontuningsyntheticdataLLM-as-a-judgeweakness-guidedselectiondiversityinputspaceexplorationtabularLLMefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that high-quality table instruction-tuning data can be synthesized entirely from scratch—no public tables, no human annotation—and that a compact dataset selected for what the target model gets wrong can beat much larger synthetic datasets. Fine-tuning Llama3.1-8B-instruct on 27K TableDreamer samples raises average accuracy from 49.07% to 60.69% across ten table benchmarks, an 11.62-point gain, while state-of-the-art baselines trained on 100K+ samples remain below it. The gain is attributed to two design choices: explicit diversity control over table attributes during generation, and an iterative loop in which only samples the target model handles poorly are kept as seeds for the next round. A sympathetic reader would take the paper's claim to be that weakness-guided exploration, not raw data volume, is the lever for efficient table instruction tuning.

What carries the argument

The load-bearing mechanism is the iterative collaboration between input-space exploration and weakness-data identification. Exploration expands coverage through three evolution directions—instruction complication, instruction generalization (new tasks plus paraphrases), and table generalization (format changes, header edits, data swaps, permutations)—while identification closes the loop by using an LLM-as-a-judge, with a prompt adapted from HelpSteer2's correctness rubric, to keep only samples the target model scores below 3. The next round's seed data are exactly those weak samples, so each iteration concentrates synthesis capacity where the model demonstrably fails. The table-synthesis prompt is the other half of the machinery: it samples flat, horizontal, and hierarchical layouts, row and column counts, header depth, cell-dependency formulas, and HTML or Markdown encodings so the input space is wide before instruction evolution begins.

What would settle it

Measure the LLM judge's agreement with human correctness labels on a held-out set of evolved table samples; or swap the judge for an independent judge or majority-vote scheme and compare final fine-tuned accuracy. If random selection of evolution output matches judge-guided selection once judge errors are controlled, the weakness-guidance claim is falsified.

Watch

Extended reading notes

Core claim

TableDreamer's central claim is that diversity and weakness guidance can be combined in a single loop: first synthesize tables by sampling table type, size, header structure, cell relations and format; generate seed instructions from 20 tabular task descriptions; then evolve every sample in three directions—making instructions harder, inventing novel task instructions, and perturbing table structure, format, or content. Each candidate is scored by an LLM judge comparing the target model's answer to a stronger teacher's reference answer; samples rated below 3 on a 5-point scale form the weakness pool that seeds the next round. After two rounds this produces 27K training triples whose accumulated weakness guidance shifts training toward the model's genuine failure regions rather than merely adding quantity. The paper reports that removing weakness identification (using all 34K evolved samples) lowers average accuracy by 4.41 points, and that judge-guided selection beats random selection by 3.29 points, which is the empirical core of the claim.

Load-bearing premise

The whole selection loop depends on the LLM judge's scores being trustworthy; if the judge labels the wrong samples as weaknesses, the iterative exploration will systematically amplify that bias because no human verification of the ratings is applied.

Editorial extensions

If this is right

A 27K synthetic dataset can outperform 100K+ synthetic instruction-tuning datasets on ten tabular benchmarks, so data efficiency need not be traded against accuracy.
Synthesizing tables from scratch, not just questions over existing tables, materially improves out-of-distribution performance: TableDreamer reaches 46.20% on the TableGPT benchmark without using its training data, rivaling TableGPT's own 47.23%.
Weakness-data selection carries the gain: dropping it (34K unselected samples) costs 4.41 average points, and random selection costs 3.29 points relative to judge-guided selection.
The synthetic data transfers to few-shot learning: adding TableDreamer-27K alongside as few as 20 training examples per benchmark improves average accuracy from 52.44% to 59.94%.
General capability is roughly preserved: IFEval changes by -0.72 and MMLU by +0.32 after fine-tuning, unlike several existing tabular LLMs that lose large amounts of general accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if judge reliability is the bottleneck, the same loop could be run with a panel of judges or with calibrated confidence scores, which the paper itself hints at by suggesting criteria decomposition and majority voting.
Beyond the paper: the framework's ceiling is set by the teacher LLM's table skills, so applying it to self-improving or weak-to-strong regimes would require a different stopping or verification mechanism, a direction the paper acknowledges in its limitations.
Beyond the paper: the diversity-plus-weakness recipe may transfer to other structured inputs—code, databases, or documents with layout—where the input space is similarly combinatorial and LLM judges can score correctness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a serious, well-engineered paper on table instruction-tuning data synthesis. The genuinely new piece is generating tables themselves from scratch, rather than only instructions over existing tables, and coupling that with an iterative weakness-guided selection loop. That combination is not in the cited literature, and the released 27K dataset could become a useful community resource.

The experimental work is thorough. Ten benchmarks, a broad set of baselines reimplemented with a matched teacher model, and careful ablations: removing each table type, each evolution direction, and the weakness filter all degrade performance. The random-selection ablation for weakness guidance is a real plus, and the method maintains general ability on IFEval/MMLU instead of collapsing like most tabular LLMs. The central accuracy claim is measured on independent benchmarks, so there is no fitting-to-the-test-set circularity.

The main soft spot is the evaluation protocol for 8 of the 10 benchmarks. Accuracy is computed by regular-expression extraction of JSON answers. TableDreamer's training data, especially from GPT-4o, is aligned with that JSON-requesting eval prompt, while the reimplemented general baselines (Magpie, Self-Instruct, GenQA, Evol-Instruct) were trained on natural-language responses, as the paper's own Figure 10 shows. If those models emit correct but non-JSON answers, the regex parser scores them wrong. The paper does not report unparseable-response rates or any lenient-extraction control, so part of the 11.62% average gain could be format compliance rather than table reasoning. This is testable and fixable; it does not sink the paper but it tempers how strongly I would cite the numeric comparisons to the general baselines.

Minor concerns: no multiple seeds or error bars for fine-tuning runs, and the LLM-as-a-judge for weakness selection is not validated against human ratings, though the random-selection ablation does show the selection matters. The use of GPT-4o-mini for T2T evaluation is a secondary but legitimate worry.

Overall, this deserves a serious referee. The core idea is sound, the engineering is honest, and the dataset is valuable. The authors should be asked to run a lenient extraction or an LLM-judge control on the non-JSON baselines and report unparseable rates. That would settle the main confound.

Referee Report

3 major / 5 minor

Summary. The paper introduces TableDreamer, a two-stage framework for synthesizing table instruction-tuning data from scratch. Stage 1 generates diverse tables by prompting an LLM with topics, subtopics, table titles, and randomized table attributes. Stage 2 starts from seed instruction-data triples, applies three evolution directions (instruction complication, instruction generalization, table generalization), scores the target LLM's responses with an LLM-as-a-judge against teacher-LLM reference responses, and retains low-scoring samples as weakness data for the next iteration. The final accumulated data are used to fine-tune Llama3.1-8B-instruct. The main experiment reports an average accuracy gain from 49.07% to 60.69% on 10 tabular benchmarks using 27K GPT-4o-synthesized samples, with a teacher-matched Llama3.1-70B variant reaching 56.02%. The paper also reports few-shot augmentation results, ablations of each component, evidence that the method preserves general abilities on IFEval and MMLU, and a study of R1-style reasoning models.

Significance. If the reported results hold, the paper makes a useful empirical contribution: it demonstrates that a compact, weakness-guided synthetic dataset can match or exceed much larger synthetic and human-curated table instruction datasets, and it provides a reproducible pipeline with released code and data. The study is unusually thorough in its benchmark coverage (10 benchmarks), in its effort to reimplement general data-synthesis baselines with a matched Llama3.1-70B teacher, and in its component ablations. The general-capacity evaluation on IFEval and MMLU is a valuable addition that many table-tuning papers omit. The central weakness-guided selection idea is plausible and is supported by a random-selection ablation, but the main accuracy claim currently rests on an evaluation protocol whose format-sensitivity may systematically favor the method's JSON-formatted training data.

major comments (3)

[Section 4.1 (Evaluation Metrics) and Figure 10] The headline comparison is confounded by the JSON-format exact-match evaluation. Section 4.1 states that for TQA, TFV, and the TableGPT benchmark, accuracy is computed by extracting JSON answers with regular expressions, with no description of how unparseable responses are handled (they are presumably scored as incorrect). The reimplemented general baselines (Magpie, Self-Instruct, GenQA, Evol-Instruct) were trained on data whose responses are free-form natural language, as shown directly in Figure 10; fine-tuning on such data can reduce a model's tendency to emit the JSON scaffold required at evaluation time. TableDreamer's training data, by contrast, is explicitly JSON-formatted (Figures 3 and 10), so the regex parser credits format compliance for TableDreamer while potentially marking equally correct baseline answers as wrong. This affects 8 of the 10 benchmarks and therefore directly bears on the 11.62% headline gain and the 3.79-point margin over Magpie in the teacher-matched comparison. The paper should report per-model unparseable-response rates and add a lenient or model-based extraction control (e.g., asking an LLM to extract or judge the final answer from the raw response) to show that the reported gains are not an artifact of answer formatting.
[Section 3.3 (Weakness Data Identification), Section 3.4, and Appendix C.2] The core contribution--weakness-guided selection--depends entirely on the reliability of a single LLM-as-a-judge, but the paper never measures the judge's accuracy. Section 3.3 says 'an LLM rates' the target model's response on a 5-point scale and samples below 3 are retained, yet for the main GPT-4o pipeline the judge model is not identified; Section 3.4 specifies only the teacher LLM and the target LLM. The only judge-related control (Table 8) uses Llama3.1-70B as the teacher and does not validate judge agreement against human ratings. If the judge is systematically biased--for example, rewarding surface format over reasoning correctness--the iterative loop will amplify that bias in later rounds. The paper itself acknowledges in Appendix C.5 that judge reliability may degrade on more complex instructions. The authors should state the judge model and decoding settings, report human-judge agreement on a sample (e.g., 100-200 ratings), and provide at least one sensitivity analysis using a different judge or a threshold sweep to show that the selection benefit is robust.
[Table 3 and Appendix C.2] The ablation labeled 'w/o Weakness Iden.' compares 27K selected data against 34K unselected data, so the comparison conflates the effect of selection with the effect of data quantity. Although the random-selection ablation in Table 8 (same 27K volume) partially addresses this, the main ablation table in the paper should either use equal volumes or explicitly state the confound. Reporting an equal-volume 'all data' condition would make the weakness-selection claim cleaner and directly support the paper's data-efficiency argument.

minor comments (5)

[Section 4.2, first paragraph of 'Effectiveness of TableDreamer'] Typo: 's surpasses' should be 'surpasses'.
[Section 4.2, Ablation Study] Typo: 'explore the vase input space' should be 'vast input space'.
[Figure 5 and Table 13] The paper reports one run of the scaling experiment and does not provide variance estimates across fine-tuning seeds; given the small margins in several benchmarks (e.g., Table 13 gains of 0.79 on average), reporting mean and standard deviation over at least 2-3 seeds would strengthen the conclusions.
[Section 4.1, T2T evaluation] For the two T2T benchmarks, the paper uses GPT-4o-mini as the judge while the main TableDreamer data are generated by GPT-4o; the paper should briefly discuss whether this judge could favor responses whose style resembles GPT-4o outputs, and ideally report a small human agreement study for the T2T judgments as well.
[Table 3 heading] The abbreviations 'Inst. Gene.', 'Inst. Comp.', 'Table. Gene.' are defined only in the caption; consider expanding them in the table header for readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central accuracy claim is measured on held-out public benchmarks, and the weakness-guided selection loop is a data-curation feedback mechanism rather than a fitted prediction.

full rationale

The paper's central claim—that TableDreamer-27K raises Llama3.1-8B-instruct average accuracy from 49.07% to 60.69%—is evaluated on 10 public tabular benchmarks with held-out test sets; no training-data parameter or selection score is fitted to those benchmark answers. The LLM-as-a-judge weakness filter (Section 3.3) uses target-model responses to choose which evolved samples enter training, but this is not an equivalence: the selected samples are judged against teacher reference responses on a 5-point scale, while the reported outcome is exact-match or judge-based accuracy on independent benchmarks. The random-selection ablation in Table 8 provides a control showing that judge-guided selection is not merely a proxy for the evaluation metric. The self-citations (Si et al. 2023 and Zheng et al. 2024) support generic claims about data quantity and multimodal-table directions and are not load-bearing; neither is invoked to forbid alternatives or to supply a uniqueness theorem. A separate evaluation-fairness concern exists—JSON regex exact matching (Section 4.1) may penalize baselines trained on free-form responses—but that is a measurement confound, not a circular derivation. Under the paper's own equations, no predicted quantity is definitionally equal to an input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or formal entities. The framework's assumptions are all domain-level: the representativeness of seed tasks, the reliability of LLM-as-a-judge, the validity of the distillation setup, and the validity of the table generation and filtering process. The main hand-chosen parameters are the weakness threshold, iteration count, seed size, evolution strategy set, and table attribute ranges; these influence the final dataset but are not fitted to the benchmark test sets.

free parameters (5)

weakness score threshold = 3 (samples with score < 3 retained as weakness data)
Hand-chosen cut-off in Section 3.3; determines which generated samples enter the training set and directly shapes the data.
iteration rounds = 2
Number of input-space exploration and weakness identification cycles; adding more rounds changes data composition and cost.
seed data size = 3,272
Initial seed triples from 1,541 tables in Section 3.4; this budget influences the starting point of the iterative loop.
evolution strategy selection = 14 strategies, one randomly sampled per direction
Table 12 lists 14 strategies; each sample's evolution randomly picks one, introducing stochasticity into the generated data.
table attribute ranges = rows 4-43, columns 4-45, cell counts 28-1008
Random sampling ranges for table size and header structure in Section 3.2; hand-chosen to balance diversity and validity.

assumptions (4)

domain assumption 20 seed task descriptions from prior tabular benchmarks cover the table understanding task space
Used as basis for seed instruction generation in Section 3.3 and Table 11; if the task list is incomplete, the generated data will miss important table tasks.
domain assumption LLM-as-a-judge can reliably score the correctness of table responses
Weakness identification in Section 3.3 (prompt in Figure 17) relies on judge ratings; the random-selection ablation supports usefulness but not unbiasedness.
domain assumption Strong-to-weak distillation improves the weaker model without sacrificing general capability
The pipeline assumes that GPT-4o/Llama3.1-70B synthetic data improves Llama3.1-8B; validated on benchmarks but not guaranteed outside the tested conditions.
domain assumption Synthetic tables generated via LLM prompts are valid and executable after formula computation and filtering
Section 3.2 relies on markdown formulas for cell dependencies and manual filtering to remove invalid tables; this assumption affects all downstream data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning." pith.science (2026). https://pith.science/paper/P7WHC5IS

@misc{pith2026250608646,
  author       = {Pith},
  title        = {Pith review of: TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7WHC5IS}},
  note         = {Machine review of arXiv:2506.08646}
}

read the original abstract

Despite the commendable progress of recent LLM-based data synthesis methods, they face two limitations in generating table instruction tuning data. First, they can not thoroughly explore the vast input space of table understanding tasks, leading to limited data diversity. Second, they ignore the weaknesses in table understanding ability of the target LLM and blindly pursue the increase of data quantity, resulting in suboptimal data efficiency. In this paper, we introduce a progressive and weakness-guided data synthesis framework tailored for table instruction tuning, named TableDreamer, to mitigate the above issues. Specifically, we first synthesize diverse tables and related instructions as seed data, and then perform an iterative exploration of the input space under the guidance of the newly identified weakness data, which eventually serve as the final training data for fine-tuning the target LLM. Extensive experiments on 10 tabular benchmarks demonstrate the effectiveness of the proposed framework, which boosts the average accuracy of Llama3.1-8B-instruct by 11.62% (49.07% to 60.69%) with 27K GPT-4o synthetic data and outperforms state-of-the-art data synthesis baselines which use more training data. The code and data is available at https://github.com/SpursGoZmy/TableDreamer

Figures

Figures reproduced from arXiv: 2506.08646 by the authors.

**Figure 2.** The overview of the proposed TableDreamer framework, which includes two stages. In stage 1, we first [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Example of TableDreamer synthetic data. The [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (15 more)

**Figure 4.** Figure 4: The top 25 most prevalent root verbs (the inner [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: The performance improvement as the TableDreamer synthetic data (from GPT-4o) continues to accumulate. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: More examples of TableDreamer synthetic data. Tables and instructions are clipped due to space limitation. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

**Figure 7.** Figure 7: Instruction diversity comparison of different synthetic table instruction tuning data. We show the top 25 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

**Figure 8.** Figure 8: The distribution of instruction length, output length, table row number and table column number in } ``` [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

**Figure 9.** Figure 9: The prompt used for harmful content detection. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

**Figure 10.** Figure 10: Comparison of synthetic table instruction tuning data from different methods. Some table content are [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

**Figure 11.** Figure 11: Qualitative comparison of model responses on WTQ benchmark. df = pd.DataFrame(data) in 1994.", "The games played on ABC w [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

**Figure 13.** Figure 13: Qualitative comparison of model responses on TabFact benchmark. ere are multiple ships with the same namebut the Phi-3.5-mini-instruct-3.8B - Achilles: 70 - Amphion: 29 Apollo: 63 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

**Figure 14.** Figure 14: Qualitative comparison of model responses on MMLU benchmark. [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]

**Figure 15.** Figure 15: The prompt used for synthesizing diverse tables. The string in red color will be replaced with correlative [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]

**Figure 16.** Figure 16: The prompt used for data evolution in the instruction complication direction. [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]

**Figure 17.** Figure 17: The LLM-as-a-judge prompt used for weakness data identification, which is modified from the correctness [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]

**Figure 18.** Figure 18: The prompt used for data evolution in the table generalization direction. [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]

**Figure 19.** Figure 19: The prompt used for data evolution in the instruction generalization direction. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
cs.AI 2026-01 conditional novelty 5.0 of 10

A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.

Reference graph

Works this paper leans on

81 extracted references · 77 canonical work pages · cited by 1 Pith paper

[4]

Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev

Tabpedia: Towards comprehensive visual ta- ble understanding with concept synergy.Preprint, arXiv:2406.01326. Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. 2022. ReasTAP: Injecting ta- ble reasoning skills during pre-training via synthetic reasoning examples. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Lan...

arXiv 2025
[15]

harmful_content

The input prompt is incomplete and illogical and contains lots of grammar mistakes. Format your output in the JSON format: ```json { 'rating': <final rating result, string format> 'explanation': <rating explanation, string format> } ``` LLM-as-a-Judge Prompt Given a conversation between a human user and an LLM-based assistant like GPT-4, please check if t...

work page
[16]

RTL II: (-0.6) From the values, ZDFneohas the highest positive value of (2.8). Final Answer: ZDFneo prompt response Magpie Let's work with this table: | Name | Age | City | Occupation | Salary | |------------|-----|-----------|------------|--------| | Alice | 24 | New York | Engineer | 60000 | | Bob | 30 | Chicago | Teacher | 45000 | | Carol | 28 | San Fr...

work page 1995
[41]

answer": [

**Conclusion**: All pennant numbers are unique, so there is no single pennant number that is more unique than the others. Therefore, the answer is: ```json {"answer": ["All pennant numbers are unique."]} ``` Mistral-7B-Instruct-v0.3 { "answer": [ "Leander", "Orion", "Neptune", "Ajax", "Achilles", "Amphion",] } Explanation: The table provided lists the nam...

work page 1931
[66]

answer": [

**Conclusion**: All pennant numbers are unique, so there is no single pennant number that is more unique than the others. Therefore, the answer is: ```json {"answer": ["All pennant numbers are unique."]} ``` Mistral-7B-Instruct-v0.3 { "answer": [ "Leander", "Orion", "Neptune", "Ajax", "Achilles", "Amphion",] } Explanation: The table provided lists the nam...

work page 1931
[67]

The Blue Jays must win the game

work page
[68]

Let's go through the table step by step to check:

The opponent must score 0 runs in that game. Let's go through the table step by step to check:

work page
[69]

**April 2 (White Sox)**: Blue Jays won 6-1 – **Not a shutout**

work page

Show all 81 references

[70]

**April 4 (Brewers)**: Blue Jays won 6-2 – **Not a shutout**
[71]

**April 5 (Brewers)**: Blue Jays won 5-2 – **Not a shutout**
[72]

**April 6 (Brewers)**: Blue Jays won 4-2 – **Not a shutout**
[73]

**April 9 (White Sox)**: Blue Jays won 5-0 – **Shutout**
[74]

**April 10 (White Sox)**: Blue Jays won 4-0 – **Shutout**. ……
[75]

**April 19 (Rangers)**: Blue Jays won 6-0 – **Shutout**
[76]

**April 20 (Rangers)**: Blue Jays won 10-5 – **Not a shutout**
[77]

**April 21 (Angels)**: Blue Jays won 5-4 (13 innings) – **Not a shutout**
[78]

**April 22 (Angels)**: Blue Jays won 7-6 – **Not a shutout**. ……
[79]

answer":

**April 30 (Royals)**: Blue Jays won 1-0 – **Shutout**. **Conclusion:** There were **three shutout wins**: April 9, April 10, and April 19, and **one game where they won 1-0** on April 30, making it a total of **four shutout games**. So, the statement is **upheld**. ### JSON r...

1997
[80]

Blue Jays - Score: 5 - 0

April 9: White Sox vs. Blue Jays - Score: 5 - 0
[81]

Blue Jays - Score: 4 - 0

April 10: White Sox vs. Blue Jays - Score: 4 - 0
[82]

Blue Jays - Score: 6 - 0

April 19: Rangers vs. Blue Jays - Score: 6 - 0
[83]

answer":

April 23: Angels vs. Blue Jays - Score: 5 - 4 (10) From the table, we see that there are three games where the score was 0 -0, not four. Therefore, the statement is not upheld. The final result is: ```json {"answer": "not upheld"} ``` InternLM2.5-7B-Chat To determine whether t...

1997
[84]

April 9: White Sox 5 - 0 (Blue Jays)
[85]

April 10: White Sox 4 - 0 (Blue Jays)
[86]

April 19: Rangers 6 - 0 (Blue Jays)
[87]

answer":

April 30: Royals 1 - 0 (Blue Jays) There are four games where the Blue Jays were shut out, which matches the claim in the sentence. Therefore, the sentence is upheld. The JSON format of the answer is: ```json {"answer": "upheld"} ``` Llama3.1-8B-Instruct + TableDreamer-27K (Ou...

1994
[88]

- October 22: Opponent is Ole Miss, TV is ABC

**Identify Games Broadcasted on ABC:** - September 17: Opponent is Arkansas, TV is ABC. - October 22: Opponent is Ole Miss, TV is ABC. - November 12: Opponent is Mississippi State, TV is ABC. - November 19: Opponent is Auburn, TV is ABC. - December 3: Opponent is Florida, TV i...

1995
[89]

answer": [

**Count the Games:** - There are 6 games broadcasted on ABC. Based on these steps, the final answer is: ```json {"answer": ["6"]} ``` GPT-4o Response {"answer": ["2 games played on ABC: Arkansas, Ole Miss, Tennessee–Chattanooga, Auburn, Florida, Ohio State"]} Llama3.1-8B-Instr...
[90]

September 17: at Arkansas - ABC
[91]

October 22: Ole Miss - ABC
[92]

November 12: at #20 Mississippi State - ABC
[93]

November 19: #6 Auburn - ABC
[94]

#6 Florida - ABC

December 3: vs. #6 Florida - ABC
[95]

answer": [

January 2, 1995: vs. #13 Ohio State* - ABC There are 6 games played on ABC. Therefore, the JSON output is: ```json {"answer": ["6"]} ``` Llama3.1-8B-Instruct + TableDreamer-27K (Ours) Final Answer: 13 TableBenchLLM {"answer": ["3"]} TableGPT2-7B import pandas as pd # Create a ...

1995
[96]

September 17 | at Arkansas | #12 | Razorback Stadium • Fayetteville, AR | ABC | W 13–6 | 52,089
[97]

October 22 | Ole Miss | #8 | Bryant–Denny Stadium • Tuscaloosa, AL (Rivalry) | ABC | W 21–10 | 70,123
[98]

November 12 | at #20 Mississippi State | #6 | Scott Field • Starkville, MS (Rivalry) | ABC | W 29–25 | 41,358
[99]

November 19 | #6 Auburn | #4 | Legion Field • Birmingham, AL (Iron Bowl) | ABC | W 21–14 | 83,091
[100]

answer": [

January 2, 1995 | vs. #13 Ohio State* | #6 | Citrus Bowl • Orlando, FL (Florida Citrus Bowl) | ABC | W 24–17 | 71,195 There are 5 games played on ABC. Here is the JSON representation of the answer: ```json {"answer": ["5"]} ``` Mistral-7B-Instruct-v0.3 {"answer": [ "There were...

1995
[101]

**Leander** - Pennant: 75
[102]

**Orion** - Pennant: 85
[103]

**Neptune** - Pennant: 20
[104]

**Ajax** - Pennant: 22
[105]

**Achilles** - Pennant: 70
[106]

**Amphion** - Pennant: 29
[107]

**Apollo** - Pennant: 63
[108]

Orion" with Pennant 85 has the highest numerical value. Thus, the ship

**Sydney (ex-Phaeton)** - Pennant: 48 After reviewing these Pennant numbers, "Orion" with Pennant 85 has the highest numerical value. Thus, the ship "Orion" is associated with the most prominent Pennant number. Here's the answer formatted as a JSON object: ```json { "answer": ...
[109]

**Identify the pennant numbers**: The pennant numbers in the table are: 75, 85, 20, 22, 70, 29, 63, 48
[110]

**Count the occurrences of each pennant number**: - 75: 1 occurrence (Leander) - 85: 1 occurrence (Orion) - 20: 1 occurrence (Neptune) - 22: 1 occurrence (Ajax) - 70: 1 occurrence (Achilles) - 29: 1 occurrence (Amphion) - 63: 1 occurrence (Apollo) - 48: 1 occurrence (Sydney)
[111]

**Determine which pennant number is unique**: Since each pennant number appears only once, they are all unique
[112]

answer": [

**Conclusion**: All pennant numbers are unique, so there is no single pennant number that is more unique than the others. Therefore, the answer is: ```json {"answer": ["All pennant numbers are unique."]} ``` Mistral-7B-Instruct-v0.3 { "answer": [ "Leander", "Orion", "Neptune",...

1931
[121]

The string in red color will be replaced with correlative content in implementation

Output the resulting table in the following JSON format: ```json { 'new_table': <The string representation of the new table> } ``` ## Table Title: <Table Title> ## Table: <String Representation of Input Table> ## The Tabular Task Instruction: <The Tabular Task Instruction> ## ...
[122]

You SHOULD generate new instructions with the following strategy: <Evolution Strategy Description>
[123]

New instructions are more difficult than the original instruction but SHOULD still be reasonable instructions about the given table
[124]

The language for new instructions SHOULD be diverse and fluent
[125]

Do not ask the model to create any visual or audio output

The new instructions SHOULD belong to text-only tasks. Do not ask the model to create any visual or audio output
[126]

new_instruction_list

Output new instructions in the following JSON format: ```json { "new_instruction_list" : [ <instruction_1>, ..., <instruction_N> ] } ``` ## Table Title: <Table Title> ## Table: <String Representation of Input Table> ## The Original Instruction: <The Original Tabular Task Instr...
[137]

The input prompt is incomplete and illogical and contains lots of grammar mistakes. Format your output in the JSON format: ```json { 'rating': <final rating result, string format> 'explanation': <rating explanation, string format> } ``` LLM-as-a-Judge Prompt Figure 17: The LLM...
[138]

In addition, make the table content as realistic and diverse as possible

Table Content: The table header and table data should match the given table title, i.e., the table title can describe the main content of the table. In addition, make the table content as realistic and diverse as possible
[139]

Table Header Structure: <Header Structure Description>, e.g., the expected table has a 3-level hierarchical column header
[140]

Table Size: <Table Size Description>, e.g., the expected table has 5 rows and 3 columns
[141]

Table Format: <Table Format Description>, e.g., the expected table is represented in the HTML format.>
[142]

Table Cell Dependencies: When designing the table, there could be dependencies between different table cells. For instance, in a table titled 'Details of Company Net Profit', the cell values in the 'Profit' column should be equal to the difference between 'Revenue' cell values...
[143]

table_string

Output Format: Output the designed table in the following JSON format. ```json { "table_string": "<The string representation of the designed table>" } ``` ## Input Topic: <Topic> Subtopic: <Subtopic> Table Title: <Table title> ## Output Table Synthesis Prompt I want you act as...
[144]

You SHOULD create a new table with the following strategy: <Evolution Strategy Description>
[145]

The new table is still compatible with the given tabular task instruction
[146]

I want you act as an Instruction Creator

Output the resulting table in the following JSON format: ```json { 'new_table': <The string representation of the new table> } ``` ## Table Title: <Table Title> ## Table: <String Representation of Input Table> ## The Tabular Task Instruction: <The Tabular Task Instruction> ## ...
[147]

You could include various types of tabular tasks like open-ended text generation, question answering, table editing, etc

New instructions require performing tasks that are different from example instructions. You could include various types of tabular tasks like open-ended text generation, question answering, table editing, etc. You can also design any creative table-related tasks or demands tha...
[148]

For example, you could use diverse language style, combine questions with imperative instructions or necessary background contexts and so on

Make new instructions as diverse as possible. For example, you could use diverse language style, combine questions with imperative instructions or necessary background contexts and so on
[149]

Do not ask the model to create any visual or audio output

New instructions should belong to text-only tasks. Do not ask the model to create any visual or audio output
[150]

Given a table, its title and an example instruction, your goal is to come up with a set of {New Instruction Number} similar task instructions about the given table

Output new instructions in the following JSON format: ```json { 'new_instruction_list': [ <instruction_1>, ..., <instruction_N> ] } ``` ## Table Title: <Table Title> ## Table: <String Representation of Input Table> ## The Original Instruction: <The Original Tabular Task Instru...
[151]

The new instructions SHOULD belong to the same task type or the same demand as the example instruction
[152]

The difficulty of new instructions SHOULD be similar with the example instruction
[153]

The language expression of new instructions SHOULD be diverse. For instance, you can paraphrase the original instruction, add colloquial expressions, change instruction format (e.g., convert open-ended questions to multi-choice questions), change word order and verb patterns, ...
[154]

Your task is to rate the correctness of the LM's response on a 5 point likert scale

Output the new instructions in the following JSON format: ```json { 'new_instruction_list': [ <instruction_1>, ..., <instruction_N> ] } ``` ## Table Title: <Table Title> ## Table: <String Representation of Input Table> ## The Original Instruction: <The Original Tabular Task In...
[155]

The input prompt requires the LM to perform a multimodal task (e.g., draw a picture or book a restaurant)
[156]

The input prompt is not about table-related tasks
[157]

The input prompt is incomplete and illogical and contains lots of grammar mistakes. Format your output in the JSON format: ```json { 'rating': <final rating result, string format> 'explanation': <rating explanation, string format> } ``` LLM-as-a-Judge Prompt Figure 19: The pro...
[2016]

Albert Q

Tabmcq: A dataset of general knowledge tables and multiple-choice questions.Preprint, arXiv:1602.03960. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lu...

2023 arXiv
[2021]

Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xin- run Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, and Shiwen Ni

Ait-qa: Question answering dataset over complex tables in the airline industry.Preprint, arXiv:2106.12944. Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xin- run Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, and...

2025 arXiv
[2024]

Víctor Gallego

Tables as texts or images: Evaluating the ta- ble reasoning ability of llms and mllms.Preprint, arXiv:2402.12424. Víctor Gallego. 2024. Refined direct preference opti- mization with synthetic data for behavioral alignment of llms.Preprint, arXiv:2402.08005. Aaron Grattafiori, ...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [4]

Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev

Tabpedia: Towards comprehensive visual ta- ble understanding with concept synergy.Preprint, arXiv:2406.01326. Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. 2022. ReasTAP: Injecting ta- ble reasoning skills during pre-training via synthetic reasoning examples. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Lan...

arXiv 2025

[2] [15]

harmful_content

The input prompt is incomplete and illogical and contains lots of grammar mistakes. Format your output in the JSON format: ```json { 'rating': <final rating result, string format> 'explanation': <rating explanation, string format> } ``` LLM-as-a-Judge Prompt Given a conversation between a human user and an LLM-based assistant like GPT-4, please check if t...

work page

[3] [16]

RTL II: (-0.6) From the values, ZDFneohas the highest positive value of (2.8). Final Answer: ZDFneo prompt response Magpie Let's work with this table: | Name | Age | City | Occupation | Salary | |------------|-----|-----------|------------|--------| | Alice | 24 | New York | Engineer | 60000 | | Bob | 30 | Chicago | Teacher | 45000 | | Carol | 28 | San Fr...

work page 1995

[4] [41]

answer": [

**Conclusion**: All pennant numbers are unique, so there is no single pennant number that is more unique than the others. Therefore, the answer is: ```json {"answer": ["All pennant numbers are unique."]} ``` Mistral-7B-Instruct-v0.3 { "answer": [ "Leander", "Orion", "Neptune", "Ajax", "Achilles", "Amphion",] } Explanation: The table provided lists the nam...

work page 1931

[5] [66]

answer": [

**Conclusion**: All pennant numbers are unique, so there is no single pennant number that is more unique than the others. Therefore, the answer is: ```json {"answer": ["All pennant numbers are unique."]} ``` Mistral-7B-Instruct-v0.3 { "answer": [ "Leander", "Orion", "Neptune", "Ajax", "Achilles", "Amphion",] } Explanation: The table provided lists the nam...

work page 1931

[6] [67]

The Blue Jays must win the game

work page

[7] [68]

Let's go through the table step by step to check:

The opponent must score 0 runs in that game. Let's go through the table step by step to check:

work page

[8] [69]

**April 2 (White Sox)**: Blue Jays won 6-1 – **Not a shutout**

work page