Synthetic Contrastive Reasoning for Multi-Table Q&A

Ankit Pratap Singh; Phillip Howard; Xin Su

arxiv: 2606.05382 · v1 · pith:SSY4AXED · submitted 2026-06-03 · cs.AI

Synthetic Contrastive Reasoning for Multi-Table Q&A

Ankit Pratap Singh , Xin Su , Phillip Howard This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 05:59 UTCgrok-4.3pith:SSY4AXED record.json open to challenge →

classification cs.AI

keywords multi-table question answeringcontrastive preference optimizationsynthetic reasoning tracespreference pairsLLM fine-tuningMMQAcompositional reasoning

0 comments

The pith

Generating synthetic positive and negative reasoning traces then applying contrastive preference optimization lifts multi-table QA accuracy by 9.7-16.3 points on average.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a synthetic dataset of reasoning traces for multi-table question answering by prompting heterogeneous LLMs to produce validated positive traces that reach correct answers and plausible negative traces that reach incorrect ones. These pairs serve as preference data for Contrastive Preference Optimization, which replaces ordinary supervised fine-tuning on question-answer pairs alone. The resulting models show consistent gains across three open-weight LLMs, with the largest lift reaching 21 points on the MMQA benchmark. Ablation results indicate that mixing different generators for the positive and negative traces strengthens the signal, while automated and human checks find the pairs largely faithful and contrastive.

Core claim

By constructing preference pairs from synthetically generated positive and negative reasoning traces for the MMQA multi-table QA task and training open-weight LLMs with Contrastive Preference Optimization, absolute average performance improves 9.7-16.3 percentage points over standard Q&A supervised fine-tuning, with gains up to 21 points.

What carries the argument

Contrastive Preference Optimization (CPO) on synthetic preference pairs of positive and negative reasoning traces generated by heterogeneous LLMs.

If this is right

The same CPO procedure produces measurable gains on Qwen3-14B, Mistral-8B, and Llama-3.1-8B.
Mixing trace generators from different model families increases the strength of the contrastive signal relative to single-generator pairs.
The improvement holds when the only change from baseline is the addition of the synthetic reasoning preference data.
Automated metrics and human ratings both support that the generated pairs remain coherent and meaningfully contrastive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could be tested on other compositional QA settings where reasoning supervision is scarce, such as knowledge-base or database query tasks.
If negative traces systematically capture common error patterns, the method may incidentally improve robustness to those specific failure modes.
Scaling the generation step with larger or more diverse LLMs might further widen the performance gap over answer-only fine-tuning.

Load-bearing premise

The synthetically generated positive and negative traces are largely faithful to correct and incorrect reasoning paths and remain coherent enough to form useful contrastive pairs.

What would settle it

A human evaluation in which a majority of the generated negative traces prove to be valid reasoning paths or the positive traces contain frequent factual errors would remove the claimed contrastive training signal.

Figures

Figures reproduced from arXiv: 2606.05382 by Ankit Pratap Singh, Phillip Howard, Xin Su.

**Figure 1.** Figure 1: Illustration of our approach for generating positive & contrastive reasoning traces for multi-table Q&A [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: Impact of Model Diversity. (Top: MMQA, Bottom: MMTU) We observe that using a distinct model (e.g., Gemini 2.0) for generating reasoning traces consistently outperforms using the same model. F BIRD Semantic Consistency Verification Prompt [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

read the original abstract

Multi-table question answering requires models to retrieve relevant evidence, link schemas, and perform compositional reasoning across relational tables. Existing multi-table Q&A resources typically provide questions and final answers but lack reasoning supervision that explains how answers are derived. To address this gap, we construct a synthetic contrastive reasoning-trace dataset for MMQA by generating validated positive traces and plausible negative traces with heterogeneous LLMs. We then use the resulting preference pairs to fine-tune open-weight LLMs with Contrastive Preference Optimization (CPO). Across Qwen3-14B, Mistral-8B, and Llama-3.1-8B, CPO achieves absolute average improvements over Q&A supervised fine-tuning ranging from 9.7%-16.3%, with gains up to 21 percentage points on MMQA. Ablations show that heterogeneous positive and negative trace generators strengthen the contrastive signal, and automated as well as human evaluations indicate that the generated pairs are largely faithful, coherent, and meaningfully contrastive.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CPO on synthetic contrastive traces from heterogeneous LLMs produces reported gains on multi-table QA, but the load-bearing claim rests on unquantified trace quality.

read the letter

The main thing here is that the authors build a new synthetic preference dataset for multi-table question answering by having different LLMs produce positive reasoning traces and plausible negative ones, then fine-tune with CPO. They report average gains of 9.7-16.3% over plain SFT across Qwen3-14B, Mistral-8B, and Llama-3.1-8B, with some tasks seeing up to 21 points. Ablations indicate the heterogeneous generators add value to the contrast.

This is a straightforward extension of synthetic data and preference optimization techniques to the multi-table setting. The work supplies a concrete dataset construction pipeline and shows measurable lifts on the target task, which is useful for anyone trying to improve compositional reasoning over relational tables.

The soft spot is exactly what the stress-test flags: the positive traces need to be verifiably correct derivations and the negatives need to be meaningfully wrong rather than just stylistically different. The abstract mentions automated and human checks for faithfulness and coherence, but gives no error rates on the positives, no inter-annotator numbers, and no description of how validation was operationalized. If those checks are weak or themselves LLM-based, the gains could be fragile. The paper would be stronger with a clearer breakdown of trace accuracy and any leakage checks.

No circularity or self-referential fitting issues appear. The results are empirical and the citation pattern looks standard.

This is for people working on LLM fine-tuning for database-style QA or synthetic preference data. A reader who cares about practical gains on multi-hop table tasks will find the numbers and ablations worth looking at. It is coherent enough on its own terms to deserve referee time, even with the data-quality questions.

Referee Report

2 major / 2 minor

Summary. The paper constructs a synthetic contrastive reasoning-trace dataset for multi-table Q&A on MMQA by using heterogeneous LLMs to generate validated positive traces and plausible negative traces. These preference pairs are then used to fine-tune open-weight models (Qwen3-14B, Mistral-8B, Llama-3.1-8B) via Contrastive Preference Optimization (CPO). The central claim is that CPO yields absolute average gains of 9.7–16.3% over standard Q&A supervised fine-tuning, with peaks of 21 points on MMQA; ablations attribute part of the signal to heterogeneous generators, and automated plus human evaluations are said to confirm that the pairs are largely faithful, coherent, and contrastive.

Significance. If the trace quality claims hold, the work supplies a scalable route to reasoning supervision for multi-table QA, an area where existing resources supply only questions and answers. The reported gains are large enough to be practically relevant, the multi-model evaluation and ablations on generator heterogeneity are positive features, and the explicit use of both automated and human checks on the synthetic data is a methodological strength.

major comments (2)

[Abstract] Abstract (dataset construction paragraph): the headline gains rest on the assumption that positive traces are verifiably correct compositional derivations and negative traces are plausible yet incorrect. The manuscript states that traces are 'validated' and that automated/human evaluations support faithfulness, but supplies no quantitative breakdown (error rate on positive traces, inter-annotator agreement, operational definition of 'validated', or judge model details). This is load-bearing for interpreting the 9.7–16.3% improvements as evidence of improved reasoning rather than artifacts.
[Experiments] Experiments / Results section: no error bars, statistical significance tests, or exact train/validation/test splits are reported for the CPO versus SFT comparisons. Without these, the reliability of the up-to-21-point MMQA gain and the cross-model average cannot be assessed.

minor comments (2)

The abstract mentions 'heterogeneous LLMs' for trace generation but does not name the specific models or prompting templates used; adding this information would improve reproducibility.
Consider adding a table that reports the automated and human evaluation metrics (e.g., faithfulness scores, coherence ratings) broken down by positive/negative traces and by generator model.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments correctly identify areas where additional quantitative details would strengthen the presentation of our dataset validation and experimental results. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

Referee: [Abstract] Abstract (dataset construction paragraph): the headline gains rest on the assumption that positive traces are verifiably correct compositional derivations and negative traces are plausible yet incorrect. The manuscript states that traces are 'validated' and that automated/human evaluations support faithfulness, but supplies no quantitative breakdown (error rate on positive traces, inter-annotator agreement, operational definition of 'validated', or judge model details). This is load-bearing for interpreting the 9.7–16.3% improvements as evidence of improved reasoning rather than artifacts.

Authors: We agree that a quantitative breakdown of the validation process is necessary to fully support the claims about trace quality. The manuscript currently states that automated and human evaluations confirm the pairs are largely faithful, coherent, and contrastive, but does not report specific metrics such as error rates on positive traces, inter-annotator agreement, the precise operational definition of 'validated', or judge-model details. In the revised manuscript we will add these quantitative details in the dataset-construction section, including error rates, agreement scores, and judge-model specifications. revision: yes
Referee: [Experiments] Experiments / Results section: no error bars, statistical significance tests, or exact train/validation/test splits are reported for the CPO versus SFT comparisons. Without these, the reliability of the up-to-21-point MMQA gain and the cross-model average cannot be assessed.

Authors: We acknowledge that the absence of error bars, statistical significance tests, and exact data splits limits the ability to assess result reliability. The current manuscript reports the absolute gains but omits these elements. In the revised Experiments and Results sections we will include error bars (standard deviations across multiple runs where applicable), statistical significance tests for the CPO versus SFT differences, and the precise train/validation/test split ratios and sizes used for each model. revision: yes

Circularity Check

0 steps flagged

Empirical fine-tuning study; no derivation chain reduces to inputs

full rationale

The paper describes dataset construction via LLM-generated traces, human/automated validation, and measured accuracy gains from CPO fine-tuning versus SFT baselines on MMQA and other benchmarks. No equations, predictions, or first-principles claims are present that could reduce by construction to fitted parameters, self-citations, or renamed inputs. All reported results are external empirical measurements. This is the standard case of a self-contained empirical ML paper.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the domain assumption that heterogeneous LLMs can produce faithful positive and plausible negative reasoning traces suitable for contrastive training. No free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption Heterogeneous LLMs can generate validated positive traces and plausible negative traces that are faithful, coherent, and meaningfully contrastive.
This premise underpins the creation of the preference pairs used for CPO fine-tuning.

pith-pipeline@v0.9.1-grok · 5698 in / 1434 out tokens · 47267 ms · 2026-06-28T05:59:58.625965+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 2 canonical work pages

[1]

InProceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21223– 21261

Uncovering the impact of chain-of-thought rea- soning for direct preference optimization: Lessons from text-to-sql. InProceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21223– 21261. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy L...

work page arXiv 2024
[2]

TableGPT2: A Large Multimodal Model with Tabular Data Integration

Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741. Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold.Advances in Neural Informat...

work page Pith review arXiv 2024
[3]

pages 6024–6044

Tablellama: Towards open large generalist models for tables. pages 6024–6044. 11 A Prompt We provide complete details of our prompts in Ta- bles 9, 10, and 11. B Additional Results Figure 2 provides an ablation study illustrating the usefulness of using multiple models for generating contrastive reasoning traces. Figure 3 compares the utility of reasoning...

2024
[4]

Stage 1 (Q&A SFT):The models were fine- tuned using Supervised Fine-Tuning (SFT) on standard Q&A pairs ( [Question], [T ables], [Answer]) from the training setT
[5]

Stage 2 (Trace SFT):We fine-tuned the mod- els on the question and synthetically gener- ated positive reasoning traces ( [Question], [T ables],+y)
[6]

This stage was initialized using the model weights from Stage 2

Stage 3 (CPO Alignment):We utilized Con- trastive Preference Optimization (CPO) (Xu et al., 2024) with contrastive reasoning traces (+y,−y) . This stage was initialized using the model weights from Stage 2. Hyperparameters and ConfigurationWe uti- lized Low-Rank Adaptation (LoRA) (Hu et al.,

2024
[7]

Yes” only when confident that the SQL query exactly captures the question’s intent, and “No

for parameter-efficient fine-tuning across all stages, though with distinct configurations for SFT and CPO to balance plasticity and stability. • SFT Configuration (Stage 1 & 2):To facili- tate the learning of complex reasoning struc- tures, we used a LoRA configuration with r= 1056 and α= 1056 . We targeted all linear projection layers, including attenti...

2025
[9]

Result:ID 2, ID 5

Identify entries in TABLE2 with attribute_id 4 and value ’1’. Result:ID 2, ID 5
[10]

2.Retrieve pricesfor these IDs from TABLE1.(Skipped constraint verification) 3.Verification: 3

Look up these IDs in TABLE1 to check catalog_level_number. 2.Retrieve pricesfor these IDs from TABLE1.(Skipped constraint verification) 3.Verification: 3. Price for ID 2 is 687.59. - ID 2 is Level 8 (✓Match) - ID 5 is Level 9 (✗Mismatch - Exclude) 4. Price for ID 5 is 616.22
[11]

5.Sum pricesof both entries

Sum prices of matching entries (only ID 2). 5.Sum pricesof both entries
[12]

Total:687.59 + 616.22 = 1303.81 Final Answer:687.59✓ Final Answer:1303.81✗ Table 13: Comparison of constraint verification

Total: 687.59 6. Total:687.59 + 616.22 = 1303.81 Final Answer:687.59✓ Final Answer:1303.81✗ Table 13: Comparison of constraint verification. 16

[1] [1]

InProceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21223– 21261

Uncovering the impact of chain-of-thought rea- soning for direct preference optimization: Lessons from text-to-sql. InProceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21223– 21261. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy L...

work page arXiv 2024

[2] [2]

TableGPT2: A Large Multimodal Model with Tabular Data Integration

Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741. Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold.Advances in Neural Informat...

work page Pith review arXiv 2024

[3] [3]

pages 6024–6044

Tablellama: Towards open large generalist models for tables. pages 6024–6044. 11 A Prompt We provide complete details of our prompts in Ta- bles 9, 10, and 11. B Additional Results Figure 2 provides an ablation study illustrating the usefulness of using multiple models for generating contrastive reasoning traces. Figure 3 compares the utility of reasoning...

2024

[4] [4]

Stage 1 (Q&A SFT):The models were fine- tuned using Supervised Fine-Tuning (SFT) on standard Q&A pairs ( [Question], [T ables], [Answer]) from the training setT

[5] [5]

Stage 2 (Trace SFT):We fine-tuned the mod- els on the question and synthetically gener- ated positive reasoning traces ( [Question], [T ables],+y)

[6] [6]

This stage was initialized using the model weights from Stage 2

Stage 3 (CPO Alignment):We utilized Con- trastive Preference Optimization (CPO) (Xu et al., 2024) with contrastive reasoning traces (+y,−y) . This stage was initialized using the model weights from Stage 2. Hyperparameters and ConfigurationWe uti- lized Low-Rank Adaptation (LoRA) (Hu et al.,

2024

[7] [7]

Yes” only when confident that the SQL query exactly captures the question’s intent, and “No

for parameter-efficient fine-tuning across all stages, though with distinct configurations for SFT and CPO to balance plasticity and stability. • SFT Configuration (Stage 1 & 2):To facili- tate the learning of complex reasoning struc- tures, we used a LoRA configuration with r= 1056 and α= 1056 . We targeted all linear projection layers, including attenti...

2025

[8] [9]

Result:ID 2, ID 5

Identify entries in TABLE2 with attribute_id 4 and value ’1’. Result:ID 2, ID 5

[9] [10]

2.Retrieve pricesfor these IDs from TABLE1.(Skipped constraint verification) 3.Verification: 3

Look up these IDs in TABLE1 to check catalog_level_number. 2.Retrieve pricesfor these IDs from TABLE1.(Skipped constraint verification) 3.Verification: 3. Price for ID 2 is 687.59. - ID 2 is Level 8 (✓Match) - ID 5 is Level 9 (✗Mismatch - Exclude) 4. Price for ID 5 is 616.22

[10] [11]

5.Sum pricesof both entries

Sum prices of matching entries (only ID 2). 5.Sum pricesof both entries

[11] [12]

Total:687.59 + 616.22 = 1303.81 Final Answer:687.59✓ Final Answer:1303.81✗ Table 13: Comparison of constraint verification

Total: 687.59 6. Total:687.59 + 616.22 = 1303.81 Final Answer:687.59✓ Final Answer:1303.81✗ Table 13: Comparison of constraint verification. 16