pith. sign in

arxiv: 2604.05114 · v1 · submitted 2026-04-06 · 💻 cs.CL · cs.AI· cs.LG

π²: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models

Quyet V. Do , Thinh Pham , Nguyen Nguyen , Sha Li , Pratibha Zunjare , Tu Vu This is my paper

Pith reviewed 2026-05-10 18:43 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG
keywords long-context reasoningdata curationsupervised fine-tuningmulti-hop reasoningWikipedia tablesreasoning tracesself-distillationLLM improvement
0
0 comments X

The pith

A pipeline that turns Wikipedia tables into verified multi-hop reasoning questions and traces improves long-context accuracy in LLMs after supervised fine-tuning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces π², a three-stage curation process that starts with tables extracted from Wikipedia, generates realistic analytical questions whose answers can be checked by running code in two independent ways, and then produces step-by-step reasoning traces that serve as training targets. Supervised fine-tuning on the resulting dataset raises average accuracy by 4.3 percent on four public long-context reasoning benchmarks for one model and by 2.7 percent for another, while the same model also improves when trained only on traces it generated itself. A sympathetic reader would care because current long-context reasoning gains often require either vastly more parameters or hand-crafted synthetic data; here the improvement comes from systematically turning existing structured web content into verifiable reasoning examples.

Core claim

The π² pipeline extracts and expands tables from Wikipedia, generates multi-hop questions whose answers are automatically verified through dual-path code execution, and back-translates structured reasoning traces into natural-language solutions given realistic web-search context; supervised fine-tuning on the resulting data produces consistent accuracy gains across long-context reasoning benchmarks.

What carries the argument

The π² curation pipeline that converts structured tables into code-verified multi-hop QA pairs paired with step-by-step reasoning traces for supervised fine-tuning.

If this is right

  • Models fine-tuned on π² data show measurable gains on multiple long-context reasoning benchmarks without changes to architecture or inference-time methods.
  • The same model can improve its own performance by training on reasoning traces it generated from the π² questions, demonstrating a self-distillation effect.
  • The approach scales with the availability of structured tables rather than requiring new human annotation for each reasoning example.
  • Gains appear for both a 20-billion-parameter model and a 4-billion-parameter model, suggesting the data benefit is not limited to a single scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The code-verification step could be extended to other structured sources such as databases or spreadsheets to generate reasoning data in additional domains.
  • Because answers are checked by independent code paths, the resulting traces may contain fewer factual errors than purely model-generated chains, offering a route to lower hallucination rates in reasoning.
  • Future experiments could test whether the same pipeline produces gains when the context length during training or evaluation is increased by another order of magnitude.

Load-bearing premise

The questions and reasoning traces produced by the pipeline are meaningfully higher quality for long-context reasoning than existing datasets or equivalent amounts of randomly scaled training data.

What would settle it

Training the same models on a control dataset that matches π² exactly in size, format, and token count but is drawn from existing long-context sources, then measuring whether the accuracy gains on the same benchmarks disappear or shrink below 1 percent.

Figures

Figures reproduced from arXiv: 2604.05114 by Nguyen Nguyen, Pratibha Zunjare, Quyet V. Do, Sha Li, Thinh Pham, Tu Vu.

Figure 1
Figure 1. Figure 1: π 2 curation pipeline. We 1) collect tables from Wikipedia and expand them with new columns when the conditions are met, then for each table, 2) generate a multi-hop analytical reasoning question paired with an executable SQL query and verify the answer with an independent Python-implemented solution. Finally, we 3) produce structured analytical reasoning traces through back-translation. Wikipedia and empl… view at source ↗
Figure 2
Figure 2. Figure 2: Histogram (in a log scale) of reasoning-trace lengths generated by [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparing output length (in tokens) of our models, base models, larger open [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The evidence as a part of the context of the question [PITH_FULL_IMAGE:figures/full_fig_p029_5.png] view at source ↗
read the original abstract

We study a pipeline that curates reasoning data from initial structured data for improving long-context reasoning in large language models (LLMs). Our approach, $\pi^2$, constructs high-quality reasoning data through rigorous QA curation: 1) extracting and expanding tables from Wikipedia, 2) from the collected tables and relevant context, generating realistic and multi-hop analytical reasoning questions whose answers are automatically determined and verified through dual-path code execution, and 3) back-translating step-by-step structured reasoning traces as solutions of QA pairs given realistic web-search context. Supervised fine-tuning with \textsc{\small{gpt-oss-20b}} and \textsc{\small{Qwen3-4B-Instruct-2507}} on $\pi^2$ yields consistent improvements across four long-context reasoning benchmarks and our alike $\pi^2$-Bench, with average absolute accuracy gains of +4.3% and +2.7% respectively. Notably, our dataset facilitates self-distillation, where \textsc{\small{gpt-oss-20b}} even improves its average performance by +4.4% with its own reasoning traces, demonstrating $\pi^2$'s usefulness. Our code, data, and models are open-source at https://github.com/vt-pi-squared/pi-squared.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces the π² pipeline for curating reasoning data from structured Wikipedia tables: table extraction and expansion, generation of multi-hop analytical QA pairs with automatic dual-path code verification for answers, and back-translation of step-by-step structured reasoning traces as solutions. Supervised fine-tuning of gpt-oss-20b and Qwen3-4B-Instruct-2507 on this data yields average absolute accuracy gains of +4.3% and +2.7% across four long-context reasoning benchmarks plus the authors' π²-Bench; the dataset also supports self-distillation (+4.4% for gpt-oss-20b). All code, data, and models are open-sourced.

Significance. If the gains prove attributable to the structure-originated curation rather than scale or format, the work supplies a reproducible method for generating verified multi-hop reasoning traces from tables, which could aid long-context training. The open release of artifacts is a clear strength that supports verification and extension.

major comments (2)
  1. [Abstract] Abstract: the reported average gains (+4.3% on gpt-oss-20b, +2.7% on Qwen3-4B-Instruct-2507) are presented without any description of baseline data volumes, total token counts, or output-format matching, so it is impossible to determine whether the deltas arise from the π² curation steps or from incidental differences in training data scale.
  2. [Abstract] Abstract: the central claim that improvements stem specifically from 'structure-originated' elements (table extraction, dual code verification, back-translated traces) requires ablations that hold example count, token budget, and formatting fixed while varying only the generation pipeline; no such controls are described, leaving the attribution to the pipeline unsupported.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'our alike π²-Bench' is ambiguous; clarify whether this is a newly introduced benchmark or a re-use of an existing one and provide its construction details.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract and the attribution of results. We address each major comment point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the reported average gains (+4.3% on gpt-oss-20b, +2.7% on Qwen3-4B-Instruct-2507) are presented without any description of baseline data volumes, total token counts, or output-format matching, so it is impossible to determine whether the deltas arise from the π² curation steps or from incidental differences in training data scale.

    Authors: We agree that the abstract is too concise on this point and does not specify the controlled conditions. The full manuscript (Experiments section) states that all compared fine-tuning runs used matched example counts (~50k) and token budgets with aligned output formats. We will revise the abstract to briefly note these controlled conditions for self-containment. revision: yes

  2. Referee: [Abstract] Abstract: the central claim that improvements stem specifically from 'structure-originated' elements (table extraction, dual code verification, back-translated traces) requires ablations that hold example count, token budget, and formatting fixed while varying only the generation pipeline; no such controls are described, leaving the attribution to the pipeline unsupported.

    Authors: The referee is correct that the manuscript does not present the exact ablations isolating only the curation pipeline while holding scale and format fixed. Our reported comparisons use data of matched volume against standard SFT baselines, but without those precise controls. We will add a dedicated ablation subsection or expanded discussion in the revised manuscript and note that the open-sourced code and data enable independent verification of the pipeline's contribution. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical SFT gains measured on held-out benchmarks

full rationale

The paper presents a data-curation pipeline (Wikipedia table extraction, multi-hop QA generation with dual code verification, back-translated reasoning traces) followed by supervised fine-tuning experiments. Reported results are absolute accuracy deltas on four external long-context reasoning benchmarks plus the authors' own π²-Bench. No mathematical derivations, equations, fitted parameters renamed as predictions, or self-referential definitions appear in the abstract or described methodology. The central claim is an empirical performance measurement that remains independent of the input curation steps and is directly falsifiable on standard benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The claim rests on the domain assumption that fine-tuning on curated high-quality reasoning traces transfers to improved long-context performance, plus the implicit assumption that the automatic verification step produces reliable labels.

axioms (1)
  • domain assumption Fine-tuning LLMs on high-quality step-by-step reasoning data improves their long-context reasoning ability.
    Standard assumption in the LLM post-training literature invoked to interpret the benchmark gains.

pith-pipeline@v0.9.0 · 5556 in / 1254 out tokens · 86693 ms · 2026-05-10T18:43:53.255717+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages · 1 internal anchor

  1. [1]

    gpt-oss-120b & gpt-oss-20b Model Card

    URLhttps://aclanthology.org/2025.emnlp-main.1025/. 11 Preprint. Under review. OpenAI. Gpt-5.4 thinking system card, 2026. URL https://deploymentsafety.openai.com/ gpt-5-4-thinking/gpt-5-4-thinking.pdf. [Accessed 30-03-2026]. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Hai...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    For a maximum stay of 15 days if the application was submitted at least 2 and a half months before arrival and if the clearance was obtained

    Turkmenistan Now I must check the specific condition for each of these candidates to see which one requires a sponsor to apply 48 hours in advance. * Step 4: Evaluating Candidates Against the Criteria Let's examine the conditions listed for each country in the table found in **Document 4** (and verified in **Document 6**): 15 Preprint. Under review. - **B...

    work page 2022

  3. [3]

    and" or

    **Question Design:** 1.1. Generate ONE single-focus and concise question. - DON'T concatenate multiple sub-questions with "and" or "which... and which". - Split multi-part questions into separate questions. For example, instead of "which X and which Y", ask only "which X". - Instead of "which X and which Y when X", just ask "When X, then which Y?" to main...

    work page 2014

  4. [4]

    the answer is

    **SQL Implementation:** - Write a SQL query assuming the table is loaded as`df`and the engine is SQLite. - Ensure the query is syntactically correct and optimized for performance. - Always wrap column names with quotes, as some column names may contain spaces or special characters. - Ensure the query returns a single definitive value or a small result set...

    work page

  5. [5]

    **Natural and exploratory**: Write as if discovering the answer in real-time

    work page

  6. [6]

    **Self-contained**: Each step should be understandable without re-reading previous steps

    work page

  7. [7]

    **Critical evaluation**: Don't accept information at face value; evaluate relevance

    work page

  8. [8]

    **Document-focused**: Ground all conclusions in the provided context

    work page

  9. [9]

    Wait, this document is from 2018, so it may be outdated

    **Conversational but precise and concise**: Use natural language while maintaining factual accuracy and efficiency ## Example Structure ``` * Step 1: [Understanding the question and planning strategy] * Step 2: [Search for candidates matching criterion A - examining documents] * Step 3: [Search for candidates matching criterion B - narrowing down] * Step ...

    work page 2018