pith. sign in

arxiv: 2606.18831 · v1 · pith:VMXOH6SEnew · submitted 2026-06-17 · 💻 cs.CL · cs.AI

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

Xiaoyue Xu , Sikui Zhang , Xiaorong Wang , Xu Han , Chaojun Xiao This is my paper

Pith reviewed 2026-06-26 21:09 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords long-context reasoningreinforcement learningdata curationGRPOretrieval tasksmulti-evidence synthesisreasoning tasksagentic tasks
0
0 comments X

The pith

A data recipe of eight datasets for retrieval, synthesis and reasoning improves long-context reasoning with only basic outcome-based RL.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that long-context reasoning improves substantially when training data is carefully composed, even with minimal reinforcement learning. It constructs a recipe around three task families—retrieval, multi-evidence synthesis, and reasoning—by curating eight datasets that total roughly 14,000 examples. When this data is paired with a simple outcome-based GRPO procedure on Qwen3 models of 4B, 8B, and 30B scale, average scores rise by several points on seven long-context benchmarks and exceed results from earlier RL data collections. The same recipe further lifts performance when applied to an already agent-tuned model on GAIA and BrowseComp.

Core claim

A simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning by targeting retrieval, multi-evidence synthesis, and reasoning tasks with eight curated datasets totaling about 14K examples.

What carries the argument

The data recipe of eight datasets spanning three complementary task families that supplies the training signal for the outcome-based GRPO procedure.

If this is right

  • Average gains of +7.2, +3.2 and +6.4 points appear across seven long-context benchmarks for the three model sizes tested.
  • The recipe surpasses the results obtained from prior RL training sets of comparable scale.
  • Continuing RL training on an agent-tuned model with the same recipe raises GAIA by 4.8 points and BrowseComp by 7.0 points.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Data curation focused on these three task families may serve as a reusable template for improving other forms of multi-step reasoning.
  • Public release of the eight datasets would let others test whether the gains hold when the GRPO details or base models are altered.
  • Emphasizing data composition over reward design could reduce the engineering effort needed to strengthen long-context capabilities in new model families.

Load-bearing premise

The performance gains are driven primarily by the quality and composition of the eight curated datasets rather than by model-specific tuning, benchmark overlap, or unstated details of the GRPO implementation.

What would settle it

Re-running the identical GRPO training loop on the same model sizes but with eight datasets of comparable size that omit the targeted retrieval, synthesis, and reasoning families and observing whether the benchmark improvements largely disappear.

Figures

Figures reproduced from arXiv: 2606.18831 by Chaojun Xiao, Sikui Zhang, Xiaorong Wang, Xiaoyue Xu, Xu Han.

Figure 1
Figure 1. Figure 1: A long-context reasoning example requiring [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training on our max-64K-token data recipe [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: LLM-as-judge prompt used during RL train [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Input length distribution (in characters) of [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: LLM-as-judge prompt used for evaluation. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: LLM-as-judge prompt used for AA-LCR and DocFinQA evaluation. Model L1 (n=42) L2 (n=66) L3 (n=19) AgentCPM-Explore 78.6 65.2 42.1 + Ours (25 steps) 81.0 63.6 52.6 + Ours (50 steps) 85.7 66.7 52.6 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
read the original abstract

Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper claims that a simple yet effective data recipe—eight curated datasets (~14K examples) targeting retrieval, multi-evidence synthesis, and reasoning—paired with a minimal outcome-based GRPO setup suffices to substantially improve long-context reasoning. On Qwen3-4B/8B/30B-A3B models this yields average gains of +7.2/+3.2/+6.4 across seven long-context benchmarks, outperforming prior RL training sets; the gains transfer to agentic tasks (+4.8 GAIA, +7.0 BrowseComp). The datasets will be released.

Significance. If the reported gains are shown to be driven by the specific composition of the eight datasets rather than GRPO implementation details, model-specific tuning, or benchmark overlap, the work would offer a practical, data-centric alternative to reward engineering for long-context RL. Releasing the datasets would be a clear reproducibility strength. The empirical focus on external benchmarks is appropriate, but the central attribution claim requires stronger isolation evidence.

major comments (3)
  1. [Experiments] Experiments section (results tables reporting +7.2/+3.2/+6.4 gains): the central claim that the data recipe alone drives the improvements is not isolated. No ablation applies the identical minimal GRPO setup to non-curated long-context data or random long-context examples, so it remains possible that the gains arise from the GRPO procedure itself or from model-specific tuning rather than the retrieval/synthesis/reasoning composition.
  2. [Data Construction] Data construction / training details: no overlap statistics or contamination analysis are supplied between the ~14K training examples and the seven evaluation benchmarks. This is load-bearing for the attribution claim; even modest leakage could account for part of the reported deltas.
  3. [Results] Results presentation (abstract and main results tables): average point gains are stated without per-benchmark variance, number of runs, or statistical significance tests. Full experimental details, baselines, and controls are required to substantiate the claim that the recipe “suffices.”
minor comments (2)
  1. [Abstract] The abstract would be clearer if it named the seven benchmarks and briefly noted the GRPO hyper-parameters used.
  2. [Introduction] Notation for the three task families (retrieval, synthesis, reasoning) should be defined once and used consistently in tables and text.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight important aspects of result attribution and experimental rigor. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Experiments] Experiments section (results tables reporting +7.2/+3.2/+6.4 gains): the central claim that the data recipe alone drives the improvements is not isolated. No ablation applies the identical minimal GRPO setup to non-curated long-context data or random long-context examples, so it remains possible that the gains arise from the GRPO procedure itself or from model-specific tuning rather than the retrieval/synthesis/reasoning composition.

    Authors: We agree that stronger isolation of the data composition would strengthen the central claim. Our existing comparisons are to prior RL training sets, but we did not run the identical minimal GRPO on a non-curated or random long-context control set. We will add this ablation to the revised manuscript. revision: yes

  2. Referee: [Data Construction] Data construction / training details: no overlap statistics or contamination analysis are supplied between the ~14K training examples and the seven evaluation benchmarks. This is load-bearing for the attribution claim; even modest leakage could account for part of the reported deltas.

    Authors: We acknowledge the omission. We will compute and report n-gram overlap and exact-match contamination statistics between the training data and all evaluation benchmarks, along with the analysis methodology, in the revised data construction section. revision: yes

  3. Referee: [Results] Results presentation (abstract and main results tables): average point gains are stated without per-benchmark variance, number of runs, or statistical significance tests. Full experimental details, baselines, and controls are required to substantiate the claim that the recipe “suffices.”

    Authors: We agree that variance, run counts, and significance testing would improve substantiation. We will expand the results tables and text to report per-benchmark standard deviations (from multiple runs where available), add statistical tests, and provide fuller baseline and control details in the revised manuscript. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical data recipe with external benchmarks

full rationale

The paper is an empirical study that curates eight datasets across retrieval, synthesis, and reasoning tasks (~14K examples), applies a minimal outcome-based GRPO setup to Qwen3 models, and reports measured gains (+7.2/+3.2/+6.4) on seven external long-context benchmarks plus transfer to agentic tasks. No equations, derivations, or fitted parameters are used to generate the results; performance deltas are directly observed against held-out benchmarks. The central claim rests on these external measurements rather than any self-definitional reduction, fitted-input prediction, or load-bearing self-citation chain. This matches the default expectation for non-circular empirical work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the empirical effectiveness of the curated data under a standard outcome-based RL setup; no free parameters, new entities, or non-standard axioms are introduced in the abstract.

axioms (1)
  • domain assumption Outcome-based GRPO constitutes a sufficient minimal RL algorithm for long-context improvement when paired with appropriate data.
    The paper adopts this setup without modification and attributes gains to data.

pith-pipeline@v0.9.1-grok · 5744 in / 1096 out tokens · 23063 ms · 2026-06-26T21:09:36.590586+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

    LoongRL: Reinforcement learning for ad- vanced reasoning over long contexts. InThe Four- teenth International Conference on Learning Repre- sentations. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McK- inney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet chal- lenging bench...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Ignore minor formatting differences (e.g., punctuation, case, extra whitespace)

  3. [3]

    Focus only on the final answer or conclusion

    The model’s prediction may contain reasoning steps. Focus only on the final answer or conclusion

  4. [4]

    If the prediction matches the standard answer in meaning, it is correct

  5. [5]

    Yes”), and the prediction is “Yes, because

    If the standard answer is short (e.g., “Yes”), and the prediction is “Yes, because...”, it is CORRECT. Output Format: If the prediction is correct, output exactly [[1]]. If the prediction is incorrect, output exactly [[0]]. Do not output any other text or explanation. Figure 4: LLM-as-judge prompt used during RL train- ing to score CrossEntity and LongMat...

  6. [6]

    You should focus on the final answer

    The Model Prediction may contain a reasoning process (e.g., within <think> tags). You should focus on the final answer

  7. [7]

    If the reasoning process is incomplete (e.g., cut off) and no final answer is provided, judge it as incorrect (0)

  8. [8]

    Output Format: <score>1 or 0</score> Figure 5: LLM-as-judge prompt used for evaluation

    If a final answer is provided and matches the standard answer (even if the reasoning is cut off or lengthy), judge it as correct (1). Output Format: <score>1 or 0</score> Figure 5: LLM-as-judge prompt used for evaluation. set sampled from the full test set due to the time cost of multi-round agentic interactions. Leakage filtering.It is a known issue that...

    2025

  9. [9]

    Do NOT re-answer the question

    The gold answer is definitely correct. Do NOT re-answer the question

  10. [10]

    For numerical answers, values that are numerically equal or very close are considered correct (e.g., 0.16 vs 0.16000, 34.8% vs 34.81%, $1.2M vs 1,200,000, -3.22% vs -3.2%)

  11. [11]

    Different but equivalent representations are acceptable (e.g., 94 vs 94.0, 14% vs 0.14)

  12. [12]

    For multi-part answers, all parts must match

  13. [13]

    Output ONLY your verdict: [[A]] if the predicted answer is CORRECT [[B]] if the predicted answer is INCORRECT Figure 6: LLM-as-judge prompt used for AA-LCR and DocFinQA evaluation

    If no clear answer can be extracted from the prediction, grade as INCORRECT. Output ONLY your verdict: [[A]] if the predicted answer is CORRECT [[B]] if the predicted answer is INCORRECT Figure 6: LLM-as-judge prompt used for AA-LCR and DocFinQA evaluation. Model L1 (n=42) L2 (n=66) L3 (n=19) AgentCPM-Explore 78.6 65.2 42.1 + Ours (25 steps) 81.0 63.652.6...

  14. [14]

    does not specify an explicit license in its repository, though its underlying sources are all under permissive licenses. Dataset Usage License HotpotQA LongDocQA CC-BY-SA-4.0 MuSiQue LongDocQA CC-BY-4.0 Qasper LongDocQA CC-BY-4.0 DeepMath-103K LongMath MIT MATH-Hard LongMath MIT DocQA-RL-1.6K Baseline Apache-2.0 LoongRL KeyChain Baseline Not specified Tab...