arxiv: 2604.06829 · v2 · submitted 2026-04-08 · 💻 cs.CL · cs.AI

Recognition: no theorem link

WRAP++: Web discoveRy Amplified Pretraining

Jiang Zhou , Yunhao Wang , Xing Wu , Tinghao Yu , Feng Zhang

Authors on Pith no claims yet

Pith reviewed 2026-05-10 17:18 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords synthetic datapretrainingcross-document reasoningweb hyperlinksjoint QA synthesisLLM knowledgeWikipediascaling laws

0 comments

The pith

WRAP++ uses web hyperlinks to discover document pairs and synthesize joint QA pairs, expanding pretraining data from 8.4B to 80B tokens while improving cross-document knowledge in LLMs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current synthetic data methods for LLM pretraining rewrite individual web documents in isolation, which restricts them to facts contained within single pages and ignores connections between pages. WRAP++ instead scans hyperlinks to identify reliable relational patterns such as dual-links and co-mentions between documents, then generates QA pairs whose answers require combining information from both sources. This produces new relational facts that cannot be found in either document alone and multiplies the volume of usable training data through the combinatorial number of entity pairs. When applied to Wikipedia, the method converts roughly 8.4 billion raw tokens into 80 billion tokens of cross-document QA. OLMo models trained with this data at 7B and 32B scales outperform those trained on single-document rewrites on the SimpleQA benchmark and continue to improve as scale increases.

Core claim

WRAP++ amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, it identifies high-confidence relational motifs including dual-links and co-mentions, then creates QA examples that require reasoning across both documents. This yields relational knowledge absent from either source document alone and creates diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, the process also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia turns 8.4B tokens of raw text into 80B of Q

What carries the argument

Discovery of high-confidence relational motifs (dual-links and co-mentions) via hyperlinks to synthesize joint QA pairs requiring cross-document reasoning.

If this is right

Produces 80B tokens of cross-document QA data from 8.4B raw Wikipedia tokens.
Generates relational knowledge absent from any single source document.
Delivers higher SimpleQA scores than single-document training at both 7B and 32B scales.
Maintains performance gains as model and data scale increase.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same hyperlink-motif discovery could be applied to broader web crawls beyond Wikipedia to increase data diversity.
Cross-document QA synthesis may improve results on downstream tasks that require multi-hop reasoning.
Combining WRAP++ with other synthetic data pipelines could produce even larger and more varied pretraining sets.

Load-bearing premise

The synthesized joint QA pairs genuinely require and teach cross-document reasoning rather than being answerable from one document or containing low-quality or noisy associations introduced during synthesis.

What would settle it

Running the same OLMo 7B and 32B training runs with single-document data versus WRAP++ data and measuring no gain in SimpleQA accuracy or no continuation of scaling improvements.

Figures

Figures reproduced from arXiv: 2604.06829 by Feng Zhang, Jiang Zhou, Tinghao Yu, Xing Wu, Yunhao Wang.

**Figure 1.** Figure 1: Overview of the WRAP++ pipeline. Unlike single-document WRAP, which rewrites individual documents, WRAP++ discovers cross-document entity relationships from web topology and amplifies them into pretraining data through joint QA synthesis. for Oppenheimer”), WRAP++ pairs these connected documents to synthesize multi-hop relational QA. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: SimpleQA pass@128 vs. training tokens. Single-document recipes (WRAP and Extended WRAP) reach a data bottleneck early, limiting further knowledge acquisition. In contrast, the combinatorial nature of WRAP++ allows it to scale effectively up to 80B tokens, improving performance without obviously plateauing. 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 k (log scale) 0.1 0.2 0.3 0.4 0.5 Pass@k (unbiased) (a) OLMo3-7B 2 0… view at source ↗

**Figure 3.** Figure 3: Evolution of pass@k performance during training. The curves illustrate the unbiased pass@k of OLMo-3-7B (a) and OLMo-3-32B (b) on SimpleQA (k ∈ [1, 128] in log scale). The color gradient (from light to dark blue) tracks the accumulation of consumed WRAP++ tokens (from 10B to 80B). The strictly monotonic upward shift across all values of k indicates robust, unsaturated knowledge internalization. 3.4 SCALING… view at source ↗

**Figure 4.** Figure 4: Per-benchmark score trajectories during mid-training with the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

**Figure 5.** Figure 5: Question vs. answer length distributions at character level (left) and word level (right). [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

**Figure 6.** Figure 6: Synthesized QA length distribution by relation type. Both subsets exhibit a similar uni [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗

**Figure 7.** Figure 7: Violin plot comparison of question length, answer length, and answer word count between [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

**Figure 8.** Figure 8: Source document length distributions for the two input passages. Passage A (the referenc [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

**Figure 9.** Figure 9: Bucketed length distributions computed over the full dataset of 240.7M instances. Left: [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

read the original abstract

Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

WRAP++ scales pretraining data by mining hyperlinks for cross-document QA pairs, but the results do not verify that the pairs actually teach relational reasoning instead of just adding volume.

read the letter

Hi, the main thing here is that WRAP++ finds document pairs on Wikipedia through dual-links and co-mentions, then generates joint QA meant to require both pages. They expand 8.4B raw tokens into 80B of this synthetic data and report that OLMo models at 7B and 32B scales beat single-document baselines on SimpleQA while showing continued scaling gains. The combinatorial pairing is a clean way to grow data without new crawling, and it extends single-page rephrasing in a direct way that makes sense for relational facts. What stands out is the reported scale and the sustained improvement across model sizes. The soft spot is exactly the one the stress test flags: no controls or checks are described to confirm the QA pairs cannot be answered from one document alone or that the synthesis avoids noise. The abstract gives no details on how the questions are created, no quality metrics, and no ablations that separate the discovery step from simple data increase. If a decent fraction of the pairs leak the answer into a single page, the performance edge could be explained by volume or phrasing quality rather than cross-document learning. This is aimed at groups building pretraining pipelines who want ideas for connected knowledge. A reader focused on data synthesis would get usable ideas from the motif approach, though they would need to add their own validation. It deserves peer review because the scaling claims are concrete and the method differs from prior work, so referees can examine the full pipeline and test the central assumption.

Referee Report

3 major / 2 minor

Summary. The paper proposes WRAP++, a method for amplifying pretraining data by discovering cross-document relationships via web hyperlinks (dual-links and co-mentions) on sources such as Wikipedia and synthesizing joint QA pairs intended to require reasoning across document pairs. This expands ~8.4B raw tokens into 80B tokens of cross-document QA data. Experiments with OLMo models at 7B and 32B scales show that training with WRAP++ data yields substantial outperformance over single-document synthetic baselines on SimpleQA, along with sustained scaling gains.

Significance. If the central empirical claims hold after verification, this work would demonstrate that hyperlink-driven discovery of relational motifs can generate scalable, combinatorially amplified training data that teaches associative knowledge beyond intra-document facts, leading to measurable gains in knowledge-intensive benchmarks and better scaling behavior. The approach offers a concrete mechanism for moving pretraining data synthesis from isolated documents to relational structures, which could influence future data pipelines if the cross-document property is confirmed.

major comments (3)

[Abstract] Abstract: The headline result attributes performance gains on SimpleQA to 'relational knowledge absent from either source document alone,' yet the manuscript provides no controls, human validation, automatic checks, or ablations confirming that synthesized QA pairs cannot be answered from a single document or that associations are not noisy. This verification is load-bearing for the mechanistic claim distinguishing WRAP++ from single-document rephrasing.
[Experiments] Experiments section (results on 7B/32B OLMo models): The reported outperformance and scaling behavior lack ablations isolating the discovery step (e.g., dual-links vs. co-mentions), data-volume controls, or comparisons to equivalent-volume single-document data; without these, gains could arise from rephrasing quality or scale rather than cross-document reasoning.
[Method] Method description of QA synthesis: No details are given on the generation process for joint QA (prompts, models used, filtering criteria for high-confidence motifs), data quality controls, or how hyperlink-derived pairs are turned into questions requiring both documents, preventing assessment of whether the synthesized data genuinely teaches the claimed relational knowledge.

minor comments (2)

[Abstract] Abstract: The expansion of the acronym WRAP++ contains an unusual internal capitalization ('discoveRy'); clarify whether this is intentional or a typographical artifact.
[Overall] Overall: The manuscript would benefit from a table summarizing the data amplification ratios, motif types, and exact evaluation setup (e.g., number of QA pairs, training tokens, baseline details) to improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. The comments correctly identify areas where additional verification, ablations, and methodological transparency would strengthen the manuscript. We address each major comment below and will revise the paper accordingly to incorporate the requested elements.

read point-by-point responses

Referee: [Abstract] Abstract: The headline result attributes performance gains on SimpleQA to 'relational knowledge absent from either source document alone,' yet the manuscript provides no controls, human validation, automatic checks, or ablations confirming that synthesized QA pairs cannot be answered from a single document or that associations are not noisy. This verification is load-bearing for the mechanistic claim distinguishing WRAP++ from single-document rephrasing.

Authors: We agree that explicit verification is essential to support the claim of relational knowledge absent from single documents. The current manuscript relies on the design of hyperlink motifs (dual-links and co-mentions) to ensure cross-document dependencies but does not include direct controls or validations. In the revision we will add: (1) an automatic evaluation where QA performance is measured using only one document as context, demonstrating clear degradation; (2) human validation on a random sample of 200 QA pairs confirming that both documents are required; and (3) an analysis of motif noise levels. These additions will directly address the load-bearing mechanistic distinction. revision: yes
Referee: [Experiments] Experiments section (results on 7B/32B OLMo models): The reported outperformance and scaling behavior lack ablations isolating the discovery step (e.g., dual-links vs. co-mentions), data-volume controls, or comparisons to equivalent-volume single-document data; without these, gains could arise from rephrasing quality or scale rather than cross-document reasoning.

Authors: We acknowledge that the existing experiments compare WRAP++ against single-document baselines but do not isolate the discovery components or fully control for volume. We will expand the Experiments section with: ablations separating dual-link from co-mention motifs; a volume-matched baseline in which single-document synthetic data is scaled to the full 80B tokens; and additional scaling curves at matched data volumes. These controls will help isolate whether gains derive from the cross-document relational structure rather than rephrasing quality or scale alone. revision: yes
Referee: [Method] Method description of QA synthesis: No details are given on the generation process for joint QA (prompts, models used, filtering criteria for high-confidence motifs), data quality controls, or how hyperlink-derived pairs are turned into questions requiring both documents, preventing assessment of whether the synthesized data genuinely teaches the claimed relational knowledge.

Authors: We will substantially expand the Method section to include the full QA synthesis pipeline. This will cover: the exact prompts used to generate joint questions (explicitly instructing the model to create queries that reference entities from both documents), the LLM and hyperparameters employed for synthesis, the filtering criteria based on motif confidence scores and post-generation answerability checks, and data quality controls such as perplexity filtering and deduplication. We will also detail the prompting strategy that enforces cross-document dependency by design. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical data pipeline evaluated externally

full rationale

The paper describes an empirical synthesis pipeline that discovers hyperlink-based document pairs and generates joint QA data, then evaluates the resulting models on the external SimpleQA benchmark. No equations, fitted parameters, or predictions are present. No self-citations are invoked as load-bearing justifications for uniqueness or ansatzes. The central performance claims rest on observable scaling behavior rather than any derivation that reduces to its own inputs by construction. This is the expected non-circular outcome for a data-generation method paper.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based solely on the abstract, the central claim rests on the domain assumption that hyperlink-derived pairs produce useful relational knowledge; no free parameters or invented entities are mentioned.

axioms (1)

domain assumption High-confidence relational motifs from web hyperlinks can be used to synthesize QA pairs that require genuine cross-document reasoning.
This premise underpins both the data amplification and the claimed performance gains.

pith-pipeline@v0.9.0 · 5537 in / 1282 out tokens · 42968 ms · 2026-05-10T17:18:52.460514+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 4 canonical work pages · 1 internal anchor

[1]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.123. URLhttps://aclanthology.org/2025.acl-long.123/. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URLhttps://arxiv.org/ abs/1811.00937. Team OLMo, Pet...

work page doi:10.18653/v1/ 2025
[2]

URLhttps://arxiv.org/abs/2411.04368. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang L...

work page internal anchor Pith review arXiv 2025
[3]

Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering

Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.493. URL https://aclanthology.org/2022.acl-long.493/. 11 A LIMITATION Our experiments instantiateWRAP++on Wikipedia, which is a clean and entity-centric corpus. Extending to noisier web corpora (e.g., Common Crawl), where hyperlinks include advertisements, navigation elements, and l...

work page doi:10.18653/v1/2022.acl-long.493 2022
[4]

Generate high-quality synthetic QA pairs that REQUIRE information from BOTH Passage A and Passage B to answer
[5]

This reasoning must explicitly bridge facts from both passages

The Answer MUST begin with a step-by-step reasoning process. This reasoning must explicitly bridge facts from both passages
[6]

Do not use external knowledge
[7]

Question:

CRITICAL CONSTRAINT: The generated QA pair will be used to train a model WITHOUT these passages provided as context. Therefore, you MUST act as an omniscient AI stating absolute 13 facts from your own inherent knowledge. - DO NOT use any attribution phrases like ‘According to Passage A’, ‘Passage B mentions’, ‘As stated in the text’, or ‘Based on the prov...

work page arXiv 2018