{"id":"479ea36f-e96f-43cf-a52d-28b9daf2f729","arxiv_id":"2508.09494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Training LLMs on self-generated, diverse 'active reading' materials improves factual recall by 160-312% and scales to a 1T-token Wikipedia expert model.","lead":"The authors propose Active Reading, a two-stage pipeline in which a language model invents learning strategies for a document and then applies them to create diverse synthetic training text. On factual QA benchmarks, 8B models trained this way gain 160-312% relative accuracy over vanilla fine-tuning, and a Wikipedia-expert model trained on 1 trillion generated tokens approaches the factual recall of much larger models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No baseline isolates self-generation: Active Reading is compared only to single fixed augmentations, so the central claim that self-generated strategies (rather than mere strategy diversity) drive gains is undersupported.","rationale":"The reader's conditional verdict is appropriate, and the leakage concern is real but not the most fundamental gap. The paper's headline contribution is 'self-generated learning strategies.' The controlled comparison in Table 1 rules out token budget, base model, and document set as explanations, and task-agnostic AR outperforming synthetic QA argues against pure benchmark-format leakage. However, the comparison does not rule out the possibility that any diverse fixed set of strategies would produce the same gains. The paper itself emphasizes diversity as the mechanism (§5.1), and the self-generation step is not isolated. This is a missing control, not an internal contradiction, and it does not invalidate the empirical finding that AR works. It does mean the specific claim about self-generation is not yet proven. The proposed fixed-strategy baseline is a direct, feasible check. If it matches AR, the contribution shifts from 'self-generated strategies' to 'diverse strategies,' which is still useful but less novel; if AR wins, the strongest claim is supported. The reader's verdict of CONDITIONAL remains appropriate, so no verdict change is needed.","tokens_in":15187,"tokens_out":7383,"duration_ms":84367,"concrete_test":"Construct a 'fixed-strategy Active Reading' baseline: use the same source documents and the same generator model, but replace the strategy-generation stage with a fixed bank of N human-authored study strategies (timeline, song/rhyme, association, active recall, analogical reasoning, etc.) sampled per chunk, controlling the number of strategies and the self-BLEU diversity to match AR-generated data. Generate the same ~4B words, train Llama-3.1-8B with identical hyperparameters to Table 1, and evaluate on SimpleWikiQA and FinanceBench. If the fixed-strategy baseline matches AR, self-generation is not necessary; if AR clearly exceeds it, the self-generation component is validated. Run at least 3 seeds with the same data-generation seed to obtain variance estimates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that 'self-generated learning strategies' — not the documents and not simple augmentation — drive the gains. Table 1 supports AR against paraphrasing and synthetic QA, but both are single fixed strategies. The paper never compares AR to a fixed, diverse bank of human-authored study strategies applied to the same documents. In §5.1 the authors attribute AR's advantage to diversity (Self-BLEU), not to self-generation per se. That is consistent with a weaker alternative: any sufficiently diverse set of document transformations would perform as well, and the model's ability to propose its own strategies is not the load-bearing ingredient. Table 3 shows self-generated data from the 8B model beats 70B-generated data, which is suggestive, but still lacks a fixed-strategy control. The reader's leakage concern is related but partial: task-agnostic AR also beats synth QA, so test-format leakage cannot fully explain the results; however, the missing fixed-strategy baseline affects both task-agnostic and task-specific variants. Without this control, the distinctive 'self-generated' aspect of the method is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Active Reading, a two-stage synthetic-data pipeline in which a language model first proposes study strategies for a given document and then applies those strategies to generate diversified training text. The authors evaluate the method in two regimes: (i) expert-domain fine-tuning of Llama-3.1-8B on Wikipedia documents relevant to SimpleWikiQA and on FinanceBench source documents, comparing against repetition, paraphrasing, and synthetic QA under a fixed ~4B-token budget; and (ii) continued pre-training at Wikipedia scale, where they train and release Meta WikiExpert-8B on 1T generated tokens and report gains on SimpleQA, NaturalQuestions, and TriviaQA. The paper's central claim is that self-generated learning strategies — not the specific documents and not simple augmentation — drive the observed gains, and that Active Reading scales better than previous fixed augmentation strategies.","tokens_in":15487,"tokens_out":4423,"duration_ms":46565,"significance":"If the result holds, Active Reading is a practically valuable method for fact-level knowledge injection. The paper's strengths are substantial: the main comparison is internally controlled (same base model, token budget, grading protocol, and 10% DCLM mix), the authors release both the WikiExpert model and the 1T-token dataset, and the coverage/diversity analysis in §5.1 is a good-faith attempt to explain the mechanism. The scaling demonstration to 1T tokens is also noteworthy. However, the central attribution to self-generation is not yet isolated: all baselines are single fixed transformations, so the evidence is equally consistent with a weaker alternative in which any sufficiently diverse set of fixed strategies would perform as well. In addition, the scaling recipe is tuned on a benchmark that is a subset of the final evaluation set. These are fixable but load-bearing gaps.","major_comments":[{"comment":"The central claim that 'self-generated' strategies, rather than strategy diversity per se, drive the gains is not isolated by the current experiments. The baselines (repeat, paraphrase, synth QA) are each single fixed transformations; the only combined baseline 'paraphrase + synth QA + AR' contains Active Reading itself. §5.1's diversity analysis (Figure 6) attributes AR's advantage to lower Self-BLEU, which is consistent with the alternative that any sufficiently diverse set of fixed, human-authored study strategies applied to the same documents would perform as well. The paper needs a control that applies a fixed bank of diverse strategies (e.g., the strategies illustrated in Figure 1 or sampled from the generation outputs) to the same source documents under the same 4B-token budget, with the strategies cycled in a fixed order. Without this control, the abstract-level attribution to se","section":"§4.1, Table 1; §5.1"},{"comment":"The scaling recipe is selected against the target benchmark. The learning rate change (1e-5 to 3e-4), the pretraining-data mixing ratio (10% DCLM to the 1:1 mix), and the relative weight of SimpleWikiQA data are all tuned using SimpleWikiQA accuracy in Figures 3 and 4. The final WikiExpert model is then evaluated on SimpleQA, the benchmark from which SimpleWikiQA was subsampled. The headline SimpleQA gain in Table 2 is therefore not out-of-sample with respect to hyperparameter selection. The paper should either hold out a portion of SimpleWikiQA for final evaluation, choose hyperparameters on an unrelated fact benchmark and transfer, or explicitly report a sensitivity analysis showing that the conclusions are robust to reasonable variations in the selected hyperparameters. This is necessary to support the general scaling claim beyond the specific configuration chosen for the released mod","section":"§4.2.1, Figures 3–4; §4.2.2, Table 2"},{"comment":"No seed variation or error bars are reported for any of the main results. The difference between task-agnostic and task-specific AR is 2.9 points (63.33 vs 66.25), and the difference between task-specific AR and the 'paraphrase + synth QA + AR' combination is 2.4 points (66.25 vs 66.66). Given that every run uses a single training run and a single generation seed, it is unclear which of these small differences are meaningful. Adding at least 2–3 seeds for the central comparisons in Table 1 and for the scaling curves in Figure 2 would substantially increase confidence in the method-level conclusions.","section":"§4.1, Table 1; §5.2"}],"minor_comments":[{"comment":"Figure 3's y-axis is labeled 'F1 score on SimpleQA' while the text and surrounding figures refer to SimpleWikiQA. If the subset was used, the label should be corrected.","section":"Figure 3"},{"comment":"The column header in Table 3 says 'SimpleQA' but the values (66.25, 62.26, etc.) match SimpleWikiQA accuracy from Table 1. Please clarify.","section":"Table 3"},{"comment":"Only the 'trivia competition' task-specific prompt is shown. The paper mentions a finance-specific task prompt for FinanceBench, but no such prompt appears in the appendix. Please include it or state explicitly if the same prompt was used for both domains.","section":"Appendix D.3.2"},{"comment":"There are two separate reference entries with the same title 'Scaling laws for fact memorization of large language models' (Lu et al. 2024a and Lu et al. 2024b). These appear to be duplicates; please merge or disambiguate.","section":"References"},{"comment":"The guardrail table shows non-trivial degradation on MMLU (-0.9), GSM8K (-5.8), and MBPP (-8.6) relative to Llama-3.1-8B. This trade-off is not discussed in the main text. A sentence acknowledging this degradation and its implications for the 'more factual base model' claim would be helpful.","section":"Appendix C"},{"comment":"The abstract reports '+313% relative over vanilla finetuning'; Table 1's repeat baseline gives 15.92 → 66.25, which is ~316% relative. The text in §1 says '312% relative.' Minor rounding is fine, but the values should be consistent.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The missing fixed-diverse-strategy control is the key risk. If the authors add that experiment and either re-run the scaling section with a held-out target or report sensitivity to the selected hyperparameters, the paper would be much stronger. I do not see a reason to reject: the experimental design is otherwise careful and the released assets are valuable. The fit to cs.CL is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth your time. Active Reading is a simple, sensible idea: instead of paraphrasing a document or generating QA pairs, you first ask the model to propose learning strategies for that document, then apply them. It's not a huge conceptual leap, but it's a genuine one, and the execution is solid. The comparisons in Table 1 are controlled—same 8B base, same 4B token budget, same grader—and the gains over vanilla finetuning, paraphrasing, and synthetic QA are large and consistent. Scaling to 1T tokens and releasing the model and data is exactly the kind of reproducible evidence that makes a paper credible. The analysis in Section 5 is honest: they check coverage and show it doesn't explain the results, and they show diversity (Self-BLEU) tracks performance. That's good science.\n\nThe main soft spot is that the central framing—that self-generated strategies are what matter—is not actually tested. The baselines are paraphrasing and synthetic QA, both single fixed strategies. There is no control with a fixed, diverse bank of human-authored study strategies applied to the same documents. Given that the authors themselves attribute the advantage to diversity, the more parsimonious story is 'diverse transformations help,' not 'the model's own ideas are special.' Table 3 (8B-generated data beats 70B-generated data) is suggestive, but it doesn't close that gap. This is a moderate concern, not fatal—the method works, and the framework is useful regardless—but the abstract and intro overstate what is established.\n\nThe task-specific prompt in D.3.2 is also worth flagging: it asks the model to 'study for a trivia competition' and cover every piece of information, which is close to the SimpleQA distribution. So part of the task-specific gain may be benchmark-style leakage. But task-agnostic Active Reading also beats synthetic QA, so leakage isn't the whole story.\n\nTwo more things. The abstract says WikiExpert 'outcompetes models with hundreds of billions of parameters on factual QA.' In Table 2 that's true only on SimpleQA; on NQ and TQA it trails the large models. That overreach should be fixed. Minor issues: no error bars or seeds, the SimpleWikiQA subset is author-constructed, and the scaling recipe is tuned on that subset. None of these are deal-breakers, but they cap the confidence I'd put in the numbers.\n\nWho this is for: anyone working on knowledge injection, synthetic data, or domain adaptation. The released 1T-token dataset alone is a resource. I'd engage with it as a peer reviewer, but I'd ask for a fixed-strategy control and a softened abstract before accepting. Worth bringing to reading group.","headline":"A real empirical result with a conceptually overreaching claim; worth engaging, but the distinctive 'self-generation' ingredient is not actually isolated.","tokens_in":15991,"tokens_out":4929,"would_cite":true,"duration_ms":43343,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM can be trained to 'study' a closed corpus by generating its own diverse learning strategies—timelines, rhymes, analogies, recall drills—and that this Active Reading pipeline lifts factual recall far beyond vani","keywords":["Active Reading","factual recall","synthetic data generation","knowledge injection","continued pretraining","tail facts","SimpleQA","learning strategies"],"falsifier":"Run Active Reading with the task-specific trivia prompt on a closed corpus, evaluate on a held-out benchmark whose fact distribution the prompt never sees (e.g., legal or medical QA), and compare against task-agnostic Active Reading and paraphrasing; if the task-specific advantage disappears, the result is prompt-to-benchmark leakage rather than general learning. Alternatively, remove or mask training examples that n-gram-overlap SimpleQA test questions and check whether the 66% score collapses.","tokens_in":15092,"feed_emoji":"🧠","tokens_out":6610,"duration_ms":63086,"temperature":0.7,"pith_summary":"The paper is trying to establish that how a model studies a text matters more than how many times it sees it. Its method, Active Reading, asks the model to propose a set of learning strategies for each source document and then applies those strategies to generate many diverse synthetic documents; training on those documents is claimed to absorb far more knowledge than repeated reading, paraphrasing, or synthetic question-answer generation. On a Wikipedia-grounded subset of SimpleQA, an 8B model trained this way reaches 66% accuracy versus 16% for vanilla finetuning; on FinanceBench it reaches 26% versus 10%. The same recipe, scaled to 1 trillion tokens of synthetic Wikipedia, produces an 8B model that answers more SimpleQA facts than models hundreds of billions of parameters larger. If true, the paper gives practitioners a scalable, domain-agnostic way to make models reliably learn a specified body of knowledge without retrieval.","feed_headline":"Active Reading lifts an 8B model's factual recall from 16% to 66%","feed_subtitle":"Self-generated study strategies beat paraphrase and synthetic QA—then scale to a 1-trillion-token Wikipedia expert.","key_machinery":"The central mechanism is the two-stage Active Reading loop: a strategy-generation prompt (task-agnostic or task-specific) and a strategy-application prompt. The model first writes several markdown-prefixed study strategies tailored to the particular document, then uses each strategy as an instruction to generate a fresh synthetic document. The work this machinery does is to force many distinct re-encodings of the same facts, which the paper measures as higher diversity (lower self-BLEU) than paraphrase or synthetic QA; the paper argues that this diversity, not answer coverage, is what lets the model internalize facts in a form that transfers to unseen questions.","core_discovery":"Active Reading is a two-stage synthetic-data generation framework. In the first stage, the model reads a source document and proposes context-specific study strategies—for example, creating a timeline of award recipients, composing a rhyme of names, or linking facts to familiar associations. In the second stage, each strategy is applied to the document to generate a distinct long-form 'study' text. Training on this mixed corpus yields substantially better fact recall than training on the raw documents, on paraphrases, or on synthetic question-answer pairs: 66% versus 16% on SimpleWikiQA and 26% versus 10% on FinanceBench's overall set. The authors argue the advantage comes from data diversit","pith_inferences":["Editorial inference: the task-specific prompt may be generating training examples that resemble the SimpleQA benchmark distribution; until the method is evaluated on corpora and query styles unrelated to the prompt (e.g., legal or medical QA), the diversity explanation is not fully separated from benchmark leakage.","Editorial inference: the result that an 8B model learns better from its own generated data than from 70B-generated data suggests the synthetic curriculum should be matched to the learner's current abilities; a direct test would vary generation model, temperature, and strategy count while holding data volume fixed.","Editorial inference: the observed recovery of target-task performance when pretraining data is mixed in hints that diverse background data acts as a plasticity regularizer; varying the pretraining mix with AR data fixed would make the mechanism testable.","Editorial inference: if diversity is the active ingredient, the framework should transfer to structured knowledge such as tables or knowledge graphs, where strategy count and strategy orthogonality should predict recall better than raw token count."],"forward_implications":["Training on Active Reading data improves factual recall on a closed corpus by 160–312% relative over vanilla finetuning, matching an oracle retrieval baseline on SimpleWikiQA.","Active Reading scales better than paraphrase and synthetic QA: gains continue as synthetic data grows to 4B words, while the baselines plateau.","At pre-training scale, 1T tokens of Active Reading-augmented Wikipedia produce an 8B base model that beats much larger models on SimpleQA and improves NaturalQuestions and TriviaQA.","The method subsumes prior augmentation strategies—paraphrase, synthetic QA, concept maps—so practitioners can replace them with one pipeline that generates all of them.","A task-specific strategy prompt (e.g., 'study for a trivia competition') yields the best tail-fact recall, suggesting the strategy can be steered toward a target task."],"supporting_citations":[{"why":"Documents that LLMs struggle to learn long-tail knowledge, motivating the target task.","marker":"(Kandpal et al., 2023)"},{"why":"Provides SimpleQA benchmark and the observed low factual recall that Active Reading improves.","marker":"(Wei et al., 2024)"},{"why":"Supplies the paraphrasing baseline and theory that repetition without variation overfits.","marker":"(Allen-Zhu and Li, 2024)"},{"why":"Supplies the synthetic question-answer baseline (PAQ-style generation).","marker":"(Lewis et al., 2021)"},{"why":"Provides FinanceBench benchmark for the expert-domain evaluation.","marker":"(Islam et al., 2023)"},{"why":"The human learning-science strategies that inspire Active Reading.","marker":"(Brown et al., 2014)"},{"why":"Prior synthetic continued pretraining (EntiGraph) that Active Reading encompasses and outperforms.","marker":"(Yang et al., 2024b)"},{"why":"Provides DCLM pre-training corpus mixed into the training runs.","marker":"(Li et al., 2024)"}],"fun_headline_variants":["Active Reading: 8B model recall jumps from 16% to 66%","Self-study strategies quadruple LLM fact recall","8B Wikipedia expert beats 100B+ on factual QA","Active Reading: 313% gain in factual accuracy at 8B"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central claim assumes the benefit comes from the diversity of self-generated study strategies, not from the task-specific prompt producing training questions that look like the evaluation benchmark; if benchmark-style leakage drives the gains, generality to unseen fact distributions is not established.","fun_headline_variants_meta":{"raw":{"variants":["Active Reading: 8B model recall jumps from 16% to 66%","Self-study strategies quadruple LLM fact recall","8B Wikipedia expert beats 100B+ on factual QA","Active Reading: 313% gain in factual accuracy at 8B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2837,"prompt_tokens":751,"completion_tokens":2086,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2023}},"tokens_in":495,"tokens_out":2086,"duration_ms":16631,"temperature":1.0,"reasoning_tokens":2023,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:00:37.518420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Active Reading with the task-specific trivia prompt on a closed corpus, evaluate on a held-out benchmark whose fact distribution the prompt never sees (e.g., legal or medical QA), and compare against task-agnostic Active Reading and paraphrasing; if the task-specific advantage disappears, the result is prompt-to-benchmark leakage rather than general learning. Alternatively, remove or mask training examples that n-gram-overlap SimpleQA test questions and check whether the 66% score collapses.","supporting_citations":[],"review_version":1}