{"id":"62691a5e-ac3b-4380-adb7-dc5c986b0823","arxiv_id":"2608.11788","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TELLME improves continual domain pre-training by training on plain text plus open-ended QA pairs while masking question tokens from the loss, with consistent finance gains and small medical gains.","lead":"This paper introduces TELLME, a continual pre-training method that mixes domain text with GPT-generated descriptive question-answer pairs and excludes the question tokens from the loss. The authors report financial benchmark gains up to 23.6% over CPT+IT and a 9.8% long-term retention improvement, but the experiments lack error bars and some baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are at risk from benchmark contamination: TELLME's GPT-4o-generated QA is never checked for overlap with FOMC, NIFTY, or MMLU subsets, so the headline improvements may reflect memorization rather than test-enhanced learning.","rationale":"The reader's weakest assumption is that GPT-4o-mini-generated QA pairs do not leak into the evaluation benchmarks, and that the 80-sample filter in Appendix E.2 is insufficient. My independent read of the paper reaches the same point: the dataset curation in Section 3.2 explicitly aims for context-independence but provides no quantitative verification against the benchmark sets, and the evaluation section reports no overlap statistics. Because the headline result is an accuracy gain on those benchmarks, a positive contamination finding would invalidate the main empirical claim. I considered an alternative concern about the retention experiment, namely that TEL(F)->CPT(M) is compared only against CPT(F)->CPT(M) without a QA-based control, so the 9.8% retention figure is confounded by different initial finance performance and training distributions. That concern is real but narrower: it affects only the retention sub-claim. The contamination concern threatens both the 23.6% acquisition gain and the benchmark component of the retention gain, so it is the single most load-bearing issue. The reader already issued CONDITIONAL on essentially this basis; my stress-test does not shift the verdict, but it sharpens the required check: an explicit n-gram overlap audit and a training run with all flagged samples removed. If that test comes back clean, the central claim would be substantially strengthened.","tokens_in":24158,"tokens_out":8652,"duration_ms":96057,"concrete_test":"Run a contamination audit before interpreting any result. Tokenize all 100k TELLME QA pairs and all evaluation items in FOMC, NIFTY, MMLU-F, HeadQA, MedMCQA, and MMLU-C; compute 8-gram containment between each benchmark question or correct-answer option and each training QA, with and without stopword removal. Also compare the source Bloomberg and PubMed excerpts against the benchmark documents to assess article-level overlap. Then remove every flagged training sample and retrain the SmolLM2-1.7B and Llama-3.1-8B comparisons from Table 2. If the TELLME advantage over +CPT+IT and INSTPT disappears or falls below the run-to-run noise level after removing flagged samples, the reported gains are contaminated and the central claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 instructs GPT-4o-mini to generate questions answerable independently of the excerpt, and Appendix E.2 filters out only 80 samples containing context-revealing phrases such as 'in this context' or 'described'. The pipeline never checks whether generated QA pairs overlap with evaluation benchmarks. This matters for two reasons. First, GPT-4o-mini was itself trained on web-scale data that almost certainly includes MMLU and similar public benchmarks, so its generated 'general domain knowledge' questions can be near-duplicates of benchmark items. Second, the 100k Bloomberg source articles used to generate the finance QA may overlap temporally or textually with the NIFTY headlines and FOMC documents used for evaluation. The +CPT, +CPT+IT, and INSTPT baselines all see the same source plain text, so an article-level overlap affects them too; however, only TELLME adds 100k GPT-generated QA pairs derived from those sources. That added QA is exactly the channel through which benchmark-like content would enter the training set. If even a small fraction of FOMC, NIFTY, or MMLU-F items appear in transformed form among the 100k QA pairs, the headline 23.6% finance gain and the benchmark portion of the retention experiment are explained by memorization rather than by test-enhanced learning. The absence of error bars and the near-zero medical gains further weaken the case that the effect is a general learning improvement rather than a benchmark-specific artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes TELLME (Test-Enhanced Learning for Language Model Enrichment), a continual pre-training method that augments domain plain text with M=3 descriptive question-answer pairs generated by GPT-4o-mini, while masking the question tokens from the loss. The method is evaluated on finance and medicine using Llama-3.2-1B/3B, Llama-3.1-8B, and SmolLM2-1.7B, with comparisons against CPT, CPT+IT, and INSTPT baselines. The main empirical claims are up to 23.6% improvement over existing methods in finance and a 9.8% long-term retention gain over standard CPT. Additional sections report ablations on loss masking, QA placement, data ratio, synthesizer choice, a LoRA-based 70B experiment, and Korean cross-lingual generalization.","tokens_in":24488,"tokens_out":4982,"duration_ms":49298,"significance":"The paper's core idea is clearly motivated: applying the psychological testing effect to continual pre-training is a natural and simple extension of QA-augmented CPT, and the authors provide useful ablations distinguishing the effect of loss masking, QA position, and data ratio. The data-generation pipeline is cheap ($12 for 100k samples) and the dataset/model release supports reproducibility. The Coverage Ratio analysis in Appendix D.3 is a good check that TELLME QA is less extractive than INSTPT QA. The method introduces no fitted parameters in the evaluative claim, so the central comparison is not circular. However, the headline claims currently outrun the evidence: all results are single runs with no error bars or significance tests, the generated QA is never checked for overlap with the evaluation benchmarks, the medical gains are small and sometimes negative at 8B scale, and the retention experiment lacks the CPT+IT and INSTPT controls needed to isolate the testing effect. These are load-bearing issues for the paper's central empirical claims.","major_comments":[{"comment":"The pipeline filters only 80 context-dependent samples (and the reported fraction 0.008% is arithmetically wrong: 80/100k = 0.08%) and never checks whether the 100k generated QA pairs overlap with the evaluation benchmarks FOMC, NIFTY, MMLU-F, HeadQA, MedMCQA, or MMLU-C. Because the QA pairs are generated by GPT-4o-mini, which was trained on web-scale data that very likely includes public benchmarks, and because the Bloomberg/PubMed source articles may overlap temporally or textually with the evaluation documents, a portion of the reported gains could be explained by near-duplicate memorization rather than test-enhanced learning. Please add a quantitative overlap analysis (e.g., 8-13 gram containment, embedding similarity) for every benchmark against both the QA pairs and the source articles, and rerun the main comparisons after removing any overlapping items.","section":"Section 3.2, Appendix E.2"},{"comment":"All reported numbers are single runs with no error bars, standard deviations, or significance tests. Several headline differences are small: in Table 2 for Llama-3.1-8B, TELLME improves over +CPT by only +0.15 on HeadQA and -0.03 on MMLU-C, and the 8B medical average for TELLME (42.63) is below the base model's average (43.99). With single runs, these differences are indistinguishable from training noise. Please report means and standard deviations over at least three seeds, and where feasible add paired significance tests or bootstrap confidence intervals for each benchmark.","section":"Section 4.1, Tables 2 and 4"},{"comment":"The long-term retention experiment compares only TEL(F)→CPT(M) against CPT(F)→CPT(M). Because the TEL model saw QA pairs during the first phase while the CPT model did not, the 9.8% retention gain could reflect the extra QA exposure rather than the specific TEL mechanism (masking questions, joint plain-text/QA samples). The appropriate controls are +CPT+IT(F)→CPT(M) and +INSTPT(F)→CPT(M), which include QA training in the first phase. Please add these conditions and also report the reverse direction (medical-to-finance) to show that the retention effect is not domain-specific.","section":"Section 4.3, Figure 3"},{"comment":"The claim that TELLME brings 'strong performance in both the finance and medical domains' is not supported by Table 2. In medicine the gains over +CPT are consistently small (e.g., at most +0.9 points on HeadQA for SmolLM2-1.7B) and negative for Llama-3.1-8B on MMLU-C; the 8B TELLME medical average is below the baseline model. The abstract's general statement that TELLME 'outperforms existing methods' is defensible for finance but not yet for medicine. Please either temper the domain-general claim or provide additional experiments showing consistent medical improvements across scales and seeds.","section":"Section 4.2, Section 6, Abstract"}],"minor_comments":[{"comment":"Table 3 appears to duplicate the Llama-3.2-3B block from Table 2; replace it with the intended results or remove it.","section":"Tables 2 and 3"},{"comment":"The text reports a 9.8% final performance gain and a 3.15-point increase, but the y-axis of Figure 3 is only labeled 'Average Score' with no definition of how the composite is computed or how the percentage is derived; specify the formula and the exact benchmark aggregation.","section":"Section 4.3, Figure 3"},{"comment":"The phrase 'two seperated forward-backward passes' should read 'two separated forward-backward passes.'","section":"Section 4.1"},{"comment":"The paper says TELLME is '1.4 times faster' than CPT, but the evidence is steps-to-same-PPL, not wall-clock time; rephrase as '1.4x fewer training steps' or report actual runtime.","section":"Section G.1, Figure 5"},{"comment":"Section 3.2 describes an LLM-as-a-judge quality evaluation, while the caption of Figure 7 calls the scores 'Human Evaluation Scores'; clarify which evaluation is actually reported and whether the 4.03 average comes from GPT-4-mini or human raters.","section":"Section G.3, Figure 7"},{"comment":"The Limitations section begins 'Model Size. Second, although we extended...' with a dangling 'Second'; the section also does not acknowledge the absence of repeated runs or the benchmark-overlap risk, which should be listed as limitations.","section":"Limitations section"},{"comment":"The SmolLM2-1.7B rows in Table 13 appear identical to the Llama-3.2-1B rows; check whether this is a copy-paste error and correct the corresponding numbers.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the paper's core idea and appreciate the clean ablations and cheap data-generation pipeline. However, the empirical case needs to be substantially strengthened before publication: the absence of repeated runs, the lack of any benchmark-overlap check for the generated QA data, and the weak medical results are all load-bearing. The title and abstract make stronger claims than the current evidence supports. There is no training code released, only the model and dataset, which limits reproducibility of the training procedure. The paper would be improved by a more conservative framing and by adding the contamination analysis and control conditions requested in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead TELLME. The new thing is a simple twist on QA-augmented CPT: generate open-ended, context-independent QA pairs with GPT-4o-mini, concatenate them with the source text in one sequence, and mask the question tokens from the loss. That combination is genuinely different from INSTPT and PIT, and the paper compares directly against both. The ablations are the strongest part: question-masking helps, ordering matters, and the method still beats INSTPT when the QA is generated by Mistral or by the target model itself. That last result is important because it argues against the idea that the gains are just distillation from a stronger teacher. The finance gains are consistent across Llama-3.2-1B/3B and 3.1-8B, and the cost estimate ($12 for 100k samples) makes the pipeline practical.\n\nThe soft spot is the one the stress-test flags. The paper never checks whether the 100k generated QA pairs overlap with the evaluation benchmarks. GPT-4o-mini almost certainly saw MMLU-style items during pretraining, and the Bloomberg articles used to generate finance QA may be temporally or textually close to FOMC and NIFTY. Only 80 obvious 'in this context' samples are filtered. If even a few hundred generated questions resemble benchmark items, the headline 23.6% could be memorization. The self-generated ablation weakens the distillation story but not the overlap story—the target model's own generations could still memorize from its own token distribution. This is a testable concern: run n-gram or embedding overlap between the TELLME dataset and each benchmark. The paper needs that.\n\nOther, smaller issues: no error bars or significance tests; the medical gains are thin, especially at 8B; the retention experiment compares only against CPT, not against the integrated baselines; and Figure 7's caption says 'Human Evaluation' when the text describes an LLM-as-a-judge. None of these are fatal on their own, but together with the contamination question they keep the central claim conditional.\n\nOverall, this is a serious empirical paper with a real design-space contribution. It deserves peer review. The referee should require the contamination analysis and a rerun with multiple seeds before the numbers are taken at face value. I would not cite the headline numbers yet, but I would cite the method if the contamination check comes back clean.\n\nRecommendation: send to review, expect major revision.","headline":"A plausible, well-ablated CPT variant whose headline numbers are conditional on ruling out benchmark contamination in the GPT-4o-mini-generated QA pairs.","tokens_in":25019,"tokens_out":2421,"would_cite":false,"duration_ms":25092,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly training on domain text and masked question-answer pairs improves continual pre-training and long-term retention of large language models.","keywords":["continual pre-training","test-enhanced learning","domain adaptation","large language models","long-term memory retention","question generation","finance domain","medical domain"],"falsifier":"Compute token and paraphrase overlap between the 100,000 TELLME training QA pairs and the six evaluation benchmarks (FOMC, NIFTY, MMLU-F, HeadQA, MedMCQA, MMLU-C); if removing near-duplicate training items erases most of the reported gains, the improvement is benchmark memorization rather than a testing effect. A second check would be to replace each question with a random unrelated question while keeping the loss mask; if retention still improves, the mechanism is the mask itself, not the quiz content.","tokens_in":23999,"feed_emoji":"🧠","tokens_out":9618,"duration_ms":90284,"temperature":0.7,"pith_summary":"The paper claims that continual pre-training of a large language model becomes both cheaper and more durable when the raw domain text is paired with descriptive question-answer quizzes, and the question tokens are masked from the prediction loss. This TELLME recipe is tested on finance and medicine with LLaMA and SmolLM models; it reports up to a 23.6% improvement in financial benchmarks over existing continual-pre-training baselines and a 9.8% improvement in long-term retention of the first domain after a second, unrelated domain is learned. The quizzes are produced by a small generative model for about $12 per 100,000 samples, so the data side of domain adaptation stops being a bottleneck. A general reader should care because domain-specific LLMs currently require large curated corpora and expensive training runs, and this points to a way to get more knowledge out of each training token.","feed_headline":"Adding quizzes to LLM domain training beats plain text by 23.6%","feed_subtitle":"Masking question tokens forces internal-knowledge recall, lifting finance scores and long-term memory.","key_machinery":"The load-bearing mechanism is the masked-loss training objective over a sample $X=(t,q,a)$ in which the plain text $t$ and answer $a$ contribute to the loss while the question $q$ does not: the indicator function is set to $\\mathbf{1}(x_i\\in t\\cup a)=1$ and $\\mathbf{1}(x_i\\in q)=0$. The dataset-generation prompt is the other half of the machinery: it forbids questions about the excerpt and asks for general domain knowledge, so each quiz is a small retrieval-and-explanation exercise. The Coverage Ratio, the percentage of answer words that also appear in the source passage, is the instrument used to show that the quiz answers are not extractive; this is what links the training objective to the testing-effect explanation rather than to simple reading comprehension.","core_discovery":"The paper's central discovery, stated as its own claim, is that the testing effect from human learning psychology transfers to continual pre-training: asking a model a question it must answer from internal knowledge, while suppressing the question from the loss, makes the model acquire the surrounding plain-text knowledge more efficiently and retain it longer. In the reported experiments TELLME beats the common CPT+IT pipeline by 10.0% on average, beats the reading-comprehension-style INSTPT method by 6.3%, and after finance-then-medicine sequential training suffers only a 0.94% drop in finance benchmark score compared with a 5.72% drop for standard CPT. The paper also documents that the generated questions are genuinely distinct from the source text: average answer token overlap with the passage falls to 14.83% in finance with stopwords removed, versus 86.60% for INSTPT, which is the evidence that the model is being pushed to retrieve knowledge rather than copy context.","pith_inferences":["A direct follow-up would be to run TELLME with an overlap-filtered version of the generated quizzes; if the gains persist, the testing effect is real, and if not, the result would be attributed to benchmark similarity rather than to the method itself.","A curriculum variant that gradually increases how much the question depends on the passage might combine the transfer benefits of TELLME with the grounding of reading-comprehension QA; the paper does not explore this.","The question-masking objective may be acting as a soft instruction-tuning signal, teaching the model to treat any prompt-like prefix as context and to answer from latent knowledge; comparing against random-token masking would separate these mechanisms.","Because TELLME works with a self-generated dataset, it could bootstrap domain knowledge in low-resource languages or specialties before any curated instruction data exists, though the paper only measures output quality with a judge score, not downstream factual accuracy."],"forward_implications":["Domain adaptation for an LLM can be done with far less data and compute: 100,000 generated quiz samples cost about $12, and TELLME reaches a given perplexity roughly 1.4 times faster than plain continual pre-training.","Sequential multi-domain training becomes less destructive: knowledge learned with TELLME survives later out-of-domain training much better than knowledge learned by standard CPT.","The benefit is not tied to a specific question format or data source: the question-masking objective also improves results when applied to INSTPT's own dataset, and the method extends to Korean with gains on KoBEST.","The masking choice matters: letting the question contribute to the loss, separating QA from the text, or placing QA before the text all lower performance, so the order and loss mask carry the effect.","At a 70B scale under parameter-efficient fine-tuning and 4-bit quantization the improvement persists, though the paper notes this compressed setting may not fully represent a dense-model run."],"supporting_citations":[{"why":"Supplies the testing-effect evidence that motivates turning continual pre-training into a retrieval exercise.","marker":"Roediger III and Karpicke, 2006"},{"why":"Supports the choice of open-ended descriptive answers over simple recall or multiple choice.","marker":"Larsen et al., 2008"},{"why":"Shows the synergy between testing and concept mapping, which the TELLME questions are designed to reproduce.","marker":"Francis et al., 2020"},{"why":"Defines INSTPT, the instruction-pre-training baseline TELLME is compared against and whose dataset is reused in ablations.","marker":"Cheng et al., 2024"},{"why":"Defines the pre-instruction tuning approach that mixes QA and plain text as separate samples, a key baseline.","marker":"Jiang et al., 2024"},{"why":"Provides the LLaMA base models used in the main domain-adaptation experiments.","marker":"Dubey et al., 2024"},{"why":"Provides the SmolLM2 base model used for scaling, perplexity, and long-term-retention experiments.","marker":"Allal et al., 2025"},{"why":"Supplies the lm-evaluation-harness used for all benchmark evaluations.","marker":"Gao et al., 2024"},{"why":"Provides the FOMC benchmark, one of the three financial evaluation sets.","marker":"Shah et al., 2023"},{"why":"Provides the MedMCQA benchmark, one of the three medical evaluation sets.","marker":"Pal et al., 2022"}],"fun_headline_variants":["Quiz-style LLM pretraining beats plain text by 23.6%","Testing effect in LLM training cuts domain forgetting 6x","TELLME: Quizzes force LLMs to recall, not copy, boosting retention","LLM domain adaptation with quizzes: 10% average gain over CPT","Masking answers in LLM pretraining yields 14.83% answer overlap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains presume the GPT-4o-mini quizzes are genuinely context-independent and do not overlap with the evaluation benchmarks, because the paper filters only 80 of 100,000 clear-cut context-dependent samples and never checks overlap with the six test sets.","fun_headline_variants_meta":{"raw":{"variants":["Quiz-style LLM pretraining beats plain text by 23.6%","Testing effect in LLM training cuts domain forgetting 6x","TELLME: Quizzes force LLMs to recall, not copy, boosting retention","LLM domain adaptation with quizzes: 10% average gain over CPT","Masking answers in LLM pretraining yields 14.83% answer overlap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3066,"prompt_tokens":877,"completion_tokens":2189,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2089}},"tokens_in":493,"tokens_out":2189,"duration_ms":14761,"temperature":1.0,"reasoning_tokens":2089,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:26:57.160711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute token and paraphrase overlap between the 100,000 TELLME training QA pairs and the six evaluation benchmarks (FOMC, NIFTY, MMLU-F, HeadQA, MedMCQA, MMLU-C); if removing near-duplicate training items erases most of the reported gains, the improvement is benchmark memorization rather than a testing effect. A second check would be to replace each question with a random unrelated question while keeping the loss mask; if retention still improves, the mechanism is the mask itself, not the quiz content.","supporting_citations":[],"review_version":1}