{"id":"27d60315-3d1b-4e76-a259-010aadbe80d3","arxiv_id":"2510.09885","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A mask-and-reconstruct fine-tuning objective lets autoregressive LLMs inject new knowledge without paraphrases and largely removes the reversal curse.","lead":"Fine-tuning an autoregressive LLM to reconstruct a masked version of new text—instead of reading it normally—lets the model answer questions about that text in both directions without costly paraphrase augmentation. The authors show this masked fine-tuning largely closes the knowledge-injection gap with diffusion LLMs and gives small gains on math fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline QA accuracy is ROUGE-1 recall with no threshold, and A.7 shows models scoring by reproducing whole training passages; this could inflate masked-fine-tuning gains over vanilla SFT, so the central 'closes the gap' claim needs re-scoring under a strict QA metric.","rationale":"I read the paper as claiming that the demasking objective alone, without paraphrases, transfers the knowledge-injection data efficiency of masked diffusion LLMs to autoregressive LLMs. The load-bearing evidence is Table 1's QA accuracies. The weakest point in that evidence is the evaluation metric. Section 3 explicitly defines accuracy as ROUGE-1 recall with no threshold, and Appendix A.7 shows high-scoring outputs that are nearly verbatim reproductions of the fine-tuning document. Because the masked-fine-tuning training target is the original document, the model is trained to reproduce full passages; evaluating with an unthresholded recall metric then rewards that behavior. This is a correctness risk, not a consensus dispute: exact match is the standard for short-fact QA, and the paper itself uses exact match for the math experiments, where its gains are far smaller. The random-token control in Figure 12 is a good design element and shows the demasking mechanism matters, but it does not resolve whether the metric measures question answering. Variance and best-checkpoint selection are additional concerns, but they are secondary because the qualitative pattern might survive; the metric issue strikes at the magnitude and interpretation of the headline numbers. I therefore agree with the reader's weakest assumption and see no reason to change the CONDITIONAL verdict. The abstract's unsupported 1.2M GPQA-diamond claim should also be fixed, but it is not the central load-bearing issue.","tokens_in":23226,"tokens_out":2986,"duration_ms":31503,"concrete_test":"Re-evaluate all Table 1 conditions (or at least the Llama-3.1-8B and LLaDA-8B rows) with a strict QA metric: exact-match or answer-string containment on the generated responses, using the same best checkpoints. Concretely, run the existing generation pipeline and compute (1) exact-match accuracy against the ground-truth answer, and (2) a human-judged 'direct answer present and not buried in a verbatim passage' score on a random 50-question subset. If masked fine-tuning retains most of its advantage under exact match (e.g., within 10 points of its ROUGE-1 score), the central claim survives; if exact-match accuracy drops toward vanilla-SFT levels, the claimed gap-closing is largely a measurement artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that masked fine-tuning enables arLLMs to reach dLLM-level QA accuracy without paraphrases. The evidence for this rests on an accuracy metric defined in Section 3 as the proportion of ground-truth answer words appearing in the generated text (ROUGE-1 recall), with no stated length or exactness threshold. The A.7 generation examples make the risk concrete: masked Llama on the Wiki forward question outputs the entire original article, and masked Llama on NameDescription backward outputs the full sentence containing the answer. Such outputs trivially score near 1.0 on ROUGE-1 recall because the answer words are contained in a verbatim reproduction of the training passage. The evaluation therefore partially measures regurgitation of the fine-tuning documents rather than the ability to answer a concise question. If a strict QA metric (exact match, or answer-only extraction) were applied, the reported gaps -- e.g., 0.377 vs. 0.980 on Wiki forward, or near-zero vs. 0.90-0.95 on NameDescription backward -- could shrink substantially. Independent support for this concern is visible in the paper's own masked-SFT math results (Section 8, Table 3), which use exact-match evaluation and show only small gains over vanilla SFT (+0.003 to +0.049). The random-token control in Appendix Figure 12 does establish that the demasking prompt matters, but it does not establish that the measured quantity is QA correctness. The missing GPQA-diamond 1.2M-sample experiment from the abstract is a separate reporting gap; the decisive validity issue remains the evaluation metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies post-training knowledge injection in autoregressive LLMs (arLLMs) versus masked diffusion LLMs (dLLMs). Across three datasets (NameDescription, Biography, Wiki), it reports that dLLMs achieve strong forward and backward QA accuracy without paraphrase augmentation, whereas arLLMs need paraphrases, and even then fail on backward questions. Motivated by the dLLM's demasking objective, the authors propose 'masked fine-tuning' for arLLMs: the model is prompted with a randomly masked document and trained to reconstruct the original text, using the original document as the supervised target. They report that this method closes most of the QA accuracy gap between arLLMs and dLLMs without paraphrases, at roughly twice the per-step training cost but with higher converged accuracy. They further extend the idea to math SFT (GSM8K, MATH) and, in the abstract, claim a result on the 1.2M-sample GPQA-diamond dataset that does not appear in the body. The main tables report QA accuracy via ROUGE-1 recall with no threshold; appendix generation examples show passages reproduced verbatim, which motivates the central concern about validity of the reported metric.","tokens_in":23437,"tokens_out":5849,"duration_ms":59683,"significance":"If the central result is borne out under a strict QA metric, the paper is significant: it offers a simple, compute-compatible way to improve knowledge-injection data efficiency in existing arLLMs, with a clean random-token control suggesting the effect is due to the demasking content rather than prompt format. The study is broad (four arLLMs, one dLLM, three knowledge datasets, plus two math datasets with exact-match evaluation), and the code/dataset repository is promised, which helps reproducibility. The empirical breadth and the existence of a control experiment are genuine strengths. However, the headline QA numbers rest on an unusually lax metric, and one advertised large-scale result (GPQA-diamond) is missing from the body. The significance of the paper therefore depends on whether the reported QA gains survive re-scoring with answer-level metrics; the paper's own exact-match math results (+0.003 to +0.049) suggest this is a real risk.","major_comments":[{"comment":"The 'accuracy' metric is ROUGE-1 recall (proportion of ground-truth words appearing in the generated text) with no length threshold. The generation examples in A.7 make the risk concrete: on the Wiki forward question, masked Llama 8B outputs the full training article, and on Biography, the dLLM emits a long passage containing the answer name; such outputs score near 1.0 by construction because the ground-truth words occur in the verbatim or near-verbatim passage. Since Table 1 compares conditions that differ greatly in verbosity (masked fine-tuning tends to reproduce whole passages; vanilla arLLM fine-tuning often produces shorter or off-topic text), the reported gaps—e.g., 0.377 vs. 0.980 on Wiki forward—may substantially overestimate the advantage. The paper's own exact-match math results in Section 8/Table 3 show only small gains (+0.003 to +0.049), which is consistent with the concer","section":"Section 3, Appendix A.7"},{"comment":"The abstract states: 'on a large-scale knowledge-intensive dataset (1.2M samples), masked SFT achieves the best downstream accuracy on GPQA-diamond among all fine-tuning variants.' No GPQA-diamond experiment, dataset, table, or evaluation protocol appears anywhere in the body or appendices; Section 8 and Appendix A.5 cover only GSM8K and MATH. This is a load-bearing result used to support the claim of broader applicability. Either add the experiment with full details or remove the claim from the abstract.","section":"Abstract, Section 8"},{"comment":"The text claims that masked fine-tuning (condition 5) 'converges at the highest accuracy with more than 2×convergence rate of all the other methods.' Table 4 reports fitted convergence rates k (Fwd/Bwd): condition (1) 0.0350/0.1337, (2) 0.0069/0.0130, (3) 0.0151/0.0495, (4) 0.0052/0.0081, (5) 0.0032/0.0029. Condition (5) has the lowest k in both directions, so the claim is contradicted by the paper's own table. The cost-comparison conclusion should be revised; if the intended argument is that condition (5) reaches a higher converged accuracy despite a lower rate, state that directly with the supporting numbers.","section":"Section 9, Table 4"}],"minor_comments":[{"comment":"Main text says the Wiki dataset contains 94 articles; Appendix A.3 says the filtering results in 96 wiki articles. Please reconcile.","section":"Section 3, Appendix A.3"},{"comment":"The tables report accuracy from 'best checkpoints' chosen by total accuracy on the evaluation set. Please report the selection rule explicitly and include variance across seeds or random restarts for all methods, not only for dLLM and masked arLLM (Figures 10–11).","section":"Figure 2 and Tables 1, 5"},{"comment":"The loss in Eq. (2) is written with m_t as a selector but the notation is not fully defined (token indices vs. a binary vector). Clarify the indexing and the normalization by the sum of m_t.","section":"Section 6, Equation (2)"},{"comment":"The paper cites 'Lin et al., 2024' for the ROUGE-1 score; the standard reference is Lin (2004). Please correct the citation.","section":"Section 3"},{"comment":"Learning rates differ across models and datasets, and some are chosen by 'smooth gains' in the text. Please state the hyperparameter search protocol more concretely (especially the number of points and the criterion for 'smooth').","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the random-token control is a good first step. However, the evaluation metric is too weak to support the headline claim as written. If the authors re-score with exact-match or answer-extraction F1 and the gains mostly persist, this could be a strong paper. The missing GPQA-diamond experiment is a separate, clear problem: an abstract-level claim must be backed by the body. I also noticed the convergence-rate claim is contradicted by Table 4; that needs a straightforward correction. I would not reject on the current version, but these issues are load-bearing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth reading and worth refereeing. The masked fine-tuning paradigm — prompt an instruct arLLM to reconstruct a randomly masked document, with the unmasked document as target — is new relative to the cited work on reversal curse and paraphrase augmentation. The random-token control (Fig 12) is a good test: it shows the benefit comes from the demasking content, not just extra input tokens. And the breadth is real: four arLLMs, one dLLM, three knowledge datasets, plus a two-model/two-dataset math check. The math results use exact match and still show a consistent small gain, which is the most trustworthy evidence in the paper.\n\nThe soft spots are specific and addressable. First, the main QA metric is ROUGE-1 recall with no threshold. The appendix examples make the problem concrete: several best-checkpoint models answer by reproducing the whole training passage. Such outputs score near perfect on this metric by construction, so the large gaps in Table 1 (e.g., 0.377 vs 0.980) could shrink under a strict exact-answer or answer-only evaluation. The paper's own math results, which use exact match, show gains of only +0.003 to +0.049 — a useful upper bound on how much of the QA gap is real. Until the QA numbers are re-scored, I'd treat the 'closes the gap' claim as promising but not proven at the stated magnitude.\n\nSecond, the abstract claims a 1.2M-sample GPQA-diamond result that never appears in the body. That's a reporting gap and should be fixed. Third, all table numbers are best-checkpoint selections (A.4), and no variance or significance is reported; that matters for the small math gains and for the dLLM comparisons. Fourth, the dLLM column is a single architecture (LLaDA-8B), so the generalization to 'dLLMs' is provisional.\n\nNone of these is load-bearing in the sense that the demasking objective is probably doing real work — the random-token control and the exact-match math results both point that way. But the metric issue is central enough that the paper needs a re-evaluation before the headline claim is accepted. Who should read it: anyone working on knowledge injection, reversal curse, or diffusion LM post-training. It deserves a serious referee: the method is simple, potentially useful, and the empirical question is well-posed. My recommendation: send it to review, but require a strict QA metric and a reproducible evaluation pipeline before acceptance.","headline":"The masked fine-tuning idea is genuinely new and the demasking objective clearly helps, but the headline QA metric (ROUGE-1 recall) is loose enough that the 'closes the gap' claim needs re-scoring before I'd trust the size of the effect.","tokens_in":24159,"tokens_out":1956,"would_cite":true,"duration_ms":16539,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masked reconstruction training lets autoregressive LLMs learn new facts from plain documents, with no paraphrase augmentation, and removes the reversal curse.","keywords":["knowledge injection","masked fine-tuning","reversal curse","diffusion language models","autoregressive language models","data efficiency","supervised fine-tuning","question answering"],"falsifier":"Run a strict exact-match evaluation on the same fine-tuned models—parse the generated answer and require it to contain exactly the ground-truth entity (e.g., a name or date) with no additional sentences—and compare masked fine-tuning against vanilla fine-tuning on the Wiki and NameDescription datasets; if the previously reported margins (e.g., 0.980 vs 0.377 forward) collapse, the central claim fails.","tokens_in":22926,"feed_emoji":"🧠","tokens_out":9306,"duration_ms":68143,"temperature":0.7,"pith_summary":"Autoregressive large language models are hard to update with new facts: fine-tuning on new documents often fails to generalize to question answering, and questions that reverse the order of the training text (the 'reversal curse') remain near-impossible without expensive paraphrase augmentation. This paper shows that masked diffusion LLMs, which learn by reconstructing masked tokens, do not have this problem: after fine-tuning on the same documents they answer both forward and backward questions accurately, with no paraphrases. To bring this advantage to autoregressive LLMs, the authors propose 'masked fine-tuning': mask a random fraction of the document, prompt the model to recover the full text, and train against the original document. Across four autoregressive models and three knowledge datasets, this single change lifts backward QA accuracy from near zero to above 0.9 and forward accuracy into the high 0.9s, closing the gap with diffusion LLMs. The same masked objective also improves supervised fine-tuning on math tasks, suggesting a general training recipe rather than a knowledge-specific trick.","feed_headline":"Masked fine-tuning lets LLMs learn new facts without paraphrases","feed_subtitle":"The demasking objective lifts backward question accuracy from near zero to over 0.9 — no paraphrase augmentation needed.","key_machinery":"The central object is the masked fine-tuning paradigm: during fine-tuning, a random fraction t of each document's tokens (uniformly sampled from 0.05 to 0.95) is replaced with a special [MASK] token; the masked document is placed in the prompt with an instruction to 'return the recovered masked passage,' and the original unmasked document serves as the supervised target, with the autoregressive loss applied only to the assistant response tokens. This emulates the mask-reconstruction loss of masked diffusion LLMs without changing the autoregressive architecture or the optimization target. The random mask ratio is crucial: it modulates task difficulty, and a control experiment with random toke","core_discovery":"The central claim is that the demasking objective alone—having the model reconstruct a randomly masked document, with the unmasked document as the supervised target—transfers the data-efficiency advantage of masked diffusion LLMs to autoregressive LLMs during fine-tuning. The paper reports that without any paraphrase augmentation, masked fine-tuning on the Wiki dataset raises forward accuracy from 0.377 to 0.980 and backward accuracy from 0.282 to 0.930 for an 8-billion-parameter autoregressive model, and lifts NameDescription backward accuracy from near zero to 0.90–0.95. This holds across four model sizes/families and three datasets, and a control experiment replacing the masked prompt wit","pith_inferences":["Beyond the paper: the reported 'accuracy' is ROUGE-1 recall without an exact-match constraint, and the appendix shows successfully fine-tuned models often reproducing the entire source passage; a strict exact-answer metric could substantially shrink the reported margins and should be run before assuming the knowledge-injection conclusion.","Beyond the paper: a sharper test of genuine knowledge acquisition would be to mask the answer token itself in the prompt during fine-tuning and check whether the model can still produce it at inference; if it cannot, the method may be inducing passage-level memory rather than a reusable association.","Beyond the paper: the fixed-mask-ratio experiments suggest a sweet spot around t ≈ 0.5–0.75; annealing the mask ratio or using a curriculum that progresses from lightly to heavily masked documents may further improve data efficiency and reduce the seed-to-seed variance visible in the paper's training curves."],"forward_implications":["If masked fine-tuning works as described, parametric knowledge injection no longer requires paraphrase-augmented training data, removing a major computational bottleneck in updating LLMs with new facts.","The method transfers the reversal-curse resistance of diffusion LLMs to autoregressive models, so models can answer questions that invert the information order of their training text.","Because the same demasking objective improves supervised fine-tuning on GSM8K and MATH across two models, the benefit is not limited to factual QA and may be a general-purpose training objective.","The convergence analysis indicates masked fine-tuning reaches higher final accuracy at comparable total compute, despite roughly doubling per-step FLOPs, because it needs fewer training steps.","This offers a practical path to lifelong learning: a small amount of new text, masked and presented as a reconstruction task, can update a deployed autoregressive LLM without relying on external retrieval or paraphrase generation."],"fun_headline_variants":["Demasking objective alone boosts fact recall in LLMs","Masked fine-tuning: no paraphrases needed for knowledge injection","Reconstructing masked text teaches LLMs new facts without paraphrases","Masked fine-tuning fixes the reversal curse without paraphrases","Demasking objective lets AR LLMs skip paraphrase augmentation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire argument rests on the evaluation metric: accuracy is measured as ROUGE-1 recall without a threshold, so a model that merely regurgitates the training passage scores near-perfect; if this metric does not reflect genuine question answering, the reported knowledge-injection gains may be illusory.","fun_headline_variants_meta":{"raw":{"variants":["Demasking objective alone boosts fact recall in LLMs","Masked fine-tuning: no paraphrases needed for knowledge injection","Reconstructing masked text teaches LLMs new facts without paraphrases","Masked fine-tuning fixes the reversal curse without paraphrases","Demasking objective lets AR LLMs skip paraphrase augmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3151,"prompt_tokens":816,"completion_tokens":2335,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":816,"completion_tokens_details":{"reasoning_tokens":2253}},"tokens_in":816,"tokens_out":2335,"duration_ms":495297,"temperature":1.0,"reasoning_tokens":2253,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:23:31.682966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a strict exact-match evaluation on the same fine-tuned models—parse the generated answer and require it to contain exactly the ground-truth entity (e.g., a name or date) with no additional sentences—and compare masked fine-tuning against vanilla fine-tuning on the Wiki and NameDescription datasets; if the previously reported margins (e.g., 0.980 vs 0.377 forward) collapse, the central claim fails.","supporting_citations":[],"review_version":2}