{"id":"36910caa-0a40-4269-8575-72d1370805e3","arxiv_id":"2505.17140","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs fine-tuned on comprehension tasks like question answering retain injected facts at more than double the rate of models fine-tuned on translation or JSON mapping, but all models struggle to apply the facts in new contexts.","lead":"A new study fine-tuned large language models on the same facts using four different task formats and found that question-answering training makes models remember facts far better than translation or JSON-format training. The result matters for anyone updating proprietary or time-sensitive knowledge in AI systems, because it suggests the training task itself, not just the data, determines how well the model learns.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe-format confound threatens the central retention gap: translation/JSON models are tested only via QA, so low scores may reflect format mismatch, not failed internalization.","rationale":"The paper makes a plausible and interesting claim, and it has real strengths: multiple model families, manual validation of atomic facts, and a judge reliability check with strong inter-annotator agreement. However, the central comparison is not yet airtight because the evaluation instrument is not format-neutral. The reader's weakest_assumption identifies exactly this issue, and I agree that it is load-bearing. A single control experiment using format-matched probes for translation and JSON would settle whether the reported gap is an artifact. Because this is an empirically addressable concern rather than a demonstrated fatal flaw, the conditional-accept verdict remains appropriate. The paper should also report training hyperparameters and release artifacts, but those are secondary to the probe confound. I do not see an internal inconsistency in the central argument; the issue is a missing control, not a contradiction.","tokens_in":8286,"tokens_out":4678,"duration_ms":38933,"concrete_test":"Using the paper's exact data and protocol, fine-tune Qwen2.5-3B separately on the translation and JSON datasets. Evaluate each model under two probe conditions: (1) the paper's direct QA set; (2) a format-matched probe—for translation, prompt 'Translate this French sentence to English: [French fact]' and use the same GPT judge to check whether the output contains the fact; for JSON, prompt 'Output a JSON object with the fields location, date, people, and fact for: [English fact]' and check field values against the fact. If models score high on format-matched probes but low on QA, the central retention gap is an artifact of probe format; if they score low on both, the gap reflects genuine non-retention rather than probe incompatibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Section 4.2 rests on an untested evaluation assumption from Section 3.2: all four fine-tuned models are scored by answering direct natural-language questions. QA and blank models are at least trained on English sentence-level comprehension formats, whereas translation models are trained on French-to-English mapping and JSON models on structured field extraction. If fine-tuning on a mapping task induces a format-specific output policy, or if those models retain facts in a form that a QA probe cannot trigger, then the observed 48% vs. 17%/20% gap is a probe-compatibility artifact, not evidence about depth of internalization. The paper never evaluates translation/JSON models on translation-style or JSON-style probes, so it cannot distinguish 'not learned' from 'learned but not QA-accessible.' The partially mitigating fact that blank-trained models (trained on cloze, not QA) score 32% does not remove the confound, because translation and JSON also differ in modality and may suffer format overfitting. A neutral fact-generation probe or a format-matched probe is therefore structurally necessary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether the format of supervised fine-tuning determines how well newly introduced facts are retained by LLMs. It constructs 126 atomic facts about 2024 events, formats them into four task types (QA, fill-in-blank, French-to-English translation, and text-to-JSON), fine-tunes several open-weight models plus four Qwen2.5 sizes, and evaluates retention with direct and generic English questions scored by a GPT-4o-mini judge. The headline result is a large retention advantage for comprehension-style tasks (QA averaging about 48%, blanks about 32%) over mapping-style tasks (translation about 17%, JSON about 20%), together with monotonic improvement across model scale and a drop on generic questions. The authors conclude that effective knowledge injection depends on depth of semantic engagement rather than mere data exposure.","tokens_in":8443,"tokens_out":5880,"duration_ms":48089,"significance":"The research question is timely and practically relevant: if the fine-tuning task format controls whether parametric knowledge updates become usable, then task selection is a key design choice for knowledge editing and continual learning. Strengths include the use of post-cutoff facts to reduce contamination, multiple architectures plus a controlled scaling family, manual validation of generated facts, and a judge whose agreement with human annotators is explicitly measured. If the central comparison were clean, the result would be a useful empirical guideline for practitioners. However, the main claim is currently undermined by an evaluation-format confound and by missing training and statistical detail; with additional probe-matched control experiments and fuller reporting, the paper could become a solid contribution.","major_comments":[{"comment":"The central comparison is confounded by the evaluation format. All four fine-tuned models are scored only with direct natural-language questions, which is the same format as the QA training data and close to the blank-training format, but not the translation or JSON training formats. A translation-fine-tuned model may store facts in a French-to-English mapping policy that a QA prompt does not trigger, and a JSON-fine-tuned model may have learned to emit structured fields rather than answer questions. The observed gap (QA 48% vs. translation 17% and JSON 20%) could therefore reflect probe compatibility rather than depth of internalization. The paper needs format-matched probes for translation and JSON models, or a single neutral probe that is equally distant from all training formats, and it should report those results before claiming that mapping-oriented tasks fail to retain injected knowledge.","section":"3.2, 4.2, Table 1"},{"comment":"No fine-tuning hyperparameters are reported for any model (learning rate, epochs, batch size, optimizer, sequence length, LoRA rank if used), and each condition appears to be a single run with no error bars, multiple seeds, or significance tests. Because all comparisons are between separately fine-tuned models, differences in training configuration or run-to-run variance could account for part of the reported gaps. The authors should report the full training setup and at least a few seeds or confidence intervals for the main comparison.","section":"4.2, 4.4"},{"comment":"The claim that knowledge retention \"follows scaling laws\" and that \"knowledge integration\" follows a power law is not substantiated. The data show four monotonically increasing points per task, but no power-law function is fitted, no parameters or goodness-of-fit are reported, and there are no error bars. Either fit the scaling relationship quantitatively and report the fitted form, or revise the claim to \"retention improves monotonically with model size in the Qwen2.5 family,\" which is what the figure actually demonstrates.","section":"4.4, Figure 1"},{"comment":"The generic-question set used for RQ2 is not well grounded in the atomic facts. For example, given the fact \"India won the tournament, defeating South Africa in the final by 7 runs,\" the generic question asks for a list of teams that lost T20 World Cup finals with minimal score difference over the last few tournaments; that list cannot be derived from the atomic fact alone and requires external knowledge. Similarly, the Trump-felony fact is paired with a question about all U.S. presidents convicted of a felony. The judge is instructed to use only the atomic fact, so a model that answers using broader knowledge will not be evaluated consistently. The generic evaluation therefore does not cleanly measure transfer of injected knowledge, and the conclusion about \"limited semantic integration\" needs to be supported by questions that are answerable from the atomic fact alone.","section":"3.2, Table 5"}],"minor_comments":[{"comment":"The text refers to \"Appendix 3\" for the baseline results; this should be Appendix B (Table 3).","section":"4.2"},{"comment":"The judge model is referred to inconsistently as \"GPT-4 O-MINI,\" \"GPT-4o-mini,\" and \"GPT-4O-MINI\"; please unify the notation.","section":"3.3, Appendix D"},{"comment":"The baseline table reports \"Percentage Answered\" without stating whether this is judged correctness under the same GPT judge protocol used in Section 4.1; please clarify the evaluation procedure for baselines.","section":"Appendix B"},{"comment":"The figure would be much more informative with numeric labels or a table of the exact values, since the four-point curves are hard to read and the 32B/72B points are close for some tasks.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is a clearly written empirical study with a relevant research question, and the strengths (post-cutoff facts, multiple architectures, scaling family, judge validation) are real. However, the central comparison currently rests on an evaluation-format confound, and the paper also omits training hyperparameters and any variance information. These are fixable within the manuscript's scope by adding format-matched or neutral probes and reporting the missing experimental detail. I therefore recommend major revision rather than rejection. I have no concerns about scope or citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper asks a question worth asking: when you fine-tune on the same factual sentences formatted as QA, blanks, translation, or JSON, does the format change what the model actually retains? The answer they give is that comprehension tasks (QA 48%, blanks 32%) beat mapping tasks (translation 17%, JSON 20%), and they show the pattern across five architectures and a Qwen size sweep. That is a new empirical comparison, and the setup is better than most: facts are post-cutoff, baselines are measured, the judge is human-validated (94% agreement, kappa 0.884), and the generic-question probe shows all models drop, which is a nice secondary result.\n\nThe problem is that the headline gap is confounded with test format. Every model is scored on direct questions — natural-language, open-ended QA. QA and blanks models were trained on that kind of surface format; translation models were trained on French-to-English mapping, and JSON models on structured field extraction. If those models store the fact in a way a QA probe cannot trigger, the 48% vs 17% gap says nothing about depth of internalization. The stress-test note is right: no format-matched probe, no neutral fact-retrieval probe, so 'not learned' and 'learned but not QA-accessible' are indistinguishable. Blank-trained models scoring 32% doesn't save it — blanks are still closer to QA than translation/JSON are.\n\nLesser issues: no hyperparameters, no error bars or significance tests, and the 'scaling laws' language is an overstatement since no functional form is fit. No code or data release either, which makes the missing details harder to fill in.\n\nNone of this is fatal. The confound is addressable — probe in the training format or use a fact-completion test that doesn't depend on question style — and the generic-question finding is independent of the confound. The paper deserves a serious referee, but the verdict should be conditional on a probe-controlled replication and artifact release. If I were citing anything, it would be the benchmark idea, not the headline.\n\nRecommendation: send to peer review, but expect major revision.","headline":"Useful question, real confound: the four-task comparison is new, but the retention gap may be a probe-format artifact.","tokens_in":8996,"tokens_out":2211,"would_cite":false,"duration_ms":18051,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on questions embeds new facts about three times better than fine-tuning on translation.","keywords":["knowledge injection","supervised fine-tuning","task format","knowledge retention","transferability","scaling laws","semantic integration","LLM evaluation"],"falsifier":"Build a matched-probe study: after fine-tuning on translation, ask the model to translate the same French sentence back to English rather than answer an English question; after JSON fine-tuning, ask it to re-emit the JSON fields for the fact. If these matched probes close most of the reported gap versus QA-trained models, the depth-of-engagement claim fails. A second check would compute per-fact rather than aggregate retention to see whether particular facts drive the task differences.","tokens_in":8053,"feed_emoji":"🧠","tokens_out":7239,"duration_ms":57849,"temperature":0.7,"pith_summary":"The paper asks whether an LLM's fine-tuning task changes whether it truly absorbs new facts. The authors create 126 facts from events after typical knowledge cutoffs, rewrite each fact into four formats—question answering, fill-in-the-blank, translation, and text-to-JSON—and fine-tune separate models on each format. They report that comprehension-demanding formats retain far more of the facts in direct questioning: about 48% for QA and 32% for blanks, versus 17% for translation and 20% for JSON. The pattern holds across model families and model sizes, and larger models do better in every format. When the facts are probed with indirect questions that require applying the knowledge, all models drop sharply, so the claim is about accessible retention, not full understanding.","feed_headline":"QA fine-tuning embeds new facts 3x better than translation","feed_subtitle":"Comprehension tasks retain 48% of injected facts; translation and JSON retain only 17-20%, at identical data exposure.","key_machinery":"The load-bearing instrument is a parallel task-format dataset built from one set of 126 atomic facts—short standalone sentences about post-cutoff events such as the 2024 U.S. election and the Men's T20 World Cup. Each fact is reformatted into four training tasks, so data exposure is identical while the required linguistic operation differs. Retention is then measured twice: direct questions that rephrase the training question, and generic questions that require transferring the fact to a new context. The gap between the two probes is used as the evidence for shallow rather than deep internalization.","core_discovery":"The central discovery is that exposure to a fact is not enough: the operation the model performs with that fact during fine-tuning determines whether it can later retrieve it. With identical atomic facts in the training data, models fine-tuned on question-answer pairs answer about 48% of direct questions correctly, blank-filling reaches about 32%, while translation and text-to-JSON conversion reach only 17% and 20%. This ordering repeats across every architecture tested and across model sizes from 1.5B to 72B parameters, with retention improving monotonically with size in all four task formats. The second half of the finding is a ceiling on integration: accuracy on indirect generic questions is much lower for every task and model, meaning the injected knowledge remains only partially transferable.","pith_inferences":["One untested reading the paper leaves open: the direct probe is always a natural-language question, so translation- or JSON-trained models may store facts in a form the probe cannot reach, which would make the retention gap a probe-compatibility effect rather than only a depth-of-engagement effect.","A fair cross-format test would probe each training format in its own output language or structure—for example, asking translation-trained models to translate the fact back from French—and would likely tighten or widen the reported gap.","The scaling pattern is demonstrated on one model family, so the claim that the task gap persists at all scales rests on that family; testing the same four formats on another family would extend the result."],"forward_implications":["Knowledge-update pipelines should treat task format as a decision variable: question-answer and blank-filling data retain roughly two to three times more facts than translation or JSON data at equal exposure.","Model scale and task format act as independent levers, since larger models improve in every task while the task ordering stays the same.","Fine-tuning on mapping tasks is an inefficient way to update factual knowledge and should be avoided when downstream use is factual recall.","No tested task format produced reliable transfer to indirect questions, so supervised fine-tuning alone cannot be assumed to give a model usable, context-flexible knowledge."],"supporting_citations":[{"why":"supplies the atomic-fact decomposition method used to build the 126 standalone facts that feed all four training tasks.","marker":"(Mecklenburg et al., 2024)"},{"why":"frames supervised fine-tuning as the knowledge-injection mechanism and provides the cost and benefit comparison against retrieval and pre-training.","marker":"(Ovadia et al., 2024)"},{"why":"prior evidence that generation and classification fine-tuning generalize differently, motivating the task-format comparison.","marker":"(Yang et al., 2024)"},{"why":"shows LLMs can rely on pattern matching rather than understanding, motivating the generic-question transfer test.","marker":"(Yan et al., 2025)"},{"why":"supplies chain-of-thought prompting used in the automatic judge to elicit and score the model's justification.","marker":"(Wei et al., 2023)"},{"why":"provides the inter-annotator agreement measure used to validate the human-judge reliability.","marker":"(Cohen, 1960)"}],"fun_headline_variants":["Comprehension tasks triple LLM knowledge retention over translation","Task choice, not data exposure, decides if LLMs retain injected facts","QA fine-tuning beats translation 3-to-1 for knowledge injection","Deep comprehension tasks triple fact retention vs shallow mapping tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that asking a single direct English question measures retained knowledge equally well for all four training formats; if translation- or JSON-trained models store facts in a form that this probe cannot access, the observed gap would reflect probe compatibility rather than depth of internalization.","fun_headline_variants_meta":{"raw":{"variants":["Comprehension tasks triple LLM knowledge retention over translation","Task choice, not data exposure, decides if LLMs retain injected facts","QA fine-tuning beats translation 3-to-1 for knowledge injection","Deep comprehension tasks triple fact retention vs shallow mapping tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3195,"prompt_tokens":848,"completion_tokens":2347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2276}},"tokens_in":464,"tokens_out":2347,"duration_ms":15062,"temperature":1.0,"reasoning_tokens":2276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:04:08.894281+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a matched-probe study: after fine-tuning on translation, ask the model to translate the same French sentence back to English rather than answer an English question; after JSON fine-tuning, ask it to re-emit the JSON fields for the fact. If these matched probes close most of the reported gap versus QA-trained models, the depth-of-engagement claim fails. A second check would compute per-fact rather than aggregate retention to see whether particular facts drive the task differences.","supporting_citations":[],"review_version":1}