{"id":"229dc3ad-efa7-418f-8502-b3731b79a327","arxiv_id":"2505.12306","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WikiDYK benchmarks knowledge injection using real-world Wikipedia facts and reports that bidirectional models memorize new knowledge more reliably than causal models.","lead":"WikiDYK, a new benchmark built from Wikipedia's 'Did You Know' entries, measures how well language models memorize newly injected facts through 77,180 questions over 12,290 facts. The authors report that bidirectional models (T5-family) memorize better than causal models (Llama, Qwen, Gemma) under continued pre-training, and an ensemble pipeline improves reliability accuracy by up to 29.1 percentage points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'BiLMs memorize better' claim is confounded by pretraining and instruction tuning; the authors' own Limitation concedes no matched-pretraining control, and internal BiLM results (e.g., roberta-large at 3.0) show bidirectionality alone is insufficient.","rationale":"The reader's weakest-assumption analysis pinpoints the same issue I consider most load-bearing: the BiLM-versus-CLM comparison does not control for scale, pretraining data, or instruction tuning. I agree with that assessment and with the CONDITIONAL verdict. My stress-test adds two concrete observations that make the concern more than a generic 'confound' worry. First, the paper's own Table 8 shows that within the BiLM category, roberta-large (a genuine bidirectional model) performs at 3.0 reliability while Flan-T5-770M reaches 73.0; if bidirectionality were the causal mechanism, this within-family spread would not be so large. Second, T5-v1.1-large (52.0) versus Flan-T5-770M (73.0) demonstrates that instruction tuning alone can produce a 21-point reliability gap within the same architecture family. These internal results support the interpretation that the headline 46.09 versus 16.09 gap in Table 4 may be driven by pretraining objective and instruction tuning rather than by attention direction. The proposed causal-mask control on a fixed T5 checkpoint is a practical way to isolate the attention variable while keeping all other factors identical. Until such an experiment is run, the benchmark and ensemble method remain valuable contributions, but the architecture claim should be labeled as a hypothesis. The verdict therefore remains CONDITIONAL: the paper should either run the control or substantially soften the causal claim.","tokens_in":19271,"tokens_out":5431,"duration_ms":61022,"concrete_test":"Run a single controlled attention-mask experiment using the same starting checkpoint, data, objective, and optimizer: take Flan-T5-770M and continue pretraining on the same WikiDYK facts with s=1000, once with the standard full bidirectional attention mask and once with a strictly causal (left-to-right) attention mask, keeping all other hyperparameters identical. Then evaluate both variants with the same protocol as Table 4. If the bidirectional variant's reliability match accuracy remains substantially higher (e.g., near 46 vs near 16), the architectural explanation is supported; if the gap collapses or reverses, the original comparison was confounded by pretraining, instruction tuning, or model family.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is not that WikiDYK is a useful benchmark, but that bidirectional architectures are intrinsically better knowledge memorizers; this inference is load-bearing and insecure. Table 4 compares Flan-T5-220M/770M against Llama, Qwen, and Gemma checkpoints that differ in pretraining corpus, tokenizer, model scale, and instruction tuning. Flan-T5 models are instruction-tuned and pretrained with T5-style span corruption, while the CLMs are base autoregressive models; the CLM span-prediction objective is an added post-hoc formulation, not their native pretraining task. The Limitation section explicitly states: 'limited by the computing resource, we are not able to completely pre-train a BiLM and a CLM under the same set of hyperparameter and data ... without further controlling the experiments.' The paper's own internal evidence reinforces the confound: in Table 8, on the same 1000-fact set, roberta-large (a bidirectional encoder) reaches only 3.00 reliability match accuracy, while T5-v1.1-large reaches 52.00 and Flan-T5-770M reaches 73.00. If 'bidirectional' were the operative variable, roberta-large should not collapse relative to other BiLMs. Similarly, CLM results are highly sensitive to training objective (e.g., Llama-3.2-1B: 16.92 with QA versus 3.03 with SP in Table 4), so the objective/format interaction is itself a major variable. The observed gap may therefore reflect instruction tuning, T5-style span-corruption pretraining, or model family differences rather than causal versus bidirectional attention. The paper's conclusion that the gain is 'related with architectural advantage' (Section 5.2) is presented as a direct finding, but the design does not isolate that variable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WikiDYK, a knowledge injection benchmark built from Wikipedia's human-curated 'Did You Know?' entries, containing 12,290 facts and 77,180 question-answer pairs spanning reliability, generality, paraphrase, portability, and locality questions. The authors evaluate continued pretraining under next-token prediction, synthetic QA, and span prediction objectives on several off-the-shelf CLMs and BiLMs, concluding that bidirectional LMs memorize injected knowledge substantially better than causal LMs (e.g., Flan-T5-770M reaches 46.09 reliability match accuracy vs. 16.09 for Llama-3.1-8B in Table 4). They further propose an ensemble pipeline that routes queries to specialized BiLM knowledge repositories with a scope classifier, reporting up to a 29.1% reliability improvement over the Flan-T5-220M base. The paper also reports static and RAG-augmented baselines, scaling analyses, and ablations of the clustering and routing components.","tokens_in":19632,"tokens_out":4728,"duration_ms":43383,"significance":"If the central comparison were clean, the paper would make a strong and surprising contribution to the knowledge injection literature: a continuously updatable, human-curated benchmark with a multi-dimensional evaluation protocol, plus a practical modular framework for using small bidirectional models as external memory. The dataset construction is thoughtful, and the public release of data and code is a clear strength. However, the headline architectural conclusion is not currently supported by the experiments, because the BiLM and CLM suites differ in scale, pretraining data, and instruction tuning, as the authors' own Limitation section concedes. This makes the paper's main claim a hypothesis to be tested rather than an established result. The benchmark and ensemble framework remain useful even if the architectural claim is softened or reframed as an observation about off-the-shelf checkpoints.","major_comments":[{"comment":"The central claim that BiLMs are intrinsically better knowledge memorizers is confounded by uncontrolled model differences. Flan-T5-220M/770M are instruction-tuned T5 models trained with span corruption, while Llama, Qwen, and Gemma are base autoregressive models of different sizes and pretraining corpora; the Limitation section explicitly states that the authors could not pre-train a BiLM and a CLM under the same hyperparameters and data. The observed 46.09 vs. 16.09 reliability gap in Table 4 could therefore reflect instruction tuning, scale, or pretraining data rather than bidirectionality. I recommend either adding controlled comparisons within the same model family (e.g., causal vs. bidirectional attention variants of the same architecture with matched data and scale) or reframing the paper's claim as an observation about off-the-shelf checkpoints rather than an architectural result.","section":"§5.2, Table 4; Limitation"},{"comment":"The claimed objective alignment between CLMs and BiLMs is incomplete. For CLMs, span prediction is implemented with the prompt 'Predict the masked words in the following sentence' and an appended mask token at test time; this is not the native T5 span-corruption objective used by Flan-T5, and Table 4 shows that CLM results are extremely sensitive to objective choice (Llama-3.2-1B: 16.92 QA vs. 3.03 SP on reliability). Thus the comparison does not isolate architecture from training objective. I would like to see the same T5-style masked span formulation applied to both a causal and a bidirectional variant of the same backbone, or at minimum a detailed discussion of why the prompt format cannot disadvantage CLMs.","section":"§4.2, §4.3, Table 4"},{"comment":"The synthetic-QA training rows may be inflated by evaluation leakage. Training questions are generated by gpt-4.1-mini from the same facts used to generate the evaluation questions (generated by GPT-4o, GPT-4.1, and o3-mini per Appendix B), and the QA-trained CLMs are evaluated on questions derived from the same source facts. If there is surface-form or template overlap between training and evaluation questions, the QA rows overestimate memorization. Please report an overlap analysis (e.g., n-gram overlap or exact match between training and evaluation questions) and quantify how many evaluation questions are identical or near-identical to training questions.","section":"§3.2, §4.2, Appendix B"},{"comment":"The paper's internal evidence weakens the attribution to bidirectionality. On the same 1,000-fact set, roberta-large (a bidirectional encoder) reaches only 3.00 reliability match accuracy, while t5-v1.1-large reaches 52.00 and Flan-T5-770M reaches 73.00. This large spread within the BiLM category suggests that pretraining objective, model family, and generation setup, rather than bidirectionality per se, drive most of the effect. The paper should analyze this spread explicitly; otherwise the term 'BiLM' is too coarse to support the stated conclusion.","section":"Appendix C, Table 8"}],"minor_comments":[{"comment":"The final paragraph of Section 6 contains a typo: 'uture work' should read 'Future work'.","section":"§6"},{"comment":"The Figure 1 caption contains garbled text ('TheBiLMs: T5/Roberta/…[M]first' and 'TheCLMs: Llama/Qwen/…Sausages') that should be cleaned before publication.","section":"Figure 1"},{"comment":"The benchmark name is rendered inconsistently as 'WIKI DYK', 'WikiDYK', and 'WIKIDYK'; please standardize the spelling.","section":"Throughout"},{"comment":"Table 7 lists 'Llama-3.1-1B' while the main text and Table 4 refer to 'Llama-3.2-1B' for the same experimental family; please confirm which model was used and correct the inconsistency.","section":"Table 7"},{"comment":"The Limitation section states 'we prove this assumption empirically,' but an uncontrolled empirical comparison does not constitute proof; I suggest rephrasing to 'provide empirical evidence consistent with' or similar.","section":"Limitation"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a solid contribution and is likely to be reused by the community; the main risk is that the strong architectural claim will be cited despite the acknowledged confound. I would encourage the editor to require either matched-pretraining control experiments or a substantial rewrite of the claims before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. First, WikiDYK is a genuinely useful resource: it takes Wikipedia's daily Did-You-Know entries, which are human-curated, temporally fresh, and verifiable, and turns them into a QA benchmark with reliability, generality, paraphrase, portability, and locality splits. That fills a real gap—most knowledge-injection benchmarks use synthetic or stale Wikipedia snapshots. The authors ship code and data, and the benchmark can be extended automatically as new DYK entries appear. I'd use it.\n\nSecond, the headline claim—that bidirectional LMs are intrinsically better knowledge memorizers than causal LMs—is not actually established by the experiments, and the authors know it. The Limitation section says plainly they could not pre-train matched BiLMs and CLMs under the same hyperparameters and data, so they used off-the-shelf models differing in scale, pretraining corpus, and instruction tuning. Flan-T5 models are instruction-tuned span-corruption models; the CLMs are base autoregressive models. The internal evidence makes the confound concrete: on the same 1,000-fact set, roberta-large (a bidirectional encoder) reaches 3.0 reliability match, while T5-v1.1-large reaches 52.0 and Flan-T5-770M reaches 73.0. If bidirectionality were the operative variable, roberta-large should not collapse relative to other BiLMs. And CLM results swing hugely with training objective—Llama-3.2-1B gets 16.92 with synthetic QA but 3.03 with span prediction—so the objective/format interaction is itself a major factor.\n\nWhat the paper does well beyond the benchmark: the systematic comparison of NTP, synthetic QA, and span prediction objectives is informative, and confirms that plain next-token prediction is poor for knowledge injection. The ensemble pipeline (routing queries to cluster-specific BiLMs via a scope classifier) is a sensible modular idea, and the ablations with an oracle classifier are honest about where the gains come from.\n\nWho should read this: anyone building knowledge-injection benchmarks or studying continual learning of facts. It deserves a serious referee—the benchmark is reproducible and valuable, and a good referee can push the authors to either soften the architecture claim to 'in our comparisons, the Flan-T5-style models with span prediction memorized better' or to add controlled pretraining. I'd accept it for peer review, but I'd tell the authors the current abstract overstates what the evidence supports.","headline":"WikiDYK is a solid, reusable benchmark, but the paper's headline claim that bidirectional architectures memorize better is confounded by pretraining and instruction-tuning differences—the authors' own limitation section concedes the key missing control.","tokens_in":20206,"tokens_out":3077,"would_cite":true,"duration_ms":27174,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bidirectional LMs memorize injected facts better than causal LLMs.","keywords":["knowledge injection","WikiDYK","bidirectional language models","causal language models","continued pretraining","span prediction","catastrophic forgetting","model ensemble"],"falsifier":"Train a causal and a bidirectional model from scratch on identical data and compute, inject the same WikiDYK facts with the same span-prediction objective, and compare reliability match accuracy; if the causal model matches or beats the bidirectional one, the architectural claim is refuted.","tokens_in":19085,"feed_emoji":"🧠","tokens_out":6672,"duration_ms":58535,"temperature":0.7,"pith_summary":"The paper introduces WikiDYK, a benchmark of 12,290 expert-curated facts and 77,180 questions built from Wikipedia's Did You Know... entries, and uses continued pre-training to test how well language models internalize that new knowledge. Its central claim is that bidirectional language models memorize injected facts far more reliably than causal language models: with matched span-prediction training, the 770M-parameter Flan-T5 reaches 46.09% reliability match accuracy while the much larger Llama-3.1-8B reaches 16.09%. The paper also proposes a modular ensemble that routes queries to clusters of small bidirectional models serving as knowledge repositories, which improves reliability accuracy by up to 29.1% over the un-ensembled base. If true, the result would refocus knowledge-update methods on bidirectional or hybrid architectures rather than assuming bigger causal LLMs are automatically better at absorbing new facts.","feed_headline":"Small bidirectional models beat 8B LLMs at memorizing new facts","feed_subtitle":"A Wikipedia-DYK benchmark finds small bidirectional models memorize new facts ~3x better than an 8B causal LLM.","key_machinery":"The load-bearing pieces are (1) the WikiDYK benchmark, which turns expert-reviewed, time-stamped DYK facts into five question types; (2) continued pre-training with span prediction (T5-style masked reconstruction) and an exhaustive masking strategy that generates varied masked inputs; and (3) the ensemble pipeline, which clusters facts by semantics or time, trains a separate BiLM per cluster, and uses a scope classifier to route queries to the right repository or defer to the base LLM. The span-prediction objective is what lets small bidirectional models encode both left and right context during injection, and the scope classifier is what lets the ensemble answer in-scope queries without overwriting the LLM's original knowledge.","core_discovery":"On its own terms, the paper establishes that, under continued pre-training on freshly written Wikipedia facts, smaller bidirectional models (Flan-T5/T5 family with span prediction) consistently outperform much larger causal models (Llama, Qwen, Gemma) across reliability, generality, and paraphrase recall, while next-token prediction is a poor injection objective for causal models. The authors argue the advantage is architectural rather than an artifact of training objective because aligning causal models to span prediction still leaves them far behind bidirectional models. A secondary discovery is that knowledge association (portability, multi-hop) improves only slightly for every method, indicating that memorizing a fact does not automatically enable reasoning over it.","pith_inferences":["If the architectural explanation holds, hybrid attention that toggles between bidirectional and causal modes could let a single LLM store facts as efficiently as a BiLM while keeping generation latency low; the paper hints at this but does not test it.","A controlled test from scratch, with identical data, compute, and hyperparameters and only the attention mask differing, would separate the architecture effect from pretraining and instruction-tuning confounds, something the authors acknowledge they could not run.","The clustering and router design suggests a production pattern: keep a library of small, cheap-to-retrain BiLM memory modules per domain or time slice and update only the relevant module when facts change, which could make knowledge updates far cheaper than full retraining.","One extension would be to test whether the BiLM advantage persists for non-English facts or for structured knowledge such as tables and triples, where bidirectional context might matter more or less."],"forward_implications":["Knowledge injection evaluations should report architecture and training objective separately, because NTP-only numbers can severely understate what causal models store.","BiLMs, despite their smaller scale, should be treated as serious candidates for dense knowledge storage, either as standalone updatable models or as plug-in repositories for larger LLMs.","Ensembles of specialized BiLMs with a router offer a practical way to add new knowledge while keeping the base LLM's locality intact.","Multi-hop association remains the bottleneck: injection methods need to target reasoning over injected facts, not just recall.","WikiDYK's automatic extension makes it possible to track knowledge injection performance continuously as new expert-reviewed facts appear."],"supporting_citations":[{"why":"Supplies the T5 span prediction objective used for training the bidirectional models and for aligning causal models to span prediction.","marker":"(Raffel et al., 2020)"},{"why":"The reversal curse motivates why unidirectional training limits relational recall, setting up the comparison with bidirectional models.","marker":"(Berglund et al., 2023)"},{"why":"Prior knowledge injection evaluation showing continued pretraining fails on association queries; the portability dimension builds on it.","marker":"(Xu et al., 2025)"},{"why":"Provides the synthetic QA training recipe that the paper adapts for causal-model knowledge injection.","marker":"(Wang et al., 2025b)"},{"why":"Plug-in specialized language models concept behind the ensemble of BiLMs as knowledge repositories.","marker":"(Feng et al., 2023)"},{"why":"Branch-train-merge modular expert training supports the paper's ensemble and clustering design.","marker":"(Li et al., 2022)"},{"why":"Comparison of fine-tuning versus retrieval for knowledge injection that the paper extends with its RAG results and architectural comparison.","marker":"(Ovadia et al., 2023)"},{"why":"Memory-based model editing at scale, related to the modular memory idea used in the ensemble framework.","marker":"(Mitchell et al., 2022)"}],"fun_headline_variants":["WikiDYK: small bidirectional models outperform 8B LLMs on fact memory","Bidirectional LMs memorize new facts ~3x better than causal LLMs","New benchmark shows bidirectional models win at knowledge injection","Surprise: smaller bidirectional models beat big causal ones at memorizing","WikiDYK: architecture matters more than size for fact memorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that bidirectional architecture is better rests on comparing off-the-shelf models that differ in scale, pretraining data, and instruction tuning; the paper's limitation section concedes it did not pre-train a BiLM and a CLM under identical hyperparameters and data.","fun_headline_variants_meta":{"raw":{"variants":["WikiDYK: small bidirectional models outperform 8B LLMs on fact memory","Bidirectional LMs memorize new facts ~3x better than causal LLMs","New benchmark shows bidirectional models win at knowledge injection","Surprise: smaller bidirectional models beat big causal ones at memorizing","WikiDYK: architecture matters more than size for fact memorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1854,"prompt_tokens":929,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":833}},"tokens_in":545,"tokens_out":925,"duration_ms":8762,"temperature":1.0,"reasoning_tokens":833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:36:12.334535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a causal and a bidirectional model from scratch on identical data and compute, inject the same WikiDYK facts with the same span-prediction objective, and compare reliability match accuracy; if the causal model matches or beats the bidirectional one, the architectural claim is refuted.","supporting_citations":[],"review_version":1}