Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bidirectional LMs memorize injected facts better than causal LLMs.

desk verdict WikiDYK is a solid, reusable benchmark, but the paper's headline claim that bidirectional architectures memorize better is confounded by pretraining and instruction-tuning differences—the authors' own limitation section concedes the key missing control. read the letter →

arxiv 2505.12306 v1 pith:KIQL3NVU submitted 2025-05-18 cs.CL

Yuwei Zhang , Wenhao Yu , Shangbin Feng , Yifan Zhu , Letian Peng , Jayanth Srinivasa , Gaowen Liu , Jingbo Shang This is my paper · ORCID
classification cs.CL
keywords knowledgeinjectionWikiDYKbidirectionallanguagemodelscausalcontinuedpretrainingspanpredictioncatastrophicforgettingmodelensemble
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces WikiDYK, a benchmark of 12,290 expert-curated facts and 77,180 questions built from Wikipedia's Did You Know... entries, and uses continued pre-training to test how well language models internalize that new knowledge. Its central claim is that bidirectional language models memorize injected facts far more reliably than causal language models: with matched span-prediction training, the 770M-parameter Flan-T5 reaches 46.09% reliability match accuracy while the much larger Llama-3.1-8B reaches 16.09%. The paper also proposes a modular ensemble that routes queries to clusters of small bidirectional models serving as knowledge repositories, which improves reliability accuracy by up to 29.1% over the un-ensembled base. If true, the result would refocus knowledge-update methods on bidirectional or hybrid architectures rather than assuming bigger causal LLMs are automatically better at absorbing new facts.

What carries the argument

The load-bearing pieces are (1) the WikiDYK benchmark, which turns expert-reviewed, time-stamped DYK facts into five question types; (2) continued pre-training with span prediction (T5-style masked reconstruction) and an exhaustive masking strategy that generates varied masked inputs; and (3) the ensemble pipeline, which clusters facts by semantics or time, trains a separate BiLM per cluster, and uses a scope classifier to route queries to the right repository or defer to the base LLM. The span-prediction objective is what lets small bidirectional models encode both left and right context during injection, and the scope classifier is what lets the ensemble answer in-scope queries without overwriting the LLM's original knowledge.

What would settle it

Train a causal and a bidirectional model from scratch on identical data and compute, inject the same WikiDYK facts with the same span-prediction objective, and compare reliability match accuracy; if the causal model matches or beats the bidirectional one, the architectural claim is refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that, under continued pre-training on freshly written Wikipedia facts, smaller bidirectional models (Flan-T5/T5 family with span prediction) consistently outperform much larger causal models (Llama, Qwen, Gemma) across reliability, generality, and paraphrase recall, while next-token prediction is a poor injection objective for causal models. The authors argue the advantage is architectural rather than an artifact of training objective because aligning causal models to span prediction still leaves them far behind bidirectional models. A secondary discovery is that knowledge association (portability, multi-hop) improves only slightly for every method, indicating that memorizing a fact does not automatically enable reasoning over it.

Load-bearing premise

The conclusion that bidirectional architecture is better rests on comparing off-the-shelf models that differ in scale, pretraining data, and instruction tuning; the paper's limitation section concedes it did not pre-train a BiLM and a CLM under identical hyperparameters and data.

Editorial extensions

If this is right

  • Knowledge injection evaluations should report architecture and training objective separately, because NTP-only numbers can severely understate what causal models store.
  • BiLMs, despite their smaller scale, should be treated as serious candidates for dense knowledge storage, either as standalone updatable models or as plug-in repositories for larger LLMs.
  • Ensembles of specialized BiLMs with a router offer a practical way to add new knowledge while keeping the base LLM's locality intact.
  • Multi-hop association remains the bottleneck: injection methods need to target reasoning over injected facts, not just recall.
  • WikiDYK's automatic extension makes it possible to track knowledge injection performance continuously as new expert-reviewed facts appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the architectural explanation holds, hybrid attention that toggles between bidirectional and causal modes could let a single LLM store facts as efficiently as a BiLM while keeping generation latency low; the paper hints at this but does not test it.
  • A controlled test from scratch, with identical data, compute, and hyperparameters and only the attention mask differing, would separate the architecture effect from pretraining and instruction-tuning confounds, something the authors acknowledge they could not run.
  • The clustering and router design suggests a production pattern: keep a library of small, cheap-to-retrain BiLM memory modules per domain or time slice and update only the relevant module when facts change, which could make knowledge updates far cheaper than full retraining.
  • One extension would be to test whether the BiLM advantage persists for non-English facts or for structured knowledge such as tables and triples, where bidirectional context might matter more or less.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces WikiDYK, a knowledge injection benchmark built from Wikipedia's human-curated 'Did You Know?' entries, containing 12,290 facts and 77,180 question-answer pairs spanning reliability, generality, paraphrase, portability, and locality questions. The authors evaluate continued pretraining under next-token prediction, synthetic QA, and span prediction objectives on several off-the-shelf CLMs and BiLMs, concluding that bidirectional LMs memorize injected knowledge substantially better than causal LMs (e.g., Flan-T5-770M reaches 46.09 reliability match accuracy vs. 16.09 for Llama-3.1-8B in Table 4). They further propose an ensemble pipeline that routes queries to specialized BiLM knowledge repositories with a scope classifier, reporting up to a 29.1% reliability improvement over the Flan-T5-220M base. The paper also reports static and RAG-augmented baselines, scaling analyses, and ablations of the clustering and routing components.

Significance. If the central comparison were clean, the paper would make a strong and surprising contribution to the knowledge injection literature: a continuously updatable, human-curated benchmark with a multi-dimensional evaluation protocol, plus a practical modular framework for using small bidirectional models as external memory. The dataset construction is thoughtful, and the public release of data and code is a clear strength. However, the headline architectural conclusion is not currently supported by the experiments, because the BiLM and CLM suites differ in scale, pretraining data, and instruction tuning, as the authors' own Limitation section concedes. This makes the paper's main claim a hypothesis to be tested rather than an established result. The benchmark and ensemble framework remain useful even if the architectural claim is softened or reframed as an observation about off-the-shelf checkpoints.

major comments (4)
  1. [§5.2, Table 4; Limitation] The central claim that BiLMs are intrinsically better knowledge memorizers is confounded by uncontrolled model differences. Flan-T5-220M/770M are instruction-tuned T5 models trained with span corruption, while Llama, Qwen, and Gemma are base autoregressive models of different sizes and pretraining corpora; the Limitation section explicitly states that the authors could not pre-train a BiLM and a CLM under the same hyperparameters and data. The observed 46.09 vs. 16.09 reliability gap in Table 4 could therefore reflect instruction tuning, scale, or pretraining data rather than bidirectionality. I recommend either adding controlled comparisons within the same model family (e.g., causal vs. bidirectional attention variants of the same architecture with matched data and scale) or reframing the paper's claim as an observation about off-the-shelf checkpoints rather than an architectural result.
  2. [§4.2, §4.3, Table 4] The claimed objective alignment between CLMs and BiLMs is incomplete. For CLMs, span prediction is implemented with the prompt 'Predict the masked words in the following sentence' and an appended mask token at test time; this is not the native T5 span-corruption objective used by Flan-T5, and Table 4 shows that CLM results are extremely sensitive to objective choice (Llama-3.2-1B: 16.92 QA vs. 3.03 SP on reliability). Thus the comparison does not isolate architecture from training objective. I would like to see the same T5-style masked span formulation applied to both a causal and a bidirectional variant of the same backbone, or at minimum a detailed discussion of why the prompt format cannot disadvantage CLMs.
  3. [§3.2, §4.2, Appendix B] The synthetic-QA training rows may be inflated by evaluation leakage. Training questions are generated by gpt-4.1-mini from the same facts used to generate the evaluation questions (generated by GPT-4o, GPT-4.1, and o3-mini per Appendix B), and the QA-trained CLMs are evaluated on questions derived from the same source facts. If there is surface-form or template overlap between training and evaluation questions, the QA rows overestimate memorization. Please report an overlap analysis (e.g., n-gram overlap or exact match between training and evaluation questions) and quantify how many evaluation questions are identical or near-identical to training questions.
  4. [Appendix C, Table 8] The paper's internal evidence weakens the attribution to bidirectionality. On the same 1,000-fact set, roberta-large (a bidirectional encoder) reaches only 3.00 reliability match accuracy, while t5-v1.1-large reaches 52.00 and Flan-T5-770M reaches 73.00. This large spread within the BiLM category suggests that pretraining objective, model family, and generation setup, rather than bidirectionality per se, drive most of the effect. The paper should analyze this spread explicitly; otherwise the term 'BiLM' is too coarse to support the stated conclusion.
minor comments (5)
  1. [§6] The final paragraph of Section 6 contains a typo: 'uture work' should read 'Future work'.
  2. [Figure 1] The Figure 1 caption contains garbled text ('TheBiLMs: T5/Roberta/…[M]first' and 'TheCLMs: Llama/Qwen/…Sausages') that should be cleaned before publication.
  3. [Throughout] The benchmark name is rendered inconsistently as 'WIKI DYK', 'WikiDYK', and 'WIKIDYK'; please standardize the spelling.
  4. [Table 7] Table 7 lists 'Llama-3.1-1B' while the main text and Table 4 refer to 'Llama-3.2-1B' for the same experimental family; please confirm which model was used and correct the inconsistency.
  5. [Limitation] The Limitation section states 'we prove this assumption empirically,' but an uncontrolled empirical comparison does not constitute proof; I suggest rephrasing to 'provide empirical evidence consistent with' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the benchmark and comparisons are empirical, and the conceded pretraining confound is a validity issue, not a self-referential derivation.

full rationale

The paper is an empirical benchmark paper, not a formal derivation chain. WikiDYK builds test questions from the same DYK facts later used for continued pretraining, but that is the intended memorization setting rather than a definitional equivalence: static models score near chance in Table 3, so the metric is not trivially satisfied by construction. The central BiLM-versus-CLM claim rests on measured reliability scores in Table 4, not on an equation that reduces to an input, and the CLM span-prediction comparison is an attempt to control the objective, albeit imperfectly. The ensemble gain of 29.1% is also a reported measurement, and the routing/classifier component is ablated against perfect-routing baselines in Table 5, so the classifier's own training labels are not being reported as the prediction. The only admitted weakness is the Limitation section: 'limited by the computing resource, we are not able to completely pre-train a BiLM and a CLM under the same set of hyperparameter and data. Instead, we choose the popular pre-trained models for experiments without further controlling the experiments.' That is a real confound: Flan-T5 is instruction-tuned and span-corruption pretrained, and roberta-large, another BiLM, reaches only 3.00 reliability in Table 8. But a confound is not circularity under the rubric: the paper does not import its conclusion through self-citation, does not invoke a uniqueness theorem, and does not rename a fitted quantity as a prediction. The self-citations to MemoryLLM/M+ and Peng et al. are contextual and are not load-bearing for the architecture claim. The derivation, such as it is, is self-contained as an empirical evaluation, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on fitted constants or invented entities. The free parameters are experimental hyperparameters that affect the comparison; the axioms are domain assumptions about benchmark validity and model representativeness.

free parameters (4)
  • upsampling s = 1000 (main); 3000/6000/10000 in analysis
    Number of times each knowledge entry is seen during continued pretraining; controls training amount and affects all results (Section 5.4).
  • span length range = min 1, max 5
    Hyperparameter for the span prediction masking strategy; affects the training examples for BiLMs and the CLM span-prediction baseline (Appendix A).
  • LoRA rank and alpha = rank 32, alpha 16
    Used for models larger than 3B (Llama-2-7B, Llama-3.1-8B, Qwen-7B); influences the parameter-efficient continued pretraining (Section 5.1).
  • learning rates = 2e-5 (full), 2e-4 (LoRA), 3e-4 (Flan-T5-220M), 1e-4 (T5 large)
    Chosen per-model hyperparameters; standard values, not fit to the benchmark.
assumptions (4)
  • domain assumption Continued pretraining on raw text or QA is a representative method for injecting knowledge into model parameters.
    The entire evaluation assumes this training setup reflects practical knowledge injection.
  • domain assumption DYK facts from 2022-2025 are outside the pretraining corpora of the evaluated models.
    The static near-chance reliability scores are used as evidence, but pretraining data for closed or undisclosed models is not verifiable.
  • domain assumption LLM-generated questions from GPT-4o/GPT-4.1/o3-mini accurately operationalize reliability, generality, paraphrase, portability, and locality.
    No human validation of the 77,180 questions is reported; prompt-based generation may introduce biases.
  • domain assumption Substring match and token F1 are adequate measures of answer correctness for open-domain QA.
    Standard metrics but can reward superficial token overlap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection." pith.science (2026). https://pith.science/paper/KIQL3NVU

@misc{pith2026250512306,
  author       = {Pith},
  title        = {Pith review of: Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KIQL3NVU}},
  note         = {Machine review of arXiv:2505.12306}
}
read the original abstract

Despite significant advances in large language models (LLMs), their knowledge memorization capabilities remain underexplored, due to the lack of standardized and high-quality test ground. In this paper, we introduce a novel, real-world and large-scale knowledge injection benchmark that evolves continuously over time without requiring human intervention. Specifically, we propose WikiDYK, which leverages recently-added and human-written facts from Wikipedia's "Did You Know..." entries. These entries are carefully selected by expert Wikipedia editors based on criteria such as verifiability and clarity. Each entry is converted into multiple question-answer pairs spanning diverse task formats from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290 facts and 77,180 questions, which is also seamlessly extensible with future updates from Wikipedia editors. Extensive experiments using continued pre-training reveal a surprising insight: despite their prevalence in modern LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge memorization capabilities compared to Bidirectional Language Models (BiLMs), exhibiting a 23% lower accuracy in terms of reliability. To compensate for the smaller scales of current BiLMs, we introduce a modular collaborative framework utilizing ensembles of BiLMs as external knowledge repositories to integrate with LLMs. Experiment shows that our framework further improves the reliability accuracy by up to 29.1%.

Figures

Figures reproduced from arXiv: 2505.12306 by the authors.

Figure 1
Figure 1. Proposed knowledge injection evaluation workflow. We inject the knowledge from via [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Evaluation results of injecting 1, 000 facts from WIKIDYK. We report performance for two CLMs and two BiLMs. A more comprehen￾sive set of results can be found in Appendix C. hensive comparative analysis of both model ar￾chitectures and training objectives for knowl￾edge injection. Given the observations on the “reverse curse” issue of Causal Language Mod￾els’ (CLMs) (Berglund et al., 2023), we aim to investigate whe… view at source ↗
Figure 3
Figure 3. Effect of number of knowledge injected with number of upsampling [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Effect of number of upsampling with 1000 injected knowledge. dedicated training on the assigned clusters and the cooperation between LLMs and the trained BiLMs. We show further analysis in Section 5.5. Knowledge association shows less improvements. As can be observed i…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A mixture-of-experts LLM trained with reinforcement learning to perform retrieval from its own parametric memory can replace external retrieval in some settings, at lower latency.

  2. Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.

  3. Data Swarms: Optimizable Generation of Synthetic Evaluation Data

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Data Swarms uses particle swarm optimization over data-generator LLM weights to produce synthetic evaluation data that scores higher on five quantitative evaluation objectives than eight baselines.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [1]

    The question should be a n s w e r a b l e using only the i n f o r m a t i o n provided in the fact

    work page

  2. [2]

    The answer should be the bold _ entity

    work page

  3. [3]

    The question should be clear , natural , and specific so that the answer can be easily i d e n t i f i e d ( i . e . , use as many details as possible from the fact )

    work page

  4. [4]

    question

    The bold entity should not be m ent io ne d in the question since it is the answer . But make sure that the question ’ s answer is the bold entity . Example : Input : {{ ’ text ’: ’ that Margrit Waltz has ferried planes to points on five c o n t i n e n t s ? ’ , ’ bold _ entity ’: ’ Margrit Waltz ’ , }} Expected output : {{ " question ": {{ " text ": " W...

    work page

  5. [5]

    a l t e r n a t i v e s

    Your output should be a JSON object with a list of d i c t i o n a r i e s under the key " a l t e r n a t i v e s ". Each sub - d i c t i o n a r y should have a " question " key and an " answer " key . Here is the pair of question and answer : Fact : { fact } Question : { question } Answer : { answer } B.4 Portability Question Generation Prompt To gener...

    work page

  6. [6]

    The answer is one aspect of the fact ( such as an entity / year / number etc .) apart from the original answer

    work page

  7. [7]

    And it shoud be a part of the fact

    The answer should be concise and direct without any re du nda nt words . And it shoud be a part of the fact

    work page

  8. [8]

    The question should utilize all the i n f o r m a t i o n in the fact and be specific

    work page

Show all 14 references
  1. [9]

    Do not use any i n f o r m a t i o n that is beyond the fact

  2. [11]

    question

    " question ": The question that c o r r e s p o n d s to the s tat em en t

  3. [12]

    answer

    " answer ": The answer to the question Example outputs :

  4. [13]

    question

    Input : Jupiter is the largest planet in our solar system . Output : {{ " question ": " What is the largest planet in our solar system ?" , " answer ": " Jupiter " }}

  5. [14]

    question

    Input : The capital of France is Paris . Output : {{ " question ": " What is the capital of France ?" , 15 " answer ": " Paris " }} Entity : { entity } D e s c r i p t i o n : { d e s c r i p t i o n } B.6 Training QA Generation Prompt We use gpt-4.1-mini for training QA gener...

    2024

  6. [2024]

    Wang, Y ., Krotov, D., Hu, Y ., Gao, Y ., Zhou, W., McAuley, J., Gutfreund, D., Feris, R., and He, Z

    OpenReview.net. Wang, Y ., Krotov, D., Hu, Y ., Gao, Y ., Zhou, W., McAuley, J., Gutfreund, D., Feris, R., and He, Z. (2025a). M+: Extending memoryllm with scalable long-term memory. Wang, Y ., Liu, X., Chen, X., O’Brien, S., Wu, J., and McAuley, J. (2025b). Self-updatable lar...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.