{"id":"039bdbd1-11e6-4728-9044-f3d77045e650","arxiv_id":"2509.00949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Language modeling objectives induce recoverable structure in both graph and text models, and periodically resetting embeddings improves their plasticity; together these two forces unify structured and unstructured knowledge engines.","lead":"This PhD thesis argues that knowledge graph models and large language models are governed by the same two forces: structure, built by language-modeling objectives, and destructure, the periodic resetting of embeddings that improves adaptation to unseen symbols. It is worth reading as a clean synthesis of the author's previously published work on relation prediction, n-gram interpretability, ReFactor GNNs, and active forgetting, unified under a single recipe.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Part II's causal explanation for active forgetting—resets improve generalization by truncating embeddings' cached infinite message-passing—is not isolated from confounds like capacity, noise, and node features; an ablation that removes only the reset is needed.","rationale":"The paper's central new claim is that 'destructure'—operationalized as periodic embedding resets—is a complementary force that improves generalization to unseen scenarios. The reader's weakest assumption correctly identifies that the causal mechanism behind this claim is not established. I agree: the formal identity in Chapter 4 is exact, but it only shows that GD updates can be viewed as message passing. The leap from that identity to 'embeddings cache infinite message-passing, and resetting them truncates that cache to improve generalization' requires causal identification. The presented experiments compare full architectures that differ along multiple dimensions, so the specific contribution of the reset is confounded. This does not undermine the published empirical results or the plausibility of the synthesis; it does mean the unifying 'destructure' narrative goes beyond what the evidence currently establishes. The thesis is honest about several limitations, and the underlying chapters are peer-reviewed with reproducible code, so I do not see grounds for rejection. A conditional verdict remains appropriate, pending the isolation ablation described.","tokens_in":51073,"tokens_out":3499,"duration_ms":48943,"concrete_test":"Ablation: take the ReFactor GNN exactly as in §4.5 trained on FB15k237_v1, including textual node features and finite L layers, but set the reset interval K to infinity (no periodic cache reset) while keeping learning rate, epochs, and all other hyperparameters identical; measure inductive test MRR on FB15k237_v1_ind. If the no-reset version matches the reported inductive MRR, then the active-forgetting reset is not the causal lever and the 'excessive cache' explanation is unsupported. A complementary check is to sweep K from small to infinity and plot inductive MRR: flat or non-monotonic dependence would further undercut the truncation narrative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Ch4's formal contribution (Eq. 4.7–4.9) establishes that gradient updates on FM entity embeddings can be rewritten as message passing. The load-bearing step is the further claim that training to convergence caches 'infinite rounds' of message passing and that periodic reset (active forgetting) improves inductive generalization by truncating this cache (Ch4 intro, §4.4; Ch5 §5.4). This step is underdetermined. ReFactor GNNs change several things at once relative to plain FMs: finite message-passing layers, input node features, layer-wise training, a global normalizer, and the external cache reset (Ch4 §4.5, Fig 4.3). The inductive gains in Fig 4.4 are measured against FMs and other GNNs, not against a no-reset ReFactor with identical features and capacity. Likewise, Ch5 compares forgetting pretraining against standard pretraining, but resetting token embeddings every K updates is a strong stochastic/regularization intervention; it may simply prevent overfitting, lower effective capacity, or act as beneficial noise, without vindicating the cache-truncation story. The thesis itself honestly flags theoretical scope (§6.2.1) but does not report an identification experiment that removes the reset alone. Unless such an ablation exists, the 'destructure' mechanism is a plausible interpretation, not an established cause.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This PhD thesis proposes a unified account of two paradigms for building knowledge engines: the structured paradigm (knowledge graphs, factorization models/GNNs) and the unstructured paradigm (transformer LMs). The claimed connection is two complementary forces: 'structure'—language-modelling objectives induce recoverable symbolic structure in both paradigms—and 'destructure'—periodically resetting embedding weights ('active forgetting') improves plasticity and generalization to unseen entities/predicates/languages. Part I (Chs 2–3) presents experiments: adding relation prediction to 1vsAll training improves KGC on several benchmarks; residual-stream decomposition of transformers yields n-gram 'databases' used for interpretability case studies. Part II (Chs 4–5) reinterprets GD on FM embeddings as message passing, introduces ReFactor GNNs for inductive KGC with periodic node-state cache resets, and applies periodic token-embedding resets during PLM pretraining to improve low-resource cross-lingual adaptation. The thesis concludes by proposing a 'structure/destructure recipe' for general knowledge engines.","tokens_in":51314,"tokens_out":5995,"duration_ms":74172,"significance":"If the central causal mechanism were established, this would be a valuable conceptual bridge: it identifies a shared architectural abstraction (the 'embedding sandwich'), gives a formal rewriting of FM training as message passing, and connects two usually separate literatures. Strengths include benchmark-backed experimental chapters (FB15k-237, WN18RR, CoDEx, XNLI, MLQA, XQuAD), statistical significance tests in Ch2, open-source code for Ch2, and generated predictions (forgetting improves low-resource adaptation) that are in principle falsifiable. However, the 'destructure' force is currently a plausible interpretation rather than an established cause: the resets are not isolated from confounds, and the integrated recipe is not validated on a fresh benchmark. The thesis's own limitations section (§6.2.1) acknowledges the theoretical scope, but the missing identification experiment is central, not peripheral.","major_comments":[{"comment":"The CoDEx-S/M/L rows reproduce, exactly, the WN18RR/FB15k-237/Aristo-v4 rows from Table 2.3 (e.g., CoDEx-S '0.487/0.441/0.501/0.580' equals WN18RR 'EP=on, RP=off'). Since CoDEx has different statistics, this is not plausible unless the table is mislabeled. Please correct and re-run/verify; as printed, the RQ1 claim 'relation prediction consistently improves MRR and Hits@1 across CoDEx' is unsupported by the reported data.","section":"Ch2, Table 2.4"},{"comment":"The formal rewriting (Eq. 4.7–4.9) shows GD updates on embeddings can be expressed as message passing. The load-bearing causal step—that converged FMs cache infinite rounds and that the periodic external cache reset improves generalization by truncating this cache—is not identified. ReFactor GNNs change several factors at once: finite L, input node features, layer-wise training, global normalizer, and reset. The inductive gains in Fig 4.4 are measured against FMs/GNNs, not against a no-reset ReFactor with identical features/capacity. The thesis should add this ablation; otherwise the performance gain could be due to reduced capacity, added noise, or node features. §6.2.1 flags scope but does not supply the experiment.","section":"§4.4–§4.5, Fig 4.4"},{"comment":"The pretraining comparison in Ch5 is standard vs forgetting PLMs. Since resetting token embeddings every K updates is a strong stochastic/regularization intervention, the observed gains in XNLI/MLQA/XQuAD do not by themselves establish the cache-truncation mechanism. Required controls include a single-reset baseline, injecting equivalent noise without resets, varying embedding capacity, and comparing reset of embeddings vs reset of other layers, plus an ablation of K. Without these, 'destructure' remains a post-hoc interpretation, not a mechanism.","section":"§5.4–§5.5, Algorithm 5"},{"comment":"The abstract and conclusion present 'a new recipe for developing general knowledge engines,' but the recipe is assembled from the author's own published experiments and is not held out against any fresh, integrated validation. The framework is therefore partly post-hoc. Either add a prospective validation of the combined structure+destructure recipe or explicitly re-frame it as an interpretive synthesis rather than a predictive recipe.","section":"§1.4, §6.3"}],"minor_comments":[{"comment":"The caption says 'Hits@1 versus embedding size' but the body text refers to 'MRR'. Please align the axis label and the description.","section":"Figure 2.2"},{"comment":"The notation is inconsistent: Eq. 4.4 defines relation embeddings as ψ[r], but Eq. 4.5 uses g(r). Define g consistently with the encoder notation in Eq. 4.2.","section":"Eq. 4.5"},{"comment":"Typo: 'fundmentally' should be 'fundamentally'.","section":"Impact Statement"},{"comment":"The repeated caption 'Accuracy comparison of forgetting and standard PLMs on XQuAD (table continued)' is redundant; one caption with a note that the table spans pages is sufficient.","section":"Tables 5.4–5.5"}],"recommendation":"major_revision","confidential_remarks":"The duplicate CoDEx table (Table 2.4) is a significant data-integrity concern and should be verified before any acceptance. The main substantive risk is the causal mechanism in Part II: the 'active forgetting' benefit is not isolated from confounds. If the author can add the missing no-reset ablation and adjust the framing, the thesis could be defensible; in current form the central 'destructure' claim is underdetermined."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a PhD thesis that re-packages four solid peer-reviewed papers inside a new conceptual framing—structure versus destructure. That framing is the genuinely new contribution, and it's worth engaging with, but the mechanism at its heart is not as established as the prose suggests.\n\nWhat's actually new is the synthesis: language modeling objectives induce recoverable structure in both knowledge-graph models and LLMs, and periodic embedding resets improve plasticity for unseen predicates and languages. The individual chapters are prior work, but they're good prior work—standard benchmarks, significance tests in Chapter 2, ablations in 4 and 5, and the code is public. Chapter 4's rewriting of gradient descent over embeddings as message-passing is a legitimate and useful perspective, and Chapter 3's data-free n-gram extraction has real value for auditing LLMs.\n\nThe soft spot is the causal claim driving Part II. The thesis says FMs cache infinite rounds of implicit message-passing when trained to convergence, and that active forgetting helps because it truncates that cache. That story is plausible, but it's not isolated from simpler confounds. ReFactor GNNs change several things at once relative to plain FMs—finite layers, node features, layer-wise training, a global normalizer, and the cache reset. The inductive gains are measured against FMs and other GNNs, not against a no-reset ReFactor with identical features and capacity. Similarly, resetting token embeddings every K updates is a strong regularizer; it may just prevent overfitting or lower effective capacity. The thesis itself honestly flags theoretical scope in Section 6.2.1, but I didn't find an identification experiment that removes only the reset. That's a missing support for the headline mechanism.\n\nAlso, the 'recipe' is built from the same experiments used to construct it, so the synthesis itself remains unfalsified. Some implementation details (K, sparsification threshold, toxic list) are under-specified.\n\nDespite that, the work is serious and honest. The framing will be useful to anyone working on embedding-level interventions or on bridging structured and unstructured knowledge. It deserves a real referee, but a referee who will demand the reset-only ablation before accepting the causal story.","headline":"A well-written thesis whose synthesis is plausible but whose causal story on active forgetting needs an ablation that removes only the reset.","tokens_in":51915,"tokens_out":2517,"would_cite":true,"duration_ms":30740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This thesis claims that knowledge-graph models and large language models are driven by the same two forces—language modelling builds structure into model weights, and periodic embedding resets restore the plasticity needed to generalize to","keywords":["knowledge engine","structure formation","active forgetting","embedding reset","knowledge graph completion","language model interpretability","model plasticity","cross-lingual transfer"],"falsifier":"Take the same forgetting pretraining recipe but replace each periodic embedding reset with additive Gaussian noise of matched magnitude at the same intervals, holding everything else fixed. If the inductive graph and cross-lingual gains match the reset condition, the cache-truncation explanation is not necessary; if the gains disappear only when resets are applied to the embedding cache, the mechanism is supported.","tokens_in":50827,"feed_emoji":"🔁","tokens_out":6110,"duration_ms":74827,"temperature":0.7,"pith_summary":"The thesis seeks to unify the structured knowledge-graph paradigm with the unstructured large-language-model paradigm by showing both are driven by the same two forces. Structure: language-modelling objectives cause models to encode recoverable relational patterns, whether the input is triples or raw text. Destructure: periodically resetting the embedding layer during training improves adaptation to unseen predicates and languages. If correct, the two paradigms share a common design recipe for knowledge engines that are simultaneously interpretable and adaptable.","feed_headline":"Structure builds, forgetting frees: one recipe for knowledge engines","feed_subtitle":"A thesis unites graph models and LLMs: language modeling builds structure, embedding resets build plasticity.","key_machinery":"The unifying object is the embedding sandwich: any model with an input embedding layer, a central body, and an output unembedding layer. The load-bearing identity is the exact rewriting of gradient descent over entity embeddings as message-passing operations; this turns embeddings into caches of symbol-interaction messages. The other central mechanism is active forgetting—periodically resetting the embedding layer—which truncates that cache. For transformers, the corresponding object is the 2^L path decomposition of recursive residual networks, which turns the network into an ensemble of factorization models whose token-interaction databases can be read out as n-grams.","core_discovery":"The central claim is that structure and destructure are complementary forces in making knowledge engines. Part I shows that a masked-language-modelling-style objective—predicting any symbol from its surrounding symbols—induces structure in both paradigms: adding relation prediction to standard entity-prediction training improves knowledge graph completion, especially on highly multi-relational graphs, and a transformer's residual computation can be decomposed into many input-to-output paths, each behaving like a factorization model, whose n-gram databases expose interpretable structure inside large language models. Part II argues that gradient descent on entity embeddings can be rewritten ex","pith_inferences":["Beyond the paper: if the cache-truncation account is right, the same periodic-reset trick should generalize to other over-trained embedding layers in continual learning or model editing, with resets scheduled by how long a symbol class has been cached rather than by a fixed interval.","Beyond the paper: the n-gram extraction method could serve as a deployment-time audit tool—for example, checking whether fine-tuning or alignment edits actually remove undesirable knowledge, since the paper's own toxicity case study shows surface detoxification can leave embedded bigram mass almost unchanged.","Beyond the paper: the structure/destructure dichotomy suggests a testable scaling prediction—the farther a new language or predicate is from the training distribution, the larger the relative benefit of active forgetting; the paper's cross-lingual results are consistent with this but do not establish the scaling law."],"forward_implications":["Adding relation prediction to standard KBC training—a masked-language-modelling-style objective over triples—improves link prediction, especially on datasets with many relation types.","A transformer's residual computation can be unrolled into many input–output paths, each resembling a factorization model; the resulting n-gram tables support data-free analysis of LLM internals, pretraining dynamics, and fine-tuning effects.","Reinterpreting gradient descent on entity embeddings as message passing shows factorization models are a special kind of GNN with infinite layers, a global normalizer, and layer-wise training; truncating this with finite layers and node features yields ReFactor GNNs, which keep transductive performance and generalize inductively.","Periodically resetting token embeddings during pretraining makes language models adapt faster and better to low-resource languages, with the largest gains for languages distant from the pretraining language.","Together the results define a recipe: model the seen through structure formation, and model the unseen through active destructuring of the learned cache."],"supporting_citations":[{"why":"Supplies the language-modelling-as-KBC objective and the experimental results that Chapter 2 builds on.","marker":"[Chen et al., 2021]"},{"why":"Supplies the jet/path-decomposition basis for extracting n-gram structures from transformers in Chapter 3.","marker":"[Chen et al., 2024]"},{"why":"Supplies the ReFactor GNN architecture and the message-passing reinterpretation of factorization models in Chapter 4.","marker":"[Chen et al., 2022]"},{"why":"Provides the 1vsAll objective and N3 regularization that the training setups in Chapters 2, 4, and 5 rely on.","marker":"[Lacroix et al., 2018]"},{"why":"Provides ComplEx, the base factorization model used in the KBC experiments.","marker":"[Trouillon et al., 2016]"},{"why":"Establishes that training strategy and hyperparameters determine factorization-model performance, motivating the carefully tuned comparisons.","marker":"[Ruffinelli et al., 2020]"},{"why":"Provides the NBFNet inductive KGC baseline and the GraIL/NBFNet results that ReFactor GNNs are measured against.","marker":"[Zhu et al., 2021]"},{"why":"Supplies the masked-language-modeling objective that the triple-level relation-prediction objective is explicitly analogized to.","marker":"[Devlin et al., 2019]"},{"why":"Provides the residual-stream view and early bigram-style analyses that Chapter 3's decomposition extends to non-linear transformers.","marker":"[Elhage et al., 2021]"}],"fun_headline_variants":["Structure and destructure: the twin forces behind AI knowledge","Build structure, reset embeddings: one recipe for general AI","Forgetting frees, structure guides: the AI knowledge recipe","Unify graphs and LLMs: structure plus forgetting"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim that periodic embedding resets help generalization because they stop embeddings from caching excessive message-passing computations over symbols—rather than because the resets add noise, shrink model capacity, or interact with node-feature inputs—is the load-bearing premise; the thesis does not isolate it with an identification experiment.","fun_headline_variants_meta":{"raw":{"variants":["Structure and destructure: the twin forces behind AI knowledge","Build structure, reset embeddings: one recipe for general AI","Forgetting frees, structure guides: the AI knowledge recipe","Unify graphs and LLMs: structure plus forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000493,"raw_usage":{"total_tokens":2204,"prompt_tokens":635,"completion_tokens":1569,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":379,"completion_tokens_details":{"reasoning_tokens":1502}},"tokens_in":379,"tokens_out":1569,"duration_ms":11623,"temperature":1.0,"reasoning_tokens":1502,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:03:38.469844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same forgetting pretraining recipe but replace each periodic embedding reset with additive Gaussian noise of matched magnitude at the same intervals, holding everything else fixed. If the inductive graph and cross-lingual gains match the reset condition, the cache-truncation explanation is not necessary; if the gains disappear only when resets are applied to the embedding cache, the mechanism is supported.","supporting_citations":[],"review_version":1}