{"id":"c9e36ef7-539d-4d95-b7df-72278fcc36bc","arxiv_id":"1908.07690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Latent Relation Language Models marginalize over hidden relation-to-span assignments to condition neural language models on knowledge graphs, improving perplexity and providing posterior entity-link scores.","lead":"This paper introduces a language model that can generate text either word by word or by copying multi-word phrases from a knowledge graph, and it learns which phrases to copy without explicit annotations. It reports lower perplexity than earlier knowledge-enhanced models on Wikipedia data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No quantitative evidence that LRLM ignores distant-supervision false positives; posterior span annotations and the semantic reading of the perplexity gains are therefore unestablished.","rationale":"The reader identified the distant-supervision span set as the weakest assumption; I agree and sharpen it. The key mechanism is that the marginal-likelihood objective can only gain probability mass from false-positive matches, so there is no optimization pressure to assign them low posterior probability. Thus the unresolved question is not merely whether the relation module 'can help' the LM, but whether the learned latent relations correspond to real entity mentions. That directly targets the paper's joint-distribution-over-entities claim and its stated entity-link application, while leaving the separate empirical question of perplexity intact. The NKLM pruning issue noted in the reader's rationale is also real and worth a baseline rerun, but it affects only the magnitude of the improvement over one baseline; the posterior-annotation concern affects the interpretation of the central model. This is why I would keep the reader's CONDITIONAL verdict rather than accept the paper as is. A single precision/recall measurement on 200 gold-annotated articles would settle the concern in either direction.","tokens_in":16139,"tokens_out":19757,"duration_ms":184448,"concrete_test":"Select 200 held-out WikiText dev/test articles and obtain gold entity mentions from Wikipedia hyperlinks or manual annotation. Run the trained LRLM (or retrain from the published configuration) and extract the highest-posterior relation span for each article text. Compute precision, recall, and F1 of these posterior spans against the gold mentions, and compare with the precision/recall of the raw string-match candidate set used in Section 4.2. If posterior precision is not clearly above the raw candidate set (or if recall is much lower), the model has not learned to ignore false positives and the entity-link claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 constructs the latent span set by string-matching KG surface forms and states: 'We rely on our model's ability to ignore such mentions by learning to assign high probabilities only on the correct mentions.' No quantitative support is provided. Because Equation (1) marginalizes over all valid segmentations, every false-positive surface-form match that occurs in the training text contributes an extra term to log P(X); the training objective contains no term that penalizes assigning high probability to a false relation path. The model can therefore raise likelihood by copying strings such as 'United States' or 'New York' through a relation in contexts where the corresponding entity is not actually mentioned. The only quality evidence offered is qualitative (Table 3 on sampled outputs, Table 4 posterior examples); Table 3 counts valid/invalid mentions in generated samples, not posterior precision/recall over real text. As a result, the paper's secondary but prominent claim that posterior span probabilities serve as entity-link annotations is unsupported, and the perplexity improvement cannot yet be attributed to knowledge-graph-aware generation rather than to a generic multi-word copy shortcut. This is load-bearing because the contribution is not merely a lower perplexity number but a joint latent-variable model of words and entities.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Latent Relation Language Models (LRLMs), which augment a word-level language model with span-level generation from a knowledge graph. A latent variable sequence selects, at each step, whether the next token is a vocabulary word or a surface form of an object related to a topic entity; training marginalizes over all valid segmentations with a forward-backward algorithm. The authors construct WikiText-S and WikiText-F by matching Wikidata surface forms against WikiText-103 articles, and report open-vocabulary perplexity on WikiFacts and these datasets with LSTM and Transformer-XL backbones. They further present posterior span probabilities as entity-link annotations and show qualitative examples.","tokens_in":16416,"tokens_out":7558,"duration_ms":163127,"significance":"The model formulation is clean: the factorization in Section 3.2 and the forward recursion in Section 3.3 make latent-variable marginalization tractable, and the framework is agnostic to the underlying sequence encoder. If the reported gains are due to KG-aware relation modeling, this would be a useful way to condition language models on structured knowledge and to obtain soft entity-link annotations. The open-vocabulary evaluation and the inclusion of two backbone architectures strengthen the empirical study. However, the paper does not yet provide quantitative evidence that the model distinguishes true from false-positive relation mentions, and the comparison to NKLM involves a span-pruning preprocessing step whose effect is not quantified; these gaps must be closed before the semantic interpretation of the perplexity gains is justified.","major_comments":[{"comment":"The paper relies on the model's ability to ignore false-positive distant-supervision spans, but this is never measured. Because Eq. (1) marginalizes over all valid segmentations, any string-match false positive ('United States' as an <origin> mention in a non-entity context) contributes an additional term to the training objective; nothing in the objective penalizes assigning high probability to a relation path for such a span. The perplexity gains in Table 2 could therefore come from a generic multi-word copy mechanism rather than from knowledge-graph relation semantics, and the posterior entity-link claim in Section 6.3 remains unsupported: Table 4 shows two favorable examples, and Table 3 counts mention validity in generated samples rather than posterior precision/recall over real text. I ask for (i) posterior precision/recall against gold entity annotations on a held-out sample, and (ii) a control experiment that replaces the relation embeddings or surface forms with random or shuffled versions; if perplexity gains persist, the mechanism is copying, not relation modeling.","section":"Section 4.2 and Eq. (1)"},{"comment":"The NKLM comparison on WikiText is conducted after a greedy span-pruning pass that removes all overlaps, preferring longer spans, earlier starts, and canonical surface forms. Because LRLM is specifically designed to marginalize over overlapping spans, this preprocessing may remove information that NKLM could have used, while LRLM still sees all matches. The paper does not report how many gold mentions are pruned or how NKLM performs under alternative pruning orders. Please provide that analysis or an ablation; otherwise the claim that LRLM outperforms NKLM on WikiText is not fully controlled.","section":"Appendix D"},{"comment":"The displayed formula for the posterior is not well-defined as written. In Section 3.3, alpha_i is a scalar forward marginal probability summing over all latent paths ending at token i; the term P(Z | x_{<l_i}) used in the posterior formula is therefore not a quantity that has been defined, and the decomposition P(X,Z) = alpha_i * P(Z | x_{<l_i}) * beta_i does not correspond to the forward-backward recursion described earlier. Please define the forward and backward variables with their precise conditioning and derive the posterior of a single span tuple (pi, sigma, rho) by summing over all compatible previous and future latent variables; otherwise the posterior examples in Table 4 are not reproducible.","section":"Section 6.3"}],"minor_comments":[{"comment":"The blanket statement that LRLM 'out-performs the baselines on all datasets' should be qualified, because the LSTM+WikiText-S row shows a small improvement over NKLM that the authors themselves report as not statistically significant; please add confidence intervals or exact p-values for all rows.","section":"Section 6.1 and Table 2"},{"comment":"The column 'Ment/Doc' is not defined; it should state whether these are gold entity mentions or surface-form matches produced by the linking procedure, since the two are very different quantities.","section":"Table 1"},{"comment":"The paper would benefit from a statement on whether code and the constructed WikiText-S/WikiText-F datasets will be released, because the matching and pruning pipelines contain many choices that are difficult to reproduce exactly from the text alone.","section":"Section 5.1 and Appendix B"},{"comment":"The notation Z is overloaded: it denotes the full latent variable sequence in Eq. (1), but in the posterior formula it appears to denote a single span tuple; please use distinct notation for these two objects.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The central derivation is sound and the paper is a reasonable fit for the journal, but the empirical claims currently outrun the evidence: the false-positive-span issue and the NKLM preprocessing issue both require new experiments or analyses, not just copy-editing. I would also encourage the editor to require code/data release, as the preprocessing choices are extensive and the comparison is delicate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid paper with a real contribution, but the entity-linking claim overstates what they actually show.\n\nThe core idea is to extend Latent Predictor Networks to language modeling conditioned on a KG subgraph, letting the model copy multi-word entity spans through relations. They marginalize over all segmentations, including overlapping spans, using forward-backward. That's a clean and correct extension, and it's nice that they demonstrate it with both an LSTM and Transformer-XL. The open-domain WikiText-F dataset is also a useful addition. The perplexity improvements over both vanilla LM and NKLM are mostly significant (Wilcoxon), with the notable exception of the LSTM on WikiText-S, which they acknowledge.\n\nThe soft spots: the stress-test note is on target. Section 4.2 says they rely on the model's ability to ignore false-positive mentions from string-matching, but they never actually measure whether the posterior over relations is correct. The qualitative posterior table is suggestive, but it is a few hand-picked examples. If the model is just using relation copying as a generic multi-word shortcut, the posterior annotations are not meaningful as entity links. So the abstract's claim about annotating posterior probability of entity spans is not established. This doesn't kill the perplexity result, but it changes what the contribution is.\n\nThe other issue is the NKLM comparison. Appendix D prunes overlapping spans greedily for NKLM, which likely handicaps it. That's a reasonable workaround given NKLM's constraints, but it makes the comparison less clean. Also, no code or data is released, so reproducibility is limited.\n\nMath and citations look fine. The factorization is standard and they acknowledge LPN and NKLM properly. This is a paper worth reading for anyone interested in knowledge-augmented LMs. The weaknesses are concrete and fixable: release code, add a posterior precision/recall evaluation on a small annotated sample, and be more careful about the NKLM baseline. It deserves a serious referee.","headline":"A clean and tractable latent-variable LM over KG relations with real but uneven gains; the posterior entity-link claim is not yet supported, and the NKLM comparison has a soft spot.","tokens_in":16894,"tokens_out":3162,"would_cite":true,"duration_ms":33655,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language model that copies entity spans from knowledge graphs through latent relations beats word-only baselines.","keywords":["latent relation language models","knowledge graph conditioned language modeling","open-vocabulary perplexity","latent variable marginalization","entity linking","span-based generation","forward-backward algorithm","distant supervision"],"falsifier":"Take a test set with hand-annotated entity mentions and compare LRLM's perplexity and posterior span accuracy when the candidate span set comes from the gold annotations versus from pure string matching; if the two perform identically, the distant-supervision linker's recall is not the driver of the reported gains, and if gold spans improve results, the matching noise is the limiting factor.","tokens_in":15958,"feed_emoji":"🔗","tokens_out":10334,"duration_ms":269469,"temperature":0.7,"pith_summary":"Latent Relation Language Models (LRLMs) are conditional language models that generate a document as a sequence of spans, each produced either from a fixed word vocabulary or by copying a surface form of an entity connected to the topic by a knowledge-graph relation. Because the text does not reveal which spans are entities or which relations are being expressed, the model treats the source choice, span boundaries, and relation as latent variables and marginalizes over all valid segmentations with the forward-backward algorithm. The paper reports that LRLM achieves lower open-vocabulary perplexity—the standard prediction-error score for language models, lower being better—than both a vanilla word-level language model and a previous hard-decision knowledge-graph language model on all three datasets tested, with either LSTM or Transformer-XL backbones. This matters because rare, topically important words and factual phrases are exactly where ordinary language models struggle, and the same machinery yields posterior probabilities over relations that can act as entity-link annotations.","feed_headline":"Hidden entity copying beats word-only language models","feed_subtitle":"A latent language model lowers open-vocabulary perplexity by copying entity spans from a knowledge graph.","key_machinery":"The central object is the latent variable sequence $Z = \\{(\\pi_t, \\sigma_t, \\rho_t)\\}$, where $\\pi_t$ is the source choice (relation or word), $\\sigma_t=(\\ell_t,r_t)$ is the span of tokens generated at step $t$, and $\\rho_t=(e_t,a_t)$ names the relation and surface form when the source is relation. A valid sequence is exactly a segmentation of the text in which every relation span equals a surface form of an object connected to the topic entity $s$ in the knowledge-graph subgraph. The model defines $P(X\\mid G',s)=\\sum_{Z} P(X,Z\\mid G',s)$, factorizes each step as source probability times span probability, and sums the lattice of segmentations with the forward-backward algorithm. This machinery is what lets multi-word entity copying, overlapping candidate spans, and posterior entity-link scores all emerge from a single training objective.","core_discovery":"The central claim is that marginalizing over latent relation spans, rather than making hard decisions about which phrases are entities, is what lets a language model actually exploit a knowledge graph. The model factorizes each step as a choice between generating a word and generating a relation span; a relation span is valid only if it matches one of the surface forms of an object connected to the topic entity in the graph, and overlapping candidate spans are allowed because they are resolved by marginalization. Training maximizes the total probability of the text summed over every valid latent segmentation, which the forward-backward algorithm makes tractable. The experiments show LRLM reaching lower open-vocabulary perplexity than the vanilla LM and the hard-copy NKLM baseline on WikiFacts, WikiText-S, and WikiText-F with LSTM and Transformer-XL backbones (with one LSTM comparison not reaching statistical significance), and qualitative examples show posterior relation probabilities that track the content of the surrounding text.","pith_inferences":["The posterior span probabilities are shown qualitatively but not measured against gold entity-link labels; a precision/recall evaluation on an annotated set would tell whether the by-product is ready for use in information extraction.","The same latent-span formulation should transfer to other structured contexts—tables, databases, program schemas—wherever candidate surface forms can be enumerated and matched against text.","Since the distant-supervision linker is the entry point for all knowledge, replacing string matching with learned mention detection or fuzzy matching is the most direct lever for improving both coverage and robustness to false positives."],"forward_implications":["Conditioning on a knowledge graph through latent relation spans lowers open-vocabulary perplexity compared with a word-only language model and with a hard-decision copy model, and the gain grows as the subgraph has more relations.","The trained model can annotate an existing text with posterior probabilities for relation-generated spans, providing entity-link-style information without separate supervision.","Because the span machinery sits on top of any sequence model, the same method works with LSTM and Transformer-XL backbones.","Sampling from LRLM produces full entity mentions rather than the partial or broken mentions that word-by-word copy models produce."],"supporting_citations":[{"why":"Introduces the Neural Knowledge Language Model baseline and the WikiFacts dataset that LRLM is compared against.","marker":"Ahn et al., 2016"},{"why":"Provides the latent predictor network factorization into source selection and span generation that LRLM adapts.","marker":"Ling et al., 2016"},{"why":"Gives the forward-backward algorithm used to marginalize over all latent span segmentations during training.","marker":"Baum et al., 1970"},{"why":"Supplies WikiText-103, the corpus from which the WikiText-S and WikiText-F datasets are built.","marker":"Merity et al., 2017b"},{"why":"Supplies the knowledge graph whose relations and surface forms define the candidate relation spans.","marker":"Vrandečić and Krötzsch, 2014"},{"why":"Establishes distant supervision, the paradigm whose noisy string matching generates the candidate mentions the model must learn to ignore.","marker":"Mintz et al., 2009"},{"why":"Provides the Transformer-XL sequence model used as one backbone in the experiments.","marker":"Dai et al., 2019"},{"why":"Supplies the surface-form embeddings used in the surface-form selection step.","marker":"Bojanowski et al., 2017"}],"fun_headline_variants":["Latent relation spans beat hard entity copying in LMs","Marginalizing over entity spans improves language models","Soft entity copying via latent relations outperforms hard copying","Language models improve with latent knowledge graph relations","LRLMs lower perplexity by marginalizing over entity spans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that string-matching the knowledge graph's surface forms against the text yields a candidate span set that covers the entity mentions the model needs, while the false positives from this distant supervision are rare or noisy enough for training to learn to ignore.","fun_headline_variants_meta":{"raw":{"variants":["Latent relation spans beat hard entity copying in LMs","Marginalizing over entity spans improves language models","Soft entity copying via latent relations outperforms hard copying","Language models improve with latent knowledge graph relations","LRLMs lower perplexity by marginalizing over entity spans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1574,"prompt_tokens":812,"completion_tokens":762,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":686}},"tokens_in":428,"tokens_out":762,"duration_ms":7671,"temperature":1.0,"reasoning_tokens":686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:59:38.948879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test set with hand-annotated entity mentions and compare LRLM's perplexity and posterior span accuracy when the candidate span set comes from the gold annotations versus from pure string matching; if the two perform identically, the distant-supervision linker's recall is not the driver of the reported gains, and if gold spans improve results, the matching noise is the limiting factor.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes distant supervision, the paradigm whose noisy string matching generates the candidate mentions the model must learn to ignore."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Transformer-XL sequence model used as one backbone in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the surface-form embeddings used in the surface-form selection step."}],"review_version":1}