{"id":"8165c064-5e6c-4892-bddc-04f31734b355","arxiv_id":"2412.00478","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LENIE generates LLM-augmented node descriptions, sampled via clustering-based triplet selection, and uses them as embeddings to improve node importance estimation in knowledge graphs.","lead":"This paper presents LENIE, a method that uses large language models to rewrite and enrich node descriptions in knowledge graphs, and shows that these richer texts improve the accuracy of graph-based importance estimation models. The key idea is that LLMs supply additional real-world context that incomplete or missing knowledge-graph descriptions lack, which helps models predict node importance scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Risk: LLM-generated descriptions may encode the target popularity labels via pretrained world knowledge (Section III-D, IV-A), so the reported SOTA gains—especially MUSIC10K—could reflect label leakage, not semantic augmentation. Anonymized-entity control is missing.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the LLM's pretrained knowledge may encode the ground-truth importance labels. This is the most serious threat to the central claim because the method's mechanism (Section III-D) is to let the LLM inject external knowledge, and the datasets' labels are themselves popularity/familiarity signals that are abundant in LLM pretraining data. The empirical pattern supports the concern: MUSIC10K, which has no node descriptions and uses artist familiarity as the label, shows the largest relative gains, and the case study confirms the LLM actively injects external world knowledge into generated descriptions. Existing ablations cannot separate legitimate semantic enrichment from target leakage, because they all retain entity names. A paired-significance test or learning-rate protocol clarification would be useful but would not settle whether the central mechanism is real; the anonymized-entity control is the decisive experiment. Therefore, the appropriate verdict remains CONDITIONAL, pending that control, rather than moving to REJECT or ACCEPT.","tokens_in":19839,"tokens_out":4040,"duration_ms":45354,"concrete_test":"Run the MUSIC10K pipeline with all entity names replaced by anonymized IDs (e.g., 'Artist #123') in both the triplet texts and the prompts, preserving the full KG structure, cluster-based sampling count, Llama3.1, text encoder, and RGTN settings. Compare the anonymized LENIE against Vanilla and against the original named LENIE. If the anonymized gains over Vanilla shrink by more than 50% on SPEARMAN or RMSE, the reported improvements are substantially driven by LLM recall of name-linked popularity, not by semantic augmentation. As a complementary check, prompt the LLM to output a standalone popularity/familiarity score for each named entity and compute its Spearman correlation with the ground-truth labels to confirm the leakage channel.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM-generated semantic augmentation, guided by KG triplets, improves NIE performance. The weakest link is target-label leakage through the LLM's parametric memory. Section III-D constructs prompts from real entity names and sampled triplets; Section IV-A defines labels as Wikipedia pageviews (FB15K), TMDB popularity (TMDB5K), and artist familiarity (MUSIC10K). Llama3.1 is pretrained on text containing exactly these popularity and familiarity signals, and its generated descriptions are then embedded (Eq. 10) and fed into the downstream GNN. Table II shows the largest gains on MUSIC10K, where entity names are the only input and the label is artist familiarity (SPEARMAN jumps from 0.459 to 0.655). This is precisely the setting in which the LLM can recall 'popular', 'renowned', or similar rank-correlated content and write it into the augmented descriptions, creating a direct shortcut to the target variable. The ablations in Table III do not rule this out: LENIE(concat) withholds LLM generation, and LENIE(random) vs LENIE(cluster) only varies triplet sampling. The paper's own case study (Section V-F, Scenario 3) shows the LLM injecting external world knowledge to correct a director name, demonstrating that parametric recall is active in the generation step, but no control separates that recall of factual relations from recall of popularity labels. Without an anonymized-entity control or an out-of-distribution test, the reported SOTA improvements cannot be attributed to semantic augmentation rather than label leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LENIE, a framework that uses LLMs to generate augmented textual descriptions of nodes in knowledge graphs for the downstream task of node importance estimation (NIE). LENIE first extracts one-hop triplets for each node, samples a subset using a clustering-based strategy in the embedding space, then builds node-specific prompts that combine the sampled triplets with any existing node description. An LLM (default Llama3.1-8B) generates an augmented description, which is encoded by a sentence transformer and used as the initial node embedding for GNN-based NIE models such as RGTN and LICAP. Experiments on FB15K, TMDB5K, and MUSIC10K report improvements over existing NIE baselines across five metrics, and the paper claims state-of-the-art performance on all three datasets. The paper also includes ablations of the augmentation components, a comparison of different LLMs, and a study applying LENIE to several downstream models.","tokens_in":20109,"tokens_out":2975,"duration_ms":30248,"significance":"If the central claim is sound, the paper makes a useful empirical contribution by showing that LLM-generated semantic augmentation can improve NIE on knowledge graphs, especially for graphs where node descriptions are missing or thin. The release of source code and processed datasets is a concrete strength that supports reproducibility. The paper is also one of the first to combine LLM text generation with GNN-based NIE, and the framing around KG semantic deficiencies (insufficient, missing, or inaccurate descriptions) is clear and actionable. However, the significance of the empirical claim depends on ruling out a leakage mechanism in which the LLM's pretrained world knowledge encodes the same popularity/familiarity signals that serve as training labels; the current experiments do not provide that control.","major_comments":[{"comment":"The main empirical claim is vulnerable to label leakage through the LLM's parametric memory. The prompts in Section III-D use real entity names, and the task labels defined in Section IV-A are Wikipedia pageviews (FB15K), TMDB popularity scores (TMDB5K), and artist familiarity (MUSIC10K). Llama3.1 is pretrained on text that contains exactly these popularity and familiarity signals, so the LLM can write rank-correlated content (e.g., 'popular', 'renowned', 'critically acclaimed') into the augmented descriptions. The largest gains occur on MUSIC10K, where entity names are the only node input and SPEARMAN rises from 0.459 (RGTN) to 0.655 (LENIE), which is precisely the setting in which leakage is most plausible. The paper provides no control experiment, such as anonymizing entity names, replacing them with placeholders, or testing on entities outside the LLM's pretraining knowledge. Without such a control, the reported SOTA improvements cannot be attributed unambiguously to semantic augmentation rather than to the LLM recalling the target variable.","section":"Section III-D, Section IV-A, Table II"},{"comment":"No significance tests or confidence intervals are reported for the headline comparisons, and several differences are small relative to the reported standard deviations. For example, on FB15K the RMSE improvement of LENIE over RGTN is 0.7826±0.0158 versus 0.7926±0.0176, and on OVER@100 LENIE is actually worse (0.498±0.0331 versus 0.502±0.0194). The paper states that LENIE 'achieved the new state-of-the-art performance' without a paired statistical test across the five cross-validation folds. The authors should report paired tests or confidence intervals for each metric, especially for FB15K where the absolute gains are around one percentage point or less.","section":"Section V-A, Table II"},{"comment":"The hyperparameter selection procedure appears to use the same cross-validation folds for both model selection and final performance reporting: the learning rate with the lowest RMSE is chosen and the corresponding results are recorded as final outcomes. This is a form of test-set reuse unless a nested or separate validation split is used. The manuscript should clarify whether the reported numbers are selected on the test folds, and if so, provide an unbiased evaluation protocol or a correction such as nested cross-validation.","section":"Section IV-D, Section V-A"},{"comment":"The ablation study does not fully isolate the source of the improvement. LENIE(concat), which excludes the LLM, is worse than Vanilla RGTN on FB15K (RMSE 0.8213 versus 0.7926) while improving on TMDB5K and MUSIC10K; LENIE(random) and LENIE(cluster) add the LLM, so any gain could come either from the LLM's parametric knowledge or from the richer textual input. On MUSIC10K, LENIE(random) and LENIE(cluster) are statistically indistinguishable on RMSE (0.0855±0.0042 versus 0.0856±0.0035), so the clustering contribution is not clearly supported on the dataset with the largest headline gain. Additional controls—for example, comparing LLM-augmented descriptions against human-written descriptions of comparable length, or against LLM descriptions generated from anonymized triplets—are needed to attribute the improvement to clustering-guided semantic augmentation.","section":"Section V-B, Table III"}],"minor_comments":[{"comment":"The headings 'The performance of LINIE' should read 'LENIE'.","section":"Table III and Table V headings"},{"comment":"There are several typos, including 'sematic deficiencies' in the conclusion and 'descrpitons' in Section V's question list; the manuscript should be proofread.","section":"Abstract and Section VI"},{"comment":"The notation 'LENIE (concate)' in Table III differs from 'LENIE (concat)' in the text; please use consistent naming.","section":"Section III-D and Table III"},{"comment":"The text refers to Figure 3 for the comparison of triplet sampling strategies, but the figure itself is not rendered in the submitted text; please ensure the figure is included and readable.","section":"Section V-A and Figure 3"},{"comment":"The dataset description says 'This dataset lacks node description text, so entity names serve as it,' which is awkward; please rephrase for clarity.","section":"Section IV-A, MUSIC10K"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is the central issue for this paper. Because the contribution is purely empirical, the authors should be required to add a control that decouples the LLM's parametric memory of popularity/familiarity signals from the proposed semantic augmentation mechanism. I would also ask for a clarification of the model-selection protocol, since a nested evaluation is essential for the reported SOTA claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: LENIE is a sensible, well-ablated first attempt to use LLMs for semantic augmentation in node importance estimation. The clustering-based triplet sampling is the most genuinely novel piece. The paper deserves review, but the headline gains on MUSIC10K need an extra control before the SOTA claim is fully safe.\n\nWhat is actually new: this is the first work to bring LLM-generated descriptions into NIE, and the clustering-based triplet sampling strategy is a reasonable mechanism for keeping prompt context diverse. The code and processed datasets are public, which is real evidence and should be credited. The paper also runs a fairly complete ablation chain: vanilla vs. concatenation vs. random sampling vs. clustering, plus three different LLMs and seven downstream models.\n\nWhere the paper does well: the experimental coverage is broad and the ablations mostly support the design choices. Clustering-based sampling usually beats random sampling, and the case study in Section V-F shows genuine semantic repair (correcting a wrong director name). The authors are also transparent that MUSIC10K lacks descriptions and that the LLM contributes extra world knowledge—that framing is honest.\n\nSoft spots, in order of importance. First, the label-leakage worry is real. Llama3.1 knows artist popularity and Wikipedia pageviews, and the largest gains appear exactly where the label is familiarity and the only input is the entity name. The Section V-F case study demonstrates the LLM actively injects parametric knowledge, so there is no reason to think it stops at factual relations and never mentions popularity. An anonymized-entity control, or an out-of-distribution set the LLM cannot know, would separate semantic augmentation from target leakage. Second, no paired significance tests are reported; on FB15K the RMSE gap is 0.010 with fold standard deviations around 0.016, so that headline improvement is within noise. Third, the learning-rate selection protocol is ambiguous—the paper says the rate with lowest RMSE was chosen, but not whether that selection happened on validation folds. If it happened on the same test folds, the reported numbers are optimistically selected.\n\nThese are real issues, but they are fixable and they do not sink the central direction. The qualitative conclusion—LLM augmentation helps, especially when original descriptions are weak—is likely correct. I would send this to peer review, asking for the control, significance tests, and a clarification of the hyperparameter protocol. The reader's conditional verdict is appropriate.","headline":"A solid, reproducible first LLM-augmentation method for NIE, with a real label-leakage question on the music benchmark; deserves peer review with a control experiment.","tokens_in":20700,"tokens_out":2166,"would_cite":true,"duration_ms":24365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that weak, missing, or inaccurate text descriptions in knowledge graphs are a bottleneck for node importance estimation (NIE), and that the bottleneck can be cleared by having a large language model rewrite each node's…","keywords":["node importance estimation","knowledge graphs","large language models","semantic augmentation","graph neural networks","triplet sampling","text embeddings","prompt engineering"],"falsifier":"Re-run LENIE with entity names anonymized — replace 'Dinosaur', '2 Fast 2 Furious', and the artist names with generic placeholders while keeping the same triplets and prompts — and compare downstream NIE accuracy against the named version. If most of the gains on TMDB5K and MUSIC10K disappear, the LLM is recalling label-relevant popularity knowledge rather than providing neutral semantic augmentation.","tokens_in":19605,"feed_emoji":"🤖","tokens_out":5747,"duration_ms":50834,"temperature":0.7,"pith_summary":"This paper claims that weak, missing, or inaccurate text descriptions in knowledge graphs are a bottleneck for node importance estimation (NIE), and that the bottleneck can be cleared by having a large language model rewrite each node's description from a diverse sample of the node's neighboring triplets. The proposed LENIE pipeline clusters the triplet sentences attached to a node, picks one triplet from each cluster so the language model sees varied relations and entity types, and prompts the model to produce a richer description that merges the graph's facts with its own world knowledge. Feeding these augmented descriptions into existing GNN-based importance estimators improves their predictions across three real-world knowledge graphs. For a reader, the point is that semantic augmentation is a transferable performance boost for NIE that does not require changing the downstream architecture.","feed_headline":"LLM-generated node blurbs sharpen node-importance prediction","feed_subtitle":"Enriching sparse knowledge-graph text with LLM knowledge lifts ranking accuracy on three benchmarks.","key_machinery":"The engine of the method is the clustering-based triplet sampler. Each node's one-hop triplets are converted to sentences, embedded with a text encoder, and clustered, and the sentence nearest each cluster center is chosen so that the LLM's prompt covers diverse relation and entity types instead of being dominated by frequent relations. This diverse triplet text is concatenated with the node's original description (when one exists) inside a node-specific adaptive prompt, and the LLM's generated description becomes the node's initial semantic embedding for the downstream GNN. The clustering step is what distinguishes LENIE from random triplet sampling in the ablations.","core_discovery":"LENIE is, to the authors' knowledge, the first method to bring large language models into node importance estimation. Its contribution is a semantic-augmentation layer that sits before any GNN-based estimator: extract all one-hop triplets of a node, cluster the embeddings of the triplet sentences, sample the triplet closest to each cluster center, build a node-specific adaptive prompt that combines the sampled triplets with any existing description, and have the LLM generate an enriched description that initializes the node's embedding. On FB15K, TMDB5K, and MUSIC10K, plugging LENIE into the RGTN and LICAP estimators improves nearly every regression and ranking metric, with the largest gains on MUSIC10K, the dataset that originally lacks node descriptions; the paper reports this as the new state of the art.","pith_inferences":["A testable extension would be an anonymization control: replacing entity names with placeholders before prompting would reveal whether the LLM is contributing neutral semantic context or recalling popularity knowledge encoded during pretraining.","The sampling-and-augmentation recipe is not tied to NIE; the same triplet-diversity prompt could enrich node descriptions for other KG tasks such as link prediction or node classification.","Because the paper uses a fixed 768-dimension sentence-transformer encoder to embed the LLM's outputs, the results suggest that a strong LLM can compensate for a modest embedding backbone, lowering the deployment cost of semantic augmentation."],"forward_implications":["The same augmented descriptions can initialize any of the seven NIE models tested, so the improvement transfers across architectures rather than belonging to one estimator.","Datasets that entirely lack node descriptions, such as MUSIC10K, see the largest gains, making the approach a practical remedy for description-poor knowledge graphs.","Selecting one triplet per semantic cluster keeps prompts short while preserving coverage, which reduces the number of LLM calls and input tokens needed for high-degree nodes.","Because LENIE is a preprocessing layer, improving the underlying LLM should translate directly into better NIE performance without altering the graph-model training procedure."],"supporting_citations":[{"why":"GENI, the first GNN-based NIE method for KGs, establishes the baseline task formulation and aggregation approach LENIE builds on.","marker":"[2]"},{"why":"RGTN is the primary downstream NIE model whose performance LENIE improves and compares against in the main experiments.","marker":"[15]"},{"why":"LICAP is the previous SOTA model, provides the experimental settings, and its contrastive pretraining is shown to combine with LENIE-augmented embeddings.","marker":"[16]"},{"why":"MultiImport is a multi-input-signal NIE baseline that LENIE is evaluated against.","marker":"[12]"},{"why":"Chen et al. supplied the original MUSIC10K dataset, which the paper processes into the evaluation format used in the experiments.","marker":"[17]"}],"fun_headline_variants":["LLM boost for node importance via semantic enrichment","First LLM-powered node importance estimator outperforms baselines","Semantic augmentation with LLMs sharpens node ranking","LLMs inject missing context into knowledge graphs for ranking","Knowledge graph text enrichment lifts node score accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the premise that the LLM's pretrained knowledge of these real-world entities does not already encode the popularity, pageview, or familiarity scores used as training labels, so the generated descriptions add neutral semantics rather than leaking the answers.","fun_headline_variants_meta":{"raw":{"variants":["LLM boost for node importance via semantic enrichment","First LLM-powered node importance estimator outperforms baselines","Semantic augmentation with LLMs sharpens node ranking","LLMs inject missing context into knowledge graphs for ranking","Knowledge graph text enrichment lifts node score accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1383,"prompt_tokens":987,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":322}},"tokens_in":603,"tokens_out":396,"duration_ms":4024,"temperature":1.0,"reasoning_tokens":322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:21:02.683519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run LENIE with entity names anonymized — replace 'Dinosaur', '2 Fast 2 Furious', and the artist names with generic placeholders while keeping the same triplets and prompts — and compare downstream NIE accuracy against the named version. If most of the gains on TMDB5K and MUSIC10K disappear, the LLM is recalling label-relevant popularity knowledge rather than providing neutral semantic augmentation.","supporting_citations":[{"cited_title":"Estimating node importance in knowledge graphs using graph neural networks,","cited_arxiv_id":null,"evidence_quote":"GENI, the first GNN-based NIE method for KGs, establishes the baseline task formulation and aggregation approach LENIE builds on."},{"cited_title":"Representation learning on knowledge graphs for node importance estimation,","cited_arxiv_id":null,"evidence_quote":"RGTN is the primary downstream NIE model whose performance LENIE improves and compares against in the main experiments."},{"cited_title":"Label informed contrastive pretraining for node importance estimation on knowledge graphs,","cited_arxiv_id":null,"evidence_quote":"LICAP is the previous SOTA model, provides the experimental settings, and its contrastive pretraining is shown to combine with LENIE-augmented embeddings."},{"cited_title":"Multiimport: Inferring node importance in a knowledge graph from multiple input signals,","cited_arxiv_id":null,"evidence_quote":"MultiImport is a multi-input-signal NIE baseline that LENIE is evaluated against."},{"cited_title":"Deep structural knowledge exploitation and synergy for estimating node importance value on heterogeneous information networks,","cited_arxiv_id":null,"evidence_quote":"Chen et al. supplied the original MUSIC10K dataset, which the paper processes into the evaluation format used in the experiments."}],"review_version":1}