{"id":"15b4408b-6e14-482f-baca-5697a24d8ef1","arxiv_id":"2501.12857","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"HierPromptLM uses hierarchical text prompts and two language-model pretraining tasks to jointly encode node text and heterogeneous graph structure, outperforming prior text-plus-GNN baselines on DBLP and OAG.","lead":"HierPromptLM is a language-model-only method that turns text-rich network neighborhoods into text prompts, then fine-tunes a BERT-style model with two custom training tasks. It reports gains over graph-neural-network baselines on two academic datasets, but the link prediction evaluation does not state whether held-out links were excluded during pretraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The link-prediction evaluation may be leaking test edges into HGA-NSP pretraining: §4.3.1 samples positives from all connected node pairs, while §5.3 holds out 40% of edges for testing without stating that those edges are excluded from pretraining.","rationale":"The reader's weakest assumption and my stress-test converge on the same point: the paper never specifies that link-prediction test edges are excluded from HGA-NSP pretraining, and this is the most load-bearing uncertainty in the empirical evaluation. I reviewed the methodology for other candidate weaknesses. The 'pure PLM' claim is partially softened by the use of a frozen PLM for graph-token distillation and predefined meta-path textualization, but those are architectural choices rather than internal contradictions. The parameter sensitivity discussion in Appendix C.4 actually shows the method is not uniformly better for all meta-path choices on DBLP link prediction, yet that is presented transparently and does not by itself undermine the headline. The ablation 'w/o HGA-MLM' on DBLP coincidentally reproduces a baseline value, which is suspicious but not sufficient to identify as the main concern without code. The leakage issue is directly tied to the largest reported gains and to the claimed edge-level modeling contribution, so it is the single load-bearing concern. My proposed check is a concrete overlap computation plus a leakage-free rerun; until then, conditional acceptance is appropriate. Therefore the verdict remains unchanged.","tokens_in":41,"tokens_out":4811,"duration_ms":146094,"concrete_test":"Obtain or derive the full HGA-NSP positive tuple list and the link-prediction test edge set; compute the overlap. If overlap > 0, leakage is confirmed. Then quantify impact by rerunning with a leakage-free protocol: split edges 10/50/40 before any pretraining; build graph tokens only from the 10% training-edge subgraph; draw HGA-NSP positives only from those training edges; tune learning rate and ratios on the 50% validation split. Compare DBLP link-prediction PR-AUC/ROC-AUC/F1 against the reported table; if margins over THLM/Heterformer shrink materially, the headline link-prediction claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim includes large link prediction gains (up to 10.84% PR-AUC on DBLP). The load-bearing assumption is that the 40% test edges of §5.3 are never used as positive HGA-NSP samples. §4.3.1 defines positive samples as tuples (u,r,v) where u and v are connected by r within the HTRN, and §5.3 splits edges as 10% training / 40% testing 'following [50]'. No sentence in §5, Appendix B, or the reproducibility section states that test edges are removed from HGA-NSP positive sampling or from the graph-token extraction. Since HGA-NSP is literally a connected/not-connected classification task, any test edge seen as a positive sample gives the model the answer during pretraining; HGA-MLM on the same relation-aware prompt can memorize the edge as well. If overlap exists, the reported ROC-AUC/PR-AUC/F1 improvements of 9-11% overall and 28-31% on paper-paper links are inflated by memorization rather than representation quality. Node classification is less directly affected, so a corrected study may still support the framework; the present paper does not establish the link-prediction claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HierPromptLM proposes a 'pure PLM' framework for representation learning on heterogeneous text-rich networks (HTRNs), replacing the usual GNN-plus-PLM pipeline with a single language model that ingests both node text and graph structure in one prompt sequence. Meta-path-based subgraphs around each node are textualized and distilled by a frozen PLM into graph tokens, which are appended to the node's own text to form a graph-aware prompt; a learnable relation token inserted between two nodes' graph-aware prompts yields a relation-aware prompt. A tunable BERT is fine-tuned with two objectives, HGA-MLM (masked token prediction over text and graph tokens) and HGA-NSP (classification of whether a node pair is connected by a given relation), and the resulting embeddings are evaluated on node classification and link prediction. On DBLP and OAG, the model reports consistent node-classification gains over nine baselines (up to 6.08% Macro-F1) and large link-prediction gains (up to 10.84% PR-AUC overall, and up to 31.51% PR-AUC for paper-paper links), alongside ablations, a training-free variant, and PLM-backbone extension experiments.","tokens_in":20018,"tokens_out":18169,"duration_ms":159210,"significance":"The central idea — encoding heterogeneous structure and text jointly in a single textual space and learning edge-level semantics through a learnable relation token — is a genuine conceptual departure from the GNN-PLM-alignment approaches (Heterformer, THLM) that the paper targets, and the two pretraining tasks are natural and sensible adaptations of MLM/NSP to HTRNs. The paper deserves credit for including a training-free variant as a falsifiable sanity check, for the per-relation link-prediction breakdown in Appendix C.1, and for disclosing several hyperparameters in Appendix B. The significance is nevertheless bounded by three factors: the evidence base is only two datasets with no code release; the headline link-prediction claim is compromised by the protocol gap described in Major Comment 1; and one ablation row (Table 6) appears corrupted. If the link-prediction results survive a leak-free protocol, the framework could be a useful new baseline for text-attributed network tasks; if not, the node-classification results alone would establish a more modest contribution.","major_comments":[{"comment":"The link-prediction evaluation is defined over the same object that HGA-NSP pretraining learns to classify. Section 4.3.1 states that positive HGA-NSP samples are tuples (u,r,v) with u and v 'connected by the relation r within the HTRN', with no qualification, while Section 5.3 holds out 40% of edges for link-prediction testing following [50]; neither Section 5, nor Appendix B, nor the reproducibility material states that held-out edges are excluded from HGA-NSP positive sampling, from HGA-MLM prompt construction, or from the meta-path subgraph and graph-token extraction of Sections 4.1–4.2. Because HGA-NSP is a binary classifier over relation-aware prompts and evaluation scores the same kind of prompt for relation validity, any test edge that appears among the pretraining positives gives the model the correct answer in advance; the edge also appears inside the endpoints' subgraph summaries, so HGA-MLM can memorize the association as well. Under this reading, the gains in Table 3 (9.19% ROC-AUC, 10.84% PR-AUC, 9.79% F1 on DBLP) and the per-relation gains in Table 5 (28.38% ROC-AUC, 31.51% PR-AUC, 21.40% F1 on paper-paper links) measure retrieval of pretraining positives rather than generalization, and the ablation drop in Table 6 when HGA-NSP is removed (DBLP ROC-AUC 85.56 to 77.68) is consistent with that channel. The authors should either document explicitly that test edges were masked during pretraining and prompt construction (including how the mask was applied to the subgraph extraction in Section 4.1), or re-run the link-prediction experiments under such masking. Node classification (Table 2) is less directly affected because class labels are never used in pretraining, so a corrected study may still support the framework; the link-prediction claim is the part that must be re-established.","section":"§4.3.1 vs. §5.3, Appendix B"},{"comment":"The w/o HGA-MLM row of the DBLP ablation (Table 6) reports node classification Micro-F1 54.62, Macro-F1 17.66, and link prediction 50.00/50.00/33.33, all with zero standard deviation; the two node-classification values are numerically identical to the HIN2Vec+Bert row in Table 2 (54.62, 17.66). This exact match suggests a copy-paste or data-recording error. As printed, the row cannot support the claim in Appendix C.2 that removing HGA-MLM causes a 'significant performance drop' on DBLP. The corresponding OAG row in Table 4 (83.07/78.93/76.85/71.35/76.16) does support the qualitative conclusion, so the fix is local, but the DBLP row must be corrected or removed before the ablation evidence can be considered reliable.","section":"Table 6 / Appendix C.2"}],"minor_comments":[{"comment":"In the paragraph above Table 5, the reported improvements for paper-paper links read '28.38% gain in ROC-AUC, 31.51% enhancement in ROC-AUC'; the second metric should be PR-AUC, not ROC-AUC.","section":"§C.1 / Table 5"},{"comment":"The '#Relation' column (67,417 for DBLP and 63,808 for OAG) does not agree with the sum of the per-relation edge counts listed in Appendix A (149,545 and 240,219, respectively); the table should clarify what '#Relation' counts, for example undirected interactions versus directed edges.","section":"Table 1 / Appendix A"},{"comment":"The link-prediction protocol is incomplete: the paper does not say what the remaining 50% of edges are used for, how negative samples for evaluation are drawn, or which output position of the relation-aware prompt (for example [CLS], mean pooling, or the relation token) is scored to produce the reported AUC and F1 values; these details are needed for reproducibility and for verifying the exclusion requested in Major Comment 1.","section":"§5.3"},{"comment":"The paper says 'the model calculates a probability p for each sample' for HGA-NSP but does not specify how p is derived from the relation-aware prompt; please state the scoring head, such as a linear layer on the [CLS] or relation-token representation.","section":"§4.3.1"},{"comment":"The training-free evaluation is not fully described: for link prediction in the frozen setting, it is unclear how an edge score or probability is computed without fine-tuning; a sentence describing the scoring procedure would allow the reader to interpret the claim that HierPromptLM-free outperforms the best baselines on both tasks.","section":"§5.5 / Figure 3(a)"},{"comment":"The reproducibility section should report the number of fine-tuning epochs or the convergence criterion, batch size, learning-rate schedule, and random seeds; currently only the optimizer (AdamW), learning-rate range, masking ratio (0.15), negative sampling ratio (1), and hidden dimension (768) are given, and no code or model checkpoint is released.","section":"Appendix B"},{"comment":"The sentence 'All models are trained for 10 times and, the mean and standard variance of test performance are reported' should read 'trained 10 times' and 'standard deviation'; in addition, the naming is inconsistent between 'M2V+Bert' in Tables 2–3 and 'M2V-Bert' in the Section 5.3 discussion.","section":"§5.2"},{"comment":"The claims of being 'for the first time' a pure PLM-based framework and introducing the two pretraining tasks should be positioned more carefully against THLM [50], which also pretrains a PLM on text-attributed heterogeneous graphs (albeit with a separate alignment step), and against LinkBERT [42].","section":"§1 / §2.2"}],"recommendation":"major_revision","confidential_remarks":"The main risk in this manuscript is the link-prediction protocol gap (Major Comment 1); I would ask the authors to state, in any revision, whether the 40% test edges were present among the HGA-NSP positive samples and in the graph tokens, and to re-run the evaluation if they were. I also noticed that the w/o HGA-MLM row of Table 6 exactly reproduces the HIN2Vec+Bert row of Table 2, which I treat as a data-integrity problem to be corrected. A further note: the novelty claims ('for the first time') are stronger than the related-work discussion supports, given THLM [50] and LinkBERT [42]; the positioning should be sharpened. Finally, with only two datasets and no released code or complete training details, the 'consistently surpasses' wording is broader than the evidence; acceptance should be conditioned on resolution of the leakage question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HierPromptLM is a plausible architectural contribution that deserves a serious referee, but the headline link prediction numbers are unreliable. The issue is concrete: HGA-NSP (Section 4.3.1) is trained on tuples (u,r,v) where u and v are connected in the HTRN, and the link prediction evaluation (Section 5.3) holds out 40% of edges for testing. The paper never states that those test edges are excluded from HGA-NSP's positive sampling or from graph-token extraction. Since HGA-NSP is literally a connected/not-connected classification, any test edge seen as a positive sample gives the model the answer during pretraining. The reported 9–11% aggregate gains and 28–31% gains on paper-paper links are inflated by memorization, not representation quality. This is a load-bearing flaw, not a minor detail. The stress-test note got this right, and the paper's silence on the split makes the ambiguity chronic rather than incidental.\n\nWhat is genuinely new: the pure-PLM formulation, where meta-path-based subgraphs are distilled into graph tokens by a frozen PLM and combined with text and a learnable relation token, so that heterogeneous structure and text live in one embedding space with no alignment step. The two pretraining tasks are tailored to the setup, and the training-free variant is a nice touch. The node classification results are much less affected by the leakage concern and are consistent across both datasets, so the framework has plausibility there. The comparison to Heterformer and THLM is appropriate, and the related work coverage is fine.\n\nThe soft spots beyond the leakage: hyperparameter selection appears to be done on test performance (Appendix C.4 discusses choosing masking and negative-sampling ratios by looking at downstream results, with no separate validation split mentioned). There is no code release, and only two datasets, both from the same academic graph family. None of these by themselves would sink the paper, but combined with the leakage, the empirical section as written overclaims.\n\nMy take: the architecture is worth reading about, and the node classification story might survive a corrected evaluation. But the link prediction claims must be re-run with the test edges masked during pretraining, and the hyperparameters need a proper validation split. A serious referee should ask for this before acceptance.","headline":"Interesting pure-PLM architecture for text-rich heterogeneous networks, but the link prediction results are likely inflated by train/test leakage in HGA-NSP; node classification is more believable, and the paper deserves a conditional review.","tokens_in":20563,"tokens_out":3131,"would_cite":false,"duration_ms":33833,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single pretrained language model handles text and graph structure in one space.","keywords":["heterogeneous text-rich networks","representation learning","prompt learning","pretrained language models","meta-path","graph tokens","HGA-MLM","HGA-NSP"],"falsifier":"Rerun the DBLP experiment with explicit bookkeeping that excludes all test edges from the HGA-NSP positive tuples, then recompute ROC-AUC and PR-AUC; if the margin over the best baseline shrinks to near zero, the headline link-prediction gains come from memorization rather than from joint text-structure representation.","tokens_in":19533,"feed_emoji":"🕸️","tokens_out":7332,"duration_ms":68610,"temperature":0.7,"pith_summary":"Heterogeneous text-rich networks—papers linked to authors, venues, and fields—are usually embedded by running a graph neural network and a language model separately and then trying to align their vector spaces. HierPromptLM argues this separation is unnecessary and harmful. It claims that one pretrained language model, prompted with textual summaries of each node's meta-path-based neighborhood and a learnable token for each relation type, can represent both nodes and edges in a single unified text space. On two academic datasets, the paper reports consistent gains over nine baselines: up to 6.08% Macro-F1 for node classification and 10.84% PR-AUC for link prediction. A training-free variant that skips fine-tuning still beats the best baselines, suggesting the prompt itself carries structural information.","feed_headline":"A single language model beats combined text+graph models","feed_subtitle":"Prompt-based encoding lifts node classification by 6.08% and link prediction by 10.84% on DBLP.","key_machinery":"The load-bearing object is the hierarchical prompt: a graph-aware prompt per node, built from the node's text plus graph tokens distilled from meta-path-based subgraph summaries by a frozen PLM, and a relation-aware prompt per edge, built by placing a learnable relation token between the two endpoints' graph-aware prompts. These prompts carry the whole argument: they translate heterogeneous structure into the PLM's own token space, so no separate GNN and no alignment step is needed. The second supporting mechanism is the two pretraining objectives—HGA-MLM, which masks both text and graph tokens, and HGA-NSP, which predicts whether a node pair is linked by a given relation—that push the tunable PLM to use the structural tokens jointly with text.","core_discovery":"The central claim is that a pure PLM-based framework—one that never trains a separate heterogeneous GNN and never aligns two embedding spaces—is sufficient for representation learning on heterogeneous text-rich networks. The method converts each node's local structure into text: meta-path-based subgraphs are summarized into natural-language sentences, distilled into compact graph tokens by a frozen language model, and then combined with the node's own text into a graph-aware prompt; for an edge, the graph-aware prompts of its two endpoints are joined by a learnable relation token. A tunable PLM is fine-tuned on two HTRN-specific objectives, one that predicts masked text and graph tokens and one that classifies whether two nodes are connected by a given relation. The claimed result is that this joint textual treatment captures the interaction between what a node says and where it sits in the network, and does so better than separate PLM-plus-HGNN pipelines on both node classification and link prediction.","pith_inferences":["Editorial inference: if this holds, the standard two-stage design of HTRN models—GNN for structure, PLM for text, plus an alignment loss—could be replaced by a single text prompt, removing a whole class of representation-mismatch failures.","Editorial inference: the meta-path choice remains manual, but the paper's DBLP result shows direct meta-paths sometimes beat longer ones for link prediction; a natural extension is to let the model or an LLM choose or generate meta-path descriptions instead of fixing them by hand.","Editorial inference: since the training-free variant is competitive, the prompts themselves might serve as a cheap baseline for evaluating whether a downstream task needs graph structure at all."],"forward_implications":["With the pure-prompt design, node classification improves by up to 4.58% Micro-F1 and 6.08% Macro-F1 on DBLP, and by 1.49% Micro-F1 and 2.44% Macro-F1 on OAG, over the best baselines.","Link prediction improves by up to 9.19% ROC-AUC, 10.84% PR-AUC, and 9.79% F1 on DBLP, with smaller but consistent gains on OAG.","Ablations attribute most of the gain to the joint masking task: removing HGA-MLM drops OAG Micro-F1 from 96.24% to 83.07%; removing graph tokens or the relation token also degrades both tasks.","A training-free version, which skips fine-tuning entirely, still surpasses all baselines on both datasets, indicating the prompt construction itself carries usable structural signal.","Swapping the frozen graph-token encoder from BERT to T5-base or GPT2-small increases performance, so the framework scales with stronger language models."],"supporting_citations":[{"why":"Supplies the BERT backbone that the framework fine-tunes and the MLM/NSP objectives that HGA-MLM and HGA-NSP adapt.","marker":"[18]"},{"why":"Defines the HTRN problem setting and represents the separate-processing PLM-plus-HGNN approach the paper claims to outperform.","marker":"[16]"},{"why":"Provides the other main HTRN baseline and the alignment-based design the paper argues is unnecessary.","marker":"[50]"},{"why":"Introduces meta-path-based neighbor aggregation, the structural idea the graph-aware prompt's subgraph design builds on.","marker":"[37]"},{"why":"Shows how to aggregate information along meta-paths including intermediate nodes, motivating the meta-path-based subgraph extraction.","marker":"[12]"}],"fun_headline_variants":["Pure PLM with graph prompts beats separate PLM and GNN models","Single language model handles text and graph via prompt-based encoding","Prompt-based graph encoding lifts node classification and link prediction","No separate GNN: pure PLM framework for heterogeneous text-rich networks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported link-prediction improvements assume that the 40% of edges held out for testing were not used as positive samples in HGA-NSP pretraining, a separation the paper never explicitly states.","fun_headline_variants_meta":{"raw":{"variants":["Pure PLM with graph prompts beats separate PLM and GNN models","Single language model handles text and graph via prompt-based encoding","Prompt-based graph encoding lifts node classification and link prediction","No separate GNN: pure PLM framework for heterogeneous text-rich networks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000734,"raw_usage":{"total_tokens":3317,"prompt_tokens":1018,"completion_tokens":2299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":2227}},"tokens_in":634,"tokens_out":2299,"duration_ms":19165,"temperature":1.0,"reasoning_tokens":2227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:42:39.128320+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the DBLP experiment with explicit bookkeeping that excludes all test edges from the HGA-NSP positive tuples, then recompute ROC-AUC and PR-AUC; if the margin over the best baseline shrinks to near zero, the headline link-prediction gains come from memorization rather than from joint text-structure representation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the HTRN problem setting and represents the separate-processing PLM-plus-HGNN approach the paper claims to outperform."},{"cited_title":"Pretraining Language Models with Text-Attributed Heterogeneous Graphs","cited_arxiv_id":"2310.12580","evidence_quote":"Provides the other main HTRN baseline and the alignment-based design the paper argues is unnecessary."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows how to aggregate information along meta-paths including intermediate nodes, motivating the meta-path-based subgraph extraction."}],"review_version":1}