{"id":"e2a82859-7c8d-423a-8007-f5040a59e270","arxiv_id":"2505.03799","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SDM-InstructGLM uses a similarity-degree biased random walk to sample graph neighborhoods for LLM prompts, reporting gains over InstructGLM on CORA and PUBMED that are confounded by a change in node features and training length.","lead":"This paper proposes a smarter way to feed graph data to a large language model, sampling important neighbors using feature similarity and node degree. It reports large accuracy gains on a citation graph, but the comparison changes two things at once, so the gains are not cleanly caused by the new sampling method.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparison is confounded: Table II varies features and training epochs alongside the sampling method, so the claimed accuracy gains cannot be attributed to SDM.","rationale":"The reader identified exactly the confound I consider load-bearing. The paper's stated goal is to show that similarity-degree sampling improves InstructGLM. The only head-to-head evidence (Table II) alters features and training duration simultaneously, so the comparison does not isolate sampling. The ablation studies show the cosine/degree components matter within SDM, but that is a different claim from outperforming the original method. No error bars or repeated runs are provided, and the claimed link-prediction results are absent, which further weakens completeness but is secondary. A controlled rerun would settle the issue; until then the central claim is unproven, so the appropriate disposition is a major revision or rejection as currently written. I do not see an internally inconsistent derivation in Section IV; the math is simple and coherent, but empirical support is the crux.","tokens_in":12697,"tokens_out":2424,"duration_ms":23439,"concrete_test":"Run a 2x2 factorial on CORA with epochs fixed: sampler (random as in InstructGLM vs SDM) x features (TF-IDF vs BERT), reporting 1/2/3-hop accuracy, plus at least 3 seeds with standard deviations. If InstructGLM-with-BERT under the same 2-epoch budget matches or exceeds SDM's numbers, the Table II gap is a feature/epoch artifact. If SDM still wins over random sampling with TF-IDF and with BERT separately, the sampling attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim, 'Across all settings, SDM-InstructGLM consistently outperforms InstructGLM,' rests on Table II. But Section V-A changes two additional variables: node features are switched from TF-IDF to BERT-based embeddings, and the baseline is trained for 4 epochs while SDM is trained for 2. On CORA, TF-IDF is known to be weak, so much of the reported 13-19 point gain may reflect the encoder, not the biased walk. The ablation (Table III) only removes cosine and degree components within SDM; it never runs InstructGLM's random sampling with BERT features or the same token budget. Therefore RQ1 is not answered and the central claim is unsupported. The abstract's link-prediction claim is also unbacked by any reported link-prediction experiment, and the node-ordering 'empirical results' are asserted without a table or comparison. None of these issues are resolved by the GitHub release, which is not independently evaluated; the community cannot reproduce or isolate the claimed effect from the paper alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SDM-InstructGLM, a modification of the InstructGLM framework for graph learning with LLMs. The method replaces InstructGLM's random neighbor sampling with a similarity-degree biased random walk, adds a structured node ordering scheme, and introduces hop-aware node selection under token constraints. The authors report node-classification accuracy on CORA and PUBMED, claiming consistent improvements over the original InstructGLM across 1-, 2-, and 3-hop settings, and they provide ablations removing the cosine-similarity and degree terms. Source code is made available on GitHub.","tokens_in":12867,"tokens_out":4196,"duration_ms":45757,"significance":"If the reported gains were attributable to the proposed sampling mechanism, the work would offer a simple, interpretable, and token-efficient way to scale LLM-only graph processing, which is a genuinely useful direction. The paper also provides public code and performs component-level ablations, which are strengths. However, the central comparison in Table II changes node features and training epochs simultaneously with the sampling method, so the headline accuracy gap cannot be attributed to SDM. The abstract additionally claims link-prediction improvements without reporting any link-prediction experiments, and the claimed benefit of structured node ordering is not supported by a dedicated experiment. As presented, the core contribution is not established.","major_comments":[{"comment":"The headline comparison is confounded. The baseline uses TF-IDF features and was typically trained for 4 epochs, while SDM-InstructGLM uses BERT-based embeddings and is trained for 2 epochs (Section V-A explicitly states both changes). On CORA, where the reported gains are 13–19 points, the improvement could plausibly come from the stronger feature encoder or the different training budget rather than from similarity-degree sampling. A controlled comparison that runs the original random-sampling InstructGLM with BERT features, 2 epochs, and the same token budget is necessary before RQ1 can be answered.","section":"Section V-A, Table II"},{"comment":"The abstract claims that the method 'enhances performance on graph-based tasks such as node classification and link prediction,' but no link-prediction experiment appears anywhere in the manuscript; Tables II–IV report only node-classification accuracy. The statement in Section V-A that readers should consult prior InstructGLM work for GNN and Graph Transformer comparisons does not supply the missing experiments, so the scope and evidence for the stated claims are mismatched.","section":"Abstract, Section V-B"},{"comment":"Section IV-B asserts that 'Empirical results demonstrate that this method outperforms random ordering,' but no experiment isolating the structured-ordering component is reported. The conclusion itself lists as future work the goal of identifying the dataset characteristics that give the node-ordering method its advantage over the random-walk method, which indicates that the ordering benefit is not yet established by the present evidence.","section":"Section IV-B, Section VI"},{"comment":"The hop-aware token allocation formula Nsample(h) = min(Nhop(h), Tmax/Tavg) applies the same global node cap to each hop rather than distributing the token budget according to the empirical hop distribution as the text claims. The manuscript also does not report any verification that total token consumption stays within Tmax under this rule, so the token-efficiency advantage over fixed-threshold sampling is not demonstrated.","section":"Section IV-C"}],"minor_comments":[{"comment":"Reference [14] appears to be a placeholder ('F. Author and S. Author') with no real venue, and references [13] and [34] are identical; reference [37] duplicates reference [8]. These should be corrected.","section":"References"},{"comment":"The InstructGLM baseline numbers in Table IV (88.26, 89.53, 89.55) differ from the corresponding numbers in Table II (91.56, 90.87, 90.85) without explanation; if the limited-prompt-node setup changes the baseline, the discrepancy should be described explicitly.","section":"Table IV"},{"comment":"The Introduction contains a typo: 'link prediction., graph reasoning' has a stray period before the comma.","section":"Section I"},{"comment":"The sentence 'Empirical evaluations confirm that this structured sampling approach improves subgraph representation' reports a result that is not tied to any table or figure in the paper; either cite the relevant experiment or remove the sentence.","section":"Section IV-C"},{"comment":"The statement that the model 'consistently surpasses previous benchmarks' is not backed by a comparison table and should be either substantiated with numbers or softened to match the reported experiments.","section":"Section V-A"}],"recommendation":"reject","confidential_remarks":"The experimental confound in Table II is severe enough that the central claim is unsupported as written, and the abstract promises link-prediction evidence that is not present. The citation list also contains a placeholder reference and duplicate entries, which compounds doubts about manuscript care. I see no basis for acceptance even with minor revision; a rejection would allow the authors to rework the controlled comparison and resubmit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2505.03799. The core idea is a similarity-degree biased random walk plus a hop-aware token budget for InstructGLM-style LLM prompts. That is a reasonable, incremental extension: InstructGLM samples neighbors randomly, so biasing toward high-similarity, high-degree nodes and allocating tokens across hops proportional to observed structure is a sensible thing to try. The paper states the method clearly and the equations are straightforward. Credit where due: the authors are upfront about not running the ArXiv dataset and about training the baseline for only 2 epochs instead of 4, and the GitHub release at least signals intent to share code.\n\nThe soft spot is the one the stress-test note pins down, and it is load-bearing. Table II compares SDM-InstructGLM against 'InstructGLM (Original)' while changing two other variables at the same time: node features go from TF-IDF to BERT, and the baseline is trained for 4 epochs while SDM gets 2. On CORA, TF-IDF is known to be weak, so the 13–19 point gap could easily come from the encoder rather than the walk. The ablation in Table III removes cosine and degree terms but never runs InstructGLM's random sampling with BERT features or the same token budget, so it does not isolate the mechanism either. The abstract's link-prediction claim is unbacked by any experiment in the paper, and the 'empirical results' for structured node ordering are asserted without a table or comparison. There is also a placeholder reference [14] ('F. Author and S. Author') and duplicate references [13]/[34]. None of this is fixable by a footnote; the central comparison needs to be redone with the feature encoder and epoch count held fixed before the headline claim is supported.\n\nBottom line: the paper contains a plausible idea and a clear write-up, but the evaluation as reported does not support the central claim. It is not a waste of a referee's time—the flaws are correctable and the idea is worth testing properly—but it should not be accepted in this form. If it comes across your desk, send it back with a request for a controlled comparison against InstructGLM with BERT features, same epoch count, same token budget, and with the ablation re-run against that baseline. I would not cite the current numbers in anything of my own.","headline":"A sensible incremental idea with a confounded evaluation that does not support the headline accuracy claims.","tokens_in":13413,"tokens_out":2089,"would_cite":false,"duration_ms":21095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing InstructGLM's random neighbor sampling with a similarity-degree biased random walk and hop-aware token allocation gives consistent node-classification gains and lower token cost, the paper argues.","keywords":["graph representation learning","large language models","biased random walk","node classification","neighbor sampling","token efficiency","instruction tuning","scalability"],"falsifier":"Re-run the CORA and PUBMED node-classification comparison with identical node features (either both TF-IDF or both BERT) and identical training epochs (both 2 or both 4), keeping only the sampling rule different; if the accuracy gap shrinks to near zero or reverses, the central claim is not supported. A second check is to run the link-prediction experiments the abstract promises, since no link-prediction results appear in the paper.","tokens_in":12486,"feed_emoji":"🕸️","tokens_out":4513,"duration_ms":40730,"temperature":0.7,"pith_summary":"This paper sets out to show that large language models can classify nodes in graphs accurately without any graph neural network, if the graph neighborhood is sampled intelligently rather than at random. It proposes SDM-InstructGLM, which biases the choice of neighboring nodes toward those with similar text features and high degree, and allocates the LLM's limited token budget across hops in proportion to the graph's observed structure. On CORA and PUBMED, the authors report that this beats the original InstructGLM at every hop depth they test, with the largest gain on CORA (roughly 19 accuracy points at 1-hop). If right, the result matters because it suggests a simple, interpretable sampling rule can remove a scalability bottleneck for LLM-only graph learning.","feed_headline":"Biased graph sampling lifts LLM accuracy on CORA by 19 points","feed_subtitle":"Replacing random neighbor picks with similarity- and degree-biased walks cuts token waste and beats the baseline at every hop depth.","key_machinery":"The load-bearing mechanism is a biased random walk with transition probability $$p(v|u) = \\text{softmax}(\\cos\\text{Sim}(h_u,h_v) \\cdot \\deg(v)).$$ The cosine similarity term keeps the walk near nodes whose text features match the anchor, and the degree term pulls it toward hubs; the same product serves as a ranking score for ordering nodes in the sequence. A hop-aware sampling rule $$N_{\\text{sample}}(h) = \\min\\left(N_{\\text{hop}}(h), \\frac{T_{\\max}}{T_{\\text{avg}}}\\right)$$ then caps how many nodes are encoded from each hop so the prompt fits the LLM's token limit while preserving a balanced spread of local and global context.","core_discovery":"The central claim is that the way neighborhoods are sampled into the prompt, not the LLM's architecture, is the main lever for making LLM-only graph learning work under token constraints. Concretely, the paper claims that computing a transition score as the product of cosine similarity between node feature embeddings and the neighbor's degree, then softmax-normalizing over neighbors, produces walks that retain both semantically relevant and structurally central nodes. Combined with ordering sampled nodes by the same score and capping per-hop node counts by the token budget, this 'similarity-degree-based' sampling consistently outperforms InstructGLM's random neighbor sampling across 1-, 2-, and 3-hop settings on both CORA and PUBMED, while using fewer tokens. The ablation studies are offered as evidence that both the similarity term and the degree term contribute to the gain, especially on the sparser CORA graph.","pith_inferences":["A testable extension the authors leave implicit is to run the same comparison on a deliberately heterophilic graph, where connected nodes tend to differ in class; the degree term could either help by finding hubs or hurt by over-weighting dissimilar neighbors.","Because the abstract claims link-prediction gains but the experiments only report node classification, the link-prediction claim is currently an unsupported inference of the paper, and a companion experiment would settle it.","The degree-bias term resembles a popularity prior, so in graphs with skewed degree distributions it may push the walk toward a few hubs and reduce coverage; that is a plausible failure mode worth checking, not something the paper tests.","The scalability claim rests on CORA and PUBMED only, and the authors state they could not run the ArXiv dataset used in the original InstructGLM because of replication issues, so scaling to large graphs remains untested."],"forward_implications":["If the sampling rule is the cause of the gains, then other LLM-only graph methods can adopt the same similarity-degree walk without changing the underlying model architecture.","Hop-aware token allocation gives a principled way to scale LLM-only graph processing to larger graphs than CORA and PUBMED, since the prompt stays within budget while coverage follows the graph's natural density.","The strong ablation results on CORA imply that on sparse, class-heavy graphs, feature similarity and hub selection matter more, while on dense, text-rich graphs like PUBMED, the choice of sampling matters less.","The consistent small gains on PUBMED at all hop depths suggest the method does not trade local accuracy for global context, which is a direct counter to the usual random-sampling information loss."],"supporting_citations":[{"why":"The InstructGLM baseline and backbone that SDM-InstructGLM builds on and compares against.","marker":"[18]"},{"why":"GraphSAGE, the source of the random neighbor sampling strategy that SDM-InstructGLM replaces.","marker":"[21]"},{"why":"Supplies the CORA and PUBMED datasets used for evaluation.","marker":"[39]"},{"why":"Provides the raw text representation used for node features.","marker":"[40]"},{"why":"PinSage, cited as the precedent for ranking and ordering sampled nodes by importance.","marker":"[38]"}],"fun_headline_variants":["Similarity-degree walks beat random sampling in InstructGLM","LLM-only graph learning: smarter sampling, fewer tokens","Why neighbor choice matters: InstructGLM with biased walks","Token-savvy graph sampling lifts LLM node classification","Key to LLM graphs: sample by similarity, not at random"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority assumes the comparison isolates the sampling mechanism, but the experiments change node features from TF-IDF to BERT and cut training from 4 to 2 epochs at the same time, so those changes could explain some or all of the accuracy gap.","fun_headline_variants_meta":{"raw":{"variants":["Similarity-degree walks beat random sampling in InstructGLM","LLM-only graph learning: smarter sampling, fewer tokens","Why neighbor choice matters: InstructGLM with biased walks","Token-savvy graph sampling lifts LLM node classification","Key to LLM graphs: sample by similarity, not at random"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2871,"prompt_tokens":982,"completion_tokens":1889,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1806}},"tokens_in":598,"tokens_out":1889,"duration_ms":12511,"temperature":1.0,"reasoning_tokens":1806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:28:27.507600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CORA and PUBMED node-classification comparison with identical node features (either both TF-IDF or both BERT) and identical training epochs (both 2 or both 4), keeping only the sampling rule different; if the accuracy gap shrinks to near zero or reverses, the central claim is not supported. A second check is to run the link-prediction experiments the abstract promises, since no link-prediction results appear in the paper.","supporting_citations":[],"review_version":1}