{"id":"d707a7eb-d056-4dee-8c52-e65e0ee046cd","arxiv_id":"1909.00958","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A survey of graph embedding methods with a small benchmark comparison of seven methods on citation and social network data, and an advertised code library that is not provided.","lead":"This review maps graph representation learning methods and compares seven popular embedding methods on citation and social network datasets. It is a possible entry point for newcomers, but the promised code library and reproducible benchmark details are missing.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed first systematic evaluation and the GRLL library are not verifiable: Section V.C reports best-of-10 K-means++ results with no code, seeds, splits, or hyperparameter values, so the central contribution cannot be checked.","rationale":"The reader's weakest assumption correctly identifies the non-reproducibility of the evaluation as the load-bearing point: the paper's novelty rests on the first-systematic-evaluation and largest-library claims, and those claims require trustworthy, rerunnable experiments. My reading of Section V.C confirms that the experimental protocol omits seeds, splits, code, and hyperparameters, and that the best-of-10 K-means++ reporting can materially change rankings, especially where reported differences are as small as 0.015 Macro-F1. The absence of any GRLL repository link is a further concrete gap, not a stylistic issue. I agree with the reader's REJECT verdict and see no reason to adjust it: the survey portion may have value as a draft, but the stated central contribution is not supported in the submitted form.","tokens_in":22323,"tokens_out":3979,"duration_ms":43803,"concrete_test":"Attempt to obtain the GRLL library from the authors' pages or GitHub; if no repository exists, the central artifact claim is unsupported. If the repository is available, rerun the Section V.C experiments with the stated default hyperparameters, a fixed 50/50 split, and 10 fixed K-means++ seeds, reporting mean and standard deviation instead of the best run. If Table 3 cannot be reproduced within about 1% accuracy, or if the best-of-10 values in Table 4 differ from the mean by more than the reported inter-method gaps (e.g., the 0.015 DeepWalk-versus-node2vec difference on YouTube), then the claimed systematic evaluation is not a reliable basis for the paper's conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is not the taxonomy but the claim in the Introduction and Section V.C of being the first survey to provide a systematic evaluation of many graph embedding methods, plus the open-source GRLL library covering the largest number of techniques. That contribution collapses if the reported tables are not reproducible. Section V.C.1 states that methods were run with 'the default hyper-parameter setting provided by each graph embedding method' and a 50/50 split, but no implementation, split seed, or hyperparameter values are given. Section V.C.2 says K-means++ was run 10 times and 'we report the best result.' This best-of-10 protocol is not a neutral comparison: it favors methods with higher run-to-run variance, and the differences that drive the conclusions are small. For example, on YouTube, DeepWalk Macro-F1 is 0.206 versus node2vec's 0.221, a gap of only 0.015, which is plausibly within K-means++ seed noise. No error bars are reported. Additionally, the GRLL library is promised 'in the Github' but no URL or repository identifier appears anywhere in the paper, so the claimed artifact cannot be inspected or rerun. The evaluation is therefore not a falsifiable, systematic benchmark as claimed; it is an unrepeatable set of numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript is a survey of graph representation learning (graph embedding) methods. It organizes methods into classical (dimensionality reduction, random walk, matrix factorization) and emerging (neural-network-based, large-graph, hypergraph, attention, and others) categories, discusses applications and future research directions, and reports an experimental comparison of seven embedding methods on small and large datasets for vertex classification and clustering. The paper claims three contributions: a broad survey with insights, the first systematic evaluation of graph embedding methods in domain-specific applications, and an open-source Python library, GRLL, unifying the largest number of graph embedding techniques. The evaluation uses public datasets (Cora, Wiki, YouTube, Flickr, BlogCatalog) and reports classification accuracy and Macro/Micro-F1 clustering scores, together with training-time measurements.","tokens_in":22576,"tokens_out":5957,"duration_ms":58264,"significance":"If the evaluation and library claims were fully substantiated, the paper would be a useful reference and benchmark resource for practitioners, because the survey covers a wide range of methods including recent neural and attention-based approaches, and a unified library would lower the barrier to comparing methods. The survey's taxonomy alone is not particularly novel, as acknowledged by the authors' citations to prior surveys [11], [44], but the systematic evaluation and the GRLL library would be a genuine additional contribution. The paper does not rely on machine-checked proofs; its contribution is empirical, so the value depends almost entirely on the reproducibility, correctness, and careful reporting of the experiments in Section V. As the paper stands, that part is not verifiable.","major_comments":[{"comment":"The paper's central claim to provide 'the first survey paper that provides systematic evaluation of a rich set of graph embedding methods' and a library 'cover[ing] the largest number of graph embedding techniques up to now' is not verifiable from the manuscript. Section V.C.1 states that methods were run with 'the default hyper-parameter setting provided by each graph embedding method' and a 50/50 split, but no repository URL, no hyperparameter values, no train/test split seed, and no implementation details are given. The phrase 'in the Github' in Section V.C contains no repository identifier. As a result, the experimental numbers in Tables 3-5 cannot be regenerated or checked, and the claimed library cannot be inspected. This is a load-bearing issue because the evaluation and library are stated as the paper's second and third contributions.","section":"Introduction, third bullet; Section V.C.1"},{"comment":"The clustering evaluation reports the 'best result' of 10 K-means++ runs (Section V.C.2) without any measure of variance, such as standard deviation or the full distribution across seeds. Reporting the best of several random initializations systematically favors methods with higher run-to-run variance, so the observed differences do not support the conclusions drawn in the surrounding text. For example, the YouTube Macro-F1 gap between DeepWalk (0.206) and node2vec (0.221) is only 0.015, which is plausibly within K-means++ seed noise. Without error bars or significance tests, the comparative statements about method superiority in Section V.C.2 are unsupported.","section":"V.C.2, Table 4"},{"comment":"Two references are seriously miscited. Reference [10], cited for the Cora dataset, is the Cabanes et al. paper on ocean temperature and salinity measurements, not the Cora citation network. Reference [75], cited for Multidimensional Scaling, is Robinson and Bennett's study of deviant workplace behaviors, not an MDS reference. In a survey paper, incorrect pointers to the primary data and method descriptions prevent readers from locating the actual sources and undermine confidence in the factual reliability of the paper.","section":"References [10] and [75]"}],"minor_comments":[{"comment":"Equation (3) is malformed: the denominator of p2(vj|vi) is missing the summation braces, and the notation mixes a conditional with an equality. Please rewrite the equation carefully.","section":"Eq. (3)"},{"comment":"Equation (7) reads 'Y = argmax min|W - YY^T|', which is not a well-defined optimization problem (argmax of a min). The intended objective function should be stated explicitly.","section":"Eq. (7)"},{"comment":"There are several typos, e.g., 'the distance characteristics among nodes is preserved' should be 'are preserved', and 'where an edge ei,j connects vertex vi to vertex vj' uses inconsistent punctuation. A thorough proofread of the notation and prose is needed.","section":"II.A, Eq. (1)"},{"comment":"The caption and text refer to 'training sample ratio' and a linear regression classifier, but the figure is not included in the manuscript text; ensure all figures are actually embedded and legible.","section":"V.C.5, Fig. 6"}],"recommendation":"major_revision","confidential_remarks":"The main concern, which I share with the reader's report, is that the evaluation and the GRLL library are presented as central contributions but are completely unverifiable: no code link, seeds, splits, or hyperparameters are provided, and the clustering results are reported as best-of-10 runs without variance. If the authors can provide the repository, the exact experimental configuration, and proper statistical reporting, the paper could potentially be made defensible. The mis-citations ([10] and [75]) are also serious and point to a lack of quality control. I would not accept the paper in its current form, but I think a major revision that makes the evaluation reproducible and corrects the factual errors could be sufficient, so I recommend major_revision rather than outright rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a survey with a decent but not novel taxonomy, and the authors' real contribution—the claimed first systematic evaluation plus the GRLL library—is the part that collapses on inspection. The paper is readable and covers a reasonable spread of classical and emerging methods, and the organization by random-walk, matrix-factorization, neural, hypergraph, and attention approaches is serviceable for a newcomer. I'd credit the authors for attempting a unified comparison, and the dimension-sweep and training-ratio plots are the kind of thing practitioners actually want. None of that is new enough to make the survey itself a must-read, but it is not worthless.\n\nThe problems are concentrated where it matters. The benchmark is presented as the paper's headline claim, yet the reported numbers cannot be checked. No code, no repository URL, no seeds, no hyperparameter values, and the clustering section explicitly reports best-of-10 K-means++ runs without error bars. That is a real flaw, not a cosmetic one: the gaps that drive the conclusions are small (e.g., YouTube Macro-F1 of 0.206 vs 0.221), and with that protocol the differences could easily be initialization noise. The GRLL library is promised but never given a location, so the promise is empty as submitted. The citations do not help: reference [10] for Cora points to an ocean dataset, and [75] for MDS is a workplace-deviance paper. There are also typos in equations—Eq. (3) has a mangled subscript in p2(vj|vi)—that undermine trust in the details.\n\nThe circularity concern does not land. The survey portion restates known material, but that is what surveys do, and the self-citations in the future-directions section are not load-bearing. The central issue is simply that the paper's distinctive contribution is unsupported as written. It is not that I think the benchmark was necessarily fabricated; it is that a reader cannot tell, and the authors made it impossible to tell.\n\nWho gets value from this? A newcomer to graph embedding who wants a high-level map of the field could skim the survey sections profitably, and the dimension-sensitivity discussion is genuinely useful. But the empirical claims and the library need to be real before the paper earns its headline. If a revision ships working code, full experimental settings, and error bars, I would take it seriously. As an editor, I would send it to review only on the condition that the artifact and experiments are verifiable; otherwise, it is a borderline desk reject. My honest recommendation: require the authors to provide the GRLL repository and complete experimental details before engaging referee time.","headline":"A survey whose useful taxonomy is undercut by a central benchmark claim that is not reproducible as written, plus sloppy citations; worth refereeing only if the authors can ship the promised GRLL code and fix the experimental reporting.","tokens_in":23122,"tokens_out":661,"would_cite":false,"duration_ms":9256,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey of graph representation learning claims to provide the first systematic head-to-head evaluation of embedding methods, with random-walk techniques leading the results.","keywords":["graph representation learning","graph embedding","graph neural networks","random walks","matrix factorization","node classification","cluster quality","survey"],"falsifier":"Run the same seven methods on the same datasets through a reimplementation of the stated protocol (default hyperparameters, 50/50 train-test split, linear-regression classifier, best of ten K-means++ runs) and check whether the rank ordering in Tables 3 and 4 and the timing ratios in Table 5 are reproduced; if the ordering flips under seed variation, the claim of a systematic evaluation collapses.","tokens_in":22116,"feed_emoji":"📊","tokens_out":5139,"duration_ms":48767,"temperature":0.7,"pith_summary":"Graphs are hard to analyze because their structure is irregular, so graph representation learning compresses nodes into low-dimensional vectors for machine learning. This survey organizes the field into classical and emerging families and argues that its own contribution is a systematic comparison: it evaluates seven embedding methods on citation graphs, social networks, and a word-co-occurrence network under a unified protocol, and it releases an open-source Python library, GRLL, that it says covers more embedding techniques than any existing library. On the paper's own evidence, random-walk methods such as DeepWalk and node2vec are the strongest all-around choice, combining high classification and clustering scores with low memory and training time. The paper would matter because it offers newcomers a way to choose among methods that had previously been evaluated under incompatible settings.","feed_headline":"Survey finds random-walk methods lead graph embedding benchmarks","feed_subtitle":"Seven embedding methods compared under one protocol; open-source GRLL library ships with the paper.","key_machinery":"The paper's argument is carried by a taxonomy of graph embedding ideas and a standardized evaluation protocol. The central objects are the first-order and second-order proximity objectives that define when an embedding preserves graph structure, and the method families built on them: dimensionality reduction, random walks, matrix factorization, neural networks, hypergraphs, and attention mechanisms. The evaluation machinery is the unified GRLL interface, which runs each method with its default hyperparameters, feeds the embeddings to a linear-regression classifier for classification, and uses K-means++ for clustering, reporting the best of ten runs.","core_discovery":"On its own terms, the paper's central claim has two parts. First, it claims to be the first survey to provide systematic evaluation of a rich set of graph embedding methods in domain-specific applications, comparing vertex classification accuracy on Cora and Wiki and clustering quality on YouTube, Flickr, and BlogCatalog under one common protocol. Second, it releases GRLL, an open-source library that it says covers the largest number of graph embedding techniques up to now. The evaluation itself finds that random-walk-based methods are the top performers on classification on both small datasets, that DeepWalk and node2vec lead clustering quality on the large datasets, and that random-walk embeddings degrade much more slowly than structural-preserving methods when the embedding dimension shrinks from 128 to 4.","pith_inferences":["Because the paper reports only the best of ten K-means++ runs, the published F1 scores may overstate typical clustering performance; reporting mean and standard deviation would be a more conservative benchmark.","The main comparison excludes neural-network methods such as GCN, GraphSAGE, and GAT, so the 'random walks win' conclusion is about the unsupervised, transductive setting; extending the same protocol to those models is a direct, testable next step.","If the GRLL code and exact hyperparameter values were released, the survey's 'first systematic evaluation' claim would turn from an assertion into a reusable community benchmark.","The dimension-robustness of random-walk embeddings suggests a practical rule of thumb: for memory-constrained deployments, start with low-dimensional DeepWalk or node2vec rather than structure-preserving embeddings."],"forward_implications":["Random-walk-with-skip-gram methods are a strong default baseline for unsupervised graph embedding, outperforming matrix-factorization and other alternatives in the tested settings.","For very large graphs, DeepWalk is the practical choice because it combines top accuracy with training time far below node2vec and LINE on YouTube and Flickr.","Embedding dimension matters less for random-walk methods; users can shrink vectors to 4 or 8 dimensions with only about 20 percent accuracy loss, while LINE, GraRep, and GF lose as much as 45 percent.","GraRep's classification quality is competitive, but its memory demands keep it off the largest graphs, so memory rather than accuracy becomes the deciding factor in those cases.","The GRLL library, if used as presented, gives a single interface for running a broad set of embedding methods and comparing them on downstream tasks."],"supporting_citations":[{"why":"DeepWalk supplies the random-walk skip-gram method that leads the classification and clustering tables.","marker":"[73]"},{"why":"node2vec supplies the biased random-walk variant that ties or beats DeepWalk in several reported results.","marker":"[46]"},{"why":"LINE is the large-scale first- and second-order proximity baseline included in all evaluation tables.","marker":"[85]"},{"why":"GraRep supplies the global-structure matrix-factorization method whose quality is competitive but memory-heavy.","marker":"[13]"},{"why":"HOPE supplies the asymmetric-transitivity embedding baseline in the clustering evaluation on BlogCatalog.","marker":"[72]"},{"why":"SDNE supplies the autoencoder-based structural deep embedding baseline on Cora and Wiki classification.","marker":"[92]"},{"why":"GF supplies the graph factorization baseline used in the clustering and embedding-dimension experiments.","marker":"[3]"}],"fun_headline_variants":["Survey: random-walk embeddings top vertex classification and clustering","Graph survey releases open-source GRLL library benchmarking 7 methods","Random-walk embeddings win on accuracy and scale robustness in survey","Largest graph embedding survey releases GRLL library; random walks lead","Survey benchmarks 7 graph embedding methods; random-walk ones dominate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported accuracy, F1, and timing numbers were produced by the claimed implementations under the stated default settings, since the paper gives no code, data splits, seeds, or hyperparameter values that would let a reader reproduce the tables.","fun_headline_variants_meta":{"raw":{"variants":["Survey: random-walk embeddings top vertex classification and clustering","Graph survey releases open-source GRLL library benchmarking 7 methods","Random-walk embeddings win on accuracy and scale robustness in survey","Largest graph embedding survey releases GRLL library; random walks lead","Survey benchmarks 7 graph embedding methods; random-walk ones dominate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3244,"prompt_tokens":795,"completion_tokens":2449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":2362}},"tokens_in":411,"tokens_out":2449,"duration_ms":17243,"temperature":1.0,"reasoning_tokens":2362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:30:12.929179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same seven methods on the same datasets through a reimplementation of the stated protocol (default hyperparameters, 50/50 train-test split, linear-regression classifier, best of ten K-means++ runs) and check whether the rank ordering in Tables 3 and 4 and the timing ratios in Table 5 are reproduced; if the ordering flips under seed variation, the claim of a systematic evaluation collapses.","supporting_citations":[{"cited_title":"MILE: A Multi-Level Framework for Scalable Graph Embedding","cited_arxiv_id":"1802.09612","evidence_quote":"DeepWalk supplies the random-walk skip-gram method that leads the classification and clustering tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LINE is the large-scale first- and second-order proximity baseline included in all evaluation tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HOPE supplies the asymmetric-transitivity embedding baseline in the clustering evaluation on BlogCatalog."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SDNE supplies the autoencoder-based structural deep embedding baseline on Cora and Wiki classification."}],"review_version":1}