{"id":"34ccc305-eb16-4959-b409-f2f18e1c713d","arxiv_id":"2412.11888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"WalkGNN, a pair-state graph neural network run on ego-nets, is reported to outperform baselines for VK friend suggestions offline and lift friend-request CTR by 12 percent online.","lead":"VK researchers describe a way to run supervised graph neural networks on huge social graphs by cutting the graph into small local neighborhoods, called ego-nets, and scoring friend suggestions inside each neighborhood. In offline tests their WalkGNN model beat existing heuristics and graph networks on a new VK dataset, and a live test on VK reportedly raised friend-request click-through by 12 percent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random train/test split of overlapping ego-nets does not prevent structural leakage; the claimed offline advantage over PPGN may be inflated.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing risk to the central claim: the offline evaluation's train/test split can leak information because ego-nets in a social graph overlap heavily. The paper's statement that 'all ego-nets are completely independent' (Section 4.1) refers only to the relabeling of nodes within each ego-net, not to statistical independence of the sampled subgraphs. Since the model is trained to predict pairwise friendship scores within ego-nets, any structural regularity shared between training and test ego-nets can be exploited, and the 20% offline margin over PPGN is exactly the kind of claim that could be inflated by such leakage. The online A/B test provides real-world support, but the paper gives no statistical detail, so the offline experiment remains the primary scientific evidence. A user-disjoint split is a concrete, feasible check that would settle the question: if the relative ranking and margin persist under this stricter split, the offline claim is robust; if not, the reported lead over baselines is not trustworthy. The reader's verdict of CONDITIONAL is appropriate, and our concern does not change it; the paper should be accepted only after this check is performed and reported. No other more serious concern emerged: the WalkGNN formulation is coherent, the framework is clearly described, and the ablation study supports the architectural choices. The main issue is evaluation integrity, not the method itself.","tokens_in":13152,"tokens_out":8130,"duration_ms":77128,"concrete_test":"Using the released Ego-VK dataset (or a re-requested version with anonymized global user IDs), compute the fraction of users appearing in both training and test ego-nets, and the distribution of pairwise node overlap between train and test ego-nets. Then create a user-disjoint split: assign each ego-net to a split by hashing the ego user's global ID, and ensure no user appears in more than one split by moving all ego-nets that contain any held-out user into the test set. Retrain WalkGNN and PPGN under this split with the same hyperparameters and report ndcg@5 and 95% confidence intervals. If WalkGNN's margin over PPGN remains outside the confidence intervals, the concern is resolved; if the margin shrinks to within noise, the offline superiority claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that WalkGNN outperforms all baselines offline and that a live A/B test shows business metric growth. The offline half rests on a random split of ego-nets (Section 4.1, Dataset). The paper asserts that all ego-nets are 'completely independent' because node IDs are ordinal per ego-net, but this only removes global identity; it does not remove statistical dependence. Ego-nets are overlapping subgraphs of the same VK graph: the same user appears as a neighbor in many ego-nets, and the same candidate pair (u,v) appears in the ego-net of every common friend. With a random split, structurally near-identical ego-nets—often differing by only one or two nodes—can fall on both sides of the train/test boundary. A learned model, especially an expressive one like WalkGNN, can then memorize common structural motifs that signal a new friendship and report optimistic ndcg@5. The no-attribute WalkGNN result (0.035±0.003) is within error bars of PPGN (0.034±0.003), so the claimed 20% relative gain in the full-attribute setting is particularly sensitive to this bias: leakage would tend to help the most expressive model. The online A/B test is not affected by this split leakage, but the paper reports only a single 12% CTR figure with no confidence interval, significance test, or experiment duration, so it cannot independently carry the central claim. Thus the load-bearing assumption—that a random split of ego-nets yields an unbiased offline comparison—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Generalized Ego-network Friendship Score (GEFS) framework, which reduces link prediction on a large social graph to a series of per-ego-net tasks followed by an aggregation step, and introduces WalkGNN, a second-order GNN whose WalkConv layers propagate pairwise node-state matrices through edge-attribute-dependent filters. The authors also release Ego-VK, a dataset of VK ego-nets with heterogeneous temporal edge attributes, and report offline experiments showing WalkGNN outperforming heuristics and GNN baselines on Ego-VK and Yeast, together with a live A/B test at VK reporting a 12% increase in friend-request CTR. The central claims are that the framework is scalable, that WalkGNN learns link-formation laws from structure alone, and that the proposed model improves a deployed recommender system.","tokens_in":13465,"tokens_out":2761,"duration_ms":27155,"significance":"If the offline result is unbiased, the paper makes a useful industrial contribution: a scalable recipe for applying supervised graph-level models to friend suggestion, a new architecture that handles heterogeneous, dynamic, featureless graphs without node identifiers, and a publicly available benchmark dataset. The code and dataset are open-sourced, which is a concrete strength. However, the offline evaluation rests on an untreated potential information-leakage problem: ego-nets drawn from the same underlying social graph are not statistically independent under a random split, so the reported margin over PPGN (0.090 ± 0.004 vs 0.075 ± 0.004) may be inflated. The no-attribute comparison (0.035 vs 0.034) is well within the confidence intervals, and the online A/B report lacks the statistical detail needed to independently corroborate the headline claim. The framework and model remain plausible and worth publishing, but the current evidence does not yet establish that WalkGNN outperforms all baselines.","major_comments":[{"comment":"The paper asserts that 'all ego-nets are completely independent' because node IDs are ordinal per ego-net. This is not a valid statistical independence argument: ego-nets of the same VK graph overlap in users and edges, and a random split can place structurally near-identical ego-nets in both training and test sets. The model could then memorize common local motifs rather than learn transferable link-formation laws, and the reported 20% relative gain over PPGN would be optimistically biased. Please quantify the overlap between ego-nets across the train/test boundary (e.g., fraction of shared nodes or shared candidate pairs), or re-run the evaluation with a node-disjoint or time-disjoint split, or otherwise provide evidence that the architecture does not exploit cross-split similarity.","section":"Section 4.1, Dataset"},{"comment":"The claim that WalkGNN 'outperforms the next-best solution by 20% and 3%' is not supported by the reported numbers in the no-attribute condition: WalkGNN achieves 0.035 ± 0.003 and PPGN 0.034 ± 0.003, so the 3% difference lies entirely within the confidence intervals and is not a statistically meaningful improvement. Please provide paired significance tests or error bars that account for paired ego-nets, and restrict the overstatement to the edge-attribute setting where a clear separation exists (0.090 ± 0.004 vs 0.075 ± 0.004).","section":"Section 4.1, Table 2 and Results"},{"comment":"The online experiment reports a single point estimate of a 12% increase in friend-request CTR, with no confidence interval, significance test, number of users per arm, experiment duration, or description of how the audience was split. Given that the abstract cites the A/B test as evidence of 'growth of business metrics', the manuscript should at least provide a standard error or p-value and state whether the 12% was the sole pre-registered metric. Without this information, the online result cannot independently carry the paper's central claim.","section":"Section 4.2, Online Experiments"}],"minor_comments":[{"comment":"The variable name 'BloomF itler' contains a typo; it should be 'BloomFilter'.","section":"Algorithm 1"},{"comment":"The sentence 'Each type of edge have numerical characteristic' has a subject-verb agreement issue; it should be 'Each type of edge has a numerical characteristic'.","section":"Section 4.1, Dataset"},{"comment":"The relation between node features and edge attributes is stated twice in slightly different ways: first 'Natural node attributes are not used', then 'As attributes of nodes, we use attributes of edges with ego node in forward and backward directions'. Please clarify once and reconcile the two statements.","section":"Section 4.1, Dataset"},{"comment":"The column header 'EGO-VK- NO-ATTR' contains an awkward spacing/hyphenation; it would be clearer as 'EGO-VK (no attrs)'.","section":"Section 4.1, Table 2"},{"comment":"The ablation table does not report the no-attribute result for the same configuration as the base model? Specifically, 'WALK GNN-NO-E-ATTR' uses 6 blocks, which is fine, but the earlier no-attribute WalkGNN result in Table 2 (0.035 ± 0.003) differs from the ablation no-attribute result (0.051 ± 0.003). Please explain this discrepancy, since the same configuration should yield the same number unless hyperparameters or preprocessing differ.","section":"Section 4.3, Ablation Study"}],"recommendation":"major_revision","confidential_remarks":"The paper is a single-author industry manuscript with a strong practical component and a useful open-source dataset. The main risk is that the offline evaluation is not yet rigorous enough for a journal-level claim of superiority; the data-leakage issue is fixable in principle (e.g., by reporting a node-disjoint split or measuring overlap), and the online A/B needs basic statistical reporting. I would not recommend rejection because the framework and architecture are of genuine interest, but the current version requires substantial additional evidence before the central claims can be accepted. There is also a question of fit: the manuscript mixes a systems/engineering contribution with an empirical ML claim; the revision should make clear which is being evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is worth a serious referee, but the offline evaluation has a real leakage risk that the authors haven't addressed, and the online result is too thinly reported to carry the claim by itself.\n\nWhat's new and good: The Generalized Ego-network Friendship Score framework is a clean, useful formalization of Epasto et al.'s ego-net approach; it turns the large-graph problem into a set of low-scale graph-level link prediction tasks. WalkGNN's WalkConv layer—per-edge MLP filters applied to pair-state matrices—is a genuine architectural contribution, not a repackaging. The Ego-VK dataset is a reasonable new benchmark for featureless, heterogeneous, temporal ego-net link prediction. The ablation is honest: edge attributes are the main driver, and the paper doesn't oversell the no-attribute result, which is within error bars of PPGN. The Yeast transfer result is a nice sanity check. The math is straightforward, the O(n^3 d^2) complexity is stated, and the related-work coverage is adequate.\n\nThe soft spot is the split. The paper says ego-nets are 'completely independent' because node IDs are ordinal per ego-net, but that only removes global identity. The same user and the same candidate pair appear in many ego-nets; a random split does not prevent near-identical local structures from appearing in both train and test. An expressive model like WalkGNN is exactly the one that would benefit from memorizing those motifs, so the 20% relative gain over PPGN in the full-attribute setting may be inflated. The online A/B is the natural counterweight, but one 12% CTR number with no confidence interval, duration, or significance test is not enough. The GitHub link has no commit hash, so I can't verify reproduction.\n\nNone of this is fatal. The framework and dataset are valuable regardless. I'd send it to review with the expectation of major revisions on evaluation: filter or measure overlap across splits, and report more online detail. For anyone working on scalable supervised link prediction, this is worth reading and citing.","headline":"A genuinely new ego-net GNN architecture and a useful new dataset, but the offline evaluation split risks structural leakage and the online result is too thin to carry the claim alone.","tokens_in":14003,"tokens_out":3240,"would_cite":true,"duration_ms":28220,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Walking the ego-net: a second-order graph neural network that scores friend pairs inside each user's local neighborhood, beats the production heuristic offline, and lifts friend-request clicks by 12% in a live A/B test.","keywords":["friend suggestion","ego-net","graph neural network","link prediction","WalkGNN","Ego-VK","A/B test","social network"],"falsifier":"Re-run the Ego-VK experiment with a split that guarantees no node appears in both a training ego-net and a test ego-net, and also report the fraction of test ego-nets that share neighbors with the training pool; if WalkGNN's offline advantage over PPGN (0.090 vs 0.075) largely disappears under such a split, the central offline claim would be shown to rely on memorized local structures.","tokens_in":12906,"feed_emoji":"🤝","tokens_out":6127,"duration_ms":50825,"temperature":0.7,"pith_summary":"The paper argues that friend suggestions on giant social graphs can be made both scalable and accurate by never thinking about the whole graph at once: cut it into each user's ego-net, run a supervised model on each small neighborhood, then paste the per-neighborhood scores together. It introduces WalkGNN, a second-order graph neural network that learns link-formation rules from a pair-of-nodes state filtered through edge attributes, and the Ego-VK dataset of 61,808 real ego-nets for measuring the task. On Ego-VK the model reaches ndcg@5 = 0.090 versus 0.075 for the strongest baseline, and a live A/B test on the VK recommendation block reports a 12% increase in friend-request click-through rate. The payoff, if the claim holds, is a practical route to supervised link prediction on social graphs with hundreds of millions of users.","feed_headline":"Ego-net GNN beats baselines and lifts friend requests 12%","feed_subtitle":"Supervised per-neighborhood ranking replaces heuristics and scales to billion-edge social graphs.","key_machinery":"The load-bearing object is WalkConv, a layer that treats each edge as an information filter: an MLP maps the edge's attribute vector (friendship age, message counts, likes, profile visits, direction) to a $[d \\times d]$ matrix, and the state of every ordered pair of nodes is advanced by summing, over each incoming edge, the state of the source pair multiplied by that edge's filter, scaled by $1/d$. Stacked with residual connections and an output MLP, WalkConv builds pair embeddings that encode the local walk structure of the ego-net without any learned node identifiers, which is what makes the model usable on anonymized, independent ego-nets. The surrounding framework contributes the decomposition that makes the model deployable: ego-net construction via a distributed triangle-counting algorithm with a Bloom filter, per-ego inference as a map step, and out-ego aggregation by sum or max.","core_discovery":"The central discovery is that graph-level link prediction inside ego-nets, treated as an independent supervised task on heterogeneous, dynamic, featureless graphs, is learnable by a second-order GNN that propagates pairwise states through edge-specific linear filters, and that aggregating these local scores across common neighbors produces better friend suggestions than the deployed heuristic. The paper states this as the Generalized Ego-network Friendship Score framework: an in-ego model maps an ego-net to a pairwise relevance matrix, and an out-ego aggregation (sum or max) combines scores for pairs appearing in several ego-nets. WalkGNN instantiates the in-ego model: each WalkConv layer updates the $[n \\times n \\times d]$ state tensor as $W^{u,v}_{k+1} = \\frac{1}{d} \\sum_{(t,v,e) \\in E} W^{u,t}_{k} \\times \\mathrm{EdgeMLP}_k(e)$, turning every typed, timestamped edge into a linear filter on the propagated relationship state. The paper reports that this outperforms Adamic-Adar, GIN, and PPGN on Ego-VK and on the adapted Yeast dataset, and that the online A/B test shows a 12% increase in friend-request CTR against the production friendship-score baseline.","pith_inferences":["The framework is not tied to friendship: any large graph whose edges carry typed, timestamped attributes and where local neighborhoods are informative could use the same in-ego/out-ego decomposition, so the method suggests a template for supervised link prediction on billion-edge graphs in other domains.","The strongest offline results depend on the assumption that randomly split ego-nets do not leak; a natural robustness check is to re-run Ego-VK with a split that removes any test ego-net sharing nodes with training ego-nets, and if the gap narrows, part of the reported gain would be attributed to memorization rather than learned link-formation laws.","The 12% online CTR gain is a business metric, not a direct measure of friendship precision; a follow-up could examine whether the model increases accepted suggestions and long-term retention, which would test whether the learned relevance scores align with genuine link formation.","WalkGNN's O(n^3 d^2) per-layer cost is a natural bottleneck; future versions could sparsify the dense state tensor or use neighborhood sampling to push the same idea to even larger ego-nets, though the paper already caps ego-nets at 300 nodes."],"forward_implications":["On Ego-VK, WalkGNN with edge attributes reaches ndcg@5 = 0.090 ± 0.004, a 20% relative improvement over the best baseline (PPGN at 0.075 ± 0.004), and a 156% improvement over its own edge-attribute-free version.","The framework makes complex supervised models deployable at industrial scale: all phases fit the MapReduce paradigm, and the production pipeline on VK runs daily on the full user graph in about 10 hours on 3k cores and 3TB RAM.","A live A/B test on the VK People You May Know block reports a 12% increase in friend-request CTR relative to a production friendship-score baseline.","Because the in-ego model learns from topology rather than node identities, the same trained model transfers to graphs outside the social domain; on the adapted Yeast dataset it reaches ndcg@5 = 0.720 versus 0.642 for PPGN.","Edge attributes carry most of the information: ablating them drops Ego-VK accuracy by more than 50% (from 0.090 to 0.051), while removing node features costs little (0.090 to 0.086)."],"supporting_citations":[{"why":"Supplies the ego-net community-mining idea that this framework generalizes and the production baseline it beats online.","marker":"[6]"},{"why":"Supplies PPGN, the provably powerful 2-GNN used as the strongest higher-order offline baseline.","marker":"[44]"},{"why":"Supplies prior work in the node-pair representation paradigm that motivates WalkGNN's design.","marker":"[8]"},{"why":"Supplies the analysis of GNN expressive power that motivates higher-order models over first-order message passing.","marker":"[28]"},{"why":"Supplies evidence that first-order GNNs rely on rich node features, which social graphs lack.","marker":"[12]"},{"why":"Supplies the Bloom filter used in the distributed ego-net construction algorithm to avoid false-positive triangles.","marker":"[45]"},{"why":"Supplies the pairwise ranking loss used for training WalkGNN.","marker":"[48]"},{"why":"Supplies the Yeast molecular-graph dataset used for the out-of-domain transfer evaluation.","marker":"[46, 47]"}],"fun_headline_variants":["WalkGNN lifts friend-request clicks 12% in live test","Ego-net GNN beats heuristics for friend suggestions","Per-ego-net supervised ranking scales to billion-edge graphs","WalkGNN: supervised ego-net link prediction outperforms baselines","Friend suggestions via ego-net GNN: 12% CTR gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that randomly dividing ego-nets into training, validation, and test sets prevents information leakage, even though in a real social graph ego-nets overlap heavily and the paper does not filter overlapping nodes across splits or measure how similar train and test ego-nets are.","fun_headline_variants_meta":{"raw":{"variants":["WalkGNN lifts friend-request clicks 12% in live test","Ego-net GNN beats heuristics for friend suggestions","Per-ego-net supervised ranking scales to billion-edge graphs","WalkGNN: supervised ego-net link prediction outperforms baselines","Friend suggestions via ego-net GNN: 12% CTR gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1453,"prompt_tokens":1037,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":653,"tokens_out":416,"duration_ms":4101,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:29:07.901526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Ego-VK experiment with a split that guarantees no node appears in both a training ego-net and a test ego-net, and also report the fraction of test ego-nets that share neighbors with the training pool; if WalkGNN's offline advantage over PPGN (0.090 vs 0.075) largely disappears under such a split, the central offline claim would be shown to rely on memorized local structures.","supporting_citations":[{"cited_title":"Mirrokni, Ismail Sebe, Ahmed Taei, and Sunita Verma","cited_arxiv_id":null,"evidence_quote":"Supplies the ego-net community-mining idea that this framework generalizes and the production baseline it beats online."},{"cited_title":"Provably powerful graph networks","cited_arxiv_id":null,"evidence_quote":"Supplies PPGN, the provably powerful 2-GNN used as the strongest higher-order offline baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies prior work in the node-pair representation paradigm that motivates WalkGNN's design."},{"cited_title":"A fair comparison of graph neural networks for graph classification","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that first-order GNNs rely on rich node features, which social graphs lack."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pairwise ranking loss used for training WalkGNN."}],"review_version":1}