Pith. sign in

REVIEW 6 major objections 5 minor 25 references

Temporal Analysis of Reddit Networks via Role Embeddings

T0 review · 6 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Reddit users change roles often; community roles hold.

desk verdict Short, honest transfer of diachronic alignment to role embeddings; the community-level 'static roles' claim rests on an unvalidated sign-flip correction. read the letter →

arxiv 1908.05192 v1 pith:PC6TK3GS submitted 2019-08-14 cs.SI cs.LG

classification cs.SIcs.LG
keywords temporalnetworksroleembeddingsstruc2vecembeddingalignmentRedditstructuralequivalenceorthogonalProcrustescommunityroles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the techniques used to track how word meanings drift over time can be turned on the structural roles people play in online communities. Applying the role-embedding algorithm struc2vec to three-month windows of Reddit interaction networks, and aligning the resulting spaces, the authors measure how much individual users and whole communities move in role space. Their central finding is that individual users frequently change structural roles, while the set of community-level roles remains comparatively stable across nine months. They also report preliminary differences between loyal and vagrant subreddits, with loyal communities showing more dispersed role clusters. If the finding holds, it would mean that online communities have durable role structures even as the people occupying those roles rotate through them.

What carries the argument

The carrying mechanism is the temporal role-embedding pipeline. struc2vec embeds each node so that structurally equivalent users, meaning users whose network neighborhoods have similar degree sequences and topologies, land close together regardless of their distance in the graph. Consecutive three-month windows are aligned with normalized orthogonal Procrustes rotation, which finds the optimal rotation of one embedding space toward another using the overlapping set of the 100 most frequent users. A PCA-based sign-flip correction repairs cases where eigenvector signs flip and community roles mirror instead of overlap. User role change is then measured by cosine distance between aligned vectors, and community role change by Euclidean distance between k-means cluster centroids found through the Elbow method, with silhouette scores quantifying how sharply roles are defined.

What would settle it

Take the same 29 subreddits, shuffle the timestamps of user comments so any temporal structure is destroyed, and run the full pipeline; if community-level centroid distances in the shuffled data are as small as those reported for the real temporal order, then the static community roles are an artifact of the alignment method rather than a property of the communities.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that user-level and community-level temporal role dynamics decouple: individual Reddit users' role embeddings move substantially between consecutive time windows, whereas the distances between community role cluster centroids stay small. The authors observed this across 29 subreddits previously classified as loyal or vagrant, using cosine distance for individual role change, Euclidean distance between aligned cluster centroids for community role change, and silhouette scores for cluster definition. A secondary result is that community roles in loyal subreddits appear more dispersed, while vagrant subreddits show more compact clusters, though the authors note the differences are small and would need a larger sample to establish significance.

Load-bearing premise

The whole analysis stands on the assumption that the procedure used to line up embedding spaces from different time periods, which relies on the 100 most frequent users to anchor the rotation and then adjusts mirrored components by hand, produces distances that reflect real role changes rather than alignment artifacts.

Editorial extensions

If this is right

  • Community-level role stability means a subreddit's coarse role structure could be monitored as a single signal, without tracking each user.
  • Diachronic alignment methods from word embeddings transfer to structural social-network roles, opening the same toolkit to network scientists.
  • If vagrant and loyal communities differ in role dispersion, subreddit type might be characterizable from structure alone, without reading content.
  • A community could support the same cast of roles for long periods even as the individuals filling those roles turn over.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would test whether the static community roles correspond to interpretable recurring behaviors, such as question-answerer, instigator, or newcomer, by comparing cluster centroids to hand-labeled user activity.
  • If the stability result is robust, platforms could use shifts in community role centroids as an early-warning signal for structural change, before individual-level churn becomes visible.
  • The same alignment pipeline could be tested on other interaction networks, such as Stack Exchange or Twitter reply graphs, to see whether static community roles are a Reddit-specific phenomenon or a general property of online communities.
  • A synthetic test with planted role changes would separate alignment artifacts from true role drift; the current data alone cannot rule out the former.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes a method for analysing temporal changes in structural roles on Reddit using the struc2vec role-embedding algorithm. The authors partition nine months of directed interaction data from 29 subreddits into three three-month windows, embed each window, and align the embedding spaces with orthogonal Procrustes rotations computed on the overlapping set of the 100 most frequent users. A PCA sign-flip correction is applied when the overlapping users are concentrated in a small region, as in the vagrant subreddit r/CityPorn. User-level role variation is measured by cosine distances between a user's embeddings across windows, and community-level role variation is measured by Euclidean distances between aligned cluster centroids obtained by PCA, elbow-based k-means, and 1-nearest-neighbour matching. The central claim is that individual users of Reddit change roles frequently, while universal community-level roles remain relatively static, and that there may be differences between loyal and vagrant subreddits.

Significance. If the central claim were fully supported, the paper would make a useful methodological transfer from diachronic word embeddings to network role analysis: it addresses the non-trivial problem of comparing role-embedding spaces over time and it studies a sociologically interesting question on a publicly available dataset. The use of an established role-embedding algorithm (struc2vec), a public interaction dataset, and the inclusion of a pre-alignment baseline in Fig. 3 are strengths. However, the statistical and methodological support is currently preliminary. The small sample of subreddits, the lack of significance tests, the unresolved class-label inconsistency in Table 1, and the unvalidated post hoc sign-flip correction mean that the findings are more a promising proof-of-concept than a settled empirical result. The authors themselves repeatedly qualify the results as preliminary, which is commendable but also indicates that the central claim is not yet established.

major comments (6)
  1. [§3.1, Fig. 3] The orthogonal Procrustes rotation is fitted on the overlapping high-frequency users, and the same users' embeddings are then used in §3.2 to compute individual-role cosine distances. This makes the individual-level variation estimate directly dependent on the alignment target. The evaluation in Fig. 3 also uses these same anchor users, so it cannot validate the alignment for non-anchor users. The authors should split the overlapping set into an anchor set for fitting and a held-out set for evaluation, and report distance distributions for both sets.
  2. [§3.1, Fig. 2(c)–(d)] The PCA sign-flip correction has no independent criterion for which principal components are 'equivalent', and it is applied specifically to make the two time points agree. Because the community-level centroid distances in §3.2 and Fig. 4(b) are computed on these sign-corrected spaces, the correction can only reduce the measured community distances relative to an arbitrary sign choice. A null model is needed: for example, random sign assignments or permutation of user identities across time windows, with a comparison of the resulting centroid-distance distribution. Without such a baseline, the 'static community roles' result is not distinguishable from an artifact of the alignment procedure.
  3. [§3, Table 1] The text states that the dataset consists of 16 loyal and 13 vagrant subreddits, while Table 1 reports 13 loyal and 16 vagrant subreddits. This inconsistency is unresolved and directly affects the class-level comparisons in Fig. 4. The authors must state which count is correct and, if necessary, recompute the loyal-versus-vagrant breakdown and all subsequent comparisons.
  4. [§3, 'we first select the 100 highest frequency participants...'] The size of the overlap set across all windows is never reported. In vagrant subreddits such as r/CityPorn, the anchor users occupy a small portion of the embedding space (Fig. 2(b)), and in all cases the anchors are by construction the most persistent users, not the transient users whose role fluctuation is the object of study. The authors should report the overlap counts for each subreddit and perform a sensitivity analysis with alternative anchor definitions (e.g., random users or lower frequency thresholds).
  5. [§4, Fig. 4] The claimed differences between loyal and vagrant subreddits, and the contrast between user-level and community-level role variation, are presented without confidence intervals, effect sizes, or significance tests. With only 13 and 16 subreddits per class, the apparent patterns in Fig. 4(a) could be noise. The text itself states that the differences are small and that further subreddits need to be examined. The authors should add permutation or bootstrap tests for the class comparison and for the user-versus-community distance contrast.
  6. [§3.2, community-role distance] The community-level distance is computed by using the Elbow method to choose the cluster count and then using 1-nearest-neighbour to measure the distance to the closest aligned centroid. This protocol does not define a matching between roles: taking the closest pair can yield a small distance even when the role structure of a subreddit has changed substantially. The authors should specify an explicit one-to-one matching (for example, Hungarian assignment on all centroids) and report all pairwise centroid distances, rather than the minimum, to support the static-community claim.
minor comments (5)
  1. [Throughout] The manuscript contains numerous typographical and OCR-like artefacts ('f_ield', '/f_indings', 'T_his', 'adpoting', 'emebeddings', 'amoung'); these should be cleaned before publication.
  2. [§4, Fig. 4] It is unclear whether the subplots are box plots or bar charts; axis labels, error bars, and the number of subreddits represented per box should be stated explicitly.
  3. [§3, data description] The description of the chain-based interaction networks and the parameters used for struc2vec (e.g., number of walks, walk length, window size) is missing, which prevents reproduction of the embeddings.
  4. [Fig. 1] The subreddit name is given as 'r/ACMillan' in the caption and 'r/ACMilan' in the text; the spelling should be made consistent.
  5. [Abstract] The phrase 'we analysis how roles evolve' should be 'we analyse how roles evolve', and the sentence structure in §3 introducing the three temporal windows is grammatically incomplete.

Circularity Check

2 steps flagged · score 4.0 of 10

The community-level 'relatively static roles' result is partly forced by the post hoc sign-flip alignment, and the Procrustes fit minimizes the same user-level distances that are later reported as role variation.

  1. fitted input called prediction [Section 3.1 and Section 3.2 (Temporal Role Alignment; Measuring Role Variation Across Time)]
    "In particular, the orthogonal Procrustes rotation between spaces is computed by mapping the overlapping sets of users to each other. ... To detect changes in an individual’s role across time, we compute the cosine distance between an actor’s embedding at time t and t + Δ: 1−cos(v_i^t, v_i^{t+Δ})."

    The rotation matrix is fitted by minimizing squared distances between the overlapping users, and the same overlapping users are then used to compute the individual cosine-distance metric. Thus the measured individual role variation is a residual after the alignment has already been optimized to make those very users close. The finding that user roles still fluctuate is conservative rather than forced, since the alignment works against it, but the numerical fluctuation values and the later contrast with community roles inherit the fit.

  2. fitted input called prediction [Section 3.1 and Section 4 (Results, Fig. 4(b))]
    "To resolve this, further alignment of roles is applied by changing the signs of equivalent principal components to agree if they do not already. ... Fig.4(b) depicts how distances between role cluster centroids for both loyal and vagrant subreddits remain small, indicating they are similar to each other."

    The sign-flip correction is chosen after observing that the PCA spaces are mirrored, with the explicit goal of making the two time points 'agree.' The community-role centroid distances in Fig.4(b) are then measured in these sign-corrected spaces. For affected subreddits such as r/CityPorn, the pre-correction projections show mirrored clusters, so the correction is what produces the apparent overlap and hence the small centroid distances. No held-out anchors, random-sign baseline, or null model is given to show that the measured 'relatively static' community roles are not an artifact of this post hoc alignment.

full rationale

The paper is not globally circular: the struc2vec embeddings, the use of overlapping users for Procrustes alignment, and the external Reddit loyalty data from Hamilton et al. are independent inputs, and the self-citations in the paper are not load-bearing. The user-level fluctuation result is also not forced, because Procrustes minimizes the very distances later reported, so the high residual distances are conservative evidence of fluctuation. However, the central community-level claim is partly constructed by the pipeline: community roles are defined on PCA projections of already-aligned spaces, and when alignment fails, a secondary sign-flip correction is applied specifically 'to agree.' The same corrected spaces are then used to measure the small centroid distances that support 'community roles remain relatively static.' Because no independent baseline quantifies the effect of this corrective alignment, the community-level finding is partially a fitted-input-called-prediction artifact rather than an independent discovery. This warrants a moderate circularity score of 4, not a higher score, since the method has independent external data and the user-level component retains real content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central measurements depend on a specific embedding algorithm, alignment anchors, cluster counts, and distance metrics. None of these are benchmarked against ground-truth role changes or varied for sensitivity, so the reported distances may reflect pipeline choices as much as real role dynamics.

free parameters (4)
  • top-100 most frequent users per window = 100
    Each time window is restricted to the 100 highest-frequency users and their overlap across windows; this choice defines the study population and the alignment anchors but is not justified.
  • embedding dimension = 128
    struc2vec is run with 128 dimensions for every subreddit; distance and clustering results likely depend on this dimension.
  • number of role clusters per embedding space = Elbow method, varies by subreddit and window
    The Elbow criterion on 2D PCA projections selects k before centroid matching; this is a data-dependent fitted number and no stability check is reported.
  • k in k-nearest-neighbour centroid matching = 1
    Community role variation is computed as the Euclidean distance to the closest aligned centroid with k=1; no sensitivity analysis is given.
assumptions (4)
  • domain assumption Structural role equivalence is captured by struc2vec embeddings.
    Sections 2 and 3 assume that embedding users by their topological context captures meaningful roles and that distances in the embedding space are comparable across windows.
  • domain assumption Overlapping top-100 users provide a sufficient anchor set for orthogonal Procrustes alignment.
    Section 3.1 uses the overlap of top-100 users to compute the Procrustes rotation; this presumes the overlap is large and representative enough to define a trustworthy rotation.
  • ad hoc to paper PCA eigenvector sign flips can be corrected by matching equivalent principal components.
    Section 3.1 changes signs of principal components when overlapping users are concentrated and eigenvectors flip; no formal criterion is given for when this correction is valid.
  • domain assumption Elbow-determined k-means clusters on 2D PCA projections represent community roles.
    Section 3.2 summarizes community roles by reducing 128-dimensional embeddings to 2 dimensions, choosing k via the Elbow method, and matching centroids with 1NN; this reduction can distort role structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Analysis of Reddit Networks via Role Embeddings." pith.science (2026). https://pith.science/paper/PC6TK3GS

@misc{pith2026190805192,
  author       = {Pith},
  title        = {Pith review of: Temporal Analysis of Reddit Networks via Role Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PC6TK3GS}},
  note         = {Machine review of arXiv:1908.05192}
}
read the original abstract

Inspired by diachronic word analysis from the field of natural language processing, we propose an approach for uncovering temporal insights regarding user roles from social networks using graph embedding methods. Specifically, we apply the role embedding algorithm, struc2vec, to a collection of social networks exhibiting either "loyal" or "vagrant" characteristics derived from the popular online social news aggregation website Reddit. For each subreddit, we extract nine months of data and create network role embeddings on consecutive time windows. We are then able to compare and contrast how user roles change over time by aligning the resulting temporal embeddings spaces. In particular, we analyse temporal role embeddings from an individual and a community-level perspective for both loyal and vagrant communities present on Reddit.

Figures

Figures reproduced from arXiv: 1908.05192 by the authors.

Figure 1
Figure 1. Visualisation of the loyal subreddit ‘r/ACMillan’ before and a er alignment and dimension reduction. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualisation of the vagrant subreddit ‘r/CityPorn’ before and a er alignment and dimension reduction. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Cosine Similarity results for alignment evaluation. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: ‡e temporal user and community role dynamics observed via three di‚erent metrics for comparing similarity: [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Subreddit PCA projections across time. role clusters are more dispersed in comparison to it’s vagrant coun￾terpart where clusters are spread and tightly compact. Œe average SilhoueŠe scores, Fig.4, indicate that it’s not an isolated scenario. However, the di‚erences ar…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debi- asing word embeddings. In Advances in Neural Information Processing Systems . 4349–4357

  2. [2]

    Hongyun Cai, Vincent W Zheng, and Kevin Chang. 2018. A comprehensive survey of graph embedding: problems, techniques and applications. IEEE Trans- actions on Knowledge and Data Engineering (2018)

  3. [3]

    Sandro Cavallari, Vincent W Zheng, Hongyun Cai, Kevin Chen-Chuan Chang, and Erik Cambria. 2017. Learning community embedding with community detection and node embedding on graphs. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management . ACM, 377–386

  4. [4]

    Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. 2018. Learning Structural Node Embeddings via Diffusion Wavelets. In KDD. ACM

  5. [5]

    Siobh´an Grayson, Maria Mulvany, Karen Wade, Gerardine Meaney, and Derek Greene. 2017. Exploring the Role of Gender in 19th Century Fiction /T_hrough the Lens of Word Embeddings. In International Conference on Language, Data and Knowledge. Springer, 358–364

  6. [6]

    Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 855–864

  7. [7]

    Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems . 1025–1035

  8. [8]

    William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 1489–1501

Show all 25 references
  1. [9]

    Hamilton, Justine Zhang, Cristian Danescu-Niculescu-Mizil, Dan Jurafsky, and Dan Jurafsky

    William L. Hamilton, Justine Zhang, Cristian Danescu-Niculescu-Mizil, Dan Jurafsky, and Dan Jurafsky. 2017. Loyalty in online communities. InInternational AAAI Conference on Weblogs and Social Media, Vol. 2017. NIH Public Access, 540

  2. [10]

    Keith Henderson, Brian Gallagher, Tina Eliassi-Rad, Hanghang Tong, Sugato Basu, Leman Akoglu, Danai Koutra, Christos Faloutsos, and Lei Li. 2012. RolX: Structural Role Extraction & Mining in Large Graphs. In Proceedings of the 18th ACM SIGKDD International Conference on Kn...

  3. [11]

    Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Sta- tistically Signi/f_icant Detection of Linguistic Change. InProceedings of the 24th International Conference on World Wide Web (WWW ’15). International World Wide Web Conferences Steering Commi/t_tee, Re...

  4. [12]

    Alex Leavi/t_t. 2015. /T_his is a /T_hrowaway Account”: Temporary Technical Identi- ties and Perceptions of Anonymity in a Massive Online Community. In Proceed- ings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15). ACM, New Y...

  5. [13]

    Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, Nov (2008), 2579–2605

  6. [14]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems . 3111–3119

  7. [15]

    Edward Newell, David Jurgens, Haji Mohammad Saleem, Hardik Vala, Jad Sassine, Caitrin Armstrong, and Derek Ruths. 2016. User Migration in Online Social Networks: A Case Study on Reddit During a Period of Community Unrest. In ICWSM. 279–288

  8. [16]

    Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. InInternational conference on machine learning. 2014–2023

  9. [17]

    SS Nishana and Subu Surendran. 2013. Graph embedding and dimensionality reduction-a survey. International Journal of Computer Science & Engineering Technology (IJCSET) 4, 1 (2013), 29–34

  10. [18]

    Ribeiro, Pedro H.P

    Leonardo F.R. Ribeiro, Pedro H.P. Saverese, and Daniel R. Figueiredo. 2017. Struc2Vec: Learning Node Representations from Structural Identity, In Pro- ceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Proceedings of the 23rd ACM S...

  11. [19]

    Rossi, Brian Gallagher, Jennifer Neville, and Keith Henderson

    Ryan A. Rossi, Brian Gallagher, Jennifer Neville, and Keith Henderson. 2013. Modeling Dynamic Behavior in Large Evolving Graphs. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM ’13) . ACM, New York, NY, USA, 667–676. h/t_tps://doi.o...

  12. [20]

    Philipp Singer, Fabian Flock, Clemens Meinhart, Elias Zeitfogel, and Markus Strohmaier. 2014. Evolution of Reddit: From the Front Page of the Internet to a Self-referential Community?. In International World Wide Web Conference. IW3C2, ACM, Seoul, Korea, 517–522

  13. [21]

    Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2016. Offline bilingual word vectors, orthogonal transformations and the inverted so/f_tmax. (2016)

  14. [22]

    /Q_uan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering 29, 12 (2017), 2724–2743

  15. [23]

    Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang. 2017. Community Preserving Network Embedding.. In AAAI. 203–209

  16. [24]

    Lekui Zhou, Yang Yang, Xiang Ren, Fei Wu, and Yueting Zhuang. 2018. Dynamic Network Embedding by Modeling Triadic Closure Process. In AAAI

  17. [25]

    Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing . 1393–1398

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.