{"id":"19f1c093-c122-4b68-a5fe-46f5624cb057","arxiv_id":"1908.05192","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying diachronic embedding alignment to struc2vec role embeddings suggests Reddit users change structural roles over time more than their communities' overall role structures do.","lead":"This paper adapts a method used to track word meaning over time and applies it to the structural roles of Reddit users, comparing how roles shift across three-month windows. It reports that individual users change roles frequently while the overall community-role structure stays fairly stable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'static community roles' result may be an artifact of the sign-flip alignment step, which is applied before the centroid distances are measured and has no independent validation.","rationale":"The reader's weakest assumption correctly identifies the alignment procedure, but my concern sharpens one specific component: the post hoc PCA sign-flip correction. The paper itself concedes this correction is needed when overlapping users are localized, and then uses the corrected spaces to compute the very distances that support the central claim. This creates a circularity risk that is not addressed by the current evaluation. I do not think this is fatal; it is an addressable methodological gap, and the paper is honest about many of its limitations. The proposed conditional verdict remains appropriate: the central claim should not be accepted without a null-model or hold-out validation of the sign-flip step. My read therefore does not change the reader's verdict, which is why I mark the verdict as unchanged rather than rejecting the paper.","tokens_in":7909,"tokens_out":4166,"duration_ms":45064,"concrete_test":"Re-run the community-role pipeline on all 29 subreddits under three conditions: (i) Procrustes alignment only, without any sign-flip correction; (ii) the current sign-flip rule; (iii) a control where sign flips are randomly assigned on degree-preserving randomized networks. Compare the Fig. 4(b) centroid distances across conditions. If condition (iii) produces distances as small as condition (ii), or if condition (i) produces large distances across many subreddits, the static-community claim is an artifact of the sign-flip step. Additionally, estimate the sign-flip correspondence using a random half of the overlapping top-100 users and evaluate centroid overlap on the held-out half; if the correction does not transfer to the held-out anchors, it is overfit to the measured time points.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that community-level roles are relatively static while user-level roles fluctuate — is only as strong as the alignment that makes the two time points comparable. Section 3.1 reports that orthogonal Procrustes alone fails for vagrant subreddits such as r/CityPorn because the overlapping users occupy a small region of the space, causing PCA eigenvector signs to flip. The authors then 'resolve' this by 'changing the signs of equivalent principal components to agree if they do not already.' The community-level distances in Fig. 4(b) are computed on these sign-corrected PCA projections. Because the sign flips are chosen specifically to make the two time points agree, the correction can only reduce the measured centroid distances; without an independent criterion for what 'equivalent' means, or a held-out set of anchors, the finding of static community roles is at least partly an artifact of the alignment procedure. The paper provides no baseline for how small Fig. 4(b) distances would be under random sign choices, nor under a null model with permuted user identities. The measured contrast between volatile user roles and static community roles therefore depends on a free post hoc alignment parameter whose effect is never quantified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for analysing temporal changes in structural roles on Reddit using the struc2vec role-embedding algorithm. The authors partition nine months of directed interaction data from 29 subreddits into three three-month windows, embed each window, and align the embedding spaces with orthogonal Procrustes rotations computed on the overlapping set of the 100 most frequent users. A PCA sign-flip correction is applied when the overlapping users are concentrated in a small region, as in the vagrant subreddit r/CityPorn. User-level role variation is measured by cosine distances between a user's embeddings across windows, and community-level role variation is measured by Euclidean distances between aligned cluster centroids obtained by PCA, elbow-based k-means, and 1-nearest-neighbour matching. The central claim is that individual users of Reddit change roles frequently, while universal community-level roles remain relatively static, and that there may be differences between loyal and vagrant subreddits.","tokens_in":8138,"tokens_out":7610,"duration_ms":73182,"significance":"If the central claim were fully supported, the paper would make a useful methodological transfer from diachronic word embeddings to network role analysis: it addresses the non-trivial problem of comparing role-embedding spaces over time and it studies a sociologically interesting question on a publicly available dataset. The use of an established role-embedding algorithm (struc2vec), a public interaction dataset, and the inclusion of a pre-alignment baseline in Fig. 3 are strengths. However, the statistical and methodological support is currently preliminary. The small sample of subreddits, the lack of significance tests, the unresolved class-label inconsistency in Table 1, and the unvalidated post hoc sign-flip correction mean that the findings are more a promising proof-of-concept than a settled empirical result. The authors themselves repeatedly qualify the results as preliminary, which is commendable but also indicates that the central claim is not yet established.","major_comments":[{"comment":"The orthogonal Procrustes rotation is fitted on the overlapping high-frequency users, and the same users' embeddings are then used in §3.2 to compute individual-role cosine distances. This makes the individual-level variation estimate directly dependent on the alignment target. The evaluation in Fig. 3 also uses these same anchor users, so it cannot validate the alignment for non-anchor users. The authors should split the overlapping set into an anchor set for fitting and a held-out set for evaluation, and report distance distributions for both sets.","section":"§3.1, Fig. 3"},{"comment":"The PCA sign-flip correction has no independent criterion for which principal components are 'equivalent', and it is applied specifically to make the two time points agree. Because the community-level centroid distances in §3.2 and Fig. 4(b) are computed on these sign-corrected spaces, the correction can only reduce the measured community distances relative to an arbitrary sign choice. A null model is needed: for example, random sign assignments or permutation of user identities across time windows, with a comparison of the resulting centroid-distance distribution. Without such a baseline, the 'static community roles' result is not distinguishable from an artifact of the alignment procedure.","section":"§3.1, Fig. 2(c)–(d)"},{"comment":"The text states that the dataset consists of 16 loyal and 13 vagrant subreddits, while Table 1 reports 13 loyal and 16 vagrant subreddits. This inconsistency is unresolved and directly affects the class-level comparisons in Fig. 4. The authors must state which count is correct and, if necessary, recompute the loyal-versus-vagrant breakdown and all subsequent comparisons.","section":"§3, Table 1"},{"comment":"The size of the overlap set across all windows is never reported. In vagrant subreddits such as r/CityPorn, the anchor users occupy a small portion of the embedding space (Fig. 2(b)), and in all cases the anchors are by construction the most persistent users, not the transient users whose role fluctuation is the object of study. The authors should report the overlap counts for each subreddit and perform a sensitivity analysis with alternative anchor definitions (e.g., random users or lower frequency thresholds).","section":"§3, 'we first select the 100 highest frequency participants...'"},{"comment":"The claimed differences between loyal and vagrant subreddits, and the contrast between user-level and community-level role variation, are presented without confidence intervals, effect sizes, or significance tests. With only 13 and 16 subreddits per class, the apparent patterns in Fig. 4(a) could be noise. The text itself states that the differences are small and that further subreddits need to be examined. The authors should add permutation or bootstrap tests for the class comparison and for the user-versus-community distance contrast.","section":"§4, Fig. 4"},{"comment":"The community-level distance is computed by using the Elbow method to choose the cluster count and then using 1-nearest-neighbour to measure the distance to the closest aligned centroid. This protocol does not define a matching between roles: taking the closest pair can yield a small distance even when the role structure of a subreddit has changed substantially. The authors should specify an explicit one-to-one matching (for example, Hungarian assignment on all centroids) and report all pairwise centroid distances, rather than the minimum, to support the static-community claim.","section":"§3.2, community-role distance"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical and OCR-like artefacts ('f_ield', '/f_indings', 'T_his', 'adpoting', 'emebeddings', 'amoung'); these should be cleaned before publication.","section":"Throughout"},{"comment":"It is unclear whether the subplots are box plots or bar charts; axis labels, error bars, and the number of subreddits represented per box should be stated explicitly.","section":"§4, Fig. 4"},{"comment":"The description of the chain-based interaction networks and the parameters used for struc2vec (e.g., number of walks, walk length, window size) is missing, which prevents reproduction of the embeddings.","section":"§3, data description"},{"comment":"The subreddit name is given as 'r/ACMillan' in the caption and 'r/ACMilan' in the text; the spelling should be made consistent.","section":"Fig. 1"},{"comment":"The phrase 'we analysis how roles evolve' should be 'we analyse how roles evolve', and the sentence structure in §3 introducing the three temporal windows is grammatically incomplete.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short workshop-style contribution with a plausible and interesting idea, but the central empirical claim is currently under-supported. The most important issues are the unvalidated sign-flip correction for community-level alignment and the lack of any null model or significance testing. These are fixable within the scope of the manuscript if the authors add the requested analyses; if the null model shows that community-level distances are no smaller than under random sign choices or permuted identities, the main claim would need to be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a short, exploratory workshop piece that takes the diachronic word embedding alignment playbook and applies it to structural role embeddings on Reddit. The new bit is the transfer itself plus a PCA sign-flip fix; the headline claim is that individual user roles fluctuate while community-level role clusters stay put.\n\nWhat it does well: it uses a public dataset, describes the pipeline clearly, reports a baseline comparison for alignment quality, and is explicit that the findings are preliminary. That is the right attitude for this kind of work.\n\nThe soft spot is the load-bearing community-level result. For vagrant subreddits, the authors find that orthogonal Procrustes alone misaligns PCA because overlapping users occupy a small region, so they flip PCA eigenvector signs to make corresponding components agree. The community-level distances in Figure 4(b) are computed after that correction. Choosing signs to maximize agreement can only shrink distances; there is no independent criterion, no random-sign baseline, no null model. So the contrast between volatile users and static communities is partly baked into the alignment. This is not a minor quibble—it is the central claim.\n\nThere are other gaps, in proportion: Procrustes is anchored on the same top-100 overlapping users whose individual variation is then measured, which can suppress the variation you are trying to detect. There are no significance tests, the sample is small and hand-picked, and the main text says 16 loyal and 13 vagrant subreddits while Table 1 says 13 loyal and 16 vagrant. That inconsistency should have been caught.\n\nThat said, the paper is honest about its limitations, and the individual-level variation result is less affected by the sign-flip issue. The finding may survive proper robustness checks; it is not obviously wrong. The paper would benefit from a permutation test on user identities, a random-sign baseline for the PCA correction, and a resolution of the table mismatch.\n\nWho is this for? People working on temporal graph embeddings or computational social science on Reddit. It is worth a serious referee—I would send it out—but I would expect major revisions before the community-level claim is credible.","headline":"Short, honest transfer of diachronic alignment to role embeddings; the community-level 'static roles' claim rests on an unvalidated sign-flip correction.","tokens_in":8618,"tokens_out":2163,"would_cite":false,"duration_ms":20641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reddit users change roles often; community roles hold.","keywords":["temporal networks","role embeddings","struc2vec","embedding alignment","Reddit","structural equivalence","orthogonal Procrustes","community roles"],"falsifier":"Take the same 29 subreddits, shuffle the timestamps of user comments so any temporal structure is destroyed, and run the full pipeline; if community-level centroid distances in the shuffled data are as small as those reported for the real temporal order, then the static community roles are an artifact of the alignment method rather than a property of the communities.","tokens_in":7699,"feed_emoji":"🔄","tokens_out":5366,"duration_ms":52130,"temperature":0.7,"pith_summary":"This paper tries to establish that the techniques used to track how word meanings drift over time can be turned on the structural roles people play in online communities. Applying the role-embedding algorithm struc2vec to three-month windows of Reddit interaction networks, and aligning the resulting spaces, the authors measure how much individual users and whole communities move in role space. Their central finding is that individual users frequently change structural roles, while the set of community-level roles remains comparatively stable across nine months. They also report preliminary differences between loyal and vagrant subreddits, with loyal communities showing more dispersed role clusters. If the finding holds, it would mean that online communities have durable role structures even as the people occupying those roles rotate through them.","feed_headline":"Reddit users change roles often; community roles hold","feed_subtitle":"Nine months of role embeddings show individual volatility alongside stable community structures in loyal and vagrant subreddits.","key_machinery":"The carrying mechanism is the temporal role-embedding pipeline. struc2vec embeds each node so that structurally equivalent users, meaning users whose network neighborhoods have similar degree sequences and topologies, land close together regardless of their distance in the graph. Consecutive three-month windows are aligned with normalized orthogonal Procrustes rotation, which finds the optimal rotation of one embedding space toward another using the overlapping set of the 100 most frequent users. A PCA-based sign-flip correction repairs cases where eigenvector signs flip and community roles mirror instead of overlap. User role change is then measured by cosine distance between aligned vectors, and community role change by Euclidean distance between k-means cluster centroids found through the Elbow method, with silhouette scores quantifying how sharply roles are defined.","core_discovery":"On the paper's own terms, the discovery is that user-level and community-level temporal role dynamics decouple: individual Reddit users' role embeddings move substantially between consecutive time windows, whereas the distances between community role cluster centroids stay small. The authors observed this across 29 subreddits previously classified as loyal or vagrant, using cosine distance for individual role change, Euclidean distance between aligned cluster centroids for community role change, and silhouette scores for cluster definition. A secondary result is that community roles in loyal subreddits appear more dispersed, while vagrant subreddits show more compact clusters, though the authors note the differences are small and would need a larger sample to establish significance.","pith_inferences":["A natural extension would test whether the static community roles correspond to interpretable recurring behaviors, such as question-answerer, instigator, or newcomer, by comparing cluster centroids to hand-labeled user activity.","If the stability result is robust, platforms could use shifts in community role centroids as an early-warning signal for structural change, before individual-level churn becomes visible.","The same alignment pipeline could be tested on other interaction networks, such as Stack Exchange or Twitter reply graphs, to see whether static community roles are a Reddit-specific phenomenon or a general property of online communities.","A synthetic test with planted role changes would separate alignment artifacts from true role drift; the current data alone cannot rule out the former."],"forward_implications":["Community-level role stability means a subreddit's coarse role structure could be monitored as a single signal, without tracking each user.","Diachronic alignment methods from word embeddings transfer to structural social-network roles, opening the same toolkit to network scientists.","If vagrant and loyal communities differ in role dispersion, subreddit type might be characterizable from structure alone, without reading content.","A community could support the same cast of roles for long periods even as the individuals filling those roles turn over."],"supporting_citations":[{"why":"Supplies the curated Reddit interaction networks and the loyal/vagrant user definitions that structure the dataset.","marker":"[9]"},{"why":"Provides struc2vec, the algorithm that produces structural role embeddings for directed, weighted networks.","marker":"[18]"},{"why":"Supplies the diachronic word-embedding framework and cosine-similarity evaluation that the method adapts.","marker":"[8]"},{"why":"Provides the orthogonal transformation approach used to align embedding spaces.","marker":"[21]"},{"why":"Alternative role-embedding method compared during method selection.","marker":"[10]"},{"why":"Alternative role-embedding method compared during method selection.","marker":"[4]"}],"fun_headline_variants":["User roles shift, community roles hold on Reddit","Reddit users change roles, communities remain steady","Individuals move, communities hold: Reddit role study","Role embeddings reveal: users change, communities don't","Subreddit roles: individuals volatile, communities stable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis stands on the assumption that the procedure used to line up embedding spaces from different time periods, which relies on the 100 most frequent users to anchor the rotation and then adjusts mirrored components by hand, produces distances that reflect real role changes rather than alignment artifacts.","fun_headline_variants_meta":{"raw":{"variants":["User roles shift, community roles hold on Reddit","Reddit users change roles, communities remain steady","Individuals move, communities hold: Reddit role study","Role embeddings reveal: users change, communities don't","Subreddit roles: individuals volatile, communities stable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000953,"raw_usage":{"total_tokens":3991,"prompt_tokens":798,"completion_tokens":3193,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":3118}},"tokens_in":414,"tokens_out":3193,"duration_ms":22217,"temperature":1.0,"reasoning_tokens":3118,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:20:27.050730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 29 subreddits, shuffle the timestamps of user comments so any temporal structure is destroyed, and run the full pipeline; if community-level centroid distances in the shuffled data are as small as those reported for the real temporal order, then the static community roles are an artifact of the alignment method rather than a property of the communities.","supporting_citations":[{"cited_title":"Hamilton, Justine Zhang, Cristian Danescu-Niculescu-Mizil, Dan Jurafsky, and Dan Jurafsky","cited_arxiv_id":null,"evidence_quote":"Supplies the curated Reddit interaction networks and the loyal/vagrant user definitions that structure the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the diachronic word-embedding framework and cosine-similarity evaluation that the method adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the orthogonal transformation approach used to align embedding spaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Alternative role-embedding method compared during method selection."}],"review_version":1}