{"id":"eac2900d-dbf8-4fc1-9940-6555b13ac3f3","arxiv_id":"2602.02838","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Behavioral policies inferred from Reddit activity detect IRA-linked troll accounts with 94.9% median macro-F1, beating a 91.2% text-embedding baseline and degrading gracefully under noise and hijacking.","lead":"This study shows that how Reddit users act — the sequence and timing of their posting and replying decisions — identifies Russian-linked influence accounts about as well as what they write, and keeps working when text is missing or corrupted. The approach models each account's activity as a decision policy and detects targeted accounts from as few as three actions, pointing to content-free detection tools for platforms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sampling mismatch between the 99 transparency-report trolls and the [49] organic pool (subreddit distribution, not just trajectory length) may drive the 94.9 vs 91.2 F1 gap; a subreddit-matched control is needed before concluding behavioral policies detect manipulation.","rationale":"The paper is unusually transparent in many respects: full hyperparameter tables (Tables B2–B7), explicit perturbation and hijacking simulations, and honest reporting of three misclassified accounts all strengthen trust in the execution. However, the central comparison lacks a control for the most obvious confound: the two classes come from different sampling frames and different subreddit ecologies. The MDP encoding intentionally excludes subreddit identity, but behavioral norms are strongly community-dependent; a classifier can learn to separate communities through action distributions (thread creation vs. reply depth) without learning anything about coordinated manipulation. The note that organic trajectories were truncated to match troll lengths addresses one distributional difference but not the community mismatch. The reader's weakest assumption identified exactly this issue, and I agree it is the load-bearing one. The proposed subreddit-matched control is a decisive and feasible test: if the behavioral advantage survives matching, the claim is substantially supported; if not, the paper must be reframed as detecting community-level behavioral distinctiveness rather than manipulation per se. This is an addressable gap, not a fatal flaw, so the conditional verdict should be retained.","tokens_in":27581,"tokens_out":8575,"duration_ms":98878,"concrete_test":"Build a matched negative set from the same subreddits as the trolls: for each of the 99 IRA accounts, sample organic users active in the same subreddits (top 20 in Table A1) during 2015–2018, matched on activity volume (e.g., number of actions) and account age; retrain the empirical and GAIL classifiers under the same CV. If macro-F1 drops toward or below the 91.2% embedding baseline, the reported advantage is largely a subreddit-sampling artifact; if it stays near 94.9%, the concern is refuted. Additionally, add a subreddit-ID feature to the behavioral model and measure the F1 gain—little gain would show the original features already encode subreddit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (Fig. 2a) defines the positive class as 99 IRA accounts with ≥10 interactions (Materials and Methods/Dataset), whose activity concentrates in r/uncen1776, r/Bad_Cop_No_Donut, r/CryptoCurrency, r/copwatch, r/The_Donald (Table A1). The negative class is drawn from [49]'s 15 subreddits (news, politics, human rights, identity), sampled by activity rank per year, then restricted to 2015–2018. The 12-state/6-action encoding contains no subreddit feature; states encode only whether the user is initial/engaged and reply polarity. Therefore any community-specific norm—e.g., link-subreddits favor thread creation, discussion subreddits favor replies—is absorbed into the 'behavioral policy.' If the organic pool is dominated by discussion-oriented subreddits while the troll pool is dominated by thread-creation-oriented subreddits, the classifier may be separating subreddit ecologies, not coordinated manipulation. The paper truncates organic trajectories to match troll lengths (Fig. 2a note) and varies observed-step counts, but it never matches the subreddit distribution or sampling frame of the two classes. The Discussion's limitation paragraph acknowledges label skew but not this sample mismatch. Because the entire claim that 'behavioral dynamics encode stable, discriminative signals of manipulation' depends on the comparison being manipulation-specific, this is the load-bearing weak point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes detecting IRA-linked troll accounts on Reddit from behavioral policies inferred from state-action trajectories, rather than text content. Using 99 labeled troll accounts and 11,965 organic users from the authors' prior dataset, it compares three policy representations (empirical state-action frequencies, GAIL, maximum-entropy deep IRL) against ModernBERT text embeddings, reporting median macro-F1 of 94.9% (GAIL) vs 91.2% (embeddings), with advantages at short traces and under random perturbation. It also clusters troll policies into three subgroups and simulates account hijacking. The manuscript includes detailed appendices on hyperparameter searches and code availability.","tokens_in":27912,"tokens_out":6927,"duration_ms":73241,"significance":"If the comparison is valid, this is a useful contribution: it demonstrates that simple, content-free behavioral features can be competitive with or better than strong text embeddings for a real-world IO campaign, and the robustness/early-detection results are practically relevant. The paper is transparent about data construction, includes extensive hyperparameter tables, and provides code. However, the central interpretation hinges on the troll/organic contrast being manipulation-specific; the unmatched sampling frames and the lack of ablations against simpler behavioral features leave this unresolved. The paper is worth publishing after the confound is addressed and the robustness claims are recalibrated.","major_comments":[{"comment":"The headline comparison is between two differently sampled populations. The positive class is 99 IRA accounts from Reddit's transparency report with ≥10 interactions; Table A1 shows their activity is concentrated in a handful of subreddits (r/uncen1776, r/Bad_Cop_No_Donut, r/CryptoCurrency, r/copwatch, r/The_Donald). The negative class is drawn from 15 news/politics/rights/identity subreddits by activity rank in [49], then restricted to 2015–2018. The state/action encoding contains no subreddit feature—states encode only initial/engaged status and reply polarity—so subreddit-specific conversation norms (thread-creation vs reply-heavy communities) are absorbed into the 'behavioral policy'. Truncating organic trajectories to troll lengths does not address this. Because the central claim is that behavioral dynamics, not content, reveal manipulation, the classifier may instead be separating","section":"Materials and Methods/Dataset; Sec. 2.2/Fig. 2a"},{"comment":"The term 'behavioral policy' overstates what is currently demonstrated. The encoding has 12 states and 6 actions, so the empirical policy in Eq. (1) is a normalized 72-dimensional state-action frequency table. The 3-action result may be largely explainable by the marginal tendency of trolls to start with 'create thread' (the text itself makes this observation). No baseline using the marginal action distribution (6 bins), a first-order action Markov model, or standard temporal features (inter-event times, hour-of-day) is reported. Without such ablations, it is unclear whether the state-conditioned policy adds information beyond simpler behavioral summaries. Please add these baselines to support the 'decision process' interpretation.","section":"Sec. 2.2/Fig. 2b; Materials and Methods/State and action space"},{"comment":"The hyperparameter tables report that configurations were used to 'guide optimal model selection', but the paper does not describe a nested cross-validation or a validation split. If hyperparameters were selected using test-fold performance, the reported median F1s and percentiles are optimistically biased. Please specify the model-selection protocol (e.g., inner CV on training folds only) and report the selected configuration and its held-out performance. This is load-bearing for the exact numerical headline (94.9% vs 91.2%), though not necessarily for the qualitative ranking.","section":"Appendix B.4-B.5; Sec. 2.2"},{"comment":"The hijacking-robustness claim is stronger than the data support. At η=10%, the policy median is 90.1% with 5th-95th percentiles 83.0-93.0, while the content median is 88.1% with percentiles 86.0-89.4; the intervals overlap substantially, so 'performs better' is not supported. At η=50%, the medians reverse (86.1 vs 86.6). Please report paired or bootstrap CIs on the difference, or soften the conclusion to 'comparable' under this synthetic attack model.","section":"Sec. 2.3/Fig. 3c"}],"minor_comments":[{"comment":"Typo: 'Poliy-based' should be 'Policy-based'.","section":"Fig. 3c, Fig. B7"},{"comment":"State explicitly how per-item ModernBERT embeddings were aggregated into a user-level feature for Fig. 2. Only the hijacking scenario defines a mean embedding.","section":"Materials and Methods/Content Embeddings"},{"comment":"The 'matched sample of 99 organic users' in the cluster analysis is not defined; specify matching variables (e.g., total action count, account age).","section":"Sec. 2.3"},{"comment":"The row beginning '10.956±0.0150.939±0.008' appears malformed; likely '0.956'.","section":"Table B7"},{"comment":"The first data row for r/uncen1776 appears to list only a percentage without a count; please align formatting.","section":"Appendix A, Table A1"},{"comment":"Clarify which components of the pipeline are inherited from [49] (organic sampling and the state/action encoding) and which are new to this paper, so the novelty boundary is unambiguous.","section":"Materials and Methods/Dataset"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope. My main concern is the unmatched sampling frame; a subreddit-matched control is essential. I am not asking for a new dataset, only for additional analyses that the current data can support. If the authors provide those, I would be willing to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: this is a well-documented, carefully executed benchmark of behavioral-policy classifiers for IRA troll detection on Reddit. The headline claim — GAIL-based policies reach 94.9% macro-F1 versus 91.2% for ModernBERT text embeddings — is plausible and internally consistent, but its interpretation as evidence that 'behavioral dynamics encode manipulation' is undermined by the fact that the troll and organic pools are drawn from different subreddit populations. That's the load-bearing weak point.\n\nWhat's actually new: policy-based detection of the Reddit IRA cohort with early-detection (n=3), noise-perturbation, and simulated-hijacking experiments, all compared against a modern text-embedding baseline on the same trajectories. The paper is unusually transparent: full hyperparameter tables for GAIL and MaxEnt IRL, perturbation equations, matched-trajectory notes, and honest reporting of the three misclassified accounts. It also makes the fair point that empirical state-action frequencies — a simple histogram — already capture much of the signal, which is an honest result even if it undercuts the 'inference' framing.\n\nThe soft spots, in order:\n\n1. Subreddit mismatch. The 99 trolls come from Reddit's 2017 transparency report with ≥10 interactions, and are heavily concentrated in subreddits like r/uncen1776, r/CryptoCurrency, r/The_Donald (Table A1). The organics come from the authors' prior [49] sampling across 15 news/politics/rights/identity subreddits. The state/action encoding has no subreddit feature. So community-specific norms — thread-creation-heavy vs reply-heavy cultures — get folded into the 'policy.' The paper truncates trajectory lengths but never matches subreddit distribution. That means part of the 3.7-point gap could be population mismatch, not coordinated manipulation. The Discussion acknowledges label skew but not this. This needs a subreddit-matched control or at least stratification by subreddit.\n\n2. 'Platform-agnostic' overstatement. It's one platform, one campaign, one encoding. They soften it in the Discussion, but the Abstract still says 'platform-agnostic framework.' That should be dialed back.\n\n3. Hyperparameter selection. F1-based selection on the same data means the reported confidence intervals are optimistic. Nested CV would fix it. Also, a simple 72-dimensional action-frequency histogram baseline would clarify what the MDP/IRL machinery adds over raw counts. The empirical policy already gets 93.9% full-trajectory, so the gain from GAIL is small.\n\nNone of these are fatal. The core measurements are solid, the reporting is above board, and the early-detection result is robust across methods. I'd send it to a serious referee with the request that they run the matched control and nested CV before the central claim is accepted.\n\nI'd bring it to our reading group, and I'd cite it in a methods section, but only after the sampling concern is resolved.","headline":"Strong empirical benchmark with a clean early-detection result, but the headline F1 gap is confounded by a subreddit-sampling mismatch that the paper doesn't address.","tokens_in":28536,"tokens_out":3365,"would_cite":true,"duration_ms":34262,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A user's pattern of actions—not their words—reliably flags IRA-linked troll accounts on Reddit, beating content-based detection.","keywords":["information operations","inverse reinforcement learning","troll detection","behavioral policy","Markov decision process","Reddit","Internet Research Agency"],"falsifier":"Rerun the troll-versus-organic classification with a negative set matched one-to-one on subreddit, posting volume, and account age; if the policy advantage over text embeddings collapses or the intervals overlap, the signal is population difference, not manipulation.","tokens_in":27339,"feed_emoji":"🕵️","tokens_out":5538,"duration_ms":51960,"temperature":0.7,"pith_summary":"The paper sets out to show that how a person acts on a social platform carries a stable signal of coordinated manipulation, independent of what they post. Using 12,064 Reddit users (99 linked to the Russian Internet Research Agency) and 38 million activity steps, it models each user's behavior as a sequence of states and actions, infers a behavioral policy, and classifies trolls versus ordinary users. The best policy-based classifier reaches a median macro-F1 of 94.9%, compared with 91.2% for a strong text-embedding baseline, with non-overlapping confidence intervals. Policy features also detect trolls from as few as three actions and degrade gracefully under noise or account hijacking. If right, this matters because content and network signals are becoming less reliable as generative text improves and platforms restrict data access.","feed_headline":"Behavior alone outscores content at finding Reddit trolls","feed_subtitle":"It reads only users' action sequences, reaches 94.9% F1, and works from as few as three actions.","key_machinery":"The central object is the behavioral policy: for each user, a probability distribution over six platform actions conditioned on a twelve-way context state, encoding how the user decides at each step (create a thread, post a root comment, reply agreeing/neutral/disagreeing, or wait for a reply). The paper casts each user's activity log as a Markov decision process, infers a policy by three routes—empirical state-action frequencies, Generative Adversarial Imitation Learning (GAIL), and maximum-entropy deep inverse reinforcement learning—and feeds the resulting policy representation into a random-forest or gradient-boosting classifier. The state-action structure is what carries the argument: it","core_discovery":"The central claim is that behavioral policy—the conditional distribution over platform actions given engagement context—is a more stable and discriminative signature of malicious actors than the content they produce. On Reddit's 2015-2018 activity window, the paper converts each user's history into a Markov decision process with 12 states (initial or engaged interactions, receiving replies, with agreement polarity) and 6 actions (create thread, root comment, reply agree/neutral/disagree, wait), then infers per-user policies by empirical frequencies, GAIL, and maximum-entropy deep inverse reinforcement learning. Feeding these policies to a classifier separates IRA-linked trolls from ordinary","pith_inferences":["Editorial inference: if the policy signal reflects operational goals rather than platform quirks, the same state-action encoding could transfer to other pseudonymous platforms such as X or TikTok, but only after redefining actions to match each platform's interface; a direct test is to train on a second platform's transparency-report labels.","Editorial inference: behavioral timing is harder for generative models to fake than text style, so policy features may stay informative even as LLM-generated content saturates platforms; one could simulate this by generating troll-like text with LLMs while keeping observed human posting rhythms and measuring how much detection accuracy remains.","Editorial inference: the three consistently misclassified accounts suggest a ceiling for behavior-only detection—actors who adopt reply-heavy, conversational patterns can hide in plain sight; pairing policy features with metadata such as account age or coordinated timing across accounts would likely recover some of them.","Editorial inference: because the positive and negative user pools differ in subreddit mix and activity volume, part of the reported gap may be population difference rather than manipulation; a matched negative set would clarify the size of the true behavioral effect."],"forward_implications":["Malicious accounts can be flagged from activity logs alone, without reading their posts, on this campaign; behavior outperforms a strong text-embedding baseline and holds up across classifiers.","Detection is possible early: with three state-action pairs the empirical-policy classifier reaches 91.4% macro-F1, so intervention can start before an account builds a long footprint.","Behavioral features degrade smoothly—staying above 80% macro-F1 with up to 50% of actions randomly replaced—so they remain usable on noisy or incomplete platform data.","Trolls are behaviorally heterogeneous: cluster analysis finds thread-creators, root-commenters, and a minority whose policies resemble organics, implying that a single behavioral profile cannot catch every actor.","Combining short behavioral summaries with text and human review would strengthen practical detection, since policy features reduce variance and catch signals text misses."],"fun_headline_variants":["Actions over text: 94.9% F1 catching Reddit trolls","How users act, not what they post, exposes IRA trolls","Behavior beats content: early and robust troll detection","Track behavior, not words: better Reddit troll spotting","Reading actions, not posts: 94.9% F1 for influence ops"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the systematic behavioral differences between the two user pools reflect coordinated manipulation rather than ordinary differences in subreddit habits, activity volume, or account age—and that the 12-state/6-action encoding captures a real policy rather than incidental platform mechanics.","fun_headline_variants_meta":{"raw":{"variants":["Actions over text: 94.9% F1 catching Reddit trolls","How users act, not what they post, exposes IRA trolls","Behavior beats content: early and robust troll detection","Track behavior, not words: better Reddit troll spotting","Reading actions, not posts: 94.9% F1 for influence ops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1133,"prompt_tokens":743,"completion_tokens":390,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":299}},"tokens_in":487,"tokens_out":390,"duration_ms":4915,"temperature":1.0,"reasoning_tokens":299,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:13:59.730793+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the troll-versus-organic classification with a negative set matched one-to-one on subreddit, posting volume, and account age; if the policy advantage over text embeddings collapses or the intervals overlap, the signal is population difference, not manipulation.","supporting_citations":[],"review_version":1}