{"id":"e4bd7190-266e-4c49-83aa-e1c096d4a2eb","arxiv_id":"2501.07368","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new classifier suite detects and levels expressions of collective action participation in Reddit comments, reaching weighted F1=0.71 for binary detection.","lead":"Researchers built text classifiers that identify when social media users express participation in collective action, such as calling others to act or reporting their own activism. The tool is meant to give social scientists a large-scale way to measure how online communities move from awareness to action across many topics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported F1 is measured only on dictionary-filtered comments from activist-named subreddits; no labeled evaluation on unfiltered or out-of-domain text supports the topic-agnostic deployment claim.","rationale":"The reader's weakest assumption already flags the dictionary-filtered, activist-subreddit sample as the key threat to the topic-agnostic claim. My concern sharpens one component: the dictionary filter applies even inside the selected subreddits, so the evaluation never measures accuracy on text without two or more dictionary matches, which is exactly the regime targeted when the model is applied to unfiltered Reddit or other platforms. The paper is transparent about data construction, releases code and data, and provides several qualitative external checks, which is real supporting evidence. However, those checks are not labeled accuracy measurements, so they cannot establish that the reported F1 transfers. The appropriate outcome remains conditional acceptance with an explicit requirement for out-of-domain, unfiltered labeled evaluation, which is the reader's verdict; no adjustment is needed.","tokens_in":25296,"tokens_out":6973,"duration_ms":72566,"concrete_test":"Re-annotate a held-out sample of roughly 800 comments drawn uniformly from random Reddit threads with no dictionary filtering and from subreddits outside the activist-seed list, using the same codebook, then run the released best BERT binary model (CS + Syn A) and compute macro and weighted F1 with bootstrap confidence intervals. If macro F1 falls below about 0.5 or drops by more than 0.10 from the reported 0.65, the topic-agnostic claim is not supported. Additionally report recall separately on the subset of comments with zero dictionary hits; near-chance recall there would show the classifier inherits the dictionary's blind spots.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation protocol in Data Curation retains only comments with at least two matches from the 47-term collective action dictionary, then feeds annotators and models the sentence with the highest term frequency plus one preceding and one following sentence. The same filter is applied to the 809-comment test set. Therefore every precision/recall number in Tables 2 and 3 is conditional on text that already contains substantial dictionary vocabulary. The abstract's binary weighted F1=0.71 and the multi-class comparison (BERT macro F1=0.44 vs. SFT 0.52) do not tell us how the classifier behaves on the unfiltered comments it actually targets in the Climate Action and sociodemographic applications, where participation can be expressed with zero or one dictionary term. The external checks do not close this gap: the stance-detection and UK parliamentary analyses report qualitative distributions rather than gold-labeled F1, and the 'Impact of Collective Action Terms' experiment removes dictionary words from an already dictionary-filtered test set, so it cannot measure recall on comments that never contained those terms. The r/intj and r/wallstreetbets validation threads are also unlabeled. Thus the topic-agnostic generalization claim rests on unmeasured transfer from a filtered, activist-skewed input distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a text-classification framework for detecting expressions of participation in collective action in social media comments, grounded in social-movement mobilization theory and operationalized with a four-level taxonomy (Problem-solution, Call-to-action, Intention, Execution). The authors construct a crowdsourced Reddit training set filtered by a 47-term collective-action dictionary, augment it with synthetic and Reddit-derived samples, and evaluate RoBERTa, zero-shot Llama3, fine-tuned Llama3 (SFT and DPO), plus baselines on a held-out test set. They report that a small RoBERTa model reaches weighted F1 0.71 for binary detection and macro F1 0.44 for four-level classification, close to the best fine-tuned LLM's 0.52, and they illustrate the method's usefulness through topic-modeling, stance-detection, parliamentary-debate, climate-action, and sociodemographic analyses.","tokens_in":25566,"tokens_out":5292,"duration_ms":52525,"significance":"If the core claims hold, the released models, annotation codebook, and training data would be a valuable resource for Computational Social Science, enabling large-scale and granular measurement of participation in collective action from text. The paper's strengths include making code, models, and datasets publicly available; grounding the annotation scheme in established mobilization theory; and systematically comparing multiple modeling strategies and augmentation techniques. However, the main evaluation is restricted to a dictionary-filtered sample from activist-named subreddits, which makes the topic-agnostic deployment claim substantially stronger than the evidence directly supports. The external validations are largely qualitative and do not provide gold-labeled F1 scores on unfiltered or out-of-domain data, leaving a measurable gap between the reported results and the stated applicability.","major_comments":[{"comment":"The evaluation reported in Tables 2 and 3 is conducted on a test set that required at least two matches from the 47-term collective-action dictionary and was sampled from subreddits whose names or descriptions contain 'activism,' 'activist,' or 'rights.' Because the same filter is used to construct the training set, the reported F1 scores are conditional on text that already contains substantial dictionary vocabulary and come from an activist-skewed community distribution. This does not support the abstract's topic-agnostic claim, which is used to justify applying the classifier to unfiltered comments in the Climate Action and sociodemographic analyses. The paper should either add a labeled evaluation on unfiltered comments from non-activist subreddits or substantially soften the generalization claim.","section":"Data Curation / Test Set"},{"comment":"The multi-class comparison rests on extremely small class sizes: the test set contains only 11 Intention and 13 Execution labels. With so few instances, the F1 differences between methods for these classes (e.g., BERT CS+SynA Intention F1=0.17 vs. SFT CS+SynA Intention F1=0.29) are within the range of chance variation, and no confidence intervals are reported. Consequently, the claim that 'the SFT and DPO models consistently outperformed others, particularly in classifying Intention and Execution' is not statistically supported; the macro-F1 differences between the best BERT (0.44) and best LLM (0.52) may also be driven largely by these unstable minority classes.","section":"Results, Multi-class Task / Tables 3 and B3"},{"comment":"The external validation experiments do not provide quantitative gold-standard evaluation on out-of-domain data. The topic-modeling comparison uses unlabeled threads from r/intj and r/wallstreetbets and reports qualitative distribution comparisons; the stance-detection experiment reports participation distributions across stance classes without gold participation labels; and the parliamentary-debate analysis provides only high/low probability examples rather than labeled F1 scores. These analyses can illustrate potential utility, but they cannot establish the classifier's transferability to unfiltered text or non-Reddit domains, which is required for the claimed broader applicability.","section":"Validation"},{"comment":"The adversarial test that removes or replaces dictionary words operates on the already dictionary-filtered test set. Since every test comment originally contained at least two collective-action terms, this experiment cannot measure recall on comments that express participation without ever using such vocabulary. The risk that the classifier's positive predictions are dependent on dictionary words therefore remains unquantified for the unfiltered target distribution, and the robustness conclusion drawn from Figure 1(f) does not close that gap.","section":"Impact of Collective Action Terms"}],"minor_comments":[{"comment":"The abstract's 'weighted F1=0.71' is not tied to a specific model configuration; Table 2 shows that zero-shot Llama3 achieves weighted F1=0.76–0.77, so the headline figure should explicitly identify the BERT model trained on CS+SynA.","section":"Abstract / Table 2"},{"comment":"The x-axis in Figure 2 is labeled 'normalized,' but neither the caption nor the surrounding text defines the normalization procedure used for the percentage values.","section":"Figure 2 / Climate Action"},{"comment":"The Paper Checklist states that an 'anonymized shared repository' is linked, but the Introduction provides a non-anonymized GitHub URL; this inconsistency should be resolved if the manuscript is intended for anonymous review.","section":"Paper Checklist / Introduction"}],"recommendation":"major_revision","confidential_remarks":"The central scientific gap is the mismatch between the filtered evaluation and the unfiltered deployment scenarios; adding a small, gold-labeled sample of unfiltered comments from non-activist subreddits would substantially strengthen the paper. The reported multi-class differences also need uncertainty quantification or larger test sets. The manuscript is otherwise transparent about its code and data, and the theoretical grounding is a positive contribution. These issues are fixable within the current scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a solid methods contribution with a real gap between what it claims (topic-agnostic) and what it measures (dictionary-filtered activist subreddits). The taxonomy is new and the model comparison is honest, but the evidence doesn't yet support the deployment claim.\n\nWhat's actually new: a four-level participation schema grounded in consensus and action mobilization theory, a small annotated dataset of Reddit comments, and a systematic comparison of dictionary, centroid, RoBERTa, and Llama3 variants. That's a useful resource for CSS. The transparency is a plus: they report many configurations, precision/recall, and discuss limitations.\n\nThe soft spot is the evaluation. All training and test data go through the same filter: subreddits with 'activism/activist/rights' in the name, and comments with at least two hits from a 47-term dictionary. Every F1 is conditioned on that distribution. The topic-agnostic claim—that the classifier works on unfiltered text across topics—has no labeled test. The external validation on the stance dataset and parliamentary debates only shows qualitative distributions, not gold-labeled accuracy. So the abstract's 'topic-agnostic' is more of an aspiration than a measured property.\n\nAlso worth noting: the test set has 11 Intention and 13 Execution labels, so the multi-class F1s for those cells are unstable. No confidence intervals anywhere. And they picked the best model from 61 combinations without a separated model-selection set, so some reported numbers might flatter the approach.\n\nNone of this is fatal for a methods paper. The core idea is sound, the theory is grounded, and the release of code, data, and models is genuinely useful. But the claims need to be matched to the evidence. A serious referee should ask for an out-of-distribution evaluation—unfiltered comments from non-activist subreddits, or a held-out set that isn't dictionary-filtered—or, failing that, a revision that drops the topic-agnostic phrasing.\n\nI'd bring it to a reading group and I'd accept it for review. The authors can likely fix the gap with additional labeling.","headline":"New participation-level taxonomy and classifier suite, but the topic-agnostic claim outruns the evaluation.","tokens_in":26096,"tokens_out":2289,"would_cite":true,"duration_ms":22592,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that participation in collective action can be detected and graded into four levels from social media text alone, using a small open-source classifier that nearly matches large language models in accuracy at a fraction…","keywords":["collective action","participation detection","text classification","Reddit","BERT","large language models","social movement mobilization","data augmentation"],"falsifier":"Take a stratified sample of comments from subreddits that do not mention activism or rights in their names—for example general news, sports, and hobby communities—annotate them with expert labelers using the paper's codebook, run the released binary classifier on the unfiltered text, and compare its macro F1 to the reported 0.65; a substantial drop would show the topic-agnostic claim depends on the dictionary filter and activist-skewed training distribution.","tokens_in":25093,"feed_emoji":"📢","tokens_out":6755,"duration_ms":58006,"temperature":0.7,"pith_summary":"This paper claims that expressions of participation in collective action can be reliably detected in social media text by a comparatively small, open-source language model, without being tied to any single topic or cause. The authors define four levels of participation—recognizing a problem and proposing solutions, issuing a call to action, stating an intention to act, and reporting actual involvement—and build a layered classifier that first decides whether a comment expresses participation and then assigns its level. Trained on crowdsourced Reddit comments augmented with synthetic and structurally similar examples, the best BERT-based model reaches a macro F1 of 0.65 for the binary detection task (weighted F1 0.71) and 0.44 for four-level classification, close to the best fine-tuned Llama3 model at 0.52 while being far cheaper to run. If the claim holds, researchers can generate large-scale, granular ground truth on who participates in collective action from text alone, at a fraction of the computational cost of LLM-based approaches.","feed_headline":"Small language models rival LLMs at detecting collective action","feed_subtitle":"A BERT classifier quantifies four participation levels in Reddit text, with F1 scores close to far larger models.","key_machinery":"The load-bearing object is the four-level participation taxonomy derived from consensus and action mobilization theory: Problem-solution, Call-to-action, Intention, and Execution, with a fifth None class for non-participation. This taxonomy turns a continuous psychological pathway into discrete labels that a classifier can learn, and it is what makes the method topic-agnostic, since the levels describe the speaker's engagement rather than the cause. The argument is carried by a layered pipeline—a binary participation detector followed by a four-way level classifier—trained on crowdsourced Reddit comments that were filtered through a collective-action word list and then augmented with synthetic minority-class examples and with labels propagated from similar comments in the same threads. The layer that does the heavy lifting is the BERT/RoBERTa classifier trained on the synthetically augmented set, which the paper selects for the full pipeline because it nearly matches fine-tuned LLMs in F1 while running two orders of magnitude faster at inference.","core_discovery":"On the paper's own terms, the central discovery is that participation in collective action is a learnable textual property that can be quantified along a theoretically grounded four-level scale, and that a fine-tuned small transformer captures most of the signal that large models do. The paper operationalizes the mobilization pathway from social movement theory as four mutually ordered classes, builds a human-annotated Reddit dataset around them, and shows that data augmentation—synthetic Llama3-generated examples for rare classes plus propagation of labels to similar in-thread comments—makes training feasible with only 369 original crowd labels. The resulting pipeline separates participation detection from participation-level classification, and its outputs are shown to be orthogonal to topic membership, stance, and keyword presence: participation is found across all stance classes and in subreddits that keyword-based proxies would miss. The authors conclude that small models can rival LLMs on this task, and that the tool can serve as a new source of granular ground truth for studying collective action dynamics online.","pith_inferences":["An implication the paper leaves implicit is that the same four-level scale could be applied to other platforms, such as X, Facebook, or TikTok transcripts, with an annotation pass to verify that platform-specific phrasing does not degrade performance.","A testable extension would be to use the classifier's per-comment timeline to trace individuals moving from Problem-solution to Intention to Execution, turning the paper's cross-sectional measurement into evidence about mobilization pathways.","One caution suggested by the paper's own design: because training comments were pre-filtered by a 47-term collective-action dictionary, the classifier may under-detect participation expressed through novel, indirect, or highly contextual language, a gap that could be probed with unfiltered annotation samples.","The demographic correlations the paper reports are aggregate community-level patterns; an editorial caveat is that individual-level inference from subreddit embeddings would need additional validation before being used to make claims about who participates."],"forward_implications":["Researchers can use the released classifiers to measure participation levels in any large text corpus without hand-labeling a new dataset for each topic.","Communities that look inactive by keyword or subreddit-membership proxies may still contain substantial collective-action participation, so prior studies based on those proxies may have underestimated activism in non-activist spaces.","The four-level output makes it possible to study mobilization as a gradual process—from recognizing an issue to reporting involvement—rather than as a binary activist/non-activist status.","Because the best small model nearly matches the best LLM at a fraction of the compute, large-scale longitudinal studies of collective action become computationally affordable.","The positive validation on UK parliamentary debates suggests the classifier transfers beyond Reddit to institutional political discourse, though the paper treats that test as preliminary."],"supporting_citations":[{"why":"Supplies the 47-term collective-action dictionary used to filter candidate comments and as the keyword baseline.","marker":"Smith, McGarty, and Thomas (2018)"},{"why":"Provides the four-step mobilization pathway from which the paper's participation-level taxonomy is derived.","marker":"Klandermans and Oegema (1987)"},{"why":"Supplies the working definition of collective action that the paper adapts to online discourse.","marker":"Wright (2009)"},{"why":"Llama3 8B Instruct is the base for zero-shot, SFT, DPO, and synthetic data generation.","marker":"Dubey et al. (2024)"},{"why":"Underpins the LLM-augmented data approach used to expand the small crowdsourced training set.","marker":"Møller et al. (2024)"},{"why":"RoBERTa is the small transformer that becomes the paper's selected classifier.","marker":"Liu (2019)"},{"why":"Sentence-BERT provides the embeddings for the centroid baseline and for propagating labels to similar in-thread comments.","marker":"Reimers (2019)"},{"why":"The climate-stance dataset is used to show participation classification is distinct from stance detection.","marker":"Luo, Card, and Jurafsky (2020)"},{"why":"Community embeddings are used to characterize the sociodemographic position of subreddits.","marker":"Waller and Anderson (2019)"},{"why":"Provides the social dimensions along which subreddit participation correlates with collective-action expressions.","marker":"Waller and Anderson (2021)"}],"fun_headline_variants":["Small AI rivals giants in spotting collective action on Reddit","Four-level participation detection from Reddit text, BERT beats LLMs","Topic-agnostic classifier reads collective action levels in posts","BERT model decodes four stages of collective action participation","Tiny transformers match LLMs on collective action detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire training and test signal comes from Reddit comments that were pre-selected through activist subreddit names and a 47-word collective-action dictionary, so the claim that the classifier is topic-agnostic rests on the assumption that this filtered sample represents how participation is expressed across topics, platforms, and ideologies.","fun_headline_variants_meta":{"raw":{"variants":["Small AI rivals giants in spotting collective action on Reddit","Four-level participation detection from Reddit text, BERT beats LLMs","Topic-agnostic classifier reads collective action levels in posts","BERT model decodes four stages of collective action participation","Tiny transformers match LLMs on collective action detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2800,"prompt_tokens":979,"completion_tokens":1821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1739}},"tokens_in":595,"tokens_out":1821,"duration_ms":12373,"temperature":1.0,"reasoning_tokens":1739,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:43:51.873640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified sample of comments from subreddits that do not mention activism or rights in their names—for example general news, sports, and hobby communities—annotate them with expert labelers using the paper's codebook, run the released binary classifier on the unfiltered text, and compare its macro F1 to the reported 0.65; a substantial drop would show the topic-agnostic claim depends on the dictionary filter and activist-skewed training distribution.","supporting_citations":[{"cited_title":"G.; McGarty, C.; and Thomas, E","cited_arxiv_id":null,"evidence_quote":"Supplies the 47-term collective-action dictionary used to filter candidate comments and as the keyword baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the four-step mobilization pathway from which the paper's participation-level taxonomy is derived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the working definition of collective action that the paper adapts to online discourse."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Community embeddings are used to characterize the sociodemographic position of subreddits."}],"review_version":1}