{"id":"8e0f416c-89f7-4cf4-9c1b-13c8e0e2a4aa","arxiv_id":"2607.18248","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Twitter's removal, reduction, and informing interventions were followed by lasting declines in bot counts and activities, but not in human sharing of cancer misinformation, over 2011–2021.","lead":"This study tracked cancer misinformation on Twitter from 2011 to 2021 and found that platform interventions were followed by sustained drops in bot activity, but no significant change in human sharing. The result matters because platform policies are often evaluated in aggregate, which can hide that humans are largely unaffected.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bot deterrence effect may be an artifact of excluding unscorable accounts: 33.1% missing Botometer scores, correlated with suspension and time.","rationale":"The reader's weakest_assumption precisely identifies the most load-bearing problem: the bot outcomes—the only significant effects in the paper—are measured on a subset of accounts that survived to data collection. Because the removal intervention suspends accounts, and Botometer cannot score suspended/non-existent/protected accounts, the post-intervention bot decline is at least partly a mechanical consequence of sample construction. This is not a minor technicality; it directly threatens the paper's central differential claim. The paper itself transparently reports the 33.1% missing rate, which is good, but it never addresses the bias. Other issues, such as unreported ARIMA orders, overlapping intervention steps, and absent code/data, are secondary; even if all were fixed, the missing-data problem would remain. The proposed sensitivity analysis—bounding missingness as all-bot/all-human, imputing from tweet features, and checking time trends in unscorable rates—would settle whether the bot coefficients are real deterrence effects or attrition artifacts. Given the current evidence, the reader's REJECT is justified; a conditional acceptance would require the sensitivity analysis to show robustness.","tokens_in":31649,"tokens_out":3752,"duration_ms":47660,"concrete_test":"Run a missing-data sensitivity analysis on the bot outcomes: (a) re-estimate the Tables 4–5 seasonal ARIMAs twice—treating all 33.1% unscorable accounts as bots, then all as humans—and compare coefficients; (b) use multiple imputation of bot scores from tweet-level features (e.g., volume, timing, content) for accounts with no Botometer score; (c) plot the monthly proportion of unscorable accounts to see if it jumps at the intervention dates (Feb 2017, Jun 2018, Sep 2020). If the removal/reduction coefficients become non-significant or shrink by >80% under any plausible missing-data assumption, the bot deterrence claim is an artifact of differential attrition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—durable bot deterrence without human deterrence—rests on bot counts/activities computed only from accounts that remained accessible to Botometer at collection time. Section 4.4 reports 33.1% of the 96,836 accounts received no bot score: 14.2% suspended, 13.3% non-existent, 4.5% protected, 1.1% tweets removed. Botometer requires current account access, so any account suspended by the removal intervention (Feb 2017 onward) is mechanically excluded from the post-intervention bot series. If bots are disproportionately suspended or deleted, the declines in Tables 4–5 (e.g., -0.804 removal, -1.762 bot activities) reflect sample attrition, not deterrence. The paper never reports how missingness varies by month or user type, nor whether bot prevalence among missing accounts differs from scored ones. Since the headline differential claim is bot decline vs. human stability, this selection bias could fully explain the pattern: bots disappear from the measurement frame, while humans remain observable. The human null result is less vulnerable, but the 'bots-only' conclusion is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper uses Twitter data from 2011–2021 on cancer misinformation, applies ML classification to identify supportive tweets, scores accounts with Botometer, and estimates seasonal ARIMA interrupted time-series models to test whether removal (Feb 2017), reduction (Jun 2018), and informing (Sep 2020) interventions have sustained deterrent effects on the number of bots/humans and their misinformation activities. The results show significant post-intervention declines for bots (all three interventions for bot count; removal and reduction for bot activity), whereas no significant effects are found for humans. The paper concludes that platform interventions durably suppress automated misinformation spread but leave human sharing unchanged.","tokens_in":1258,"tokens_out":1602,"duration_ms":93826,"significance":"If the bot-related findings were valid, the paper would make a meaningful contribution: it is one of the few longitudinal evaluations that disaggregate intervention effects by user type, and it uses a decade of real-world data. The human-null result is interesting and is less vulnerable to the missing-data problem. However, the headline bot claim is compromised by a load-bearing selection issue: 33.1% of accounts received no Botometer score, and the removal intervention operates by suspending accounts, so the post-intervention bot decline may be partly mechanical. The paper's strengths include its large dataset, the explicit post hoc aggregated comparison, and the longitudinal design, but the central bot conclusion is not established as written.","major_comments":[{"comment":"The bot series are computed only for accounts with Botometer scores; 33.1% of the 96,836 supportive accounts are unscorable, including 14.2% suspended and 13.3% non-existent. Because the removal intervention operates by suspending accounts, the post-Feb-2017 decline in bot counts/activities may reflect mechanical attrition from the measurement frame rather than deterrence. The paper never reports missingness by month or user type, nor does it compare the scored bot series with the total supportive-account series. Please add missingness diagnostics, attempt bounding/robustness checks, and reframe the bot conclusions accordingly. As written, H1a–H1c and H2a–H2b are not established.","section":"§4.4, Tables 4–5"},{"comment":"The ARIMA specifications are not reported: the orders ((p,d,q)×(P,D,Q)), the exact intervention coding (separate vs. joint models), and the estimation details are missing. Because the three interventions overlap in time (removal Feb 2017, reduction Jun 2018, informing Sep 2020), separate step models confound later interventions with earlier accumulated effects. Report full model formulas, estimation output, and sensitivity to alternative intervention start dates.","section":"§4.5, Appendix A, Tables 4–7"},{"comment":"The estimated effects are extremely large: removal implies an 84% decrease in bot count and a 98% decrease in bot activities. These magnitudes are hard to reconcile with the reported 14.2% suspended-account share even if all suspended accounts were bots. The gap suggests either that the measured decline is dominated by attrition of newly created bots or that the model is misspecified; the paper should reconcile these numbers or provide a mechanism.","section":"Table 4, Table 5"}],"minor_comments":[{"comment":"The text says 'supporting H2a and H2b, but not H3c'; H3c is a human hypothesis. This should be H2c.","section":"§5, Table 5 paragraph"},{"comment":"The total number of tweets is inconsistent: the text reports 268,280 collected, 2,000 used for training, and 266,280 unlabeled, but Table 3 lists 264,280 unlabeled and a total of 266,280. Verify and correct the arithmetic.","section":"§4.3, Table 3"},{"comment":"Table B.2 appears to be a copy of Table 7 (same estimates and model assessment), yet the text describes it as modeling the number of all users. Clarify the intended analysis and correct the table.","section":"Appendix B, Table B.2"},{"comment":"The limitations section does not mention the 33.1% of accounts without Botometer scores or the direct relationship between the removal intervention and the measurement of bot presence. This omission should be addressed explicitly.","section":"§6.3 Limitations"},{"comment":"The caption refers to 'the introduction of Twitter’s intervention measures' but the figure appears to mark only one date; specify which intervention the vertical line denotes.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The reader's core concern about missing Botometer scores is valid and should be prominently communicated. I recommend major revision rather than outright rejection because the authors could add missing-data diagnostics, re-run the analysis on the full account set, and re-scope the claims; if such analyses are not possible, the paper's central bot conclusions cannot stand."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim—durable bot deterrence, no human deterrence—is only half-supported. The human side is credible; the bot side is not established because 33.1% of accounts never got a Botometer score, and the missing accounts (suspended, deleted, protected) are exactly the ones the interventions would remove. The paper never checks whether missingness varies by month or user type, so the post-2017 collapse in bot counts and activity is partly mechanical attrition, not deterrence.\n\nWhat the paper does well: the longitudinal setup is genuinely different from the short-term studies it cites; separating bots from humans is overdue. The post hoc aggregate analysis is a good idea and shows the masking effect. The ML classification pipeline is described in enough detail to be plausible; the classifier quality is modest but acceptable. The literature review is broad and mostly current. The human null result—no sustained effect for any of the three interventions—is the most sound piece of the analysis. It doesn't depend on the bot-vs-human boundary in the same way, and it aligns with other work on the difficulty of changing human sharing behavior.\n\nSoft spots, in order: (1) the missing-account problem is the big one. Section 4.4 gives the numbers, but the analysis never addresses them. The removal intervention suspended accounts; Botometer needs accessible accounts; so the outcome is defined on a surviving sample. The 84–87% \"decrease\" in bot presence in Table 4 is too large to trust without a missingness model or a bounds check. The reduction and informing results are somewhat less suspect because those interventions don't directly suspend, but the same measurement gap applies to all series. (2) The ARIMA orders are not reported, there is no code or data, and intervention months are chosen from blog posts without any sensitivity to alternative dates. These are fixable, but as-is the results are not reproducible. (3) Minor: H2c is mislabeled as H3c in one place, and the \"deterrence\" framing for bots stretches the theory—better to talk about elimination or adaptation than rational deterrence for automated accounts.\n\nOverall: worth refereeing because the question matters and the human result is a useful corrective to aggregate studies. But the bot claim should not survive in its current form. A revision that reframes the bot result as conditional on observable accounts, adds a missing-data analysis or bounds, and reports full model specs could be publishable and genuinely useful. I'd send it to review, but I'd expect referee reports to ask for substantial work, not minor edits.","headline":"The bot deterrence finding is likely an artifact of sample attrition; the human null is the real contribution, but the paper overstates the bot side.","tokens_in":32354,"tokens_out":3134,"would_cite":false,"duration_ms":37972,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Platform interventions durably reduced bot accounts and bot activity in cancer misinformation over a decade, while human sharers showed no significant sustained change.","keywords":["health misinformation","social media interventions","bots","deterrence theory","interrupted time series","ARIMA","cancer misinformation","Twitter"],"falsifier":"Recompute bot counts and activities while imputing bot scores for the 33.1% of unscored accounts (suspended, non-existent, protected, tweet-removed), or track account status over time; if the post-intervention bot decline disappears or shrinks to non-significance once those accounts are accounted for, the sustained deterrence claim for bots is an artifact of missing-data mechanics.","tokens_in":31535,"feed_emoji":"🤖","tokens_out":3112,"duration_ms":33469,"temperature":0.7,"pith_summary":"This paper tries to establish that Twitter's three anti-misinformation interventions—removal, reduction, and informing—produced lasting deterrence for automated bot accounts but not for human accounts. Analyzing tweets endorsing unproven cancer claims from 2011 to 2021, the authors find sustained declines in bot counts after all three interventions and in bot activity after removal and reduction, with estimated drops of 72–98%. For human users, no intervention produced a statistically significant sustained change in either the number of sharers or their activity. The point of the distinction is practical: aggregate studies can make interventions look successful when the effect is concentrated in bots, masking unchanged human behavior that may need different strategies.","feed_headline":"Twitter crackdowns curbed bots for years, not humans","feed_subtitle":"A decade of cancer-misinformation tweets: enforcement shrank bot activity, while human sharing held steady.","key_machinery":"An interrupted time-series design using seasonal ARIMA models, in which a step-change indicator switches from 0 to 1 at the month each intervention was announced (February 2017 for removal, June 2018 for reduction, September 2020 for informing). The coefficient on that indicator is the estimated sustained level shift in logged monthly bot/human counts and activity. Around this, the analysis relies on identifying supportive misinformation tweets with a machine-learning classifier and separating accounts into bots and humans with a bot-scoring tool; the estimated effect percentages are derived by back-transforming the intervention coefficients.","core_discovery":"The central claim is that social media interventions act as deterrent mechanisms with different long-term trajectories for bots and humans. Using monthly counts of accounts and posts supporting cancer misinformation, interrupted time-series models with a step-change indicator for each intervention show that bot user numbers fell significantly after removal, reduction, and informing interventions, and bot activity fell after removal and reduction; the informing intervention did not significantly reduce bot activity. The same models found no significant sustained effect on the number of human sharers or the volume of human misinformation activity for any intervention. A post hoc aggregate anal","pith_inferences":["Editorial inference: A large share of accounts (about 33%) had no bot score because they were suspended, deleted, protected, or scrubbed; if those missing accounts skew toward bots, the post-removal decline in measured bot counts is partly mechanical — the intervention removes accounts from observation — rather than evidence that remaining or future bot operators were deterred.","Editorial inference: The paper's logic for bots implies a testable prediction: bot creators' behavior should shift toward evasion, so one would expect detection-resistant bot designs or migration to other platforms to appear after each intervention; the paper does not test this.","Editorial inference: The same step-change design could be applied to other persistent health misinformation topics or to newer LLM-based bots, which the authors flag as future work; re-running the analysis on post-2022 data would tell whether the bot-deterrence pattern survives the arrival of generative AI.","Editorial inference: The informing intervention's significant effect on bot counts but not bot activity hints that labels mainly suppress account creation or existence, while active bots that remain keep posting at similar rates; this asymmetry could be checked by stratifying bot activity by account age."],"forward_implications":["If the paper is right, platform enforcement can durably shrink the automated side of health misinformation, so bot-oriented harms like scale amplification and distorted popularity can be reduced for years.","The null results for humans imply that removal, reduction, and informing as deployed on Twitter do not, by themselves, durably change human sharing of health misinformation.","Evaluations that mix bots and humans can misattribute bot reductions to overall success; disaggregated measurement is needed before concluding an intervention worked.","Since informing alone did not reduce bot activity even though it reduced bot counts, persuasive or soft interventions may be weaker than coercive or situational ones for curbing bot output.","The findings suggest platform resources should be split: automated-account enforcement for bots, and different, perhaps cognitive or prebunking, strategies for humans."],"fun_headline_variants":["Bots stop sharing health lies for years; humans keep at it","Crackdowns curb bot misinformation long-term, not human activity","Health misinformation: penalties work on bots, not humans","Social media bans deter bot spreaders, humans remain undeterred","Study: bot misinformation drops for years, humans ignore penalties"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The analysis assumes that bot and human counts are measured consistently before and after each intervention, even though a third of accounts could not be scored because they were suspended, deleted, protected, or had their tweets removed — and the removal intervention itself suspends accounts.","fun_headline_variants_meta":{"raw":{"variants":["Bots stop sharing health lies for years; humans keep at it","Crackdowns curb bot misinformation long-term, not human activity","Health misinformation: penalties work on bots, not humans","Social media bans deter bot spreaders, humans remain undeterred","Study: bot misinformation drops for years, humans ignore penalties"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1597,"prompt_tokens":651,"completion_tokens":946,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":874}},"tokens_in":395,"tokens_out":946,"duration_ms":9212,"temperature":1.0,"reasoning_tokens":874,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:33:23.588751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute bot counts and activities while imputing bot scores for the 33.1% of unscored accounts (suspended, non-existent, protected, tweet-removed), or track account status over time; if the post-intervention bot decline disappears or shrinks to non-significance once those accounts are accounted for, the sustained deterrence claim for bots is an artifact of missing-data mechanics.","supporting_citations":[],"review_version":1}