REVIEW 2 major objections 4 minor 3 cited by
Labeled Datasets for Research on Information Operations
T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper releases labeled datasets that pair platform-verified information-operation posts with over 13 million control posts from 303,000 accounts discussing the same topics at the same times, for 26 campaigns across 16 state actors.
desk verdict A genuinely useful multi-campaign IO dataset with an honest limitations section, but the unquantified timeline asymmetry between IO and control accounts is a real confound for detection benchmarking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The method that carries the paper is a hashtag-anchored control-selection pipeline. For each campaign, the authors extract all hashtags used by the IO accounts, query the platform API for accounts that used the same hashtags on the same dates, and then pull those control accounts' timelines for those dates, truncated at 100 posts per day. The working assumption is that hashtag co-use on the same day selects organic accounts engaging with the same topics at the same time, making 'is control' a usable negative label for supervised detection and a baseline for descriptive comparison. The pipeline also merges the IO and control records into a single schema with aligned field names and an explicit is_control column.
What would settle it
A concrete check would be to compute, per campaign, the fraction of IO posts that contain no hashtag and the fraction of IO hashtags that never appear in control posts; the paper's own numbers give a median hashtag coverage of only 31%, so campaigns whose distinctive hashtags are largely absent from the control data would yield biased benchmarks. One could also manually audit a random sample of control accounts for spam-like or coordinated patterns; any substantial share of non-organic control accounts would directly invalidate the negative labels.
Extended reading notes
Core claim
The central delivery is a set of 26 labeled datasets, one per campaign, each with a binary flag indicating whether a post comes from a platform-verified IO account (false) or from a matched control account (true). The IO side contains the full timelines of the accounts the platform attributed to each campaign, not just the posts judged inauthentic; the control side contains daily timelines (up to 100 posts per day) of accounts that used the IO campaigns' hashtags on the same dates. The authors report 100% coverage of control data, unlike earlier control sets drawn from small samples, and they anonymize all identifiers with a consistent one-way hash so that mentions and reposts remain linkable across the IO/control boundary. Their claim is that this combination—verified malicious accounts, topic- and time-matched organic accounts, full timeline context, and cross-campaign breadth—supplies what detection research lacked: a reusable, re-hydration-free benchmark for distinguishing coordinated from organic behavior.
Load-bearing premise
The control labels are only as good as the assumption that the hashtags used by IO accounts capture the content of each campaign, and that accounts sharing those hashtags on the same dates are genuinely organic—an assumption the paper itself notes fails when IO tactics avoid hashtags or when generic hashtags pull in spammers and other coordinated actors.
Editorial extensions
If this is right
- Researchers can train supervised detectors to distinguish IO accounts from organic accounts across dozens of campaigns without needing to collect or re-hydrate posts through now-inaccessible platform APIs.
- The multi-country, multi-actor structure allows cross-campaign generalization tests, such as training on some operations and evaluating on held-out campaigns from different states or languages.
- Including full daily timelines for control accounts, rather than only topic-matched posts, enables behavior-sequence and account-history methods that earlier campaign-specific control sets did not support.
- The consistent anonymization hash lets investigators reconstruct mention, repost, and reply networks across the IO/control boundary, supporting network-based coordination detection and descriptive studies of engagement tactics.
- The datasets establish a shared benchmark where future detection methods can be compared against the same verified labels and control samples across 26 campaigns.
Reading between the lines
- An implication the authors leave implicit is that the dataset's value is uneven across campaigns: because control selection depends on hashtags, campaigns whose IO accounts rarely hashtag (the paper reports hashtag coverage varying from 3% to 73%, with a median of 31%) will produce weaker controls, so benchmarks should report per-campaign results rather than pooled accuracy.
- Since the anonymization scheme preserves account linkage across IO and control posts, a natural extension not pursued in the paper would be to release derived temporal interaction graphs, enabling direct evaluation of coordination-detection algorithms that operate on network structure.
- Because the control labels are based on absence from a platform-verified IO list rather than on any independent audit, evaluation protocols that treat some control accounts as potentially noisy negatives—for example, by manually reviewing a sample or using robust ranking metrics—are more trustworthy than protocols that assume all control labels are clean.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This dataset paper introduces and documents a new collection of labeled social-media datasets covering 26 state-sponsored information operations (IOs) attributed to 16 state actors. For each campaign, the release contains platform-verified IO posts (full account timelines) and a control set of posts by accounts that used the same hashtags as the IO accounts on the same dates. The control data reportedly comprise over 13 million posts from 303k accounts. The data are anonymized, merged into a common schema with an `is_control` flag, and released on Zenodo with a DOI. The paper describes the collection pipeline, provides descriptive statistics in Table 1, reports coverage statistics in Fig. 2, and discusses limitations including temporal misalignment and hashtag-proxy quality. The stated contribution is that these datasets enable researchers to characterize IO tactics and benchmark IO detection algorithms without needing to re-hydrate posts through now-inaccessible APIs.
Significance. If the control data are valid, the release is a useful community resource: it spans more campaigns and countries than most previous control sets, avoids the need for API access, and includes anonymized post content rather than only IDs. Strengths include the breadth of the release, the explicit coverage statistics, the careful acknowledgment of several limitations, and the inclusion of a data DOI and ethical documentation. The paper is also internally consistent in its reported aggregate totals: the sums in Table 1 approximate the abstract's 13M+ control posts. However, the central benchmarking claim depends on the control accounts being a usable negative class. The structural asymmetry between full IO timelines and cropped control timelines, plus the dependence on hashtag quality, means the datasets as released can support descriptive comparative studies but need additional validation or qualification before they can support reliable detection benchmarks.
major comments (2)
- [Discussion (limitations)] The main load-bearing issue for the benchmarking claim is the structural asymmetry between IO and control timelines. In 'Control Data Collection', IO accounts contribute their full platform-verified timelines, whereas control accounts contribute only up to 100 messages per day on dates when they used an IO hashtag, and the Discussion states that 'control account timelines are cropped at 100 posts and exclude posts that occurred after the date on which they met the inclusion criteria.' This makes per-account features such as total post volume, active-day count, and observed timeline span systematically different between the classes by construction. The Related Work claim that the datasets include, for control accounts, 'other posts from their timelines' is inconsistent with this procedure. No quantification of the resulting distributional difference is provided, and Fig. 2 reports only aggregate collective coverage. Because a detection algorithm could separate the classes on this artifact alone, the paper should either release matched temporal subsets and per-class feature distributions, or explicitly restrict the benchmarking claim to controls for which the confounding is addressed.
- [Discussion (limitations)] The choice to define control accounts by the hashtags used by IO accounts makes the validity of the negative class depend on the quality of the hashtag proxy, as the paper itself acknowledges: 'the quality of the control sample depends on IO hashtags being a quality proxy for IO content, which is not always the case.' This is a genuine threat to the central contribution because campaigns with little or no hashtag use may lack valid control counterparts, and generic hashtags can attract non-IO coordinated or inauthentic activity into the control set. The paper gives an illustrative #crypto example but does not quantify the problem: per campaign, what fraction of IO posts contain no hashtagged content, and how specific are the hashtags used? Without such diagnostics, downstream users cannot tell which of the 26 campaigns support a reliable detection benchmark. Please add per-campaign hashtag-coverage statistics and, if possible, a control subset that excludes very generic hashtags.
minor comments (4)
- [Related Work] The phrase 'developed ntify messages' is truncated and should read 'developed to identify messages'.
- [Related Work] The sentence 'does not need require re-hydration' should be corrected to 'does not require re-hydration'.
- [Discussion] The sentence beginning 'Research who wish to mitigate' should read 'Researchers who wish to mitigate'.
- [Related Work] The claim 'Our datasets have 100% coverage of control data' should be qualified, since the collection procedure caps control timelines at 100 posts per day and excludes posts after the first matched date; the statement is stronger than what the data collection section describes.
Circularity Check
No circularity: the paper's contribution is a released dataset, and its control-selection choices are acknowledged validity limitations rather than circular derivation.
full rationale
This is a dataset-description paper with no fitted model, no predicted quantity, and no uniqueness theorem; the central claim is the release of new labeled IO and control datasets at a DOI. The control accounts are selected by querying hashtags used by IO accounts on the same dates, which is an intentional sampling design for topic-matched negative examples, not a definitional equivalence between the input and the claimed output. The paper explicitly acknowledges the resulting limitations: 'the quality of the control sample depends on IO hashtags being a quality proxy for IO content, which is not always the case,' and 'control account timelines are cropped at 100 posts and exclude posts that occurred after the date on which they met the inclusion criteria,' noting a 'significant misalignment in the active days for individual control accounts compared to individual IO accounts.' These are empirical validity and confound concerns for downstream benchmarks, not circular reasoning: the labels are not defined in terms of any detection outcome, no parameter is fit to a subset and then reported as a prediction, and the self-citations in the related work are background context rather than load-bearing support for the dataset's construction. The derivation chain is therefore self-contained with respect to circularity, and any criticism of the control-group comparability belongs to correctness risk rather than circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Platform-provided IO labels are accurate ground truth
- domain assumption IO hashtags are a good proxy for campaign content
- domain assumption Accounts sharing hashtags on the same dates are a valid negative class
Cite this review
Pith. "Pith review of Labeled Datasets for Research on Information Operations." pith.science (2026). https://pith.science/paper/X4ETBDSZ
@misc{pith2026241110609,
author = {Pith},
title = {Pith review of: Labeled Datasets for Research on Information Operations},
year = {2026},
howpublished = {\url{https://pith.science/paper/X4ETBDSZ}},
note = {Machine review of arXiv:2411.10609}
}
read the original abstract
Social media platforms have become a hub for political activities and discussions, democratizing participation in these endeavors. However, they have also become an incubator for manipulation campaigns, like information operations (IOs). Some social media platforms have released datasets related to such IOs originating from different countries. However, we lack comprehensive control data that can enable the development of IO detection methods. To bridge this gap, we present new labeled datasets about 26 campaigns, which contain both IO posts verified by a social media platform and over 13M posts by 303k accounts that discussed similar topics in the same time frames (control data). The datasets will facilitate the study of narratives, network interactions, and engagement strategies employed by coordinated accounts across various campaigns and countries. By comparing these coordinated accounts against organic ones, researchers can develop and benchmark IO detection algorithms.
Figures
Forward citations
Cited by 3 Pith papers
-
Bridging the Narrative Divide: Cross-Platform Discourse Networks in Fragmented Ecosystems
A new platform-agnostic network method reveals that 0.33% of users introduce nearly 70% of narratives that migrate between Truth Social and X during the 2024 U.S. election.
-
IOHunter: Graph Foundation Model to Uncover Online Information Operations
IOHunter detects information-operation drivers by fusing frozen language-model text embeddings with a GNN over a fused user-similarity network, reporting SOTA Macro-F1 on six Twitter IO datasets and cross-country tran...
-
Social Media Information Operations
A tutorial that frames social media information operations as an optimization problem and surveys the analytics, threat models, and countermeasures that support it.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Addawood, A.; Badawy, A.; Lerman, K.; and Ferrara, E. 2019. Linguistic Cues to Deception: Identifying Political Trolls on Social Media . In Proc. Intl. AAAI Conf. on Web and Social Media, volume 13, 15--25
work page 2019
-
[4]
N.; Buntain, C.; and Tucker, J
Alizadeh, M.; Shapiro, J. N.; Buntain, C.; and Tucker, J. A. 2020. Content-based Features Predict Social Media Influence Operations . Science Advances, 6(30)
work page 2020
-
[5]
Badawy, A.; Addawood, A.; Lerman, K.; and Ferrara, E. 2019. Characterizing the 2016 Russian IRA Influence Campaign . Social Network Analysis and Mining, 9: 1--11
work page 2019
-
[6]
Bradshaw, S.; and Howard, P. 2017. Troops, Trolls and Troublemakers: A Global Inventory of Organized Social Media Manipulation . Technical report, University of Oxford
work page 2017
-
[7]
Cima, L.; Mannocci, L.; Avvenuti, M.; Tesconi, M.; and Cresci, S. 2024. Coordinated behavior in information operations on Twitter . IEEE Access
work page 2024
-
[8]
Elmas, T. 2023. Analyzing Activity and Suspension Patterns of Twitter Bots Attacking Turkish Twitter Trends by a Longitudinal Dataset . In Companion Proc. ACM Web Conf., 1404--1412
work page 2023
Show all 46 references
-
[9]
Ezzeddine, F.; Ayoub, O.; Giordano, S.; Nogara, G.; Sbeity, I.; Ferrara, E.; and Luceri, L. 2023. Exposing Influence Campaigns in the Age of LLMs: A Behavioral-based AI Approach to Detecting State-Sponsored Trolls . EPJ Data Science, 12(1): 46
2023
-
[10]
Facebook. 2021. Threat Report: The State of Influence Operations 2017-2020 . Technical report, Facebook
2021
-
[11]
Fecher, L.; Reich, T.; Taylor, J.; and Warren, P. 2022. Oh, The Places You'll Guo! The Tactics and Impact of a Chinese Multilingual Narrative Flooding Campaign through Political Cartoons . https://open.clemson.edu/cgi/viewcontent.cgi?article=1000&context=mfh_ci_reports
2022
-
[12]
Guo, X.; and Vosoughi, S. 2022. A large-scale longitudinal multimodal dataset of state-backed information operations on Twitter . In Proc. of the Intl. AAAI Conf. on Web and Social Media, volume 16, 1245--1250
2022
-
[13]
Im, J.; Chandrasekharan, E.; Sargent, J.; Lighthammer, P.; Denby, T.; Bhargava, A.; Hemphill, L.; Jurgens, D.; and Gilbert, E. 2020. Still Out There: Modeling and Identifying Russian Troll Accounts on Twitter . In 12th ACM Conf. on Web Science
2020
-
[14]
B.; Schoch, D.; Stier, S.; and Yang, J
Keller, F. B.; Schoch, D.; Stier, S.; and Yang, J. 2020. Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign . Political communication
2020
-
[15]
Kong, Q.; Calderon, P.; Ram, R.; Boichak, O.; and Rizoiu, M.-A. 2023. Interval-censored Transformer Hawkes: Detecting Information Operations using the Reaction of Social Systems . In Proc. ACM Web Conf., 1813–--1821
2023
-
[16]
Lazer, D.; Baum, M.; Benkler, Y.; Berinsky, A.; Greenhill, K.; Menczer, F.; Metzger, M.; Nyhan, B.; Pennycook, G.; Rothschild, D.; Schudson, M.; Sloman, S.; Sunstein, C.; Thorson, E.; Watts, D.; and Zittrain, J. 2018. The science of fake news. Science, 359(6380): 1094--1096
2018
-
[17]
L.; and Warren, P
Linvill, D. L.; and Warren, P. L. 2020. Engaging with Others: How the IRA Coordinated Information Operation Made Friends . Misinformation Review
2020
-
[18]
Luceri, L.; Boniardi, E.; and Ferrara, E. 2024. Leveraging Large Language Models to Detect Influence Campaigns in Social Media. In Companion Proc. of the 2024 ACM Web Conf
2024
-
[19]
Luceri, L.; Giordano, S.; and Ferrara, E. 2020. Detecting Troll Behavior via Inverse Reinforcement Learning: A Case Study of Russian Trolls in the 2016 US Election . In Intl. AAAI Conf. on Web and Social Media, volume 14, 417--427
2020
-
[20]
Luceri, L.; Pantè, V.; Burghardt, K.; and Ferrara, E. 2024. Unmasking the web of deceit: Uncovering coordinated activity to expose information operations on Twitter. In Proc. of the 2024 ACM Web Conf
2024
-
[21]
Merhi, M.; Rajtmajer, S.; and Lee, D. 2023. Information Operations in Turkey: Manufacturing Resilience with Free Twitter Accounts . In Proc. Intl. AAAI Conf. on Web and Social Media
2023
-
[22]
Murtfeldt, R.; Alterman, N.; Kahveci, I.; and West, J. D. 2024. RIP Twitter API: A eulogy to its vast research contributions. arXiv preprint arXiv:2404.07340
2024
-
[23]
Ng, L. H. X.; Moffitt, J.; and Carley, K. M. 2022. Coordinated through a Web of Images: Analysis of Image-based Influence Operations from China, Iran, Russia, and Venezuela . Preprint arXiv:2206.03576
2022 arXiv
-
[24]
C.; Flammini, A.; and Menczer, F
Nwala, A. C.; Flammini, A.; and Menczer, F. 2023. A Language Framework for Modeling Social Media Account Behavior . EPJ Data Science, 12(1): 33
2023
-
[25]
C.; and Caba \ n es, J
Ong, J. C.; and Caba \ n es, J. V. A. 2018. Architects of Networked Disinformation: Behind the Scenes of Troll Accounts and Fake News Production in the Philippines . Tech. rep., UMass Amherst. https://doi.org/10.7275/2cq4-5396
2018 doi
-
[26]
T.; Flammini, A.; and Menczer, F
Pacheco, D.; Hui, P.-M.; Torres-Lugo, C.; Truong, B. T.; Flammini, A.; and Menczer, F. 2021. Uncovering Coordinated Networks on Social Media: Methods and Case Studies. In Proc. Intl. AAAI Conf. on Web and Social Media, volume 21, 455--466
2021
-
[27]
Rowett, G. 2018. The Strategic Need to Understand Online Memes and Modern Information Warfare Theory . In IEEE Big Data
2018
-
[28]
H.; Ali, S.; Paudel, P.; Blackburn, J.; and Stringhini, G
Saeed, M. H.; Ali, S.; Paudel, P.; Blackburn, J.; and Stringhini, G. 2024. Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twitter . arXiv preprint arXiv:2407.18098
2024 arXiv
-
[29]
Senate Select Committee on Intelligence . 2019. Report of the Select Committee on Intelligence United States Senate on Russian Active Measures Campaigns and Interference in the 2016 U.S. Election Volume 2: Russia's Use of Social Media with Additional Views . https://www.intell...
2019
-
[30]
L.; Varol, O.; Yang, K.; Flammini, A.; and Menczer, F
Shao, C.; Ciampaglia, G. L.; Varol, O.; Yang, K.; Flammini, A.; and Menczer, F. 2018. The Spread of Low-credibility Content by Social bots . Nature Communications, 9: 4787
2018
-
[31]
Sharma, K.; Zhang, Y.; Ferrara, E.; and Liu, Y. 2021. Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours . In 27th ACM SIGKDD Conf. on Knowledge Discovery & Data Mining, 1441--1451
2021
-
[32]
H.; Ehrett, C.; and Warren, P
Smith, D. H.; Ehrett, C.; and Warren, P. L. 2024. Unsupervised Detection of Coordinated Information Operations in the Wild . arXiv preprint arXiv:2401.06205
2024 arXiv
-
[33]
Stanford Internet Observatory . 2021. Published reports of the Stanford Internet Observatory. https://github.com/stanfordio/publications
2021
-
[34]
Starbird, K. 2019. Disinformation's spread: bots, trolls and all of us. Nature, 571(7766): 449--450
2019
-
[35]
G.; Arif, A.; and Starbird, K
Stewart, L. G.; Arif, A.; and Starbird, K. 2018. Examining Trolls and Polarization with a Retweet Network . In Proc. ACM WSDM Workshop on Misinformation and Misbehavior Mining on the Web (MIS2). https://api.semanticscholar.org/CorpusID:44033303
2018
-
[36]
Torres-Lugo, C.; Pote, M.; Nwala, A.; and Menczer, F. 2022. Manipulating Twitter through Deletions . In Proc. Intl. AAAI Conf. on Web and Social Media, 1029--1039
2022
-
[37]
J.; and Carley, K
Uyheng, J.; Cruickshank, I. J.; and Carley, K. M. 2022. Mapping State-sponsored Information Operations with Multi-view Modularity Clustering . EPJ Data Science, 11(1): 25
2022
-
[38]
Vargas, L.; Emami, P.; and Traynor, P. 2020. On the Detection of Disinformation Campaign Activity with Network Analysis . In Proc. of the 2020 ACM SIGSAC Conf. on Cloud Computing Security Workshop, 133--146
2020
-
[39]
Vosoughi, S.; Roy, D.; and Aral, S. 2018. The Spread of True and False News Online . Science
2018
-
[40]
Wilson, T.; and Starbird, K. 2021. Cross-platform Information Operations: Mobilizing Narratives & Building Resilience through both'Big'&'Alt'Tech . Proc. of the ACM on Human-Computer Interaction
2021
-
[41]
C.; and Howard, P
Woolley, S. C.; and Howard, P. N. 2018 a . Computational Propaganda: Political Parties, Politicians, and Political Manipulation on Social Media . Oxford University Press
2018
-
[42]
C.; and Howard, P
Woolley, S. C.; and Howard, P. N. 2018 b . Conclusion: Political Parties, Politicians, and Computational Propaganda . In Computational propaganda: Political parties, politicians, and political manipulation on social media, 241--248. Oxford University Press
2018
-
[43]
Yang, K.-C.; and Menczer, F. 2024. Anatomy of an AI-powered Malicious Social Botnet . Journal of Quantitative Description: Digital Media, 4
2024
-
[44]
Zannettou, S.; Caulfield, T.; Bradlyn, B.; De Cristofaro, E.; Stringhini, G.; and Blackburn, J. 2020. Characterizing the Use of Images in State-Sponsored Information Warfare Operations by Russian Trolls on Twitter. Proc. of the Intl. AAAI Conf. on Web and Social Media
2020
-
[45]
Zannettou, S.; Caulfield, T.; De Cristofaro, E.; Sirivianos, M.; Stringhini, G.; and Blackburn, J. 2019 a . Disinformation Warfare: Understanding State-sponsored Trolls on Twitter and their Influence on the Web . In Companion Proc. WWW Conf., 218--226
2019
-
[46]
Zannettou, S.; Caulfield, T.; Setzer, W.; Sirivianos, M.; Stringhini, G.; and Blackburn, J. 2019 b . Who Let the Trolls Out? Towards Understanding State-sponsored Trolls . In Proc. 10th ACM Conf. on Web Science, 353--362
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.