REVIEW 3 major objections 5 minor 47 references
On the efficacy of old features for the detection of new bots
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Cheap, decade-old Twitter features still distinguish newly evolved bots, including coordinated teams, the paper claims.
desk verdict A useful, plainly written empirical comparison showing cheap timeline/profile features still work on recent coordinated-bot datasets, but the 'novel bot' claim needs a temporal or fully held-out evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central objects are two 'old' feature sets inherited from earlier fake-follower detection: Class A, computed only from account profile fields such as friends and followers counts, account age, and the presence of a name, image, bio, or URL; and Class B, computed from the last 400 tweets, namely the rates of posts containing hashtags, URLs, and mentions, plus the retweet rate. These are contrasted with Botometer's CAP score, a calibrated bot probability derived from more than 1,000 features, and with the percentage of tweets posted from unofficial Twitter clients. The load-bearing mechanism is that timeline content statistics compress enough behavioral signal to separate evolved bots from humans without requiring the friend-and-follower graphs that are expensive to collect.
What would settle it
A strict temporal holdout experiment: train the same five classifiers with Class A and Class B features on accounts labeled in 2018-2019, then evaluate on a dataset of accounts newly labeled after 2020; if ROC-AUC for the timeline features drops to near chance on the new accounts, the paper's central claim fails.
Extended reading notes
Core claim
On its own terms, the paper claims that a set of simple features obtained from the user's timeline are effective in distinguishing novel social bots, whether the bots act individually, like those in the Vendor dataset, or in teams, like the retweeters in Rtbust and the low-value-stock promoters in Stock. Timeline features reach ROC-AUC values above 0.85 on the coordinated-bot datasets, and on the easier self-declared-bot datasets both profile and timeline features exceed 0.95, typically matching or beating Botometer's CAP score when the same classifiers are used. The Twitter-client heuristic, by contrast, works only on self-declared bots and collapses on fake-follower and coordinated-bot datasets. The authors read the results as evidence that detection can keep pace with bot evolution without expensive network or relationship features.
Load-bearing premise
The reasoning assumes that test scores on these particular 2018-2019 datasets tell us how well the features will spot bots that appear later.
Editorial extensions
If this is right
- On these datasets, timeline features rival or beat the Botometer CAP score when both are fed to the same classifiers, suggesting that expensive feature engineering is not always necessary.
- A two-stage detection pipeline becomes plausible: cheap profile and timeline features skim likely bots first, and specialized coordination or network detectors then inspect only the shortlist.
- The Twitter-client heuristic should be treated as a weak signal that is useful only for naive, self-declared bots, not for sophisticated or coordinated accounts.
- Because Class A and Class B features need only profile data and up to 400 recent tweets, they can be gathered at far lower API cost than relationship-based features, making large-scale screening more practical.
Reading between the lines
- Inference: The strongest untested implication is temporal transfer; because the paper uses 10-fold cross-validation within each dataset, the decisive next experiment is a strict time-split evaluation that trains on pre-2020 accounts and tests on accounts labeled later.
- Inference: If timeline statistics are the signal, bot operators could respond by randomizing hashtag, mention, URL, and retweet rates to imitate human timelines, so the durability of Class B features should be monitored on fresh datasets.
- Inference: A practical pipeline suggested by the results is to use timeline features as a low-cost pre-filter and then apply expensive coordination analysis only to flagged accounts, though the paper does not test this end-to-end.
- Inference: Combining timeline features with sequence-based encodings of account behavior could yield both individual bot scores and team similarity measures, potentially turning the first-stage detector into a group detector; the paper gestures in this direction but does not implement it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares four feature sets for Twitter bot detection: the Botometer CAP_UNI* score, profile-based Class A features, timeline-based Class B features, and the proportion of tweets from unofficial clients. Using five classifiers and 10-fold cross-validation on five constructed training sets drawn from six datasets (Celebrity, Verified, Botwiki, Vendor, Stock, Rtbust), it reports that Class A and especially Class B features perform well, including on coordinated-bot datasets, while the Twitter-client feature performs poorly. The authors conclude that cheap, old features can help detect novel bots and propose that such classifiers could serve as a first-stage screening tool before specialized coordinated-behavior analysis.
Significance. If the result held as stated, it would be practically useful: profile and timeline features are inexpensive to collect and could support general-purpose first-stage bot detection. The paper has real strengths: it uses publicly available datasets, standard tooling with Weka default settings, reports several metrics (Balanced Accuracy, precision, recall, MCC, PR-AUC, ROC-AUC), and includes robustness checks over timeline length and a feature-importance analysis. However, the evaluation protocol does not establish the central 'novel bot' claim, and the Botometer arm is affected by label-derived adjustments and optimistic threshold selection. The contribution is therefore best described as an exploratory benchmark with a generalization claim that currently outruns the evidence.
major comments (3)
- [4.1, 6.2, Table 11]
- [4.4, 6.3]
- [6.4, Figure 1]
minor comments (5)
- [Table 11]
- [Section 5.5]
- [Section 6.3]
- [Abstract and Introduction]
- [Section 4.4]
Circularity Check
Botometer baseline is partly circular: CAP* embeds the dataset's label prior and Table 11's threshold is fit on the same accounts; the central Class A/B finding is independent and the 'novel bot' claim is an external-validity gap, not circularity.
-
fitted input called prediction
[Section 4.1, CAP* definition]
"However, if one knows the background level of bots in the sample (and this is our case, since the datasets are tagged), CAP can be adjusted with a corrective factor: CAP∗=CAP P d(Bot)/0.15 where P d(Bot) is the actual probability that in the domain d under investigation there is a specific number of bots. We know in advance the nature of our datasets, so it is possible to compute P d(Bot)."
P d(Bot) is the proportion of bot labels in the dataset, so the CAP* feature is defined in terms of the target labels. In the 10-fold CV experiments, the same full-dataset P d(Bot) is applied to training and test folds, meaning the CAP* value of a test account is computed from a prior that includes that test account's own label. The Botometer-arm scores therefore partly measure the classifier's access to the label distribution through the feature, rather than the standalone discriminating power of Botometer's 1,000 features. The paper presents CAP* as one of the four feature sets for detecting novel bots, so this evaluation is partially self-referential.
-
fitted input called prediction
[Section 6.2, Table 11]
"Once having CAP_UNI* for every account in the training sets, we measured the Botometer's performances by applying a threshold-based rule: if CAP_UNI*≥th, then the account is tagged as a bot. Then, we evaluated the performances of the rule varying the threshold in the interval [0, ...1]. Finally, for each training set, we selected the threshold that gave the best Balanced Accuracy."
The threshold th is chosen to maximize Balanced Accuracy on the very accounts whose scores are then reported in Table 11. The rule-based Botometer results are therefore in-sample fitted values, not held-out predictions; the reported precision, recall, and balanced accuracy are near-optimal by construction on those same accounts. This makes the comparison with the 10-fold CV results of the learning models unequal, although it does not affect the paper's central Class A/B finding.
full rationale
The central claim about Class A and Class B features is not circular: those features are computed from account profiles/timelines without using the bot/human labels, and 10-fold cross-validation is a legitimate separability test within each labeled dataset. The paper's admission that Botwiki bots are 'very easy to recognise' weakens the 'novel bots' inference but is an honest dataset characterization, not a circular derivation. The acknowledged lack of testing on Sayyadiharikandeh et al.'s datasets is an external-validity gap, not a circularity. No load-bearing self-citation or imported uniqueness theorem is present. The partial circularity is confined to the Botometer arm: CAP* is rescaled by the actual label prevalence in the sample, and the threshold used for the rule-based Botometer comparison is fitted on the full data. Both steps inflate the Botometer baseline's reported performance but do not drive the article's principal conclusion that cheap timeline/profile features separate the tested bot and human accounts.
Assumptions & free parameters
free parameters (3)
- CAP_UNI* prevalence correction P_d(Bot) =
Dataset-specific: 0.128 (Celebrity-Botwiki), 0.251 (Verified-Botwiki), 0.102 (Verified-Vendor), 0.591 (Stock), 0.496…
- Botometer decision threshold =
0.015, 0.025, 0.025, 0.4, 0.02 per training set
- Minimum tweets per account =
400
assumptions (4)
- domain assumption The a priori labels in the six datasets are correct and the datasets represent 'novel' or 'evolved' bots.
- domain assumption The Botometer CAP formulation (prior P(Bot)=0.15 and Bayes correction) is a valid score for bot probability.
- domain assumption Weka default hyperparameters are adequate for fair comparison across feature sets.
- domain assumption The proportion of tweets from unofficial clients is a meaningful automation signal, per Bovet and Makse.
Cite this review
Pith. "Pith review of On the efficacy of old features for the detection of new bots." pith.science (2026). https://pith.science/paper/3Y324AOR
@misc{pith2026250619635,
author = {Pith},
title = {Pith review of: On the efficacy of old features for the detection of new bots},
year = {2026},
howpublished = {\url{https://pith.science/paper/3Y324AOR}},
note = {Machine review of arXiv:2506.19635}
}
read the original abstract
For more than a decade now, academicians and online platform administrators have been studying solutions to the problem of bot detection. Bots are computer algorithms whose use is far from being benign: malicious bots are purposely created to distribute spam, sponsor public characters and, ultimately, induce a bias within the public opinion. To fight the bot invasion on our online ecosystem, several approaches have been implemented, mostly based on (supervised and unsupervised) classifiers, which adopt the most varied account features, from the simplest to the most expensive ones to be extracted from the raw data obtainable through the Twitter public APIs. In this exploratory study, using Twitter as a benchmark, we compare the performances of four state-of-art feature sets in detecting novel bots: one of the output scores of the popular bot detector Botometer, which considers more than 1,000 features of an account to take a decision; two feature sets based on the account profile and timeline; and the information about the Twitter client from which the user tweets. The results of our analysis, conducted on six recently released datasets of Twitter accounts, hint at the possible use of general-purpose classifiers and cheap-to-compute account features for the detection of evolved bots.
Figures
Reference graph
Works this paper leans on
-
[1]
E. Mustafaraj, P. T. Metaxas, From obscurity to prominence in minutes: Political speech and real-time search, in: Web Science: Extending the Frontiers of Society On-Line, 2010
work page 2010
- [2]
-
[3]
G. Caldarelli, R. Nicola, F. D. Vigna, M. Petrocchi, F. Saracco, The role of bot squads in the political propaganda on Twitter, Communications Physics 3 (2019) 1–15
work page 2019
-
[4]
Flow of online misinformation during the peak of the COVID-19 pandemic in Italy
G. Caldarelli, R. D. Nicola, M. Petrocchi, M. Pratelli, F. Saracco, Analysis of online misinformation during the peak of the COVID-19 pandemics in italy, CoRR abs/2010.01913 (2020).arXiv:2010. 01913. URLhttps://arxiv.org/abs/2010.01913
work page Pith review arXiv 2020
-
[5]
E. Ferrara, O. Varol, C. Davis, F. Menczer, A. Flammini, The rise of social bots, Commun. ACM 59 (7) (2016) 96–104.doi:10.1145/2818717. URLhttp://doi.acm.org/10.1145/2818717
doi:10.1145/2818717 2016
- [6]
-
[7]
Sonic Research Group at Nortwestern Univ., Episode 5: The bits and bots of the web with fil menczer, online: https://tinyurl.com/y49gmchk (2020)
work page 2020
- [8]
Show all 47 references
-
[9]
K. Yang, O. Varol, P. Hui, F. Menczer, Scalable and generalizable social bot detection through data selection, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 20...
2020
-
[10]
Varol, E
O. Varol, E. Ferrara, C. A. Davis, F. Menczer, A. Flammini, Online human-bot interactions: Detection, estimation, and characterization, in: Proceedings of the Eleventh International Conference on Web and Social Media, ICWSM 2017, Montréal, Québec, Canada, May 15-18, 2017., 201...
2017
-
[11]
K. Yang, O. Varol, C. Davis, E. Ferrara, A. Flammini, F. Menczer, Arming the public with artificial intelligence to counter social bots, Human Behavior and Emerging Technologies 48–61
-
[12]
Cresci, R
S. Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, M. Tesconi, Fame for sale: efficient detection of fake twitter followers, Decision Support Systems 80 (2015) 56–71
2015
-
[13]
Vico, La Scienza Nuova, 1744
G. Vico, La Scienza Nuova, 1744
-
[14]
Yardi, D
S. Yardi, D. M. Romero, G. Schoenebeck, danah boyd, Detecting spam in a twitter network, First Monday 15 (1) (2010)
2010
-
[15]
Cresci, A decade of social bot detection, Commun
S. Cresci, A decade of social bot detection, Commun. ACM 63 (10) (2020) 72–83. 22
2020
-
[16]
Viswanath, M
B. Viswanath, M. A. Bashir, M. B. Zafar, S. Bouget, S. Guha, K. P. Gummadi, A. Kate, A. Mislove, Strength in numbers: Robust tamper detection in crowd computations, in: Proceedings of the 2015 ACM on Conference on Online Social Networks, ACM, 2015, pp. 113–124
2015
-
[17]
Cresci, R
S. Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, M. Tesconi, DNA-inspired online behavioral modeling and its application to spambot detection, IEEE Intelligent Systems 31 (5) (2016) 58–64
2016
-
[18]
R. Yu, X. He, Y. Liu, GLAD: Group anomaly detection in social media analysis, ACM Transactions on Knowledge Discovery from Data (TKDD) 10 (2) (2015) 1–22
2015
-
[19]
Cresci, R
S. Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, M. Tesconi, Social fingerprinting: detection of spambot groups through dna-inspired behavioral modeling, IEEE Transactions on Dependable and Secure Computing 15 (4) (2018) 561–576
2018
-
[20]
Sayyadiharikandeh, O
M. Sayyadiharikandeh, O. Varol, K. Yang, A. Flammini, F. Menczer, Detection of novel social bots by ensembles of specialized classifiers, in: M. d’Aquin, S. Dietze, C. Hauff, E. Curry, P. Cudré-Mauroux (Eds.), CIKM ’20: The 29th ACM International Conference on Information and ...
2020
-
[21]
R. J. Schuchard, A. T. Crooks, Insights into elections: An ensemble bot detection coverage framework applied to the 2018 U.S. midterm elections, PLOS ONE 16 (1) (2021) 1–19.doi:10.1371/journal. pone.0244309. URLhttps://doi.org/10.1371/journal.pone.0244309
2021 doi
-
[22]
Chavoshi, H
N. Chavoshi, H. Hamooni, A. Mueen, Debot: Twitter bot detection via warped correlation, in: 2016 IEEE 16th International Conference on Data Mining (ICDM), 2016, pp. 817–822.doi:10.1109/ICDM. 2016.0096
2016
-
[23]
Beskow, K
D. Beskow, K. Carley, H. Bisgin, A. Hyder, C. Dancy, R. Thomson, Introducing Bothunter: A tiered approach to detection and characterizing automated activity on Twitter, in: International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior R...
2018
-
[24]
Ferrara, Disinformation and social bot operations in the run up to the 2017 french presidential election, First Monday 22 (8) (2017)
E. Ferrara, Disinformation and social bot operations in the run up to the 2017 french presidential election, First Monday 22 (8) (2017). URLhttps://firstmonday.org/ojs/index.php/fm/article/view/8005
2017
-
[25]
Pedregosa, et al., Scikit-learn: Machine learning in python, Journal of Machine Learning Research 12 (85) (2011) 2825–2830
F. Pedregosa, et al., Scikit-learn: Machine learning in python, Journal of Machine Learning Research 12 (85) (2011) 2825–2830. URLhttp://jmlr.org/papers/v12/pedregosa11a.html
2011
-
[26]
El-Mawass, P
N. El-Mawass, P. Honeine, L. Vercouter, Similcatch: Enhanced social spammers detection on twitter using markov random fields, Information Processing & Management 57 (6) (2020) 102317.doi:https: //doi.org/10.1016/j.ipm.2020.102317. URLhttp://www.sciencedirect.com/science/articl...
2020
-
[27]
P. Hui, K. Yang, C. Torres-Lugo, F. Menczer, Botslayer: DIY real-time influence campaign detection, in: M. D. Choudhury, R. Chunara, A. Culotta, B. F. Welles (Eds.), Proceedings of the Fourteenth International AAAI Conference on Web and Social Media, ICWSM 2020, Held Virtually...
2020
-
[28]
Giatsoglou, D
M. Giatsoglou, D. Chatzakou, N. Shah, A. Beutel, C. Faloutsos, A. Vakali, ND-Sync: Detecting syn- chronized fraud activities, in: PAKDD, Springer, 2015
2015
-
[29]
Jiang, P
M. Jiang, P. Cui, C. Faloutsos, Suspicious behavior detection: Current trends and future directions, IEEE Intelligent Systems 31 (1) (2016) 31–39
2016
-
[30]
Q. Cao, X. Yang, J. Yu, C. Palow, Uncovering large groups of active malicious accounts in online social networks, in: ACM SIGSAC Conference on Computer and Communications Security, ACM, 2014, pp. 477–488
2014
-
[31]
K. Yang, O. Varol, C. A. Davis, E. Ferrara, A. Flammini, F. Menczer, Arming the public with AI to counter social bots, CoRR abs/1901.00912 (2019). URLhttp://arxiv.org/abs/1901.00912 23
2019 arXiv
-
[32]
Mazza, S
M. Mazza, S. Cresci, M. Avvenuti, W. Quattrociocchi, M. Tesconi, RTbust: Exploiting temporal patterns for botnet detection on Twitter, in: 11th ACM Conference on Web Science, 2019, pp. 183– 192.doi:10.1145/3292522.3326015
2019
-
[33]
Cresci, F
S. Cresci, F. Lillo, D. Regoli, S. Tardelli, M. Tesconi, Cashtag piggybacking: Uncovering spam and bot activity in stock microblogs on Twitter, ACM Trans. Web 13 (2) (Apr. 2019).doi:10.1145/3313184. URLhttps://doi.org/10.1145/3313184
2019 doi
-
[34]
Breiman, Random forests, Machine Learning 45 (1) (2001) 5–32.doi:10.1023/A:1010933404324
L. Breiman, Random forests, Machine Learning 45 (1) (2001) 5–32.doi:10.1023/A:1010933404324
2001 doi
-
[35]
Bovet, F
A. Bovet, F. Morone, H. A. Makse, Validation of twitter opinion trends with national polling aggregates: Hillary clinton vs donald trump, Scientific Reports 8 (1) (2018) 8673
2018
-
[36]
CE, Basic principles of ROC analysis, Seminars in Nuclear Medicine 8 (4) (1978) 283–298
M. CE, Basic principles of ROC analysis, Seminars in Nuclear Medicine 8 (4) (1978) 283–298
1978
-
[37]
G. M. Weiss, F. Provost, Learning when training data are costly: The effect of class distribution on tree induction, Journal of artificial intelligence research 19 (2003) 315–354
2003
-
[38]
Saito, M
T. Saito, M. Rehmsmeier, The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, PLOS ONE 10 (3) (2015) 1–21.doi:10.1371/ journal.pone.0118432. URLhttps://doi.org/10.1371/journal.pone.0118432
2015 doi
-
[39]
S. K. Pal, S. Mitra, Multilayer perceptron, fuzzy sets, and classification, IEEE Transactions on Neural Networks 3 (5) (1992) 683–697.doi:10.1109/72.159058
1992 doi
-
[40]
W. W. Cohen, Fast effective rule induction, in: A. Prieditis, S. Russell (Eds.), Machine Learning Proceedings 1995, Morgan Kaufmann, San Francisco (CA), 1995, pp. 115 – 123.doi:https://doi. org/10.1016/B978-1-55860-377-6.50023-2. URLhttp://www.sciencedirect.com/science/article...
1995 doi
-
[41]
G. H. John, P. Langley, Estimating continuous distributions in bayesian classifiers, in: Proceedings of theEleventhConferenceonUncertaintyinArtificialIntelligence, UAI’95, MorganKaufmannPublishers Inc., San Francisco, CA, USA, 1995, p. 338–345
1995
-
[42]
Breiman, Random forests, Machine Learning 45 (2001) 5–32
L. Breiman, Random forests, Machine Learning 45 (2001) 5–32
2001
-
[43]
I. H. Witten, E. Frank, M. A. Hall, Data mining: practical machine learning tools and techniques, 3rd Edition, Morgan Kaufmann, Elsevier, 2011
2011
-
[44]
D. Aha, D. Kibler, M. Albert, Instance-based learning algorithms, Machine Learning 6 (1991) 37–66
1991
-
[45]
Ozenne, F
B. Ozenne, F. Subtil, D. Maucort-Boulch, The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases, Journal of Clinical Epidemiology 68 (8) (2015) 855–859.doi:https://doi.org/10.1016/j.jclinepi.2015.02.010
2015 doi
-
[46]
J. T. Kent, Information gain and a general measure of correlation, Biometrika 70 (1) (1983) 163–173
1983
-
[47]
T. M. Mitchell, et al., Machine learning (1997). 24
1997
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.