Pith. sign in

REVIEW 2 major objections 4 minor 52 references

Towards Ethical Content-Based Detection of Online Influence Campaigns

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A content-based influence-campaign detector misclassifies English sentences by native Russian speakers at nearly double the rate of native English speakers, and named entity masking only partially fixes the disparity.

desk verdict A useful bias measurement with a real confound: the L1 Russian vs. L1 English comparison is not topic-matched, so the language-bias claim is conditional, not settled. read the letter →

arxiv 1908.11030 v1 pith:D2KMV4MO submitted 2019-08-29 cs.CY cs.CL

Evan Crothers , Nathalie Japkowicz , Herna Viktor This is my paper · ORCID
classification cs.CYcs.CL
keywords influencecampaigndetectionalgorithmicbiasfalsepositiveratesnativelanguageidentificationnamedentitymaskingBERTsentenceembeddingsRedditcommentscontent-basedclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a text-only classifier trained to spot online influence accounts can be deployed without unfairly discriminating against particular language communities. It trains a BERT-based sentence classifier on comments from accounts linked to a Russian influence operation, then measures false positives on English written by native Russian speakers versus native English speakers. The central finding is that the classifier flags Russian-native English at a far higher rate (about 64% vs 39%), that named entity masking lowers both rates but leaves a persistent gap, and that the gap grows when the text mentions frequent named entities such as US politics or cryptocurrency. The authors conclude that content-based influence detection without safeguards risks automated suppression of speech by a language group, and that evaluation sets of L2 English writers should be part of model development.

What carries the argument

The machinery is a sentence-level classifier built from BERT embeddings: each sentence is transformed into a fixed vector from the model's [CLS] token and passed through a single-layer classifier trained to separate suspect influence-account sentences from randomly sampled Reddit comments. The intervention is named entity masking (NEM), which replaces recognized named entities with their part-of-speech tags before embedding, a technique borrowed from native-language identification research to reduce topic bias caused by entities over-represented in influence-campaign text. NEM is the object that carries the fairness argument: it demonstrably lowers false positives when frequent named entities are present, but the residual language gap shows that entity content is not the only source of the disparity.

What would settle it

Build a topic-matched version of the evaluation set by pairing Russian-native and English-native sentences on topic categories (US politics, cryptocurrency, general discussion) and recompute false-positive rates with the same classifiers. If the gap falls to near zero within each topic category, the claimed language-community discrimination is not supported; if the gap persists after matching, the language-bias claim survives this check.

Watch

Extended reading notes

Core claim

The paper's central claim is that content-based detection models trained on past influence campaigns inherit a language bias: they misclassify English sentences written by native Russian speakers at a markedly higher false-positive rate than English sentences by native English speakers. On a held-out evaluation set, the unmasked BERT classifier produced false positives for 63.82% of Russian-native sentences versus 38.82% for English-native sentences; when sentences were filtered to contain frequent named entities (US, Trump, Bitcoin, Russia, and similar), the rates rose to 70.09% and 54.46%. Replacing named entities with their part-of-speech tags reduced all rates (43.72% vs 36.10% overall, 56.55% vs 51.97% on frequent-entity sentences) while keeping classification accuracy nearly unchanged (0.7266 versus 0.7409 AUC). The authors interpret this as evidence that masking helps with topic-driven false positives but does not remove the elevated risk for Russian-native writers, particularly when they discuss US politics or cryptocurrency.

Load-bearing premise

The comparison assumes that the Russian-native and English-native evaluation sentences are alike except for the writer's first language; because the two sets were not topic-matched, the higher false-positive rate could be partly a topic effect (more politics or cryptocurrency talk among Russian-native users) rather than purely a language effect.

Editorial extensions

If this is right

  • A content-only influence detector trained on historical campaigns will systematically over-flag native Russian speakers writing in English, with the highest risk on sentences about US politics or cryptocurrency.
  • Named entity masking is a partial mitigation: it reduces false positives in the presence of frequent named entities, but a statistically significant language gap remains.
  • Evaluation pipelines for such detectors should include L2 English speaker test sets, built from contextual cues such as subreddit flair, to measure and track this bias.
  • Mitigations to explore include adding genuine L2 English comments as negative training examples and combining content features with account metadata in an ensemble.
  • Similar bias is likely for any influence campaign attributed to a country whose population largely writes English as a second language, putting other language communities at risk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the gap is linguistic rather than topical would require topic matching; if the gap survives, it points to subtle L2 syntactic or lexical cues shared between Russian-native writers and influence-account text, and NEM alone would not address those cues.
  • The same measurement protocol could be applied to other native-language groups and to newer detectors, including generative-text detectors, where false-positive risk for non-native writers may be even larger because the models are often trained on native-like web text.
  • The frequent-entity list (US, Trump, Bitcoin, Russia, ISIS) is itself a snapshot of one campaign; retraining the detector on future campaigns would shift the list, but the fairness-testing protocol transfers unchanged.
  • Platforms could operationalize this result by auditing any content-based moderation model on L2 English communities before deployment, using the false-positive gap as a release criterion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper trains a BERT-based sentence classifier to distinguish comments from Reddit's 2017 suspicious-accounts transparency list (Corpus I) from randomly sampled comments (Corpus II), then evaluates false positive rates on a third corpus of self-identified L1 Russian and L1 English Reddit users. It reports that false positive rates are substantially higher for L1 Russian than for L1 English users, that the gap grows when sentences mention frequent named entities from the training data, and that replacing named entities with part-of-speech tags reduces but does not eliminate the gap. The authors argue that this demonstrates an ethical risk of content-based influence detection for Russian-speaking users, and they propose mitigation directions such as including L2 English negative examples and tracking language-group error rates.

Significance. If the topic-confound issue is addressed, this paper makes a significant contribution: it provides a reproducible, empirically grounded demonstration that a state-of-the-art content classifier has disparate false positive rates across language groups, and that named-entity masking is only a partial mitigation. The use of the official Reddit transparency list, repeated cross-validation with significance testing, and public code are concrete strengths. The paper also usefully transfers the 'topic bias' concept from native language identification to influence campaign detection, which is an important and understudied fairness concern.

major comments (2)
  1. [§4.1.3, Table 3, §6] The central claim that L1 Russian writers are at higher false-positive risk is confounded by the construction of Corpus III. The L1 Russian evaluation set is assembled from European-subreddit self-identified flairs and from AskARussian, whereas the L1 English set is drawn from users in English-speaking countries; no topic or subreddit matching is reported. Because the model is topic-sensitive (the FNE rows in Table 3 shift FPR by about 6 to 16 points relative to the unfiltered rows), the gap between 63.82% and 38.82% (unmasked) and between 43.72% and 36.10% (masked) may reflect differences in topic distribution rather than the author's L1. This is load-bearing for the Section 6 statement that 'users with Russian as a first language... are at increased risk of false positive classification.' A matched evaluation set (same subreddits or posts, or topic-controlled sampling) or a topic-adjusted analysis is needed to separate language from topic; as it stands, the data support only the narrower claim that users in these particular Russian-associated communities and topics are at higher risk.
  2. [§5, Table 3] The significance testing for the false-positive-rate comparisons is under-specified. The paper reports 10 runs of 10-fold cross-validation and then states that the Table 3 statistics were 'computed using a paired-sample t-test,' but it does not state how many observations entered each t-test. If the paired units are the 10 runs, the smallest t-statistic in Table 3 (3.67) gives a two-tailed p-value of approximately 0.005, not p < 0.001 as claimed. If the units are the 100 fold models, the corrected repeated-CV adjustment should be described. Please report the number of units, the correction used, and exact p-values or confidence intervals.
minor comments (4)
  1. [§1.1, §4.1.3, §4.4] There are several typos: 'significant ethically considerations' should be 'significant ethical considerations' in Section 1.1; 'The results in a total of 774,702 comments' should be 'This results in a total...' in Section 4.1.3; and 'named-entitiy recognition' should be 'named-entity recognition' in Section 4.4.
  2. [§4.1.3, Table 3] Please report the number of sentences in each evaluation cell (L1En, L1Ru, L1En-FNE, L1Ru-FNE). The paper gives 1.9 million for the full L1 Russian sentence set but not the counts for the other subsets, which is needed to assess the stability of the reported proportions.
  3. [Table 3] The false-positive rates are reported only as point estimates with t-statistics. Adding confidence intervals or effect sizes would make the magnitudes of the differences easier to interpret, especially for the masked-model comparisons where the gaps are smaller.
  4. [§4.2] For reproducibility, please include the exact version or commit hash of the code repository [23] and specify the spaCy model used for named entity recognition, as different versions can produce different entity annotations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims rest on external ground truth, independent train/evaluation splits, and a measured preprocessing intervention.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. The positive class is defined externally by Reddit's 2017 transparency report list of suspected Russian Internet Research Agency accounts, the negative training class is an external random-comment corpus, and the L1 Russian versus L1 English evaluation corpus is assembled from self-reported user flair and country metadata rather than from model outputs. The BERT classifier is trained and evaluated on standard 10x10-fold cross-validation splits, and the false-positive rates in Table 3 are measured results, not fitted values renamed as predictions. Named entity masking (NEM) is a preprocessing transformation that replaces named entities with part-of-speech tags, and the 'frequent named entities' (FNE) filter is an evaluation subset derived from entity frequencies in the training corpus; this does not constitute fitting a parameter to the evaluation set and then reporting that parameter as a finding. The masking literature and prior NLI work are cited for technique context, but the paper's actual contribution—quantifying the FPR disparity and showing that masking only partially reduces it—is not an input of those citations. The reader's flagged concern about a topic mismatch between the L1Russian and L1English evaluation sets is a potential validity/confounding issue, not a circularity issue: the source distributions are external and independent of the model's fitted values. No self-citation chain, uniqueness theorem, or ansatz smuggled in by citation is load-bearing here. The ethical conclusion is a direct interpretation of measured classifier behavior on externally sourced data, so no significant circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The experiment is a standard supervised learning setup with external ground truth. No new physical or conceptual entities are introduced. The main assumptions are about the reliability of the ground truth labels, the representativeness of the random and L2 corpora, and the sufficiency of BERT embeddings as features. The FNE list is a manually chosen evaluation filter, not a fitted model parameter.

free parameters (2)
  • Frequent named entity (FNE) list = US, TIE, Trump, Bitcoin, Hillary, America, Russia, Russian, ISIS, BTC
    Manually selected top ten named entities in Corpus I, omitting DATE, CARDINAL, PERCENT and the emoji ':D'. This list defines the FNE evaluation subsets; changing the list would change the reported FPR values on those subsets. It is a hand-chosen selection rather than a fitted model parameter.
  • Sentence length threshold = 10 characters
    Sentences shorter than 10 characters are discarded during preprocessing; the cutoff is a design choice that affects the evaluation sets.
assumptions (4)
  • domain assumption The Reddit transparency report list of 944 accounts correctly identifies influence accounts.
    The paper treats the official list as ground truth positive labels (Section 4.1.1). If this list is noisy or incomplete, the classifier learns something other than influence behavior.
  • domain assumption Corpus II and Corpus III contain no influence-account comments.
    The paper assumes random comments and L2-user comments are negative examples based on the scale of past campaigns and Reddit volume (Sections 4.1.2 and 4.1.3). If influence accounts are present in these corpora, the measured false positive rates are biased.
  • domain assumption Self-identified flair indicates the user's native language.
    Corpus III labels L1 Russian and L1 English based on subreddit flair in the L2 corpus and AskARussian (Section 4.1.3). Flair may not reliably correspond to true L1.
  • domain assumption BERT sentence embeddings capture features useful for distinguishing influence-account text.
    The method relies on the pretrained BERT model's [CLS] representation as a sufficient feature space (Section 4.3). If BERT embeddings do not encode relevant stylistic signals, the measured bias is about the embedding model rather than text content generally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Ethical Content-Based Detection of Online Influence Campaigns." pith.science (2026). https://pith.science/paper/D2KMV4MO

@misc{pith2026190811030,
  author       = {Pith},
  title        = {Pith review of: Towards Ethical Content-Based Detection of Online Influence Campaigns},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2KMV4MO}},
  note         = {Machine review of arXiv:1908.11030}
}
read the original abstract

The detection of clandestine efforts to influence users in online communities is a challenging problem with significant active development. We demonstrate that features derived from the text of user comments are useful for identifying suspect activity, but lead to increased erroneous identifications when keywords over-represented in past influence campaigns are present. Drawing on research in native language identification (NLI), we use "named entity masking" (NEM) to create sentence features robust to this shortcoming, while maintaining comparable classification accuracy. We demonstrate that while NEM consistently reduces false positives when key named entities are mentioned, both masked and unmasked models exhibit increased false positive rates on English sentences by Russian native speakers, raising ethical considerations that should be addressed in future research.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 49 canonical work pages

  1. [1]

    in- fluence accounts

    INTRODUCTION 1.1. Overview The recent surge in media coverage of nation-state efforts t o influ- ence public perception in online communities, as well as the release of large datasets by major social media platforms such as Fac ebook [1], Reddit [2], and Twitter [3], has led to a notable increas e in re- search into the detection of online influence campaig...

    work page 2017

  2. [2]

    topic bias

    RELATED WORK Past work during the 2017 NLI Shared Task [10] has explored th e state-of-the-art in NLI, demonstrating successful combin ations of semantic and syntactic features for differentiating langu age learn- ers from native English speakers. However, this task did not reflect highly fluent advanced non-native speakers, which represen t a much more cha...

    work page 2017

  3. [3]

    of suspected Russian Internet Res earch Agency origin

    METHODOLOGY This paper focuses on analyzing the language characteristics of com- ments posted by accounts “of suspected Russian Internet Res earch Agency origin” released by link-aggregation and discussio n web- site Reddit within their 2017 transparency report on April 1 0, 2018 [7]. Using state-of-the-art natural language processing ( NLP) model BERT [2...

    work page 2017

  4. [4]

    of suspec ted Russian Internet Research Agency origin

    EXPERIMENTAL SETUP All experiments were conducted on a n1-standard-2 (2 vCPUs, 7.5 GB memory) Google Cloud instance, with a Tensor Processing U nit (TPU) v3-8. The code for the experiment is available online [ 23]. 4.1. Datasets This analysis relies on three corpora, as detailed below. 4.1.1. Corpus I: Comments from 2017 Reddit transparency rep ort This d...

    work page 2017

  5. [5]

    Normalize datasets into similar formats using regular ex pres- sions to remove extraneous escape characters

    work page

  6. [6]

    Perform sentence tokenization using Python NLTK [29] to extract sentences from comments

    work page

  7. [7]

    Remove newline characters, Reddit quote markdown charac - ters, and horizontal tab characters ( &#009;)

    work page

  8. [8]

    Remove all URLs and replace with [URL] token

    work page

Show all 52 references
  1. [9]

    V ery small sen- tences are poorly-suited to the classifier and may introduce noise

    Discard sentences shorter than 10 characters. V ery small sen- tences are poorly-suited to the classifier and may introduce noise

  2. [10]

    sentence embeddings

    Run full BERT tokenization pipeline [30], which includes converting to lowercase, WordPiece [31] tokenization, pun c- tuation splitting, and invalid character removal. 4.3. BERT Sentence Embedding Classification Sentence embedding refers to the family of techniques where by sen...

  3. [11]

    The un- masked model does however retain a slight advantage on the tr ained classification task

    RESULTS Table 2 illustrates that the performance of the NE masked mod el (NEMM) is comparable to that of the unmasked model when dis- tinguishing between sentences written by randomly sampled Reddit users and sentences written by suspected influence accounts . The un- masked mo...

  4. [12]

    Wh en both of these conditions coincide, the effect is magnified su bstan- tially, giving the highest percentage of false positives in the evalua- tion set

    ANALYSIS The results described in §5 indicate that models trained exclusively on content features of existing influence campaigns disprop ortion- ately misclassify speakers of that language, as well as user s who use specific named entities common to past influence accounts. Wh e...

  5. [13]

    ACKNOWLEDGEMENTS Research supported with Cloud TPUs from Google’s TensorFlo w Research Cloud (TFRC)

  6. [14]

    CONCLUSION We conclude that the use of content-based features without s afe- guards creates the potential for discrimination against us ers of spe- cific language backgrounds, especially when they are engage d in speech that contains common named entities that often reflec t po-...

  7. [15]

    Facebook launches n ew initiative to help scholars assess social media’s impact on elec- tions,

    Elliot Schrage and David Ginsberg, “Facebook launches n ew initiative to help scholars assess social media’s impact on elec- tions,” Apr 2018

    2018

  8. [16]

    Reddit transparency report: Suspicious accounts,

    “Reddit transparency report: Suspicious accounts,” https://www.reddit.com/wiki/suspiciousaccounts, Accessed: 2019-04-20

    2019

  9. [17]

    Twitter elections integrity dataset,

    “Twitter elections integrity dataset,” https://about.twitter.com/en_us/values/elections-in tegrity.html, Accessed: 2019-04-20

    2019

  10. [18]

    United states alexa rankings,

    “United states alexa rankings,” https://www.alexa.com/topsites/countries/US, Accessed: 2019-06-03

    2019

  11. [19]

    Canada alexa rankings,

    “Canada alexa rankings,” https://www.alexa.com/topsites/countries/CA, Accessed: 2019-06-03

    2019

  12. [20]

    Tweeting made easier: official blog post,

    Aliza Rosen, “Tweeting made easier: official blog post,” https://blog.twitter.com/en_us/topics/product/2017/tweetingmadeeasier.html, Accessed: 2019-06-05

    2017

  13. [21]

    Reddit 2017 transparency re- port and suspect account findings,

    “Reddit 2017 transparency re- port and suspect account findings,” https://www.reddit.com/r/announcements/comments/8bb85p/reddits_2017_transparency_report_and_suspect/, Accessed: 2019-05-23

    2017

  14. [22]

    V erifying online user identity using stylometric analysis for short messages,

    Marcelo Brocardo, Issa Traore, Sherif Saad, and Isaac Wo un- gang, “V erifying online user identity using stylometric analysis for short messages,” Journal of Networks, vol. 9, 12 2014

    2014

  15. [23]

    Identifying russian trol ls on reddit with deep learning and bert word embeddings,

    Henry Weller and Jeffrey Woo, “Identifying russian trol ls on reddit with deep learning and bert word embeddings,” http://web.stanford.edu/class/cs224n/reports/custom/15739845.pdf, Accessed: 2019-05-22

    2019

  16. [24]

    A report on the 2017 native language identification shared task,

    Shervin Malmasi, Keelan Evanini, Aoife Cahill, Joel Te treault, Robert Pugh, Christopher Hamill, Diane Napolitano, and Yao Qian, “A report on the 2017 native language identification shared task,” in Proceedings of the 12th W orkshop on In- novative Use of NLP for Building Educa...

    2017

  17. [25]

    N ative language cognate effects on second language lexical choice ,

    Ella Rabinovich, Y ulia Tsvetkov, and Shuly Wintner, “N ative language cognate effects on second language lexical choice ,” TACL, vol. 6, pp. 329–342, 2018

    2018

  18. [26]

    Automatically assess- ing lexical sophistication: Indices, tools, findings, and a pplica- tion,

    Kristopher Kyle and Scott A. Crossley, “Automatically assess- ing lexical sophistication: Indices, tools, findings, and a pplica- tion,” TESOL Quarterly, vol. 49, no. 4, pp. 757786, Sep 2014

    2014

  19. [27]

    Nati ve lan- guage identification with user generated content,

    Gili Goldin, Ella Rabinovich, and Shuly Wintner, “Nati ve lan- guage identification with user generated content,” in Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - Novem- ber 4, 2018, 2018, pp. 3591–3601

    2018

  20. [28]

    Measuring interlangu age: Native language identification with l1-influence metrics,

    Julian Brooke and Graeme Hirst, “Measuring interlangu age: Native language identification with l1-influence metrics,” in LREC, 2012

    2012

  21. [29]

    Arabic native language identification,

    Shervin Malmasi and Mark Dras, “Arabic native language identification,” in Proceedings of the EMNLP 2014 W orkshop on Arabic Natural Language Processing (ANLP), Doha, Qatar, Oct. 2014, pp. 180–186, Association for Computational Lin- guistics

    2014

  22. [30]

    Subdialectal differences in Sorani Kur- dish,

    Shervin Malmasi, “Subdialectal differences in Sorani Kur- dish,” in Proceedings of the Third W orkshop on NLP for Similar Languages, V arieties and Dialects (V arDial3), Osaka, Japan, Dec. 2016, pp. 89–96, The COLING 2016 Organizing Committee

    2016

  23. [31]

    Still out there: Modeling and identifying russian troll accounts on twitter,

    Jane Im, Eshwar Chandrasekharan, Jackson Sargent, Pai ge Lighthammer, Taylor Denby, Ankit Bhargava, Libby Hemphill, David Jurgens, and Eric Gilbert, “Still out there: Modeling and identifying russian troll accounts on twitter,” CoRR, vol. abs/1901.11162, 2019

    1901 arXiv

  24. [32]

    A holistic system for troll de - tection on twitter,

    Paolo Fornacciari, Monica Mordonini, Agostino Poggi, Laura Sani, and Michele Tomaiuolo, “A holistic system for troll de - tection on twitter,” Computers in Human Behavior, vol. 89, pp. 258–268, Dec. 2018

    2018

  25. [33]

    Predicting russian trolls using reddit comments,

    “Predicting russian trolls using reddit comments,” https://towardsdatascience.com/predicting-russian-t rolls-using-reddit -comments-57a707653184 , Accessed: 2019-04-20

    2019

  26. [34]

    BERT: pre-training of deep bidirectional transformers for language understanding,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018

    2018 arXiv

  27. [35]

    Attention is all you need,

    Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszko - reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Il- lia Polosukhin, “Attention is all you need,” CoRR, vol. abs/1706.03762, 2017

    2017 arXiv

  28. [36]

    The Application of Forensic Linguistic s in Cybercrime Investigations,

    Ria C Perkins, “The Application of Forensic Linguistic s in Cybercrime Investigations,” Policing: A Journal of Policy and Practice, 12 2018

    2018

  29. [37]

    Code for experimental evaluation,

    “Code for experimental evaluation,” https://github.com/ecrows/l2-reddit-experiment , Accessed: 2019-06-15

    2019

  30. [38]

    Reddit suspicious accounts dataset,

    “Reddit suspicious accounts dataset,” https://github.com/ALCC01/reddit-suspicious-account s, Accessed: 2019-04-20

    2019

  31. [39]

    Josh russel reddit investigation,

    “Josh russel reddit investigation,” https://www.reddit.com/user/eye_josh/comments/843beq/russian_reddit_accounts_and_links/, Accessed: 2019-04-20

    2019

  32. [40]

    Reddit-trolls repository of brandon punturo on githu b,

    “Reddit-trolls repository of brandon punturo on githu b,” https://github.com/brandonjpunturo/Reddit-Trolls, Accessed: 2019-05-28

    2019

  33. [41]

    Reddit comment archive datasets,

    “Reddit comment archive datasets,” https://files.pushshift.io/reddit/comments/, Accessed: 2019-04-20

    2019

  34. [42]

    The reddit l2 corpus,

    “The reddit l2 corpus,” http://cl.haifa.ac.il/projects/L2/, Accessed: 2019-04-20

    2019

  35. [43]

    Nltk: The natural langua ge toolkit,

    Edward Loper and Steven Bird, “Nltk: The natural langua ge toolkit,” in In Proceedings of the ACL W orkshop on Effective Tools and Methodologies for Teaching Natural Language Pro- cessing and Computational Linguistics. Philadelphia: Ass oci- ation for Computational Linguistics , 2002

    2002

  36. [44]

    Google research bert git repository,

    “Google research bert git repository,” https://github.com/google-research/bert, Accessed: 2019-05-28

    2019

  37. [45]

    Google sentencepiece git repository,

    “Google sentencepiece git repository,” https://github.com/google/sentencepiece, Accessed: 2019-05-28

    2019

  38. [46]

    A neural probabilistic language model,

    Y oshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Chris- tian Janvin, “A neural probabilistic language model,” J. Mach. Learn. Res., vol. 3, pp. 1137–1155, Mar. 2003

    2003

  39. [47]

    Distributed representations of words and phrases and their compositionality,

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeffrey Dean, “Distributed representations of words and phrases and their compositionality,” CoRR, vol. abs/1310.4546, 2013

    2013 arXiv

  40. [48]

    Spacy python library,

    “Spacy python library,” https://github.com/explosion/spaCy, version 2.0.16

  41. [49]

    Simple and acc u- rate dependency parsing using bidirectional lstm feature r epre- sentations,

    Eliyahu Kiperwasser and Y oav Goldberg, “Simple and acc u- rate dependency parsing using bidirectional lstm feature r epre- sentations,” Transactions of the Association for Computational Linguistics, vol. 4, pp. 313–327, 2016

    2016

  42. [50]

    Inference for the gen eral- ization error,

    Claude Nadeau and Y oshua Bengio, “Inference for the gen eral- ization error,” Machine Learning, vol. 52, no. 3, pp. 239–281, Sep 2003

    2003

  43. [51]

    Evaluating the repl i- cability of significance tests for comparing learning algo- rithms,

    Remco R. Bouckaert and Eibe Frank, “Evaluating the repl i- cability of significance tests for comparing learning algo- rithms,” in Advances in Knowledge Discovery and Data Min- ing, Honghua Dai, Ramakrishnan Srikant, and Chengqi Zhang, Eds., Berlin, Heidelberg, 2004, pp. 3–12, ...

    2004

  44. [52]

    Language models are unsuper- vised multitask learners,

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dari o Amodei, and Ilya Sutskever, “Language models are unsuper- vised multitask learners,” 2018

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.