Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Hatevolution: What Static Benchmarks Don't Tell Us

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Static hate speech benchmark scores misalign with how well 20 language models handle evolving hate speech.

desk verdict Solid empirical study of temporal hate speech benchmarking that overstates its central correlation claim; worth reviewing with a required statistical revision. read the letter →

arxiv 2506.12148 v1 pith:USBEMR6S submitted 2025-06-13 cs.CL

Chiara Di Bonaventura , Barbara McGillivray , Yulan He , Albert Meroño-Peñuela This is my paper
classification cs.CL
keywords hatespeechdetectionlanguageevolutionstaticbenchmarkstime-sensitiveevaluationtemporalbiasneologismscounterfactualinvariancezero-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hate speech language changes as society changes: words shift meaning, gain or lose offensiveness, and new words appear. Yet the benchmarks used to test how safely language models handle hate speech are frozen snapshots, tied to the moment they were built. This paper asks whether a model's score on such static benchmarks predicts how well it handles hate speech as it evolves, and evaluates 20 language models on two time-sensitive tasks: year-by-year shifts in hateful language on a 2017–2022 corpus, and sentence pairs in which an ordinary word is replaced by a recently coined one. The two kinds of evaluation barely agree: rank correlations between static and time-sensitive scores are negative or near zero, while the static benchmarks agree with each other. The authors conclude that static evaluations can overstate the safety of hate speech detectors and argue for time-sensitive benchmarks instead.

What carries the argument

The argument is carried by three measurement objects. First, time-sensitive macro F1, defined as $\frac{1}{T}\sum_{t} F1_t$ over the years 2017–2022 on the Singapore Online Attacks corpus, rewards models that perform well every year rather than on a single frozen test set. Second, counterfactual invariance — the requirement that swapping an irrelevant part of the input, here a target word replaced by a neologism, should not change the prediction — is operationalised on 341 sentence pairs as a label flip rate (how often the swap changes the model's verdict) and a hallucination rate (how often the model stops following the classification instruction). Third, Spearman's rank correlation between the model rankings produced by the time-sensitive measures and the rankings produced by four static benchmarks is the statistic that carries the conclusion of misalignment.

What would settle it

Compute the same Spearman correlations on a much larger timestamped hate speech corpus (tens of thousands of posts per year instead of hundreds) and on thousands of neologism pairs instead of 341, with confidence intervals tight enough to exclude zero; if the static-versus-time-sensitive coefficients come out clearly positive, or if the negative values vanish within sampling noise, the claimed misalignment is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that static hate speech benchmarks and time-sensitive evaluations are misaligned to the point of negative or zero correlation. Across 20 zero-shot prompted language models, Spearman's rank correlation between performance on each of four static benchmarks (HateCheck, Dynabench, HateXplain, Implicit Hate Corpus) and a time-sensitive macro F1 averaged over 2017–2022 is negative in every case, from about -0.15 to -0.28; correlations against the neologism experiment are negative or near zero, from -0.31 to +0.19. In contrast, the four static benchmarks correlate positively among themselves, with an average coefficient of 0.36. The authors read this as evidence that the model which excels on a static snapshot of hateful language is not the model that best tracks hate speech as it evolves, and they document the behaviour underneath: per-year F1 for detecting hateful content swings widely and largely declines from 2017 to 2022, and 6 of 20 models change their verdict on more than 10% of sentences where a target word is swapped for a neologism.

Load-bearing premise

The load-bearing premise is that the time-sensitive numbers are true signals rather than noise: the Singapore corpus provides only about 500 posts per year for 2017–2022, and the neologism experiment rests on 341 sentence pairs annotated with a Cohen's Kappa of 0.67, so the per-year F1 swings, flip rates, and the resulting rank correlations must reflect language evolution rather than sampling variation or annotation disagreement.

Editorial extensions

If this is right

  • If the misalignment is real, any safety evaluation that ranks models on static hate speech benchmarks will misorder them with respect to robustness to evolving language.
  • High static scores do not imply robustness to new or shifting terminology; several of the best static performers show some of the highest label flip rates on neologism counterfactuals.
  • Dynamic adversarial training on successive Dynabench rounds does not confer robustness to evolving language, since the later-round RoBERTa-dyna models score lower on time-sensitive macro F1 than the first-round model.
  • Adopting time-sensitive evaluation, such as per-year macro F1 and neologism counterfactual tests, would give a more reliable picture of a deployed hate speech detector's safety.
  • Larger model size improves classification performance but does not fix robustness: GPT-4o flips labels on neologisms more often than smaller RoBERTa-dyna models, and size mainly reduces hallucination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editor's inference: extending the paper's lexical-overfit explanation, a model's static benchmark score should degrade more steeply as the gap widens between its training data cutoff and the language of the test text, a prediction testable by applying today's benchmarks to newly collected text.
  • Editor's inference: the two-experiment design transfers to other domains with drifting language, such as misinformation, political abuse, or trolling, where timestamped corpora already exist; per-year F1 plus neologism counterfactuals would reveal whether this misalignment is generic.
  • Editor's inference: tracking the Spearman coefficient across successive releases of static benchmarks would separate two possibilities the paper does not distinguish, namely that all static benchmarks age versus that some model families are inherently more time-robust.
  • Editor's inference: because the per-year estimates rest on roughly 500 posts each, re-running the correlation analysis on a larger timestamped corpus is the natural robustness check, and if the negative coefficients persist at scale the case for time-sensitive benchmarking is much stronger.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether performance on static hate speech benchmarks predicts robustness to language change. It conducts two time-sensitive experiments: Experiment 1 measures yearly macro-F1 on the Singapore Online Attacks dataset (2017–2022) for 20 language models, and Experiment 2 measures label-flip and hallucination rates on 341 sentence pairs from NeoBench in which a target word is replaced by a neologism. The models are also evaluated on four static hate speech benchmarks (HateCheck, Dynabench, HateXplain, Implicit Hate). The paper reports Spearman rank correlations between static and time-sensitive evaluation rankings, finding point estimates that are mostly negative or near zero for Experiment 1, and concludes that there is a 'clear misalignment' between static and time-sensitive evaluations, so that static benchmarks are unreliable predictors of robustness to evolving language.

Significance. If the central claim were fully supported, the paper would make a useful methodological contribution by showing that static hate speech benchmarks can give a misleading picture of model safety under language change. The study has clear strengths: it introduces two time-sensitive evaluation settings, annotates a new subset of NeoBench for hate speech, releases the code and data, and includes a contamination analysis. The descriptive results (yearly volatility, high label-flip rates for several models) are plausible and interesting. However, the main statistical evidence for the headline 'misalignment' claim is currently too weak to carry the paper's conclusions.

major comments (3)
  1. [Section 3, Table 5] The claim of a 'clear misalignment' and of a 'negative correlation' between static and time-sensitive evaluations is not supported by the reported confidence intervals. Every 90% CI in Table 5 includes zero; for example, the Experiment 1 vs. HateCheck correlation is -0.266 with CI (-0.586, 0.126), and the Experiment 2 vs. Implicit Hate correlation is 0.191 with CI (-0.203, 0.532). The data are therefore consistent with zero correlation (and even with modest positive correlation for some pairs). The point estimates skew negative for Experiment 1, which is suggestive, but they do not establish a negative correlation. Please report a formal test against the null of zero correlation or a bootstrap/permutation-based uncertainty measure, and revise the abstract and conclusions to state that the data show 'no evidence of positive alignment' rather than a demonstrated negative misalignment.
  2. [Appendix D, Tables A7–A10; Section 3] The static benchmark rankings used in the correlation analysis include models with very low instruction-following rates. For example, DeepSeek LLM-7b produces expected outputs on only 0.54% of HateCheck, 7.86% of Dynabench, 10.01% of HateXplain, and 2.75% of Implicit Hate samples; mT0-small also has a low expected-output rate on multiple benchmarks. Because macro-F1 is computed only on the valid outputs, these models' static scores are based on extremely small and likely non-representative subsamples, making their ranks effectively arbitrary. Including such ranks in the Spearman correlations of Table 5 can distort the coefficients. The authors should either exclude models with insufficient valid outputs, treat their scores as missing, or report a sensitivity analysis with these models removed.
  3. [Section 3, Tables 3 and 4] The time-sensitive metrics are presented without any uncertainty quantification, which weakens the reliability of the rankings that feed into Table 5. In Experiment 1, annual macro-F1 values are computed from roughly 500 posts per year, and the claim of 'significant changes in the macro F1 scores year by year' is not supported by any significance test; the reported standard deviations alone do not establish that year-to-year differences are meaningful. In Experiment 2, with 341 sentence pairs, the label-flip and hallucination rates carry binomial uncertainty that is not reported; for instance, a flip rate of 14.14% (FLAN-Alpaca-xl) versus 3.98% (FLAN-Alpaca-large) may not be reliably different at this sample size. Reporting bootstrap confidence intervals for the time-sensitive metrics would help assess the stability of the model rankings.
minor comments (5)
  1. [Section 2, Models; Tables 1, 3, 4] TimeLMs appears in Tables 3 and 4 but not in Table 1, and the paper says 20 models while the tables list 21 rows when TimeLMs is included. Please clarify whether TimeLMs is counted among the 20 models and whether it is included in the correlation analyses.
  2. [Section 3, Experiment 2] The definition of hallucination in the text is the conditional rate 'the model does not follow the instruction when given the counterfactual s2 but does follow the instruction when given s1,' which matches the formula, but Table A6 also reports an OR variant (hals1,s2). Please make explicit which definition is used in Table 4 and keep notation consistent throughout.
  3. [Section 2, Table 2] Table 2 lists 'NeoBench 682' while the text says 341 sentence pairs; making explicit that 682 is the total number of sentences (341 pairs) would avoid confusion.
  4. [Section 6, Limitations] The limitations section is candid, but it does not mention the small annual sample sizes in Experiment 1 or the moderate inter-annotator agreement (Kappa = 0.67) in Experiment 2 as potential threats to the stability of the time-sensitive metrics. Adding these would strengthen the paper's transparency.
  5. [Throughout] There are several presentation issues: 'Plaza-del arco' should be capitalized as 'Plaza-del Arco'; 'Carlini et al.' in the text lacks a year; and the reference to 'Xue et al., 2021' for T5 appears to actually cite the mT5 paper, so the citation should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the misalignment claim is an empirical correlation between independently measured model rankings, with no fitted parameter or load-bearing self-citation.

full rationale

This paper is an empirical measurement study rather than a derivation. The time-sensitive metrics (year-wise macro F1; counterfactual label-flip and hallucination rates) are defined directly on the Singapore Online Attack and NeoBench-derived data, and the static benchmark scores are obtained on four separate established datasets. Table 5 then reports Spearman correlations between the resulting model rankings; no equation in the paper defines a static score in terms of a time-sensitive score, and no parameter is fitted to one evaluation and then renamed as a prediction of the other. The self-citations that appear (e.g., Di Bonaventura et al. 2025 for benchmark-dimension framing, and McGillivray et al. 2022 for the 'gammon' example) are contextual and are not load-bearing for the central empirical claim: the correlations would be the same if those citations were removed. The manual annotation of the NeoBench sample by the authors is a data-construction step, not a circular fit, since the resulting ground truth is then used to score all models externally against an independent static benchmark ranking. The paper's main weakness is statistical—the 90% confidence intervals in Table 5 all span zero—but that is a question of whether the evidence supports the strength of the claim, not a circularity in the derivation chain. Accordingly, no circular step is present and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three domain assumptions: dataset representativeness, validity of the zero-shot prompt protocol, and quality of the manual annotations. These are reasonable but not independently verified. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption The two chosen datasets (Singapore Online Attacks and NeoBench extension) adequately represent evolving hate speech in English.
    The study's conclusions generalize only if these datasets capture the relevant language change; Section 2 introduces them as the basis for Experiments 1 and 2.
  • domain assumption Zero-shot prompting with the prompt 'Classify the input text as respectful or toxic' is a fair and valid evaluation protocol for all 20 models.
    The authors adopt Plaza-del arco et al. (2023) verbalization; differences in instruction-following could confound comparisons; Section 2 'Models' and App A.
  • domain assumption Manual hate speech annotations with Cohen's Kappa 0.67 provide reliable ground truth for the neologism counterfactuals.
    Section 2 'Experiment 2' reports Kappa 0.67; disagreement is resolved by majority vote, and errors in this gold set directly affect label flip and F1 estimates.
  • standard math The statistical inference using Spearman's rank correlation with n=20 and Fisher z-transformation is appropriate for the comparison.
    App E uses Spearman with Fisher z confidence intervals; valid but low power.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hatevolution: What Static Benchmarks Don't Tell Us." pith.science (2026). https://pith.science/paper/USBEMR6S

@misc{pith2026250612148,
  author       = {Pith},
  title        = {Pith review of: Hatevolution: What Static Benchmarks Don't Tell Us},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USBEMR6S}},
  note         = {Machine review of arXiv:2506.12148}
}
read the original abstract

Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains under-explored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporal misalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 29 canonical work pages

  1. [1]

    Rabab Alkhalifa, Elena Kochkina, and Arkaitz Zubiaga. 2023. Building for tomorrow: Assessing the temporal persistence of text classifiers. Information Processing & Management, 60(2):103200

    work page 2023

  2. [2]

    Eduardo G Altmann, Janet B Pierrehumbert, and Adilson E Motter. 2009. Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words. PLOS one, 4(11):e7678

    work page 2009

  3. [3]

    Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis Espinosa-Anke, Jiaxin Pei, and Jose Camacho-Collados. 2023. Supertweeteval: A challenging, unified and heterogeneous benchmark for social media nlp research. In Findings of the Association for Computational Linguistics: EMNLP 2023

    work page 2023

  4. [4]

    Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. 2022. https://doi.org/10.18653/v1/2022.findings-acl.88 Entropy-based attention regularization frees unintended bias mitigation from lists . In Findings of the Association for Computational Linguistics: ACL 2022, pages 1105--1119, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.findings-acl.88 2022

  5. [5]

    Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662--678

    work page 2020

  6. [6]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403

    arXiv 2024

  7. [7]

    Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662

    arXiv 2023

  8. [8]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations

    work page

Show all 57 references
  1. [9]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37--46

    1960

  2. [10]

    Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Dan Gillick, Jacob Eisenstein, and William Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257--273

    2022

  3. [11]

    Chiara Di Bonaventura, Lucia Siciliani, Pierpaolo Basile, Albert Merono-Penuela, and Barbara McGillivray. 2025. From detection to explanation: Effective learning strategies for LLM s in abusive language research. In Proceedings of the 31st International Conference on Computati...

    2025

  4. [12]

    Felix Drinkall, Eghbal Rahimikia, Janet Pierrehumbert, and Stefan Zohren. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.208 Time machine GPT . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3281--3292, Mexico City, Mexico. Association f...

    2024 doi

  5. [13]

    Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. PloS one, 9(11):e113114

    2014

  6. [14]

    Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process...

    2021

  7. [15]

    Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. Test of time: A benchmark for evaluating llms on temporal reasoning. arXiv preprint arXiv:2406.09170

    2024 arXiv

  8. [16]

    Komal Florio, Valerio Basile, Marco Polignano, Pierpaolo Basile, and Viviana Patti. 2020. Time of your hate: The challenge of time in hate speech detection on social media. Applied Sciences, 10(12):4180

    2020

  9. [17]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.301 R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models . In Findings of the Association for Computational Linguist...

    2020 doi

  10. [18]

    Janosch Haber, Bertie Vidgen, Matthew Chapman, Vibhor Agarwal, Roy Ka-Wei Lee, Yong Keong Yap, and Paul R \"o ttger. 2023. https://doi.org/10.18653/v1/2023.acl-long.711 Improving the detection of multilingual online attacks with rich social media data from S ingapore . In Proc...

    2023 doi

  11. [19]

    Justin T Huang, Masha Krupenkin, David Rothschild, and Julia Lee Cunningham. 2023. The cost of anti-asian racism during the covid-19 pandemic. Nature human behaviour, 7(5):682--695

    2023

  12. [20]

    Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215

    2021 arXiv

  13. [21]

    Mali Jin, Yida Mu, Diana Maynard, and Kalina Bontcheva. 2023. Examining temporal bias in abusive language detection. arXiv preprint arXiv:2309.14146

    2023 arXiv

  14. [22]

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021. Dynabench: Rethinking benchmarking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of t...

    2021

  15. [23]

    Hirokazu Kiyomaru, Issa Sugiura, Daisuke Kawahara, and Sadao Kurohashi. 2024. https://aclanthology.org/2024.inlg-main.45/ A comprehensive analysis of memorization in large language models . In Proceedings of the 17th International Natural Language Generation Conference, pages ...

    2024

  16. [24]

    William Labov. 2011. Principles of linguistic change, volume 3: Cognitive and cultural factors, volume 3. John Wiley & Sons

    2011

  17. [25]

    Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural In...

    2021

  18. [26]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research

    2023

  19. [27]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364

    2019 arXiv

  20. [28]

    Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022. https://doi.org/10.18653/v1/2022.acl-demo.25 T ime LM s: Diachronic language models from T witter . In Proceedings of the 60th Annual Meeting of the Association for Comput...

    2022 doi

  21. [29]

    Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. 2022. Time waits for no one! analysis and challenges of temporal misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisti...

    2022

  22. [30]

    Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020. https://aclanthology.org/2020.lrec-1.592/ Leveraging contextual embeddings for detecting diachronic semantic shift . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4811--4819, Marseill...

    2020

  23. [31]

    Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867--14875

    2021

  24. [32]

    Barbara McGillivray, Malithi Alahapperuma, Jonathan Cook, Chiara Di Bonaventura, Albert Mero \ n o-Pe \ n uela, Gareth Tyson, and Steven Wilson. 2022. Leveraging time-dependent lexical features for offensive language detection. In Proceedings of the First Workshop on Ever Evol...

    2022

  25. [33]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...

    2023 doi

  26. [34]

    Flor Miriam Plaza-del arco, Debora Nozza, and Dirk Hovy. 2023. https://doi.org/10.18653/v1/2023.woah-1.6 Respectful or toxic? using zero-shot learning with language models to detect hate speech . In The 7th Workshop on Online Abuse and Harms (WOAH), pages 60--68, Toronto, Cana...

    2023 doi

  27. [35]

    Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023. On the challenges of using black-box apis for toxicity evaluation in research. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7595--7609

    2023

  28. [36]

    Jing Qian, Hong Wang, Mai ElSherief, and Xifeng Yan. 2021. Lifelong learning of hate speech classification on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages ...

    2021

  29. [37]

    Wenjun Qiu and Yang Xu. 2022. Histbert: A pre-trained language model for diachronic lexical semantic analysis. arXiv preprint arXiv:2202.03612

    2022 arXiv

  30. [38]

    Rosin and Kira Radinsky

    Guy D. Rosin and Kira Radinsky. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.112 Temporal attention for language models . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1498--1508, Seattle, United States. Association for Computational ...

    2022 doi

  31. [39]

    Paul R \"o ttger and Janet Pierrehumbert. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.206 Temporal adaptation of BERT and performance on downstream document classification: Insights from social media . In Findings of the Association for Computational Linguistics: EMN...

    2021 doi

  32. [40]

    Paul R \"o ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int...

    2021

  33. [41]

    Oscar Sainz, Jon Campos, Iker Garc \' a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.722 NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark . In Findings of th...

    2023 doi

  34. [42]

    Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, and Eneko Agirre. 2023 b . https://hitz-zentroa.github.io/lm-contamination/blog/ Did chatgpt cheat on your test?

    2023

  35. [43]

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.acl-long.828 Towards benchmarking and improving the temporal reasoning capability of large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  36. [44]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  37. [45]

    Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274--38290

    2022

  38. [46]

    Victor Veitch, Alexander D Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/8710ef761bbb29a6f9d12e4ef8e4379c-Paper.pdf Counterfactual invariance to spurious correlations in text classification . In Advances in Neura...

    2021

  39. [47]

    Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. https://doi.org/10.18653/v1/2021.acl-long.132 Learning from the worst: Dynamically generated datasets to improve online hate detection . In Proceedings of the 59th Annual Meeting of the Association for Comput...

    2021 doi

  40. [48]

    Ekaterina Vylomova and Nick Haslam. 2021. Semantic changes in harm-related concepts in english. Computational approaches to semantic change, 6:93

    2021

  41. [49]

    Yuqing Wang and Yun Zhao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.382 TRAM : Benchmarking temporal reasoning for large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 6389--6415, Bangkok, Thailand. Association for Com...

    2024 doi

  42. [50]

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. In International Conference on Learning Representations

    2021

  43. [51]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association...

    2021

  44. [52]

    Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, and Dacheng Tao. 2024. Safebench: A safety evaluation framework for multimodal large language models. arXiv preprint arXiv:2410.18927

    2024 arXiv

  45. [53]

    Jonathan Zheng, Alan Ritter, and Wei Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.749 NEO - BENCH : Evaluating robustness of large language models with neologisms . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2024 doi

  46. [54]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  47. [55]

    Eszter Zsisku, Arkaitz Zubiaga, and Haim Dubossarsky. 2024. Hate speech detection and reclaimed language: Mitigating false positives and compounded discrimination. In Proceedings of the 16th ACM Web Science Conference, pages 241--249

    2024

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.