Pith. sign in

REVIEW 6 major objections 5 minor 47 references

Beyond Binary Moderation: Identifying Fine-Grained Sexist and Misogynistic Behavior on GitHub with Large Language Models

T0 review · 6 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that prompt-tuned GPT-4o can classify GitHub comments into twelve sexism and misogyny labels with 84.1% accuracy, beating binary and keyword filters.

desk verdict The label-phrasing insight is real, but the headline MCC is an in-sample artifact and needs a held-out re-estimate before I'd trust the performance claim. read the letter →

arxiv 2507.20358 v1 pith:V5Y35T2D submitted 2025-07-27 cs.SE

Tanni Dev , Sayma Sultana , Amiangshu Bosu This is my paper
classification cs.SE
keywords sexismdetectionmisogynyGitHubmoderationmulti-classclassificationlargelanguagemodelspromptengineeringfew-shotlearningcontent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that instruction-tuned large language models, guided by carefully engineered prompts, can move beyond binary sexism detection and sort GitHub comments into twelve fine-grained labels: eleven distinct forms of sexism and misogyny plus a Neutral class. The authors argue this granularity matters because different harms call for different moderation responses, and because keyword filters and binary classifiers miss subtle, context-dependent slights that push women and gender minorities out of open-source communities. Their best configuration, GPT-4o with a prompt refined through 20 iterative versions, reaches 84.1% accuracy and a Matthews correlation coefficient (MCC) of 0.501 on 1,440 labeled GitHub issue comments, with a binary F1-score of 86.1%. The paper further claims this is the first multiclass classifier built specifically for sexism and misogyny in GitHub communications, and that its gains come mainly from prompt design—behavior-based category definitions, few-shot examples, and rules for sarcasm and overlapping harms—rather than from model scale alone.

What carries the argument

The load-bearing mechanism is a structured multi-class prompt frame, iterated through 20 versions, that wraps a 12-category rubric with role clarification, behavior-based definitions, a balanced set of few-shot examples (33 by the final version), explicit rules for resolving overlapping categories and reading sarcastic or dismissive tone, and a strict output format that requires category labels, confidence scores, and a short reasoning statement. The reasoning output is itself part of the machinery: the authors inspect the model's justifications to locate misclassifications and revise the prompt accordingly. The primary evaluation metric is the Matthews correlation coefficient (MCC), chosen because the class distribution is heavily imbalanced and MCC summarizes the full confusion matrix in a single number.

What would settle it

Re-annotate the 1,440 test comments—especially the 146 newly collected ones—with three independent annotators using the original rubric; if multi-label agreement lands near the source dataset's κ=0.421, label noise is large enough that the reported MCC of 0.501 could be an artifact of the chosen gold labels rather than a stable property of the model.

Watch

Extended reading notes

Core claim

The central claim is that prompt engineering can make an off-the-shelf LLM competitive at fine-grained moderation of sexist and misogynistic content in developer communication, without the large manually annotated training sets that supervised classifiers require. On the 1,440-comment evaluation set, the best model accurately distinguishes frequent and distinct categories such as Anti-LGBTQ+ and Dismissing while still misclassifying more than 70% of the comments in the subtle categories of Discredit, Physical Appearance, Sexual Harassment, and Sexual Objectification. The authors read this pattern as evidence that the remaining errors are concentrated in implicit, sarcastic, or overlapping harms rather than random noise, and they trace concrete improvements to specific prompt changes—for instance, renaming the abstract label 'Victim Blaming' to a behaviorally concrete one and adding tone indicators moved that category from zero F1 to strong F1 scores across models.

Load-bearing premise

The evaluation treats the hand-assigned labels as gold, including 146 newly collected comments verified by only two authors; if those labels are inconsistent with the original SGID schema, the reported MCC and F1 scores are not a true measure of the model's ability.

Editorial extensions

If this is right

  • GitHub maintainers could deploy a prompt-tuned LLM as a first-pass screener that flags suspicious comments and offers a short reason, leaving only flagged comments for human review.
  • The twelve-label output could replace keyword filters in code-of-conduct enforcement, reducing both missed microaggressions and false alarms triggered by surface keywords.
  • Because the approach does not require a large annotated training set, new harm categories or platforms could be handled by editing the definitions and examples in the prompt rather than retraining a classifier.
  • Rare or previously missed categories—such as victim blaming—become detectable when labels are renamed to describe observable behavior, given enough few-shot examples and overlap rules.
  • Binary precision of 98.25% with 76.6% recall means the method is safe for flagging, since it rarely brands neutral comments as harmful, but it still misses about a quarter of harmful comments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves open is feeding the surrounding issue thread or pull-request context into the prompt; several confusion pairs (Sexual Harassment versus Anti-LGBTQ+, Discredit versus Dismissing) hinge on whether a remark targets a person, which often only thread context reveals.
  • The paper's own numbers imply that label noise may set the current ceiling: multi-label annotator agreement in the source dataset was only κ=0.421, so part of what the authors count as model 'misclassification' could be disagreement between the model and one chosen gold label.
  • The prompt-refinement recipe—rename abstract labels, add behavioral cues, balance few-shot examples, resolve overlaps—should transfer to other platform-specific harm taxonomies, but the paper gives no evidence for cross-platform transfer, so that generalization remains untested.
  • Because inference was run once per prompt-model pair at temperature 0.1, a cheap robustness check would be to rerun the best prompt several times and measure the variance of the MCC; the paper does not report run-to-run stability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper presents a prompt-engineering study in which GPT-4o, LLaMA 3.3, and Mistral 7B are used to classify GitHub issue comments into twelve fine-grained categories of sexist and misogynistic behavior, plus Neutral. The authors construct a balanced 1,440-comment dataset from the SGID corpus and 146 newly collected samples, iteratively refine a classification prompt over 20 versions, and report that GPT-4o with Prompt 19 achieves an MCC of 0.501, 84.1% accuracy, and a binary F1 of 86.1%. They also provide per-category error analyses, confusion matrices, and practical lessons for building LLM-based moderation classifiers. The central claim is that the optimized prompt 'significantly outperforms' baseline approaches and constitutes the first multiclass classifier for sexism and misogyny in GitHub communications.

Significance. If the reported performance were an unbiased, out-of-sample estimate, the paper would offer a useful contribution to automated moderation of developer communities: it demonstrates that instruction-tuned LLMs can be steered toward fine-grained harm categories, it releases prompts and evaluation data, and its per-category confusion analysis gives actionable guidance for future classifiers. The paper also makes a methodological point that abstract category labels such as 'Victim Blaming' hinder LLM comprehension and that behaviorally descriptive labels and few-shot examples improve performance; this is a genuinely transferable insight. However, the headline MCC and F1 numbers are computed on the same evaluation set used to select the prompt through error-driven iteration, so the absolute performance figures and the 'significantly outperforming' wording are not supported by the reported experiments. The contribution is therefore a promising case study with useful engineering lessons, but the central empirical claim needs either a proper held-out evaluation or a substantial reframing.

major comments (6)
  1. [II-F, II-I, III-B] The reported MCC of 0.501 is an in-sample optimum, not an unbiased performance estimate. Section II-F describes 20 iterations of prompt refinement driven by error analysis on the evaluation data, and Section II-I and Table IV report scores for every prompt version on the same full 1,440-comment Phase 2 set. Selecting the best of 20 correlated in-sample scores and reporting it as the method's performance is selection overfitting; it measures fit to this particular label set, not expected performance on future GitHub comments. The claim of 'significantly outperforming' is also unsupported because each condition used a single inference run at temperature 0.1, which is not deterministic, and no variance or significance test is reported. Please provide a held-out validation set or a nested evaluation protocol, or explicitly reframe the reported numbers as in-sample optimization results rather than generalizable performance.
  2. [II-C, II-F, III-B, V] There is a direct contradiction about whether the gold labels were modified. Section II-F and Section III-B state that the category 'Victim Blaming' was renamed to 'Deflection' and then to 'Dismissing' during prompt development, and Table I lists 'Victim Blaming' as a category with 40 samples. Section V states 'We used the original labels without modification.' The manuscript never clarifies whether the 40 gold labels were remapped to 'Dismissing' before computing the confusion matrix and F1 scores. If they were not remapped, the reported F1 for Dismissing is computed against mismatched gold labels; if they were, the construct being evaluated changed, making the comparison to SGID's original labels invalid. This must be resolved for the evaluation to be interpretable.
  3. [II-C] The 146 newly collected samples, which constitute one-third of the 440 harmful comments, were shortlisted by one author and verified by a second author, with no inter-rater reliability statistic reported. These labels are treated as gold in all reported metrics, so label noise or a shift in category interpretation in these 146 samples directly affects the MCC and F1 values. Please report agreement for the verification step or at least discuss the potential impact of this label noise on the reported performance.
  4. [III-C, Abstract] The claim that the optimized approach 'significantly outperforms baseline approaches' is not supported by the evidence presented. The only direct comparison is to SGID4SE [15], and the authors themselves state in Section III-C that 'an apples-to-apples comparison is not feasible.' No statistical test is given, and the baseline was not evaluated under the same prompt-refinement protocol or on the same evaluation set. The word 'significantly' should be removed or replaced by a comparison on a common held-out test set with an appropriate significance test.
  5. [III-B, Table IV] The identity of the winning prompt is inconsistently reported. Section III-B states that 'culminating in Prompt 10, which achieved the highest MCC scores overall (MCC=0.501), with GPT-4o,' while Table IV and the abstract attribute the 0.501 MCC to Prompt 19. This ambiguity must be corrected, since the paper's central result depends on which prompt configuration is being claimed as optimal.
  6. [I, VI] The contribution list claims 'First multiclass classifier designed to categorize sexism and misogyny in GitHub communications,' but this is difficult to reconcile with the paper's own description of Sultana et al. [15], which the authors characterize as a manually annotated multiclass dataset with automated identification and a trained model (SGID4SE). The novelty claim needs to be qualified with an explicit comparison to [15] showing what is genuinely new, such as the use of instruction-tuned LLMs or the specific twelve-category scheme.
minor comments (5)
  1. [Figure 3] The axis label 'Mistrial-7B' should be 'Mistral-7B,' and the label 'GPT-40' should be 'GPT-4o.'
  2. [III-A] The name 'Sutana et al.' is a typo for 'Sultana et al.'
  3. [Figure 2] The caption lists prompts from 'Prompt: 01' through 'Prompt: 20,' but the text describes twenty iterations from Prompt 00 through Prompt 19; the numbering scheme should be made consistent.
  4. [IV, Lesson #1] The example 'the grammar on this is a bit funny maybe That every woman has the ability to code' appears to have a formatting or transcription error; the quotation and capitalization should be corrected so the example is readable.
  5. [II-G] The statement that a single inference run at temperature 0.1 is 'deterministic' is inaccurate; temperature 0.1 still uses stochastic sampling. Please either set temperature to 0 for determinism or describe how output variability was measured across runs.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline MCC 0.501 is an in-sample maximum over 20 prompt variants scored on the same 1,440 comments; no held-out evaluation supports the reported performance.

  1. fitted input called prediction [Section II-F (Prompt Refinement), Section II-I (Evaluation Design), Table IV]
    "The prompt underwent 20 iterations to improve classification performance through error analysis (Figure 2). ... In Phase 2, we used the full 1,440-comment dataset and the top three models. We tested prompt configurations in three stages: zero-shot (Prompt 00, category definitions only), one-shot (Prompt 01, one example with label and reasoning), and refined few-shot prompts (Prompts 02–19, with enhanced tone cues, overlap resolution, and example diversity; Table IV)."

    Prompt 19 was not evaluated on fresh data: every refinement shown in Table IV (definition expansions, overlap-resolution rules, few-shot examples) was motivated by errors observed on the same Phase 2 evaluation set, and Table IV lists MCC scores for all prompt versions on that same set. The reported MCC=0.501 is therefore the best of an in-sample search over 20 prompt variants, not an unbiased estimate of performance on unseen comments. Section II-F makes the mechanism explicit: a gold-labeled comment from the evaluation data was misclassified under Prompt 14, and Prompt 19 was edited to fix it (expanded Stereotyping, added a rule, added a similar example), with success then measured on the same set.

full rationale

The paper's central empirical claim is the in-sample MCC of a prompt selected by iterating on the same 1,440-comment test set. Section II-F describes 20 iterations of prompt refinement driven by error analysis; Section II-I and Table IV evaluate every prompt version on the full Phase 2 dataset. Consequently, the headline MCC=0.501 is a selection maximum over correlated in-sample evaluations, not an out-of-sample performance estimate. This is the only substantive circularity found. The dataset citation to Sultana et al. [15] is to an externally published corpus and is not used as a uniqueness theorem, so it is not load-bearing self-citation in the sense of forcing the result. The paper honestly reports its limitations (e.g., nuanced categories remain hard, and the authors note the comparison to SGID4SE is not apples-to-apples), but it does not provide the independent validation that would make the headline claim non-circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the reliability of the manually annotated labels and on the prompt being a fair representation of the task. The prompt itself, including 33 few-shot examples, is the main hand-fitted component.

free parameters (4)
  • Few-shot example count = 33 (final prompt)
    The number and selection of few-shot examples were chosen based on error analysis on the evaluation set, effectively fitting the prompt to the test data.
  • Temperature = 0.1
    Set after exploratory tests at 0.7 produced inconsistent outputs; not fitted to the target metric but a hand-chosen hyperparameter.
  • Top-p = 0.9
    Chosen to balance diversity and coherence; fixed across models.
  • Max tokens = 150
    Set to limit response length for consistency; not systematically tuned.
assumptions (4)
  • domain assumption The SGID dataset labels, including the 146 newly collected comments verified by two authors, are accurate and reliable ground truth.
    The evaluation treats these labels as gold; if they are noisy, the reported MCC is inflated. Invoked in Section II-C.
  • domain assumption The 12-category taxonomy from Sultana et al. is an appropriate and complete schema for sexist/misogynistic comments on GitHub.
    The framework maps every comment to one of 12 categories; if the schema is not exhaustive or boundaries are arbitrary, classification performance is less meaningful. Invoked in Section II-A/B.
  • domain assumption A single inference run at temperature 0.1 is representative enough to compare prompts.
    The authors assert deterministic outputs make one run sufficient (Section II-G); stochastic variation is not quantified.
  • domain assumption LLM outputs can be parsed reliably into the predefined categories and confidence scores.
    The evaluation relies on exact label matching; no manual verification of all outputs is reported. Invoked in Section II-E/F.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Binary Moderation: Identifying Fine-Grained Sexist and Misogynistic Behavior on GitHub with Large Language Models." pith.science (2026). https://pith.science/paper/V5Y35T2D

@misc{pith2026250720358,
  author       = {Pith},
  title        = {Pith review of: Beyond Binary Moderation: Identifying Fine-Grained Sexist and Misogynistic Behavior on GitHub with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5Y35T2D}},
  note         = {Machine review of arXiv:2507.20358}
}
read the original abstract

Background: Sexist and misogynistic behavior significantly hinders inclusion in technical communities like GitHub, causing developers, especially minorities, to leave due to subtle biases and microaggressions. Current moderation tools primarily rely on keyword filtering or binary classifiers, limiting their ability to detect nuanced harm effectively. Aims: This study introduces a fine-grained, multi-class classification framework that leverages instruction-tuned Large Language Models (LLMs) to identify twelve distinct categories of sexist and misogynistic comments on GitHub. Method: We utilized an instruction-tuned LLM-based framework with systematic prompt refinement across 20 iterations, evaluated on 1,440 labeled GitHub comments across twelve sexism/misogyny categories. Model performances were rigorously compared using precision, recall, F1-score, and the Matthews Correlation Coefficient (MCC). Results: Our optimized approach (GPT-4o with Prompt 19) achieved an MCC of 0.501, significantly outperforming baseline approaches. While this model had low false positives, it struggled to interpret nuanced, context-dependent sexism and misogyny reliably. Conclusion: Well-designed prompts with clear definitions and structured outputs significantly improve the accuracy and interpretability of sexism detection, enabling precise and practical moderation on developer platforms like GitHub.

Figures

Figures reproduced from arXiv: 2507.20358 by the authors.

Figure 1
Figure 1. Common Structure of our LLM prompts E. Prompt Design [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Iterative prompt-design structure highlighting the key refinements introduced at each version [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of F1-scores based on Zero-Shot, One-Shot, and Few-Shot prompts [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Confusion matrix for GPT-4o using Prompt 19 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 44 canonical work pages

  1. [15]

    Automated identification of sexual orientation and gender identity discriminatory texts from issue comments,

    S. Sultana, J. Sarker, F. Israt, R. Paul, and A. Bosu, “Automated identification of sexual orientation and gender identity discriminatory texts from issue comments,” ACM Transactions on Software Engineering Methodology (TOSEM), vol. TBD, 2025

    work page 2025

  2. [1]

    Octoverse: Ai leads python to top language as the number of global developers surges,

    GitHub, “Octoverse: Ai leads python to top language as the number of global developers surges,” 2024, accessed: April 2, 2025. [Online]. Available: https://github.blog/news-insights/octoverse/octoverse-2024/

    work page 2024

  3. [2]

    Gender differences and bias in open source: Pull request acceptance of women versus men,

    J. Terrell, A. Kofink, J. Middleton, C. Rainear, E. Murphy-Hill, C. Parnin, and J. Stallings, “Gender differences and bias in open source: Pull request acceptance of women versus men,”PeerJ Computer Science, vol. 3, p. e111, 2017

    work page 2017

  4. [3]

    Paradise unplugged: Identifying barriers for female participation on stack overflow,

    D. Ford, J. Smith, P. J. Guo, and C. Parnin, “Paradise unplugged: Identifying barriers for female participation on stack overflow,” in Proceedings of the 2016 24th ACM SIGSOFT International symposium on foundations of software engineering , 2016, pp. 846–857

    work page 2016

  5. [4]

    Codes of conduct in open source,

    H. Frluckaj and J. Howison, “Codes of conduct in open source,” in Equity, Diversity, and Inclusion in Software Engineering: Best Practices and Insights. Apress Berkeley, CA, 2024, pp. 295–308

    work page 2024

  6. [5]

    Code reviews in open source projects : How do gender biases affect participation and outcomes?

    S. Sultana, A. K. Turzo, and A. Bosu, “Code reviews in open source projects : How do gender biases affect participation and outcomes?” Empirical Software Engineering , vol. 28, no. 4, p. 92, 2023

    work page 2023

  7. [6]

    Discrimination, misogyny and harassment: Examples from oss: content analysis of women-focused online discus- sion forums,

    V . Singh and W. Brandon, “Discrimination, misogyny and harassment: Examples from oss: content analysis of women-focused online discus- sion forums,” in Proceedings of the Third Workshop on Gender Equality, Diversity, and Inclusion in Software Engineering , 2022, pp. 71–79

    work page 2022

  8. [7]

    Diversity and inclusion in open source software (oss) projects: Where do we stand?

    A. Bosu and K. Z. Sultana, “Diversity and inclusion in open source software (oss) projects: Where do we stand?” in 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Mea- surement (ESEM). IEEE, 2019, pp. 1–11

    work page 2019

Show all 47 references
  1. [8]

    Fixing the leaky pipeline for talented women in stem,

    A. Almukhambetova, D. H. Torrano, and A. Nam, “Fixing the leaky pipeline for talented women in stem,” International Journal of Science and Mathematics Education , pp. 1–20, 2021

    2021

  2. [9]

    Ashcraft, B

    C. Ashcraft, B. McLain, and E. Eger, Women in tech: The facts . National Center for Women & Technology (NCWIT) Colorado, CO, USA, 2016

    2016

  3. [10]

    Codes of conduct in open source software—for warm and fuzzy feelings or equality in community?

    V . Singh, B. Bongiovanni, and W. Brandon, “Codes of conduct in open source software—for warm and fuzzy feelings or equality in community?” Software Quality Journal , pp. 1–40, 2022

    2022

  4. [11]

    Code of conduct conversations in open source software projects on github,

    R. Li, P. Pandurangan, H. Frluckaj, and L. Dabbish, “Code of conduct conversations in open source software projects on github,” Proceedings of the ACM on Human-computer Interaction , vol. 5, no. CSCW1, pp. 1–31, 2021

    2021

  5. [12]

    Code of conduct in open source projects,

    P. Tourani, B. Adams, and A. Serebrenik, “Code of conduct in open source projects,” in 2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER). IEEE, 2017, pp. 24–33

    2017

  6. [13]

    Building and sustaining ethnically, racially, and gender diverse software engineering teams: A study at google,

    E. Dagan, A. Sarma, A. Chang, S. D’Angelo, J. Dicker, and E. Murphy- Hill, “Building and sustaining ethnically, racially, and gender diverse software engineering teams: A study at google,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Sympos...

    2023

  7. [14]

    Implicit bias and negative stereotyping in global software development and why it is time to move on!

    S. Matthiesen, P. Bjørn, and C. Trillingsgaard, “Implicit bias and negative stereotyping in global software development and why it is time to move on!” Journal of Software: Evolution and Process, vol. 35, no. 5, p. e2435, 2023

    2023

  8. [16]

    Exploring chatgpt for identifying sexism in the communication of software developers,

    S. Sultana and M. Begum Kali, “Exploring chatgpt for identifying sexism in the communication of software developers,” in Proceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments, 2024, pp. 400–403

    2024

  9. [17]

    Automated identification of toxic code reviews using toxicr,

    J. Sarker, A. K. Turzo, M. Dong, and A. Bosu, “Automated identification of toxic code reviews using toxicr,” ACM Transactions on Software Engineering and Methodology , vol. 32, Feb 2023

    2023

  10. [18]

    Incivility detection in open source code review and issue discussions,

    I. Ferreira, A. Rafiq, and J. Cheng, “Incivility detection in open source code review and issue discussions,” Journal of Systems and Software , vol. 209, p. 111935, 2024

    2024

  11. [19]

    A meta-analysis of procedures to change implicit measures

    P. S. Forscher, C. K. Lai, J. R. Axt, C. R. Ebersole, M. Herman, P. G. Devine, and B. A. Nosek, “A meta-analysis of procedures to change implicit measures.” Journal of personality and social psychology , vol. 117, no. 3, p. 522, 2019

    2019

  12. [20]

    Reducing implicit prejudice,

    C. K. Lai, K. M. Hoffman, and B. A. Nosek, “Reducing implicit prejudice,” Social and Personality Psychology Compass , vol. 7, no. 5, pp. 315–330, 2013

    2013

  13. [21]

    Toxispanse: An explainable toxicity detection in code review comments,

    J. Sarker, S. Sultana, S. R. Wilson, and A. Bosu, “Toxispanse: An explainable toxicity detection in code review comments,” in 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . IEEE, 2023, pp. 1–12

    2023

  14. [22]

    A rubric to identify misogynistic and sexist texts from software developer communications,

    S. Sultana, J. Sarker, and A. Bosu, “A rubric to identify misogynistic and sexist texts from software developer communications,” in Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , 2021, pp. 1–6

    2021

  15. [23]

    Challenges and frontiers in abusive content detection,

    B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. Hale, and H. Margetts, “Challenges and frontiers in abusive content detection,” in Proceedings of the third workshop on abusive language online . Association for Computational Linguistics, 2019

    2019

  16. [24]

    Advancing single-and multi-task text classification through large language model fine-tuning,

    H. Zhao, Q. P. Chen, Y . B. Zhang, and G. Yang, “Advancing single-and multi-task text classification through large language model fine-tuning,” arXiv preprint arXiv:2412.08587 , 2024

    2024 arXiv

  17. [25]

    Large language models for text classification: Case study and comprehensive review,

    A. Kostina, M. D. Dikaiakos, D. Stefanidis, and G. Pallis, “Large language models for text classification: Case study and comprehensive review,” arXiv preprint arXiv:2501.08457 , 2025

    2025 arXiv

  18. [26]

    Design science research,

    A. Dresch, D. P. Lacerda, and J. A. V . Antunes Jr, “Design science research,” in Design science research: A method for science and tech- nology advancement. Springer, 2014, pp. 67–102

    2014

  19. [27]

    Automatic identification and classification of misogynistic language on twitter,

    M. Anzovino, E. Fersini, and P. Rosso, “Automatic identification and classification of misogynistic language on twitter,” in Natural Language Processing and Information Systems. Springer International Publishing, 2018, pp. 57–64

    2018

  20. [28]

    Overview of the task on automatic misogyny identification at ibereval 2018

    E. Fersini, P. Rosso, and M. Anzovino, “Overview of the task on automatic misogyny identification at ibereval 2018.” Ibereval@ sepln, vol. 2150, pp. 214–228, 2018

    2018

  21. [29]

    Ami@ evalita2020: Automatic misogyny identification

    E. Fersini, D. Nozza, and P. Rosso, “Ami@ evalita2020: Automatic misogyny identification.” in EVALITA, 2020

    2020

  22. [30]

    Unibo@ ami: A multi-class approach to misogyny and aggressiveness identification on twitter posts using alberto,

    A. Muti and A. Barron-Cede, “Unibo@ ami: A multi-class approach to misogyny and aggressiveness identification on twitter posts using alberto,” EVALITA Evaluation of NLP and Speech Tools for Italian- December 17th, 2020 , p. 29, 2020

    2020

  23. [31]

    Developing a multilingual annotated corpus of misogyny and aggression,

    S. Bhattacharya, S. Singh, R. Kumar, A. Bansal, A. Bhagat, Y . Dawer, B. Lahiri, and A. Ojha, “Developing a multilingual annotated corpus of misogyny and aggression,” 03 2020

    2020

  24. [32]

    Exploring misogyny across the manosphere in reddit,

    T. Farrell, M. Fernandez, J. Novotny, and H. Alani, “Exploring misogyny across the manosphere in reddit,” ser. WebSci ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 87–96. [Online]. Available: https://doi.org/10.1145/3292522.3326045

    2019

  25. [33]

    When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data,

    A. Jha and R. Mamidi, “When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data,” in Proceedings of the Second Workshop on NLP and Computational Social Science . Vancouver, Canada: Association for Computational Linguistics, Aug....

    2017

  26. [34]

    An expert annotated dataset for the detection of online misogyny,

    E. Guest, B. Vidgen, A. Mittos, N. Sastry, G. Tyson, and H. Margetts, “An expert annotated dataset for the detection of online misogyny,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , 2021, pp. 1336–1350

    2021

  27. [35]

    Overview of exist 2024—learning with disagreement for sexism identification and characterization in tweets and memes,

    L. Plaza, J. Carrillo-de Albornoz, V . Ruiz, A. Maeso, B. Chulvi, P. Rosso, E. Amig ´o, J. Gonzalo, R. Morante, and D. Spina, “Overview of exist 2024—learning with disagreement for sexism identification and characterization in tweets and memes,” in International Conference of ...

    2024

  28. [36]

    Detecting sexist meme on the web: A study on textual and visual cues,

    E. Fersini, F. Gasparini, and S. Corchs, “Detecting sexist meme on the web: A study on textual and visual cues,” in 2019 8th International Conference on Affective Computing and Intelligent Interaction Work- shops and Demos (ACIIW) . IEEE, 2019, pp. 226–231

    2019

  29. [37]

    A contrastive learning based approach to detect sexism in memes,

    F. Maqbool and E. Fersini, “A contrastive learning based approach to detect sexism in memes,” Working Notes of CLEF , 2024

    2024

  30. [38]

    Concatenated trans- former models based on levels of agreements for sexism detection,

    V . Ruiz, J. Carrillo-de Albornoz, and L. Plaza, “Concatenated trans- former models based on levels of agreements for sexism detection,” Working Notes of CLEF , 2024

    2024

  31. [39]

    Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos,

    N. AlDahoul, M. J. T. Tan, H. R. Kasireddy, and Y . Zaki, “Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos,” arXiv preprint arXiv:2411.17123, 2024

    2024 arXiv

  32. [40]

    Watch your lan- guage: Investigating content moderation with large language models,

    D. Kumar, Y . A. AbuHashem, and Z. Durumeric, “Watch your lan- guage: Investigating content moderation with large language models,” in Proceedings of the International AAAI Conference on Web and Social Media, vol. 18, 2024, pp. 865–878

    2024

  33. [41]

    Towards safer social media platforms: Scalable and performant few- shot harmful content moderation using large language models,

    A. Bonagiri, L. Li, R. Oak, Z. Babar, M. Wojcieszak, and A. Chhabra, “Towards safer social media platforms: Scalable and performant few- shot harmful content moderation using large language models,” arXiv preprint arXiv:2501.13976, 2025

    2025 arXiv

  34. [42]

    Re-ranking using large language models for mitigating exposure to harmful content on social media platforms,

    R. Oak, M. Haroon, C. Jo, M. Wojcieszak, and A. Chhabra, “Re-ranking using large language models for mitigating exposure to harmful content on social media platforms,” arXiv preprint arXiv:2501.13977 , 2025

    2025 arXiv

  35. [43]

    Stress and burnout in open source: Toward finding, understanding, and miti- gating unhealthy interactions,

    N. Raman, M. Cao, Y . Tsvetkov, C. K ¨astner, and B. Vasilescu, “Stress and burnout in open source: Toward finding, understanding, and miti- gating unhealthy interactions,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emer...

    2020

  36. [44]

    ” did you miss my comment or what?

    C. Miller, S. Cohen, D. Klug, B. Vasilescu, and C. KaUstner, “” did you miss my comment or what?” understanding toxicity in open source discussions,” in Proceedings of the 44th international conference on software engineering, 2022, pp. 710–722

    2022

  37. [45]

    Motivated and capable but no space for error,

    V . Singh and B. Bongiovanni, “Motivated and capable but no space for error,” The International journal of information, diversity, & inclusion , vol. 5, no. 3, pp. 98–126, 2021

    2021

  38. [46]

    Floss as a source for profanity and insults: Collecting the data,

    M. Squire and R. Gazda, “Floss as a source for profanity and insults: Collecting the data,” in 2015 48th Hawaii International Conference on System Sciences. IEEE, 2015, pp. 5290–5298

    2015

  39. [47]

    Exploring chatgpt for toxicity detection in github,

    S. Mishra and P. Chatterjee, “Exploring chatgpt for toxicity detection in github,” in Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) , 2024, pp. 6–10. [Online]. Available: https://doi.org/10.11...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.