Pith. sign in

REVIEW 3 major objections 6 minor 77 references

A single small-LLM call plus a trained error-spotter makes GitHub toxicity labeling at scale practical, producing a 124,757-conversation dataset.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:02 UTC pith:MG2HXJET

load-bearing objection A genuinely useful HITL annotation pipeline and a large toxicity dataset, but the headline statistics ignore the wide uncertainty from a small audit sample. the 3 major comments →

arxiv 2607.25946 v2 pith:MG2HXJET submitted 2026-07-28 cs.SE

A Low-Cost Human-in-the-Loop Investigation of Toxicity on GitHub at Scale

Rahat Rizvi Rahman , Mia Mohammad Imran , Kostadin Damevski This is my paper
classification cs.SE
keywords toxicityGitHubhuman-in-the-loop annotationLLM-based labelingopen source softwareempirical software engineeringconversation analysisdataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that large-scale, context-calibrated toxicity labeling of GitHub discussions can be made practical with one call per conversation to a small local language model, followed by a lightweight validator that flags the sliver of conversations most likely to be mislabeled for human review. Applied to 124,757 issue and pull request conversations, the pipeline needed human review of only about 1.5% of the corpus and yielded a dataset with 946 toxic conversations (0.76%). Using that dataset, the paper revisits prior small-sample findings: the link between conversation length and toxicity is much weaker than reported, accepted pull requests are about five times less toxic than non-accepted ones, and toxicity concentrates in weakly governed repositories. If correct, the method and dataset give software-engineering and social-computing researchers a reusable, low-cost annotation recipe and a reference snapshot of toxicity in modern open source development.

Core claim

On the paper's own terms, the discovery is that the judge of annotation difficulty need not be a second model or a confidence score: the same LLM's structured 'event category scores' — eight interpretable dimensions such as escalation, subtle criticism, and process hostility — carry enough signal for a small Random Forest to predict when the LLM's toxicity label is wrong. Trained on 898 prior human-labeled conversations, the validator flags only the most confusion-prone conversations; in deployment it routed 1,874 of 124,757 conversations (1.5%) to human review, where 169 labels were corrected, almost all (155) from non-toxic to toxic. A manual audit of 5,000 unflagged conversations found 11

What carries the argument

The central mechanism is the two-stage annotation-verification loop. Stage one: a single call to a small, locally run language model outputs both a binary toxicity label and scores on eight named event categories (escalation, subtle criticism, process hostility, dismissiveness, person hostility, hostility density, resolution language, behavior callout). Stage two: a Random Forest validator trained only on those eight scores learns which conversations are hard for the LLM — primarily those high in escalation and subtle criticism — and flags them for human review. The validator does not relabel; it concentrates human effort on likely errors, achieving better error capture than confidence thres

Load-bearing premise

The whole pipeline rests on the assumption that the validator's ability to detect LLM labeling errors — learned on 898 conversations collected before 2024 — transfers to the 2024 corpus, where the flag rate itself changes by roughly six-fold and the key toxic-stratum audit is based on only 7 observed false positives.

What would settle it

A direct check would be to have human annotators review a random sample (say 5,000 to 10,000) of the conversations the validator did not flag in the 2024 corpus, compute the false-negative rate among predicted-non-toxic conversations, and compare it to the 0.083% observed in the paper's own 5,000-conversation audit. If the rate is materially higher, for instance above 0.5%, the estimated 129 residual errors and the 0.76% prevalence figure would need substantial upward revision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the pipeline's error detection generalizes, any future toxicity study on GitHub can annotate hundreds of thousands of conversations with roughly 1.5% manual review, making large-scale, GitHub-calibrated datasets the default rather than the exception.
  • The 124,757-conversation dataset becomes a reference point: toxicity prevalence of 0.76% slightly exceeds the 0.6% reported by prior longitudinal work, and monthly rates drift upward in late 2024.
  • Accepted pull requests are about five times less toxic (0.23% vs 1.19%), and toxic PRs are accepted at a quarter of the rate of non-toxic ones, reinforcing that toxicity and rejection co-occur.
  • The conversation-length effect is real but modest (median 4 vs 3 comments), correcting the much larger gap in small-sample studies.
  • Weakly governed repositories — fewer administrators, lower health scores, individual ownership — concentrate toxicity, while highly active repositories show little high-toxicity presence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the validator's training set predates the 2024 corpus and the flag rate dropped from 8.9% to 1.5%, the pipeline's transfer depends on the 2024 corpus being similar in difficulty; a straightforward extension would be to retrain or recalibrate the validator on a small sample of the target corpus and measure how flag rate and residual error change.
  • The paper's extrapolated ~129 residual errors are not propagated into the 0.76% prevalence estimate; a re-analysis that corrects prevalence by the audit's false-negative and false-positive rates, with confidence bounds, would give a more honest uncertainty band around the headline rate.
  • The finding that administrators both author toxic comments and moderate in a toxic tone suggests moderation quality, not just presence, is the target for interventions; a testable extension is whether repositories with explicit, enforced codes of conduct show lower toxic-administrator behavior.
  • The event-category scores are proposed as a general 'annotation difficulty' feature set; they could be tested on other subjective labeling tasks, such as sentiment or harassment detection, where a single small LLM does the first pass.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a human-in-the-loop (HITL) annotation pipeline for GitHub toxicity. A single Ministral LLM call per conversation produces a binary toxicity prediction plus eight event-category scores; a Random Forest validator, trained on Imran et al.'s 898 annotated conversations, flags the conversations most likely to be mislabeled; human annotators review only the flagged subset. The validator is evaluated with stratified 5-fold cross-validation and on Raman et al.'s 314-conversation external dataset. The pipeline is then applied to 124,757 issue/PR conversations from 340 repositories collected during 2024–2025. After reviewing 1,874 flagged conversations (about 1.5%) and auditing 5,000 unflagged conversations, the authors report 946 toxic conversations. Empirically, the paper revisits prior findings: prevalence of 0.76%, non-monotonic association with contributor participation, higher toxicity in deeper mention chains, a weaker conversation-length association than Imran et al., and lower toxicity in accepted PRs. It also reports new observations on programming-language ecosystems, governance, and highly active users. Section 6 quantifies effect sizes and the impact of manual review.

Significance. If the labels are trustworthy, the dataset would be one of the largest GitHub toxicity corpora to date, and the pipeline would be a practical, low-cost annotation protocol for rare, context-dependent phenomena. The methodological core is solid in several respects: the validator comparison in §2.2 uses 5-fold CV and reasonable baselines; the paper reports inter-annotator agreement; an external dataset check is included; and the replication package is promised via Zenodo. The effect-size analysis in §6 is a useful addition and shows awareness that manual correction matters. However, the empirical conclusions rest on a small audit sample whose uncertainty is not propagated into the prevalence and effect-size estimates. The specific concern that residual false negatives may be length-biased and could alter Observation #4 is real and should be addressed with a sensitivity analysis. With that addition, the paper would be a strong reference for OSS toxicity research.

major comments (3)
  1. [§3.3, §4, §6 (Tables 6 and 8)] The audit that underpins the residual-error claim is too small to support the point estimates used downstream. In §3.3, the manual audit found 4 false negatives in 4,800 predicted-non-toxic conversations and 7 false positives in 200 predicted-toxic conversations; the paper itself reports 95% CIs of 0.023%–0.213% and 1.42%–7.07%. These intervals translate to roughly 40–320 residual errors, not a single point of 129, and a true toxic count that could be materially above the reported 946. Section 4.3 reports prevalence as 0.76% with no interval, and Table 8 treats the corrected labels as ground truth. This is not a hypothetical concern: §6 shows that manual correction changed the conversation-length Cliff's delta from 0.077 to 0.195, i.e., the corrections were length-biased. If the residual false negatives are similarly length-biased, Observation #4 ('much weaker than Imran et al.') could b
  2. [§2.2, §3.2] The validator threshold τ=0.5 is selected on Imran et al.'s 898 conversations, where it flags 8.9% of instances, but on the 2024 corpus it flags only 1.5% of conversations. This six-fold difference indicates a distribution shift, and the supporting transfer evidence is thin: the external Raman et al. check is only 314 instances, and the audit's toxic stratum is based on n=7. To support the central claim that the validator 'directs human review only where it is needed' on the new corpus, the paper should report the validator's estimated operational precision and recall for error detection on the 2024 data, with CIs, using the flagged set and the audit sample. Otherwise the reader cannot tell whether applying the training-set threshold unchanged is justified.
  3. [§5.2 (Figure 2, Observation #7)] The governance-quadrant analysis is not reproducible as written. The thresholds on 'new issue volume' and 'proportion of toxic issues' are described as fixed by manual inspection, but the actual values are never reported. The claim that no repository falls in the High Volume–High Toxicity quadrant—and the conclusion that active repositories avoid sustained toxicity—is sensitive to where those thresholds are drawn. Please state the exact thresholds, report the number of repositories in each quadrant, and include a robustness check across neighboring threshold values. Otherwise Observation #7 cannot be verified from the replication package.
minor comments (6)
  1. [§5.2] The proxy for administrators uses Owners, Members, and Collaborators, but 'Member' in GitHub organization terms does not necessarily imply write or admin access. A sentence justifying why this proxy is appropriate would strengthen the governance analysis.
  2. [Table 3] Only the best threshold per baseline is reported. For fair comparison, especially for the confidence-threshold baseline, include the full threshold sweep or precision-recall curves; the current presentation makes the margin between approaches hard to assess.
  3. [Figure 1] The monthly prevalence plot has no uncertainty bands. Given that the underlying labels themselves are estimates (see Major Comment 1), the reader cannot tell whether the late-2024 peaks are meaningful or noise. Adding CIs or error bars would help.
  4. [§3.3] The comparison with ToxiShield mixes units: ToxiShield is described as a comment-level detector, while the proposed pipeline is evaluated at conversation level. Please clarify the unit of analysis for the baseline and report its performance on the same conversation-derived inputs.
  5. [§2.1.1] The prompt's toxicity definition is paraphrased in the paper. Since prompt detail is a core part of the methodology and reproducibility, include the full prompt verbatim in an appendix or in the replication package.
  6. [§4.3] The 2024 GitHub survey statistic (64.23% of developers experienced or witnessed negative interactions) measures a different construct from conversation-level prevalence; the paper labels it 'suggestive,' but even that framing may overstate the connection. A one-sentence clarification would be useful.

Circularity Check

0 steps flagged

No significant circularity: the validator is trained on a temporally disjoint prior dataset, checked against an external dataset and a 5,000-conversation audit, and the empirical findings are not entailed by the pipeline's construction.

full rationale

The paper's derivation chain is: (1) a single LLM call produces toxicity predictions and eight event-category scores; (2) a Random Forest validator trained on Imran et al.'s 898 manually annotated conversations (pre-2024) flags 1,874 of 124,757 new 2024 conversations for human review; (3) human annotators correct 169 labels; (4) a stratified audit of 5,000 unflagged conversations estimates a residual error rate of 0.105%; (5) downstream observations are computed from the corrected labels. No step reduces to its own input. The validator is not fitted to the 2024 corpus and does not relabel anything; it only selects conversations for review, and the final labels are the product of human judgment. The external check on Raman et al.'s independently sampled dataset and the manual audit of the new corpus provide genuine out-of-sample evidence. The toxicity definition is inherited from prior literature, but confirming or qualifying prior empirical findings on a new, larger corpus is not circular: the findings are not entailed by the definition. The use of Imran et al.'s dataset (which shares authors) to train the validator is a self-citation, but it is not load-bearing because the central empirical claims rest on the new 2024 corpus and are supported by external validation and audit. The small sample sizes in the audit (e.g., n=7 for the false-positive stratum) raise uncertainty about residual error and could bias prevalence or the length-toxicity association, but this is a validity concern, not a circularity. There is no equation or construction in which a predicted quantity is identical to a fitted input, and no uniqueness theorem or ansatz is imported solely from the authors' prior work.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The pipeline rests on the transfer of the validator trained on same-group 898 pre-2024 conversations to the 2024 corpus, on the representativeness of the 5,000-thread audit, on the validity of the prompt-based toxicity definition inherited from the very studies being tested, and on the completeness of the eight LLM-generated event-category features as error predictors. No new entities are postulated. The hand-set thresholds in §5.2 and the validator threshold τ=0.5 are the main fitted quantities beyond standard ML hyperparameters.

free parameters (3)
  • Validator threshold τ = 0.5
    Selected from 5-fold CV on the 898-conversation Imran et al. dataset as the best balance (Macro F1 0.83, Incorrect F1 0.68). Applied to the 2024 corpus it flags only 1.5% of conversations vs 8.9% in CV, indicating distribution shift.
  • Random Forest hyperparameters = Not reported
    RF with SMOTE and imbalanced-learn/scikit-learn defaults implied, but no grid search or parameter values are given; these choices affect validator performance.
  • Governance quadrant thresholds (issue volume and toxicity ratio) = Manual cut-offs
    §5.2 sets fixed thresholds by manually inspecting the empirical distribution, assigning repositories to LV-LT/HV-LT/LV-HT; this hand-chosen partition shapes Observation #7.
axioms (6)
  • domain assumption The 898 Imran et al. conversations are a valid training signal for predicting LLM mislabels in future GitHub conversations.
    §2.2.2 trains the validator entirely on this same-group dataset; transfer to the 2024 corpus is checked only via a 314-instance external set and a 5,000-instance audit.
  • domain assumption Two expert annotators provide an unbiased gold standard for toxicity.
    §3.2–3.3: reviewers were shown the Ministral prediction and explanation before labeling, so label independence from the LLM is not established; κ=0.78 indicates substantial but not strong agreement.
  • domain assumption The eight LLM-generated event-category scores are sufficient features for detecting the LLM's own mislabels.
    §2.2.1: the validator uses only these eight scores; error signals living in other features (length, author roles, text) are invisible to it.
  • standard math The 5,000-conversation stratified audit is representative of the 122,883 unflagged conversations.
    §3.3: extrapolation uses 4 errors in 4,800 predicted non-toxic and 7 in 200 predicted toxic; the toxic-stratum CI (1.42%–7.07%) is wide, so the 129-error estimate is uncertain.
  • domain assumption The prompt-based toxicity definition, inherited from Miller et al., Ferreira et al., Imran et al., and Raman et al., is the correct construct.
    §2.1.1: all prevalence and observation results are relative to this definition, which is also the definition used by the prior studies being tested.
  • domain assumption Repositories with ≥1,000 stars, ≥100 issues, recent activity, and a locked conversation or code of conduct represent the GitHub population of interest.
    §3.1: prevalence and governance results are conditional on this population; acknowledged as selection bias in §8.

pith-pipeline@v1.3.0-alltime-deepseek · 21693 in / 22787 out tokens · 185392 ms · 2026-08-01T01:02:16.083512+00:00 · methodology

0 comments
read the original abstract

Toxic interactions in open source discussions can alienate contributors and threaten project sustainability, yet prior empirical studies of GitHub toxicity have been limited in scale, raising questions about their generalizability. Scaling up is difficult because toxicity on GitHub is often implicit and context-dependent, making both fully manual annotation and LLM-based labeling unreliable. We present a human-in-the-loop (HITL) annotation methodology that makes large-scale, domain-calibrated toxicity labeling practical. A single call to a small, local LLM produces both a toxicity prediction and a set of interpretable event category scores. A lightweight Random Forest validator then uses those scores to flag the small subset of conversations most likely to be mislabeled, directing human review only where it is needed. The validator outperforms confidence-based and multi-LLM baselines while adding low annotation cost. We apply this pipeline to over 124,000 GitHub issue and pull request conversations. Using the resulting dataset, we evaluate key findings from prior small-scale research, confirming some and qualifying others, and present new insights into the prevalence, characteristics, and dynamics of toxicity across diverse open source projects.

Figures

Figures reproduced from arXiv: 2607.25946 by Kostadin Damevski, Mia Mohammad Imran, Rahat Rizvi Rahman.

Figure 1
Figure 1. Figure 1: Monthly percentage of toxic conversations in [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Repository groups by issue activity and proportion [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 3 canonical work pages

  1. [1]

    Nuri Almarimi, Ali Ouni, Moataz Chouchen, Islem Saidani, and Mohamed Wiem Mkaouer. 2020. On the detection of community smells using genetic programming-based ensemble classifier chain. InProceedings of the 15th Interna- tional Conference on Global Software Engineering. 43–54

    2020

  2. [2]

    Md Awsaf Alam Anindya, Showvik Biswas, Anindya Iqbal, Jaydeb Sarker, and Amiangshu Bosu. 2026. ToxiShield: Promoting Inclusive Developer Communica- tion through Real-Time Toxicity Filtering.Proceedings of the ACM on Software Engineering3, FSE (2026), 2767–2789

    2026

  3. [3]

    Giusy Annunziata, Carmine Ferrara, Stefano Lambiase, Fabio Palomba, Gemma Catolino, Filomena Ferrucci, and Andrea De Lucia. 2024. An Empirical Study on the Relation Between Programming Languages and the Emergence of Community Smells. In2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 268–275

    2024

  4. [4]

    Atijit Anuchitanukul, Julia Ive, and Lucia Specia. 2022. Revisiting Contextual Toxicity Detection in Conversations.Journal of Data and Information Quality15, 1, Article 6 (Dec. 2022), 22 pages. doi:10.1145/3561390

    work page doi:10.1145/3561390 2022

  5. [5]

    Ekaterina Artemova, Akim Tsvigun, Dominik Schlechtweg, Natalia Fedorova, Konstantin Chernyshev, Sergei Tilga, and Boris Obmoroshev. 2025. Hands- On Tutorial: Labeling with LLM and Human-in-the-Loop.31st International Conference on Computational Linguistics (COLING)(2025)

    2025

  6. [6]

    Passonneau, and Ninareh Mehrabi

    Berk Atil, Rebecca J. Passonneau, and Ninareh Mehrabi. 2026. Robust Persona- Aware Toxicity Detection with Prompt Optimization and Learned Ensembling. arXiv:2601.02337 [cs.CL] https://arxiv.org/abs/2601.02337

    arXiv 2026

  7. [7]

    Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khat- tar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Alek- sander Ficek, et al. 2025. NVIDIA Nemotron 3: Efficient and Open Intelligence. arXiv preprint arXiv:2512.20856(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Kevin Daniel André Carillo, Josianne Marsan, and Bogdan Negoita. 2016. Towards developing a theory of toxicity in the context of free/open source software & peer production communities.SIGOPEN 2016(2016)

    2016

  9. [9]

    CMUSTRUDEL. 2021. Raman et al. Dataset. https://github.com/CMUSTRUDE L/toxicity-detector/blob/master/data/training/train_comments.csv Accessed: 2025-07-15

    2021

  10. [10]

    GitHub Docs. [n. d.]. CommentAuthorAssociation Enum. https://docs.github.co m/en/graphql/reference/enums#commentauthorassociation

  11. [11]

    Coraline Ada Ehmke. 2017. Contributor Covenant: A Code of Conduct for Open Source Projects. https://www.contributor-covenant.org/version/1/4/code-of- conduct/. Accessed: 2025-07-19

    2017

  12. [12]

    Ramtin Ehsani, Preetha Chatterjee, et al. 2024. Analyzing Toxicity in Open Source Software Communications Using Psycholinguistics and Moral Foundations The- ory.NLBSE(2024)

    2024

  13. [13]

    Ramtin Ehsani, Mia Mohammad Imran, Robert Zita, Kostadin Damevski, and Preetha Chatterjee. 2024. Incivility in Open Source Projects: A Comprehensive Annotated Dataset of Locked GitHub Issue Threads. In2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR). IEEE, 515–519

    2024

  14. [14]

    Isabella Ferreira, Bram Adams, and Jinghui Cheng. 2022. How heated is it? understanding GitHub locked issues. InProceedings of the 19th International Conference on Mining Software Repositories (MSR ’22). Association for Computing Machinery, New York, NY, USA. doi:10.1145/3524842.3527957

    arXiv 2022

  15. [15]

    Isabella Ferreira, Bram Adams, and Jinghui Cheng. 2022. How Heated is it? Understanding GitHub Locked Issues. In19th International Conference on Mining Software Repositories

    2022

  16. [16]

    shut the f** k up

    Isabella Ferreira, Jinghui Cheng, and Bram Adams. 2021. The" shut the f** k up" phenomenon: Characterizing incivility in open source code review discussions. Proceedings of the ACM on Human-Computer InteractionCSCW2 (2021)

    2021

  17. [17]

    Isabella Ferreira, Ahlaam Rafiq, and Jinghui Cheng. 2024. Incivility detection in open source code review and issue discussions.Journal of Systems and Software (2024)

    2024

  18. [18]

    Anna Filippova and Hichang Cho. 2015. Mudslinging and manners: Unpacking conflict in free and open source software. InProceedings of the 18th ACM Confer- ence on Computer Supported Cooperative Work & Social Computing. 1393–1403

    2015

  19. [19]

    Anna Filippova and Hichang Cho. 2016. The effects and antecedents of conflict in free and open source software development. InProceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

    2016

  20. [20]

    O’Reilly Media, Inc

    Karl Fogel. 2005.Producing open source software: How to run a successful free software project. " O’Reilly Media, Inc. "

    2005

  21. [21]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences of the United States of America120, 30 (2023), e2305016120

    2023

  22. [22]

    GitHub. 2025. GitHub GraphQL API. https://docs.github.com/en/graphql Accessed: 2025-02-09

    2025

  23. [23]

    GitHub, Inc., Kenyatta Forbes, Kevin Xu, Jeffrey Luszcz, Margaret Tucker, Eva Maxfield Brown, Peter Cihon, Mike Linksvayer, Ashley Wolf, Lukas Spieß, Kevin Crosby, and Jason Meridth. 2024. GitHub Open Source Survey 2024. doi:10.5281/zenodo.13989018

    work page doi:10.5281/zenodo.13989018 2024

  24. [24]

    Sanuri Dananja Gunawardena, Peter Devine, Isabelle Beaumont, Lola Piper Gar- den, Emerson Murphy-Hill, and Kelly Blincoe. 2022. Destructive criticism in software code review impacts inclusion.Proceedings of the ACM on Human- Computer Interaction6, CSCW2 (2022), 1–29

    2022

  25. [25]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum...

    2024

  26. [26]

    Andreas Holzinger. 2016. Interactive machine learning for health informatics: when do we need the human-in-the-loop?Brain informatics3, 2 (2016), 119–131

    2016

  27. [27]

    Nip it in the Bud

    Jane Hsieh, Joselyn Kim, Laura Dabbish, and Haiyi Zhu. 2023. " Nip it in the Bud": Moderation Strategies in Open Source Software Projects and the Role of Bots. Proceedings of the ACM on Human-Computer Interaction7, CSCW2 (2023), 1–29

    2023

  28. [28]

    Mia Mohammad Imran, Preetha Chatterjee, and Kostadin Damevski. 2024. Un- covering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Ma- chinery, New York, NY, USA, Article 182, 13 pages. ...

    arXiv 2024

  29. [29]

    Mia Mohammad Imran, Yashasvi Jain, Preetha Chatterjee, and Kostadin Damevski

  30. [30]

    Mia Mohammad Imran, Robert Zita, Rahat Rizvi Rahman, Preetha Chatterjee, and Kostadin Damevski. 2026. Toxicity Ahead: Forecasting Conversational De- railment on GitHub. InProceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE ’26). Association for Computing Machinery, New York, NY, USA

    2026

  31. [31]

    Jack Jamieson, Naomi Yamashita, and Eureka Foong. 2024. Predicting open source contributor turnover from value-related discussions: An analysis of GitHub issues. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering

    2024

  32. [32]

    German, and Daniela Damian

    Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2014. The promises and perils of mining GitHub. InProceedings of the 11th Working Conference on Mining Software Repositories (Hyderabad, India)(MSR 2014). Association for Computing Machinery, New York, NY, USA, 92–101. doi:10.1145/2597073.2597074

    arXiv 2014

  33. [33]

    Marzena Karpinska and Mohit Iyyer. 2023. Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist. InProceedings of the Eighth Conference on Machine Translation. 419–451

    2023

  34. [34]

    Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang

  35. [35]

    Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha

  36. [36]

    Renee Li, Pavitthra Pandurangan, Hana Frluckaj, and Laura Dabbish. 2021. Code of conduct conversations in open source software projects on github.Proceedings of the ACM on Human-computer Interaction5, CSCW1 (2021), 1–31

    2021

  37. [37]

    Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. 2026. Ministral 3.arXiv preprint arXiv:2601.08584 (2026)

    Pith/arXiv arXiv 2026

  38. [38]

    In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA)(CHI EA ’24)

    LLM-Mod: Can Large Language Models Assist Content Moderation?. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA)(CHI EA ’24). Association for Computing Machinery, New York, NY, USA, Article 217, 8 pages. doi:10.1145/3613905.3650828

    arXiv

  39. [39]

    Did you miss my comment or what?

    Courtney Miller, Sophie Cohen, Daniel Klug, Bogdan Vasilescu, and Christian KaUstner. 2022. "Did you miss my comment or what?": understanding toxicity in open source discussions. InProceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 710–722. doi:10.1145/3510003.3510111

    arXiv 2022

  40. [40]

    Shyamal Mishra and Preetha Chatterjee. 2024. Exploring ChatGPT for Tox- icity Detection in GitHub. InProceedings of the 2024 ACM/IEEE 44th Interna- tional Conference on Software Engineering: New Ideas and Emerging Results (ICSE- NIER’24). Association for Computing Machinery, New York, NY, USA, 6–10. doi:10.1145/3639476.3639777

    arXiv 2024

  41. [41]

    Zhiwei Liu, Kailai Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou

  42. [42]

    InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Barcelona, Spain) (KDD ’24)

    EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Barcelona, Spain) (KDD ’24). Association for Computing Machinery, New York, NY, USA, 5487–5496. doi:10.1145/3637528.3671552

    arXiv

  43. [43]

    Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2014. Towards discovering the role of emotions in stack overflow. InProceedings of the 6th international workshop on social software engineering

    2014

  44. [44]

    Siobhán O’mahony and Fabrizio Ferraro. 2007. The emergence of governance in an open source community.Academy of Management Journal50, 5 (2007), 1079–1106

    2007

  45. [45]

    Audris Mockus, Roy T Fielding, and James D Herbsleb. 2002. Two case studies of open source software development: Apache and Mozilla.ACM Transactions on A Low-Cost Human-in-the-Loop Investigation of Toxicity on GitHub at Scale ASE ’26, October 12–16, 2026, Munich, Germany Software Engineering and Methodology (TOSEM)11, 3 (2002), 309–346

    2002

  46. [46]

    2021.Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

    Robert Munro Monarch. 2021.Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster

    2021

  47. [47]

    Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python.Journal of Machine Learning Re...

    2011

  48. [48]

    Huilian Sophie Qiu, Anna Lieb, Jennifer Chou, Megan Carneal, Jasmine Mok, Emily Amspoker, Bogdan Vasilescu, and Laura Dabbish. 2023. Climate Coach: A Dashboard for Open-Source Maintainers to Overview Community Dynamics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York,...

    arXiv 2023

  49. [49]

    Marco Ortu, Bram Adams, Giuseppe Destefanis, Parastou Tourani, Michele March- esi, and Roberto Tonelli. 2015. Are bullies more productive? Empirical study of affectiveness vs. issue fixing time. In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE

    2015

  50. [50]

    Nick Pangakis and Sam Wolken. 2025. Keeping Humans in the Loop: Human- Centered Automated Annotation with Generative AI. InInternational AAAI Conference on Web and Social Media

    2025

  51. [51]

    Rahat Rizvi Rahman, Seung Pyo Kang, Se Jin Jang, Seungmoo Lee, Chungyean Cho, and Kostadin Damevski. 2026. When Pair Programming Gets Emotional: A Case Study of What LLMs Still Miss.International Journal of Software Engineering and Knowledge Engineering(2026), 1–15

    2026

  52. [52]

    Naveen Raman, Minxuan Cao, Yulia Tsvetkov, Christian Kästner, and Bogdan Vasilescu. 2020. Stress and burnout in open source: Toward finding, understand- ing, and mitigating unhealthy interactions. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results. 57–60

    2020

  53. [53]

    Huilian Sophie Qiu, Bogdan Vasilescu, Christian Kästner, Carolyn Egelman, Ciera Jaspan, and Emerson Murphy-Hill. 2022. Detecting interpersonal conflict in issues and code review: cross pollinating open-and closed-source approaches. InProceedings of the 2022 ACM/IEEE 44th International Conference on Software Engineering: Software Engineering in Society. 41–55

    2022

  54. [54]

    Md Shamimur Rahman, Zadia Codabux, and Chanchal K Roy. 2024. Do Words Have Power? Understanding and Fostering Civility in Code Review Discussion. Proceedings of the ACM on Software Engineering1, FSE (2024), 1632–1655

    2024

  55. [55]

    Jaydeb Sarker, Asif Kamal Turzo, and Amiangshu Bosu. 2025. The Landscape of Toxicity: An Empirical Investigation of Toxicity on GitHub.FSE(2025)

    2025

  56. [56]

    Jaydeb Sarker, Asif Kamal Turzo, Ming Dong, and Amiangshu Bosu. 2023. Auto- mated Identification of Toxic Code Reviews Using ToxiCR.ACM Trans. Softw. Eng. Methodol.32, 5 (July 2023), 118:1–118:32. doi:10.1145/3583562

    doi:10.1145/3583562 2023

  57. [57]

    Aimee Kendall Roundtree. 2022. Facial recognition technology codes of ethics: Content analysis and review. In2022 IEEE International Professional Communica- tion Conference (ProComm). IEEE, 211–220

    2022

  58. [58]

    Jaydeb Sarker. 2022. Identification and mitigation of toxic communications among open source software developers. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–5

    2022

  59. [59]

    Margaret-Anne Storey, Alexey Zagalsky, Fernando Figueira Filho, Leif Singer, and Daniel M German. 2016. How social and communication channels shape and challenge a participatory culture in software development.IEEE Transactions on Software Engineering43, 2 (2016), 185–204

    2016

  60. [60]

    Damian A Tamburri, Philippe Kruchten, Patricia Lago, and Hans van Vliet. 2015. Social debt in software engineering: insights from industry.Journal of Internet Services and Applications6, 1 (2015), 10

    2015

  61. [61]

    Martin Saveski, Brandon Roy, and Deb Roy. 2021. The Structure of Toxic Conversations on Twitter. InProceedings of the Web Conference 2021 (WWW ’21). Association for Computing Machinery, New York, NY, USA, 1086–1097. doi:10.1145/3442381.3449861

    arXiv 2021

  62. [62]

    Chang, Cristian Danescu-Niculescu-Mizil, and Karen Levy

    Charlotte Schluger, Jonathan P. Chang, Cristian Danescu-Niculescu-Mizil, and Karen Levy. 2022. Proactive Moderation of Online Discussions: Existing Practices and the Potential for Algorithmic Support.Proc. ACM Hum.-Comput. Interact.6, CSCW2 (Nov. 2022), 370:1–370:27. doi:10.1145/3555095

    doi:10.1145/3555095 2022

  63. [63]

    Bianca Trinkenreich, Klaas-Jan Stol, Anita Sarma, Daniel M German, Marco A Gerosa, and Igor Steinmacher. 2023. Do i belong? modeling sense of virtual community among linux kernel contributors. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 319–331

    2023

  64. [64]

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao

  65. [65]

    Damian A Tamburri, Fabio Palomba, and Rick Kazman. 2019. Exploring com- munity smells in open-source: An automated approach.IEEE Transactions on Software Engineering47, 3 (2019), 630–652

    2019

  66. [66]

    Forgotten Server Development Team. 2018. Issue Comment on GitHub: Forgotten Server #2494. https://github.com/otland/forgottenserver/issues/2494/#issueco mment-417775273

    2018

  67. [67]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  68. [68]

    Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. LL- MaAA: Making Large Language Models as Active Annotators. InFindings of the Association for Computational Linguistics: EMNLP 2023. 13088–13103

    2023

  69. [69]

    InProceedings of the CHI Conference on Human Factors in Computing Systems

    Human-LLM collaborative annotation through effective verification of LLM labels. InProceedings of the CHI Conference on Human Factors in Computing Systems. 1–21

  70. [70]

    Zijie J Wang, Dongjin Choi, Shenyu Xu, and Diyi Yang. 2021. Putting Humans in the Natural Language Processing Loop: A Survey. InProceedings of the First Work- shop on Bridging Human–Computer Interaction and Natural Language Processing. 47–52

    2021

  71. [71]

    Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022. A survey of human-in-the-loop for machine learning.Future Generation Computer Systems135 (2022), 364–381

    2022

  72. [72]

    Frances Zlotnick. 2017. GitHub Open Source Survey 2017. http://opensourcesu rvey.org/2017/. doi:10.5281/zenodo.806811 Received 2026-03-26; accepted 2026-06-18

    work page doi:10.5281/zenodo.806811 2017

  73. [74]

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. Senti- ment Analysis in the Era of Large Language Models: A Reality Check. InFindings of the Association for Computational Linguistics: NAACL 2024, Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 3881–3906. https://aclan...

    2024

  74. [75]

    Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. InProceedings of the 2020 conference on fairness, accountability, and transparency. 295–305

    2020

  75. [76]

    Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2024. Exploring the capability of chatgpt to reproduce human labels for social com- puting tasks. InInternational Conference on Advances in Social Networks Analysis and Mining. Springer, 13–22

    2024

  76. [2022]

    InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

    Data augmentation for improving emotion recognition in software en- gineering communication. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–13

  77. [2024]

    InPro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations

    MEGAnno+: A Human-LLM Collaborative Annotation System. InPro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 168–176