Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By treating the prosociality definition as an editable artifact and letting a calibrated cheap classifier abstain on ambiguous cases, this paper claims to reach ~0.90 precision on game-chat prosocial detection while cutting large-model usag

desk verdict A genuinely useful applied pipeline for LLM-assisted annotation, but the headline precision and cost numbers currently rest on a gold standard that was partly built with the very model being evaluated—so treat them as promising, not proven. read the letter →

arxiv 2508.05938 v1 pith:W6HZROZU submitted 2025-08-08 cs.CL cs.AIcs.CY

Rafal Kocielnik , Min Kim , Penphob (Andrea) Boonyarungsrit , Fereshteh Soltani , Deshawn Sambrano , Animashree Anandkumar , R. Michael Alvarez This is my paper
classification cs.CLcs.AIcs.CY
keywords prosocialbehaviorin-gamechatLLM-assistedannotationhuman-AIcollaborationselectiveroutingdefinitionrefinementcontentmoderationcost-efficientclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prosociality in game chat—messages that affirm, support, or encourage other players—is a useful social signal, but it has no established definition or labeled corpus. The paper tries to show that a practical system can be built anyway, using three steps: pick the best LLM prompting strategy against a small human-labeled seed set, refine the operational definition through expert review of model–human disagreements, and then synthesize 10k labels to train a cheap classifier with a selective fallback to a large language model. Across six definition iterations, model–human disagreement fell from 25.4% to 9.7%. The deployed two-stage system routes only ~35% of cases to GPT-4o, cutting large-model inference cost by ~70% while reaching precision around 0.90—higher than the large model achieves on its own. If true, this gives content-safety teams a recipe for scaling detection of a new social construct without a massive annotation budget.

What carries the argument

The calibrated SVM acting as a risk-aware abstainer: it emits well-calibrated probabilities (ECE 0.0352), and a midpoint–tolerance window reroutes low-confidence predictions to GPT-4o. The second mechanism is the definition artifact D1–D6, a version-controlled operational definition of prosociality that the human–LLM loop repeatedly rewrites to shrink ambiguity before labels are generated. The third is the RAG labeling configuration—retrieval-augmented generation with 16 similar examples per class, a generic label token instead of 'PROSOCIAL', and no persona framing—which aligns LLM outputs with human judgments in the seed set.

What would settle it

Take a fresh random sample from the Deployment Dataset, have new human annotators label it from scratch without seeing GPT-4o's predictions or the D6 definition, and compare the hybrid's precision against standalone GPT-4o on that independent set. If the hybrid no longer exceeds GPT-4o's precision—or if the fresh annotators disagree with the adjudicated labels on more than ~10% of items—the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that prosocial chat detection can be made production-ready by treating the task definition as an optimizable artifact and by letting a calibrated classifier abstain on ambiguous cases. Six guided revisions of the prosociality definition—adding outcome-based categories, exclusions, and anti-patterns—reduced GPT-4o–human disagreement from 25.4% to 9.7%. The refined definition, combined with retrieval-augmented prompting (16 similar examples per class, generic label token, no persona), let GPT-4o label 10k new chat instances used to train a linear SVM. At a routing midpoint of 0.7 and tolerance 0.3, about 35% of items go to GPT-4o; the hybrid reaches precision ~0.90

Load-bearing premise

The whole evaluation is scored against an adjudicated gold standard created by human reviewers who manually reviewed disagreements between GPT-4o and other models; if those reviewers were anchored by the same GPT-4o outputs and refined definition, the reported precision and the 'hybrid beats GPT-4o' result are not independent measurements.

Editorial extensions

If this is right

  • Live game-chat moderation can use a cheap local classifier for most traffic and reserve an LLM for the ambiguous tail, cutting cost by ~70% at precision ~0.90.
  • The definition-refinement loop can be reused for other underdefined social constructs, such as respect or support, where labeling guides are absent.
  • Prompt details—label wording, persona framing, retrieval examples—shift LLM agreement by up to 8 AUROC points, so they should be tuned and audited, not assumed neutral.
  • Selective routing should be the default comparison baseline for LLM annotation systems, rather than full LLM inference or full local inference.
  • Reward programs that amplify prosocial behavior in games become feasible because detection can run at scale and at cost that does not scale linearly with message volume.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to swap the GPT-4o fallback for a smaller quantized model; if precision stays at ~0.90, the routing mechanism, not the oracle's size, is doing the work.
  • The same neutral-label trick could be turned into a screening test: before trusting any LLM label on a socially loaded category, compare accuracy with a generic token and audit the gap.
  • If the adjudicated gold standard was created by humans who had already seen GPT-4o's predictions, the hybrid's precision edge over GPT-4o should be re-measured on an independently annotated sample; this is the paper's load-bearing condition.
  • The six definition versions imply a cheap diagnostic: track agreement on held-out anchor examples after every prompt edit; a drop flags definition drift before deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a three-stage pipeline for prosocial behavior detection in player game chat: (1) prompt-design selection using 960 human-labeled instances, (2) iterative human-AI refinement of the prosociality definition (D1-D6) driven by GPT-4o-versus-human disagreements, and (3) synthesis of 10k labels with GPT-4o, followed by training a calibrated SVM that routes low-confidence predictions to GPT-4o. The authors claim that this hybrid system reduces LLM inference cost by roughly 70% while achieving precision around 0.90, exceeding any single model in their comparison. They also report that definition refinement reduced model-human disagreement from 25.4% to 9.7%.

Significance. If the evaluation were independent, this would be a useful applied contribution: it addresses a real underdefined task, uses a sizable deployment-oriented dataset, includes controlled prompt ablations (persona, label terminology, RAG counts), and provides a cost-quality routing analysis grounded in a calibrated classifier. The paper also identifies practical phenomena such as label-name semantic anchoring and definition drift. However, the headline quantitative claims rest on a gold standard that is not fully independent of GPT-4o, and the definition-refinement curve is measured in-sample. These concerns bear directly on the central claims and need to be resolved before the results can be accepted.

major comments (4)
  1. [§3.4/§4.3] The Deployment Dataset gold standard is not independent of GPT-4o. The text states that annotators 'manually reviewed disagreements between GPT-4o and other model predictions using the updated prosociality definition, yielding an adjudicated ground truth set used in all evaluations.' Since the 10k labels were initially produced by GPT-4o and the other models were trained on those same labels, instances on which models agree retain the GPT-4o label. Tables 3-4 therefore compare models against a target that is largely GPT-4o's own output; the claim that the hybrid 'exceeds the precision of any single model' is at risk of measuring self-agreement. Please re-evaluate on a held-out set of independent human labels, or report results separately on the human-adjudicated subset with a clear statement of coverage.
  2. [§4.2/Table 2] The 63% disagreement reduction (25.4% to 9.7%) is measured on the same 960 Initial Dataset instances whose model-human disagreements motivated the D1-D6 edits. This is an in-sample curve, not a generalization result. Moreover, 'Label Chg.' rows show that ground-truth labels themselves were revised during the loop, so part of the reduction may reflect relabeling rather than definition improvement. To support the definition-refinement claim, freeze a held-out validation set before edits and report disagreement for each D1-D6 on that set.
  3. [§3.4/§4.3.1] The feature representation used for the SVM and other classical models is not specified. The paper reports SVM AUC 0.77, ECE 0.0352, and uses SVM probabilities as the routing signal, but never states whether inputs are TF-IDF vectors, sentence-embedding features, or something else. Without this, the calibration and routing results are not reproducible and the model comparison in Table 4 is not interpretable. Please specify the full input pipeline and preprocessing.
  4. [§4.3.1/Tables 3-5] The task has three classes (prosocial, not-prosocial, unclear; see Table 5), yet Tables 3-4 report binary metrics (AUC, precision, recall) without explaining how the 'unclear' class is treated. If 'unclear' is collapsed into non-prosocial, or excluded, this materially changes precision/recall and the routing trade-offs. Please state the binarization rule explicitly and, ideally, report multiclass results or sensitivity analyses.
minor comments (5)
  1. [§2.2/References] The text cites 'Smith et al. [20]', but reference [20] is Liu et al. (2025). Please correct the citation or reference list.
  2. [Figures 3-4] Figure 3 and Figure 4 are referenced but not placed in the text; ensure captions define all colors and markers. Figure 4's caption mentions green/orange points that are not visible in the manuscript.
  3. [Table 3] The headline comparison (e.g., precision 0.892 vs 0.870) is reported as a point estimate without confidence intervals. Given that model results are averaged over three seeds in §3.3, please add intervals or error bars for the routing trade-off table.
  4. [Appendix B] In Tables 6 and 7, 'Changes made from precious definition iteration' should be 'previous definition iteration'.
  5. [Abstract and Figure 1] The '70% cost reduction' claim depends on the cost model used (per-token vs. per-call latency vs. dollar cost). Please state the cost model explicitly so the figure can be interpreted.

Circularity Check

2 steps flagged · score 6.0 of 10

Deployment 'ground truth' is built from GPT-4o's own labels, reviewed only where models disagreed (§3.4 Step 3), then GPT-4o is benchmarked on it; the 25.4%→9.7% D1–D6 disagreement drop is measured on the same 960 instances that drove the definition edits.

  1. self definitional [Section 3.4, Step 3 (Label Expansion & Cost Efficient Labeling at Scale); evaluated in §4.3.1 Table 4 and §4.3.3]
    "Annotators manually reviewed disagreements between GPT-4o and other model predictions using the updated prosociality definition, yielding an adjudicated ground truth set used in all evaluations. GPT-4o was also reapplied with the same prompting strategy to benchmark its own performance on this new gold standard."

    By the paper's own protocol, humans reviewed only instances where GPT-4o disagreed with the other models; every agreement instance therefore inherits the GPT-4o synthetic label with no human review. The other models are not independent arbiters—'These high-fidelity synthetic labels served as supervision for training a suite of classical and neural models'—so their agreement with GPT-4o largely reflects shared bias from the same training labels. GPT-4o and the hybrid are then benchmarked 'on this new gold standard': the reported GPT-4o AUC 0.90 / precision 0.87 and the hybrid precision 0.89–0.90 ('exceeds the precision of any standalone model in Table 4') are measured against a target that, on every un-adjudicated instance, equals the evaluated system's own output. Systematic GPT-4o errors

  2. fitted input called prediction [Section 3.4, Step 2 (Human-AI Collaboration for Prosocial Definition Refinement); reported in §4.2, Table 2 / Figure 3]
    "Overall, disagreement was reduced by 63%—from 218 instances (25.4%) in D1 to 93 (9.7%) in D6—demonstrating that structured, iterative updates guided by human-LLM disagreements can enhance annotation consistency and clarify semantic boundaries."

    The D2–D6 definition edits were derived from the very disagreements being measured, on the same 960 Initial Dataset instances ('Each disagreement instance was either used to refine the label or update the prosociality definition'), and the loop's own stopping rule was 'until we achieved a satisfactory disagreement rate of under 10%'. The reported 9.7% final disagreement is thus the stopping criterion achieved by in-sample optimization—edits were selected to remove exactly these disagreements—not a held-out estimate of model–human alignment. Moreover, 1–2% of labels per iteration changed ('Label Chg. %' in Table 2) under the updated definition, so the human-reference target itself moved toward the model. The 63% 'reduction' therefore conflates improving the model with redefining both the de

full rationale

The paper is not wholly circular: the Initial Dataset carries genuine two-annotator human labels (Section 3.1), Step 1's strategy comparison (AUC 0.85, precision 0.93, Table 1) is anchored to those human labels, expert adjudicators genuinely reviewed a subset of disagreements, and the SVM calibration analysis (ECE 0.0352, Appendix C) is an independent measurement. However, the central deployment claims rest on two load-bearing reductions. First, the Deployment Dataset gold standard used in 'all evaluations' (Section 3.4 Step 3) was created by adjudicating only GPT-4o-vs-other-model disagreements; all agreement instances retain GPT-4o's label. Since the other models were trained on GPT-4o's synthetic labels, model-model agreement is expected on shared-bias instances, so GPT-4o's 0.90 AUC / 0.87 precision and the hybrid's 0.89–0.90 precision are measured, in substantial part, against labels the evaluated system itself produced. The 'hybrid beats GPT-4o' comparison is computed on the same contaminated target, so it does not provide the independent human anchor the claim requires. The fraction of instances actually adjudicated is never reported, but the design guarantees non-independence for the complement. Second, the headline definition-refinement result—a 63% disagreement reduction to 9.7%—is measured in-sample on the same 960 instances that drove edits D2–D6, with a stopping rule of under 10% disagreement, and with 1–2% of gold labels themselves changed each iteration; the number converges to its own stopping criterion by construction. These two steps make the headline precision/cost and alignment claims partially self-referential, though the pipeline architecture itself (selective routing, calibration) remains independently meaningful. A blind, fully human-labeled held-out gold set would resolve the shortfall. Score 6: one or more 'predictions' reduce by construction; partial, not total, circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The pipeline introduces no physical or latent entities. The main burdens are the operational definition of prosociality, the assumption that GPT-4o synthetic labels are reliable supervision, and the validity of the adjudicated gold set. The two listed free parameters are the most consequential tunable choices.

free parameters (2)
  • RAG few-shot count k = 16 per class
    Chosen by grid search on the Initial Dataset; final labeling of the 10k deployment set used 16x2 RAG examples.
  • Routing midpoint and tolerance = midpoint 0.7, tolerance 0.3 (~35% data routed)
    Operating point highlighted in Table 3 and Figure 4, selected from development data to maximize precision at moderate cost; other midpoints (0.3, 0.5) give different precision/recall trade-offs.
assumptions (4)
  • domain assumption The final definition D6 is a valid operationalization of prosociality in game chat.
    D6 is the product of six in-sample edits (Appendix B, Tables 6-7); no independent construct validation against external prosociality measures is provided.
  • domain assumption GPT-4o labels generated under D6 are accurate enough to serve as training supervision for the SVM.
    Section 4.3 uses 10k GPT-4o labels as high-fidelity supervision; only a subset is audited via disagreement review, not fully human-verified.
  • domain assumption The adjudicated Deployment Dataset gold labels are a valid, independent ground truth.
    Section 4.3: gold set created by annotators reviewing GPT-4o-vs-model disagreements; this admits anchoring and is not a purely human-only held-out benchmark.
  • domain assumption The Initial Dataset from Li et al. [18] is representative enough for prompt selection and definition refinement.
    The paper uses 960 COD:MWII entries from 2023 to choose prompts and refine definitions, then applies them to COD:WZ2 data from 2024; distribution shift is not directly quantified for definitional choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale." pith.science (2026). https://pith.science/paper/W6HZROZU

@misc{pith2026250805938,
  author       = {Pith},
  title        = {Pith review of: Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6HZROZU}},
  note         = {Machine review of arXiv:2508.05938}
}
abstract

Detecting prosociality in text--communication intended to affirm, support, or improve others' behavior--is a novel and increasingly important challenge for trust and safety systems. Unlike toxic content detection, prosociality lacks well-established definitions and labeled data, requiring new approaches to both annotation and deployment. We present a practical, three-stage pipeline that enables scalable, high-precision prosocial content classification while minimizing human labeling effort and inference costs. First, we identify the best LLM-based labeling strategy using a small seed set of human-labeled examples. We then introduce a human-AI refinement loop, where annotators review high-disagreement cases between GPT-4 and humans to iteratively clarify and expand the task definition-a critical step for emerging annotation tasks like prosociality. This process results in improved label quality and definition alignment. Finally, we synthesize 10k high-quality labels using GPT-4 and train a two-stage inference system: a lightweight classifier handles high-confidence predictions, while only $\sim$35\% of ambiguous instances are escalated to GPT-4o. This architecture reduces inference costs by $\sim$70% while achieving high precision ($\sim$0.90). Our pipeline demonstrates how targeted human-AI interaction, careful task formulation, and deployment-aware architecture design can unlock scalable solutions for novel responsible AI tasks.

Figures

Figures reproduced from arXiv: 2508.05938 by the authors.

Figure 1
Figure 1. End-to-end pipeline for scalable, cost-efficient prosocial behavior classification in multiplayer game chat. Step 1: [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Impact of individual prompt design factors on LLM annotation performance on [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. LLM–human disagreement and definition drift [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Trade-off between routing threshold tolerance, performance, and inference cost across uncertainty midpoints on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Calibration of Predicted Probabilities for SVM and BERT Classifiers. The histogram bars (blue for SVM, green for [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    LLMs correct only 34.8% of zero-shot annotation errors via prompting, and Definition-Specific Familiarity correlates positively with performance (partial r = +0.41) while memorization metrics do not.

Reference graph

Works this paper leans on

49 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    Nicole A Beres, Julian Frommel, Elizabeth Reid, Regan L Mandryk, and Madison Klarkowski. 2021. Don’t you know that you’re toxic: Normalization of toxicity in online gaming. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–15

    work page 2021

  2. [2]

    Jeremy Blackburn and Haewoon Kwak. 2014. STFU NOOB! predicting crowd- sourced decisions on toxic behavior in online games. In Proceedings of the 23rd international conference on World wide web . 877–888

    work page 2014

  3. [3]

    Nan-Chen Chen, Margaret Drouhard, Rafal Kocielnik, Jina Suh, and Cecilia R Aragon. 2018. Using machine learning to support qualitative coding in social sci- ence: Shifting the focus to ambiguity. ACM Transactions on Interactive Intelligent Systems (TiiS) 8, 2 (2018), 1–20

    work page 2018

  4. [4]

    Margaret Drouhard, Nan-Chen Chen, Jina Suh, Rafal Kocielnik, Vanessa Pena- Araya, Keting Cen, Xiangyi Zheng, and Cecilia R Aragon. 2017. Aeonium: Visual analytics to support collaborative qualitative coding. In 2017 IEEE Pacific Visualization Symposium (PacificVis). IEEE, 220–229

    work page 2017

  5. [5]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al

  6. [6]

    Ran El-Yaniv et al. 2010. On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research 11, 5 (2010)

    work page 2010

  7. [7]

    Fahimeh Fakour, Ali Mosleh, and Ramin Ramezani. 2024. A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learning. arXiv preprint arXiv:2406.00332 (2024)

    arXiv 2024

  8. [8]

    David Farr, Iain Cruickshank, Nico Manzonelli, Nicholas Clark, Kate Starbird, and Jevin West. 2024. LLM Confidence Evaluation Measures in Zero-Shot CSS Classification. arXiv preprint arXiv:2410.13047 (2024)

    work page Pith review arXiv 2024

Show all 49 references
  1. [9]

    David Farr, Nico Manzonelli, Iain Cruickshank, Kate Starbird, and Jevin West

  2. [10]

    Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating Label Biases for In-context Learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 14014–14031

    2023

  3. [11]

    Nate Garrelts and Matthew Wilhelm Kapell. 2017. Responding to Call of Duty: Critical essays on the game franchise . McFarland

    2017

  4. [12]

    Pengrui Han, Peiyang Song, Haofei Yu, and Jiaxuan You. 2024. In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024. 5624–5643

    2024

  5. [13]

    Tomas Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, and Timo Spinde. 2024. The Promises and Pitfalls of LLM Annota- tions in Dataset Labeling: a Case Study on Media Bias Detection. arXiv preprint arXiv:2411.11081 (2024)

    2024 arXiv

  6. [14]

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. InThe Twelfth Internation...

    2024

  7. [15]

    Rafal Kocielnik, Sara Kangaslahti, Shrimai Prabhumoye, Meena Hari, Michael Alvarez, and Anima Anandkumar. 2023. Can you label less by using out-of- domain data? Active & transfer learning with few-shot instructions. In Transfer Learning for Natural Language Processing Workshop...

    2023

  8. [16]

    Bastian Kordyaka, Katharina Jahn, and Bjoern Niehaves. 2020. Towards a unified theory of toxic behavior in video games.Internet Research 30, 4 (2020), 1081–1102

    2020

  9. [17]

    Deepak Kumar, Yousef Anees AbuHashem, and Zakir Durumeric. 2024. Watch your language: Investigating content moderation with large language models. In Proceedings of the International AAAI Conference on Web and Social Media , Vol. 18. 865–878

    2024

  10. [18]

    Zhuofang Li, Rafal Kocielnik, Fereshteh Soltani, Animashree Anandkumar, R Michael Alvarez, et al . 2025. Self-Anchored Attention Model for Sample- Efficient Classification of Prosocial Text Chat. arXiv preprint arXiv:2506.09259 (2025)

    2025 arXiv

  11. [19]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024)

    2024

  12. [20]

    Xiner Liu, Andres Felipe Zambrano, Ryan S Baker, Amanda Barany, Jaclyn Ocumpaugh, Jiayi Zhang, Maciej Pankiewicz, Nidhi Nasiar, and Zhanlan Wei

  13. [21]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  14. [22]

    Aniket Nagle, Peter Wolf, Robert Riener, and Domen Novak. 2014. The use of player-centered positive reinforcement to schedule in-game rewards increases enjoyment and performance in a serious game. The International Journal of Serious Games 1, 4 (2014), 35–47

    2014

  15. [23]

    Guruprasad Nookala. 2024. Adaptive Data Governance Frameworks for Data- Driven Digital Transformations. Journal of Computational Innovation 4, 1 (2024)

    2024

  16. [24]

    Nicolas Posocco and Antoine Bonnefoy. 2021. Estimating expected calibration errors. In Artificial Neural Networks and Machine Learning–ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, Sep- tember 14–17, 2021, Proceedings, Part IV 3...

    2021

  17. [25]

    Shrimai Prabhumoye, Rafal Kocielnik, Mohammad Shoeybi, Anima Anandku- mar, and Bryan Catanzaro. 2021. Few-shot instruction prompts for pretrained language models to detect social biases. arXiv preprint arXiv:2112.07868 (2021)

    2021 arXiv

  18. [26]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing . Association for Computational Linguistics. https://arxiv.org/abs/1908.10084

    2019 arXiv

  19. [27]

    K Andrew R Richards and Michael A Hemphill. 2018. A practical guide to collaborative qualitative data analysis. Journal of Teaching in Physical education 37, 2 (2018), 225–231

    2018

  20. [28]

    Hope Schroeder, Marianne Aubin Le Quéré, Casey Randazzo, David Mimno, and Sarita Schoenebeck. 2025. Large Language Models in Qualitative Research: Uses, Tensions, and Intentions. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems . 1–17

    2025

  21. [29]

    Kayla Schroeder and Zach Wood-Doughty. 2024. Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge. arXiv preprint arXiv:2412.12509 (2024)

    2024 arXiv

  22. [30]

    Meghna Sharma, Kritarth Drall, Tanvi Tanvi, Tridev Parida, and Valliti Lokesh

  23. [31]

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large Language Models for Data Annotation and Synthesis: A Survey. InProceed- ings of the 2024 Conference on Empirical Methods in Na...

    2024

  24. [32]

    Sian Tomkinson and Benn Van Den Ende. 2022. ‘thank you for your compliance’: Overwatch as a disciplinary system. Games and culture 17, 2 (2022), 198–218

    2022

  25. [33]

    Josiane Van Dorpe, Zachary Yang, Nicolas Grenon-Godbout, and Grégoire Win- terstein. 2023. Unveiling Identity Biases in Toxicity Detection: A Game-Focused Dataset and Reactivity Analysis Approach. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language P...

    2023

  26. [34]

    Sarah Van Hoeyweghen. 2024. Speaking of games: AI-based content modera- tion of real-time voice interactions in video games under the DSA. Interactive Entertainment Law Review 7, 1 (2024), 30–46

    2024

  27. [35]

    AI-Powered Smart Social Content Filter for Identifying Harmful Online Content. (2025)

    2025

  28. [36]

    Ximing Wang and Panos M Pardalos. 2014. A survey of support vector machines with uncertainties. Annals of Data Science 1 (2014), 293–309. Kocielnik et al

    2014

  29. [37]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems 36 (2023), 15614–15638

    2023

  30. [38]

    Henry Weld, Guanghao Huang, Jean Lee, Tongshu Zhang, Kunze Wang, Xinghong Guo, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2021. CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection. arXiv preprint arXiv:2106.06213 (2021)

    2021 arXiv

  31. [39]

    Michel Wijkstra, Katja Rogers, Regan L Mandryk, Remco C Veltkamp, and Ju- lian Frommel. 2023. Help, My Game Is Toxic! First Insights from a Systematic Literature Review on Intervention Systems for Toxic Behaviors in Online Video Games. In Companion Proceedings of the Annual Sy...

    2023

  32. [40]

    Luis Vila-Henninger, Claire Dupuy, Virginie Van Ingelgom, Mauro Caprioli, Ferdinand Teuber, Damien Pennetreau, Margherita Bussi, and Cal Le Gall. 2024. Abductive coding: Theory building and qualitative (re) analysis. Sociological Methods & Research 53, 2 (2024), 968–1001

    2024

  33. [41]

    Zachary Yang, Yasmine Maricar, MohammadReza Davari, Nicolas Grenon- Godbout, and Reihaneh Rabbany. 2023. Toxbuster: In-game chat toxicity buster with BERT. arXiv preprint arXiv:2305.12542 (2023)

    2023 arXiv

  34. [42]

    Zachary Yang, Domenico Tullo, and Reihaneh Rabbany. 2025. Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource- Efficient Toxicity Detection. arXiv preprint arXiv:2506.06347 (2025)

    2025 arXiv

  35. [43]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Pr...

    2023

  36. [45]

    Lingling Xu, Haoran Xie, Zongxi Li, Fu Lee Wang, Weiming Wang, and Qing Li

  37. [46]

    ACM Transac- tions on Intelligent Systems and Technology 14, 4 (2023), 1–34

    Contrastive learning models for sentence representations. ACM Transac- tions on Intelligent Systems and Technology 14, 4 (2023), 1–34

    2023

  38. [2022]

    In International conference on machine learning

    Glam: Efficient scaling of language models with mixture-of-experts. In International conference on machine learning . PMLR, 5547–5569

  39. [2023]

    Advances in Neural Information Processing Systems 36 (2023), 46534–46594

    Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2023), 46534–46594

    2023

  40. [2024]

    In 2024 IEEE International Conference on Big Data (BigData)

    Llm chain ensembles for scalable and accurate data annotation. In 2024 IEEE International Conference on Big Data (BigData) . IEEE, 2110–2118

    2024

  41. [2025]

    Qualitative Coding with GPT-4: Where It Works Better.Journal of Learning Analytics 12, 1 (2025), 169–185

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.