Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Iterative prompt refinement lifts ChatGPT's agreement with expert raters on targeting language from a Cohen's Kappa of 0.40 to 0.66, while inappropriate-language detection already aligns closely.

desk verdict The headline Kappa 0.66 is a selection-on-the-evaluation-set artifact; the paper is a useful prompt-engineering case study but does not establish its central improvement claim. read the letter →

arxiv 2505.21710 v1 pith:BPGCUCTG submitted 2025-05-27 cs.CL

Barbarestani Baran , Maks Isa , Vossen Piek This is my paper
classification cs.CL
keywords contentmoderationtargetinglanguageinappropriateChatGPTpromptengineeringCohen'sKappacrowdannotationsReddit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether ChatGPT can be made useful for automated content moderation by detecting 'targeting' language—attacks on a person or group based on attributes such as race, gender, or politics—and 'inappropriate' language such as profanity and slurs. It claims that iterative refinement of the labeling prompt and model settings, across six versions, raises ChatGPT's agreement with expert annotations on the gold set from Cohen's Kappa 0.40 to 0.66. On the full data set, the best version achieves Kappa 0.516 against crowd annotations, which the paper interprets as moderate but meaningful agreement. The paper also reports that ChatGPT over-detects targeting relative to experts, with false positives falling from 61 to 27 across the prompt versions, and that its agreement with experts on inappropriate language is much higher, Kappa 0.88. The paper's contribution is a demonstration that prompt iteration can align a general-purpose conversational model with human moderation judgments, plus an honest account of where that alignment remains incomplete.

What carries the argument

The carrying mechanism is a prompt-version ladder: six successive versions of a system prompt that tell ChatGPT to read post, context, and comment, decide whether a specific person or group is targeted, and answer with exactly one of two labels. The ladder works by simplifying the decision space from a three-way code to binary classification, switching between gpt-4 and gpt-4o, tightening the temperature from 0.8 to 0.3 and then to 0, and adding modular code with retries and logging. Agreement is measured with Cohen's Kappa, a chance-corrected statistic that the paper uses to compare ChatGPT, experts, and crowd annotators. The decisive variables turn out to be the binary label scheme, the underlying model, and deterministic decoding; the code-level additions such as logging and retry handling make smaller contributions.

What would settle it

A concrete check would be to re-annotate a random sample of the full data set with expert moderators and compare Version 6's labels against those experts. If the resulting Kappa is substantially below the 0.516 reported against the crowd, or if the over-detection pattern worsens, the claim that ChatGPT is aligned with human judgments on targeting language fails; if the expert-based Kappa stays near 0.66, the crowd benchmark was the weak link.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the largest gains in ChatGPT's moderation performance come from prompt and model configuration, not from additional training. The original setup asked for a three-way targeting decision and agreed with experts at Kappa 0.40; Version 1 simplified the task to a binary TARGETING/NOT TARGETING judgment and switched to gpt-4, raising Kappa to 0.62. Later changes produced Kappa scores of 0.61, 0.65, 0.54, 0.65, and finally 0.66: switching from gpt-4 to gpt-4o in Version 4 degraded agreement, while reverting to gpt-4 and lowering the temperature to 0.3 and then to 0 restored and slightly improved it. Against expert labels, Version 6 still over-detects targeting, marking 27 comments as targeting that experts did not and missing 14 that experts flagged. On the full crowd-annotated data, the same version reaches Kappa 0.516, missing 377 crowd-positive targeting comments and adding 231 that the crowd did not mark, which the paper reads as a strong basis for using ChatGPT as a complement to human review rather than a replacement.

Load-bearing premise

The full-data comparison assumes crowd annotators are a reliable benchmark for targeting language, even though their agreement with experts was only moderate (Cohen's Kappa 0.58) and their internal agreement was much lower (0.36), so the headline 0.516 score is only as trustworthy as those crowd labels.

Editorial extensions

If this is right

  • A tuned ChatGPT version can serve as a scalable first-pass filter that flags likely targeting and inappropriate comments for human moderators, with the final decision left to people.
  • The version-by-version trajectory implies that simple binary labels, a stable model choice, and temperature 0 matter more than infrastructure refinements like retry logic and logging.
  • Because ChatGPT over-detects targeting, a moderation pipeline built on it would catch more harmful content but would also send more benign comments to human review, increasing workload.
  • The strong inappropriate-language agreement (Kappa 0.88 against experts) suggests that the model is closer to deployable accuracy for profanity and slurs than for attributional targeting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the crowd labels are less reliable than the paper assumes, the full-data Kappa of 0.516 may understate ChatGPT's true agreement with the underlying phenomenon, because the yardstick itself is noisy.
  • A natural extension the paper does not run is to use ChatGPT-crowd disagreement as a triage signal: comments where the two disagree would be exactly the ones worth sending to expert adjudication.
  • The tuned prompts were developed on Reddit data in English, so an immediate testable extension is whether the same version ladder transfers to other platforms and languages, or whether the binary targeting definition needs local recalibration.
  • The reported Version 4 drop (0.65 to 0.54 when moving from gpt-4 to gpt-4o) suggests model version choice can outweigh prompt wording, which is a caution for anyone building moderation tools on fast-moving commercial APIs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript evaluates ChatGPT's ability to identify targeting and inappropriate language in Reddit comments, comparing its annotations against expert and crowd annotations. It documents an iterative prompt-refinement process across six versions and reports that Version 6 achieves a Cohen's Kappa of 0.66 against the expert gold set and 0.516 against crowd annotations on the full data set. The paper concludes that iterative refinements, particularly Version 6, notably improve targeting detection while acknowledging higher false-positive rates. Additional analyses examine the onset, prevalence, and development of targeting behavior, the effect of contextual cues, and annotation scopes.

Significance. The paper addresses a practically relevant question: can ChatGPT be systematically prompted to align with human judgments for content moderation? Its strengths include the use of an existing annotated dataset, transparent documentation of all six prompt versions in the appendix, and a detailed error analysis separating false positives and false negatives. However, the headline quantitative claim is not established by the reported evaluation. The best prompt version is selected by maximizing Kappa on the expert gold set, and that same gold-set Kappa is then reported as the result, which is a selection-on-evaluation-set procedure. The full-data Kappa of 0.516 is less affected by this issue but is measured against crowd annotations whose reliability for targeting is itself weak. If the evaluation were corrected, a moderate-agreement finding could still be useful, but as presented the improvement claim is not supported.

major comments (3)
  1. [§5.1.7 and §5.2] The central improvement claim is undermined by selection on the evaluation set. Section 5.1.7 reports the gold-set Kappa for each of six prompt versions, and Section 5.2 states that Version 6 was applied to the broader data 'given that Version 6 achieved the highest Cohen's Kappa score of 0.66' on that same gold set. The gold expert set is therefore used both to select the best version and to report its final performance. The expected value of the maximum of six noisy Kappas exceeds the true agreement, and the 0.01 difference between Version 5 (0.65) and Version 6 (0.66) is plausibly within sampling noise. No confidence interval, bootstrap, or held-out validation is provided. The Section 7 limitations list does not acknowledge this circularity. The authors should either validate the selected version on a separate expert-annotated set or explicitly treat the gold-set numbers as exploratory and base any improvement claim on a pre-specified evaluation, such as comparing the original prompt and Version 6 on the crowd-annotated full data.
  2. [§5.1.7, Version 5 paragraph] The numerical reporting in the Version 5 paragraph is internally inconsistent. The text states that the number of consistently classified 'NOT TARGETING' comments 'increased from 96 to 85' and that false positives 'decreased from 16 to 27'; both statements describe changes in the wrong direction. The intended comparison to Version 4 appears to be reversed. Because the narrative of monotonic improvement across Versions 4-6 relies on these counts, they must be corrected and re-verified against the underlying annotations.
  3. [§3 and §4] The full-data evaluation uses crowd annotations as the benchmark, but the reliability of that benchmark for targeting is not established. Section 3 reports that crowd annotators had an internal Cohen's Kappa of 0.36 for targeting and that crowd-expert Kappa was 0.58. Section 4 interprets these as indicating that crowd annotations are 'sufficiently reliable and align well with expert judgments,' which is not supported: 0.36 is weak agreement, and 0.58 is moderate at best. Consequently, the all-data Kappa of 0.516 between ChatGPT and crowd annotations should be understood as agreement with a noisy benchmark, not as a direct measure of accuracy against expert judgments. The manuscript should temper its conclusions and, ideally, report agreement measures that account for benchmark unreliability.
minor comments (5)
  1. [§4.1] The sentence 'The results are detailed in Tables 3, 5, 7, and 8)' contains a stray closing parenthesis and should be corrected.
  2. [§3, Table 1 and Table 2] The captions say 'Annotation Results ... by Crowd, Experts, and ChatGPT' and 'by Crowd and ChatGPT,' but the tables do not make clear which ChatGPT version produced the counts; this should be stated explicitly, since the original and Version 6 prompts differ materially.
  3. [§5.1.1, Table 4] The token-type labels 'offensive' and 'non-offensive' are applied to individual tokens such as 'gay' and 'the' without a definition of the annotation procedure; a sentence describing how token types were assigned would improve reproducibility.
  4. [§4.4] The description of Version 1 as introducing a binary 'TARGETING' versus 'NOT TARGETING' distinction is clear, but the original prompt in Figure 15 already shows that distinction; the text should clarify that the change was from a three-way I/O/N scheme to the binary scheme.
  5. [§5.1.6] Table 11 reports agreement percentages without sample sizes or confidence intervals; given the small number of comments in some scenarios, adding counts or intervals would help the reader assess the stability of the comparisons.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline gold-set Kappa is selected on the same gold set used to report it, so the central improvement claim (0.4 to 0.66) is an optimistic fitted value rather than a held-out estimate; the full-data comparison is more honest but inherits a weak crowd benchmark.

  1. fitted input called prediction [Section 4.4 and Section 5.2]
    "After validating these refinements with the expert set, we tested the improved prompts on a larger data set with crowd annotations, ensuring that the refinements were both accurate and broadly applicable. [...] Given that Version 6 achieved the highest Cohen’s Kappa score of 0.66 in targeting language identification by ChatGPT compared to the expert annotations on the gold set, we applied the same model and prompts to the broader data set."

    The six prompt versions were evaluated against the same expert gold set that is then used for the headline metric. Section 5.1.7 reports each version's Cohen's Kappa on that gold set, and Section 5.2 selects Version 6 because its gold-set Kappa is the highest, then presents that same gold-set Kappa as the result. The reported 0.66 is therefore max_i κ_i over the six configurations measured on the evaluation set, not an unbiased held-out estimate; the expected maximum of six noisy Kappas exceeds the true agreement, and the comparison to the original 0.4 conflates tuning with measured performance. The 0.01 difference from Version 5 is within sampling noise, and the small gold set (248 targeting comments) makes the selection-on-evaluation-set effect material.

full rationale

The paper is an empirical evaluation rather than a mathematical derivation, so most circularity patterns do not apply. The main circular step is the prompt-selection protocol: Section 4.4 says the six prompt versions were developed and 'validated' against the expert gold set, Section 5.1.7 reports the Kappa of each version on that same set, and Section 5.2 chooses Version 6 for having the highest gold-set Kappa and reports that same number as the headline result. This is selection on the evaluation set, making the reported improvement over the original version statistically forced rather than a prediction on unseen data. The full-data evaluation against crowd annotations (Kappa 0.516) provides some independent content, which prevents the paper from being wholly circular. The self-citation to Barbarestani et al. (2024) supplies the dataset and original prompt, but that is normal provenance and is not itself the circularity; the load-bearing issue is the same-set selection and evaluation. The paper's benchmark-reliability assumption (crowd-expert Kappa 0.58 for targeting) is a correctness risk, not a circularity. Overall, the central improvement claim is substantially weakened by the fitted-selection protocol, but the broader empirical comparison retains partial independent value, so the circularity score is 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central evaluation rests on a benchmark built by the same authors in a prior paper, plus several hand-chosen analytical thresholds (conversation bins, low/high comment cutoff). The prompt version is effectively a fitted parameter because it is chosen by highest Kappa on the gold set. No new entities are introduced, but the benchmark and annotation process are entirely endogenous to the authors' own program.

free parameters (3)
  • Prompt version (including model choice and temperature) = Version 6: gpt-4 with temperature 0
    The authors selected the prompt version with the highest Cohen's Kappa on the expert gold set (Section 4.4 and 5.1.7). This is a free parameter fitted to the evaluation set.
  • Threshold for low vs high comment numbers = 3 comments
    In Section 5.2, the authors define 'low' as the first two comments and 'high' as third onwards, because the minimum thread length is 3. This is an arbitrary analytical choice, not justified by data.
  • Bin boundaries for beginning, middle, final conversation segments = 34%/33%/33% (targeting), 33%/33%/33% (inappropriateness)
    Section 4.1 defines the bins differently for targeting and inappropriateness. These cutoffs are chosen by hand, with no sensitivity analysis.
assumptions (5)
  • domain assumption Crowd annotations are a reliable benchmark for evaluating ChatGPT on the full dataset
    Section 4 claims crowd-expert agreement of 58% Kappa for targeting indicates 'sufficiently reliable' alignment, despite this being only moderate agreement and crowd internal Kappa being 0.36 (Section 3).
  • domain assumption The gold set is representative of toxic online interactions
    Section 3 describes a gold set of 39 subthreads from banned subreddits, oversampled from high-toxicity bins. This is a small, skewed sample and may not generalize.
  • standard math Cohen's Kappa is an appropriate agreement measure for these imbalanced classification tasks
    The paper uses Cohen's Kappa throughout without discussing its behavior under class imbalance or base-rate dependence, which is relevant given the large differences in targeting prevalence.
  • ad hoc to paper The dataset and human annotations from Barbarestani et al. (2024) are accepted as ground truth
    The authors rely on their own prior paper for the dataset, the expert labels, and the crowd labels, with no external validation or independent annotation effort.
  • domain assumption ChatGPT responses are stable enough for single-run evaluation
    Each prompt version is evaluated with a single run (or at least no variance is reported). Even with temperature 0, API endpoints and model weights can change, and no repeated runs or confidence intervals are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study." pith.science (2026). https://pith.science/paper/BPGCUCTG

@misc{pith2026250521710,
  author       = {Pith},
  title        = {Pith review of: Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPGCUCTG}},
  note         = {Machine review of arXiv:2505.21710}
}
read the original abstract

This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate language in online comments. With the increasing challenge of moderating vast volumes of user-generated content on social network sites, the role of AI in content moderation has gained prominence. We compared ChatGPT's performance against crowd-sourced annotations and expert evaluations to assess its accuracy, scope of detection, and consistency. Our findings highlight that ChatGPT performs well in detecting inappropriate content, showing notable improvements in accuracy through iterative refinements, particularly in Version 6. However, its performance in targeting language detection showed variability, with higher false positive rates compared to expert judgments. This study contributes to the field by demonstrating the potential of AI models like ChatGPT to enhance automated content moderation systems while also identifying areas for further improvement. The results underscore the importance of continuous model refinement and contextual understanding to better support automated moderation and mitigate harmful online behavior.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 26 canonical work pages

  1. [1]

    Barbarestani, B., Maks, I., and Vossen, P. T. (2024). Content moderation in online platforms: A study of annotation methods for inappropriate language. In Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying@ LREC- COLING-2024, pp. 96–104

    work page 2024

  2. [2]

    Hurtlex: A multilingual lexicon of words to hurt

    Bassignana, E., Basile, V., Patti, V., and others (2018). Hurtlex: A multilingual lexicon of words to hurt. In CEUR Workshop proceedings, volume 2253, pp. 1–6. CEUR-WS

    work page 2018

  3. [3]

    Bhattacharya, A., Chakrabarti, T., Basu, S., Knott, A., Pedreschi, D., Chatila, R., Leavy, S., Eyers, D., Teal, P. D. , and Biecek, P. (2024). Towards a crowdsourced framework for online hate speech moderation-a case study in the indian political scenario. In Companion Publication of the 16th ACM Web Science Conference, pp. 75–84

    work page 2024

  4. [4]

    Demetis, D. S. (2020). Breaking bad online: A synthesis of the darker sides of social networking sites. European Management Journal, 38(1):33–44

    work page 2020

  5. [5]

    , Jones, B

    Dinakar, K. , Jones, B. , Havasi, C. , Lieberman, H. , and Picard, R. (2012). Common sense reasoning for detection, prevention, and mitigation of cyberbullying. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(3):1–30

    work page 2012

  6. [6]

    S., and Yadav, P

    Goar, V., Yadav, N. S., and Yadav, P. S. (2023). Conversational ai for natural language processing: An review of chatgpt. International Journal on Recent and Innovation Trends in Computing and Communication, 11:109–117

    work page 2023

  7. [7]

    U., Munot, M

    Gongane, V. U., Munot, M. V., and Anuse, A. D. (2022). Detection and moderation of detrimental content on social media platforms: current status and future directions. Social Network Analysis and Mining, 12(1):129

    work page 2022

  8. [8]

    Gorwa, R., Binns, R., and Katzenbach, C. (2020). Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data & Society, 7(1):2053951719897945

    work page 2020

Show all 27 references
  1. [9]

    Haque, M. A. and Li, S. (2024). Exploring chatgpt and its impact on society. AI and Ethics, pp. 1–13. Horta Ribeiro, M., Jhaver, S., Zannettou, S., Blackburn, J., Stringhini, G., De Cristofaro, E., and West, R. (2021). Do platform migrations compromise content moderation? evid...

    2024

  2. [10]

    , AbuHashem, Y

    Kumar, D. , AbuHashem, Y. , and Durumeric, Z. (2023). Watch your language: large language models and content moderation. arXiv preprint arXiv:2309.14517

    2023 arXiv

  3. [11]

    V., Zhang, Y., and Tan, C

    Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., and Tan, C. (2022). Human-ai collaboration via conditional del- egation: A case study of content moderation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–18

    2022

  4. [12]

    Lee, C. S. and Lee, A. Y. (2020). Clinical applications of continual learning machine learning. The Lancet Digital Health, 2(6):e279–e281. Llansó, E. J. (2020). No amount of “ai” in content moderation will solve filtering’s prior -restraint problem. Big Data & Society, 7(1):20...

    2020

  5. [13]

    Lo, L. S. (2023). The clear path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4):102720

    2023

  6. [14]

    Molina, M. D. and Sundar, S. S. (2024). Does distrust in humans predict greater trust in ai? role of individual differences in user responses to content moderation. New Media & Society, 26(6):3638–3656

    2024

  7. [15]

    Paakki, H., Vepsäläinen, H., and Salovaara, A. (2021). Disruptive online communication: How asymmetric trolling -like response strategies steer conversation off the track. Computer Supported Cooperative Work (CSCW), 30(3):425–461

    2021

  8. [16]

    and Schitter, C

    Palan, S. and Schitter, C. (2018). Prolific. ac—a subject pool for online experiments. Journal of behavioral and experimental finance, 17:22–27. 32 Natural Language Processing

    2018

  9. [17]

    , Sayeed, A

    Pusse, F. , Sayeed, A. , and Demberg, V. (2016). Lingoturk: managing crowdsourced tasks for psycholinguistics. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pp. 57–61

    2016

  10. [18]

    Roumeliotis, K. I. and Tselikas, N. D. (2023). Chatgpt and open -ai models: A preliminary review. Future Internet , 15(6):192

    2023

  11. [19]

    P., Danescu-Niculescu-Mizil, C., and Levy, K

    Schluger, C., Chang, J. P., Danescu-Niculescu-Mizil, C., and Levy, K. (2022). Proactive moderation of online discussions: Existing practices and the potential for algorithmic support. Proceedings of the ACM on Human -Computer Interaction, 6(CSCW2):1–27

    2022

  12. [20]

    F., Barbarestani, B., Tufa, W., Vossen, P., and Markov, I

    Schouten, S. F., Barbarestani, B., Tufa, W., Vossen, P., and Markov, I. (2023). Cross-domain toxic spans detection. In International Conference on Applications of Natural Language to Information Systems, pp. 533–545. Springer

    2023

  13. [21]

    P., Hom, G

    Singh, R. P., Hom, G. L., Abramoff, M. D., Campbell, J. P., Chiang, M. F., and others (2020). Current challenges and barriers to real -world artificial intelligence adoption for the healthcare system, provider, and the patient. Translational Vision Science & Technology, 9(2):45–45

    2020

  14. [22]

    Thompson, J. B. (2020). Mediated interaction in the digital age. Theory, Culture & Society, 37(1):3–28

    2020

  15. [23]

    and Derczynski, L

    Vidgen, B. and Derczynski, L. (2020). Directions in abusive language training data, a systematic review: Garbage in, garbage out. Plos one, 15(12):e0243300

    2020

  16. [24]

    Wiegand, M., Ruppenhofer, J., Schmidt, A., and Greenberg, C. (2018). Inducing a lexicon of abusive words–a feature-based approach. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,...

    2018

  17. [25]

    , Ngiam, K

    Yin, J. , Ngiam, K. Y. , and Teo, H. H. (2021). Role of artificial intelligence applications in real -life clinical practice: systematic review. Journal of medical Internet research, 23(4):e25759

    2021

  18. [26]

    and Zhang, L

    Zhang, L. and Zhang, L. (2022). Artificial intelligence for remote sensing data analysis: A review of challenges and opportunities. IEEE Geoscience and Remote Sensing Magazine, 10(2):270–294

    2022

  19. [27]

    , and Xu, R

    Zhu, Q., Lin, Z., Zhang, Y., Sun, J., Li, X., Lin, Q., Dang, Y. , and Xu, R. (2021). Hitsz-hlt at semeval -2021 task 5: Ensemble sequence labeling and span boundary detection for toxic span detection. In Proceedings of the 15th international workshop on semantic evaluation (Se...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.