Understanding and Analyzing Inappropriately Targeting Language in Online Discourse: A Comparative Annotation Study

Baran Barbarestani; Isa Maks; Piek Vossen

REVIEW 3 major objections 4 minor 13 references

Understanding and Analyzing Inappropriately Targeting Language in Online Discourse: A Comparative Annotation Study

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChatGPT over-identifies targeting language in Reddit discussions, labeling 75% of gold-data comments as targeting versus 55% by expert majority, while the study adds four new target categories.

desk verdict A transparent but small annotation study whose central claim about ChatGPT is actually about text-davinci-003 (GPT-3); the new target categories and error analysis are the most useful parts. read the letter →

arxiv 2505.16847 v1 pith:TJ6PFCPZ submitted 2025-05-22 cs.CL

Baran Barbarestani , Isa Maks , Piek Vossen This is my paper · ORCID

classification cs.CL

keywords targetinglanguagehatespeechdetectionannotationinter-annotatoragreementChatGPTRedditcontentmoderationtargetcategories

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that inappropriately targeting language in online conversations is best detected by comparing three annotation routes—trained experts, a crowd, and ChatGPT—rather than trusting any single one. Using 498 Reddit conversation threads drawn from banned subreddits, the authors build an annotation framework that marks whether a comment targets someone, whether that target is inside or outside the thread, and which target category and tokens are involved. The central empirical finding is that ChatGPT over-identifies targeting: it labels 75% of the gold-data comments as targeting while expert majority vote labels only 55%, and its agreement with the experts is lower ($\kappa = 0.40$ at comment level) than crowd-vs-expert agreement ($\kappa = 0.58$). The authors also identify four target categories absent from standard hate speech schemas: social belief, body image, addiction, and socioeconomic status. If correct, the study implies that automated moderation tools built on current large language models will over-block neutral content unless they are calibrated against human judgment.

What carries the argument

The load-bearing mechanism is the three-tier annotation framework: a shared guideline defines 'targeting' as language that inappropriately directs abuse at individuals or groups, distinguishes targets inside versus outside the conversation thread, and asks annotators to mark both a target category and the specific target tokens. Expert annotations are adjudicated by majority vote to form AdjExpert, crowd annotations by majority vote to form AdjCrowd, and ChatGPT receives the same task through a chain of prompts that feed its earlier outputs forward. Agreement among annotators is quantified with Cohen's $\kappa$ at comment and subthread levels, and confusion matrices are used to separate over-identification from under-identification. The framework's role is to make the three annotation sources comparable on the same units, so that differences in labels can be attributed to the annotator type rather than to the task definition.

What would settle it

Re-annotate a random sample of the 39 gold subthreads with an independent panel of, say, nine experts using the same guidelines; if the new panel's majority labels disagree with AdjExpert on more than 25% of comments, or if pairwise agreement stays near 0.58, then the measured over-identification of ChatGPT is an artifact of one particular adjudication rather than a property of the model.

Watch

Extended reading notes

Core claim

The paper's central claim is that ChatGPT, when given the same annotation task as human annotators, cannot substitute for expert judgment in recognizing targeting language: it is more sensitive but less specific, flagging comments as targeting that experts unanimously consider neutral, such as a factual remark about airplane models, and missing cases where the target is a famous individual mentioned indirectly. The claim is supported by a comparison of three annotation sets on 39 gold subthreads adjudicated by expert majority vote: moderate expert-expert agreement ($\kappa = 0.58$ comment-level), lower crowd agreement ($\kappa = 0.36$), ChatGPT-vs-expert agreement of $0.40$, and a confusion-matrix pattern in which ChatGPT flags 75% of gold-data comments as targeting versus 55% for experts. The paper further claims that the standard target-category list is incomplete, and proposes social belief, body image, addiction, and socioeconomic status as categories observed in unanimous 'other' annotations.

Load-bearing premise

The load-bearing premise is that the three experts' majority-vote adjudication (AdjExpert) is a reliable ground truth; but the experts only reach moderate pairwise agreement ($\kappa = 0.58$ at comment level) and disagree on whole categories, so if that reference standard is unstable, the ChatGPT and crowd comparisons lose their benchmark.

Editorial extensions

If this is right

If ChatGPT over-identifies targeting as described, deploying current large language model annotation directly in content moderation will produce many false positives, so human review or threshold calibration remains necessary.
Because the four new categories arose from unanimous 'other' annotations, standard hate speech taxonomies miss real targeting categories; adding them should improve coverage in detection systems.
Subthread-level agreement is higher than comment-level agreement for both experts and crowd, suggesting that conversation-level aggregation is a more reliable unit for moderation decisions than isolated comments.
The gold-data subset, though small, can serve as a benchmark for comparing future automated annotation systems against human adjudication.
ChatGPT's near-zero agreement on inside-versus-outside targeting suggests that structural understanding of conversation context, not just toxicity detection, is the main bottleneck for AI annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the over-identification pattern implies that precision-oriented metrics for large language model moderation should be reported alongside sensitivity; a system that flags 75% of content will overwhelm human review queues even if it catches more true positives.
The new categories suggest a testable extension: building lexicon or few-shot classifiers for social belief, body image, addiction, and socioeconomic status and measuring whether they recover comments that current toxicity filters mark as neutral.
Because the expert gold standard itself rests on a three-person majority with only moderate pairwise agreement, a practical extension would be to collect soft labels or disagreement scores from a larger panel and treat annotation uncertainty as a first-class output.
The fact that ChatGPT interpreted titles and first comments as non-targeting hints at a positional bias; a direct follow-up could swap thread order or prompt structure to see whether over-identification shifts with position.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth knowing: the paper's headline finding—ChatGPT over-identifies targeting language—is actually about text-davinci-003, a GPT-3 engine. The abstract and intro say ChatGPT, but Section 3.5 names the real model. That oversells the referent. The genuinely new content is the qualitative discovery of four target categories (social belief, body image, addiction, socioeconomic status) and a careful error analysis of expert disagreements. That part is useful.

What the paper does well: it's transparent with kappa numbers and confusion matrices, it distinguishes comment-level vs subthread-level agreement, and it gives concrete examples of why experts disagree. The comparison of expert, crowd, and LLM annotations is a reasonable setup, and the finding that the LLM over-flags (75% vs 55% targeting in gold data) is visible in the tables. The authors also acknowledge the small dataset and limited language scope in the Limitations section, which is honest.

Soft spots. The ChatGPT/GPT-3 naming is the biggest one; it's not a nit because the central claim is tied to the engine. If they rerun on a chat-tuned model, fine; otherwise they should consistently say 'GPT-3 (text-davinci-003)' and stop saying ChatGPT. Second, the dataset and code aren't shipped, despite being promised as a 'benchmark data set'—that's a reproducibility gap. Third, the gold standard is only 39 subthreads, and expert agreement is moderate (comment-level kappa 0.58). That doesn't kill the comparative finding, but it does mean the ground truth is fragile. The crowd tables also look unpolished (age 120-130, 'CONSENT REVOKED' entries)—minor data-cleaning issue, but it suggests the appendix wasn't fully checked.

The new target categories come from expert interpretations of 'other', and they're plausibly defined, but the sample is small, so treat them as tentative. I'm not worried about circularity—building on their own TRAC corpus is normal, and they cite prior work.

Who it's for: people working on hate speech annotation, especially comparing LLM vs human annotation. It deserves peer review because it's a legitimate empirical study with a clear method and mostly honest reporting. But the model-name issue and missing data require revision before acceptance.

Referee Report

3 major / 4 minor

Summary. The paper presents a comparative annotation study of 'inappropriately targeting language' in Reddit subthreads from banned communities, using expert annotators, crowd annotators, and an LLM. It reports moderate expert inter-annotator agreement (comment-level Cohen's kappa 0.58), lower agreement for the LLM (0.40 against the adjudicated expert set), and a tendency of the LLM to over-identify comments as targeting (188 LLM-targeting annotations vs. 133.67 expert average). It also claims to uncover new target categories (social belief, body image, addiction, socioeconomic status) and discusses sources of expert disagreement.

Significance. If the central findings held for the model actually named, the study would be a useful caution against substituting LLM annotations for expert judgment in detecting targeted abuse, and it would extend prior hate-speech taxonomies. Strengths of the paper include transparent reporting of kappa scores, confusion matrices for expert pairs, and detailed annotation prompts in the appendix. The dataset, though small, is a concrete resource. However, the significance is materially weakened by the mismatch between the model used (text-davinci-003) and the model named throughout the paper (ChatGPT), and by the lack of statistical support for the new-category claims.

major comments (3)

[Abstract, Section 3.5, Section 4.2] The paper's central claim is stated as 'ChatGPT tends to over-identify comments as targeting' (Section 4.2), but the only LLM evaluated is OpenAI's GPT-3 text-davinci-003 engine, as explicitly stated in Section 3.5 ('We integrated OpenAI's GPT-3 language model into our annotation process, using the text-davinci-003 engine'). Text-davinci-003 is not ChatGPT; the abstract, introduction, 'Main contributions,' and conclusion all attribute the results to ChatGPT. This mismatch directly affects the scope and validity of the headline finding: the reported statistics may hold for text-davinci-003, but they do not support claims about ChatGPT. The authors should either rename the model throughout to 'GPT-3 (text-davinci-003)' and adjust the title and abstract, or rerun the annotation pipeline on an actual ChatGPT model and report those results.
[Section 4.1, Table 5] The paper lists 'new targeting categories such as social belief, body image, addiction, and socioeconomic status' as a main contribution, but these categories appear only in a single sentence in Section 4.1. They are absent from the annotation schema in Section 3.2, from the prompts in Appendix E, and from Table 5's category counts. No statistics, examples, inter-annotator reliability scores, or case counts are provided for these alleged new categories. As written, this claim is unsupported. The authors need to specify how these categories were derived, how often they occur in the gold data, and whether annotators can reliably identify them.
[Sections 3.4, 3.6, Table 4, Figure 1] The comparative evaluation treats the expert majority-vote adjudication (AdjExpert) as ground truth, but expert pairwise agreement is only moderate (comment-level kappa 0.58; Table 1), and Section 4.1 documents substantial substantive disagreements, including whether 'Sunday Gunday: Self-Defense' is targeting at all. With such a noisy reference, the raw kappa differences (0.40 for AdjExpert vs. ChatGPT, 0.58 for AdjExpert vs. AdjCrowd) and the over-identification claim in Figure 1 rest on a fragile foundation. Additionally, Table 5 reports counts (e.g., 188 vs. 133.67) without the number of comments in the gold set, so the statement that 'ChatGPT marks 75% of cases as targeting' cannot be verified. The authors should report the comment-level denominator, provide proportions with confidence intervals, and present agreement against each individual expert as well as against the majority-vote adjudication.

minor comments (4)

[Appendix D] Several crowd-annotator demographic tables contain implausible or pipeline-artifact entries: Table D.9 lists an age range of 120-130, and tables D.6-D.14 list 'CONSENT REVOKED' as a fluent language, primary language, nationality, sex, ethnicity, student status, employment status, and country of residence, with 'DATA EXPIRED' appearing as a student status and employment status. These should be cleaned or explicitly reported as missing-data placeholders.
[Section 3.5.3] The sentence 'These tokens, extracted from targeting comments or titles provided to ChatGPT along with their associated categories (as described in 3.5.3, offer additional context...' contains a self-reference to Section 3.5.3, but the target categories were described in Section 3.5.2. Please correct the cross-reference.
[Figures 1-4] Section 4.2 heavily relies on Figures 1 and 2 (confusion matrices for ChatGPT vs. AdjExpert), but these figures are not included in the manuscript text; only captions/placeholders appear. Please ensure the actual figures are embedded and that their content is consistent with the reported counts.
[Throughout] The manuscript contains several typos and inconsistencies: 'inapropriately' in the Highlights, 'CONSENTREV OKED' in Table D.8, 'Dann' in Example 4.3, and inconsistent use of 'ChatGPT' vs. 'GPT-3' in the Highlights (which correctly mentions GPT-3 text-davinci-003) and the abstract (which says ChatGPT). A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an observational comparison of annotations, and the central over-identification finding is computed from new annotation data rather than reduced to its inputs.

full rationale

The paper does not derive a mathematical result or fit parameters, so most circularity patterns do not apply. The central claim that the LLM over-identifies targeting language is based on comparing ChatGPT-generated annotations with expert adjudications (Section 4.2, Figures 1-2, Table 4), which is an independent measurement rather than a construction. The only self-citations are the use of the authors' prior dataset [9] and annotator-selection method [11]; these provide inputs to the study but do not by themselves determine the reported agreement scores or the over-identification conclusion. The newly named target categories (social belief, body image, addiction, socioeconomic status) are a post-hoc relabeling of expert-unanimous 'other target category' instances, not a prediction derived from the annotation framework, so no definitional circularity is present. The mismatch between 'ChatGPT' in the title/abstract and 'text-davinci-003' in Section 3.5 is a correctness and scope concern about which model was actually evaluated, but it is not a circularity issue because the empirical comparison remains a comparison of annotator outputs. Overall, the derivation chain is observational and self-contained: expert annotations, crowd annotations, and LLM annotations are compared against each other, and the paper's conclusions follow from those comparisons rather than being equivalent to any input assumption.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The study introduces no new theoretical entities or parameters in the physics sense. The free parameters are the hand-chosen dataset construction thresholds, and the axioms are the domain assumptions that the inherited corpus, adjudicated gold labels, and kappa metric are valid. No machine-checked proofs or released data support these assumptions.

free parameters (2)

Subthread selection thresholds (min 3, max 17 comments; 51-1,276 tokens; max 38 tokens per comment) = 3-17 comments, 51-1,276 tokens, 38 tokens per comment
Hand-chosen filtering criteria in Section 3.1 that define which Reddit conversations enter the dataset and therefore shape all subsequent annotation and agreement statistics.
Toxicity sampling cutoff (top 400 toxic, 98 non-toxic subthreads) = 400 and 98 subthreads
Hand-selected numbers in Section 3.1 for balancing toxic and non-toxic threads; no justification is given for these exact counts.

assumptions (3)

domain assumption The prior corpus from reference [9] provides a valid sampling frame of English Reddit threads from banned subreddits.
Invoked in Section 3.1; the entire dataset is inherited from the authors' own prior work without external validation of representativeness.
domain assumption Majority-vote adjudication yields a reliable gold standard for targeting labels.
Used in Sections 3.3-3.4; moderate expert kappa (0.58) makes this assumption load-bearing for all comparisons.
standard math Cohen's Kappa is an appropriate agreement metric for these categorical annotations.
Used throughout Section 3.6; standard but sensitive to prevalence, which the paper itself notes as a factor in the subthread versus comment discrepancy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding and Analyzing Inappropriately Targeting Language in Online Discourse: A Comparative Annotation Study." pith.science (2026). https://pith.science/paper/TJ6PFCPZ

@misc{pith2026250516847,
  author       = {Pith},
  title        = {Pith review of: Understanding and Analyzing Inappropriately Targeting Language in Online Discourse: A Comparative Annotation Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TJ6PFCPZ}},
  note         = {Machine review of arXiv:2505.16847}
}

read the original abstract

This paper introduces a method for detecting inappropriately targeting language in online conversations by integrating crowd and expert annotations with ChatGPT. We focus on English conversation threads from Reddit, examining comments that target individuals or groups. Our approach involves a comprehensive annotation framework that labels a diverse data set for various target categories and specific target words within the conversational context. We perform a comparative analysis of annotations from human experts, crowd annotators, and ChatGPT, revealing strengths and limitations of each method in recognizing both explicit hate speech and subtler discriminatory language. Our findings highlight the significant role of contextual factors in identifying hate speech and uncover new categories of targeting, such as social belief and body image. We also address the challenges and subjective judgments involved in annotation and the limitations of ChatGPT in grasping nuanced language. This study provides insights for improving automated content moderation strategies to enhance online safety and inclusivity.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages

[1]

Countering Online Hate Speech: An NLP Perspective

M. Chaudhary, C. Saxena, H. Meng, Countering online hate speech: An nlp perspective, arXiv preprint arXiv:2109.02941 (2021)

work page Pith review arXiv 2021
[2]

M. S. Jahan, M. Oussalah, A systematic review of hate speech automatic de - tection using natural language processing., Neurocomputing (2023) 126232

work page 2023
[3]

Fanton, H

M. Fanton, H. Bonaldi, S. S. Tekiroglu, M. Guerini, Human -in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech, arXiv preprint arXiv:2107.08720 (2021)

arXiv 2021
[4]

Lemmens, I

J. Lemmens, I. Markov, W. Daelemans, Improving hate speech type and target detection with hateful metaphor features, in: Proceedings of the fourth work - shop on NLP for internet freedom: censorship, disinformation, and propaganda, 2021, pp. 7–16

work page 2021
[5]

ElSherief, V

M. ElSherief, V. Kulkarni, D. Nguyen, W. Y. Wang, E. Belding, Hate lingo: A target-based linguistic analysis of hate speech in social media, in: Proceedings of the international AAAI conference on web and social media, Vol. 12, 2018

work page 2018
[6]

Zampieri, S

M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, Pre - dicting the type and target of offensive posts in social media, arXiv preprint arXiv:1902.09666 (2019)

arXiv 2019
[7]

ElSherief, S

M. ElSherief, S. Nilizadeh, D. Nguyen, G. Vigna, E. Belding, Peer to peer hate: Hate speech instigators and their targets, in: Proceedings of the International AAAI Conference on Web and Social Media, Vol. 12, 2018

work page 2018
[8]

Vidgen, D

B. Vidgen, D. Nguyen, H. Margetts, P. Rossini, R. Tromble, Introducing cad: the contextual abuse dataset (2021)

work page 2021

Show all 13 references

[9]

Barbarestani, I

B. Barbarestani, I. Maks, P. Vossen, Content moderation in online platforms: A study of annotation methods for inappropriate language, in: Proceedings of the Third Workshop on Threat, Aggression and Cyberbullying (TRAC 2024), 2024. 34

2024
[10]

W. T. Tufa, I. Markov, P. Vossen, The constant in hate: Analyzing toxicity in reddit across topics and languages (2024). arXiv:2404.18726

2024 arXiv
[11]

Barbarestani, I

B. Barbarestani, I. Maks, P. Vossen, Annotating targets of toxic language at the span level, in: Proceedings of the Third Workshop on Threat, Aggression and Cyberbullying (TRAC 2022), 2022, pp. 43 –51

2022
[12]

Pusse, A

F. Pusse, A. Sayeed, V. Demberg, Lingoturk: managing crowdsourced tasks for psycholinguistics, in: Proceedings of the 2016 Conference of the North Ameri - can Chapter of the Association for Computational Linguistics: Demonstrations, 2016, pp. 57–61

2016
[13]

Palan, C

S. Palan, C. Schitter, Prolific. ac—a subject pool for online experiments, Journal of Behavioral and Experimental Finance 17 (2018) 22 –27

2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Countering Online Hate Speech: An NLP Perspective

M. Chaudhary, C. Saxena, H. Meng, Countering online hate speech: An nlp perspective, arXiv preprint arXiv:2109.02941 (2021)

work page Pith review arXiv 2021

[2] [2]

M. S. Jahan, M. Oussalah, A systematic review of hate speech automatic de - tection using natural language processing., Neurocomputing (2023) 126232

work page 2023

[3] [3]

Fanton, H

M. Fanton, H. Bonaldi, S. S. Tekiroglu, M. Guerini, Human -in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech, arXiv preprint arXiv:2107.08720 (2021)

arXiv 2021

[4] [4]

Lemmens, I

J. Lemmens, I. Markov, W. Daelemans, Improving hate speech type and target detection with hateful metaphor features, in: Proceedings of the fourth work - shop on NLP for internet freedom: censorship, disinformation, and propaganda, 2021, pp. 7–16

work page 2021

[5] [5]

ElSherief, V

M. ElSherief, V. Kulkarni, D. Nguyen, W. Y. Wang, E. Belding, Hate lingo: A target-based linguistic analysis of hate speech in social media, in: Proceedings of the international AAAI conference on web and social media, Vol. 12, 2018

work page 2018

[6] [6]

Zampieri, S

M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, Pre - dicting the type and target of offensive posts in social media, arXiv preprint arXiv:1902.09666 (2019)

arXiv 2019

[7] [7]

ElSherief, S

M. ElSherief, S. Nilizadeh, D. Nguyen, G. Vigna, E. Belding, Peer to peer hate: Hate speech instigators and their targets, in: Proceedings of the International AAAI Conference on Web and Social Media, Vol. 12, 2018

work page 2018

[8] [8]

Vidgen, D

B. Vidgen, D. Nguyen, H. Margetts, P. Rossini, R. Tromble, Introducing cad: the contextual abuse dataset (2021)

work page 2021