REVIEW 4 major objections 6 minor 17 references
The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Emoji swaps lower tweet offensiveness without changing meaning, human raters say
desk verdict Useful descriptive analysis of emoji roles in offensive tweets, but the pipeline's claimed emoji-only mechanism is unverified because the prompt allows text rewriting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a four-role taxonomy of emoji function in offensive tweets—offensive in itself, intensifies offense, mitigates offense, unrelated to offense—paired with a four-step LLM pipeline. The pipeline first classifies whether a tweet is offensive and which of five offense types it belongs to, then assigns every emoji a role using the taxonomy and prior distributional findings, then recommends surrogate emojis only for emojis labeled offensive-in-themselves or intensifying, and finally condenses the justifications into a user-facing summary. The role taxonomy is what makes replacement selective rather than destructive: it directs the LLM to leave mitigating and unrelated emojis untouched, which is why semantic measures remain stable while offensiveness falls.
What would settle it
Take a random sample of the 9,285 offensive tweets, ask a group of human annotators to label every emoji with the paper's four-role taxonomy, and measure agreement with GPT-4's Step-2 labels. If agreement is near chance, the pipeline is targeting the wrong emojis and its observed offensiveness drop would not generalize. A second falsifier: run the pipeline only on tweets human-raters scored 5 (extremely offensive); the paper itself reports no offensiveness reduction there, so the central claim should be scoped to mild and moderate offenses, and a replication restricted to score-5 tweets would be expected to show zero effect.
Extended reading notes
Core claim
Emojis carry offense through at least four distinct roles—being the offense itself, intensifying the offense, mitigating it, or being unrelated to it—and these roles are predictable enough from context that a prompted LLM can use them to selectively replace harmful emojis. In a within-subject human experiment, the pipeline's targeted replacements reduced perceived offensiveness from 3.00 to 2.58 (p<0.05) with no statistically significant change in sentiment, arousal, clarity, or fluency. A direct-prompting baseline that simply asks the LLM to replace offensive emojis without the role taxonomy produced only a small, insignificant drop (3.00 to 2.94), indicating that the taxonomy-driven stepwise procedure is what carries the effect. The paper also documents that emoji-offense associations are type-specific: body-part emojis cluster in sexual content, dehumanizing emojis in personal and political attacks, and the pipeline's substitutions successfully reduce those specific markers even when the overall offensiveness score does not move. Removing emojis only changes perceived offensiveness for mildly offensive tweets; when verbal abuse alone earns a score of 4 or 5, emoji replacement has little effect.
Load-bearing premise
The pipeline's second step assumes that GPT-4 can reliably tell, from the tweet text alone, which emoji is offensive by itself, which intensifies the attack, which softens it, and which is irrelevant, but the paper never checks these role labels against human judgment.
Editorial extensions
If this is right
- Content moderation systems should treat emojis as first-class carriers of offensive meaning rather than as noise, and can leverage role taxonomies to target them selectively.
- An LLM given only a generic instruction to reduce offensiveness through emoji replacement fails to lower offense scores, whereas the same model guided by an explicit role taxonomy succeeds, showing that prompt structure, not model capability alone, drives the outcome.
- Emoji moderation is most effective for mildly offensive content; for tweets already rated 4-5 on offensiveness, replacing emojis does not move the needle, so deployment should prioritize borderline cases.
- The effect is offense-type specific: the pipeline reduces dehumanizing emojis in personal and political attacks (by 12.8% and 22.5%) and body-symbol emojis in sexual content (by 15.0%), without inflating sarcasm.
- Replacing emojis that carry implicit offense (body symbols, dehumanizing objects, sarcastic faces) can preserve the semantic intent of a tweet while lowering its perceived harm, offering a less invasive alternative to full-text rewriting.
Reading between the lines
- The entire pipeline hinges on GPT-4's Step-2 role labels, and since the paper never validates those labels against human judgments, the 3.00-to-2.58 effect could be an artifact of the model's own consistent but incorrect role assignments; a human-annotation check of emoji roles on a random sample would settle this.
- Because the paper's own limitation section notes cultural and individual variation in emoji interpretation, the pipeline is likely to be far less effective outside U.S. English contexts; a natural extension is to build role taxonomies per language or culture rather than one global taxonomy.
- The finding that emoji replacement works only for mild offense suggests a deployment strategy: platforms could use this pipeline as a low-stakes nudge for borderline tweets while leaving high-severity content to the ordinary removal or flagging process.
- The paper's taxonomy could be imported into hate-speech datasets to create emoji-role annotations at scale, which would let future work train smaller, faster classifiers to do the replacement without calling an LLM for every tweet.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the role of emojis in offensive tweets and proposes a multi-step LLM pipeline that selectively replaces emojis judged to be offensive in themselves or to intensify offense, while preserving the tweet's semantics. Using a collected dataset of 9,285 offensive tweets, the authors analyze emoji roles and distributions across offense types, then run a human evaluation (20 annotators, 200 tweets in three conditions) comparing original tweets, pipeline-processed tweets, and a direct-prompting baseline. The headline result is that the pipeline reduces perceived offensiveness from 3.00 to 2.58 on a 1–5 scale with p<0.05, with no notable changes in semantic measures. The paper also reports heterogeneous effects across offense types and provides case studies and prompts in the appendix.
Significance. If the main result holds, the paper is a useful contribution to an under-studied area: it provides descriptive evidence that emojis have distinct functional roles in offensive content and proposes a more conservative moderation alternative than full-tweet rewriting. The descriptive findings on emoji distributions by offense type and role (Figures 1 and Tables 1–3) are valuable for future research on emoji semantics and content moderation. However, the central causal claim—that the offensiveness reduction is specifically due to emoji substitution—is not yet verified, because the pipeline does not demonstrably leave non-emoji text untouched. In addition, the statistical reporting is too thin to support the strength of the conclusions, and the paper does not release data or code, which limits reproducibility. These issues are fixable, but they are load-bearing for the paper's main contribution.
major comments (4)
- [Section 4.2, Step 3; Appendix B.2] The claim that the pipeline reduces offensiveness through emoji substitution alone is not supported, because the Step 3 prompt instructs the LLM to output 'the full tweet after all necessary replacements' without requiring that non-emoji tokens be preserved verbatim, and the direct-prompting baseline has the same ambiguity. The only manipulation check in Section 4.4 (Table 9) compares emoji frequency distributions; no token-level diff or edit-distance analysis is reported. Consequently, the offensiveness drop from 3.00 to 2.58 in Table 5 could be partially or wholly caused by incidental text rewriting (e.g., softening profanity or rephrasing), not by emoji replacement. Because the paper's stated contribution is targeted emoji moderation, this missing control is load-bearing; the authors should either constrain the prompt to forbid non-emoji edits and verify with a diff, or add a text-only rewriting condition to isolate the emoji effect.
- [Section 4.5, Tables 5 and 6] The statistical evidence for the headline result is underreported. The paper reports only mean scores and a 'p<0.05 (paired t-test)' asterisk, with no standard deviations, confidence intervals, effect sizes, or inter-annotator agreement (e.g., Krippendorff's alpha or Cohen's kappa) despite each tweet being rated by two annotators. The analysis also lacks any correction for multiple comparisons: Table 6 tests five offense types across roughly ten variables (about 50 pairwise tests), so the significant offensiveness deltas (racial, sexual, violence) and the body-symbol/dehumanization/extra-meaning deltas are expected by chance at the 0.05 level. Without these details, the claim that the pipeline 'significantly reduces offensiveness' is not adequately established.
- [Section 3.1 and Section 4.2, Step 2] The GPT-4-based emoji role labels are used both as the descriptive analysis (Figure 1, Table 1) and as prior knowledge injected into the pipeline prompt (the lists of 'Emojis Often Associated with Specific Offense Types' and 'Popular Emojis by Typical Functionality'). The paper never validates these labels against human judgments, so the pipeline may be systematically replacing or retaining emojis based on noisy or biased role assignments. This matters because Step 2 determines which emojis are candidates for replacement; if the role labels are unreliable, the observed offensiveness reduction may not generalize to other datasets or models. A small human-annotation reliability study for the four-way role taxonomy would address this.
- [Section 4.5, Evaluation Design] The within-subject design presents the same tweet in original, pipeline-processed, and baseline-processed forms to the same annotators. Although the paper randomizes presentation order, it does not describe counterbalancing or whether annotators could recognize repeated content; repeated exposure may induce demand characteristics and inflate the apparent effect. At minimum, the authors should report whether they used a Latin-square design or between-subject split, and analyze order effects.
minor comments (6)
- [Tables 1, 2, 3, and 9] These tables are empty in the manuscript text as provided because the emoji glyphs are missing, so the reader cannot verify the actual emoji lists; please include Unicode names or a rendered figure.
- [Section 3.3, Table 3] The thresholds for 'High/Moderate/Low frequency' (30% and 20%) are arbitrary and not justified; since these categories are later fed into the Step 2 prompt, the threshold choice is a free parameter that should be justified or examined in a sensitivity analysis.
- [Section 4.5, Experimental Setup] The description of the annotation procedure is ambiguous: 'Each annotator assesses 60 tweets ... consisting of 20 original tweets, 20 versions ...' could mean each annotator sees triples of the same 20 tweets, or 60 distinct items; please clarify the assignment and the unit of the paired t-test (tweet-level or annotator-level).
- [Appendix A.2, Extra Meaning question] The question 'Do you agree ... meanings of all emojis ... disclosed by the text' uses a two-point Disagree/Agree scale; the 'Extra Meaning (% Yes)' in Table 5 presumably reflects agreement, but the mapping is not stated.
- [Section 7, Limitations] The Limitations section acknowledges cultural and LLM-bias concerns but does not mention the possibility that the LLM rewrites non-emoji text; adding this limitation and a corresponding analysis would strengthen the paper.
- [References] Several references lack arXiv IDs or DOIs (e.g., Davidson et al. 2019, Barbieri et al. 2020); please standardize the bibliography.
Circularity Check
Minor circularity in the qualitative emoji-removal check; the central human-evaluation claim is independent and not circular.
-
fitted input called prediction
[Section 4.2 (Step 2), Appendix B.2; cf. Section 4.4]
"Crucially, we incorporate findings from our analysis (Section 3), such as common emojis for each offense type and role, and their general offensive frequency, into the prompt to guide the LLM towards more contextually accurate role identification."
The Step 2 prompt in Appendix B.2 literally feeds the Section 3 outputs back into the pipeline: it lists 'Emojis Often Associated with Specific Offense Types', 'Popular Emojis by Typical Functionality', and 'Emoji Frequency in Offensive Tweets', and Step 3 replaces only emojis classified as roles 1 or 2. Therefore, the later claim in Section 4.4 that 'the multi-step pipeline successfully eliminates most item-based emojis frequently used for offense' is, for those emojis, true by construction: the prompt already designated them as targets. This makes the distributional check a partial restatement of its own inputs rather than an independent confirmation of the Section 3 taxonomy.
full rationale
The paper's main claim is that the multi-step pipeline reduces perceived offensiveness (3.00 to 2.58) while preserving meaning, and this is supported by a within-subject human evaluation with randomized presentation of original, pipeline-processed, and direct-prompting tweets. Human offensiveness and semantic-preservation judgments are external to the LLM prompts and are not determined by the Section 3 emoji lists, so the core contribution is not circular. The mild circularity is confined to the qualitative emoji-distribution check: the pipeline prompts incorporate the Section 3 findings, so observing that the pipeline removes the very emojis those findings flagged is expected by construction. Self-citations in the paper are supportive rather than load-bearing, and no uniqueness theorem or imported ansatz is used to force the paper's conclusions. The skeptic's concern that Step 3 may also rewrite non-emoji text is a correctness/robustness risk, not a circularity, because even if true it would weaken the mechanism attribution without making the evaluation logically dependent on the paper's own outputs.
Assumptions & free parameters
free parameters (3)
- RoBERTa offensive probability threshold =
0.5
- Emoji frequency category thresholds =
20% and 30%
- BERTopic minimum cluster size =
20 documents
assumptions (4)
- domain assumption RoBERTa and GPT-4 accurately detect offensive content in English tweets.
- domain assumption The four-category emoji role taxonomy is valid and exhaustive.
- domain assumption Human 1-5 offensiveness ratings are interval-scale and comparable across annotators and conditions.
- domain assumption The 2019 English Twitter corpus is representative of the claimed phenomenon.
Cite this review
Pith. "Pith review of The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation." pith.science (2026). https://pith.science/paper/YNXGNYKG
@misc{pith2026250600583,
author = {Pith},
title = {Pith review of: The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YNXGNYKG}},
note = {Machine review of arXiv:2506.00583}
}
read the original abstract
Social media platforms have become central to modern communication, yet they also harbor offensive content that challenges platform safety and inclusivity. While prior research has primarily focused on textual indicators of offense, the role of emojis, ubiquitous visual elements in online discourse, remains underexplored. Emojis, despite being rarely offensive in isolation, can acquire harmful meanings through symbolic associations, sarcasm, and contextual misuse. In this work, we systematically examine emoji contributions to offensive Twitter messages, analyzing their distribution across offense categories and how users exploit emoji ambiguity. To address this, we propose an LLM-powered, multi-step moderation pipeline that selectively replaces harmful emojis while preserving the tweet's semantic intent. Human evaluations confirm our approach effectively reduces perceived offensiveness without sacrificing meaning. Our analysis also reveals heterogeneous effects across offense types, offering nuanced insights for online communication and emoji moderation.
Figures
Reference graph
Works this paper leans on
-
[1]
Sexual Content and Gender Issues: In- cludes sexual harassment, gender discrimina- tion, body shaming, objectification, gender- based insults, and derogation
-
[2]
In the past month, how often do you use so- cial media platforms (e.g., Twitter, Reddit, Instagram, TikTok, Messenger)? Please select the option that best describes the fre- quency. • Never: Never in the past month • Rarely: Fewer than once a week • Sometimes: Two or three times a week • Often: More than three times a week, but not everyday • Always: Almo...
-
[3]
In the past month, how often do you see content that includes emojis on social me- dia platforms? Please select the option that best describes the frequency. • Never: Never in the past month • Rarely: Fewer than once a week • Sometimes: Two or three times a week • Often: More than three times a week, but not everyday • Always: Almost everyday
-
[4]
People from [Group X] are lazy and don’t want to work
In the past month, how often do you use emojis yourself when engaging on social media platforms? Please select the option that best describes the frequency. • Never: Never in the past month • Rarely: Fewer than once a week • Sometimes: Two or three times a week • Often: More than three times a week, but not everyday • Always: Almost everyday Guidelines fo...
-
[5]
Violence and Abuse: Includes topics related to physical or verbal abuse, violence, threats, and aggressive behaviors. Your Response Should Include: • Whether the tweet is offensive (Yes/No). • If offensive, the offense category (e.g., Sexual Content and Gender Issues). • A brief justification for the chosen category. Examples: • Tweet: @sinnersworldxxx RT...
-
[8]
Personal Attacks and Disrespect: Ranges from direct insults to subtle disre- spect/derogation targeting individuals/groups based on personal characteristics
-
[9]
Racial and Ethnic Offense: Includes racial slurs, ethnic stereotyping, and discrimina- tion/prejudice based on race or ethnicity. Offensive Type Topics Personal Attacks and Disrespect 0_Personal Confrontations and Profanity, 1_Explicit Con- tent Solicitation, 3_Offensive Language Usage, 6_Casual Slang and Swearing, 9_Offensive Language and Slang Us- age, ...
-
[10]
Political and Social Issues: Encompasses po- litical attacks, social discrimination, harass- ment, and aggression over political views or social status
Show all 17 references
-
[15]
• Assign functionality category/categories (1- 4)
Emoji isnot directly relatedto the offense Use the following data (emojis based on pro- vided list) to inform your analysis: Emojis Often Associated with Specific Of- fense Types: • Sexual Content: , , , , , , , , , , , , , , , , , , , , , • Personal Attacks: , , , , , , , , ,...
-
[16]
Emoji represents theoffense itself
-
[17]
Emojiintensifiesthe offense
-
[18]
Emojimitigatesthe offense
-
[19]
Emojis categorized as3 (Miti- gates)or4 (Unrelated)should be kept
Emoji isnot directly relatedto the offense Your task is to recommend replacements only for emojis categorized as1 (Offense Itself)or2 (In- tensifies Offense). Emojis categorized as3 (Miti- gates)or4 (Unrelated)should be kept. For emojis needing replacement: • Suggest a replace...
-
[20]
Your Response Should Include: • A summary paragraph integrating the offense type, the functionality of the replaced emojis, and the reason for their replacements
Unrelated to Offense We only replaced emojis categorized as1or2. Your Response Should Include: • A summary paragraph integrating the offense type, the functionality of the replaced emojis, and the reason for their replacements. Inputs Provided: • Original Tweet:{tweet} • Emoji...
-
[2015]
blissfully happy
The evolution of cognitive bias.The handbook of evolutionary psychology, pages 724–746. Tianran Hu, Han Guo, Hao Sun, Thuy-vy Nguyen, and Jiebo Luo. 2017. Spice up your chat: the intentions and sentiment effects of using emojis. InICWSM 2017. Fan Huang, Haewoon Kwak, and Jisun...
2017 arXiv
-
[2016]
InMobileHCI 2016
Sender-intended functions of emojis in us messaging. InMobileHCI 2016. Thomas Davidson, Debasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias in hate speech and abusive language detection datasets.arXiv preprint arXiv:1905.12516. Thomas Davidson, Dana Warmsley, Michael...
2016 arXiv
-
[2019]
Ruiqi Zhong, Yanda Chen, Desmond Patton, Charlotte Selous, and Kathy McKeown
Predicting the type and target of of- fensive posts in social media.arXiv preprint arXiv:1902.09666. Ruiqi Zhong, Yanda Chen, Desmond Patton, Charlotte Selous, and Kathy McKeown. 2019. Detecting and reducing bias in a high stakes domain.arXiv preprint arXiv:1908.11474. Li Zhou...
1902 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.