{"id":"419a675f-478e-4aac-9af0-848f9be89c1a","arxiv_id":"2501.11241","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-4o systematically overestimates the likelihood that emojis are used ironically, correlating only weakly with human-perceived scores from the Ciron dataset.","lead":"This paper compares how GPT-4o and humans interpret irony in emojis, using a Chinese social media dataset. It finds that GPT-4o assigns higher irony scores on average, with only a weak correlation to human judgments, and that specifying an older age in the prompt lowers the model's scores.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The overestimation claim compares inverse conditionals: GPT-4o rates P(use emoji | ironic intent) while Ciron human scores estimate P(ironic intent | emoji), so the reported gap may be an artifact.","rationale":"The reader's verdict of CONDITIONAL is reasonable, but the specific weakness identified — the transfer of post-level irony ratings to individual emojis — is only part of a larger construct-validity problem. The more fundamental issue is that the GPT-4o prompt asks for a conditional likelihood P(use emoji | ironic intent), whereas the human benchmark is derived from the inverse conditional E[irony | emoji appears], as computed in Equation (1). These quantities are related by Bayes' rule, and the highly skewed emoji frequency distribution in Ciron means the reported systematic overestimation could arise even if GPT-4o perfectly matched human usage patterns. This is not a problem of statistical testing but of comparing two different constructs. The paper's own worked example with the smirk emoji illustrates the mismatch: the model reasons about potential ironic uses, while the human score reflects actual usage frequencies in posts. A matched human likelihood survey using the same prompt would settle whether the overestimation is genuine. The paper deserves credit for using a public dataset, transparent statistical tests, and candid examples, and the demographic analysis, though imperfect, is secondary. Because the central claim is unsupported without such a matched comparison, the CONDITIONAL verdict stands, provided the condition is understood as requiring a direct human-model comparison on the same task rather than merely a re-annotation of emojis within posts.","tokens_in":6717,"tokens_out":11026,"duration_ms":107968,"concrete_test":"Run a human survey (N ≥ 50 per emoji, e.g., on Prolific) in which participants rate the same 82 emojis on the exact GPT-4o prompt and 11-point scale. Aggregate per emoji (mean human likelihood rating) and repeat the Wilcoxon signed-rank and Spearman analyses against GPT-4o's ratings. If the significant overestimation and weak correlation persist on this matched task, the paper's conclusion survives; if the gap shrinks or reverses, the original finding is an artifact of comparing a conditional likelihood judgment against a usage-based posterior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison is not between like quantities. Equation (1) defines the human irony score S(e) as the mean post-level irony rating R(p) over posts containing emoji e, i.e., an estimate of E[irony | emoji appears]. The model prompt asks GPT-4o to 'rate your likelihood of using this emoji if your intention is to express irony,' which elicits P(use e | ironic intent), the inverse conditional. By Bayes' rule, E[I|e] = P(e|I) * P(I) / P(e). Because emoji frequencies in Ciron are highly skewed (82 emojis over about 3,000 posts), an emoji can have high P(e|I) as a candidate for ironic use while having low E[I|e] because it rarely appears in ironic posts. The Wilcoxon signed-rank test on these two different quantities does not establish that GPT-4o 'systematically overestimates' human perception; it may merely reflect the direction of the conditionals and base-rate skew. The paper's own example supports this reading: GPT-4o justifies a high rating for the smirk emoji by describing potential ironic contexts, while Ciron posts containing that emoji are mostly non-ironic. The transfer assumption flagged by the reader is one facet of this mismatch; even with perfect emoji-level human annotations, the model would still be answering a different question unless the human measure is obtained under the same prompt and context. Thus the strongest claim, as stated, is not a valid inference from the current design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares GPT-4o's irony ratings for emojis with \"human-perceived\" irony scores obtained from the Ciron corpus. The human score S(e) in Eq. (1) is the average post-level irony rating over posts containing emoji e. GPT-4o is prompted to rate, on an 11-point scale, how likely it would be to use a given emoji to express irony; these ratings are rescaled to a 5-point scale and compared with S(e) using a Wilcoxon signed-rank test and a Spearman correlation. The paper reports that GPT-4o assigns significantly higher irony scores than humans (W = 918.5, p < .001) and that the two sets of scores are weakly but significantly correlated (rho = 0.28, p < .05). It also explores prompts with demographic information and reports an age-related trend in GPT-4o's scores.","tokens_in":6988,"tokens_out":5968,"duration_ms":61235,"significance":"If the comparison were valid, the finding that GPT-4o systematically overestimates the ironic potential of emojis relative to human usage would be practically useful for emoji-aware sentiment analysis and human-behavior simulation. The paper has several strengths: it uses an external human-annotated corpus rather than fitting model parameters to the human data, the model is a frozen system queried with a transparent prompt, and the nonparametric rank tests are appropriate for the discrete, skewed scores involved. The limitation paragraph honestly acknowledges the single-model and Chinese-corpus constraints. However, the central comparison is undermined by a conceptual mismatch between what the human score measures and what the prompt elicits, so the headline overestimation claim is not currently supported.","major_comments":[{"comment":"The sentence \"We use the irony rating of a post to represent the irony level of the emojis found within it\" is an untested and load-bearing assumption. A post-level irony rating is assigned to the entire post, not to each emoji in it; an ironic post may contain emojis that are not themselves ironic, and a non-ironic post may contain an emoji used ironically. Because S(e) is the entire human benchmark used in the Wilcoxon and Spearman comparisons, this assumption must be validated with emoji-level annotations or at least with a per-emoji agreement analysis. Without such validation, the comparison is not a clean measure of emoji-level human perception.","section":"Human Perception of Emoji Irony, Eq. (1)"},{"comment":"The prompt asks GPT-4o to \"rate your likelihood of using this emoji if your intention is to express irony,\" which elicits P(use e | ironic intent), whereas Eq. (1) computes S(e) = E[irony rating | e appears], an estimate of P(ironic intent | e). These are inverse conditionals related by Bayes' rule, and they can diverge substantially when emoji base rates are skewed, as they are here (82 emojis across about 3,000 posts). Consequently, the Wilcoxon result W = 918.5 does not establish that GPT-4o \"systematically overestimates\" human perception; it may simply reflect the difference between asking \"how likely would you use this emoji ironically?\" and \"how often does this emoji appear in ironic posts?\" The paper's own smirk-emoji example in Results illustrates exactly this distinction: GPT-4o's high rating is justified by possible ironic contexts, while the Ciron posts containing that emoji are mostly non-ironic. A valid comparison would require eliciting human likelihood ratings under the same prompt, or annotating emoji irony directly in context.","section":"GPT-4o's Classification of Emoji Irony and Results"},{"comment":"The paper states that the model's ratings are \"rescaled from a range of 1–11 to align with the 1–5 scale\" but does not give the transformation. This matters because the Wilcoxon signed-rank test operates on the signs of paired differences, and an affine transformation of the model scores with a nonzero intercept can change the sign of some differences. For example, the natural mapping x' = 0.4x + 0.6 adds a positive constant to all model scores, which can flip small negative differences to positive ones and alter W and the reported p < .001. The authors should specify the exact rescaling, justify it, and show that the statistical conclusions are robust to reasonable alternatives, or use a comparison that does not depend on the arbitrary intercept.","section":"Results, rescaling paragraph"},{"comment":"The demographic section states that \"no significant differences in irony scores are observed between prompts specifying female or male gender\" and that scores \"tend to decrease on average as the specified age in the prompt increases,\" but no test statistic, p-value, effect size, or confidence interval is reported for any of these comparisons. If these demographic findings are part of the paper's contribution, they need inferential support; as written, the claims rest on visual inspection of Figure 2. The paper should also explain how the five age groups and the male/female conditions were compared (e.g., repeated-measures tests across the 82 emojis) and whether any multiple-comparison correction was applied.","section":"Prompts with Demographic Information, Figure 2"}],"minor_comments":[{"comment":"The text says the \"median irony score assigned by GPT-4o is significantly higher\" than the human score, but the Wilcoxon signed-rank test tests the median of paired differences (pseudomedian), not the difference of the two marginal medians. The wording should be adjusted to avoid a technically incorrect interpretation.","section":"Results, Wilcoxon reporting"},{"comment":"The Spearman correlation is reported only as p < .05; the exact p-value and the sample size (N = 82) should be stated, and the caption should note that the correlation uses tied discrete scores.","section":"Figure 1"},{"comment":"The paper does not report how the 82 unique emojis were extracted or their frequency distribution across the approximately 3,000 emoji-containing posts. Many emojis likely appear only once or twice, making their S(e) estimates very noisy; reporting the counts, or at least their minimum, median, and maximum, would help readers assess the reliability of Eq. (1).","section":"Human Perception of Emoji Irony"},{"comment":"The paper averages three GPT-4o queries per emoji at temperature 0.5 but does not report the variance or agreement across those queries. Reporting the standard deviation or the range of the three ratings would strengthen the claim that the model scores are stable.","section":"Experiment Setting"},{"comment":"The speculation that GPT-4o \"may have been trained on data with a disproportionate representation of ironic emoji usage\" is presented without evidence; it is a plausible hypothesis, but it should be labeled as a hypothesis rather than a conclusion drawn from the current results.","section":"Discussion and Conclusions"},{"comment":"Several emoji glyphs do not render in the plain-text version of the manuscript (e.g., the kiss emoji in Eq. (2) and the smirk emoji in Results). The final camera-ready version must ensure the glyphs display correctly, since the emojis are the experimental stimuli.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central comparison conflates inverse conditional probabilities, so the headline claim about overestimation cannot be accepted as stated. I would require either new human data collected under the same likelihood prompt, or a substantial reframing of the paper as a comparison between a model's conditional use likelihood and a corpus-based co-occurrence measure, with the rescaling and demographic analyses fully specified. The paper is within scope for a computational social science or NLP venue, but the conceptual mismatch needs to be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this one before you cite it: the headline claim that GPT-4o systematically overestimates emoji irony is not supported by the design. The human score S(e) in Eq. (1) is the average post-level irony rating over posts containing e, which estimates E[irony | emoji appears]. The model prompt asks \"rate your likelihood of using this emoji if your intention is to express irony,\" which elicits P(use e | ironic intent). Those are inverse conditionals. An emoji can be a plausible ironic choice (high P(e|I)) yet rarely appear in ironic posts (low E[I|e]) because of base rates. The Wilcoxon test on these two different quantities tells you the direction of the conditionals, not that the model overestimates human perception. The paper's own smirk example illustrates the point: GPT-4o justifies a high rating by describing potential ironic contexts, while Ciron posts containing that emoji are mostly non-ironic. Even with perfect emoji-level human annotations, the model would still be answering a different question unless the human measure is elicited under the same prompt. So the strongest claim, as stated, is not a valid inference.\n\nThat said, the paper does some things well. It is a clean, incremental extension of Lyu et al. (2024b) from general emoji discrepancy to irony, and it adds demographic prompting. The rank-based tests are appropriate for the paired data, the p-values are consistent with the reported statistics, and the method is transparent and reproducible. The authors also explicitly flag the post-to-emoji transfer assumption and acknowledge the English-versus-Chinese cultural mismatch. Those are honest limitations.\n\nThe soft spots beyond the inverse-conditional problem: the demographic analysis compares model outputs across prompt conditions, not against human demographic data, and the age trend is never statistically tested. The sample is 82 emojis from one platform, and the effect size (rho=0.28) is weak. The transfer assumption is the reader's weakest point; the stress-test note is a second, deeper facet of the same mismatch.\n\nThis paper deserves a serious referee because the topic is timely, the artifact is reproducible, and the flaw is addressable by reframing the claim or adding a human benchmark collected under the same conditional. But as is, it should not be accepted without substantial revision. Who gets value from it: people working on emoji sentiment or LLM evaluation who want a cautionary example of conditional mismatch. I would not cite it as a finding, though I might mention it in a methods discussion as a case where prompt direction and base rates confound comparison.","headline":"Useful extension of prior emoji-LLM work, but the central overestimation claim compares inverse conditionals and does not hold as stated.","tokens_in":7562,"tokens_out":1507,"would_cite":false,"duration_ms":16245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that GPT-4o systematically overestimates the likelihood that emojis are used ironically, relative to human usage patterns measured in a Chinese social media corpus, and that model and human scores agree only weakly.","keywords":["emoji irony","GPT-4o","LLM interpretation","human perception","irony detection","Ciron dataset","demographic factors","social media"],"falsifier":"Collect a set of posts, have human annotators rate the irony of each emoji directly in context rather than the whole post, and compare emoji-level human scores to GPT-4o's likelihood ratings on the same emojis; if the model's overestimation disappears or reverses under direct emoji annotation, the paper's central gap is an artifact of the post-to-emoji transfer.","tokens_in":6477,"feed_emoji":"😏","tokens_out":4480,"duration_ms":39154,"temperature":0.7,"pith_summary":"The paper asks whether GPT-4o interprets the ironic potential of emojis the way human social media users do. Using the Ciron dataset of Chinese Weibo posts, the authors derive a human irony score for each of 82 emojis by averaging the irony ratings of posts that contain it, then prompt GPT-4o to rate how likely it would use each emoji to express irony. They report that GPT-4o's median irony score is significantly higher than the human-derived score (Wilcoxon W = 918.5, p < .001) and that the two sets of scores correlate only weakly (Spearman ρ = 0.28, p < .05). In other words, the model overestimates how ironic emojis look, and its ordering of emojis by irony aligns poorly with human usage. The paper also finds that when prompts specify a demographic, simulated age shifts GPT-4o's scores downward while simulated gender has little effect.","feed_headline":"GPT-4o sees more irony in emojis than humans do","feed_subtitle":"On 82 emojis, model scores run significantly higher and correlate only weakly with human ratings.","key_machinery":"The transfer rule of Equation (1) is the hinge: it assigns to each emoji e the average irony rating R(p) of all posts containing e, so human emoji-level irony is never directly annotated but inferred from post-level ratings. The paper compares these derived human scores with GPT-4o's ratings obtained from an 11-point likelihood prompt, with candidates presented as images, and then rescales the model ratings to the 1–5 human scale. The statistical machinery consists of the Wilcoxon signed-rank test for median difference and the Spearman rank correlation for agreement, with prompt variations that insert gender and age labels.","core_discovery":"On the paper's own terms, the central finding is a measurable gap: GPT-4o, prompted to act as a social media user, assigns significantly higher likelihood-of-irony ratings to emojis than the irony scores humans produce through natural usage in the Ciron corpus, and the agreement between model and human scores is positive but weak (ρ = 0.28). This overestimation appears across the emoji set and is accompanied by larger variance in the model's ratings. The paper attributes the gap to possible training-data skew toward ironic emoji usage, overgeneralization of irony patterns, and the English-centric orientation of GPT-4o relative to the Chinese-language dataset, and it treats the demographic prompt results as evidence that the model tends to assign lower irony scores when prompted with older ages.","pith_inferences":["A cleaner test would annotate emojis directly in context; if such annotation lowered human emoji-level irony scores, the GPT-4o overestimation gap would widen, and if it raised them, the gap could narrow or disappear. This is my inference, not the paper's.","The rescaling of the 11-point model scale to a 1–5 scale is monotone but arbitrary; a different mapping could change the magnitude of the reported median difference, although it would not affect Spearman correlation.","The study's binary gender and five age bins likely compress real demographic variation; a prompt study with non-binary genders and finer age gradations could reveal effects the current design cannot see.","Because the correlation is computed on only 82 emojis with many ties, a permutation or bootstrap test would give a more robust sense of whether ρ = 0.28 is stable."],"forward_implications":["Applications that rely on GPT-4o to gauge emoji sentiment can expect it to flag irony more often than users intend, risking over-detection in chatbot and sentiment pipelines.","The weak correlation means the model's ranking of which emojis are most ironic is not a reliable proxy for human rankings.","If the age effect is real and stable, LLM-based social-media personas for older demographics will produce more literal emoji interpretations, which may matter for behavioral simulation.","The larger variance in model scores suggests its interpretations are less anchored than human usage patterns, so single-shot ratings should not be treated as calibrated.","The use of a Chinese-language dataset with an English-trained model means the observed overestimation may partly reflect cross-linguistic transfer rather than a general property of the model."],"supporting_citations":[{"why":"Supplies the Ciron dataset, the post-level human irony ratings, and the definition of irony as meaning opposite to literal interpretation; the human benchmark is built entirely from it.","marker":"Xiang et al. (2020)"},{"why":"The alternative English irony dataset considered but rejected for having only 494 tweets with emojis, which motivates using Ciron.","marker":"Van Hee, Lefever, and Hoste (2018)"},{"why":"Establishes the prior finding of human–model emoji interpretation discrepancy that this paper extends, and supplies the prompt-with-image approach.","marker":"Lyu et al. (2024b)"},{"why":"Motivates presenting emojis in image format to GPT-4o.","marker":"Częstochowska et al. (2022)"},{"why":"Supplies the demographic prompt manipulation approach for gender groups in LLM experiments.","marker":"Kotek, Dockum, and Sun (2023)"},{"why":"Supplies prior evidence that age and gender affect human emoji comprehension, against which the demographic prompt results are interpreted.","marker":"Chen et al. (2024)"}],"fun_headline_variants":["GPT-4o overrates irony in emojis vs humans","Model sees more emoji irony than people do","GPT-4o's emoji irony scores exceed humans'","Emoji irony: GPT-4o overshoots human judgment","GPT-4o and humans diverge on emoji irony"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on treating the irony rating of a post as the irony level of every emoji inside it, so that one post-level number stands in for each emoji's human score; if that transfer is wrong, the human benchmark is not actually measuring emoji irony.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o overrates irony in emojis vs humans","Model sees more emoji irony than people do","GPT-4o's emoji irony scores exceed humans'","Emoji irony: GPT-4o overshoots human judgment","GPT-4o and humans diverge on emoji irony"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1407,"prompt_tokens":845,"completion_tokens":562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":461,"tokens_out":562,"duration_ms":5009,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:28:51.366308+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of posts, have human annotators rate the irony of each emoji directly in context rather than the whole post, and compare emoji-level human scores to GPT-4o's likelihood ratings on the same emojis; if the model's overestimation disappears or reverses under direct emoji annotation, the paper's central gap is an artifact of the post-to-emoji transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Ciron dataset, the post-level human irony ratings, and the definition of irony as meaning opposite to literal interpretation; the human benchmark is built entirely from it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The alternative English irony dataset considered but rejected for having only 494 tweets with emojis, which motivates using Ciron."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies prior evidence that age and gender affect human emoji comprehension, against which the demographic prompt results are interpreted."}],"review_version":1}