Pith. sign in

REVIEW 4 major objections 5 minor 30 references

Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Large language models can define queer slang reliably when given an example sentence or a domain cue, but without either they fall back on the word's everyday meaning.

desk verdict A genuinely useful new dataset and a plausible first result on LLMs and queer slang, but the quantitative core is thin and the single-annotator labeling is the load-bearing risk. read the letter →

arxiv 2608.04847 v1 pith:XPSFPSTL submitted 2026-08-05 cs.CL

classification cs.CL
keywords queerslangLGBTQIA+languageuser-generatedcontentlargemodelsdefinitiongenerationpromptdesigntaxonomypolysemy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models actually understand queer slang as it appears in real user-generated text, and it answers with a new evaluation resource. The authors build Slang-Q, a manually curated set of 1,024 English sentences from an online slang dictionary, paired with 118 queer-related terms, definitions, and a two-level taxonomy. They then test four instruction-tuned models on a definition-generation task under four prompt conditions. The consistent finding is that models score below a human reference bound, and performance drops most when the model receives only the bare term with no indication that it is queer slang; adding a domain cue, an example sentence, or both pushes definitions toward the intended queer-specific sense. Manual review shows fully incorrect answers drop sharply in the context-plus-framing condition, which matters because it means simple prompting choices can change whether an LLM gives accurate information about community-specific language.

What carries the argument

The load-bearing object is Slang-Q: a manually curated dataset of 1,024 user-generated English sentences, each paired with one of 118 queer-related terms and a reference definition, organized by a two-level taxonomy with broad categories (Identity, Slang, Intersectional) and optional subcategories (Reclaimed, Shorthand, Idiomatic expression, Pronoun, Spelling variation). The evaluation machinery is the crossed design of two binary prompt dimensions—domain framing (generic language expert versus queer internet slang expert) and context (term only versus term plus example sentence)—which isolates the effect of each on definition generation. Scores are computed with a lexical-overlap metric and a semantic-similarity metric against three reference definitions per term, then aggregated at the term level so that frequent terms do not dominate the results.

What would settle it

Independently re-annotate a random sample of the 1,024 sentences with several annotators who are familiar with online queer slang, and measure inter-annotator agreement; if a substantial share of the queer-sense and harmfulness labels are not reproduced, the gold standard and the prompt-condition scores built on it collapse.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that what an LLM knows about queer slang is not a fixed quantity: it is strongly conditioned by whether the prompt identifies the domain and whether an example sentence is present. Across all four evaluated models, the bare-term baseline scores lowest on both automatic metrics, while slang-informed framing alone and contextual grounding alone each improve scores, and the two together perform best. Manual inspection of 200 outputs from the two conditions with the strongest contrast supports this: in the bare-term condition, roughly a quarter to a third of definitions miss the intended sense, while in the slang-informed context condition the share of fully incorrect definitions drops to zero in one model and to a small fraction in the other, though some definitions still omit sociocultural nuance. The paper also finds that all models fall below the human reference bound, that no model dominates, and that intersectional terms—those shared with other communities such as AAVE or fandom—are the hardest category. A data-contamination probe finds that one proprietary model appears to have memorized far more of the source sentences than the open-weight models, yet this does not translate into better definitional performance.

Load-bearing premise

The whole evaluation rests on one author's manual judgment, without measured inter-annotator agreement, of which sentences use each term in its intended queer sense and which content is harmful; if those labels are wrong or inconsistent, the gold definitions and all model scores built on them shift.

Editorial extensions

If this is right

  • A model asked to define a slang term in isolation will make noticeably more errors than the same model given a single example sentence, so user-facing systems that explain community slang should avoid bare-term queries.
  • Explicitly telling the model that it is dealing with queer slang is enough by itself to shift some definitions toward the queer-specific sense, and combining that framing with an example sentence yields the strongest correctness in the manual evaluation.
  • Intersectional terms that circulate in multiple communities, such as AAVE or fandom, are the hardest to define, meaning queer-slang evaluation should treat multi-community usage as a first-class difficulty rather than noise.
  • Since all four tested models cluster within a narrow performance band, the results point to broadly similar queer-slang knowledge across model families and sizes, with none reaching the human reference bound.
  • The Slang-Q dataset provides a reusable evaluation set for future work on definition generation, prompt design, and slang handling in large language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the model with the highest estimated memorization of the source corpus does not outperform the others, the paper's data point implicitly suggests that exposure to slang-heavy text during training is not the limiting factor for community-specific understanding; prompt and context design may be more actionable.
  • The taxonomy's 'Intersectional' label acknowledges that queer slang overlaps with AAVE and fandom usage; a fairer scoring protocol might compare model definitions against multiple gold definitions, one per community, since a single reference definition will penalize valid senses.
  • A direct testable extension is to apply the same two-by-two prompt design to other dynamic sociolects, such as disability community language or regional slang, to see whether the context-and-framing effect generalizes beyond queer slang.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Slang-Q, a dataset of 1,024 naturally occurring English sentences paired with 118 queer slang terms and reference definitions drawn from external lexicons, together with a taxonomy of queer slang. Using this resource, the authors evaluate four LLMs (Claude Sonnet 4.6, LLaMA 3.3 70B, LLaMA 4 Scout, Qwen3 32B) on a definition-generation task under four prompt conditions that vary whether the prompt is slang-informed and whether an example sentence is provided. Automatic evaluation with ROUGE-L and BERTScore is supplemented by a manual error analysis on two models. The paper concludes that models fall below a human upper bound, that withholding domain framing and sentential context hurts performance, and that both slang-informed prompting and contextual grounding help models converge on the intended queer-specific meaning.

Significance. If the results are reliable, this is a useful contribution to an underserved area: queer slang is underrepresented in NLP, and Slang-Q could become a reference resource for evaluating LLM understanding of community-specific language. The authors are transparent about data sources, make the repository available, include a data-contamination check, and provide a manual error analysis that partially corroborates the automatic findings. However, the current evidence is exploratory: the automatic metric differences are small, the 'human' bound is not a true human-performance measurement, and the entire dataset rests on single-annotator judgments without reliability checks. These issues are central to the paper's headline claims rather than cosmetic.

major comments (4)
  1. [Section 3.3] The annotation was carried out by the first author alone, with only 'fewer ambiguous cases' discussed with the second author, and no inter-annotator agreement or reliability statistic is reported. Because every sentence-term pairing, every judgment that a term is used in its intended queer sense, and every harmfulness decision is single-annotator, the two headline findings--that models score below a human bound and that context/framing improve performance--inherit any systematic bias in these labels. If the annotator retained sentences in which the queer sense is strongly disambiguated by context and discarded genuinely ambiguous ones, the context-benefit effect could be inflated or even manufactured. The manuscript should report a reliability study (e.g., a second annotator on a stratified sample with Cohen's kappa or equivalent) and an adjudication protocol, or the conclusions must be substantially softened.
  2. [Section 5, Table 5] The row labeled 'Human' is not a measure of human performance; it is the mean similarity of the gold reference definition to the two GPT-5.5-generated alternative definitions. The conclusion in Section 6 that 'models fall below the human upper bound' is therefore not supported by the data as presented. Either collect and score actual human-written definitions for a sample of terms, or relabel this quantity as a reference-agreement ceiling and revise the wording of the finding accordingly.
  3. [Section 5, Table 5] The reported condition differences are very small relative to the standard deviations (e.g., ROUGE-L 0.16 vs 0.18, BERTScore 0.85 vs 0.86, with SDs around 0.05-0.07), and no significance tests are reported. The claim in Section 6 that withholding domain framing and sentential context 'consistently hurts performance' requires paired significance testing across terms (e.g., Wilcoxon signed-rank or bootstrap) and effect sizes. The manual evaluation is suggestive but covers only two models and two conditions, so it cannot by itself establish the cross-model claim.
  4. [Section 4.4 and Section 5] Claude Sonnet 4.6 shows a high verbatim contamination range (57.30-62.00%) with the source corpus, and the context condition supplies the exact example sentences from that corpus. Although the authors argue that memorization of a sentence does not entail ability to solve the definition task, memorized examples could plausibly inflate performance specifically in the context conditions. Please provide a per-model breakdown of the context benefit and discuss how contamination affects the interpretation, or explicitly control for it in the analysis.
minor comments (5)
  1. [Section 3.3] The discard counts (1,833 semantically unrelated + 327 harmful/vulgar = 2,160) leave 1,024 of 3,184 sentences; stating the retained percentage explicitly (about 32%) would prevent reader confusion.
  2. [Section 5] There is a typo in 'showes' (should be 'shows'), and the condition naming is inconsistent between 'Term baseline' in the text and 'Term (Base)' in the first result paragraph.
  3. [Table 5] The 'Human' row would benefit from a footnote stating explicitly that it measures agreement among the gold and paraphrased reference definitions, not human-written outputs.
  4. [Figures 1 and 2] The category panels aggregate across all models and conditions, which can mask important interactions; adding error bars or significance markers would make the visual claims more interpretable.
  5. [Section 6] The limitations paragraph should mention the single-annotator reliability issue and the lack of significance tests, since these are the main threats to the stated conclusions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: model scores are compared against externally sourced gold definitions and the dataset is not derived from the evaluated models.

full rationale

The paper's central claim compares model-generated definitions against gold references assembled from external lexical sources (Lexicon Library.LGBT, Wiktionary, Wikipedia, and prior NLP inventories), not from the evaluated models: "When a term is added from a lexical source, such as Lexicon Library.LGBT, Wiktionary, or Wikipedia's LGBTQ slang category, we use the definition provided by that source" (§3.1.1). The GPT-5.5-generated alternative references were manually validated for meaning preservation and are used uniformly as additional references, so they do not force the observed condition effects. The only self-citations (e.g., Savoldi et al. [3] and Piergentili et al. [18]) are background references, not load-bearing evidence for the dataset or the evaluation conclusions. The single-annotator labeling described in §3.3 ("Annotation was carried out by the first author") is a genuine validity and reliability risk, but it is not circular: the annotation decisions determine which sentences enter the dataset, not the models' scores by construction, and the paper explicitly notes that contamination does not entail task-solving ability (§4.4). Since no fitted parameter is renamed as a prediction and no claim reduces to its own input by definition, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters are fitted to data in this paper; the numbers reported are measurements, not fits. The main assumptions are domain assumptions about data representativeness, metric validity, and annotation reliability. No new physical or conceptual entities are postulated; the taxonomy is a classification scheme, not an independently evidenced entity.

assumptions (5)
  • domain assumption Urban Dictionary sentences extracted from the SLANG benchmark are a suitable and representative source of naturally occurring queer slang in user-generated content.
    All sentences in Slang-Q come from this corpus (Section 3); the paper does not validate representativeness against other platforms or genres.
  • domain assumption ROUGE-L and BERTScore, averaged over the gold definition and two GPT-5.5 paraphrases, are valid proxies for definition quality.
    The main quantitative results use these metrics (Section 4.3); validation only checks length sensitivity, not correlation with human judgments on the full set.
  • domain assumption The single annotator's classification of intended queer sense, harmfulness, and taxonomy category is reliable enough to serve as ground truth.
    Section 3.3 reports annotation by the first author with ambiguous cases discussed with the second author and no inter-annotator agreement.
  • domain assumption The 'human upper bound' measured as gold-vs-GPT-5.5-paraphrase similarity is a meaningful comparator for model performance.
    Table 5 and Section 4.3; this is a paraphrase-consistency measure, not a measure of humans defining the terms.
  • domain assumption Verbatim contamination detected by DCQ does not benefit the definition-generation task.
    Section 4.4 asserts this for Claude's 57-62% contamination without direct evidence; if false, Claude's scores could be inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content." pith.science (2026). https://pith.science/paper/XPSFPSTL

@misc{pith2026260804847,
  author       = {Pith},
  title        = {Pith review of: Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPSFPSTL}},
  note         = {Machine review of arXiv:2608.04847}
}
read the original abstract

Despite its cultural relevance and diffusion, queer slang remains underrepresented in Natural Language Processing research. Towards addressing this gap, we introduce Slang-Q, a manually curated dataset of naturally user-generated English sentences paired with queer slang terms and reference definitions, built upon a newly constructed taxonomy of 118 queer terms. We use this resource to conduct a first exploratory evaluation of language models on their ability to understand and define queer slang under varying prompting conditions. Slang-Q is intended as a basis for studying how current models handle sensitive, community-specific language and whether they can provide accurate and reliable information about such forms of identity and linguistic expression.

Figures

Figures reproduced from arXiv: 2608.04847 by the authors.

Figure 1
Figure 1. BERTScore F1 scores by taxonomy category, aggregated across all models and conditions (A) and broken down by prompting condition (B). intersectional slang identity 0.0 0.1 0.2 0.3 0.4 0.5 ROUGE Score (A) ROUGE by Category intersectional slang identity 0.0 0.1 0.2 0.3 0.4 0.5 ROUGE Score (B) ROUGE by Condition context context-s term term-s [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. ROUGE scores by taxonomy category, aggregated across all models and conditions (A) and broken down by prompting condition (B). Manual Evaluation We complement our automatic evaluation with a manual error analysis on a subset of models and conditions. Namely, we focus on the two most contrasting conditions—Term base￾line and slang-informed Context—for Claude Sonnet 4.6 and Llama 4 Scout. Indeed, the two models show t… view at source ↗
Figure 3
Figure 3. shows the thirty most frequent matched terms in the final dataset. The distribution is highly skewed, with a small number of terms accounting for a large share of the matched occurrences [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

  1. [1]

    C. Chen, K. Shu, Combating misinformation in the age of llms: Opportunities and challenges, AI magazine 45 (2024) 354–368

  2. [2]

    Chatterji, T

    A. Chatterji, T. Cunningham, D. J. Deming, Z. Hitzig, C. Ong, C. Y. Shan, K. Wadman, How people use chatgpt, Technical Report, National Bureau of Economic Research, 2025

  3. [3]

    Savoldi, G

    B. Savoldi, G. Attanasio, O. Gorodetskaya, M. M. Manerba, E. Bassignana, S. Casola, M. Negri, T. Caselli, L. Bentivogli, A. Ramponi, A. Muti, N. Balbo, D. Nozza, Generative ai practices, literacy, and divides: An empirical analysis in the italian context, 2026. URL: https://arxiv.org/abs/2512. 03671.arXiv:2512.03671

  4. [4]

    Basoah, D

    J. Basoah, D. Chechelnitsky, T. Long, K. Reinecke, C. Zerva, K. Zhou, M. Díaz, M. Sap, Not like us, hunty: Measuring perceptions and behavioral effects of minoritized anthropomorphic cues in llms, in: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 2025, pp. 710–745

  5. [5]

    L. Mei, S. Liu, Y. Wang, B. Bi, X. Cheng, SLANG: New concept comprehension of large language models, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 12558–12575. URL: https://aclanthology.org/20...

  6. [6]

    C. C. Eble, Slang & sociability: In-group language among college students, Univ of North Carolina Press, 1996

  7. [7]

    Kulick, Gay and lesbian language, Annual review of anthropology 29 (2000) 243–285

    D. Kulick, Gay and lesbian language, Annual review of anthropology 29 (2000) 243–285

  8. [8]

    Motschenbacher, Taking queer linguistics further: Sociolinguistics and critical heteronormativ- ity research., International journal of the sociology of language 2011 (2011)

    H. Motschenbacher, Taking queer linguistics further: Sociolinguistics and critical heteronormativ- ity research., International journal of the sociology of language 2011 (2011)

Show all 30 references
  1. [9]

    Cifor, K

    M. Cifor, K. Rawson, Mediating queer and trans pasts: The homosaurus as queer information activism, Information, Communication & Society 26 (2023) 2168–2185

  2. [11]

    R. Dorn, L. Kezar, F. Morstatter, K. Lerman, Harmful speech detection by language models exhibits gender-queer dialect bias, in: Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 2024, pp. 1–12

  3. [12]

    Vásquez, G

    J. Vásquez, G. Bel-Enguix, S. T. Andersen, S.-L. Ojeda-Trueba, Heterocorpus: A corpus for heteronormative language detection, in: Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), 2022, pp. 225–234

  4. [13]

    Tint, Guardrails, not guidance: Understanding responses to LGBTQ+ language in large language models, in: A

    J. Tint, Guardrails, not guidance: Understanding responses to LGBTQ+ language in large language models, in: A. Pranav, A. Valentine, S. Bhatt, Y. Long, A. Subramonian, A. Bertsch, A. Lauscher, A. Gupta (Eds.), Proceedings of the Queer in AI Workshop, Association for Computatio...

  5. [14]

    Weber, A

    S. Weber, A. Wang, A. Gupta, A. Subramonian, D. Ulmer, E. Tanwar, G. Aich, H. Devinney, J. Hobbs, J. Mickel, J. Tint, M. Sosto, R. Groshan, S. Astarita, V. Gautam, V. Blaschke, W. Agnew, W. Y. Lee, Y. Long, Queer nlp: A critical survey on literature gaps, biases and trends, 20...

  6. [15]

    Sosto, D

    M. Sosto, D. S. M. Pandiani, L. Hollink, QueerGen: How LLMs reflect societal norms on gender and sexuality in sentence completion task, in: V. Demberg, K. Inui, L. Marquez (Eds.), Findings of the Association for Computational Linguistics: EACL 2026, Association for Computation...

  7. [16]

    URL: https://lexicon.library.lgbt/

    Library.LGBT, Lexicon library.lgbt, n.d. URL: https://lexicon.library.lgbt/

  8. [17]

    Veloso, L

    L. Veloso, L. Hirlimann, P. Wicke, H. Schütze, Slaying: Towards queer language processing, 2025. URL: https://arxiv.org/abs/2509.17449.arXiv:2509.17449

  9. [18]

    Piergentili, B

    A. Piergentili, B. Savoldi, M. Negri, L. Bentivogli, Enhancing gender-inclusive machine translation with neomorphemes and large language models, in: C. Scarton, C. Prescott, C. Bayliss, C. Oakley, J. Wright, S. Wrigley, X. Song, E. Gow-Smith, R. Bawden, V. M. Sánchez-Cartagena...

  10. [19]

    E. L. Ungless, S. Dev, C. L. Bennett, R. Gulotta, J. Bastings, R. Denton, Amplifying trans and nonbinary voices: A community-centred harm taxonomy for LLMs, in: W. Che, J. Nabende, E. Shutova, M. T. Pilehvar (Eds.), Proceedings of the 63rd Annual Meeting of the Association for...

  11. [20]

    Locatelli, G

    D. Locatelli, G. Damo, D. Nozza, A cross-lingual study of homotransphobia on Twitter, in: S. Dev, V. Prabhakaran, D. I. Adelani, D. Hovy, L. Benotti (Eds.), Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), Association for Computational Linguis...

  12. [21]

    Pofcher, C

    J. Pofcher, C. M. Homan, R. Sell, A. R. KhudaBukhsh, Hope vs. hate: Understanding user interactions with LGBTQ+ news content in mainstream US news media through the lens of hope speech, in: C. Christodoulopoulos, T. Chakraborty, C. Rose, V. Peng (Eds.), Proceedings of the 2025...

  13. [22]

    Ghosal, A

    A. Ghosal, A. Gupta, V. Srikumar, Unequal voices: How llms construct constrained queer narratives,

  14. [23]

    Ramesh, S

    K. Ramesh, S. Kumar, A. Khudabukhsh, Revisiting queer minorities in lexicons, in: K. Narang, A. Mostafazadeh Davani, L. Mathias, B. Vidgen, Z. Talat (Eds.), Proceedings of the Sixth Work- shop on Online Abuse and Harms (WOAH), Association for Computational Linguistics, Seat- t...

  15. [24]

    Subramonian, V

    A. Subramonian, V. Gautam, P. Seshadri, D. Klakow, K.-W. Chang, Y. Sun, Agree to disagree? a meta- evaluation of llm misgendering, 2025. URL: https://arxiv.org/abs/2504.17075.arXiv:2504.17075

  16. [25]

    Team, Qwen3 technical report, 2025

    Q. Team, Qwen3 technical report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388

  17. [26]

    by arXiv, The llama 4 herd: Architecture, training, evaluation, and deployment notes, 2026

    R. by arXiv, The llama 4 herd: Architecture, training, evaluation, and deployment notes, 2026. URL: https://arxiv.org/abs/2601.11659.arXiv:2601.11659

  18. [27]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sra- vankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spatar...

  19. [28]

    Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp

    C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013/

  20. [29]

    Zhang, V

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, 2020. URL: https://arxiv.org/abs/1904.09675.arXiv:1904.09675

  21. [30]

    A Chinese internet slang term referring to a person whose sex- ual orientation or gender iden- tity is perceived as fluid or am- biguous

    S. Golchin, M. Surdeanu, Data contamination quiz: A tool to detect and estimate contamination in large language models, Transactions of the Association for Computational Linguistics 13 (2025) 809–830. URL: https://aclanthology.org/2025.tacl-1.37/. doi:10.1162/tacl.a.20. A. Ter...

  22. [2025]

    URL: https://arxiv.org/abs/2507.15585.arXiv:2507.15585

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.