Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Tell Me What You Know About Sexism: Expert-LLM Interaction Strategies and Co-Created Definitions for Zero-Shot Sexism Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-generated definitions outperform expert-written ones in zero-shot sexism detection, with co-creation helping only some experts.

desk verdict A genuinely useful pipeline and resource paper whose central co-creation claim is undercut by its own appendix: for most experts the co-created definition is nearly identical to the LLM-generated one. read the letter →

arxiv 2504.15392 v1 pith:RFOD2ZTM submitted 2025-04-21 cs.CL cs.CY

Myrthe Reuver , Indira Sen , Matteo Melis , Gabriella Lapesa This is my paper · ORCID
classification cs.CLcs.CY
keywords sexismdetectionzero-shotclassificationlargelanguagemodelshybridintelligenceco-createddefinitionsexpertknowledgepromptengineeringbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether sexism researchers can improve zero-shot sexism detection by writing, selecting, or co-creating the definition of sexism that an LLM is prompted with. Nine sexism researchers each produced three definitions: their own, the best one the LLM generated, and a version they edited together with the LLM. Prompting GPT-4o with these 27 definitions on 2,500 texts from five sexism benchmarks shows that expert-written definitions do markedly worse on average than LLM-generated ones. At the same time, some experts, including at least one with little LLM experience, get better results from the co-created definition than from the model's alone. The central claim is that the definition's provenance and content substantively change zero-shot detection performance, and that expert knowledge does not automatically transfer into a better prompt.

What carries the argument

The load-bearing object is the definition-as-prompt: each expert contributes three textual definitions of sexism (expert-written, LLM-generated, co-created), and each is inserted into a fixed prompt template that asks GPT-4o to label a text 'sexist,' 'non-sexist,' or 'can't say.' The pipeline—survey, interactive assessment of the model's knowledge, definition co-creation, and zero-shot classification—ties the qualitative interaction strategies to measured $F_1$ scores, so the paper can attribute performance differences to how the definition was produced.

What would settle it

A direct test: take the nine expert-written definitions, expand each to the average length of the LLM-generated definitions without adding new substantive criteria, and re-run GPT-4o on the same 2,500 texts; if $F_1$ jumps to the LLM-generated level, the advantage is length. A second test: compare only the experts whose co-created and LLM-generated definitions are semantically identical and check whether any performance difference remains.

Watch

Extended reading notes

Core claim

On the paper's own terms, the core discovery is that in zero-shot sexism classification, the provenance of the prompt definition changes performance: LLM-generated definitions (mean macro $F_1 = .765$) and co-created definitions (mean $F_1 = .762$) outperform expert-written definitions (mean $F_1 = .532$) across five benchmarks. The pattern is dataset-dependent, with near-tied results on CallMeSexist and a large gap on RedditGuest. A few experts nevertheless obtain their best results from co-created definitions, including Expert 6, who reported low confidence with LLMs. The authors read this as partial evidence for hybrid intelligence: expert and model can sometimes jointly produce a better detection prompt than either alone.

Load-bearing premise

The claim that definition provenance changes detection rests on the assumption that the measured $F_1$ differences come from the definition content, not from confounds such as definition length (expert-written definitions average 34 tokens versus 110–120 for the others) or from the high similarity between co-created and LLM-generated definitions.

Editorial extensions

If this is right

  • Definition provenance changes zero-shot sexism detection: model-generated definitions are better prompts on average than short expert definitions.
  • Expert knowledge can still enter a prompt usefully, but only through co-creation, and success depends on the individual expert, not on self-reported LLM confidence.
  • Prompt-sensitivity varies by dataset: on CallMeSexist all definition types perform similarly, while on RedditGuest expert-written definitions fall far behind.
  • The co-created definition can lift performance above the majority-class baseline on some datasets and experts, so hybrid definitions are not uniformly useless.
  • The effect is tied to the model and temperature: at higher temperature or with LLaMa the definition-type differences shrink, so the finding is not model-independent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because expert-written definitions average 34 tokens versus roughly 110–120 for the other two types, a plausible reading is that length, not authorship, drives the gap; this is an inference, not the paper's claim.
  • Since most co-created definitions are near-identical to LLM-generated ones (cosine similarities up to 1.00), the observed co-creation advantage may be an expert-selection effect—which model output the expert chose to keep—rather than an editing effect.
  • The strong dataset dependence suggests that prompt definitions are a high-variance intervention on rare-class benchmarks like RedditGuest; practitioners should probably evaluate several definition variants before relying on one.
  • Extending the same pipeline to other contested constructs would show whether the expert-written deficit is specific to sexism or a general feature of short expert definitions in zero-shot prompting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how sexism researchers interact with an LLM (GPT-3.5) to produce definitions of sexism, and whether those definitions improve zero-shot sexism detection when placed in a prompt template for GPT-4o. Nine experts each produced three definitions: an expert-written definition, a preferred LLM-generated definition, and a co-created definition. These 27 definitions were used to classify 2,500 texts from five sexism benchmarks, yielding 67,500 classification decisions. The main reported results are that expert-written definitions perform much worse (mean macro-F1 = .532) than LLM-generated definitions (.765) and co-created definitions (.762), and that some experts, including an LLM-inexperienced one, improve performance with co-created definitions. The paper also presents qualitative taxonomies of expert interaction strategies and definition similarity analyses, and it releases the interaction framework, definitions, and code.

Significance. The paper is methodologically interesting and fills a real gap: it connects open-ended expert–LLM interaction, definition co-creation, and downstream benchmark evaluation in one pipeline. Its strengths include a clearly described and reproducible experimental protocol, a released corpus of definitions and interactions, multiple benchmarks, and robustness checks with a second model and a different temperature. The large gap between expert-written and LLM-generated definitions is a robust and potentially useful finding for prompt design. However, the more distinctive claim about co-created definitions is not currently established. The manuscript's own Appendix E shows that most co-created definitions are identical or nearly identical to LLM-generated definitions, and Section 4.4.1 reports no confidence intervals or significance tests. The aggregate co-created vs. LLM-generated difference (.762 vs. .765) is within any reasonable estimate of sampling noise, and the per-expert comparisons promoted in the abstract and Section 4.5 are therefore likely to reflect repeated-run variation rather than a co-creation effect.

major comments (3)
  1. [Section 4.5, Table 8, Appendix E] The claim that "some experts do improve classification performance with their co-created definitions" is not supported by the evidence as presented. For seven of nine experts (1, 3, 5, 6, 9, 10, 11), the co-created definition is identical to or a lightly cleaned version of the LLM-generated definition, with TF-IDF cosine similarity at or above .95 (Experts 3 and 11 have similarity 1.00). The paper itself labels these cases "robustness tests" in Appendix E. Under these conditions, any F1 difference between the co-created and LLM-generated conditions is repeated-prompt noise, not a co-creation effect. Section 4.5 nevertheless treats Expert 6's difference as evidence of co-creation success despite a TF-IDF similarity of .97 between that expert's co-created and LLM-generated definitions. The analysis should be restricted to the experts whose co-created definitions genuinely differ from the LLM-generated ones (notably Experts 2 and 4), or the claim should be reformulated as a comparison of editing behavior rather than an independent contribution of expert knowledge.
  2. [Section 4.4.1 and Appendix C.2] No confidence intervals, significance tests, or repeated-run variance are reported for the main temperature=0 GPT-4o results, yet the central comparisons are close. The aggregate means are LLM-generated F1 = .765 and co-created F1 = .762, a difference of three thousandths that is almost certainly within sampling noise. The manuscript notes in Appendix C.2 that the full experiment was run twice, so run-level variance is available and should be reported. Without a proper uncertainty quantification, the abstract's statement that "some experts do improve" is indistinguishable from a noise-driven claim. I would expect per-expert bootstrap confidence intervals, a paired significance test, or an explicit effect size with uncertainty for the co-created vs. LLM-generated difference.
  3. [Section 4.4.1, Appendix E.1] The interpretation of the expert-written vs. LLM-generated gap is incomplete because definition length and content are confounded. Expert-written definitions average 34.44 tokens versus 119.89 for LLM-generated and 110.55 for co-created definitions. Appendix E.1 shows negative within-type correlations between length and F1, which weakens a simple "longer is better" explanation, but the cross-type comparison still conflates length, lexical style, and content. The paper would be stronger if it included a control analysis (e.g., length-matched definitions, or a comparison restricted to experts who wrote substantially longer definitions) or explicitly acknowledged that the expert-written gap may be driven by informativeness and style rather than by expert knowledge per se.
minor comments (5)
  1. [Appendix H.2] The sentence "the LLM-generated definition performs slightly lower (F1 = .699) than the co-written definition (F1 = .584) or participant-written definitions (F1 = .703)" is internally inconsistent: .584 is the lowest of the three values, not lower than .699. The numbers or the wording should be corrected.
  2. [Appendix G.2] The text reports "M F1 = .760 vs M F1 = .68 for the temperature=0 run," but Section 4.4.1 reports a mean F1 of .765 for the temperature=0 condition. This inconsistency should be reconciled.
  3. [Table 9] The last row entry "Similarity (TF-IDF) - LLM-generated" on HateCheck shows "33" rather than a decimal value; it should presumably be ".33".
  4. [Section 3.4] The text uses European decimal notation "2.500 texts" where an English-language paper should use "2,500 texts".
  5. [Throughout] The terminology for the three definition types is inconsistent: "expert-written," "participant definitions," and "hybrid" are all used in different places to refer to the same conditions. Please standardize the labels.

Circularity Check

1 steps flagged · score 6.0 of 10

The co-created condition is defined as a copy/edit of the LLM-generated definition, so the reported per-expert co-creation gains reduce to LLM-generated prompts by construction.

  1. self definitional [Section 3.3 (Part III), Appendix E 'Comparing LLM-generated and co-created definitions', Table 8; Section 4.5]
    "At the end of the interaction, the participants are asked to review the full interaction and perform two actions: a) select their preferred definition among those generated by the LLMs (referred to in the paper as LLM-generated definition) and b) copy and edit their preferred definition (we refer to this definition as co-created). ... It is true for the large majority of the experts that the co-created definition is either identical to the LLM-generated one (experts 3 and 11) or just an edited, cleaned-up version of the LLM-generated one (experts 1, 5, 6, 9, 10). ..."

    Co-created is operationally defined as 'copy and edit their preferred definition' where the preferred definition is one generated by the LLM. Appendix E and Table 8 confirm that for seven of the nine experts (1, 3, 5, 6, 9, 10, 11) the co-created text is identical to or a lightly cleaned-up version of the LLM-generated text (TF-IDF cosine .95-1.00; SBERT .62-1.00). The abstract's highlighted claim that 'some experts do improve classification performance with their co-created definitions ... also experts who are inexperienced in using LLMs' relies on per-expert differences such as Expert 6, whose co-created and LLM-generated definitions have TF-IDF similarity .97 and SBERT similarity .87.

full rationale

The main zero-shot modeling pipeline is not circular in the usual sense: the 27 definitions were elicited before classification, no definition or prompt parameter was fitted to the 2,500 benchmark labels, and the expert-written vs LLM-generated comparison is an external evaluation with real (if confounded) content. No load-bearing self-citation chain or uniqueness theorem appears; Sen et al. (2023) is only cited for the prompt template. The circularity is confined to the co-created condition. By the design in Section 3.3, the co-created definition is a copy/edit of the LLM-selected definition, and Appendix E states that for most experts it is identical or nearly identical to the LLM-generated one. Yet Section 4.5 and the abstract attribute per-expert F1 differences to co-creation, including for LLM-inexperienced Expert 6. Those differences are between effectively the same prompt, so the 'co-creation improves some experts' finding reduces by construction to the LLM-generated prompt's own performance. This is partial circularity in the paper's most distinctive claim, hence score 6 rather than a higher score.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The modeling comparisons rely on benchmark labels as ground truth, on prompt-based GPT4o outputs as a valid measurement, and on 500-text samples being representative. The study treats the co-created definition as a distinct condition even though most co-created definitions are near-identical to LLM-generated ones. No fitted numerical parameters or invented entities are introduced besides a hand-set sample size and design choices.

free parameters (1)
  • per-dataset evaluation sample size = 500 texts per dataset (2,500 total)
    Chosen to control API cost (Appendix C.2) rather than from a power analysis; all F1 comparisons are computed on these samples and no sampling seed is reported in the text.
assumptions (6)
  • domain assumption Benchmark labels in the five sexism datasets are treated as ground truth for sexism.
    Evaluation computes F1 against the original annotations of CallMeSexist, EDOS, RedditGuest, EXIST, and HateCheck (Section 3.4).
  • domain assumption GPT4o zero-shot outputs with the prompt template are a valid measurement of sexism detection.
    The modeling component assumes that prompt-based classification reflects construct detection (Section 3.4, Appendix C.3).
  • domain assumption The 500-text samples per benchmark are representative of the full datasets.
    The paper states the samples fairly represent performance and class balance (Section 3.4), but no sampling seed or repeated sampling is reported.
  • domain assumption The grounded-theory taxonomy derived by the authors is a valid characterization of expert strategies.
    Two author-annotators coded interactions and a third author coded a subset, but no inter-annotator agreement statistics are reported (Sections 4.2 and 4.3).
  • domain assumption Self-reported sexism expertise reflects actual expertise.
    Part I uses self-identification and adapted CSEM items to establish expertise (Section 3.1).
  • ad hoc to paper The co-created definition condition is meaningfully distinct from the LLM-generated condition.
    Appendix E and Table 8 show cosine similarity of 0.94-1.00 between co-created and LLM-generated definitions for most experts, yet the analysis treats them as separate conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tell Me What You Know About Sexism: Expert-LLM Interaction Strategies and Co-Created Definitions for Zero-Shot Sexism Detection." pith.science (2026). https://pith.science/paper/RFOD2ZTM

@misc{pith2026250415392,
  author       = {Pith},
  title        = {Pith review of: Tell Me What You Know About Sexism: Expert-LLM Interaction Strategies and Co-Created Definitions for Zero-Shot Sexism Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RFOD2ZTM}},
  note         = {Machine review of arXiv:2504.15392}
}
read the original abstract

This paper investigates hybrid intelligence and collaboration between researchers of sexism and Large Language Models (LLMs), with a four-component pipeline. First, nine sexism researchers answer questions about their knowledge of sexism and of LLMs. They then participate in two interactive experiments involving an LLM (GPT3.5). The first experiment has experts assessing the model's knowledge about sexism and suitability for use in research. The second experiment tasks them with creating three different definitions of sexism: an expert-written definition, an LLM-written one, and a co-created definition. Lastly, zero-shot classification experiments use the three definitions from each expert in a prompt template for sexism detection, evaluating GPT4o on 2.500 texts sampled from five sexism benchmarks. We then analyze the resulting 67.500 classification decisions. The LLM interactions lead to longer and more complex definitions of sexism. Expert-written definitions on average perform poorly compared to LLM-generated definitions. However, some experts do improve classification performance with their co-created definitions of sexism, also experts who are inexperienced in using LLMs.

Figures

Figures reproduced from arXiv: 2504.15392 by the authors.

Figure 1
Figure 1. Experts participate in a survey (part I) as well [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Explanation of the interactive experiments of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. F1 (macro) performance of GPT4o per participant over the three definitions (upper plot) and over the five datasets (bottom row). tion. When connecting results from all parts in the pipeline, we obtain additional insights. For instance, Expert 6 - less experienced with LLMs - was sometimes successful at co-creating definitions that perform better than the LLM-written defini￾tions. In contrast, the co-created definiti… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Heatmap of Likert scale on participants experience on LLMs. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of Likert scale auto-reported experience on sexism research. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Heatmap of Likert scale on suitability of the model in detecting sexism. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of Likert scale on the suitability fo the co-created definition. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Length comparison, per expert: Expert(-written), (LLM-)generated, Co-created [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Cosine Similarity Heatmaps between Datasets definitions and Expert-written (left) Co-created (center) [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Cosine Similarity Heatmaps between Datasets definitions and Expert-written (left), Co-created (center) [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Showing per-definition macro F1 for each dataset, with a plotted line for the macro-F1 of the majority class in each dataset. For most, this is non-sexist, though for some it is sexist. 0.0 0.2 0.4 0.6 0.8 1.0 accuracy on top of majority class % CallMeSexist 0.0 0.2 0…
Figure 12
Figure 12. Figure 12: Showing per dataset accuracy with a plotted line for the majority class in each dataset. For most datasets, the majority class is non-sexist, though for some it is sexist. G Effects of Temperature The temperature hyperparameter in generative Large Language Models affe…
Figure 13
Figure 13. Figure 13: Difference per participant over definition types (above) and datasets (bottom) for the modelling [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Difference per participant over definition types (above) and datasets (bottom) for the modelling [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 32 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2024. Prompt design matters for computational social science tasks but in unpredictable ways. arXiv preprint arXiv:2406.11980

    arXiv 2024

  4. [4]

    Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. https://aclanthology.org/2024.eacl-long.159 Sensitivity, Performance , Robustness : Deconstructing the Effect of Sociodemographic Prompting . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages ...

    2024

  5. [5]

    Godfred O Boateng, Torsten B Neilands, Edward A Frongillo, Hugo R Melgar-Qui \ n onez, and Sera L Young. 2018. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in public health, 6:149

    work page 2018

  6. [6]

    John Brewer. 2000. Ethnography. McGraw-Hill Education (UK)

    work page 2000

  7. [7]

    Kathy Charmaz. 2015. Grounded theory. Qualitative psychology: A practical guide to research methods, 3:53--84

    work page 2015

  8. [8]

    Gabriele de Seta, Matti Pohjonen, and Aleksi Knuutila. 2024. Synthetic ethnography: Field devices for the qualitative study of generative models. Big Data & Society, 11(4):20539517241303126

    work page 2024

Show all 56 references
  1. [9]

    Dominik Dellermann, Philipp Ebel, Matthias S \"o llner, and Jan Marco Leimeister. 2019. Hybrid intelligence. Business & Information Systems Engineering, 61(5):637--643

    2019

  2. [10]

    Alain D \'e silets, Louise Brunette, Christiane Melan c on, and Genevi \`e ve Patenaude. 2008. Reliable innovation: A tecchie’s travels in the land of translators. In Proceedings of the 8th Conference of the Association for Machine Translation in the Americas: Government and C...

    2008

  3. [11]

    Andrew Schwartz

    Gourab Dey, Adithya V Ganesan, Yash Kumar Lal, Manal Shah, Shreyashee Sinha, Matthew Matero, Salvatore Giorgi, Vivek Kulkarni, and H. Andrew Schwartz. 2024. https://aclanthology.org/2024.eacl-short.40 SOCIALITE - LLAMA : An instruction-tuned model for social scientific tasks ....

    2024

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Bahareh Fatemi, Fazle Rabbi, and Andreas L. Opdahl. 2023. https://doi.org/10.1109/ACCESS.2023.3345414 Evaluating the effectiveness of gpt large language model for news classification in the iptc news ontology . IEEE Access, 11:145386--145394

    2023

  6. [14]

    Nathan Gagn \'e and L \'e on Franzen. 2023. How to run behavioural experiments online: Best practice suggestions for cognitive psychology and neuroscience. Swiss Psychology Open: the official journal of the Swiss Psychological Society, 3(1)

    2023

  7. [15]

    Mirta Galesic. 2006. Dropouts on the web: Effects of interest and burden experienced during an online survey. Journal of official statistics, 22(2):313

    2006

  8. [16]

    Ella Guest, Bertie Vidgen, Alexandros Mittos, Nishanth Sastry, Gareth Tyson, and Helen Margetts. 2021. An expert annotated dataset for the detection of online misogyny. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisti...

    2021

  9. [17]

    Greg Guest, Arwen Bunce, and Laura Johnson. 2006. How many interviews are enough? an experiment with data saturation and variability. Field methods, 18(1):59--82

    2006

  10. [18]

    Monique Hennink and Bonnie N Kaiser. 2022. Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social science & medicine, 292:114523

    2022

  11. [19]

    Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61--83

    2010

  12. [20]

    John Hughes, Val King, Tom Rodden, and Hans Andersen. 1994. https://doi.org/10.1145/192844.193065 Moving out from the control room: ethnography in system design . In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, CSCW '94, page 429–439, New York...

    1994

  13. [21]

    Abigail Z Jacobs and Hanna Wallach. 2021. Measurement and fairness. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 375--385

    2021

  14. [22]

    Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. https://doi.org/10.1145/3544548.3581196 Co-writing with opinionated language models affects users’ views . In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '...

    2023

  15. [23]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  16. [24]

    Aiqi Jiang, Nikolas Vitsakis, Tanvi Dinkar, Gavin Abercrombie, and Ioannis Konstas. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.887 Re-examining sexism and misogyny classification with annotator attitudes . In Findings of the Association for Computational Linguistics...

    2024 doi

  17. [25]

    Urja Khurana, Eric Nalisnick, and Antske Fokkens. 2025. https://aclanthology.org/2025.coling-main.293/ D ef V erify: Do hate speech models reflect their dataset`s definition? In Proceedings of the 31st International Conference on Computational Linguistics, pages 4341--4358, Ab...

    2025

  18. [26]

    Urja Khurana, Ivar Vermeulen, Eric Nalisnick, Marloes Van Noorloos, and Antske Fokkens. 2022. https://doi.org/10.18653/v1/2022.woah-1.17 Hate speech criteria: A modular approach to task-specific hate speech definitions . In Proceedings of the Sixth Workshop on Online Abuse and...

    2022 doi

  19. [27]

    Hannah Rose Kirk, Wenjie Yin, Bertie Vidgen, and Paul R \"o ttger. 2023. Semeval-2023 task 10: Explainable detection of online sexism. arXiv preprint arXiv:2303.04222

    2023 arXiv

  20. [28]

    Katerina Korre, Arianna Muti, Federico Ruggeri, and Alberto Barr \'o n-Cede \ n o. 2025. Untangling hate speech definitions: A semantic componential analysis across cultures and domains. Findings of the Association for Computational Linguistics: NAACL 2025

    2025

  21. [29]

    Kalle Kusk and Claus Bossen. 2022. https://doi.org/10.1145/3492823 Working with wolt: An ethnographic study of lenient algorithmic management on a food delivery platform . Proc. ACM Hum.-Comput. Interact., 6(GROUP)

    2022 doi

  22. [30]

    Anne Lauscher, Debora Nozza, Ehm Miltersen, Archie Crowley, and Dirk Hovy. 2023. https://doi.org/10.18653/v1/2023.acl-long.23 What about `` em '' ? how commercial machine translation fails to handle (neo-)pronouns . In Proceedings of the 61st Annual Meeting of the Association ...

    2023 doi

  23. [31]

    Andreas Liesenfeld, Alianda Lopez, and Mark Dingemanse. 2023. Opening up chatgpt: Tracking openness, transparency, and accountability in instruction-tuned text generators. In Proceedings of the 5th international conference on conversational user interfaces, pages 1--6

    2023

  24. [32]

    Leping Liu. 2023. https://doi.org/10.18785/jetde.1601.03 Analyzing the Text Contents Produced by ChatGPT : Prompts , Feature - Components in Responses , and a Predictive Model . Journal of Educational Technology Development and Exchange, 16(1):49--70

    2023 doi

  25. [33]

    Riia Luhtanen and Jennifer Crocker. 1992. A collective self-esteem scale: Self-evaluation of one's social identity. Personality and social psychology bulletin, 18(3):302--318

    1992

  26. [34]

    Dhavalkumar Patel, Prem Timsina, Ganesh Raut, Robert Freeman, Matthew Levin, Girish Nadkarni, Benjamin S Glicksberg, and Eyal Klang. 2024. Exploring temperature effects on large language models across various clinical tasks. medRxiv, pages 2024--07

    2024

  27. [35]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? Proceedings of the 15th International Conference on Computational Creativity (ICCC'24)

    2024

  28. [36]

    Youri Peskine, Damir Korenčić, Ivan Grubisic, Paolo Papotti, Raphael Troncy, and Paolo Rosso. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.267 Definitions Matter : Guiding GPT for Multi -label Classification . In Findings of the Association for Computational Linguisti...

    2023 doi

  29. [37]

    N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  30. [38]

    Michael V Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. arXiv preprint arXiv:2304.11085

    2023 arXiv

  31. [39]

    Matthew Renze. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.432 The effect of sampling temperature on problem solving in large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346--7356, Miami, Florida, USA. Associatio...

    2024 doi

  32. [40]

    Myrthe Reuver, Antske Fokkens, and Suzan Verberne. 2021. https://www.aclweb.org/anthology/2021.hackashop-1.7 No NLP Task Should be an Island : Multi -disciplinarity for Diversity in News Recommender Systems . In Proceedings of the EACL Hackashop on News Media Content Analysis ...

    2021

  33. [41]

    Francisco Rodriguez-Sanchez, Jorge Carrillo de Albornoz, Laura Plaza, Julio Gonzalo, Paolo Rosso, Miriam Comet, and Trinidad Donoso. 2021. Overview of exist 2021: sexism identification in social networks. Procesamiento del Lenguaje Natural, 67(0)

    2021

  34. [42]

    Paul R \"o ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. https://doi.org/10.18653/v1/2021.acl-long.4 H ate C heck: Functional tests for hate speech detection models . In Proceedings of the 59th Annual Meeting of the Associatio...

    2021 doi

  35. [43]

    Call me sexist, but

    Mattia Samory, Indira Sen, Julian Kohne, Fabian Flöck, and Claudia Wagner. 2021. https://doi.org/10.1609/icwsm.v15i1.18085 “ Call me sexist, but...” : Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples . Proceedings of the International AAAI Confere...

    2021 doi

  36. [44]

    Anna Schj tt Hansen and Jannie M ller Hartley. 2023. Designing what’s news: An ethnography of a personalization algorithm and the data-driven (re) assembling of the news. Digital Journalism, 11(6):924--942

    2023

  37. [45]

    Indira Sen, Dennis Assenmacher, Mattia Samory, Isabelle Augenstein, Wil Aalst, and Claudia Wagner. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.649 People Make Better Edits : Measuring the Efficacy of LLM - Generated Counterfactually Augmented Data for Harmful Language De...

    2023 doi

  38. [46]

    Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.acl-long.244 On Second Thought , Let 's Not Think Step by Step ! Bias and Toxicity in Zero - Shot Reasoning . In Proceedings of the 61st Annual Meeting of the Ass...

    2023 doi

  39. [47]

    Carlo Simon, Stefan Haag, and Lara Zakfeld. 2023. Experiments on gpt-3 assisted process model development. ECMS, 2023:37th

    2023

  40. [48]

    Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11--21

    1972

  41. [49]

    Maartje Ter Hoeve, Julia Kiseleva, and Maarten Rijke. 2022. https://doi.org/10.18653/v1/2022.naacl-main.4 What Makes a Good and Useful Summary ? Incorporating Users in Automatic Summarization Research . In Proceedings of the 2022 Conference of the North American Chapter of the...

    2022 doi

  42. [50]

    Eddie Ungless, Bjorn Ross, and Anne Lauscher. 2023. https://doi.org/10.18653/v1/2023.findings-acl.502 Stereotypes and smut: The (mis)representation of non-cisgender identities by text-to-image models . In Findings of the Association for Computational Linguistics: ACL 2023, pag...

    2023 doi

  43. [51]

    Ding Wang, Santosh D Kale, and Jacki O'Neill. 2020. Please call the specialism: Using wechat to support patient care in china. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1--13

    2020

  44. [52]

    Maximilian Weber and Merle Reichardt. 2023. https://doi.org/10.48550/arXiv.2401.00284 Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences . A Primer using Open Models . ArXiv:2401.00284 [cs]

    2023 doi arXiv

  45. [53]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  46. [54]

    Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Chris Tanner. 2022. https://doi.org/10.18653/v1/2022.insights-1.10 What GPT knows about who is who . In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 75--81, Dublin, Ireland. Association for...

    2022 doi

  47. [55]

    JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1--21

    2023

  48. [56]

    Lining Zhang, Mengchen Wang, Liben Chen, and Wenxin Zhang. 2022. Probing gpt-3’s linguistic knowledge on semantic tasks. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 297--304

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.