Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Human raters systematically prefer text variants containing the lexical items that an LHF-trained Llama model overproduces relative to its base model, supporting the hypothesis that learning from human feedback drives LLM word overuse.

desk verdict Useful automated tool and a clean preference result, but the causal claim that LHF drives lexical overuse outruns the Base-vs-Instruct design. read the letter →

arxiv 2508.01930 v1 pith:DO2DVI67 submitted 2025-08-03 cs.CL cs.AI

Tom S. Juzek , Zina B. Ward This is my paper
classification cs.CLcs.AI
keywords lexicaloveruselearningfromhumanfeedbackRLHFdirectpreferenceoptimizationLlamapreferenceswordfrequencyalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why large language models overuse certain words such as 'delve' and 'intricate'—a phenomenon widely observed but not well understood. Its central claim is that learning from human feedback (LHF), the preference-training step that follows base training, is at least partly responsible. To support this, the authors first compare Meta's Llama 3.2-3B Base and Instruct models generating continuations of PubMed abstracts, identifying words the Instruct model uses far more often than the Base model. They then emulate the LHF task with 400 human raters drawn from Global South countries, who are asked to choose between abstract variants containing more or fewer of those words; the raters prefer the high-LHF-score variants 52.4% to 47.6%, a statistically significant difference. This result makes a causal story concrete: a slight human preference for certain words, amplified through preference optimization, becomes the overuse we see in deployed models.

What carries the argument

The paper's central instrument is the 'LHF-Score', a token-level weight assigned as the percent increase in a word's frequency between Llama Base and Llama Instruct, divided by 1000. Variants of abstracts generated by Llama Instruct are scored by summing these weights, and pairs are built by taking the lowest- versus highest-scoring variants for the same abstract. The experiment then treats those pairs in a binary preference task, analyzed by chi-square and a mixed-effects regression; the identification step relies on part-of-speech tagged corpora (spaCy) and GPT-4o for cleaning and summary-note generation.

What would settle it

If a model pair were tested in which the instruction-tuned variant was trained with the same data but without any preference optimization and the same lexical items still appeared overproduced, the LHF attribution would be undercut; equivalently, failure to reproduce the Base-vs-Instruct overuse pattern in another model family (e.g., OLMo) would show the result does not generalize.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that lexical overuse in LLMs is not a random artifact of training data but a consequence of the LHF procedure aligning models to human evaluator preferences. The authors show that the words overrepresented in Llama Instruct relative to Llama Base—'nuanced', 'firstly', 'reliance', 'generalizability', 'underscore', and hundreds more—are precisely the words that human raters prefer when choosing between otherwise matched abstract pairs. Emulating the LHF set-up (raters from the Global South, where LHF workers are often recruited, comparing A/B text variants), the experiment finds a 52.4% vs 47.6% preference for variants with more LHF-linked words. The authors interpret this as evidence that LHF is a primary source of the lexical biases observed in LLM-generated scientific text, and that the overuse reflects a divergence between the lexical expectations of LHF workers and of LLM users.

Load-bearing premise

That the word-usage differences between Llama Base and Llama Instruct are caused by learning from human feedback rather than by other training differences such as instruction tuning, tooling optimization, or safety mitigation.

Editorial extensions

If this is right

  • If LHF is a primary driver of lexical overuse, then models trained via DPO or RLHF on similar preference data will tend to overproduce whatever words their evaluators favor, making overuse a predictable and systematic outcome of alignment.
  • The LHF-Score procedure offers a low-cost, automated way to identify candidate overused words before deployment, which developers could use to spot-check or filter LHF datasets.
  • The divergence between LHF-worker preferences and LLM-user complaints implies that alignment is population-relative: a model can be well-aligned to its raters while seeming misaligned to its end users.
  • The overlap between the identified words and the literature's list of LLM-associated words (28 of 32) suggests that the Base-vs-Instruct comparison captures the same phenomenon that users perceive as word overuse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors' intergenerational language-change interpretation suggests a testable extension: comparing the same preference task across age groups and speaker populations to see whether preference strength varies with rater demographics.
  • Since the experiment used raters from the Global South to emulate LHF workers, a natural follow-up would test Global North raters on the same pairs; if the preference reverses or vanishes, the misalignment is specifically between populations rather than between humans and models.
  • The procedure's reliance on a single model family could be checked against other base/instruct pairs (e.g., OLMo or Falcon) to test whether the same words emerge as LHF-promoted across architectures and LHF implementations.
  • The authors leave open whether the effect is driven by the words themselves or by their stylistic correlates; a controlled version inserting the target words into otherwise identical human-written text would isolate the lexical contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether Learning from Human Feedback (LHF), including RLHF and DPO, is a cause of lexical overuse in LLMs. The authors first compare word frequencies in continuations generated by Llama 3.2-3B Base versus Instruct, identifying words with large relative increases (e.g., 'nuanced', 'firstly', 'reliance'). They define a per-token LHF-Score based on these percentage increases and use it to select 30 pairs of generated abstract variants that differ maximally in LHF-Score. They then recruit 400 Prolific participants from the Global South, who rate which variant they prefer in a forced-choice task. Participants prefer the high-LHF-Score variant at 52.4% versus 47.6% (chi-square p<0.01; mixed-effects model intercept p<0.001). The paper concludes that LHF is 'at least partly responsible' for lexical overuse in LLM outputs.

Significance. If the causal claim were established, this would be an important empirical contribution to the ongoing discourse on LLM lexical overuse, with implications for alignment research, data transparency, and intervention strategies. The study combines a corpus-based word-identification procedure with an original human experiment, and the authors release code and data, which is commendable. The paper also explicitly discusses population differences (Global South LHF workers vs. Global North users) and situates the work in xAI. However, the central inference from the Base-versus-Instruct contrast to LHF specifically is not warranted by the current design, and the experiment's post-hoc 'nuanced' finding complicates the main result.

major comments (3)
  1. [Section 2, paragraph beginning 'There are other differences between Llama Base and Llama Instruct'] The causal attribution to LHF is load-bearing but unsupported. Llama 3.2 Instruct differs from Llama 3.2 Base not only by DPO but also by instruction tuning (SFT), optimization for tooling, and safety mitigation, as the paper itself acknowledges. The assertion that 'none of these, including instruction tuning, are known to contribute to lexical overrepresentation' is an argument from ignorance, and the cited reference [38] is the authors' own supplementary material rather than independent evidence. If SFT alone shifts the model toward the same lemmas, then the Section 3 preference result would implicate instruction tuning, not LHF. The conclusion in Section 6 that LHF is 'at least partly responsible' therefore overstates what the comparison can establish. The authors should either compare against a model that differs only in LHF (e.g., a Base+SFT checkpoint without DPO/RLHF) or substantially temper the causal claim to 'post-training' or 'instruction-tuned model' effects.
  2. [Section 3.3 (Results) and Figure 3(b)] The post-hoc finding about 'nuanced_ADJ' is not a minor anomaly. 'nuanced_ADJ' is the word with the largest relative increase in Table 1 (+8342%), so it contributes heavily to the LHF-Score Deltas used to select the 30 item pairs. The paper reports that items containing 'nuanced' show a reversed preference (46.6% for the high-LHF-Score variant) compared to items without it (54.5%). This suggests that the overall preference effect is not uniform across the identified words and may be driven by a subset of items while the most heavily weighted word actually reverses the effect. The authors should report the main analysis excluding the 'nuanced' items, and ideally a per-word or per-item analysis, to establish that the effect is not an artifact of a few items. Without this, the statement that 'participants systematically and significantly prefer variants containing more of the items identified' is potentially misleading.
  3. [Section 3.1, Eq. (1) (LHF-Score definition) and stimulus selection] The LHF-Score weights each token by its percentage increase between Base and Instruct, which makes extremely rare words with large relative increases (e.g., 'radar_NOUN', 'theanine_NOUN', 'staffing_NOUN' in Table 1) dominate the score. Because the 30 item pairs are then selected to maximize LHF-Score Deltas, the stimulus set may be heavily influenced by idiosyncratic topic-related words rather than the general, widely discussed 'delve'-type overuse. The paper acknowledges in Section 5 that some identified items are likely artifacts of corpus size, and it removes a list of known AI words from the variants, but it does not restrict the LHF-Score to words that are also overused relative to human text or that appear in the literature-derived list (Section 4). The experimental result may therefore show only that participants prefer texts with more of these particular artifacts, not that LHF promotes words humans generally prefer. The authors should demonstrate that the preference result holds when restricting the analysis to words that overlap with the literature list or with the human-baseline comparison.
minor comments (5)
  1. [Section 3.3] In the paragraph beginning 'Although we did not initially intend...', the term 'high RP-Score' appears to be a typo for 'high LHF-Score'; please correct for consistency.
  2. [Table 1] The lemma 'multifacete_VERB' appears to be a misspelling (likely 'multifacet' or 'multifaceted'); check the lemma and the spaCy output.
  3. [Section 3.1] The description of removing variants containing 'the 21 most overused AI words as discussed in [9]' is vague; please specify the exact list and criterion, and clarify whether this removal applied to both variants in each pair (as implied) or only to one condition.
  4. [Section 3.1 (Exclusions)] The exclusion rate of 46.8% is high, and while prior work is cited as precedent, it would strengthen the report to include a robustness analysis with alternative exclusion thresholds or with no exclusions, to show that the main result is not dependent on the specific filtering decisions.
  5. [Figure 3 caption] The caption 'High LHF-Score Delta pairs contained "nuanced_ADJ"' is confusing; it is unclear whether all pairs with high Delta contain that word or only the points circled/colored in the figure. Please clarify the visual encoding.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the human-preference experiment is an external, falsifiable test; only the causal label 'LHF-induced' rests on a self-cited assumption.

full rationale

The paper's central experimental result is not circular. The LHF-Score used to construct stimulus pairs is computed from the Section 2 Base-versus-Instruct frequency differences, but the outcome variable — participants' binary preferences between high- and low-scoring variants — is an external behavioral measurement that could have come out the other way. The reported 52.4% preference for high-LHF-Score variants is an empirical fact, not a consequence of the scoring formula. The preference result therefore does not reduce by construction to the corpus contrast that generated the word list. The main weakness is the causal inference: Section 2 labels the words 'potentially LHF-induced' purely from the Base/Instruct comparison, and the paper dismisses instruction tuning, tooling optimization, and safety mitigation by citing its own supplementary material [38] rather than a controlled comparison. This is a self-citation and a construct-validity concern, making 'LHF remains the most plausible contributor' an assumption rather than a demonstrated fact. However, this is a confound or correctness risk, not a circular derivation: the experimental preference data are independent of that assumption. The paper itself acknowledges in the Limitations section that 'other contributing factors remain to be systematically investigated' and that the connection to human language use is 'somewhat preliminary.' No equation is defined in terms of the conclusion, and no fitted parameter is relabeled as a prediction. Accordingly, no significant circularity is present; the score of 2 reflects the minor self-citation used in the causal interpretation without making the experimental finding circular.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the assumption that the Base-Instruct difference isolates LHF, and on the proxy assumption for LHF workers. No new physical or conceptual entities are introduced. The free parameters listed are hand-chosen thresholds that affect exclusions and scoring but are not fitted to the outcome.

free parameters (2)
  • LHF-Score scaling divisor (1000) = 1000
    Chosen by hand for interpretability; arbitrary scaling that does not affect the ordering of variants.
  • Quick-reading exclusion threshold = 40% of predicted minimum reading time
    Exclusion threshold chosen by the authors; affects which participants are retained and could influence the main result.
assumptions (3)
  • domain assumption Base and Instruct models differ primarily in LHF, and instruction tuning, tooling, and safety mitigation do not affect lexical choice.
    The paper acknowledges these other differences but asserts, without evidence, that they do not contribute to lexical overrepresentation. This is load-bearing for attributing the observed word list to LHF.
  • domain assumption Prolific participants from the Global South are a valid proxy for actual LHF workers.
    The paper recruits from countries resembling the presumed LHF worker population, but provides no evidence that these participants perform similar tasks or hold similar preferences to real LHF workers.
  • standard math The chi-square test and mixed-effects model are appropriate for the binary preference data.
    Standard statistical tools for categorical preference data; the test choice is not controversial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback." pith.science (2026). https://pith.science/paper/DO2DVI67

@misc{pith2026250801930,
  author       = {Pith},
  title        = {Pith review of: Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DO2DVI67}},
  note         = {Machine review of arXiv:2508.01930}
}
read the original abstract

Large Language Models (LLMs) are known to overuse certain terms like "delve" and "intricate." The exact reasons for these lexical choices, however, have been unclear. Using Meta's Llama model, this study investigates the contribution of Learning from Human Feedback (LHF), under which we subsume Reinforcement Learning from Human Feedback and Direct Preference Optimization. We present a straightforward procedure for detecting the lexical preferences of LLMs that are potentially LHF-induced. Next, we more conclusively link LHF to lexical overuse by experimentally emulating the LHF procedure and demonstrating that participants systematically prefer text variants that include certain words. This lexical overuse can be seen as a sort of misalignment, though our study highlights the potential divergence between the lexical expectations of different populations -- namely LHF workers versus LLM users. Our work contributes to the growing body of research on explainable artificial intelligence and emphasizes the importance of both data and procedural transparency in alignment research.

Figures

Figures reproduced from arXiv: 2508.01930 by the authors.

Figure 1
Figure 1. An illustration of the procedure used to identify lexical preferences that are potentially induced by Learning from Human Feedback (LHF); created with Canva. pointed to Learning from Human Feedback (LHF) as a significant contributor to these lexical choices [14,15], conclusive evidence is still missing. Learning from Human Feedback is a procedure applied after initial model training during which human evaluators ind… view at source ↗
Figure 2
Figure 2. The rating interface for our experiment. The Task. The task began with IRB information (full instructions can be found on our GitHub), followed by an introduction to the task (“In the following, you will read a series of research summaries, with two alternatives next to each other. Please express which alternative you overall prefer. Some of the items are hard, do the best you can!”, with an example as per [PITH_FU… view at source ↗
Figure 3
Figure 3. (a) Experimental results: Preferences between low LHF-Score variant vs. high LHF-Score variant, for the 30 items. (b) Participant preferences for pairs with different LHF-Score Deltas. Each dot represents the mean preference for one of 30 abstract pairs. High LHF-Score Delta pairs contained "nuanced_ADJ." [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Introduces a triangulation-based metric to quantify lexical shifts attributable to preference tuning without requiring manual curation of examples.

Reference graph

Works this paper leans on

73 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [38]

    Wordoveruseandalignment in large language models: The influence of learning from human feedback

    Juzek,T.S.,Ward,Z.B.:Supplementarymaterialsfor"Wordoveruseandalignment in large language models: The influence of learning from human feedback". OSF, https://osf.io/4nvjk https://doi.org/10.17605/OSF.IO/4NVJK (2025) Word Overuse, LLMs, and LHF 15

    work page doi:10.17605/osf.io/4nvjk 2025

  2. [1]

    https://x.com/PKoppenburg/status/ 1774757167045788010, last accessed 2024/08/12

    Koppenburg, P.: Tweet on 01 April 2024. https://x.com/PKoppenburg/status/ 1774757167045788010, last accessed 2024/08/12

    work page 2024

  3. [2]

    https://x.com/JeremyNguyenPhD/status/ 1774021645709295840, last accessed 2024/08/12

    Nguyen, J.: Tweet on 30 March 2024. https://x.com/JeremyNguyenPhD/status/ 1774021645709295840, last accessed 2024/08/12

    work page 2024

  4. [3]

    https://pshapira.net/2024/03/31/ delving-into-delve/, last accessed 2024/09/21

    Shapira, P.: Delving into "delve". https://pshapira.net/2024/03/31/ delving-into-delve/, last accessed 2024/09/21

    work page 2024

  5. [4]

    contamination

    Gray, A.: ChatGPT "contamination": Estimating the prevalence of LLMs in the scholarly literature. arXiv preprint arXiv:2403.16887 (2024)

    arXiv 2024

  6. [5]

    arXiv preprint arXiv:2406.07016 (2024)

    Kobak, D., González Márquez, R., Horvát, E.-Á., Lause, J.: Delving into Chat- GPT usage in academic writing through excess vocabulary. arXiv preprint arXiv:2406.07016 (2024)

    arXiv 2024

  7. [6]

    et al.: Mapping the increasing use of LLMs in scientific papers

    Liang, W. et al.: Mapping the increasing use of LLMs in scientific papers. arXiv preprint arXiv:2404.01268 (2024)

    arXiv 2024

  8. [7]

    arXiv:2404.15799 (2024)

    Liu, J., Bu, Y.: Towards the relationship between AIGC in manuscript writing and author profiles: Evidence from preprints in LLMs. arXiv:2404.15799 (2024)

    arXiv 2024

Show all 73 references
  1. [8]

    medRxiv (2024)

    Matsui, K.: Delving into PubMed Records: Some Terms in Medical Writing Have Drastically Changed after the Arrival of ChatGPT. medRxiv (2024)

    2024

  2. [9]

    In Proceedings of the 31st International Conference on Computational Linguistics (pp

    Juzek, T.S., Ward, Z.B.: Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics (pp. 6397-6411). https://doi.org/10.48550/arXiv.2412.11385 (2025)

    2025 doi arXiv

  3. [10]

    In: Proc

    Degaetano-Ortlieb, S., Teich, E.: Using relative entropy for detection and analysis of periods of diachronic linguistic change. In: Proc. 2nd Joint SIGHUM Workshop, pp. 22–33 (2018)

    2018

  4. [11]

    In: From Data to Evidence in English Language Research, pp

    Degaetano-Ortlieb,S.,Kermes,H.,Khamis,A.,Teich,E.:Aninformation-theoretic approach to modeling diachronic change in scientific English. In: From Data to Evidence in English Language Research, pp. 258–281. Brill, Leiden (2018)

    2018

  5. [12]

    Bizzoni, Y., Degaetano-Ortlieb, S., Fankhauser, P., Teich, E.: Linguistic variation and change in 250 years of English scientific writing: A data-driven approach. Front. Artif Intell. 3(73) (2020)

    2020

  6. [13]

    In: Corpus Pragmatic Studies on the History of Medical Discourse, pp

    Menzel, K.: Medical discourse in Late Modern English: Insights from a multidis- ciplinary corpus of scientific journal articles. In: Corpus Pragmatic Studies on the History of Medical Discourse, pp. 79–104. John Benjamins, Amsterdam (2022)

    2022

  7. [14]

    https://www.theguardian.com/technology/2024/apr/16/ techscape-ai-gadgest-humane-ai-pin-chatgpt, last accessed 2024/08/12

    Hern, A.: TechScape: How cheap, outsourced labour in Africa is shap- ing AI English. https://www.theguardian.com/technology/2024/apr/16/ techscape-ai-gadgest-humane-ai-pin-chatgpt, last accessed 2024/08/12

    2024

  8. [15]

    https: //hesamsheikh.substack.com/p/why-does-chatgpt-use-delve-so-much,lastaccessed 2025/01/14 14 T

    Sheikh, H.: Why does ChatGPT use “Delve” so much? Mystery Solved. https: //hesamsheikh.substack.com/p/why-does-chatgpt-use-delve-so-much,lastaccessed 2025/01/14 14 T. S. Juzek & Z. B. Ward

    2025

  9. [16]

    Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. In: Adv. Neural Inf. Process. Syst. (30) (2017)

    2017

  10. [17]

    arXiv preprint arXiv:1909.08593 (2019)

    Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Chris- tiano, P., Irving, G.: Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019)

    2019 arXiv

  11. [18]

    et al.: Direct preference optimization: Your language model is secretly a reward model

    Rafailov, R. et al.: Direct preference optimization: Your language model is secretly a reward model. In: Adv. Neural Inf. Process. Syst. (36) (2024)

    2024

  12. [19]

    He, Z., Guo, S., Rao, A., Lerman, K.: Whose Emotions and Moral Sentiments Do Language Models Reflect? arXiv preprint arXiv:2402.11114 (2024)

    2024 arXiv

  13. [20]

    2021 ACM Conf

    Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: Proc. 2021 ACM Conf. on Fairness, Accountability, and Transparency, pp. 610–623 (2021)

    2021

  14. [21]

    Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., Hashimoto, T.: Whose opinions do language models reflect? In: Int. Conf. on Machine Learning (ICML), pp. 29971–30004 (2023)

    2023

  15. [22]

    et al.: Towards measuring the representation of subjective global opin- ions in language models

    Durmus, E. et al.: Towards measuring the representation of subjective global opin- ions in language models. arXiv preprint arXiv:2306.16388 (2023)

    2023 arXiv

  16. [23]

    et al.: Training language models to follow instructions with human feedback

    Ouyang, L. et al.: Training language models to follow instructions with human feedback. In: Adv. Neural Inf. Process. Syst. (35), 27730–27744 (2022)

    2022

  17. [24]

    et al.: On the opportunities and risks of foundation models

    Bommasani, R. et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

    2021 arXiv

  18. [25]

    arXiv preprint arXiv:2409.13686 (2024)

    Geng, M., Chen, C., Wu, Y., Chen, D., Wan, Y., Zhou, P.: The impact of large language models in academia: from writing to speaking. arXiv preprint arXiv:2409.13686 (2024)

    2024 arXiv

  19. [26]

    arXiv preprint arXiv:2409.01754 (2024)

    Yakura, H., Lopez-Lopez, E., Brinkmann, L., Serna, I., Gupta, P., Rahwan, I.: Empirical evidence of large language model’s influence on human spoken communi- cation. arXiv preprint arXiv:2409.01754 (2024)

    2024 arXiv

  20. [27]

    arXiv preprint arXiv:2506.01407 (2025)

    Zamaraeva, O., Flickinger, D., Bond, F., Gómez-Rodríguez, C.: Comparing LLM- generated and human-authored news text using formal syntactic theory. arXiv preprint arXiv:2506.01407 (2025)

    2025 arXiv

  21. [28]

    et al.: Hidden technical debt in machine learning systems

    Sculley, D. et al.: Hidden technical debt in machine learning systems. In: Adv. Neural Inf. Process. Syst. (28) (2015)

    2015

  22. [29]

    et al.: Explainability for large language models: A survey

    Zhao, H. et al.: Explainability for large language models: A survey. ACM Trans- actions on Intelligent Systems and Technology 15(2), 1–38 (2024)

    2024

  23. [30]

    et al.: XAI meets LLMs: A survey of the relation between explainable AI and large language models

    Cambria, E. et al.: XAI meets LLMs: A survey of the relation between explainable AI and large language models. arXiv preprint arXiv:2407.15248 (2024)

    2024 arXiv

  24. [31]

    arXiv:2409.11704 (2024)

    Zhang, X., Xiong, W., Chen, L., Zhou, T., Huang, H., Zhang, T.: From lists to emojis: How format bias affects model alignment. arXiv:2409.11704 (2024)

    2024 arXiv

  25. [32]

    Erdocia, I., Migge, B., Schneider, B.: Language is not a data set—Why overcoming ideologies of dataism is more important than ever in the age of AI. J. Sociol. (2024)

    2024

  26. [33]

    et al.: The LLaMA 3 herd of models

    Dubey, A. et al.: The LLaMA 3 herd of models. arXiv:2407.21783 (2024)

    2024 arXiv

  27. [34]

    et al.: Transformers: State-of-the-Art Natural Language Processing

    Wolf, T. et al.: Transformers: State-of-the-Art Natural Language Processing. arXiv preprint arXiv:1910.03771 (2020)

    2020 arXiv

  28. [35]

    Hugging Face Team: Open LLM Leaderboard. (2024). https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

  29. [36]

    Allen Institute for AI. (2024). OLMo 2. https://allenai.org/olmo

    2024

  30. [37]

    TechnologyInnov.Inst.(2024).Falcon3.https://falconllm.tii.ae/falcon3/index.html

    2024

  31. [39]

    https://www.python.org/, accessed 2024

    Python Software Foundation: Python 3. https://www.python.org/, accessed 2024

    2024

  32. [40]

    https://pubmed.ncbi.nlm.nih

    National Library of Medicine: PubMed Database. https://pubmed.ncbi.nlm.nih. gov/, last accessed 2024/11/24

    2024

  33. [41]

    et al.: GPT-4 technical report

    Achiam, J. et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  34. [42]

    Version 1.57

    OpenAI: OpenAI Python API. Version 1.57. https://platform.openai.com/docs/, last accessed 2025/01/18

    2025

  35. [43]

    Montani, I., Honnibal, M., Boyd, A., Van Landeghem, S., Peters, H.: explo- sion/spaCy: v3.7.2: Fixes for APIs and requirements. Zenodo. https://doi.org/10. 5281/zenodo.10009823 (2023)

    2023

  36. [44]

    https://books.google.com/ngrams/, last ac- cessed 2025/01/02

    Google: Google Books Ngram Viewer. https://books.google.com/ngrams/, last ac- cessed 2025/01/02

    2025

  37. [45]

    Pan 8(4), pp

    Lavergne, T., Urvoy, T., Yvon, F.: Detecting fake content with relative entropy scoring. Pan 8(4), pp. 27–31 (2008)

    2008

  38. [46]

    et al.: On the possibilities of AI-generated text detection

    Chakraborty, S. et al.: On the possibilities of AI-generated text detection. arXiv preprint arXiv:2304.04736 (2023)

    2023 arXiv

  39. [47]

    Mitchell, E., Lee, Y., Khazatsky, A., Manning, C.D., Finn, C.: DetectGPT: Zero- shot machine-generated text detection using probability curvature. In: Int. Conf. on Machine Learning (ICML), pp. 24950–24962 (2023)

    2023

  40. [48]

    et al.: MAGRET: Machine-generated Text Detection with Rewritten Texts

    Huang, Y. et al.: MAGRET: Machine-generated Text Detection with Rewritten Texts. In: Proc. COLING 2025, pp. 8336–8346 (2025)

    2025

  41. [49]

    Springer, CH (2016)

    Haslwanter, T.: An Introduction to Statistics with Python. Springer, CH (2016)

    2016

  42. [50]

    Race Class 60(4), 3–26 (2019)

    Kwet, M.: Digital colonialism: US empire and the new imperialism in the Global South. Race Class 60(4), 3–26 (2019)

    2019

  43. [51]

    Time Magazine (18) (2023)

    Perrigo, B.: Exclusive: OpenAI used Kenyan workers on less than $2 per hour to make ChatGPT less toxic. Time Magazine (18) (2023)

    2023

  44. [52]

    et al.: Broadening the perspective for sustainable artificial intelligence: Sustainability criteria and indicators for Artificial Intelligence systems

    Rohde, F. et al.: Broadening the perspective for sustainable artificial intelligence: Sustainability criteria and indicators for Artificial Intelligence systems. Curr. Opin. Environ. Sustain. (66), 101411 (2024)

    2024

  45. [53]

    Sage, Thousand Oaks (1997)

    Cowart, W.: Experimental Syntax. Sage, Thousand Oaks (1997)

    1997

  46. [54]

    Berinsky, A.J., Margolis, M.F., Sances, M.W.: Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. Am. J. Polit. Sci. 58(3), 739–753 (2014)

    2014

  47. [55]

    Maniaci, M.R., Rogge, R.D.: Caring about carelessness: Participant inattention and its effects on research. J. Res. Pers. (48), 61–83 (2014)

    2014

  48. [56]

    In: Proc

    Friedman, H.H., Herskovitz, P.J., Pollack, S.: The biasing effects of scale-checking styles on response to a Likert scale. In: Proc. Amer. Stat. Assoc. Conf. on Survey Research Methods (792), pp. 792–795 (1994)

    1994

  49. [57]

    Chyung, S.Y., Kennedy, M., Campbell, I.: Evidence-based survey design: The use of ascending or descending order of Likert-type response options. Perform. Improv. 57(9), 9–16 (2018)

    2018

  50. [58]

    Language 92(3), 619–635 (2016)

    Mahowald, K., Graff, P., Hartman, J., Gibson, E.: SNAP judgments: A small N acceptability paradigm (SNAP) for linguistic acceptability judgments. Language 92(3), 619–635 (2016)

    2016

  51. [59]

    In: Featherston, S., Hörnig, R., Steinberg, R., Umbreit, B., Wallis, J

    Häussler, J., Juzek, T.: Hot topics surrounding acceptability judgement tasks. In: Featherston, S., Hörnig, R., Steinberg, R., Umbreit, B., Wallis, J. (eds.) Linguistic Evidence 2016: Empirical, Theoretical, and Computational Perspectives. University of Tübingen, Tübingen. htt...

    2017 doi

  52. [60]

    In: Proc

    Downs,J.S.,Holbrook,M.B.,Sheng,S.,Cranor,L.F.:Areyourparticipantsgaming the system? Screening Mechanical Turk workers. In: Proc. SIGCHI Conf. on Human Factors in Computing Systems, pp. 2399–2402 (2010) 16 T. S. Juzek & Z. B. Ward

    2010

  53. [61]

    In: SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pp

    Zhu,D.,Carterette,B.:Ananalysisofassessorbehaviorincrowdsourcedpreference judgments. In: SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pp. 17–20 (2010)

    2010

  54. [62]

    In: Proc

    Kazai, G., Kamps, J., Milic-Frayling, N.: Worker types and personality traits in crowdsourcing relevance labels. In: Proc. 20th ACM Int. Conf. on Information and Knowledge Management, pp. 1941–1944 (2011)

    2011

  55. [63]

    Thomas, K.A., Clifford, S.: Validity and Mechanical Turk: An assessment of ex- clusion methods and interactive experiments. Comp. Hum. Behav. (77), 184–197 (2017)

    2017

  56. [64]

    ACM Comput

    Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., Allahbakhsh, M.: Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Surv. 51(1), 1–40 (2018)

    2018

  57. [65]

    3: Cognitive and Cultural Factors

    Labov, W.: Principles of Linguistic Change, vol. 3: Cognitive and Cultural Factors. Wiley, Hoboken (2011)

    2011

  58. [66]

    et al.: The Role of AI in Peer Support for Young People: A Study of PreferencesforHuman-andAI-GeneratedResponses.In:Proc.CHIConf.onHuman Factors in Computing Systems, pp

    Young, J. et al.: The Role of AI in Peer Support for Young People: A Study of PreferencesforHuman-andAI-GeneratedResponses.In:Proc.CHIConf.onHuman Factors in Computing Systems, pp. 1–18 (2024)

    2024

  59. [67]

    Wu, M., Aji, A.F.: Style Over Substance: Evaluation Biases for Large Language Models. In Proc. COLING 2025, pp. 297–312. Association for Computational Lin- guistics, Abu Dhabi, UAE. https://aclanthology.org/2025.coling-main.21/ (2025)

    2025

  60. [68]

    hMensa, P.A.: Artificial intelligence and the future of sociolinguistic research: An African contextual review. J. Socioling. (2024)

    2024

  61. [69]

    Anthropic (2024)

    Templeton, A.: Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Anthropic (2024)

    2024

  62. [70]

    Toxtli, C., Suri, S., Savage, S.: Quantifying the invisible labor in crowd work. Proc. ACM Hum.-Comput. Interact. 5 (CSCW2), 1–26 (2021)

    2021

  63. [71]

    https://www.accel.ai/ anthology/2023/5/22/jyzu7sbpzyxufu5l1ekidxj0g7jafh, last accessed 2023

    Roberts, J.: The Precarious Human Work Behind AI. https://www.accel.ai/ anthology/2023/5/22/jyzu7sbpzyxufu5l1ekidxj0g7jafh, last accessed 2023

    2023

  64. [72]

    https: //change-links.org/a-i-s-dirty-secret-its-powered-by-digital-sweatshops/, last ac- cessed 2023

    Novick, M.: A.I.’s Dirty Secret: It’s Powered by Digital Sweatshops. https: //change-links.org/a-i-s-dirty-secret-its-powered-by-digital-sweatshops/, last ac- cessed 2023

    2023

  65. [73]

    Jim the AI Whisperer: How One Sentence Pattern Can Expose AI Writing. Medium. https://generativeai.pub/ how-to-spot-ai-writing-with-one-sentence-pattern-8aa5b3ec5a63, accessed 2024/12 A Appendix Permitted Countries : Bangladesh, Belize, Botswana, Cameroon, Ethiopia, Fiji, Gamb...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.