Pith. sign in

REVIEW 3 major objections 4 minor 4 references

Beyond "AI Language": The case for the idiolectal nature of LLM output

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM output is not one 'AI language': each model carries a stable idiolect, and the 2026 generation shares a new stylistic voice.

desk verdict A genuinely new 2026 corpus and a clean diachronic comparison, but the idiolect claim overreaches because stability across topics is never tested. read the letter →

arxiv 2608.06589 v1 pith:YID4QKRE submitted 2026-08-06 cs.CL cs.AI

Karolina Rudnicka , Thomas Stephan Juzek This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMidiolectsstylometryprincipalcomponentanalysiscontractionslanguagevariationandchangeAI-generatedtextcorpuslinguisticsmodelgenerations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the output of a single large language model is best understood not as part of one collective 'AI language' but as a model-specific variety similar to a human idiolect: a stable, distinctive way of using language. It backs this with two corpora of model-generated essays on societal topics, one from 2024 and one newly generated from 2026 using the same prompts. Stylometric principal component analysis separates the two cohorts on the first axis, while each model occupies its own region of the stylistic space. A contraction-frequency case study sharpens the point: rates range from 120 to over 30,000 per million words, and models differ in which contraction types they favor. If right, this reframes AI-text comparison, detection, and forensic attribution around individual tools and their versions rather than a single generic 'AI voice'.

What carries the argument

The load-bearing concept is the LLM idiolect: an individual model's distinctive, stable, and emergent way of using language, following the Richards, Platt and Platt definition. It is made measurable by two instruments. First, a stylometric PCA on the 150 most frequent words, culled at 50% and computed on a correlation matrix, using 10,000-word samples from the health-misinformation topic; PC1 separates generations and per-model ellipses measure internal consistency. Second, a contraction taxonomy of six categories ('t, 's, 'm, 're, 've, 'd) searched with a custom script covering eight apostrophe glyphs, giving normalized per-million-word rates that expose model-specific fingerprints. The replication rerun of Mistral-7B with different seeds is the control showing that the recovered signature is stable across generation runs.

What would settle it

Recompute the same PCA and contraction counts on texts the same 13 models produce for a second, unrelated topic (for instance, climate change) under the same prompt-and-sampling protocol. If the 95% confidence ellipses of different models overlap on that topic, or if PC1 no longer separates the 2024 from the 2026 cohort, the observed 'idiolects' are topic-bound registers rather than stable model voices.

Watch

Extended reading notes

Core claim

The central claim is that each LLM-based tool has its own idiolect—defined as an emergent, relatively stable, unique way of communicating—and that alongside these individual idiolects there is a cohort-level generational shift. On health-misinformation essays, PCA of the 150 most frequent words separates all six 2024 models on the positive side of PC1 (37.6% of variance) and five of six 2026 models on the negative side, with only Mistral-Nemo-12B placed with the older cohort. Within that space each model's samples form its own ellipse; re-running Mistral-7B with different seeds lands near the original, indicating a recoverable signature. Contraction counts reinforce the picture: overall frequencies span 120.2 to 30,611.9 per million words, negation contractions are roughly 32 times more frequent in the 2026 cohort, and individual models show distinct category profiles, such as Gemini-3-Flash skewing toward "n't" while other models favor "'s". The paper concludes the idiolect frame captures stability, uniqueness, and emergence better than style or a single AI super-variety.

Load-bearing premise

The paper assumes a model's idiolect stays relatively stable across topics and contexts, but all the profile evidence comes from one topic (health misinformation), so the model-specific profiles could be topic-register effects rather than stable individual voices.

Editorial extensions

If this is right

  • LLM-generated text detection and attribution can treat each model version as a separate author class instead of hunting for a single generic 'AI voice.'
  • Human-vs-AI comparative studies should name the specific model and version, since the 120-to-30,000-per-million contraction spread shows one cohort contains radically different defaults.
  • The clean 2024/2026 split on PC1 means model generation is a real variable: detectors and style corpora calibrated on older output will misjudge newer text unless time-stamped.
  • Variationist and usage-based linguistics can study model families as lineages, with family-level continuities occupying a middle level between the individual idiolect and the super-variety.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cross-topic replication using the same 13 models on a second, unrelated prompt set would test the definitional claim of stability: if per-model ellipses separate within a topic but merge across topics, the reported profiles are topic-register effects.
  • If the idiolects survive cross-topic testing, model versions become usable as 'speakers' for corpus-based studies of language change, letting researchers trace family-level stylistic drift across releases and generations.
  • The contraction taxonomy is a cheap attribution signal—Gemini's negation-heavy profile versus Claude's broad contraction use—but it needs validation across prompts, temperatures, and lengths before it can support forensic claims.
  • Some of the 2026-vs-2024 shift could be an artifact of deployment rather than training-data drift, since closed-API models are queried through vendor systems that may add their own preprompts; a controlled open/closed comparison with identical system-prompt strings would isolate this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper argues that the output of individual LLM-based tools is better analysed as a model-specific idiolect than as a single collective 'AI language'. Using the 2024 TextualLLMap corpus and a newly generated 2026 corpus with the same prompts and topics, the authors report computational descriptors (length, sentence length, TTR, vocabulary size, disclaimers), a stylometric PCA on word-frequency profiles, and a targeted contraction analysis. They find a generational shift on PC1 (37.6% of variance) separating most 2024 from most 2026 models, while also reporting model-specific profiles, illustrated most dramatically by contraction frequencies ranging from 120.2 to 30,611.9 per million words. The chapter concludes that the idiolect notion is empirically warranted and useful for variation, detection, and forensic linguistics.

Significance. If its central claim holds, the paper makes a useful contribution by shifting attention from an undifferentiated 'AI language' to model-specific, stable stylistic profiles, with concrete implications for LLM-generated text detection, forensic attribution, and usage-based accounts of variation and change. The study has notable strengths: it uses a newly generated 2026 corpus with prompts and settings matched as closely as possible to the 2024 corpus; it includes a Mistral-7B replication control that lands in nearly identical PCA coordinates; it documents and handles a genuine technical problem (the multiplicity of Unicode apostrophe glyphs in LLM output); and the empirical analyses are direct measurements of newly generated text rather than quantities derived to force the conclusion. The principal weakness is that the empirical core—PCA and contraction analyses—is restricted to a single topic (health misinformation), so the paper's own definition of idiolect as stable 'across topics and contexts' is not tested.

major comments (3)
  1. [§2.1, §5.1, §6.1] The paper's definition of idiolect requires a profile that 'stays relatively stable across topics and contexts' (Section 2.1), but the load-bearing evidence for distinct model profiles comes exclusively from one topic. The PCA in Section 5.1 deliberately restricts the corpus to health misinformation, and the contraction analysis in Section 6.1 uses the same single-domain corpus. The result could therefore be a topic-register effect—each model having a characteristic way of writing about health misinformation—rather than a stable cross-topic voice. The data needed to test stability already exist: Section 3.1 states that four topics were generated per model, and Section 4 uses all four for length, TTR, and disclaimer analyses, but no cross-topic stylometric or contraction comparison is reported. Without such a comparison, the central idiolect claim overreaches the evidence.
  2. [§5.2, Fig. 5.1] The claim that 'each individual model maintains a unique linguistic profile' is not supported by any statistical separability or significance test. The interpretation rests on visual inspection of 95% confidence ellipses in Figure 5.1, and the figure's own alt-text acknowledges that some ellipses are 'larger, more elongated, or overlapping.' For overlapping models, 'unique profile' is not demonstrated. I would ask for a quantitative separation analysis—for example, a permutation test on pairwise Mahalanobis distances, a MANOVA on the component scores, or a cross-validated classifier accuracy—so that readers can see which model pairs are actually distinguishable and at what confidence level.
  3. [§6.3, Table 6.1] The contraction results, while vividly illustrating variation, are also based only on the health-misinformation corpus and are reported as aggregate corpus-level counts per model. The strong generational claim for negation contractions ('t, a ratio of roughly 32:1 between 2026 and 2024 means) is thus subject to the same single-topic limitation as the PCA. It is also heavily driven by one model, Claude-Haiku-4-5; the authors do note that excluding it leaves a 9:1 ratio, but no confidence intervals or per-text variability measures are provided. Treating each model's full output as a single pooled corpus does not allow assessment of within-model consistency across texts or topics, which is precisely what the idiolect claim requires.
minor comments (4)
  1. [§7] In the answer to RQ3, the text says 'the dominant finding from the PCA (Section 7)' but the PCA results are in Section 5, not Section 7; this cross-reference should be corrected.
  2. [§3.1, Table 6.1, Table 6.2] Model names are inconsistent across the manuscript: the 2024 cohort is described as including 'Llama-3-8B' in Section 3.1 but appears as 'Llama-3.8B' in Table 6.1, and the 2026 GPT model is rendered both as 'GPT-5.4 Mini' (Section 3.1) and 'GPT-5-4-Mini' (Sections 5 and 6). Please standardise model names.
  3. [§6.2] The enumeration of apostrophe glyphs is clear and useful, but it would help to state explicitly whether the custom regular-expression pattern builder was validated against a manually annotated sample beyond the AntConc checks, and how many texts were manually inspected.
  4. [§4.2(iv)] The sentence reporting that 'within-speaker variation in a given domain is greater than within-model variation, while between-speaker variation is greater than between-model variation' is presented without supporting numbers or a reference to a table or figure; this is a substantial claim and should either be quantified in the text or moved to the repository with a pointer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the empirical analyses are direct measurements on externally anchored corpora; the idiolect label is an interpretation, not a fitted input.

full rationale

The derivation chain is empirical rather than definitional. The 2026 corpus is generated by querying contemporary models with the same prompts and settings as the external 2024 TextualLLMap corpus (Improta et al. 2024); the descriptors in Section 4, the PCA in Section 5, and the contraction counts in Section 6 are direct measurements on these texts. Nothing is fitted to the idiolect conclusion and then reported as a prediction. The PCA is an unsupervised word-frequency projection, with the generational interpretation drawn from where models land on PC1; the contraction analysis is an independent frequency count on the same cleaned corpus, and although it was prompted by the tokenisation artefact 't' on PC1, it does not reuse a fitted parameter as evidence. Self-citations to Rudnicka (2025a,b) and Juzek (2026) are motivational and review-level, not load-bearing: the central empirical content of the chapter stands on the new 2026 corpus and the external 2024 corpus. The fact that the PCA and contraction analyses are restricted to one topic, while the paper's own definition of idiolect requires stability across topics and contexts, is a limitation on generalizability and construct validity; it does not make the output equivalent to its input by construction. Consequently, no circular step meeting the quoted-evidence standard can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central evidence consists of direct corpus measurements; the main auxiliary commitments are the stylometric equivalence assumptions (word-frequency PCA as style, single-topic generalizability) and the comparability of 2024 and 2026 generation. No physical entities are introduced.

free parameters (4)
  • MFW count and culling threshold for PCA = 150 most frequent words, culled at 50%
    Analyst-chosen settings determine which tokens enter the stylometric space; different choices could change ellipse positions and the PC1 generation split.
  • Stylometric sample size = 10,000 non-overlapping words per sample
    Chosen to balance sample count and corpus size; sample sizes vary from 30 to 125 per model, which may affect confidence ellipse estimates.
  • Contraction form list = 35 forms in 6 categories
    Hand-selected list of contractions; the 's and 'd categories are acknowledged as ambiguous between is/has and would/had, so category counts are partly definitional.
  • Generation sampling parameters = temperature 0.5, top-p 0.95, cap 2,048 tokens
    Inherited from Improta et al. (2024) to ensure comparability; top-k and penalties were left at defaults, so small uncontrolled differences across harnesses are possible.
assumptions (5)
  • domain assumption PCA on word-frequency profiles captures stylistic identity.
    The stylometric method treats the distribution of the 150 most frequent words as a proxy for authorial voice; this is standard in stylometry but is an assumption about what constitutes style.
  • domain assumption Identical prompts and topics produce comparable texts across models and cohorts.
    Section 3.1 states the 2026 corpus uses 'exactly the same prompts and topics' and 'generation parameters as close as possible'; this assumes prompt and topic equivalence is sufficient to isolate model-specific style.
  • domain assumption Idiolect, defined for humans, can be applied to LLM output as an analytical category.
    Section 7 explicitly says the term is adopted 'not as a metaphysical claim about AI agency or personhood, but as an analytical framework'; this is a framing assumption on which the central argument rests.
  • ad hoc to paper Single-topic stylometric profiles generalize across topics.
    Section 5.1 restricts the PCA to health misinformation texts, yet Section 2.1 defines idiolect as stable 'across topics and contexts'; no cross-topic stylometric validation is reported.
  • domain assumption Truncation at 2,048 tokens does not bias stylistic measures for cap-heavy models.
    Section 3.1 reports GPT-5.4 Mini hits the cap in 4-16% of texts depending on topic; the analysis keeps these texts after a truncation detector, implicitly assuming partial responses are stylistically representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond "AI Language": The case for the idiolectal nature of LLM output." pith.science (2026). https://pith.science/paper/YID4QKRE

@misc{pith2026260806589,
  author       = {Pith},
  title        = {Pith review of: Beyond "AI Language": The case for the idiolectal nature of LLM output},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YID4QKRE}},
  note         = {Machine review of arXiv:2608.06589}
}
read the original abstract

While large language model outputs are frequently analysed as a collective super variety termed "AI language," this chapter argues that this perspective coexists with distinct, model-specific linguistic signatures akin to human idiolects. We analyse two datasets of LLM-generated texts on societal topics: a 2024 corpus of six models (Improta et al. 2024) and a newly generated 2026 corpus using the same prompts featuring six contemporary models. Our findings, utilising computational descriptors and stylometric principal component analysis reveal a generational shift between the style of the 2024 and 2026 cohorts, while demonstrating that each individual model maintains a unique linguistic profile. This multi-layered interplay is illustrated by contraction frequencies, which vary from over 1,200 to over 30,000 per million words within the same cohort of models (2026). Ultimately, we conclude that treating LLM output as idiolectal in nature provides a valuable framework with potential implications for research on variation and change, LLM-generated text detection, forensic linguistics and usage-based approaches to language.

Figures

Figures reproduced from arXiv: 2608.06589 by the authors.

Figure 4.1
Figure 4.1. The percentage of texts that were categorised as containing repetition loops, for the models and topics. Years refer to the year of when the datasets were generated (2024 vs 2026). Alt text: Shown is a heatmap of 13 models times 4 topics, showing the percentage of texts that were categorised as containing repetition loops. Most cells are near 0%; a few rows have values of up to 12.2%, mainly in the health-misinforma… view at source ↗
Figure 4.2
Figure 4.2. Text length measured in words per text, and mean words per sentence, per model, with years of dataset generation. Alt text: Shown are two box-plot panels, containing one row per model, with points coloured by topic, within a box’s whiskers. Word counts rise sharply for the 2026 models; average sentence length varies to a lesser degree. (ii) With respect to lexical diversity, type-token ratio and vocabulary size are … view at source ↗
Figure 4.3
Figure 4.3. Left: Type-token ratio (TTR) per text, per model. Right: Vocabulary size in unique lemmas per model and topic. Years in both graphs refer to year of data generation. Alt text: Left: Shown are box plots of type-token ratio per model; whilst there is only some degree of variation, two models have a notably lower TTR, namely GTP-5.4 and Qwen-3, both of which are part of the 2026 cohort. Right: Shown is a heatmap for vo… view at source ↗
Figures from the paper (2 more)
Figure 4.4
Figure 4.4. Figure 4.4: Left: The percentage of texts opening with a safety disclaimer, per model and topic. Right: The percentage of texts containing an AI self-reference, per model and topic. Years in both graphs refer to year of data generation. Alt text: Shown are two heatmaps with the …
Figure 5
Figure 5. Figure 5: was created using the ggplot2 package for R (Wickham 2016). [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith

  1. [148]

    Meanings Attached to Intergenerational Language Shift Processes in the Context of Migrant Families

    https://doi.org/10.33423/jhetp.v22i13.5514. Smith, Genevieve, Eve Fleisig, Ishita Rustagi & Xavier Yin. 2025. Standard Language Ideology in AI -Generated Language. ArXiv preprint https://doi.org/10.48550/arXiv.2406.08726. Stiennon, Nisan, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei & Paul F. Christiano. ...

    arXiv doi:10.48550/arxiv.2406.08726 2025

  2. [429]

    Bergs, Alexander

    https://doi.org/10.1075/ijcl.22007.bak. Bergs, Alexander. 2025. Construction Grammar and Literature. In Miriam Fried & Kiki Nikiforidou (eds.), The Cambridge handbook of construction grammar [Cambridge Handbooks in Language and Linguistics], 623–647. Cambridge: Cambridge University Press. https://doi.org/10.1017/9781009049139. Bitton, Yehonatan, Elad Bitt...

    arXiv doi:10.48550/arxiv.2503.01659 2025

  3. [2017]

    Advances in Neural Information Processing Systems 30: 4302–4310

    Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems 30: 4302–4310. https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd 4e49-Abstract.html. 27 Crothers, Evan N., Nathalie Japkowicz & Herna L. Viktor. 2023. Machine -generated text: A comprehensive survey of threat models a...

    arXiv 2017

  4. [2023]

    Scientific Reports 13: 18617

    A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports 13: 18617. https://doi.org/10.1038/s41598-023-45644-9. Hickey, Raymond. 2012. Internally and externally motivated language change. In Juan Manuel Hernández-Campoy & Juan Camilo Conde -Silvestre (eds.), The handbook of historical sociolinguistics, 401 –421. Malden...

    arXiv doi:10.48550/arxiv.2403.01152 2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.