REVIEW 5 major objections 8 minor 13 references
ChatGPT as speechwriter for the French presidents
T0 review · 5 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read ChatGPT-written French presidential speeches are statistically distinct from real ones, yet intertextual-distance attribution no longer flags them when a single model speech is supplied.
desk verdict Useful descriptive profile of ChatGPT's French output, but missing generation details and length-matched controls leave the style claims underdetermined. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by three quantitative instruments applied to corpora whose words were reduced to dictionary lemmas. Part-of-speech densities per thousand words expose the noun-versus-verb imbalance. Rank-frequency tables of the most frequent lemmas in the verb, pronoun, adverb, noun, adjective, and determiner categories reveal which words ChatGPT over- or under-uses. Sentence-length distributions, summarized by mode, median, mean, standard deviation, coefficient of variation, and the ninth-to-first-decile ratio, show that generated texts cluster around the mean. The defining diagnostic is the inequality $\text{mode} < \text{median} < \text{mean}$, which the authors treat as a property of natural sentence-length distributions and which ChatGPT's near-Gaussian output violates. For the detection question, the central object is intertextual distance, a measure between zero and one computed from the absolute differences between lemma frequencies divided by total text length; below a threshold it supports single authorship, and the paper shows it no longer separates GPT from human text when one homogeneous model is supplied.
What would settle it
Re-run the same presidential speech-generation protocol on several dated ChatGPT versions with temperature and prompts recorded; if the overuse of nouns, possessive determiners, and numbers, the underuse of pronouns, adverbs, and subordinate clauses, and the compression of sentence lengths do not reproduce, the claimed signature is version- or prompt-specific.
Extended reading notes
Core claim
On its own terms, the paper establishes that ChatGPT's French writing carries a measurable stylistic signature, not just isolated oddities. Compared with the presidents' addresses, generated texts lean toward the noun side of the language: common nouns rise by about 7%, adjectives by 13%, possessive determiners by 30%, and numbers by 36%, while verb-linked categories shrink, with pronouns down 22%, adverbs down 32%, and subordinating conjunctions down 25%. Lexically, ChatGPT overuses devoir, continuer, nous, année, défi, and valiant, and underuses être, vouloir, falloir, aller, dire, and third-person pronouns. The clearest single signature is sentence length: real speeches obey the natural inequality $\text{mode} < \text{median} < \text{mean}$ and have a coefficient of variation near 78%, whereas ChatGPT's near-Gaussian distribution yields $\text{mode} \approx \text{median} \approx \text{mean}$ and pulls the coefficient down to about 50%. Despite these divergences, ChatGPT given a single homogeneous model text produces output that intertextual distance cannot flag as machine-generated, so the paper concludes this classical detection method is no longer adequate.
Load-bearing premise
The load-bearing premise is that the outputs collected from chat.openai.com, whose model version, temperature, and exact prompts are not reported, represent a stable 'ChatGPT style' rather than an artifact of one model snapshot or one prompting strategy.
Editorial extensions
If this is right
- ChatGPT's French prose has a describable statistical baseline: noun-heavy, verb-shy, with fewer pronouns, adverbs, and subordinate clauses than human political writing.
- Real presidential addresses and ChatGPT outputs can be told apart on part-of-speech and sentence-length grounds, even when the machine was given a real speech as a model.
- Intertextual-distance authorship attribution, which previously detected machine-generated and fraudulent scientific papers, fails when the prompt supplies one homogeneous model text.
- Plagiarism detectors that compare n-gram overlaps will likely miss ChatGPT output because the generator rearranges the model's vocabulary rather than copying it.
- The paper itself notes these limitations may be mitigated in future generator versions and calls for further experiments on detection features and other languages.
Reading between the lines
- Because the corpus holds one generated output per presidential speech, a natural next test is to sample many outputs from the same prompt under different model versions, temperatures, and random seeds; this would quantify the stylistic variance behind the reported averages.
- The overuse of 'défi' and the feminized formula 'nos concitoyennes et nos concitoyens' suggests ChatGPT generalizes beyond the prompt from generic presidential discourse in its training data, so detectors aimed at stereotyped collocations (such as 'relever les défis qui') might work where frequency profiles fail.
- If the sentence-length compression ($\text{mode} \approx \text{median} \approx \text{mean}$) generalizes beyond French, a simple variance-based readability statistic could serve as a cheap, language-independent screening test for generated prose.
- A more demanding test would be corpus-level authorship attribution, comparing the generated text against many writings of the suspected author rather than a single supplied model; the paper's negative result concerns the single-model case.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a corpus study of 20 ChatGPT-generated French presidential New Year's addresses, each prompted with a real presidential address as a model, compared with the 20 original addresses. Using lemmatized POS tagging, lemma frequency tables, sentence-length distributions, and intertextual distance, the authors report that ChatGPT overuses nouns, possessive determiners, and numbers; underuses verbs, pronouns, and adverbs; produces more uniform sentence lengths; and that intertextual distance fails to distinguish the generated texts from their models when a single homogeneous text is provided. The paper concludes that ChatGPT cannot easily be detected by existing distance-based methods.
Significance. The study addresses an important and timely question: whether LLM-generated political speech can be stylistically distinguished from human-authored text. The descriptive frequency tables are useful and the authors provide a transparent corpus of French political speech. If the findings survive rigorous controls, they would inform both stylometry and AI-text detection. However, the current analysis lacks basic experimental controls (text length, prompt variability, model version) and the significance index is under-specified, so the central claims are plausible but not yet established.
major comments (5)
- [Section 4, Table 2] The aggregate comparison of POS frequencies between the NT corpus (30,935 words) and the GPT corpus (16,699 words) does not control for text length. Because the average GPT output is 835 words versus 1,547 for a presidential address (Table 1), and because lexical density, pronoun frequency, and syntactic complexity are known to vary with text length, the reported differences (e.g., common nouns +7%, adverbs -32%) may be an artifact of output length rather than a stable property of ChatGPT. Please include a length-matched human baseline (e.g., first 835 words of each presidential speech) or otherwise demonstrate that the effects persist when text length is held constant.
- [Section 3] The paper does not report the exact prompts used to generate the GPT texts, the model version (e.g., GPT-3.5 vs GPT-4), the sampling temperature, or the number of independent generations per presidential speech. Without this information, the results are not reproducible, and the abstract's general characterization of 'ChatGPT style' may apply only to a particular snapshot and prompting strategy. The authors should disclose the full generation protocol and ideally run multiple independent generations to assess stability.
- [Section 6, Table 10 and Figure 1] The sentence-length analysis compares entire corpora of different sizes and does not control for text length. The lower standard deviation and coefficient of variation in GPTs may simply reflect the fact that shorter texts are more uniform in sentence length; a length-matched comparison with human-written texts is needed. Additionally, the claim that GPT sentence lengths follow a Gaussian distribution is not tested with a formal normality test, and the mode/median/mean equality is merely stated.
- [Section 7] The intertextual distance analysis merges all GPT outputs per president (2,843-5,235 words) to circumvent the 1,000-word minimum distance requirement that the authors themselves note. While this avoids individual short texts, the merged GPT corpora are still much shorter than the corresponding NT corpora (5,631-9,818 words), so the distances are computed on unequal-length texts. More importantly, the conclusion that 'intertextual distance is therefore no longer able to identify texts generated by ChatGPT' is an overgeneralization: only one condition (a single homogeneous model text) was tested, with no control condition such as generation without a model or with multiple heterogeneous examples. The conclusion should be restricted to the tested setting, and the current wording in the abstract should be qualified.
- [Tables 2-9] The significance index S is undefined. The text states thresholds for 5% and 1% risk but does not give the formula or the underlying statistical test. Without this, the reader cannot assess whether S-values such as 0.009 or 0.977 support the claimed significant differences. Please define S explicitly (e.g., the probability under a stated null hypothesis) and justify the thresholds.
minor comments (8)
- [Section 6] The phrase 'It is well-kwon' contains a typo; it should be 'well-known'.
- [Table 7] The word 'mounth' should be 'month'.
- [Table 9] The entries 'firs' and 'eigth' should be 'first' and 'eighth'.
- [Table 6] The word 'ensembe' should be 'ensemble' (or 'together' in the English gloss).
- [Section 3] The sentence 'the characteristics of ChatGPT's vocabulary (see Section 4)' points to Section 4, which is about POS; vocabulary lemmas are actually discussed in Section 5. Please correct the cross-reference.
- [Section 5.1] The phrase 'ChatGPT has a little trouble with simple verbs' is informal; consider 'some difficulty with simple verbs'.
- [Figure 1] The axes are not fully labeled: the x-axis should be 'sentence length (words)' and the y-axis 'percentage of sentences'.
- [References] The reference to 'Vaswami et al. 2017' contains a typo; it should be 'Vaswani et al. 2017'.
Circularity Check
No significant circularity: the paper's empirical comparisons are self-contained, and no load-bearing derivation reduces to its own inputs.
full rationale
The paper's central claims about ChatGPT's overuse of nouns, possessive determiners, and numbers and its underuse of verbs, pronouns, and adverbs are direct corpus-frequency comparisons. No parameter is fitted, no statistical model is trained on the target texts, and no 'prediction' is derived from a quantity that already contains the claimed outcome. The S index in Table 2 is a significance test applied to observed frequencies, not a fitted input. The Section 7 claim that intertextual distance can no longer identify ChatGPT outputs is an empirical stress-test result: the authors explicitly condition on 'a single homogeneous text' being submitted as the model, and the distance values and classifications are computed rather than assumed. Although the experimental design is self-referential (ChatGPT is given the exact target text as its prompt, and the distance metric is the authors' own from prior work), this does not constitute derivation-level circularity. The corpus-length imbalance and the undisclosed prompt details are validity limitations, not cases where a result is equivalent to its inputs by construction. No load-bearing step reduces to a self-citation chain or a fitted parameter renamed as a prediction, so the score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The lemmatizer and Muller (1977) normalization correctly disambiguate French homographs and token variants, so observed differences are not tagging artifacts.
- ad hoc to paper The 20 ChatGPT outputs, generated with undisclosed prompts and model settings, are representative of 'ChatGPT' as a stable authorial style.
- domain assumption Intertextual distance computed on texts merged per president (eight texts) is valid despite the texts being shorter than the 1,000-word guideline and despite merging heterogeneous speeches.
- domain assumption The S significance index assumptions, including independence of word occurrences, are valid for comparing the two corpora.
Cite this review
Pith. "Pith review of ChatGPT as speechwriter for the French presidents." pith.science (2026). https://pith.science/paper/54DQ5YWU
@misc{pith2026241118382,
author = {Pith},
title = {Pith review of: ChatGPT as speechwriter for the French presidents},
year = {2026},
howpublished = {\url{https://pith.science/paper/54DQ5YWU}},
note = {Machine review of arXiv:2411.18382}
}
read the original abstract
Generative AI proposes several large language models (LLMs) to automatically generate a message in response to users' requests. Such scientific breakthroughs promote new writing assistants but with some fears. The main focus of this study is to analyze the written style of one LLM called ChatGPT by comparing its generated messages with those of the recent French presidents. To achieve this, we compare end-of-the-year addresses written by Chirac, Sarkozy, Hollande, and Macron with those automatically produced by ChatGPT. We found that ChatGPT tends to overuse nouns, possessive determiners, and numbers. On the other hand, the generated speeches employ less verbs, pronouns, and adverbs and include, in mean, too standardized sentences. Considering some words, one can observe that ChatGPT tends to overuse "to must" (devoir), "to continue" or the lemma "we" (nous). Moreover, GPT underuses the auxiliary verb "to be" (^etre), or the modal verbs "to will" (vouloir) or "to have to" (falloir). In addition, when a short text is provided as example to ChatGPT, the machine can generate a short message with a style closed to the original wording. Finally, we reveal that ChatGPT style exposes distinct features compared to real presidential speeches.
Reference graph
Works this paper leans on
-
[1]
to must” (devoir), “to continue
- 1 - ChatGPT as speechwriter for the French presidents Dominique Labbé, Cyril Labbé, Jacques Savoy Science Po Laboratoire LIG Computer Science Dept. Univ. Grenoble Alpes Univ. Grenoble Alpes University of Neuchatel 1030 av. Centrale 700 av. Centrale rue Emile Argand 11 38400 St Martin-d’Hères 38000 Grenoble 2000 Neuchatel, Switzerland Dominique.Labbé@umr...
work page 2000
-
[3]
the foundation model GPT is able, based on a sequence of initial tokens, to produce a ranked list of the next plausible token (e.g., word or punctuation symbol), list elaborated based on the training documents. For example, after sequence “the prime minister of”, the model can define a list of the next token as {UK, India, England, Canada, France, Austral...
work page 2023
-
[4]
suggest that ChatGPT does not really know French grammar and all the vocabulary, but that it works with a wide variety of tokens (graphic forms) and seems to have some difficulty with certain homographs. To have an overview of our corpus, Table 1 provides the president names and Nnt, the number of words in each presidential message, followed by Ngpt, the ...
work page 2012
-
[5]
- 4 - performance drops clearly
(Soto et al., 2024), the 4 This random aspect is under the control of the temperature parameter. - 4 - performance drops clearly. In addition, such detectors must take account of updated LLMs in response to their known weaknesses. Therefore, there is a clear requirement to acquire a style description or to derive some stylistic features associated to such...
work page 1977
-
[9]
The twenty most frequent determiners by presidents, compared with GPTs (frequencies per thousand words) The two corpora share the first four determiners in the same order. Thereafter, the differences are quite significant. Most notable is the overuse of “notre” (our), “mon” (mine) and, to a lesser extent, “votre” (your) in connection with the overuse of t...
work page 2020
-
[10]
indicates that this dispersion around the mean is lower in the corpus of generated texts than in that by the presidents. This finding explains why, on the figure, the mode of sentences produced by ChatGPT is clearly higher than that of natural sentences. Finally, the absence of very long sentences in GPTs may be linked to the low frequency of complex cons...
work page 2016
-
[11]
Tree classification by presidents How much confidence can one place in these classifications? A quality index is used to answer this question (Labbé & Labbé, 2006). For the tree (Figure 3), all the paths on the tree have an index greater than 90%, and the quality of the whole tree is greater than 96%. In other words, there is less than a 5% chance of bein...
arXiv 2023
-
[30]
- 22 - Wolfram, S. (2023). What is GPT-4 Doing… and What Does it Work?. Orlando: Wolfram Re-search Inc., Champaign (IL). Zhao, W., Zhou, K., Li, J., Tang, T., Wang, X, Hou, Y. Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Peiyu, P., Nie, J.Y., and Wen, J.R. (2023). A survey of large la...
arXiv 2023
Show all 13 references
-
[2012]
The calculation compares the vocabulary of two texts (A, B), measuring the absolute difference between the frequencies of each lemma in A and B
or well as in authorship attribution (Savoy, 2018). The calculation compares the vocabulary of two texts (A, B), measuring the absolute difference between the frequencies of each lemma in A and B. The sum of these differences is related to the total length of the two texts. Th...
2018
-
[2013]
According to Gao et al
even in generating tortured phrases (Cabanac et al., 2021). According to Gao et al. (2023), the scientific abstracts generated by GPT are hard to detect by expert in the field (success rate around 68%, high-impact journals). In this case, GPT abstracts appear vague, superficia...
2023
-
[2014]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I
https://doi.org/10.1038/nature.2014.14763. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Pro-cessing Systems,
2017
-
[2021]
https://arxiv.org/abs/2107.06751 Cover T. M. & Hart P. E. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory. 13(1):21–27. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. Boston: The MIT Press. Gao, C.A., Howard, F.M.,...
1967 arXiv
-
[2023]
produces outputs that are essentially indistinguishable from (even better than) what humans could produce
trained on huge corpora, the machine could generate short answers to users’ requests. The produced messages are clear, coherent, plausible, and without spelling errors. Facing such successes, Bubeck et al. (2023) assert that GPT “produces outputs that are essentially indisting...
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.