Pith. sign in

REVIEW 4 major objections 5 minor 30 references

EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read The paper claims that ChatGPT's responses to tweets about generative AI are 90% positive while human comments are 72% negative, reflecting LLMs' built-in tendency to stay constructive and positive.

desk verdict The 90%-positive ChatGPT claim is not derivable from the described K-means pipeline because the paper never explains how clusters were labeled positive or negative. read the letter →

arxiv 2501.06597 v1 pith:AUYF263F submitted 2025-01-11 cs.LG cs.CLcs.HC

classification cs.LGcs.CLcs.HC
keywords sentimentanalysisgenerativeAIChatGPTLLM-generatedresponsesK-meansclusteringBERTembeddingsemotionalintelligenceTwitter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EmoXpt, a sentiment-analysis pipeline that compares human Twitter comments about generative AI with ChatGPT's replies to the same tweets. Its central empirical claim is that ChatGPT-generated responses are consistently positive, with about 90% classified positive, while human comments are mostly negative, at about 72%. The authors argue this reflects ChatGPT's design to maintain a constructive, positive tone, and that this makes AI responses more cohesive but less emotionally varied than human language. The result matters because it quantifies a suspected asymmetry: conversational AI is trained to avoid negativity, so automated positivity can mask rather than mirror public opinion.

What carries the argument

The load-bearing mechanism is an unsupervised sentiment-analysis pipeline: BERT word embeddings of 768 dimensions are computed for every token and averaged to sentence embeddings, then K-means clustering with k=2 is applied separately to human words, ChatGPT words, human sentences, and ChatGPT sentences. Cluster quality is measured with the Silhouette score, and t-SNE projects the high-dimensional clusters to two dimensions for visualization. The cleaning steps differ between the two sides: human comments get URLs, emojis, and @mentions removed, while ChatGPT responses additionally have frequent neutral words such as 'AI', 'language', and 'model' stripped out before the same clustering is run.

What would settle it

Recompute Table V with a documented cluster-labeling rule, for example assigning the cluster whose centroid has higher average overlap with positive sentiment-lexicon words as 'positive'; if the resulting percentages are close to 90% and 27% the claim stands, and if they differ sharply the polarity assignment was arbitrary. A second concrete check is to rerun the full pipeline with identical stopword and neutral-word removal for both human comments and ChatGPT responses, since the paper removes frequent neutral words only from ChatGPT responses.

Watch

Extended reading notes

Core claim

On its own terms, EmoXpt claims to show that human comments about generative AI on Twitter are predominantly negative (72%), while ChatGPT's responses to those same tweets are overwhelmingly positive (90%). The sentence-level K-means clusters for ChatGPT reach a Silhouette score of 0.58 versus 0.19 for human comments, which the paper reads as ChatGPT's responses being more focused, cohesive, and uniformly toned. The paper interprets the split as evidence that LLMs exhibit a positivity bias, responding constructively even to negative or fearful prompts, and concludes that AI-generated text is efficient and positive but does not capture the full emotional nuance of human interaction.

Load-bearing premise

The central load-bearing premise is that the two K-means clusters correspond to positive and negative sentiment, and that the reported 90% and 27% figures follow from that correspondence; the paper gives no rule for which cluster is which.

Editorial extensions

If this is right

  • If the claim is right, ChatGPT's public-facing text is systematically skewed toward positive sentiment compared with human expression, so treating LLM output as representative human opinion would overestimate support for generative AI.
  • The 90% versus 27% split suggests that alignment or safety training suppresses negative framing in ChatGPT even when the input tweet is critical or worried.
  • The large gap in sentence-level Silhouette scores (0.58 for ChatGPT versus 0.19 for humans) implies that ChatGPT's emotional tone is more uniform and less varied than human tone.
  • The framework gives a label-free way to compare emotional variance across language models and social-media populations, without needing manually annotated sentiment data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the headline percentages depend on an unstated rule for assigning 'positive' and 'negative' to the two K-means clusters; labeling the same clusters using a sentiment lexicon could confirm or reverse the 90% versus 27% split.
  • Because frequent neutral words were removed from ChatGPT responses but not from human comments, the same pipeline with symmetric cleaning on both sides is a direct test of whether the positivity gap is a property of the model or an artifact of preprocessing.
  • If the claim holds, downstream analytics that use LLM-generated replies as proxies for user sentiment will systematically overestimate approval, which matters for product feedback, public-opinion tracking, and content moderation.
  • The same two-cluster pipeline could be applied to other language models and to non-English posts to test whether near-universal positivity is model-specific or a general property of aligned conversational AI.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EmoXpt, an unsupervised sentiment analysis framework that uses BERT embeddings and k-means clustering to compare the sentiment of human Twitter comments about generative AI with ChatGPT-generated responses to those tweets. The central quantitative claim, stated in the abstract, conclusion, and Table V, is that ChatGPT responses are overwhelmingly positive (90% positive, 10% negative) while human comments are mostly negative (27% positive, 72% negative). The paper also reports silhouette scores for word- and sentence-level clustering and interprets these as evidence that ChatGPT responses are more efficient, cohesive, and consistently positive than human comments. The manuscript includes data collection details over a two-month period in 2023, preprocessing steps, a description of the k-means pipeline, and a discussion of threats to validity.

Significance. If the headline result were properly supported, the paper would offer a useful, albeit modest, data point on the sentiment of LLM-generated responses relative to human commentary. The proposed framework itself, however, is a straightforward application of standard BERT embeddings plus k-means, and the paper's distinctive contribution is entirely the empirical comparison in Table V. Unfortitingly, the paper provides no reproducible pipeline, no code or data, and no validation of the claimed sentiment percentages. The central claim therefore rests on an unspecified methodological step, and the internal data inconsistencies further undermine confidence in the reported numbers. The strengths are the clear motivation to compare human and LLM sentiment and the explicit acknowledgment of some validity threats, but these do not compensate for the missing analysis.

major comments (4)
  1. [§IV-B, Table V] The central result of the paper, that ChatGPT responses are 90% positive and human comments are 27% positive, is not a determinate consequence of the described method. The only modeling step described is k-means with k=2 on BERT embeddings (Section III-D), which outputs unlabeled clusters. Neither the Silhouette Scores in Table IV nor the t-SNE plots in Figures 9–10 assign polarity to clusters. The paper does not specify any cluster-labeling rule, such as centroid inspection against a sentiment lexicon, a labeled validation set, or a bootstrapping procedure. Therefore the percentages in Table V cannot be derived from the described pipeline, and the abstract's and conclusion's claims that LLM-generated responses are 'notably more positive' are unsupported. The authors must specify exactly how clusters were mapped to positive/negative labels and how the percentages were computed.
  2. [§III-C] The preprocessing is asymmetric. Human comments are cleaned with steps shown in Figure 5, while ChatGPT responses additionally have 'frequently occurring neutral words' such as 'AI', 'language', and 'model' removed (Figure 6). This step is applied only to ChatGPT responses, not to human comments. Removing words like 'AI' from one corpus but not the other changes the token distributions and resulting BERT embeddings, which in turn can systematically alter the k-means clusters and any derived sentiment percentages. The paper provides no justification for why these words are 'neutral' in ChatGPT responses but not in human comments, and no sensitivity analysis is given to show that the headline 90% versus 27% comparison is robust to this asymmetric cleaning. This is a second load-bearing bias that could inflate ChatGPT's apparent positivity even if the cluster-labeling problem were resolved.
  3. [§III-A and §IV-B] Several data statistics in the paper are internally inconsistent. The paper states there are 512 human tweets, 429 user comments, and ChatGPT responses for each tweet (Section III-A). It also states that 75% of tweets have no comments and that the average number of comments is 28.4 with a maximum of 29; with 429 comments over 512 tweets the mean is approximately 0.84, and an average of 28.4 is arithmetically impossible. Additionally, in Section III-C the sentence embedding matrix is described as '(768, 429) where 768 is the number of data points and 429 is the features extracted for each data point,' which confuses the embedding dimension with the sample count; the correct dimensions would be (429, 768) for human comments and (512, 768) for ChatGPT responses. These inconsistencies indicate that the data handling and reporting are not reliable, which further erodes confidence in Table V.
  4. [§IV-B, RQ1 and RQ2] The interpretation of the silhouette scores is not supported by the metric. The paper states that ChatGPT words have a silhouette score of 0.053 and human words 0.13, and interprets the lower ChatGPT score as indicating 'a more focused, repetitive vocabulary' and 'consistent and direct' language. However, a silhouette score near zero actually indicates that clusters are not well separated and that data points are about as close to their own cluster as to the neighboring cluster; it does not measure vocabulary diversity or focus. The same misunderstanding applies to the sentence-level scores in Table IV. The paper's claim that ChatGPT responses are 'cohesive' based on these scores is therefore not justified.
minor comments (5)
  1. [Abstract / Conclusion] The claim that ChatGPT responses are 'more efficient, cohesive, and consistently positive' uses terms that are not defined or measured in the experimental section; the paper only measures clustering quality and sentiment polarity, not efficiency or cohesion in any operational sense.
  2. [§III-C heading] The subsection title 'Data Prepossessing' is a typo for 'Data Preprocessing'.
  3. [§III-D] The phrase 'which is usesful for identifying contrasting sentiments' contains a typo ('usesful' should be 'useful').
  4. [§V] In 'Threats to Conclusion Validity,' the sentence 'To reduce the conclusion validity of this work' should read 'To reduce the threat to conclusion validity' or similar.
  5. [§VI references] Several references are incomplete or formatted inconsistently; for example, reference [9] has the title 'Usecase' rather than a meaningful title, and references [15], [23], and [24] mix URL text with the actual resource names.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction is present in the derivation chain; the RQ3 positivity claim is underdetermined by the described K-means pipeline because cluster-to-polarity labeling is never specified, and the only self-citations ([26], [27]) are non-load-bearing.

full rationale

The paper's central claim, that ChatGPT responses are about 90% positive versus 27% for human comments (Table V, Section IV-B), is presented as the outcome of an unsupervised sentiment analysis pipeline (Section III-D) that trains K-means with k=2 on BERT embeddings. That pipeline outputs unlabeled partitions, and no equation or rule in the paper maps the two clusters to the polarity labels 'positive' and 'negative'; Section IV-B simply states 'Based on the percentages in Table V, we infer...', so the percentages are not shown to be a consequence of the described method. This is an underdetermination or completeness gap, not circularity: the conclusion is not an input to the method, no parameter was fitted to the 90%/27% target, and the claim does not reduce by construction to the K-means output or to the BERT embeddings. The paper's own Section V concedes that 'ChatGPT often generates positive sentiments, even using negative sentiment words in a positive way, making it challenging for an unsupervised model to predict', corroborating that the polarity assignment is an unreported extra step rather than a forced derivation. The asymmetric cleaning step (Section III-C) that removes frequent words such as 'AI', 'language', and 'model' from ChatGPT responses only is a plausible source of bias, but removing tokens does not by construction force the 90%/27% split, so it is a validity concern, not a circular one. The only self-citations are [26] and [27], both co-authored by T. S. Zaman, cited in Section V merely to frame the four standard threats-to-validity categories; the sentiment pipeline itself rests on independent references (BERT [6], K-means [14], the unsupervised sentiment pipeline [5] and [15], silhouette scores [16], and t-SNE [17]), none of which are self-citations. Accordingly, the derivation chain is not self-referential, and the score of 2 reflects the presence of minor non-load-bearing self-citations rather than any circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an unstated cluster-labeling rule, on a domain assumption that K-means clusters over BERT embeddings separate sentiment, and on a cleaning asymmetry between human and ChatGPT text; these choices are not validated against labeled data.

free parameters (3)
  • Number of K-means clusters = 2
    The paper sets k=2 to separate positive and negative sentiment but provides no validation that two clusters match sentiment categories; a different k would change the percentages.
  • ChatGPT response generation settings = Not reported
    The model version, API parameters, and prompting strategy used to generate ChatGPT responses are not described, making the comparison irreproducible and potentially dependent on undocumented choices.
  • Removed 'neutral' word list = e.g., 'AI', 'language', 'model'
    The authors select frequent words to remove from ChatGPT responses based on word-frequency analysis of the same responses; this hand-chosen list differs from the cleaning applied to human comments and can affect the sentiment balance.
assumptions (4)
  • domain assumption BERT embeddings capture sentiment-relevant semantics
    The method assumes that distance in BERT embedding space reflects sentiment similarity, which is not guaranteed for short, informal tweets.
  • domain assumption K-means clusters correspond to positive and negative sentiment
    The analysis treats the two clusters as sentiment categories, but K-means merely finds spherical groups; without labeled validation, the clusters could reflect topic or style rather than sentiment (Section IV-B).
  • domain assumption The cleaning procedures do not differentially bias sentiment
    Human comments have emojis, URLs, and mentions removed, while ChatGPT responses additionally have frequent 'neutral' words removed; the paper assumes this asymmetry does not change the relative positivity (Section III-C).
  • domain assumption The collected tweets are representative
    The dataset of 512 English tweets with specific hashtags collected by two graduate students over two months is assumed to represent public opinion about generative AI (Section III-A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses." pith.science (2026). https://pith.science/paper/AUYF263F

@misc{pith2026250106597,
  author       = {Pith},
  title        = {Pith review of: EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUYF263F}},
  note         = {Machine review of arXiv:2501.06597}
}
read the original abstract

The widespread adoption of generative AI has generated diverse opinions, with individuals expressing both support and criticism of its applications. This study investigates the emotional dynamics surrounding generative AI by analyzing human tweets referencing terms such as ChatGPT, OpenAI, Copilot, and LLMs. To further understand the emotional intelligence of ChatGPT, we examine its responses to selected tweets, highlighting differences in sentiment between human comments and LLM-generated responses. We introduce EmoXpt, a sentiment analysis framework designed to assess both human perspectives on generative AI and the sentiment embedded in ChatGPT's responses. Unlike prior studies that focus exclusively on human sentiment, EmoXpt uniquely evaluates the emotional expression of ChatGPT. Experimental results demonstrate that LLM-generated responses are notably more efficient, cohesive, and consistently positive than human responses.

Figures

Figures reproduced from arXiv: 2501.06597 by the authors.

Figure 1
Figure 1. The Overview of EmoXpt D.(2023) Human Tweet ChatGPt Response Human Comments Mar. 16 Is ChatGPT the cornerstone of Skynet by “Cyberdyne Systems”? Blessing or curse at once #ChatGPT No, ChatGPT is not the cornerstone of Skynet by Cyberdyne Systems, nor is it in any way related to the fictional ... This is nothing you need to be con￾cerned with... April 28 Wait. Umm is any of #ChatGPT training data based on what was re… view at source ↗
Figure 4
Figure 4. Word Frequency of ChatGPT Responses. of subsequent analyses, such as sentiment analysis or topic modeling. Word Count Distribution and Outlier Detection: Analyzing the distribution of word counts in comments and responses helps us understand typical response lengths and identify unusually long or short responses that may serve as outliers. Detecting outliers in response length is valuable, as it flags data points th… view at source ↗
Figure 3
Figure 3. Topic Analysis of Human Comments [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Cleaning Steps For Human Comments. In the second step, we clean the human comments [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: Cleaning Steps For ChatGPT Responses. Lastly, the third method customizes the cleaning process for ChatGPT responses [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 8
Figure 8. Figure 8: represents the data points of ChatGPT words that are clustered into two groups using K-means. The interpre￾tation suggests that the model struggled to effectively cluster the data points due to significant noise, which hindered its ability to discern meaningful pattern…
Figure 7
Figure 7. Figure 7: K-means on Human Words [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 10
Figure 10. Figure 10: K-means on Human Comments. TABLE IV SENTENCE LEVEL K-MEANS MODEL EVALUATION HCmnt CRspn Silhouette Score 0.19 0.58 Based on these scores, we analyzed ChatGPT’s emotional expressions and compared them with those of humans. RQ3. LLM-generated responses are more positive…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 25 canonical work pages

  1. [1]

    [Online]

    Introducing chatgpt. [Online]. Available: https://openai.com/index/ chatgpt/

  2. [2]

    Available: https: //www.cloudflare.com/learning/ai/what-is-large-language-model/

    What is a large language model (llm)? [Online]. Available: https: //www.cloudflare.com/learning/ai/what-is-large-language-model/

  3. [3]

    Alpaydin, Machine learning

    E. Alpaydin, Machine learning . MIT press, 2021

  4. [4]

    Generative ai,

    S. Feuerriegel, J. Hartmann, C. Janiesch, and P. Zschech, “Generative ai,” Business & Information Systems Engineering , vol. 66, no. 1, pp. 111–126, 2024

  5. [5]

    Unsupervised sentiment analysis with emotional signals,

    X. Hu, J. Tang, H. Gao, and H. Liu, “Unsupervised sentiment analysis with emotional signals,” in Proceedings of the 22nd International Conference on World Wide Web, ser. WWW ’13. New York, NY , USA: Association for Computing Machinery, 2013, p. 607–618. [Online]. Available: https://doi.org/10.1145/2488388.2488442

  6. [6]

    Homophobic and hate speech detection using multilingual-bert model on turkish social media,

    H. Karayi ˘git, A. Akdagli, and C ¸ . ˙I. Aci, “Homophobic and hate speech detection using multilingual-bert model on turkish social media,” Information Technology and Control , vol. 51, no. 2, pp. 356–375, 2022

  7. [7]

    Comparison of bert models and machine learning methods for sentiment analysis on turkish tweets,

    Z. A. Guven, “Comparison of bert models and machine learning methods for sentiment analysis on turkish tweets,” in 2021 6th International Conference on Computer Science and Engineering (UBMK) , 2021, pp. 98–101

  8. [8]

    Review of research on text sentiment analysis based on deep learning,

    W. Li et al. , “Review of research on text sentiment analysis based on deep learning,” Open Access Library Journal , vol. 7, no. 03, p. 1, 2020

Show all 30 references
  1. [9]

    [Online]

    Usecase. [Online]. Available: https://www.nytimes.com/2023/02/16/ technology/bing-chatbot-microsoft-chatgpt.html

  2. [10]

    I think this is the most disruptive technology: Exploring sentiments of chatgpt early adopters using twitter data,

    M. U. Haque, I. Dharmadasa, Z. T. Sworna, R. N. Rajapakse, and H. Ahmad, “I think this is the most disruptive technology: Exploring sentiments of chatgpt early adopters using twitter data,” 2022. [Online]. Available: https://arxiv.org/abs/2212.05856

  3. [11]

    Public perception of generative ai on twitter: an empirical study based on occupation and usage,

    K. Miyazaki, T. Murayama, T. Uchiba, J. An, and H. Kwak, “Public perception of generative ai on twitter: an empirical study based on occupation and usage,” EPJ Data Science , vol. 13, no. 1, p. 2, 2024

  4. [12]

    Analyzing the user’s sentiments of chatgpt using twitter data,

    A. Korkmaz, C. Akturk, and T. Talan, “Analyzing the user’s sentiments of chatgpt using twitter data,” Iraqi Journal F or Computer Science and Mathematics, vol. 4, no. 2, p. 202–214, May 2023. [Online]. Available: https://journal.esj.edu.iq/index.php/IJCM/article/view/618

  5. [13]

    [Online]

    Nltk. [Online]. Available: https://www.nltk.org/https://huggingface.co/

  6. [14]

    An unsupervised fuzzy clustering method for twitter sentiment analysis,

    H. Suresh and G. R. S., “An unsupervised fuzzy clustering method for twitter sentiment analysis,” in 2016 International Conference on Com- putation System and Information Technology for Sustainable Solutions (CSITSS), 2016, pp. 80–85

  7. [15]

    Twitter, myspace, digg: Unsupervised sentiment analysis in social media,

    G. Paltoglou and M. Thelwall, “Twitter, myspace, digg: Unsupervised sentiment analysis in social media,” ACM Trans. Intell. Syst. Technol., vol. 3, no. 4, Sep. 2012. [Online]. Available: https: //doi.org/10.1145/2337542.2337551

  8. [16]

    Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,

    P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics , vol. 20, pp. 53–65, 1987. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0377042787901257

  9. [17]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008

  10. [18]

    A survey of text representation and embedding techniques in nlp,

    R. Patil, S. Boit, V . Gudivada, and J. Nandigam, “A survey of text representation and embedding techniques in nlp,” IEEE Access, vol. 11, pp. 36 120–36 146, 2023

  11. [19]

    Euclidean distance mapping,

    P.-E. Danielsson, “Euclidean distance mapping,” Computer Graphics and image processing , vol. 14, no. 3, pp. 227–248, 1980

  12. [20]

    [Online]

    Python 3.10. [Online]. Available: https://www.python.org/downloads/ release/python-3100/

  13. [21]

    [Online]

    Pandas. [Online]. Available: https://pandas.pydata.org/

  14. [22]

    [Online]

    Numpy. [Online]. Available: https://numpy.org/

  15. [23]

    [Online]

    Hugging face. [Online]. Available: https://huggingface.co/https: //scikit-learn.org/

  16. [24]

    [Online]

    Scikit-learn. [Online]. Available: https://scikit-learn.org/https:// matplotlib.org/

  17. [25]

    Matplotlib in python,

    N. Ari and M. Ustazhanov, “Matplotlib in python,” in 2014 11th International Conference on Electronics, Computer and Computation (ICECCO). IEEE, 2014, pp. 1–6

  18. [26]

    Extracting implicit programming rules: comparing static and dynamic approaches,

    T. S. Zaman and T. Yu, “Extracting implicit programming rules: comparing static and dynamic approaches,” in Proceedings of the 7th International Workshop on Software Mining , ser. SoftwareMining 2018. New York, NY , USA: Association for Computing Machinery, 2018, p. 1–7. [Onli...

  19. [27]

    Netminer: Identifying failure-inducing patterns in the logs generated by network monitoring software,

    G. R. Sathi, L. Vedullapalli, M. H. Kishan, T. S. Zaman, M. T. Islam, and M. M. Badr, “Netminer: Identifying failure-inducing patterns in the logs generated by network monitoring software,” in 2023 14th International Conference on Computing Communication and Networking Technol...

  20. [28]

    [Online]

    Gemini. [Online]. Available: https://www.gemini.com/

  21. [29]

    [Online]

    Claude. [Online]. Available: https://www.anthropic.com/claude

  22. [30]

    [Online]

    Llama. [Online]. Available: https://www.llama.com/

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.