{"id":"4ab36a1a-cce9-4cab-98e8-f3464668b685","arxiv_id":"2501.06597","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A small BERT-plus-K-means study reports ChatGPT responses to AI-related tweets are 90% positive versus 27% for human comments, but the cluster-labeling method is undisclosed.","lead":"A short study compares the sentiment of human tweets about generative AI with the sentiment of ChatGPT's responses to those tweets, using BERT embeddings and K-means clustering. It reports that ChatGPT responses come out overwhelmingly positive while human comments are mostly negative, but the procedure for producing those percentages is not described in the paper.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90%-positive ChatGPT claim cannot be derived from the described K-means pipeline, because the paper never specifies how the two unlabeled clusters were mapped to 'positive' and 'negative.'","rationale":"The reader's weakest assumption already identified the core problem: the K-means clusters are assumed to correspond to positive and negative sentiment, but no labeling rule is given. My stress-test confirms that this is the load-bearing gap. The paper's headline claim, that LLM-generated responses are notably more positive than human comments, depends entirely on Table V. Since no method connects the unlabeled clusters to sentiment polarity, the central quantitative finding is not reproducible from the paper alone. The qualitative direction of the finding is plausible given ChatGPT's well-known tendency to produce helpful, non-confrontational text, and the paper does provide some independent support in the form of silhouette scores and sample outputs. However, silhouette scores only measure cluster cohesion, not sentiment, so they do not validate the polarity assignment. The asymmetric cleaning of ChatGPT responses is a further confound, but even with perfectly symmetric cleaning the missing label mapping would still invalidate the specific percentages. Therefore the reader's REJECT verdict stands; no adjustment is needed. I would add that the paper's own Section V threat-to-validity statement, acknowledging that ChatGPT's use of negative words in positive contexts makes unsupervised prediction challenging, directly undermines the confidence with which Table V is presented.","tokens_in":8691,"tokens_out":2912,"duration_ms":31089,"concrete_test":"Obtain the authors' dataset, cleaning scripts, and K-means code, then re-run the exact pipeline with the same BERT embeddings and k=2. Label the two resulting clusters by the sign of the mean VADER sentiment of their member texts, and separately by manual inspection of the top centroid words. If the resulting positive/negative percentages do not reproduce roughly 90% for ChatGPT and 27% for humans, or if the two labeling rules disagree, then Table V is not a determinate consequence of the described method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests entirely on Table V, yet the only modeling step described is K-means with k=2 on BERT embeddings (Section III-D). K-means outputs unlabeled partitions; neither the Silhouette Scores nor the t-SNE plots assign polarity. The paper provides no lexicon, no labeled validation set, no centroid-inspection rule, and no explanation of how 429 human comments or 512 ChatGPT responses yield the stated percentages. Section V even concedes that ChatGPT often uses negative sentiment words in a positive way, making unsupervised prediction challenging. Without an explicit cluster-to-polarity labeling rule, the 90% vs. 27% numbers are not a determinate consequence of the described method; they could be reversed, arbitrary, or based on post hoc inspection. The asymmetric cleaning rule that removes frequent neutral words like 'AI,' 'language,' and 'model' from ChatGPT responses only (Section III-C) is a second bias that could inflate ChatGPT's apparent positivity, but the missing labeling rule is the more fundamental gap because it leaves the headline claim unsupported even before cleaning differences are considered.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EmoXpt, an unsupervised sentiment analysis framework that uses BERT embeddings and k-means clustering to compare the sentiment of human Twitter comments about generative AI with ChatGPT-generated responses to those tweets. The central quantitative claim, stated in the abstract, conclusion, and Table V, is that ChatGPT responses are overwhelmingly positive (90% positive, 10% negative) while human comments are mostly negative (27% positive, 72% negative). The paper also reports silhouette scores for word- and sentence-level clustering and interprets these as evidence that ChatGPT responses are more efficient, cohesive, and consistently positive than human comments. The manuscript includes data collection details over a two-month period in 2023, preprocessing steps, a description of the k-means pipeline, and a discussion of threats to validity.","tokens_in":8960,"tokens_out":3883,"duration_ms":36479,"significance":"If the headline result were properly supported, the paper would offer a useful, albeit modest, data point on the sentiment of LLM-generated responses relative to human commentary. The proposed framework itself, however, is a straightforward application of standard BERT embeddings plus k-means, and the paper's distinctive contribution is entirely the empirical comparison in Table V. Unfortitingly, the paper provides no reproducible pipeline, no code or data, and no validation of the claimed sentiment percentages. The central claim therefore rests on an unspecified methodological step, and the internal data inconsistencies further undermine confidence in the reported numbers. The strengths are the clear motivation to compare human and LLM sentiment and the explicit acknowledgment of some validity threats, but these do not compensate for the missing analysis.","major_comments":[{"comment":"The central result of the paper, that ChatGPT responses are 90% positive and human comments are 27% positive, is not a determinate consequence of the described method. The only modeling step described is k-means with k=2 on BERT embeddings (Section III-D), which outputs unlabeled clusters. Neither the Silhouette Scores in Table IV nor the t-SNE plots in Figures 9–10 assign polarity to clusters. The paper does not specify any cluster-labeling rule, such as centroid inspection against a sentiment lexicon, a labeled validation set, or a bootstrapping procedure. Therefore the percentages in Table V cannot be derived from the described pipeline, and the abstract's and conclusion's claims that LLM-generated responses are 'notably more positive' are unsupported. The authors must specify exactly how clusters were mapped to positive/negative labels and how the percentages were computed.","section":"§IV-B, Table V"},{"comment":"The preprocessing is asymmetric. Human comments are cleaned with steps shown in Figure 5, while ChatGPT responses additionally have 'frequently occurring neutral words' such as 'AI', 'language', and 'model' removed (Figure 6). This step is applied only to ChatGPT responses, not to human comments. Removing words like 'AI' from one corpus but not the other changes the token distributions and resulting BERT embeddings, which in turn can systematically alter the k-means clusters and any derived sentiment percentages. The paper provides no justification for why these words are 'neutral' in ChatGPT responses but not in human comments, and no sensitivity analysis is given to show that the headline 90% versus 27% comparison is robust to this asymmetric cleaning. This is a second load-bearing bias that could inflate ChatGPT's apparent positivity even if the cluster-labeling problem were resolved.","section":"§III-C"},{"comment":"Several data statistics in the paper are internally inconsistent. The paper states there are 512 human tweets, 429 user comments, and ChatGPT responses for each tweet (Section III-A). It also states that 75% of tweets have no comments and that the average number of comments is 28.4 with a maximum of 29; with 429 comments over 512 tweets the mean is approximately 0.84, and an average of 28.4 is arithmetically impossible. Additionally, in Section III-C the sentence embedding matrix is described as '(768, 429) where 768 is the number of data points and 429 is the features extracted for each data point,' which confuses the embedding dimension with the sample count; the correct dimensions would be (429, 768) for human comments and (512, 768) for ChatGPT responses. These inconsistencies indicate that the data handling and reporting are not reliable, which further erodes confidence in Table V.","section":"§III-A and §IV-B"},{"comment":"The interpretation of the silhouette scores is not supported by the metric. The paper states that ChatGPT words have a silhouette score of 0.053 and human words 0.13, and interprets the lower ChatGPT score as indicating 'a more focused, repetitive vocabulary' and 'consistent and direct' language. However, a silhouette score near zero actually indicates that clusters are not well separated and that data points are about as close to their own cluster as to the neighboring cluster; it does not measure vocabulary diversity or focus. The same misunderstanding applies to the sentence-level scores in Table IV. The paper's claim that ChatGPT responses are 'cohesive' based on these scores is therefore not justified.","section":"§IV-B, RQ1 and RQ2"}],"minor_comments":[{"comment":"The claim that ChatGPT responses are 'more efficient, cohesive, and consistently positive' uses terms that are not defined or measured in the experimental section; the paper only measures clustering quality and sentiment polarity, not efficiency or cohesion in any operational sense.","section":"Abstract / Conclusion"},{"comment":"The subsection title 'Data Prepossessing' is a typo for 'Data Preprocessing'.","section":"§III-C heading"},{"comment":"The phrase 'which is usesful for identifying contrasting sentiments' contains a typo ('usesful' should be 'useful').","section":"§III-D"},{"comment":"In 'Threats to Conclusion Validity,' the sentence 'To reduce the conclusion validity of this work' should read 'To reduce the threat to conclusion validity' or similar.","section":"§V"},{"comment":"Several references are incomplete or formatted inconsistently; for example, reference [9] has the title 'Usecase' rather than a meaningful title, and references [15], [23], and [24] mix URL text with the actual resource names.","section":"§VI references"}],"recommendation":"reject","confidential_remarks":"The paper's headline claim is not reproducible from the described method, and the internal inconsistencies in the reported data statistics are severe enough that I cannot see a path to acceptance without a substantial rewrite. The missing cluster-labeling rule alone would require a major revision and reanalysis, and the asymmetric cleaning and numerical contradictions would need to be addressed as well. Even then, the absence of code or data would make verification difficult. The paper may have a legitimate research question, but as submitted it does not provide a sound empirical answer."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's central quantitative claim—ChatGPT responses 90% positive vs human comments 27% positive—does not follow from the method as written. K-means outputs unlabeled clusters, and the paper never specifies how those clusters were mapped to 'positive' and 'negative.' That missing step leaves the headline numbers unsupported, and the stress-test note is right. The paper also cleans human and ChatGPT text asymmetrically, removing 'neutral' words like 'AI' and 'model' only from ChatGPT responses, which could bias the comparison.\n\nWhat's genuinely new: unlike prior work that analyzes human sentiment about ChatGPT on Twitter, this paper looks at the sentiment of ChatGPT's own responses, and it does so with a simple BERT-plus-K-means pipeline. The qualitative conclusion that ChatGPT tends to stay positive is plausible and consistent with the paper's own threat-to-validity note, which admits ChatGPT often uses negative words in a positive way. The authors also engage honestly with related work and list reasonable threats.\n\nThe soft spots are not minor. First, the cluster-labeling gap: without a rule to label the two K-means clusters, the percentages in Table V could be reversed or arbitrary. Silhouette scores only measure cluster compactness, not polarity. Second, the asymmetric cleaning: removing 'AI,' 'language,' and 'model' from ChatGPT text only changes the vocabulary distribution, and it is not controlled. Third, there are numerical inconsistencies: the text says the average number of comments is 28.4 when the max is 29 and most tweets have none, and the hashtag percentages add to roughly 120%. These look like careless errors, but they do not inspire confidence. No code or data is released, so nothing can be checked.\n\nWho benefits: a reader interested in a quick example of why unsupervised sentiment analysis needs careful validation, or someone looking for a baseline on ChatGPT's emotional tone. The paper is not citable for its specific numbers. It deserves a serious referee only if the authors can supply the labeling procedure and release the data; otherwise the central claim is a conjecture, not a result.","headline":"The 90%-positive ChatGPT claim is not derivable from the described K-means pipeline because the paper never explains how clusters were labeled positive or negative.","tokens_in":9402,"tokens_out":2798,"would_cite":false,"duration_ms":25431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ChatGPT's responses to tweets about generative AI are 90% positive while human comments are 72% negative, reflecting LLMs' built-in tendency to stay constructive and positive.","keywords":["sentiment analysis","generative AI","ChatGPT","LLM-generated responses","K-means clustering","BERT embeddings","emotional intelligence","Twitter sentiment"],"falsifier":"Recompute Table V with a documented cluster-labeling rule, for example assigning the cluster whose centroid has higher average overlap with positive sentiment-lexicon words as 'positive'; if the resulting percentages are close to 90% and 27% the claim stands, and if they differ sharply the polarity assignment was arbitrary. A second concrete check is to rerun the full pipeline with identical stopword and neutral-word removal for both human comments and ChatGPT responses, since the paper removes frequent neutral words only from ChatGPT responses.","tokens_in":8433,"feed_emoji":"📊","tokens_out":4661,"duration_ms":44136,"temperature":0.7,"pith_summary":"The paper introduces EmoXpt, a sentiment-analysis pipeline that compares human Twitter comments about generative AI with ChatGPT's replies to the same tweets. Its central empirical claim is that ChatGPT-generated responses are consistently positive, with about 90% classified positive, while human comments are mostly negative, at about 72%. The authors argue this reflects ChatGPT's design to maintain a constructive, positive tone, and that this makes AI responses more cohesive but less emotionally varied than human language. The result matters because it quantifies a suspected asymmetry: conversational AI is trained to avoid negativity, so automated positivity can mask rather than mirror public opinion.","feed_headline":"LLM replies to AI tweets are 90% positive, humans 72% negative","feed_subtitle":"The gap suggests AI's built-in positivity may hide how users actually feel about generative AI.","key_machinery":"The load-bearing mechanism is an unsupervised sentiment-analysis pipeline: BERT word embeddings of 768 dimensions are computed for every token and averaged to sentence embeddings, then K-means clustering with k=2 is applied separately to human words, ChatGPT words, human sentences, and ChatGPT sentences. Cluster quality is measured with the Silhouette score, and t-SNE projects the high-dimensional clusters to two dimensions for visualization. The cleaning steps differ between the two sides: human comments get URLs, emojis, and @mentions removed, while ChatGPT responses additionally have frequent neutral words such as 'AI', 'language', and 'model' stripped out before the same clustering is run.","core_discovery":"On its own terms, EmoXpt claims to show that human comments about generative AI on Twitter are predominantly negative (72%), while ChatGPT's responses to those same tweets are overwhelmingly positive (90%). The sentence-level K-means clusters for ChatGPT reach a Silhouette score of 0.58 versus 0.19 for human comments, which the paper reads as ChatGPT's responses being more focused, cohesive, and uniformly toned. The paper interprets the split as evidence that LLMs exhibit a positivity bias, responding constructively even to negative or fearful prompts, and concludes that AI-generated text is efficient and positive but does not capture the full emotional nuance of human interaction.","pith_inferences":["Going beyond the paper, the headline percentages depend on an unstated rule for assigning 'positive' and 'negative' to the two K-means clusters; labeling the same clusters using a sentiment lexicon could confirm or reverse the 90% versus 27% split.","Because frequent neutral words were removed from ChatGPT responses but not from human comments, the same pipeline with symmetric cleaning on both sides is a direct test of whether the positivity gap is a property of the model or an artifact of preprocessing.","If the claim holds, downstream analytics that use LLM-generated replies as proxies for user sentiment will systematically overestimate approval, which matters for product feedback, public-opinion tracking, and content moderation.","The same two-cluster pipeline could be applied to other language models and to non-English posts to test whether near-universal positivity is model-specific or a general property of aligned conversational AI."],"forward_implications":["If the claim is right, ChatGPT's public-facing text is systematically skewed toward positive sentiment compared with human expression, so treating LLM output as representative human opinion would overestimate support for generative AI.","The 90% versus 27% split suggests that alignment or safety training suppresses negative framing in ChatGPT even when the input tweet is critical or worried.","The large gap in sentence-level Silhouette scores (0.58 for ChatGPT versus 0.19 for humans) implies that ChatGPT's emotional tone is more uniform and less varied than human tone.","The framework gives a label-free way to compare emotional variance across language models and social-media populations, without needing manually annotated sentiment data."],"supporting_citations":[{"why":"Supplies the BERT embeddings used to convert words and sentences into the 768-dimensional feature vectors that K-means clusters.","marker":"[6]"},{"why":"Provides the K-means clustering algorithm that the pipeline uses to separate the data into two sentiment groups.","marker":"[14]"},{"why":"Gives the unsupervised sentiment-analysis pipeline pattern that EmoXpt adapts for both human and ChatGPT text.","marker":"[15]"},{"why":"Defines the Silhouette score used to evaluate whether the two clusters are well separated for each dataset.","marker":"[16]"},{"why":"Supplies t-SNE, the dimensionality-reduction technique used to visualize the 768-dimensional clusters in two dimensions.","marker":"[17]"}],"fun_headline_variants":["ChatGPT replies are 90% positive, humans 72% negative on AI tweets","AI positivity gap: ChatGPT 90% positive vs humans 72% negative","EmoXpt shows ChatGPT's sunny replies vs human gloom on AI tweets","LLMs respond 90% positive to tweets humans see 72% negative","Study: ChatGPT's AI tweet replies 90% positive, humans 72% negative"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central load-bearing premise is that the two K-means clusters correspond to positive and negative sentiment, and that the reported 90% and 27% figures follow from that correspondence; the paper gives no rule for which cluster is which.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT replies are 90% positive, humans 72% negative on AI tweets","AI positivity gap: ChatGPT 90% positive vs humans 72% negative","EmoXpt shows ChatGPT's sunny replies vs human gloom on AI tweets","LLMs respond 90% positive to tweets humans see 72% negative","Study: ChatGPT's AI tweet replies 90% positive, humans 72% negative"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1421,"prompt_tokens":806,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":422,"tokens_out":615,"duration_ms":5508,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:56:27.198009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Table V with a documented cluster-labeling rule, for example assigning the cluster whose centroid has higher average overlap with positive sentiment-lexicon words as 'positive'; if the resulting percentages are close to 90% and 27% the claim stands, and if they differ sharply the polarity assignment was arbitrary. A second concrete check is to rerun the full pipeline with identical stopword and neutral-word removal for both human comments and ChatGPT responses, since the paper removes frequent neutral words only from ChatGPT responses.","supporting_citations":[{"cited_title":"Homophobic and hate speech detection using multilingual-bert model on turkish social media,","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT embeddings used to convert words and sentences into the 768-dimensional feature vectors that K-means clusters."},{"cited_title":"An unsupervised fuzzy clustering method for twitter sentiment analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the K-means clustering algorithm that the pipeline uses to separate the data into two sentiment groups."},{"cited_title":"Twitter, myspace, digg: Unsupervised sentiment analysis in social media,","cited_arxiv_id":null,"evidence_quote":"Gives the unsupervised sentiment-analysis pipeline pattern that EmoXpt adapts for both human and ChatGPT text."}],"review_version":1}