{"id":"d45d0225-b130-4555-9e2e-5ff97bc7100e","arxiv_id":"2412.00530","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GPT-3.5's creativity ratings of short stories barely correlate with human ratings, and when scoring its own stories it leans on emotional features while humans lean on semantic network structure.","lead":"This study maps short stories as networks of word connections and counts emotional words, then asks whether these features explain creativity scores given by humans and by GPT-3.5. It finds GPT-3.5 rates stories very differently from humans, favoring its own emotionally rich stories and relying on emotional cues when scoring them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The emotion-shift claim is confounded: GPT-rated GPT stories differ from human stories in length and emotional features, so SHAP differences may reflect the texts being rated, not a change in GPT-3.5's rating criteria.","rationale":"The reader correctly identifies the differing class encodings and the absence of a word-length control as fragile assumptions. I see an even more basic confound underlying both: the emotion-shift comparison in Section 4.3 is across different story sets, not within a fixed set of texts. SHAP importance depends on the feature distribution of the training data, so the fact that GPT stories are longer and emotionally richer than human stories can by itself make emotion features more important in the GPT-rating-GPT model, without any change in GPT-3.5's internal evaluation criteria. The paper's rating-level claims are well supported: human and GPT-3.5 ratings of the same human stories are uncorrelated (r = 0.006, p = 0.848), and the distributions differ significantly. Those findings justify the cautionary conclusion. But the specific claim that GPT-3.5 shifts from network to emotional features when rating its own stories needs the proposed length-matched, common-encoding control before it can be treated as evidence about GPT-3.5's evaluation policy. Since the reader already returned CONDITIONAL, my concern does not move the verdict; it strengthens the conditions under which the paper should be accepted.","tokens_in":26992,"tokens_out":4469,"duration_ms":74403,"concrete_test":"Retrain the GPT-rating-GPT model on a length-matched subset of GPT stories (e.g., GPT stories at most 85 words, or propensity-matched to human stories on word count) and with the same three-class encoding used for the human-rated models (1–2/3/4–5); recompute SHAP. If emotion features no longer dominate, the paper's central 'shift to emotional features' is an artifact of text length or stimulus properties rather than GPT-3.5's evaluation criteria; if they still dominate, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's headline finding—that GPT-3.5 relies on emotional rather than network features when rating its own stories—is obtained by comparing SHAP values from two models trained on different text populations with different target encodings. Model B (GPT-rating-human) is trained on human stories (average 70 words; Section 4.1) with classes 1–2/3/4–5, while Model C (GPT-rating-GPT) is trained on GPT stories (average 121 words) with classes 3/4/5 (Section 3, Figure 2). These datasets also differ significantly in clustering coefficient, joy, trust, anticipation, fear, and disgust (Table 2). SHAP importance is a property of a fitted model plus its training distribution, not a direct measurement of a rater's stable evaluation policy. If GPT stories have more variance or more predictive signal in emotion features, SHAP will rank those features higher even if GPT-3.5's judgment process is identical. Thus the 'shift from network to emotion' conflates a stimulus effect (GPT stories are longer and emotionally richer) with a rater effect. The absence of confidence intervals for SHAP differences and the different class encodings make the comparison even harder to interpret. The rating-level divergence (r = 0.006) is credible, but the feature-shift claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares how human raters and GPT-3.5 rate short stories for creativity, using textual forma mentis network (TFMN) features: five network measures (diameter, average shortest path length, clustering coefficient, degree centrality, PageRank centrality) and eight emotion z-scores derived from the stories. Three XGBoost classifiers are trained: humans rating human stories, GPT-3.5 rating human stories, and GPT-3.5 rating GPT-3.5-generated stories. The authors report near-zero correlations between human and GPT-3.5 ratings of the same human stories, and use SHAP feature importance to conclude that network features dominate human ratings and GPT-3.5's ratings of human stories, while emotion features dominate GPT-3.5's ratings of its own stories. The paper interprets this as evidence that GPT-3.5 uses different evaluation criteria when assessing its own output and cautions against using GPT-3.5 for creativity assessment.","tokens_in":27123,"tokens_out":9046,"duration_ms":82759,"significance":"If the rating-level results are taken alone—human and GPT-3.5 ratings of the same stories are essentially uncorrelated, and GPT-3.5 assigns no low scores to its own stories—the paper provides a useful, falsifiable descriptive comparison with an open dataset. The interpretable-feature approach is a strength: rather than a black-box embedding, the authors use network and emotion variables grounded in Mednick's associative theory, and they make the GPT-generated stories and rating code publicly available. However, the paper's headline contribution, the SHAP-based feature shift from network structure to emotion when GPT-3.5 rates its own stories, is not currently supported because the three models differ in class encoding and in the text population being rated. These confounds must be addressed before the qualitative XAI comparison can be accepted.","major_comments":[{"comment":"The three models are trained on different target encodings. For human-rating-human and GPT-rating-human, class 0 = ratings 1–2, class 1 = 3, class 2 = 4–5; for GPT-rating-GPT, class 0 = 3, class 1 = 4, class 2 = 5. The SHAP comparison in Figures 5 and 6 treats feature importances across these models as directly comparable, but SHAP values summarize a fitted model's decision boundary, which depends on the class definitions and class priors. The 'low' class in the GPT-rating-GPT model is actually a mid-creativity rating, and the class balance is very different (Table 6 shows recall of 0.10 for class 0). The shift from network to emotion features in model C could therefore be an artifact of the encoding and class imbalance rather than a change in GPT-3.5's rating policy. The authors should re-run the comparison under a common encoding (e.g., regression on the raw 1–5 scale, or a three-class model with the same cut points applied to all datasets) and show that the feature-shift result survives.","section":"Section 3, Figure 2, and Section 4.3"},{"comment":"The feature-shift claim is also confounded by the fact that the three models are trained on different text populations. GPT-generated stories average 121 words versus 70 for human stories, and Table 2 shows significant differences in clustering coefficient, joy, trust, anticipation, disgust, fear, and PageRank. SHAP importance is a property of the model and its training distribution, so if GPT stories are longer and have more variance or predictive signal in emotion features, the emotion features will rank higher even if GPT-3.5's latent judgment process is identical to when it rates human stories. Thus the conclusion that GPT-3.5 'relies on' emotions when rating its own stories conflates a stimulus effect with a rater effect. A minimal control would be to include story length as a feature, to match or stratify stories by length, or to apply models trained on one population to a common evaluation set (e.g., score human and GPT stories with the same model) to isolate the rater contribution.","section":"Section 4.1, Table 2, and Section 4.3"},{"comment":"No uncertainty is reported for the SHAP feature importances. The authors use 4-fold cross-validation for model evaluation but do not state whether SHAP values are computed on the full-data model or per fold, and no confidence intervals or statistical tests accompany the qualitative ranking differences in Figures 5 and 6. Given the small number of stories in the low-creativity class of the GPT-rating-GPT model (Table 6), the 'striking difference' between model B and model C could be driven by a few influential stories. Reporting per-fold or bootstrapped SHAP distributions, or at least the variance of the mean absolute SHAP values, is necessary to establish that the feature shift is robust.","section":"Section 4.3, Figures 5 and 6"}],"minor_comments":[{"comment":"The text reports nH = nGPT = 1071 for the story-length Mann-Whitney test, but the reduced human dataset used elsewhere contains 997 stories (Section 2.1.1) and Table 2 reports sH = 997. Please clarify which sample was used for the length comparison.","section":"Section 4.1"},{"comment":"The claim that XGBoost 'consistently outperformed' the other classifiers is contradicted by the GPT-rating-GPT column, where Gradient Boost yields 0.757 versus XGBoost's 0.752; the authors acknowledge this, but the wording should be adjusted.","section":"Section 4.2 and Table 3"},{"comment":"The generation procedure uses the interactive web interface and does not report temperature or other sampling parameters; providing these (or the exact API calls) would improve reproducibility, although the OSF repository mitigates this concern.","section":"Section 2.1.2"},{"comment":"The interpretation of PageRank direction is confusing. Section 1.4 hypothesizes that higher degree and clustering correlate with higher creativity, but Section 5.1 argues that lower PageRank centrality corresponds to higher creativity; the SHAP direction is not made explicit in Figures 5–6, so the reader cannot verify this reading.","section":"Section 5.1"},{"comment":"Multiple Mann-Whitney tests are run without multiple-comparison correction; while many p-values are far below 0.001, the two marginal features near 0.05 should be interpreted cautiously.","section":"Table 2"},{"comment":"The three columns use different class definitions for the GPT-rating-GPT model, so row labels such as 'Low Creativity Stories' are not comparable across columns; this should be stated clearly in the caption and taken into account in the analysis.","section":"Figure 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's central XAI claim needs a reanalysis with common class encoding and matched text populations. The rating-level null result is a useful benchmark, but the feature-shift result is not yet established. No concerns about citation or authorship conduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi X,\n\nQuick take: the paper gives you two separate results. The first, that GPT-3.5's creativity ratings of human stories are essentially uncorrelated with human ratings (r = 0.006) and differ in distribution, is clean and credible. The Mann-Whitney and correlation analyses are straightforward, and the descriptive statistics are fine. The second result, that GPT-3.5 shifts from network features to emotion features when rating its own stories, is the headline and it is not supported by the current comparison.\n\nThe stress-test note has it right. Model B (GPT rating human) is trained on 70-word human stories with classes 1-2/3/4-5; Model C (GPT rating GPT) is trained on 121-word GPT stories with classes 3/4/5. The text populations differ systematically in clustering coefficient, joy, trust, fear, disgust, and length. SHAP values describe the fitted model plus the training distribution, not the rater's stable policy. So the 'shift' could be a stimulus effect. Without a word-length control, a matched comparison, or at least confidence intervals on the SHAP differences, the feature-shift conclusion is a hypothesis, not a finding.\n\nWhat is genuinely new is the GPT-3.5 generated story dataset (1071 stories) and the three-way rater comparison. That is a useful artifact, and they put it on OSF. The model performance tables with error bounds are reported transparently, and they acknowledge the class imbalance in the GPT-rating-GPT model. The qualitative appendix showing the repetitive story openings is a nice touch.\n\nThe soft spots beyond the confound: model selection on the full dataset weakens the 'predictive' framing, the class binning is post hoc, and the absence of sampling parameters for GPT-3.5 makes replication harder. None of these are fatal for the rating-level result. The citation pattern is honest; they build directly on Johnson et al. and Semeraro et al., which is appropriate.\n\nWho benefits: anyone working on automated creativity assessment, LLM evaluation, or AI-in-education. The rating divergence is a useful cautionary data point. The feature-shift claim needs a follow-up with controls before it should be cited as established.\n\nI'd send it to peer review, not desk-reject it, but the revision should face the confound head-on. A matched-length control or a rater-policy test on the same text population would settle it.\n\nRegards,\n\n[Your name]","headline":"The rating-level divergence between GPT-3.5 and humans is credible, but the headline SHAP-based emotion-shift claim is confounded by different text populations and class encodings.","tokens_in":27847,"tokens_out":1743,"would_cite":true,"duration_ms":99400,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-3.5's creativity ratings diverge from human ratings, and the features that explain its ratings change when it judges its own stories: network structure for human stories, emotions for GPT-generated ones.","keywords":["creativity assessment","GPT-3.5","textual forma mentis networks","explainable AI","SHAP","XGBoost","semantic networks","emotion features"],"falsifier":"Re-run the GPT-rating-GPT analysis with matched conditions: ask GPT-3.5 to generate stories at human-like lengths around 70 words, obtain low ratings by instructing it to write deliberately uncreative stories, and encode ratings with the same 1-2/3/4-5 class boundaries used for human stories. If SHAP feature importance under these matched conditions still shows emotional features dominating structural network features, the paper's central claim survives; if the profile flips back to PageRank and degree centrality, the claimed emotion-centric shift is an artifact of length and label encoding.","tokens_in":26632,"feed_emoji":"🤖","tokens_out":8834,"duration_ms":84279,"temperature":0.7,"pith_summary":"This paper asks whether the features that predict human creativity ratings of short stories also predict GPT-3.5's ratings, and whether the same features explain GPT-3.5's ratings of its own AI-generated stories. The authors extract textual forma mentis networks, interpretable networks of syntactic and semantic word associations plus emotion scores, from roughly a thousand human stories and a thousand GPT-3.5 stories, then use XGBoost with SHAP explanations to compare three rating scenarios. They find that human ratings and GPT-3.5 ratings of human stories are best predicted by network-structural features such as PageRank and degree centrality, while GPT-3.5's ratings of its own stories are best predicted by emotional features such as joy, anger, and anticipation. They also find that GPT-3.5's numeric ratings of human stories are not correlated with human ratings and that GPT-3.5 systematically favors its own stories. The message is that GPT-3.5 is not yet aligned with human creativity judgments and should be used cautiously as a creativity evaluator or generator.","feed_headline":"GPT-3.5 rates its own stories by emotion, not structure","feed_subtitle":"Network structure predicts human and GPT ratings of human stories; emotional cues dominate GPT's self-assessments.","key_machinery":"The central machinery is the textual forma mentis network (TFMN) combined with SHAP attribution. A TFMN is an interpretable cognitive-network representation of a text in which words are nodes, syntactic dependency relations within a limited distance become edges, lemmatisation merges word variants, and negations flip emotional valence, with eight emotion z-scores computed against a random baseline. XGBoost predicts creativity rating classes from these network and emotion features, and SHAP values convert the ensemble's decisions into per-feature importance estimates, allowing the authors to compare which textual properties drive ratings in each scenario.","core_discovery":"The paper's central discovery is that the measurable signature of a creativity rating depends on who is judging and who wrote the story. Using textual forma mentis networks built from syntactic dependency trees, the authors quantify each story through five network features (diameter, average shortest path length, clustering coefficient, degree centrality, PageRank centrality) and eight emotion z-scores. Trained on these thirteen features, an XGBoost classifier predicts human creativity ratings with network centrality features as the strongest SHAP contributors, and it predicts GPT-3.5's ratings of human stories with a similar network-heavy profile. But when GPT-3.5 rates stories it generated itself, the same pipeline shifts to emotion features, especially anger, anticipation, and joy, as the dominant predictors. The authors interpret this as evidence that GPT-3.5's self-assessment of creativity relies on emotional tone rather than on the associative network structure that human raters appear to use, and they note a clear evaluative bias: no GPT-3.5-produced story received the lowest ratings, whereas human stories were rated across the full scale.","pith_inferences":[],"forward_implications":["If the result holds, GPT-3.5's creativity scores cannot be treated as a stand-in for human judgment, since its ratings of the same human stories are statistically uncorrelated with human ratings.","Using GPT-3.5 to evaluate creative writing in educational or professional settings risks rewarding a different, emotion-focused notion of creativity than the network-structural patterns human raters respond to.","The same interpretable feature pipeline can be applied to newer language models to determine whether the emotion-centric shift is specific to GPT-3.5 or persists as models improve.","The finding that GPT-3.5 uses network features when rating human stories but emotional features when rating its own stories suggests that generation and evaluation in large language models may rely on different internal criteria, which matters for AI-assisted creative writing.","Creativity assessments reported by AI raters should be accompanied by source-specific summary statistics, because the model's preference for its own stories can change the rating distribution dramatically.","An implication the paper leaves implicit: the emotion-centric profile for GPT-rated GPT stories could be tested by asking GPT-3.5 to rate its own stories after trimming them to human-like length and after forcing identical class encodings, which would separate a genuine evaluation preference from the confound of longer, more emotionally dense texts.","An editor's inference is that the SHAP comparisons across models are qualitative, so a permutation-based statistical test of feature-rank differences would tell whether the network-versus-emotion shift is robust or a threshold artifact of the thirteen-feature set.","A further extension, following the paper's future-direction idea, is to instruct GPT-3.5 to write deliberately uncreative stories; if such stories receive low ratings based on structural features, then GPT can recognise uncreative structure even while its default output emphasises emotion."],"supporting_citations":[{"why":"Supplies the human-authored short-story dataset and the human creativity ratings that two of the three models predict.","marker":"Johnson et al. (2023)"},{"why":"Introduces textual forma mentis networks, the representation used to extract syntactic and semantic structure from texts.","marker":"Stella (2020)"},{"why":"Provides the EmoAtlas feature extraction pipeline, including emotion z-scores and network measures used as model inputs.","marker":"Semeraro et al. (2024)"},{"why":"Supplies the associative theory of creativity that motivates using network distance and structure as creativity predictors.","marker":"Mednick (1962)"},{"why":"Provides the SHAP method used to determine feature importance for interpreting the XGBoost predictions.","marker":"Lundberg & Lee (2017)"},{"why":"Provides the XGBoost classifier that generates the predictions explained by SHAP.","marker":"Chen & Guestrin (2016)"},{"why":"Supplies the framework of eight basic emotions on which the emotion features are based.","marker":"Plutchik (1980)"},{"why":"Provides the EmoLex lexicon used to assign emotional values to words during TFMN construction.","marker":"Mohammad & Turney (2013)"}],"fun_headline_variants":["GPT-3.5 judges its own stories by emotion, not structure","Emotion beats structure when GPT-3.5 rates its own stories","GPT-3.5 self-ratings lean on emotion, humans on network structure","For self-assessment, GPT-3.5 keys on emotion, not network structure","Network structure drives human creativity scores; emotion drives GPT-3.5's own"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that comparing SHAP feature importances across the three models is valid even though the creativity scores are grouped differently for GPT-rated GPT stories (3/4/5 instead of 1-2/3/4-5) and the stories differ in average length (70 versus 121 words), so the shift from network to emotional features could be an artifact of grouping or length rather than a genuine difference in GPT-3.5's evaluation criteria.","fun_headline_variants_meta":{"raw":{"variants":["GPT-3.5 judges its own stories by emotion, not structure","Emotion beats structure when GPT-3.5 rates its own stories","GPT-3.5 self-ratings lean on emotion, humans on network structure","For self-assessment, GPT-3.5 keys on emotion, not network structure","Network structure drives human creativity scores; emotion drives GPT-3.5's own"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000913,"raw_usage":{"total_tokens":3974,"prompt_tokens":1052,"completion_tokens":2922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":2833}},"tokens_in":668,"tokens_out":2922,"duration_ms":18134,"temperature":1.0,"reasoning_tokens":2833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:16:08.757132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the GPT-rating-GPT analysis with matched conditions: ask GPT-3.5 to generate stories at human-like lengths around 70 words, obtain low ratings by instructing it to write deliberately uncreative stories, and encode ratings with the same 1-2/3/4-5 class boundaries used for human stories. If SHAP feature importance under these matched conditions still shows emotional features dominating structural network features, the paper's central claim survives; if the profile flips back to PageRank and degree centrality, the claimed emotion-centric shift is an artifact of length and label encoding.","supporting_citations":[{"cited_title":"R., Kaufman, J","cited_arxiv_id":null,"evidence_quote":"Supplies the human-authored short-story dataset and the human creativity ratings that two of the three models predict."}],"review_version":1}