REVIEW 3 major objections 6 minor 12 references
Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read GPT-3.5's creativity ratings diverge from human ratings, and the features that explain its ratings change when it judges its own stories: network structure for human stories, emotions for GPT-generated ones.
desk verdict The rating-level divergence between GPT-3.5 and humans is credible, but the headline SHAP-based emotion-shift claim is confounded by different text populations and class encodings. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the textual forma mentis network (TFMN) combined with SHAP attribution. A TFMN is an interpretable cognitive-network representation of a text in which words are nodes, syntactic dependency relations within a limited distance become edges, lemmatisation merges word variants, and negations flip emotional valence, with eight emotion z-scores computed against a random baseline. XGBoost predicts creativity rating classes from these network and emotion features, and SHAP values convert the ensemble's decisions into per-feature importance estimates, allowing the authors to compare which textual properties drive ratings in each scenario.
What would settle it
Re-run the GPT-rating-GPT analysis with matched conditions: ask GPT-3.5 to generate stories at human-like lengths around 70 words, obtain low ratings by instructing it to write deliberately uncreative stories, and encode ratings with the same 1-2/3/4-5 class boundaries used for human stories. If SHAP feature importance under these matched conditions still shows emotional features dominating structural network features, the paper's central claim survives; if the profile flips back to PageRank and degree centrality, the claimed emotion-centric shift is an artifact of length and label encoding.
Extended reading notes
Core claim
The paper's central discovery is that the measurable signature of a creativity rating depends on who is judging and who wrote the story. Using textual forma mentis networks built from syntactic dependency trees, the authors quantify each story through five network features (diameter, average shortest path length, clustering coefficient, degree centrality, PageRank centrality) and eight emotion z-scores. Trained on these thirteen features, an XGBoost classifier predicts human creativity ratings with network centrality features as the strongest SHAP contributors, and it predicts GPT-3.5's ratings of human stories with a similar network-heavy profile. But when GPT-3.5 rates stories it generated itself, the same pipeline shifts to emotion features, especially anger, anticipation, and joy, as the dominant predictors. The authors interpret this as evidence that GPT-3.5's self-assessment of creativity relies on emotional tone rather than on the associative network structure that human raters appear to use, and they note a clear evaluative bias: no GPT-3.5-produced story received the lowest ratings, whereas human stories were rated across the full scale.
Load-bearing premise
The load-bearing assumption is that comparing SHAP feature importances across the three models is valid even though the creativity scores are grouped differently for GPT-rated GPT stories (3/4/5 instead of 1-2/3/4-5) and the stories differ in average length (70 versus 121 words), so the shift from network to emotional features could be an artifact of grouping or length rather than a genuine difference in GPT-3.5's evaluation criteria.
Editorial extensions
If this is right
- If the result holds, GPT-3.5's creativity scores cannot be treated as a stand-in for human judgment, since its ratings of the same human stories are statistically uncorrelated with human ratings.
- Using GPT-3.5 to evaluate creative writing in educational or professional settings risks rewarding a different, emotion-focused notion of creativity than the network-structural patterns human raters respond to.
- The same interpretable feature pipeline can be applied to newer language models to determine whether the emotion-centric shift is specific to GPT-3.5 or persists as models improve.
- The finding that GPT-3.5 uses network features when rating human stories but emotional features when rating its own stories suggests that generation and evaluation in large language models may rely on different internal criteria, which matters for AI-assisted creative writing.
- Creativity assessments reported by AI raters should be accompanied by source-specific summary statistics, because the model's preference for its own stories can change the rating distribution dramatically.
- An implication the paper leaves implicit: the emotion-centric profile for GPT-rated GPT stories could be tested by asking GPT-3.5 to rate its own stories after trimming them to human-like length and after forcing identical class encodings, which would separate a genuine evaluation preference from the confound of longer, more emotionally dense texts.
- An editor's inference is that the SHAP comparisons across models are qualitative, so a permutation-based statistical test of feature-rank differences would tell whether the network-versus-emotion shift is robust or a threshold artifact of the thirteen-feature set.
- A further extension, following the paper's future-direction idea, is to instruct GPT-3.5 to write deliberately uncreative stories; if such stories receive low ratings based on structural features, then GPT can recognise uncreative structure even while its default output emphasises emotion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares how human raters and GPT-3.5 rate short stories for creativity, using textual forma mentis network (TFMN) features: five network measures (diameter, average shortest path length, clustering coefficient, degree centrality, PageRank centrality) and eight emotion z-scores derived from the stories. Three XGBoost classifiers are trained: humans rating human stories, GPT-3.5 rating human stories, and GPT-3.5 rating GPT-3.5-generated stories. The authors report near-zero correlations between human and GPT-3.5 ratings of the same human stories, and use SHAP feature importance to conclude that network features dominate human ratings and GPT-3.5's ratings of human stories, while emotion features dominate GPT-3.5's ratings of its own stories. The paper interprets this as evidence that GPT-3.5 uses different evaluation criteria when assessing its own output and cautions against using GPT-3.5 for creativity assessment.
Significance. If the rating-level results are taken alone—human and GPT-3.5 ratings of the same stories are essentially uncorrelated, and GPT-3.5 assigns no low scores to its own stories—the paper provides a useful, falsifiable descriptive comparison with an open dataset. The interpretable-feature approach is a strength: rather than a black-box embedding, the authors use network and emotion variables grounded in Mednick's associative theory, and they make the GPT-generated stories and rating code publicly available. However, the paper's headline contribution, the SHAP-based feature shift from network structure to emotion when GPT-3.5 rates its own stories, is not currently supported because the three models differ in class encoding and in the text population being rated. These confounds must be addressed before the qualitative XAI comparison can be accepted.
major comments (3)
- [Section 3, Figure 2, and Section 4.3] The three models are trained on different target encodings. For human-rating-human and GPT-rating-human, class 0 = ratings 1–2, class 1 = 3, class 2 = 4–5; for GPT-rating-GPT, class 0 = 3, class 1 = 4, class 2 = 5. The SHAP comparison in Figures 5 and 6 treats feature importances across these models as directly comparable, but SHAP values summarize a fitted model's decision boundary, which depends on the class definitions and class priors. The 'low' class in the GPT-rating-GPT model is actually a mid-creativity rating, and the class balance is very different (Table 6 shows recall of 0.10 for class 0). The shift from network to emotion features in model C could therefore be an artifact of the encoding and class imbalance rather than a change in GPT-3.5's rating policy. The authors should re-run the comparison under a common encoding (e.g., regression on the raw 1–5 scale, or a three-class model with the same cut points applied to all datasets) and show that the feature-shift result survives.
- [Section 4.1, Table 2, and Section 4.3] The feature-shift claim is also confounded by the fact that the three models are trained on different text populations. GPT-generated stories average 121 words versus 70 for human stories, and Table 2 shows significant differences in clustering coefficient, joy, trust, anticipation, disgust, fear, and PageRank. SHAP importance is a property of the model and its training distribution, so if GPT stories are longer and have more variance or predictive signal in emotion features, the emotion features will rank higher even if GPT-3.5's latent judgment process is identical to when it rates human stories. Thus the conclusion that GPT-3.5 'relies on' emotions when rating its own stories conflates a stimulus effect with a rater effect. A minimal control would be to include story length as a feature, to match or stratify stories by length, or to apply models trained on one population to a common evaluation set (e.g., score human and GPT stories with the same model) to isolate the rater contribution.
- [Section 4.3, Figures 5 and 6] No uncertainty is reported for the SHAP feature importances. The authors use 4-fold cross-validation for model evaluation but do not state whether SHAP values are computed on the full-data model or per fold, and no confidence intervals or statistical tests accompany the qualitative ranking differences in Figures 5 and 6. Given the small number of stories in the low-creativity class of the GPT-rating-GPT model (Table 6), the 'striking difference' between model B and model C could be driven by a few influential stories. Reporting per-fold or bootstrapped SHAP distributions, or at least the variance of the mean absolute SHAP values, is necessary to establish that the feature shift is robust.
minor comments (6)
- [Section 4.1] The text reports nH = nGPT = 1071 for the story-length Mann-Whitney test, but the reduced human dataset used elsewhere contains 997 stories (Section 2.1.1) and Table 2 reports sH = 997. Please clarify which sample was used for the length comparison.
- [Section 4.2 and Table 3] The claim that XGBoost 'consistently outperformed' the other classifiers is contradicted by the GPT-rating-GPT column, where Gradient Boost yields 0.757 versus XGBoost's 0.752; the authors acknowledge this, but the wording should be adjusted.
- [Section 2.1.2] The generation procedure uses the interactive web interface and does not report temperature or other sampling parameters; providing these (or the exact API calls) would improve reproducibility, although the OSF repository mitigates this concern.
- [Section 5.1] The interpretation of PageRank direction is confusing. Section 1.4 hypothesizes that higher degree and clustering correlate with higher creativity, but Section 5.1 argues that lower PageRank centrality corresponds to higher creativity; the SHAP direction is not made explicit in Figures 5–6, so the reader cannot verify this reading.
- [Table 2] Multiple Mann-Whitney tests are run without multiple-comparison correction; while many p-values are far below 0.001, the two marginal features near 0.05 should be interpreted cautiously.
- [Figure 6 caption] The three columns use different class definitions for the GPT-rating-GPT model, so row labels such as 'Low Creativity Stories' are not comparable across columns; this should be stated clearly in the caption and taken into account in the analysis.
Circularity Check
No circular derivation: ratings and features are externally grounded; SHAP/GPT comparisons are descriptive and confounded but not circular.
full rationale
The derivation chain is self-contained. Text features are computed from raw stories through a transparent pipeline (spaCy syntactic parsing, TFMN construction, EmoLex-based z-scores) that does not encode the creativity labels or the paper's conclusions. Human creativity ratings come from an independent external dataset (Johnson et al., 2023); GPT ratings are collected from separate API calls with documented prompts. XGBoost is fitted to those external ratings and SHAP values summarize the fitted model; no parameter is fitted to one subset and then presented as a prediction of that same subset, and the authors explicitly state that their aim is not generalization but feature description. The central GPT-versus-human divergence rests on direct correlations (r = 0.006), independent of any model. Self-citations to TFMN and EmoAtlas are methodology and tool citations with reproducible code and externally validated lexicons, not unverified uniqueness theorems, so they do not make the argument circular. The acknowledged limitations (different class encodings across models, GPT stories longer and emotionally richer, and low frequency of low-rated GPT stories) are genuine threats to the causal interpretation of SHAP comparisons, but they are confounds and validity risks, not reductions of the claim to its own inputs by construction.
Assumptions & free parameters
free parameters (3)
- Syntax-tree link distance threshold =
3 steps
- Rating class binning thresholds =
human models: 1-2, 3, 4-5; GPT-GPT model: 3, 4, 5
- GPT-3.5 generation and rating sampling parameters =
not reported
assumptions (4)
- domain assumption Syntactic dependency edges in TFMNs reflect associative closeness relevant to creativity
- domain assumption GPT-3.5 API single-number ratings are a valid operationalization of creativity judgments
- ad hoc to paper Different label codings across models are comparable for SHAP analysis
- domain assumption Manual chat resets yield independent GPT-3.5 participants
Cite this review
Pith. "Pith review of Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters." pith.science (2026). https://pith.science/paper/WMRQK34Y
@misc{pith2026241200530,
author = {Pith},
title = {Pith review of: Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters},
year = {2026},
howpublished = {\url{https://pith.science/paper/WMRQK34Y}},
note = {Machine review of arXiv:2412.00530}
}
read the original abstract
Creativity is a fundamental skill of human cognition. We use textual forma mentis networks (TFMN) to extract network (semantic/syntactic associations) and emotional features from approximately one thousand human- and GPT3.5-generated stories. Using Explainable Artificial Intelligence (XAI), we test whether features relative to Mednick's associative theory of creativity can explain creativity ratings assigned by humans and GPT-3.5. Using XGBoost, we examine three scenarios: (i) human ratings of human stories, (ii) GPT-3.5 ratings of human stories, and (iii) GPT-3.5 ratings of GPT-generated stories. Our findings reveal that GPT-3.5 ratings differ significantly from human ratings not only in terms of correlations but also because of feature patterns identified with XAI methods. GPT-3.5 favours 'its own' stories and rates human stories differently from humans. Feature importance analysis with SHAP scores shows that: (i) network features are more predictive for human creativity ratings but also for GPT-3.5's ratings of human stories; (ii) emotional features played a greater role than semantic/syntactic network structure in GPT-3.5 rating its own stories. These quantitative results underscore key limitations in GPT-3.5's ability to align with human assessments of creativity. We emphasise the need for caution when using GPT-3.5 to assess and generate creative content, as it does not yet capture the nuanced complexity that characterises human creativity.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Acar, S. (2023). Creativity Assessment, Research, and Practice in the Age of Artificial Intelligence. Creativity Research Journal, 1-7. Amabile, T. M., Barsade, S. G., Mueller, J. S., & Staw , B. M. (2005). Affect and creativity at work. Administrative science quarterly, 50(3), 367-403. Amancio, D. R. (2015). A complex network approach to stylometry. PloS...
work page 2023
-
[7]
Kopcsó, K., & Láng, A. (2017). Regulated divergence: Textual patterns, creativity and cognitive emotion regulation. Creativity Research Journal, 29(2), 218-223. Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems,
work page 2017
-
[10]
Vrana, S. R., Bono, R. S., Konig, A., & Scalzo, G. C. (2019). Assessing the coherence of narratives of traumatic events with latent semantic analysis. Psychological Trauma: Theory, Research, Practice, and Policy, 11(5),
work page 2019
-
[30]
M., Erion, G., Chen, H., DeGrave, A., Prutkin, J
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., ... & Lee, S. I. (2019). Explainable AI for trees: From local explanations to global understanding. arXiv preprint arXiv:1905.04610. Mednick, S. (1962). The associative basis of the creative process. Psychological review, 69(3),
arXiv 2019
-
[144]
Hox, J., Moerbeek, M., & Van de Schoot, R. (2017). Multilevel analysis: Techniques and applications. Routledge. Improta, R., Veltri, G. A., & Stella, M. (2024). TextualLLMap: A dataset of 28,000 Large Language Models’ writings designed to expose their biases on societal issues. PsyArXiv. https://doi.org/10.31234/osf.io/xwpe8 Ivcevic, Z., & Brackett, M. A....
-
[220]
Miall, D. S. (2011). Emotions and the structuring of narrative responses. Poetics Today, 32(2), 323-348. Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word –emotion association lexicon. Computational intelligence, 29(3), 436-465. Newman, M. E. J. (2010). Networks: An introduction. Oxford University Press. Orwig, W., Diez, I., Vannini, P., Beaty...
work page 2011
-
[407]
Dietterich, T. G. (2000, June). Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Berlin, Heidelberg: Springer Berlin Heidelberg. D'Souza, R. (2021). What characterises creativity in narrative writing, and how do we assess it? Research findings from a systematic literature search. Thinking Skills an...
work page 2021
-
[480]
Johnson, D. R., Kaufman, J. C., Baker, B. S., Patterson, J. D., Barbot, B., Green, A. E., ... & Beaty, R. E. (2023). Divergent semantic integration (DSI): Extracting creativity from narratives with distributional semantic modeling. Behavior Research Methods , 55(7), 3726-3759. Kenett, Y. N. (2019). What can quantitative measures of semantic distance tell ...
work page 2023
Show all 12 references
-
[521]
Wang, Y., Chen, J., & Yue, Z. (2017). Positive emotion facilitates cognitive flexibility: An fMRI study. Frontiers in psychology, 8,
2017
-
[779]
E., & Johnson, D
Beaty, R. E., & Johnson, D. R. (2021). Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior research methods, 53(2), 757-
2021
-
[780]
Berahmand, K., Bouyer, A., & Samadi, N. (2018). A new centrality measure based on the negative and positive effects of clustering coefficient for identifying influential spreaders in complex networks. Chaos, Solitons & Fractals, 110, 41-54. Boden, M. A. (1998). Creativity and ...
2018
-
[1832]
stamp",
Weinstein, T. J., Ceh, S. M., Meinel, C., & Benedek, M. (2022). What’s creative about sentences? A computational approach to assessing creativity in a sentence generation task. Creativity Research Journal, 34(4), 419-430. Zedelius, C. M., Mills, C., & Schooler, J. W. (2019). B...
2022
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.