Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read GPT-3.5's creativity ratings diverge from human ratings, and the features that explain its ratings change when it judges its own stories: network structure for human stories, emotions for GPT-generated ones.

desk verdict The rating-level divergence between GPT-3.5 and humans is credible, but the headline SHAP-based emotion-shift claim is confounded by different text populations and class encodings. read the letter →

arxiv 2412.00530 v1 pith:WMRQK34Y submitted 2024-11-30 cs.AI cs.CLcs.LG

classification cs.AIcs.CLcs.LG
keywords creativityassessmentGPT-3.5textualformamentisnetworksexplainableAISHAPXGBoostsemanticemotionfeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the features that predict human creativity ratings of short stories also predict GPT-3.5's ratings, and whether the same features explain GPT-3.5's ratings of its own AI-generated stories. The authors extract textual forma mentis networks, interpretable networks of syntactic and semantic word associations plus emotion scores, from roughly a thousand human stories and a thousand GPT-3.5 stories, then use XGBoost with SHAP explanations to compare three rating scenarios. They find that human ratings and GPT-3.5 ratings of human stories are best predicted by network-structural features such as PageRank and degree centrality, while GPT-3.5's ratings of its own stories are best predicted by emotional features such as joy, anger, and anticipation. They also find that GPT-3.5's numeric ratings of human stories are not correlated with human ratings and that GPT-3.5 systematically favors its own stories. The message is that GPT-3.5 is not yet aligned with human creativity judgments and should be used cautiously as a creativity evaluator or generator.

What carries the argument

The central machinery is the textual forma mentis network (TFMN) combined with SHAP attribution. A TFMN is an interpretable cognitive-network representation of a text in which words are nodes, syntactic dependency relations within a limited distance become edges, lemmatisation merges word variants, and negations flip emotional valence, with eight emotion z-scores computed against a random baseline. XGBoost predicts creativity rating classes from these network and emotion features, and SHAP values convert the ensemble's decisions into per-feature importance estimates, allowing the authors to compare which textual properties drive ratings in each scenario.

What would settle it

Re-run the GPT-rating-GPT analysis with matched conditions: ask GPT-3.5 to generate stories at human-like lengths around 70 words, obtain low ratings by instructing it to write deliberately uncreative stories, and encode ratings with the same 1-2/3/4-5 class boundaries used for human stories. If SHAP feature importance under these matched conditions still shows emotional features dominating structural network features, the paper's central claim survives; if the profile flips back to PageRank and degree centrality, the claimed emotion-centric shift is an artifact of length and label encoding.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the measurable signature of a creativity rating depends on who is judging and who wrote the story. Using textual forma mentis networks built from syntactic dependency trees, the authors quantify each story through five network features (diameter, average shortest path length, clustering coefficient, degree centrality, PageRank centrality) and eight emotion z-scores. Trained on these thirteen features, an XGBoost classifier predicts human creativity ratings with network centrality features as the strongest SHAP contributors, and it predicts GPT-3.5's ratings of human stories with a similar network-heavy profile. But when GPT-3.5 rates stories it generated itself, the same pipeline shifts to emotion features, especially anger, anticipation, and joy, as the dominant predictors. The authors interpret this as evidence that GPT-3.5's self-assessment of creativity relies on emotional tone rather than on the associative network structure that human raters appear to use, and they note a clear evaluative bias: no GPT-3.5-produced story received the lowest ratings, whereas human stories were rated across the full scale.

Load-bearing premise

The load-bearing assumption is that comparing SHAP feature importances across the three models is valid even though the creativity scores are grouped differently for GPT-rated GPT stories (3/4/5 instead of 1-2/3/4-5) and the stories differ in average length (70 versus 121 words), so the shift from network to emotional features could be an artifact of grouping or length rather than a genuine difference in GPT-3.5's evaluation criteria.

Editorial extensions

If this is right

  • If the result holds, GPT-3.5's creativity scores cannot be treated as a stand-in for human judgment, since its ratings of the same human stories are statistically uncorrelated with human ratings.
  • Using GPT-3.5 to evaluate creative writing in educational or professional settings risks rewarding a different, emotion-focused notion of creativity than the network-structural patterns human raters respond to.
  • The same interpretable feature pipeline can be applied to newer language models to determine whether the emotion-centric shift is specific to GPT-3.5 or persists as models improve.
  • The finding that GPT-3.5 uses network features when rating human stories but emotional features when rating its own stories suggests that generation and evaluation in large language models may rely on different internal criteria, which matters for AI-assisted creative writing.
  • Creativity assessments reported by AI raters should be accompanied by source-specific summary statistics, because the model's preference for its own stories can change the rating distribution dramatically.
  • An implication the paper leaves implicit: the emotion-centric profile for GPT-rated GPT stories could be tested by asking GPT-3.5 to rate its own stories after trimming them to human-like length and after forcing identical class encodings, which would separate a genuine evaluation preference from the confound of longer, more emotionally dense texts.
  • An editor's inference is that the SHAP comparisons across models are qualitative, so a permutation-based statistical test of feature-rank differences would tell whether the network-versus-emotion shift is robust or a threshold artifact of the thirteen-feature set.
  • A further extension, following the paper's future-direction idea, is to instruct GPT-3.5 to write deliberately uncreative stories; if such stories receive low ratings based on structural features, then GPT can recognise uncreative structure even while its default output emphasises emotion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper compares how human raters and GPT-3.5 rate short stories for creativity, using textual forma mentis network (TFMN) features: five network measures (diameter, average shortest path length, clustering coefficient, degree centrality, PageRank centrality) and eight emotion z-scores derived from the stories. Three XGBoost classifiers are trained: humans rating human stories, GPT-3.5 rating human stories, and GPT-3.5 rating GPT-3.5-generated stories. The authors report near-zero correlations between human and GPT-3.5 ratings of the same human stories, and use SHAP feature importance to conclude that network features dominate human ratings and GPT-3.5's ratings of human stories, while emotion features dominate GPT-3.5's ratings of its own stories. The paper interprets this as evidence that GPT-3.5 uses different evaluation criteria when assessing its own output and cautions against using GPT-3.5 for creativity assessment.

Significance. If the rating-level results are taken alone—human and GPT-3.5 ratings of the same stories are essentially uncorrelated, and GPT-3.5 assigns no low scores to its own stories—the paper provides a useful, falsifiable descriptive comparison with an open dataset. The interpretable-feature approach is a strength: rather than a black-box embedding, the authors use network and emotion variables grounded in Mednick's associative theory, and they make the GPT-generated stories and rating code publicly available. However, the paper's headline contribution, the SHAP-based feature shift from network structure to emotion when GPT-3.5 rates its own stories, is not currently supported because the three models differ in class encoding and in the text population being rated. These confounds must be addressed before the qualitative XAI comparison can be accepted.

major comments (3)
  1. [Section 3, Figure 2, and Section 4.3] The three models are trained on different target encodings. For human-rating-human and GPT-rating-human, class 0 = ratings 1–2, class 1 = 3, class 2 = 4–5; for GPT-rating-GPT, class 0 = 3, class 1 = 4, class 2 = 5. The SHAP comparison in Figures 5 and 6 treats feature importances across these models as directly comparable, but SHAP values summarize a fitted model's decision boundary, which depends on the class definitions and class priors. The 'low' class in the GPT-rating-GPT model is actually a mid-creativity rating, and the class balance is very different (Table 6 shows recall of 0.10 for class 0). The shift from network to emotion features in model C could therefore be an artifact of the encoding and class imbalance rather than a change in GPT-3.5's rating policy. The authors should re-run the comparison under a common encoding (e.g., regression on the raw 1–5 scale, or a three-class model with the same cut points applied to all datasets) and show that the feature-shift result survives.
  2. [Section 4.1, Table 2, and Section 4.3] The feature-shift claim is also confounded by the fact that the three models are trained on different text populations. GPT-generated stories average 121 words versus 70 for human stories, and Table 2 shows significant differences in clustering coefficient, joy, trust, anticipation, disgust, fear, and PageRank. SHAP importance is a property of the model and its training distribution, so if GPT stories are longer and have more variance or predictive signal in emotion features, the emotion features will rank higher even if GPT-3.5's latent judgment process is identical to when it rates human stories. Thus the conclusion that GPT-3.5 'relies on' emotions when rating its own stories conflates a stimulus effect with a rater effect. A minimal control would be to include story length as a feature, to match or stratify stories by length, or to apply models trained on one population to a common evaluation set (e.g., score human and GPT stories with the same model) to isolate the rater contribution.
  3. [Section 4.3, Figures 5 and 6] No uncertainty is reported for the SHAP feature importances. The authors use 4-fold cross-validation for model evaluation but do not state whether SHAP values are computed on the full-data model or per fold, and no confidence intervals or statistical tests accompany the qualitative ranking differences in Figures 5 and 6. Given the small number of stories in the low-creativity class of the GPT-rating-GPT model (Table 6), the 'striking difference' between model B and model C could be driven by a few influential stories. Reporting per-fold or bootstrapped SHAP distributions, or at least the variance of the mean absolute SHAP values, is necessary to establish that the feature shift is robust.
minor comments (6)
  1. [Section 4.1] The text reports nH = nGPT = 1071 for the story-length Mann-Whitney test, but the reduced human dataset used elsewhere contains 997 stories (Section 2.1.1) and Table 2 reports sH = 997. Please clarify which sample was used for the length comparison.
  2. [Section 4.2 and Table 3] The claim that XGBoost 'consistently outperformed' the other classifiers is contradicted by the GPT-rating-GPT column, where Gradient Boost yields 0.757 versus XGBoost's 0.752; the authors acknowledge this, but the wording should be adjusted.
  3. [Section 2.1.2] The generation procedure uses the interactive web interface and does not report temperature or other sampling parameters; providing these (or the exact API calls) would improve reproducibility, although the OSF repository mitigates this concern.
  4. [Section 5.1] The interpretation of PageRank direction is confusing. Section 1.4 hypothesizes that higher degree and clustering correlate with higher creativity, but Section 5.1 argues that lower PageRank centrality corresponds to higher creativity; the SHAP direction is not made explicit in Figures 5–6, so the reader cannot verify this reading.
  5. [Table 2] Multiple Mann-Whitney tests are run without multiple-comparison correction; while many p-values are far below 0.001, the two marginal features near 0.05 should be interpreted cautiously.
  6. [Figure 6 caption] The three columns use different class definitions for the GPT-rating-GPT model, so row labels such as 'Low Creativity Stories' are not comparable across columns; this should be stated clearly in the caption and taken into account in the analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ratings and features are externally grounded; SHAP/GPT comparisons are descriptive and confounded but not circular.

full rationale

The derivation chain is self-contained. Text features are computed from raw stories through a transparent pipeline (spaCy syntactic parsing, TFMN construction, EmoLex-based z-scores) that does not encode the creativity labels or the paper's conclusions. Human creativity ratings come from an independent external dataset (Johnson et al., 2023); GPT ratings are collected from separate API calls with documented prompts. XGBoost is fitted to those external ratings and SHAP values summarize the fitted model; no parameter is fitted to one subset and then presented as a prediction of that same subset, and the authors explicitly state that their aim is not generalization but feature description. The central GPT-versus-human divergence rests on direct correlations (r = 0.006), independent of any model. Self-citations to TFMN and EmoAtlas are methodology and tool citations with reproducible code and externally validated lexicons, not unverified uniqueness theorems, so they do not make the argument circular. The acknowledged limitations (different class encodings across models, GPT stories longer and emotionally richer, and low frequency of low-rated GPT stories) are genuine threats to the causal interpretation of SHAP comparisons, but they are confounds and validity risks, not reductions of the claim to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim does not require a derivation with fitted constants, but it depends on several modeling choices: the 3-step syntax window, the post hoc class binning, GPT-3.5 default sampling, and the assumption that SHAP values from differently encoded classifiers can be compared. None of these are invented entities; they are assumptions about measurement and comparability.

free parameters (3)
  • Syntax-tree link distance threshold = 3 steps
    Chosen by hand in TFMN construction (Section 2.3, step 3). All network features derive from edges defined by this threshold; a different threshold would change every network feature and the subsequent SHAP results.
  • Rating class binning thresholds = human models: 1-2, 3, 4-5; GPT-GPT model: 3, 4, 5
    Chosen post hoc to balance classes after GPT-3.5 never assigned ratings 1 or 2 (Section 3). The different encodings weaken cross-model SHAP comparability.
  • GPT-3.5 generation and rating sampling parameters = not reported
    The paper does not report temperature, top_p, or API model version, which can affect story content and rating distributions.
assumptions (4)
  • domain assumption Syntactic dependency edges in TFMNs reflect associative closeness relevant to creativity
    Section 2.3 builds networks from spaCy syntax trees and Section 1.1 connects ASPL to Mednick's associative theory; the mapping from grammar to association is assumed, not validated against behavioral norms.
  • domain assumption GPT-3.5 API single-number ratings are a valid operationalization of creativity judgments
    Section 2.2.2 only checks output format; no instruction comprehension checks, no calibration, and the resulting near-zero correlation with humans is interpreted as a property of GPT-3.5 rather than as a measurement artifact.
  • ad hoc to paper Different label codings across models are comparable for SHAP analysis
    Section 3 creates class 0/1/2 with different boundaries for GPT-rating-GPT than for human-rating scenarios; Section 4.3 compares SHAP importance across these models without testing whether the encoding changes the ranking.
  • domain assumption Manual chat resets yield independent GPT-3.5 participants
    Section 2.1.2 resets anonymous chats but does not randomize the order or control model state; the formulaic story openings reported in Table 11 suggest outputs are not fully independent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters." pith.science (2026). https://pith.science/paper/WMRQK34Y

@misc{pith2026241200530,
  author       = {Pith},
  title        = {Pith review of: Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WMRQK34Y}},
  note         = {Machine review of arXiv:2412.00530}
}
read the original abstract

Creativity is a fundamental skill of human cognition. We use textual forma mentis networks (TFMN) to extract network (semantic/syntactic associations) and emotional features from approximately one thousand human- and GPT3.5-generated stories. Using Explainable Artificial Intelligence (XAI), we test whether features relative to Mednick's associative theory of creativity can explain creativity ratings assigned by humans and GPT-3.5. Using XGBoost, we examine three scenarios: (i) human ratings of human stories, (ii) GPT-3.5 ratings of human stories, and (iii) GPT-3.5 ratings of GPT-generated stories. Our findings reveal that GPT-3.5 ratings differ significantly from human ratings not only in terms of correlations but also because of feature patterns identified with XAI methods. GPT-3.5 favours 'its own' stories and rates human stories differently from humans. Feature importance analysis with SHAP scores shows that: (i) network features are more predictive for human creativity ratings but also for GPT-3.5's ratings of human stories; (ii) emotional features played a greater role than semantic/syntactic network structure in GPT-3.5 rating its own stories. These quantitative results underscore key limitations in GPT-3.5's ability to align with human assessments of creativity. We emphasise the need for caution when using GPT-3.5 to assess and generate creative content, as it does not yet capture the nuanced complexity that characterises human creativity.

Figures

Figures reproduced from arXiv: 2412.00530 by the authors.

Figure 3
Figure 3. Comparison of mean values with error bars of network features (top) and emotion features (bottom) for human-generated and GPT-3.5-generated stories [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figure 4
Figure 4. 4-fold cross validated confusion matrices for the three different models. For the human￾generated stories: low creativity = rating 1 or 2; mid creativity = rating 3; high creativity = rating 4 or 5. For the GPT-rating-GPT model: low creativity = rating 3; mid creativity = rating 4; high creativity = rating 5 [PITH_FULL_IMAGE:figures/full_fig_p029_4.png] view at source ↗
Figure 5
Figure 5. Feature importance plots for the three different models. For the human-generated stories: class 0 = low creativity (ratings 1 or 2); class 1 = mid creativity (rating 3); class 2 = high creativity (ratings 4 or 5). For the GPT-rating-GPT model: class 0 = low creativity (rating 3); class 1 = mid creativity (rating 4); class 2 = high creativity (rating 5). The panels in [PITH_FULL_IMAGE:figures/full_fig_p031_5.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Feature importance beeswarm plots for the human-rating-human model (left), the GPT￾rating-human model (middle) and the GPT-rating-GPT model (right) separated by class (low, mid and high creativity stories). Feature value intensity is indicated as weak (blue), moderate …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [1]

    Acar, S. (2023). Creativity Assessment, Research, and Practice in the Age of Artificial Intelligence. Creativity Research Journal, 1-7. Amabile, T. M., Barsade, S. G., Mueller, J. S., & Staw , B. M. (2005). Affect and creativity at work. Administrative science quarterly, 50(3), 367-403. Amancio, D. R. (2015). A complex network approach to stylometry. PloS...

  2. [7]

    Kopcsó, K., & Láng, A. (2017). Regulated divergence: Textual patterns, creativity and cognitive emotion regulation. Creativity Research Journal, 29(2), 218-223. Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems,

  3. [10]

    R., Bono, R

    Vrana, S. R., Bono, R. S., Konig, A., & Scalzo, G. C. (2019). Assessing the coherence of narratives of traumatic events with latent semantic analysis. Psychological Trauma: Theory, Research, Practice, and Policy, 11(5),

  4. [30]

    M., Erion, G., Chen, H., DeGrave, A., Prutkin, J

    Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., ... & Lee, S. I. (2019). Explainable AI for trees: From local explanations to global understanding. arXiv preprint arXiv:1905.04610. Mednick, S. (1962). The associative basis of the creative process. Psychological review, 69(3),

  5. [144]

    Hox, J., Moerbeek, M., & Van de Schoot, R. (2017). Multilevel analysis: Techniques and applications. Routledge. Improta, R., Veltri, G. A., & Stella, M. (2024). TextualLLMap: A dataset of 28,000 Large Language Models’ writings designed to expose their biases on societal issues. PsyArXiv. https://doi.org/10.31234/osf.io/xwpe8 Ivcevic, Z., & Brackett, M. A....

  6. [220]

    Miall, D. S. (2011). Emotions and the structuring of narrative responses. Poetics Today, 32(2), 323-348. Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word –emotion association lexicon. Computational intelligence, 29(3), 436-465. Newman, M. E. J. (2010). Networks: An introduction. Oxford University Press. Orwig, W., Diez, I., Vannini, P., Beaty...

  7. [407]

    Dietterich, T. G. (2000, June). Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Berlin, Heidelberg: Springer Berlin Heidelberg. D'Souza, R. (2021). What characterises creativity in narrative writing, and how do we assess it? Research findings from a systematic literature search. Thinking Skills an...

  8. [480]

    R., Kaufman, J

    Johnson, D. R., Kaufman, J. C., Baker, B. S., Patterson, J. D., Barbot, B., Green, A. E., ... & Beaty, R. E. (2023). Divergent semantic integration (DSI): Extracting creativity from narratives with distributional semantic modeling. Behavior Research Methods , 55(7), 3726-3759. Kenett, Y. N. (2019). What can quantitative measures of semantic distance tell ...

Show all 12 references
  1. [521]

    Wang, Y., Chen, J., & Yue, Z. (2017). Positive emotion facilitates cognitive flexibility: An fMRI study. Frontiers in psychology, 8,

  2. [779]

    E., & Johnson, D

    Beaty, R. E., & Johnson, D. R. (2021). Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior research methods, 53(2), 757-

  3. [780]

    Berahmand, K., Bouyer, A., & Samadi, N. (2018). A new centrality measure based on the negative and positive effects of clustering coefficient for identifying influential spreaders in complex networks. Chaos, Solitons & Fractals, 110, 41-54. Boden, M. A. (1998). Creativity and ...

  4. [1832]

    stamp",

    Weinstein, T. J., Ceh, S. M., Meinel, C., & Benedek, M. (2022). What’s creative about sentences? A computational approach to assessing creativity in a sentence generation task. Creativity Research Journal, 34(4), 419-430. Zedelius, C. M., Mills, C., & Schooler, J. W. (2019). B...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.