Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Comparing generated stories against human-written references, test by test, makes LLM creativity judgments align with human experts more closely than rating stories alone.

desk verdict A plausible reference-based creativity evaluator whose headline gain is likely inflated by tuning on the test set; worth a revision, not a reject. read the letter →

arxiv 2504.15784 v1 pith:D24G6AOJ submitted 2025-04-22 cs.CL cs.AI

Ruizhe Li , Chiwei Zhu , Benfeng Xu , Xiaorui Wang , Zhendong Mao This is my paper · ORCID
classification cs.CLcs.AI
keywords creativityevaluationlargelanguagemodelsTorranceTestofCreativeWritingreference-basedLikertscaleLLM-as-judgestorygenerationautomated
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM judges can evaluate creative writing almost as reliably as human experts when the judgment is reframed as a relative comparison rather than an absolute rating. Each machine-written story is scored against a high-quality human-written reference on the 14 binary tests of the Torrance Test of Creative Writing, covering Fluency, Flexibility, Originality, and Elaboration. The judge model analyzes both stories against the criterion before choosing a five-level Likert verdict, the presentation order is alternated to cancel position bias, and a test counts as passed when the average verdict clears a cutoff of -2. On 12 storylines with human expert rankings as ground truth, the method lifts average pairwise accuracy to 0.75, a 15 percent improvement over direct prompting, and raises the best judge's average Spearman correlation from 0.15 to 0.49. If correct, this offers a scalable automated benchmark for creativity that does not require manual annotation.

What carries the argument

The machinery is a reference-based Likert comparison over the 14 binary TTCW tests. The tests are organized into four dimensions: Fluency, Flexibility, Originality, and Elaboration, each with a guiding question about the story. The judge sees the candidate and reference together, evaluates one test at a time, and outputs one of five verdicts; two orderings are averaged; the test is passed if the mean score is at least -2; and the sum over the 14 tests becomes the creativity score. This converts subjective judgment into a finite countable score, which is what makes automated and scalable evaluation possible.

What would settle it

Run the proposed reference-based TTCW protocol on a new set of 50 storylines where human rankings come from multiple independent expert judges with measured inter-annotator agreement; if the method's pairwise accuracy against one noisy judge set drops below 0.60, the claimed 0.75 alignment is not generalizable.

Watch

Extended reading notes

Core claim

The central claim is that creativity in generated stories can be measured automatically as a product-level property using the TTCW's fourteen binary questions, provided each question is answered comparatively rather than absolutely. For each question, an LLM judge reads the candidate and a human-authored reference, states which is better on a five-level scale, and repeats the comparison with the order swapped; the question counts as passed only when the average verdict meets a cutoff of -2. The total creativity score is the number of tests passed. With this protocol plus an analyze-then-rate prompt, the paper reports that three LLM judges align better with human expert rankings than the original direct-prompt TTCW, with the best judge reaching 0.75 pairwise accuracy, 0.49 average Spearman, and 0.44 average Kendall's tau.

Load-bearing premise

The human expert rankings from Chakrabarty et al. are treated as reliable and noise-free ground truth, but the paper does not report inter-annotator agreement for those rankings, so all reported correlations inherit whatever noise or bias those rankings contain.

Editorial extensions

If this is right

  • LLM creativity can be tracked across model versions cheaply and without human annotation, as long as high-quality reference texts exist for the relevant domain.
  • Model rankings become more comparable across research groups because scores are anchored to the same published reference set rather than to each judge's private taste.
  • The procedure can be ported to other creative writing forms, such as poetry, dialogue, or advertising copy, by adapting TTCW-style test batteries and reference collections.
  • The choice of Likert granularity matters: the paper reports that the 5-point scale outperforms 3-point and 7-point scales, so scale design is a practical parameter for future evaluation suites.
  • Automated creativity scoring could support reference-based quality control in generation systems, flagging outputs that fall below the quality bar of a chosen reference style.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because every score is computed relative to a single high-quality reference per storyline, the method probably measures proximity to the reference's aesthetic as much as creativity in the abstract; using multiple or deliberately diverse reference anchors could reveal how much of the ranking is style-dependent.
  • The 0.75 pairwise accuracy is estimated from only 12 storylines, so its uncertainty is unknown; re-running the protocol on 50 or more storylines with bootstrap confidence intervals would give a more stable effect size and would test whether the 15-point gain over baseline persists.
  • The paper itself flags, in its limitation section, that reference dependence restricts open-ended use and that the method collapses when all candidates are far worse than the reference; a practical extension would be adaptive reference selection or multiple reference anchors to recover ranking signal in that regime.
  • The same reference-based comparison could be repurposed as a distillation monitor, scoring a student model's creative outputs against a teacher model's outputs; the paper mentions this direction only in passing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an automated creativity evaluation method for LLMs based on the Torrance Test of Creative Writing (TTCW). For each of 14 binary TTCW tests, an LLM judge compares a candidate story against a high-quality reference story using a Likert scale, with the order of candidate and reference alternated across two trials. A test is counted as passed if the score difference meets a cutoff, and the total number of passed tests ranks the candidate models. The authors compare these rankings with human expert rankings using Spearman's correlation, Kendall's tau, and pairwise accuracy. With an analyze-rate prompting strategy, they report a pairwise accuracy of 0.75 (+15%) for Claude 3.5, with smaller gains for GPT-4o and Qwen-2-72B-Chat, and they include ablations and a generalization experiment on a second dataset.

Significance. If the reported gains hold under proper validation, the reference-based Likert-style TTCW evaluation would be a practical, scalable alternative to manual creativity annotation, and the paper would make a useful contribution to automated evaluation of creative writing. The method is simple, grounded in established psychometric tests, and the inclusion of ablations and a second dataset are strengths. However, the central quantitative claim is currently not protected from selection effects: the binary-conversion cutoff is chosen on the same test set used to report the headline results, the Likert scale granularity is chosen on the same data, and only three of ten evaluator models are reported for the proposed method. As a result, the reported +15% pairwise-accuracy improvement may reflect test-set fitting rather than a genuine property of the method. The paper would be significant after the empirical claims are made robust through held-out validation or equivalent safeguards.

major comments (4)
  1. [Section 5.1, Table 4, Eq. (1)] The score cutoff is selected by searching Cutoff = -3, -2, -1, 0 on the same test set that is later used to report the main results. For example, Claude 3.5's Spearman correlation under the proposed method ranges from 0.20 to 0.49 across cutoffs, and the reported 0.49 is the maximum. With only 12 stories and 3-4 candidate texts ranked per story, the variance of Spearman and pairwise accuracy is large, and selecting the cutoff, Likert scale, and evaluator model on the same data makes the reported maximum likely to exceed the true expected value. Please provide a held-out validation procedure, cross-validation, or at minimum report all cutoff results with confidence intervals or bootstrap estimates so the reader can assess the selection effect.
  2. [Section 5.2] The 5-point Likert scale is selected using qwen2-72b-chat on the same TTCW test data, with 3-point and 7-point scales yielding lower Spearman correlations. This is an additional hyperparameter selected on the test set, and the reported results for all three models use that selected scale. The claim that the 5-point scale is the 'more effective choice' is therefore not validated independently. Please report the sensitivity of all reported metrics to the Likert granularity, or validate the choice on a separate development set.
  3. [Section 4.3, Table 1] The 'Ours' rows are reported for only three of the ten evaluator models listed in the baseline, and these three models appear to be the ones with the largest improvements. The abstract and conclusion claim that 'our method significantly improves the alignment between LLM evaluations and human assessments' in general, but no results are shown for the other seven models. Reporting only the favorable subset, without a multiple-comparison correction, materially weakens the generalization claim. Please report results for all ten models, or clearly frame the claim as applying to the three models tested.
  4. [Section 4.1 and Section 4.3] The human expert rankings from Chakrabarty et al. (2024) are used as ground truth without reporting inter-annotator agreement or any measure of their reliability. If those rankings are noisy or biased, all reported Spearman, Kendall, and pairwise accuracy values inherit that noise, and the 0.75 pairwise accuracy is not a meaningful measure of evaluation quality in isolation. No confidence intervals or significance tests are provided for any of the headline numbers, which is particularly concerning given the small sample of 12 stories. Please include reliability statistics for the human ground truth and uncertainty estimates for the reported alignment metrics.
minor comments (4)
  1. [Section 3.2, Eq. (1)] The notation is inconsistent: the subscripts in L^{k,+}_{i,j} are later written as L^{k,+}_{i,k}, and the prose says a test passes if the average score across two assessments is higher than the cutoff, while Eq. (1) uses the difference L^{k,+} - L^{k,-}. Please clarify whether the two trials are averaged or differenced, and align the notation and the equation.
  2. [Appendix A.1] The prompt template shown in Appendix A.1 contains the placeholder 'Stories and Question...' instead of the actual instruction block. Please provide the complete prompt used in the experiments so the method is reproducible.
  3. [Appendix A.4, Figures 2-4] The per-story results are only shown as sorted averages in the figures; it would be more informative to display the individual story-level values or add error bars so that variability across the 12 stories is visible.
  4. [References] In the reference for Summers-Stay et al. (2023), the author name is rendered as 'Clare R. V oss' with an extra space; please fix the typesetting.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline alignment is selected, not predicted: the binary cutoff and Likert granularity are tuned on the same human rankings used to report the main results.

  1. fitted input called prediction [Section 3.2 (score equation), Section 5.1, Table 4, and Table 1]
    "To determine the optimal cutoff score, we conducted a hyperparameter search, as detailed in Appendix A.5. The results indicate that setting the cutoff at -2 yields the best ranking similarity, which aligns with our expectations."

    The paper's score is Score_k_i = sum_j I[(L+ - L-) >= score_cutoff], and the cutoff is selected by maximizing ranking similarity against the same human rankings later reported in Table 1. Table 4 shows Claude 3.5-Ours Spearman values 0.20, 0.49, 0.37, 0.30 for cutoffs -3, -2, -1, 0; the reported headline result (Spearman 0.49, pairwise accuracy 0.75) is the maximum over this grid. Reporting the best value of a parameter searched on the evaluation labels as evidence that the method 'improves alignment' is a fitted input called a prediction: the measured alignment is not an independent estimate because the target labels were used to choose the parameter. No held-out validation, confidence interval, or significance test is provided.

  2. fitted input called prediction [Section 5.2 and Table 1 (qwen2-72b-chat row)]
    "we evaluated the performance of 3-point, 5-point, and 7-point Likert scales to determine the optimal level of granularity for our evaluation framework. The Spearman's correlation for the 5-point scale is 0.22, outperforming both the 3-point scale (-0.07) and the 7-point scale (0.01)."

    The 5-point Likert scale is selected because it maximizes Spearman correlation on the same qwen2-72b-chat evaluation of the human-ranked test stories, and Table 1 then reports qwen2-72b-chat(ours) Spearman = 0.22, exactly the selected maximum. The scale granularity is therefore chosen using the target human rankings and then presented as part of the method's achieved performance, so the reported improvement for this configuration partially reduces to a fitted choice rather than an out-of-sample prediction.

full rationale

The paper's core claim is an empirical alignment result, not a formal derivation, and most of the pipeline—reference-based Likert scoring, analyze-rate prompting, and the TTCW rubric—is independently motivated and ablated. However, the central quantitative claims are partially circular in the fitted-input sense: the binary cutoff (Section 5.1) and the Likert granularity (Section 5.2) are selected by maximizing agreement with the same human rankings used to report Table 1, and the headline pairwise accuracy of 0.75 is the best value from that search. The additional dataset in Appendix A.7 provides some independent grounding, and there is no self-citation chain or definitional equivalence, which prevents a higher score; but the absence of any held-out tuning or uncertainty quantification means the reported +15% gain is substantially a selection artifact. Score 6 reflects partial circularity: one or more reported predictions reduce to fitted hyperparameters chosen on the evaluation target.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method introduces no new theoretical entities. The free parameters are the cutoff and the Likert granularity, both tuned on the evaluation set. The axioms are standard assumptions about the ground truth, the TTCW battery, reference quality, prompting strategy, and position-bias handling.

free parameters (2)
  • score_cutoff = -2
    The paper performs a hyperparameter search over cutoff values on the same test set used to report results, and selects -2 because it yields the best ranking similarity (Section 5.1, Appendix A.5).
  • Likert scale granularity = 5-point
    The 5-point scale is chosen after comparing 3, 5, and 7 point scales using qwen2-72b-chat on the same evaluation data, where 5 points gave the highest Spearman correlation (Section 5.2).
assumptions (5)
  • domain assumption Human expert rankings from Chakrabarty et al. (2024) are valid ground truth for creativity ordering.
    The paper measures alignment against these rankings without reporting inter-annotator agreement or otherwise validating the ground truth (Sections 4.1 and 4.3).
  • domain assumption TTCW's 14 binary tests across Fluency, Flexibility, Originality, and Elaboration capture the creativity of a written product.
    The test battery is adopted from prior work and is not independently validated in this paper (Section 3.2 and Appendix A.3).
  • domain assumption A single human-authored New Yorker reference story per plot is a suitable and stable anchor for relative creativity judgments.
    The method assumes reference quality is high and comparable across all stories; the limitation section notes this dependence but does not test anchor stability (Sections 3.1 and 7).
  • domain assumption The analyze-rate prompting strategy improves LLM evaluator reliability without introducing systematic bias.
    The paper relies on prior observations from Chiang and Lee (2023) and its own experiments, but provides no independent analysis of how the strategy affects score distribution (Section 3.3).
  • domain assumption Averaging two position-flipped assessments sufficiently removes positional bias.
    The method alternates order and averages the two scores, but no residual position-bias analysis is reported (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach." pith.science (2026). https://pith.science/paper/D24G6AOJ

@misc{pith2026250415784,
  author       = {Pith},
  title        = {Pith review of: Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D24G6AOJ}},
  note         = {Machine review of arXiv:2504.15784}
}
read the original abstract

Creative writing is a key capability of Large Language Models (LLMs), with potential applications in literature, storytelling, and various creative domains. However, evaluating the creativity of machine-generated texts remains a significant challenge, as existing methods either rely on costly manual annotations or fail to align closely with human assessments. In this paper, we propose an effective automated evaluation method based on the Torrance Test of Creative Writing (TTCW), which evaluates creativity as product. Our method employs a reference-based Likert-style approach, scoring generated creative texts relative to high-quality reference texts across various tests. Experimental results demonstrate that our method significantly improves the alignment between LLM evaluations and human assessments, achieving a pairwise accuracy of 0.75 (+15\%).

Figures

Figures reproduced from arXiv: 2504.15784 by the authors.

Figure 1
Figure 1. Distribution of TTCW test scores across different models. The dashed lines indicate the average number [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Complete Spearman correlation results across individual stories and models. Models labeled ’ours’ [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Complete Kendall’s tau results across individual stories and models. Models labeled ’ours’ indicate [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Complete Pairwise accuracy results across individual stories and models. Models labeled ’ours’ indicate [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Spearman correlation performance under different cutoff scores. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Complete Spearman correlation results across individual stories and models. Models labeled ’ours’ [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Complete Kendall’s tau results across individual stories and models. Models labeled ’ours’ indicate [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Complete Pairwise accuracy results across individual stories and models. Models labeled ’ours’ indicate [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LLM judges of scientific ideas are measurably swayed by writing style; a style-detecting module reduces but does not remove the bias.

  2. ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A unified evaluation framework for proactive dialogue agents, built with 328 synthetic environments across six domains, shows that thinking modes improve target planning but not dialogue guidance in a 22-model comparison.

  3. LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Small reward models trained on LitBench reach 78% agreement with upvote-derived human preferences in creative writing, beating all zero-shot LLM judges tested.

Reference graph

Works this paper leans on

22 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    John Baer. 1993. https://api.semanticscholar.org/CorpusID:145663312 Creativity and divergent thinking: A task-specific approach

    work page 1993

  4. [4]

    Beaty and Dan R

    Roger E. Beaty and Dan R. Johnson. 2021. Automating creativity assessment with semdis: An open platform for computing semantic distance. Behavior Research Methods, 53(2):757--780

    work page 2021

  5. [5]

    Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1--34

    work page 2024

  6. [6]

    Cheng-Han Chiang and Hung yi Lee. 2023. https://arxiv.org/abs/2310.05657 A closer look into automatic evaluation using large language models . Preprint, arXiv:2310.05657

    arXiv 2023

  7. [7]

    B Cramond. 2020. Choosing a creativity assessment that is fit for purpose. Assessing Creativity: A palette of possibilities, pages 58--63

    work page 2020

  8. [8]

    Guilford

    J.P. Guilford. 1967. Creativity: Yesterday, today and tomorrow. The Journal of Creative Behavior, 1(1):3--14

    work page 1967

Show all 22 references
  1. [9]

    Guzik, Christian Byrge, and Christian Gilde

    Erik E. Guzik, Christian Byrge, and Christian Gilde. 2023. https://doi.org/10.1016/j.yjoc.2023.100065 The originality of machines: Ai takes the torrance test . Journal of Creativity, 33(3):100065

    2023

  2. [10]

    Carlos Gómez-Rodríguez and Paul Williams. 2023. https://arxiv.org/abs/2310.08433 A confederacy of models: a comprehensive evaluation of llms on creative writing . Preprint, arXiv:2310.08433

    2023 arXiv

  3. [11]

    Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. 2024. https://arxiv.org/abs/2411.02316 Evaluating creative short story generation in humans and large language models . Preprint, arXiv:2411.02316

    2024 arXiv

  4. [12]

    Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81--93

    1938

  5. [13]

    Edenbaum, Joshua D

    William Orwig, Emma R. Edenbaum, Joshua D. Greene, and Daniel L. Schacter. 2024. https://doi.org/10.1002/jocb.636 The language of creativity: Evidence from humans and large language models . The Journal of Creative Behavior, 58(1):128--136

    2024 doi

  6. [14]

    A. Roy. 2020. https://books.google.de/books?id=sWQOzgEACAAJ A Comprehensive Guide for Design, Collection, Analysis and Presentation of Likert and Other Rating Scale Data: Analysis of Likert Scale Data . Amazon Digital Services LLC - KDP Print US

    2020

  7. [15]

    Charles Spearman. 1904. ``General Intelligence,'' Objectively Determined and Measured . The American Journal of Psychology, 15(2):201--292

    1904

  8. [16]

    Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. 2022. https://arxiv.org/abs/2206.08932 Putting gpt-3's creativity to the (alternative uses) test . Preprint, arXiv:2206.08932

    2022 arXiv

  9. [17]

    Lukin, and Clare R

    Douglas Summers-Stay, Stephanie M. Lukin, and Clare R. Voss. 2023. https://api.semanticscholar.org/CorpusID:259305709 Brainstorm, then select: a generative language model improves its creativity score

    2023

  10. [18]

    E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and psychological measurement

    1966

  11. [19]

    Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. https://doi.org/10.18653/v1/2023.inlg-main.23 The next chapter: A study of large language models in storytelling . In Proceedings of the 16th International Natural Language Generation Conference, pages 323--351, Prague, Czechia....

    2023 doi

  12. [20]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://arxiv.org/abs/2106.11520 Bartscore: Evaluating generated text as text generation . Preprint, arXiv:2106.11520

    2021 arXiv

  13. [21]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  14. [22]

    Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, Qi Guo, Ling Li, and Yunji Chen. 2024. https://arxiv.org/abs/2401.12491 Assessing and understanding creativity in large language models . Preprint, arXiv:2401.12491

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.