Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Do LLMs Agree on the Creativity Evaluation of Alternative Uses?

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that four large language models produce mutually consistent and oracle-matched creativity judgments for alternative uses of everyday objects, and that no model favors its own outputs.

desk verdict The cross-model agreement result is real; the oracle-based accuracy claim is circular and the paper overstates it. read the letter →

arxiv 2411.15560 v2 pith:GWYACGRD submitted 2024-11-23 cs.AI cs.CL

Abdullah Al Rabeyah , Fabrício Góes , Marco Volpe , Talles Medeiros This is my paper · ORCID
classification cs.AIcs.CL
keywords AlternativeUsesTestLLM-as-a-judgecreativityevaluationSpearmancorrelationoraclebenchmarkinter-modelagreementself-preferencescoringvsranking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether different large language models agree with one another when they judge the creativity of alternative uses in the Alternative Uses Test, and whether they judge fairly when the responses come from other models. It builds a dataset of 300 alternative uses for five objects, generated by four LLMs under three prompt conditions labeled common, creative, and highly creative, and defines an oracle ordering from those labels. Each model then scores and ranks all responses, alone and in smaller batches. The paper reports average inter-model rank correlations above 0.7, correlations with the oracle above 0.77, and no systematic self-preference, concluding that LLMs can be reliable and impartial automated creativity assessors.

What carries the argument

The evaluation oracle is the load-bearing object: a fixed ordering that places all highly_creative responses first, then all creative responses, then all common responses, with each of the four models' responses kept together inside each tier. The paper measures every LLM's output against this oracle and against every other model using Spearman's rank correlation, so 'accuracy' reduces to whether the model reconstructs the oracle tier order and 'agreement' reduces to pairwise rank similarity. The forceful-prompt generation procedure from [2] is the machinery that produces the three creativity tiers in the first place.

What would settle it

Have human raters rank or score the same 300 alternative uses and compare their average ordering with the oracle. If many highly_creative responses fall below creative or common responses in human judgment, the claim that LLMs accurately evaluate creativity fails, even if inter-model agreement remains high.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that four different LLMs converge on a shared ordering of alternative uses by creativity and that this ordering matches a benchmark oracle built from the prompt conditions used to generate the responses. In the comprehensive scoring condition, every model's Spearman correlation with the oracle was above 0.95 for the average across objects, and two objects produced a perfect correlation of 1.0; across all four experimental conditions, correlations with the oracle averaged above 0.77 and inter-model correlations averaged above 0.7. The paper also reports that no model assigned systematically higher scores or ranks to its own generated responses over those of other models, and it interprets this as impartiality. It further finds that scoring outperforms ranking in oracle alignment and that the comprehensive setting generally matches or exceeds the segmented setting.

Load-bearing premise

The load-bearing premise is that the oracle's tier order—highly_creative above creative above common, based on the generation prompt's forcefulness—really reflects increasing creativity; if it does not, the reported accuracy correlations only show agreement with a label ordering, not with creative quality.

Editorial extensions

If this is right

  • A single LLM evaluator can be used in place of a panel of different models for AUT scoring tasks, since the models' judgments are near-interchangeable.
  • Because no self-preference was observed, LLM-generated alternative uses can be judged by another LLM without applying a bias correction for authorship.
  • Scoring should be preferred over full ranking when designing automated creativity evaluation prompts, because scoring produced higher oracle correlations in this study.
  • Batch size had limited impact: scoring 60 responses at once performed at least as well as scoring groups of 12, which supports using longer evaluation prompts when needed.
  • The benchmark construction can be reused to compare future LLM judges on the same 300-response dataset or on new objects generated under the same prompt tiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence I draw is that the inter-model agreement may reflect shared training on similar human judgments, so it establishes consensus among LLMs but does not by itself establish that the consensus matches human creativity judgments.
  • A test the paper leaves open is human validation: scoring the same 300 alternative uses by human raters would show whether the oracle's tier order matches human judgment or only the forcefulness of the generation prompts.
  • The framework could be extended to other divergent-thinking tasks, such as poetry or joke evaluation; in that extension, the same oracle construction would need independent per-task creativity levels rather than inherited AUT prompts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether four LLMs (Claude 3.5 Sonnet, ChatGPT-4, ChatGPT-4o, Gemini 1.5 Flash) agree in evaluating the creativity of Alternative Uses Test (AUT) responses. AUs were generated by prompting each model to produce common, creative, and highly creative uses for five objects, using a forceful-prompt technique from a prior paper [2]. Each model then evaluated all AUs under four conditions: scoring vs. ranking and comprehensive (60 AUs) vs. segmented (12 AUs). Spearman rank correlations were computed between each model's evaluations and an oracle defined by the generation categories, and between models. The paper reports average oracle correlations above 0.77 and inter-model correlations above 0.7, and concludes that LLMs are impartial and reliable for automated creativity assessment.

Significance. If the inter-model agreement result is robust, the paper provides a useful descriptive finding about consensus among current LLMs in ranking AUT responses. The benchmark construction methodology (category-based oracle from forceful prompts) is also of interest, although its validity as ground truth is not established. The main contribution is a descriptive consensus study, not a validation of LLM creativity assessment, because no human ground truth is involved. The work is potentially relevant to practitioners considering LLM-based evaluation, but the accuracy/reliability claims overreach the evidence.

major comments (3)
  1. [§3.2.4, Eq. (1)] The oracle is defined entirely by the generation-prompt categories, and its ordering is justified only by reference to [2], which shares authors with this manuscript. There is no independent human validation that the 'highly_creative', 'creative', and 'common' categories actually differ in creativity. Consequently, the reported oracle correlations (0.77–0.97) largely measure whether the evaluator reproduces the assumed category ordering, not accuracy in assessing creativity. Because the oracle assigns tied positions to all four highly_creative entries, all four creative entries, and all four common entries, a perfect Spearman correlation only requires the three category-level means to be ordered; within-category ordering is never tested. This is load-bearing for the abstract's claim that the results 'validate the reliability of LLMs in creativity assessment'. The paper should either supply human ratings of the AUs to validate the oracle or explicitly restrict its claims to inter-model agreement.
  2. [§3.2.3, Tables 2–4, Figure 7] All Spearman correlations are computed on n=12 aggregated values per object (the average score or rank of five AUs per model-category cell), and no confidence intervals, standard errors, or significance tests are reported. The abstract's claim of correlations 'averaging above 0.7' is a descriptive statistic with no uncertainty quantification. Given the small effective sample size, the aggregation can inflate correlation estimates, and the high values may be driven by one or two objects. The authors should report bootstrap confidence intervals, per-object correlation distributions, or a multilevel analysis to support the claimed magnitude and stability of the correlations.
  3. [§4.5, Table 5] The statement that 'LLMs do not favor their own responses' is not supported by a statistical test. The evidence consists of informal comparisons of average scores in Table 5 and in the appendix tables. A model may rate its own AUs higher because its AUs are genuinely more creative, and a proper impartiality test would compare each model's self-evaluation with the average evaluation of the same AUs by the other models, with an error term accounting for per-AU variability. Without such a test, the impartiality claim in the abstract and Section 4.5 is not established.
minor comments (6)
  1. [Title] The manuscript header reads 'DO LLM S AGREE' instead of 'DO LLMs AGREE'; this should be corrected.
  2. [§3.2.3] The computation of the Spearman correlation on the 12 averaged values per object should be stated explicitly before the results are presented; currently the reader must infer this from Figure 6 and the described aggregation.
  3. [§4.1, Table 2] The text refers to 'top left in the table' for the heatmap; ensure that all heatmap entries include numeric values and that a color scale legend is provided for accessibility.
  4. [§4.1] The phrase 'the highest score observed in our evaluation experiments' is vague; specify the exact correlation value and the condition to which it refers.
  5. [§4.2] The sentence 'the standard deviation ranged between 0.10 and 1.03' does not specify whether this is across models, across AUs, or across categories; clarify the population over which the standard deviation is computed.
  6. [§2.2] The statement that LLM evaluation performances are 'far superior' to semantic-distance-based evaluations is strong; consider citing the relevant empirical comparisons or softening the wording.

Circularity Check

1 steps flagged · score 7.0 of 10

Oracle-based 'accuracy' is circular: the ground truth in §3.2.4 is the prompt-category ordering, justified only by self-cited prior LLM work, so the reliability claim reduces to agreement with that assumed order.

  1. self citation load bearing [Section 3.2.4 (AUs Evaluation Oracle), Eq. (1); see also Section 2.2]
    "By relying on the results of [2], we expect the creativity levels (common, creative and highly_creative) used for prompting to be able to actually produce outputs with increasing levels of creativity. Therefore, the oracle was constructed based on the AUs generated by the different creativity levels as follows: AUs Evaluation Oracle = [highly_creative1, highly_creative2, highly_creative3, highly_creative4, creative1, creative2, creative3, creative4, common1, common2, common3, common4] (1)"

    The oracle—the sole ground truth for the paper's 'accuracy' correlations—is exactly the ordering induced by the generation-prompt labels. The paper's only justification for treating that ordering as true creativity is a citation to [2], a paper with overlapping authors (Góes and Volpe) whose 'results' are themselves LLM evaluations from forceful prompts, not human creativity judgments. The reported oracle correlations (0.77–0.97) therefore measure whether each evaluator reproduces the assumed highly_creative > creative > common order, not whether evaluations match independently established creativity. The central reliability claim in the abstract—'validating the reliability of LLMs in creativity assessment'—thus reduces to agreement with a self-cited assumption.

full rationale

The paper has a genuinely independent core: inter-model agreement (Figure 7, Table 2) and the absence of self-favoritism are descriptive, self-contained findings that do not depend on the oracle. However, the paper's stronger claim—that LLMs are 'accurate' and 'reliable' at creativity assessment—is measured exclusively as Spearman correlation against an oracle whose ordering is defined by the generation-prompt categories in §3.2.4. That ordering is asserted to reflect actual creativity levels 'by relying on the results of [2]', a prior work sharing two co-authors with this paper and whose own evidence was LLM-based, not human-validated. No independent human baseline is reported anywhere in the paper, and no uncertainty estimates accompany the oracle correlations. Because the accuracy measure is defined as agreement with an oracle that itself encodes the expected category order, the reliability claim largely reduces to whether each model reproduces the prompt-induced ordering. I therefore score the oracle-based 'accuracy/reliability' portion as substantially circular (7/10), while noting that the inter-model agreement result would survive even if the oracle were removed.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central accuracy claim rests on the assumption that prompt-based creativity categories reflect true creativity; this assumption is inherited from self-cited prior work and is not independently validated. The aggregation of 60 evaluations into 12 means is a methodological choice that strengthens correlations but is not justified in the paper.

assumptions (2)
  • domain assumption The three prompt categories (common, creative, highly_creative) correspond to increasing true creativity levels.
    Section 3.2.4 defines the oracle by this ordering, citing [2] by two of the present authors; no human ratings are collected in this paper.
  • ad hoc to paper Spearman correlation computed on 12 category-averaged values (5 AUs averaged per model-category cell) is an adequate measure of evaluation agreement.
    Section 3.2.3 aggregates 60 individual evaluations into 12 means before computing correlations, which reduces noise and tends to inflate apparent agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do LLMs Agree on the Creativity Evaluation of Alternative Uses?." pith.science (2026). https://pith.science/paper/GWYACGRD

@misc{pith2026241115560,
  author       = {Pith},
  title        = {Pith review of: Do LLMs Agree on the Creativity Evaluation of Alternative Uses?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWYACGRD}},
  note         = {Machine review of arXiv:2411.15560}
}
read the original abstract

This paper investigates whether large language models (LLMs) show agreement in assessing creativity in responses to the Alternative Uses Test (AUT). While LLMs are increasingly used to evaluate creative content, previous studies have primarily focused on a single model assessing responses generated by the same model or humans. This paper explores whether LLMs can impartially and accurately evaluate creativity in outputs generated by both themselves and other models. Using an oracle benchmark set of AUT responses, categorized by creativity level (common, creative, and highly creative), we experiment with four state-of-the-art LLMs evaluating these outputs. We test both scoring and ranking methods and employ two evaluation settings (comprehensive and segmented) to examine if LLMs agree on the creativity evaluation of alternative uses. Results reveal high inter-model agreement, with Spearman correlations averaging above 0.7 across models and reaching over 0.77 with respect to the oracle, indicating a high level of agreement and validating the reliability of LLMs in creativity assessment of alternative uses. Notably, models do not favour their own responses, instead they provide similar creativity assessment scores or rankings for alternative uses generated by other models. These findings suggest that LLMs exhibit impartiality and high alignment in creativity evaluation, offering promising implications for their use in automated creativity assessment.

Figures

Figures reproduced from arXiv: 2411.15560 by the authors.

Figure 1
Figure 1. Prompt to generate common uses of an object (from [2]). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Prompt to generate highly_creative alternative uses of an object using forceful prompts (from [2]). [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Prompt example for evaluating alternative uses of an object by score. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: The bar chart where every color represents the creativity category, which are used to visualize the LLMs [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Average SRC heatmap between the LLMs over all the four experiments. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LLM judges of scientific ideas are measurably swayed by writing style; a style-detecting module reduces but does not remove the bias.

Reference graph

Works this paper leans on

34 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [2]

    Pushing gpt’s creativity to its limits: Alternative uses and torrance tests,

    F. Góes, P. Sawicki, M. Grzes, M. V olpe, and J. Watson, “Pushing gpt’s creativity to its limits: Alternative uses and torrance tests,” in Proceedings of the 14th International Conference on Computational Creativity, Ontario, Canada, June 19-23, 2023 , A. Pease, J. M. Cunha, M. Ackerman, and D. G. Brown, Eds. Association for Computational Creativity (ACC)...

    work page 2023

  2. [1]

    Automatic assessment of divergent thinking in chinese language with transdis: A transformer-based language model approach,

    T. Yang, Q. Zhang, Z. Sun, and Y . Hou, “Automatic assessment of divergent thinking in chinese language with transdis: A transformer-based language model approach,” Behavior Research Methods, vol. 56, no. 6, p. 5798–5819, Dec. 2023. [Online]. Available: http://dx.doi.org/10.3758/s13428-023-02313-z

    work page doi:10.3758/s13428-023-02313-z 2023

  3. [3]

    Using large language models to evaluate alternative uses task flexibility score,

    E. Hadas and A. Hershkovitz, “Using large language models to evaluate alternative uses task flexibility score,” Thinking Skills and Creativity , vol. 52, p. 101549, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1871187124000877

    work page 2024

  4. [4]

    A standardised procedure for evaluating creative systems: Computational creativity evaluation based on what it is to be creative,

    A. Jordanous, “A standardised procedure for evaluating creative systems: Computational creativity evaluation based on what it is to be creative,” Cognitive Computation, vol. 4, no. 3, pp. 246–279, 2012. [Online]. Available: https://kar.kent.ac.uk/42379/

    work page 2012

  5. [5]

    Regent-dependent creativity: A domain independent metric for the assessment of creative artifacts,

    C. França, L. F. W. Góes, A. Amorim, R. Rocha, and A. R. Da Silva, “Regent-dependent creativity: A domain independent metric for the assessment of creative artifacts,” inProceedings of the Seventh International Conference on Computational Creativity. Citeseer, 2016, pp. 68–75

    work page 2016

  6. [6]

    Evaluating the evaluator: Measuring llms’ adherence to task evaluation instructions,

    B. Murugadoss, C. Poelitz, I. Drosos, V . Le, N. McKenna, C. S. Negreanu, C. Parnin, and A. Sarkar, “Evaluating the evaluator: Measuring llms’ adherence to task evaluation instructions,” 2024. [Online]. Available: https://arxiv.org/abs/2408.08781

    arXiv 2024

  7. [7]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” 2023. [Online]. Available: https://arxiv.org/abs/2306.05685

    arXiv 2023

  8. [8]

    Automatic scoring of metaphor creativity with large language models,

    P. V . DiStefano, J. D. Patterson, and R. E. Beaty, “Automatic scoring of metaphor creativity with large language models,” Creativity Research Journal , pp. 1–15, 2024. [Online]. Available: https://doi.org/10.1080/10400419.2024.2326343

    arXiv 2024

Show all 34 references
  1. [9]

    Is GPT-4 good enough to evaluate jokes?

    F. Góes, P. Sawicki, M. Grzes, M. V olpe, and D. Brown, “Is GPT-4 good enough to evaluate jokes?” inProceedings of the 14th International Conference on Computational Creativity, Ontario, Canada, June 19-23, 2023 , A. Pease, J. M. Cunha, M. Ackerman, and D. G. Brown, Eds. Assoc...

    2023

  2. [10]

    On the power of special-purpose GPT models to create and evaluate new poetry in old styles,

    P. Sawicki, M. Grzes, F. Góes, A. Jordanous, D. Brown, S. Paraskevopoulou, M. Peeperkorn, and A. Khatun, “On the power of special-purpose GPT models to create and evaluate new poetry in old styles,” inProceedings of the 14th International Conference on Computational Creativity...

    2023

  3. [11]

    Large language models are not fair evaluators,

    P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y . Cao, Q. Liu, T. Liu, and Z. Sui, “Large language models are not fair evaluators,” 2023. [Online]. Available: https://arxiv.org/abs/2305.17926

    2023 arXiv

  4. [12]

    Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges,

    A. S. Thakur, K. Choudhary, V . S. Ramayapally, S. Vaidyanathan, and D. Hupkes, “Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12624

    2024 arXiv

  5. [13]

    Can large language models be an alternative to human evaluations?

    C.-H. Chiang and H. yi Lee, “Can large language models be an alternative to human evaluations?” 2023. [Online]. Available: https://arxiv.org/abs/2305.01937 11 Do LLMs Agree on the Creativity Evaluation of Alternative Uses?

    2023 arXiv

  6. [14]

    Creative beam search: Llm-as-a-judge for improving response generation,

    G. Franceschelli and M. Musolesi, “Creative beam search: Llm-as-a-judge for improving response generation,”

  7. [15]

    Gptscore: Evaluate as you desire,

    J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” 2023. [Online]. Available: https://arxiv.org/abs/2302.04166

    2023 arXiv

  8. [16]

    A closer look into automatic evaluation using large language models,

    C.-H. Chiang and H. yi Lee, “A closer look into automatic evaluation using large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.05657

    2023 arXiv

  9. [17]

    Evalullm: Llm assisted evaluation of generative outputs,

    M. Desmond, Z. Ashktorab, Q. Pan, C. Dugan, and J. M. Johnson, “Evalullm: Llm assisted evaluation of generative outputs,” in Companion Proceedings of the 29th International Conference on Intelligent User Interfaces, ser. IUI ’24 Companion. New York, NY , USA: Association for C...

    2024

  10. [18]

    A confederacy of models: a comprehensive evaluation of LLMs on creative writing,

    C. Gómez-Rodríguez and P. Williams, “A confederacy of models: a comprehensive evaluation of LLMs on creative writing,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Lin...

    2023

  11. [19]

    Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation,

    Q. Li, L. Cui, L. Kong, and W. Bi, “Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation,” 2023. [Online]. Available: https://arxiv.org/abs/2310.19740

    2023 arXiv

  12. [20]

    Automated scoring of scientific creativity in german,

    B. Goecke, P. V . DiStefano, W. Aschauer, K. Haim, R. Beaty, and B. Forthmann, “Automated scoring of scientific creativity in german,” The Journal of Creative Behavior , 2024. [Online]. Available: https://doi.org/10.1002/jocb.658

    2024 doi

  13. [21]

    Automated scoring of open-ended question complexity: A large language model approach,

    T. Raz, S. Luchini, R. Beaty, and Y . Kenett, “Automated scoring of open-ended question complexity: A large language model approach,” 2024. [Online]. Available: https://doi.org/10.21203/rs.3.rs-3890828/v1

    2024 doi

  14. [22]

    Assessing and understanding creativity in large language models,

    Y . Zhao, R. Zhang, W. Li, D. Huang, J. Guo, S. Peng, Y . Hao, Y . Wen, X. Hu, Z. Du, Q. Guo, L. Li, and Y . Chen, “Assessing and understanding creativity in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2401.12491

    2024 arXiv

  15. [23]

    Creativity and machine learning: A survey,

    G. Franceschelli and M. Musolesi, “Creativity and machine learning: A survey,” ACM Comput. Surv., vol. 56, no. 11, Jun. 2024. [Online]. Available: https://doi.org/10.1145/3664595

    2024 doi

  16. [24]

    Creativity: Yesterday, today and tomorrow,

    J. P. Guilford, “Creativity: Yesterday, today and tomorrow,”The Journal of Creative Behavior, vol. 1, no. 1, pp. 3–14, 1967. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2162-6057.1967.tb00002.x

    1967

  17. [25]

    Automating creativity assessment with semdis: An open platform for computing semantic distance,

    R. E. Beaty and D. R. Johnson, “Automating creativity assessment with semdis: An open platform for computing semantic distance,” Behavior Research Methods , vol. 53, pp. 757–780, 2021. [Online]. Available: https://doi.org/10.3758/s13428-020-01453-w

    2021 doi

  18. [26]

    Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models,

    P. Organisciak, S. Acar, D. Dumas, and K. Berthiaume, “Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models,” Thinking Skills and Creativity , vol. 49, p. 101356, 2023. [Online]. Available: https://www.sciencedirect.com/...

    2023

  19. [27]

    Putting gpt-3’s creativity to the (alternative uses) test,

    C. Stevenson, I. Smal, M. Baas, R. Grasman, and H. van der Maas, “Putting gpt-3’s creativity to the (alternative uses) test,” in Proceedings of the International Conference on Computational Creativity 2022 . Association for Computational Creativity (ACC), 2022, pp. 164–168. [O...

    2022 arXiv

  20. [28]

    Torrance tests of creative thinking,

    E. P. Torrance, “Torrance tests of creative thinking,”Educational and Psychological Measurement, 1966. [Online]. Available: https://doi.org/10.1037/t05532-000

    1966 doi

  21. [29]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,”ACM Trans. Intell. Syst. Technol., vol. 15, no. 3, Mar. 2024. [Online]. Availa...

    2024 doi

  22. [30]

    Less is more for long document summary evaluation by LLMs,

    Y . Wu, H. Iso, P. Pezeshkpour, N. Bhutani, and E. Hruschka, “Less is more for long document summary evaluation by LLMs,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (V olume 2: Short Papers) , Y . Graham and M...

    2024

  23. [31]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024. [Online]. Available: https://aclantholo...

    2024

  24. [32]

    Leveraging llm-respondents for item evaluation: a psychometric analysis,

    Y . Liu, S. Bhandari, and Z. A. Pardos, “Leveraging llm-respondents for item evaluation: a psychometric analysis,”

  25. [34]

    [Online]. Available: https://arxiv.org/abs/2407.10899 12 Do LLMs Agree on the Creativity Evaluation of Alternative Uses? 6 Appendix Experiments AUs evaluation order Spearman’s Rank Correlation Heatmap Comprehensive evaluation by scores (60 Alternative Uses) Comprehensive evalu...

    arXiv

  26. [2024]

    Available: https://arxiv.org/abs/2405.00099

    [Online]. Available: https://arxiv.org/abs/2405.00099

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.