REVIEW 4 major objections 3 minor 30 references
Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models
T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The in-loop evaluator, not search depth or temperature, is what determines creativity scores in iterative LLM recipe search.
desk verdict A transparent pilot with a plausible but unproven central claim: the in-loop evaluator seems to matter more than iteration or temperature, but the evidence rests on unvalidated LLM evaluation and a scorer/judge overlap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the FunSearch loop adapted to a subjective domain. Semi-isolated islands preserve different recipe lineages; best-shot prompting rebuilds prompts from high-scoring candidates; and an in-loop selection scorer applies a weighted Pillsbury-style rubric (recipe content 70%, story 30%) to decide which candidates survive into the next generation. The final creativity analysis is kept separate from the search objective, using four post hoc LLM evaluators with TTCT-derived definitions of fluency, flexibility, originality, and elaboration. The controlled contrast that carries the argument is the in-loop scorer's model size, since the generator stays fixed: the same search loop is run with a smaller and a larger evaluator, and the smaller one wins on most dimensions.
What would settle it
Have a panel of human cooks rate, blind, the same retained recipes from the small-scorer and large-scorer conditions; the central claim fails if average human creativity ratings do not favor the small-scorer condition, or if the official Bake-Off winner is not rated at or near the top of the human reference set.
Extended reading notes
Core claim
The central claim is that when creativity is judged subjectively, the in-loop evaluator shapes how much iterative search improves outputs more than search depth or sampling randomness does. The paper supports this by adapting FunSearch to recipe generation: multi-island search keeps separate lineages, high-scoring candidates seed later prompts, and an in-loop selection scorer applies a weighted Pillsbury-style rubric (70% recipe, 30% story) to admit or discard candidates. Final creativity is assessed separately by four LLM evaluators using TTCT-derived product-level dimensions: fluency, flexibility, originality, and elaboration. In the main comparison, the smaller 8B selection scorer produced significantly higher final scores on creativity, fluency, flexibility, and elaboration than the larger 17B scorer, whereas moving from 5 to 30 iterations produced no clear monotonic gain and lowering generator temperature only reduced originality. The paper interprets this as evidence that the evaluative ecology, not the amount of generation, determines whether iterative novelty accumulates in subjective creative domains.
Load-bearing premise
The load-bearing premise is that the automated judges used to score creativity are a valid stand-in for human taste in recipes; if they mostly reward rubric compliance or their own preferences, the reported advantage of the smaller in-loop scorer would not carry over to human judgment.
Editorial extensions
If this is right
- Adding more FunSearch iterations to a recipe-generation loop does not by itself raise final creativity scores; the near-one-shot baseline and the 5, 15, and 30 iteration conditions land at similar levels.
- Switching the in-loop selection scorer from a larger to a smaller model reliably raises final creativity, fluency, flexibility, and elaboration scores under TTCT-style LLM evaluation.
- Lower generator temperature mainly costs originality; it does not improve any other TTCT dimension at a significant level.
- Iterative search with a well-chosen evaluator can produce recipe artifacts whose LLM-judged creativity is comparable to a human reference set from the same competition.
- Creative LLM systems should treat the diversity, architecture, and incentive structure of in-loop evaluators as a central design variable rather than relying on more generation cycles.
Reading between the lines
- A natural next test is whether the evaluator-size effect survives human judging: because the paper's own benchmark shows only partial agreement with the official competition outcome, a human panel could plausibly reverse the smaller-scorer advantage.
- The paper's framing suggests varying the in-loop evaluator during search, for example rotating several scorers or changing rubrics between islands, as a way to escape convergence on conservative recipes, but this extension is not tested here.
- Because the larger in-loop scorer also served as one of the final evaluators, a cleaner replication with fully disjoint generator, scorer, and final judge models is needed to rule out self-preference as the source of the size effect.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts the FunSearch evolutionary generation loop to a subjective creative domain—recipe generation for the 2024 Pillsbury Bake-Off—and evaluates the resulting artifacts with four TTCT-style LLM evaluators. Two experiments manipulate iteration count, generator temperature, and in-loop selection-scorer model size. The authors report that iterative generation can reach creativity scores comparable to a human reference set, that additional iterations do not improve scores, that the smaller Meta-8B in-loop scorer yields significantly higher scores than the Meta-17B scorer on most TTCT dimensions, and that lower temperature mainly reduces originality. They conclude that evaluator design is a first-order design variable in subjective creative search.
Significance. If the main finding holds, it would be a valuable reframing for computational creativity: the design of the in-loop evaluator, rather than additional search depth or sampling stochasticity, may determine the creativity of iteratively generated artifacts. The paper has real strengths: it transfers a well-known program-search method to a non-objective domain, provides detailed appendices with the recipe skeleton, in-loop rubric, evaluation prompt structure, and full regression output, and explicitly acknowledges its main limitation—that Meta-17B serves as both the in-loop scorer and one of the final evaluators. The use of four LLM evaluators and a partial human benchmark is also a reasonable pilot-level design. However, the statistical support is weak and the evaluator-overlap issue is load-bearing, so the contribution is best assessed as a promising pilot rather than a fully established result.
major comments (4)
- [Study Design and Evaluation; Limitations and Future Work; Appendix D, Table 1] The central negative LargeEvaluator coefficient in Appendix D, Table 1 is confounded by the fact that Meta-17B is both the in-loop selection scorer and one of the four final TTCT evaluators. The paper acknowledges this overlap in the Limitations section, but it does not address it quantitatively. Because the outcome is the average of four LLM evaluators, one of which is identical to the manipulated scorer, the observed 8B-over-17B difference could partly reflect self-preference or rubric alignment rather than perceived creativity. I request a re-analysis that excludes the Meta-17B evaluator from the outcome variable, or a small human-rating study on a subset of the generated recipes, before the evaluator-design claim is made.
- [Appendix D, Table 1; Results and Interpretation] The regression evidence is statistically fragile. Adjusted R2 values range from 0.006 to 0.037, meaning the manipulated factors explain almost none of the variance in the outcomes, and the five outcome models together involve fifteen significance tests with no multiple-comparison correction. The originality model, which is the basis for the claim that temperature mainly reduces originality, has adjusted R2 = 0.006. The paper should report standardized effect sizes, confidence intervals, and either corrected p-values or a pre-specified analysis plan before claiming that evaluator size 'matters most.' It should also include iteration count and scorer size in a single comparative model if the conclusion is to be that evaluator design matters more than search depth.
- [Results and Interpretation, Experiment 1] Experiment 1 reports mean final creativity scores of 3.921, 3.835, and 3.927 for 5, 15, and 30 iterations, respectively, but does not provide error bars, confidence intervals, or inferential tests for these means. The claim that additional iterations do not improve creativity, and that iterative search reaches scores comparable to the human benchmark, is therefore not supported by the statistics reported. The one-iteration baseline comparison (in-loop score dropping from 4.71 to 4.00, final creativity roughly 4.1) is also given only as point estimates in the text. I recommend reporting per-condition uncertainty and formal comparisons, and ideally placing Experiment 1 and Experiment 2 in a single model that allows a direct test of the relative importance of iteration count and evaluator size.
- [Study Design and Evaluation; Limitations and Future Work] The benchmark check validates only the in-loop rubric against the human competition, and even there the agreement is partial: the human winning recipe is placed 'near the top' rather than at rank one, and the stories are absent from the comparison. The post hoc TTCT-style evaluators are not validated against human creativity judgments of the generated recipes. Since the paper's central comparative claim ('evaluator design matters more than search depth') is expressed in terms of creativity scores from these LLM evaluators, this validation gap is load-bearing. The authors explicitly concede that the paper can say more about behavior under LLM evaluation than about human judgment; I would like that qualification to be reflected in the abstract and conclusion, or the missing human ratings to be supplied.
minor comments (3)
- [Figure 3] The caption mentions a red cross marking the winning Pillsbury recipe, but the figure would be easier to read with explicit panel labels and significance annotations; error bars are shown as one standard deviation, but no statistical tests are indicated on the figure itself.
- [Results and Interpretation, Experiment 1] The one-iteration baseline values (in-loop weighted score 4.71 versus 4.00, final creativity roughly 4.1) appear only in the text; a small table or figure would improve reproducibility and make the comparison easier to verify.
- [References] The reference to Taylor et al. (2024) concerns clinical tasks and is not an obvious citation for the model-size effect in creative generation; a more directly relevant scaling-law or model-capability reference would strengthen the motivation in the Introduction.
Circularity Check
Central scorer-size effect is partially circular: the in-loop Meta-17B scorer is also one of the four final TTCT evaluators, so the outcome measure overlaps with the manipulated variable.
-
other
[Limitations and Future Work; Study Design and Evaluation; Appendix D Table 1]
"Meta-17B also served as both the in-loop scorer and as one of the final evaluators, introducing methodological overlap. Averaging across four models softens but does not remove this dependence between optimization and outcome assessment."
The headline regression result (Appendix D Table 1) estimates LargeEvaluator = -0.161 on overall creativity, with the outcome defined as the average of four TTCT-style LLM evaluators. One of those four evaluators is Meta-17B, the same model that acts as the in-loop selection scorer when LargeEvaluator=1. Thus the final creativity score for the large-scorer condition is partly produced by the very model whose size is the manipulated factor; the negative coefficient can arise from self-consistency between search-time and evaluation-time preferences rather than from a genuine effect on independently judged creativity. The paper concedes this overlap, so the central 'evaluator design matters more than search depth' claim is not fully independently identified.
full rationale
The derivation chain is otherwise largely self-contained: the FunSearch adaptation, the in-loop rubric, and the TTCT prompting structure are described in enough detail to be reproduced, and the comparison against the Pillsbury reference set provides an external, if partial, benchmark. The main circular content is the overlap between the in-loop scorer and one of the four final evaluators, which the authors explicitly acknowledge in Limitations. Because three other evaluators and a partial human benchmark check are also used, the central claim retains independent content and the overlap is softened rather than fatal. The minor self-citation (Choudhury, Vanneste, and Zohrehvand 2024) is used only for a chain-of-thought prompt-design detail and is not load-bearing. On balance the paper is not definitionally circular, but the key quantitative comparison is partially self-referential, warranting a score of 4.
Assumptions & free parameters
free parameters (4)
- in-loop rubric weights =
recipe 70%, story 30%
- search topology constants =
7 islands, batch size 5
- recipe skeleton constraints =
at most 10 ingredients, 1 Pillsbury ingredient, prep under 30 minutes, at most 2000 character instructions, at most…
- temperature levels =
0.5, 1.0, 1.5
assumptions (4)
- domain assumption TTCT-style dimensions can be applied to recipe products as measures of creativity.
- domain assumption LLM-based TTCT evaluation is a useful proxy for human creativity judgment.
- domain assumption The 30 human recipes from the 2024 Bake-Off are a meaningful human creativity reference set.
- standard math Dummy-coded OLS regression is appropriate for 1-5 Likert score outcomes.
Cite this review
Pith. "Pith review of Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models." pith.science (2026). https://pith.science/paper/L7ASQBB6
@misc{pith2026260807243,
author = {Pith},
title = {Pith review of: Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/L7ASQBB6}},
note = {Machine review of arXiv:2608.07243}
}
read the original abstract
Generative models are often evaluated through singular artifacts, whereas human creativity typically emerges through iterative generation, appraisal, and refinement. This pilot study examines whether iterative search improves LLM creativity by adapting FunSearch to recipe generation for the 2024 Pillsbury Bake-Off and evaluating outputs against human benchmarks using TTCT-based LLM evaluation. Across two experiments, we test iteration count, generator temperature, and in-loop selection-scorer model size. Results show that iterative generation-selection can produce recipes with creativity scores comparable to human benchmarks, but additional iterations alone do not improve creativity. The in-loop evaluator matters most: a smaller selection scorer yields significantly higher scores across most TTCT dimensions, while temperature has limited effects except for originality. These findings suggest that evaluator design is a first-order design variable in subjective creative search.
Figures
Reference graph
Works this paper leans on
-
[1]
Al Rabeyah, A.; G \'o es, F.; Volpe, M.; and Medeiros, T. 2025. Do LLMs agree on the creativity evaluation of alternative uses? In Proceedings of the International Conference on Computational Creativity (ICCC'25)
work page 2025
-
[2]
Bae, M., and Kim, H. 2024. Collective critics for creative story generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 18784--18819
work page 2024
-
[3]
Boden, M. A. 2004. The Creative Mind: Myths and Mechanisms . Routledge, 2nd edition
work page 2004
-
[4]
Chakrabarty, T.; Laban, P.; Agarwal, D.; Muresan, S.; and Wu, S. 2024. Art or artifice? Large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems
work page 2024
-
[5]
Chen, W.-L.; Wei, Z.; Zhu, X.; Feng, S.; and Meng, Y. 2025. Do LLM evaluators prefer themselves for a reason? arXiv
work page 2025
-
[6]
Choudhury, P.; Vanneste, B.; and Zohrehvand, A. 2024. The Wade test: Generative AI and CEO communication. Harvard Business School Working Paper
work page 2024
-
[7]
Csikszentmihalyi, M. 1988. Society, culture, and person: A systems view of creativity. In Sternberg, R. J., ed., The Nature of Creativity: Contemporary Psychological Perspectives . Cambridge University Press. 325--339
work page 1988
-
[8]
Dub\' e , J., and Xu, A. 2026. Large language models and creative content design: A case study of email marketing at wine access. Quantitative Marketing and Economics
work page 2026
Show all 30 references
-
[9]
Franceschelli, G., and Musolesi, M. 2024. Creative beam search: LLM-as-a-Judge for improving response generation. In Proceedings of the 15th International Conference on Computational Creativity (ICCC'24)
2024
-
[10]
F.; Volpe, M.; Sawicki, P.; Grzes, M.; and Watson, J
Goes, L. F.; Volpe, M.; Sawicki, P.; Grzes, M.; and Watson, J. 2023. Pushing GPT 's creativity to its limits: Alternative uses and torrance tests. In Proceedings of the International Conference on Computational Creativity (ICCC'23)
2023
-
[11]
Gong, Y.-J.; Tang, K.; Li, J.-Y.; and Yao, X. 2015. Distributed evolutionary algorithms and their models: A survey of the state-of-the-art. Applied Soft Computing 34:286--300
2015
-
[12]
Haase, J., and Hanel, P. H. P. 2023. Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity. Journal of Creativity 33(3):100066
2023
-
[13]
Harada, T., and Alba, E. 2020. Parallel genetic algorithms: A useful survey. ACM Computing Surveys 53(4)
2020
-
[14]
Hubert, K.; Awa, K.; and Zabelina, D. 2024. The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks. Scientific Reports 14:3440
2024
-
[15]
G.; and Clarke, C
Lamb, C.; Brown, D. G.; and Clarke, C. L. A. 2018. Evaluating computational creativity: An interdisciplinary tutorial. ACM Computing Surveys 51(2):28:1--28:34
2018
-
[16]
P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P
Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-refine: Iterative refinement with self-feedback. arXiv
2023
-
[17]
S.; Dayan, P.; and Stevenson, C
Nath, S. S.; Dayan, P.; and Stevenson, C. 2024. Characterising the creative process in humans and large language models. In Proceedings of the International Conference on Computational Creativity (ICCC'24)
2024
-
[18]
Pandhare, H. 2024. Evaluating large language models: Frameworks and methodologies for AI / ML system testing. International Journal of Scientific Research and Management
2024
-
[19]
Peeperkorn, M.; Kouwenhoven, T.; Brown, D.; and Jordanous, A. 2024. Is temperature the creativity parameter of large language models? In Proceedings of the International Conference on Computational Creativity (ICCC'24)
2024
-
[20]
Perez Y Perez, R., and Sharples, M. 2001. Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental and Theoretical Artificial Intelligence 13(2):119--139
2001
-
[21]
Romera-Paredes, B.; Barekatain, M.; Novikov, A.; Balog, M.; Kumar, M.; Dupont, E.; Ruiz, F.; Ellenberg, J.; Wang, P.; Fawzi, O.; Kohli, P.; and Fawzi, A. 2024. Mathematical discoveries from program search with large language models. Nature
2024
-
[22]
Saunders, R. 2019. Multi-agent based models of social creativity. In Readings in Computational Creativity . Springer
2019
-
[23]
Sawyer, R. 2021. The iterative and improvisational nature of the creative process. Journal of Creativity 31:100002
2021
-
[24]
Stevenson, C.; Smal, I.; Baas, M.; Grasman, R.; and van der Maas , H. 2022. Putting GPT -3's creativity to the (alternative uses) test. In Proceedings of the International Conference on Computational Creativity (ICCC'22)
2022
-
[25]
Taylor, N.; Ghose, U.; Rohanian, O.; Nouriborji, M.; Kormilitzin, A.; Clifton, D.; and Nevado-Holgado, A. 2024. Efficiency at scale: Investigating the performance of diminutive language models in clinical tasks. Artificial Intelligence in Medicine
2024
-
[26]
Torrance, E. P. 1966. Torrance Tests of Creative Thinking: Norms-Technical Manual . Princeton, NJ: Personnel Press
1966
-
[27]
Wataoka, K.; Takahashi, T.; and Ri, R. 2024. Self-preference bias in LLM -as-a-judge. arXiv
2024
-
[28]
Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems
2022
-
[29]
Zhao, Y.; Zhang, R.; Li, W.; Huang, D.; Guo, J.; Peng, S.; Hao, Y.; Wen, Y.; Hu, X.; Du, Z.; Guo, Q.; Li, L.; and Chen, Y. 2024. Assessing and understanding creativity in large language models. arXiv
2024
-
[30]
Zhou, E., and Lee, D. 2024. Generative artificial intelligence, human creativity, and art. PNAS Nexus
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.