Pith. sign in

REVIEW 4 major objections 3 minor 30 references

Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read The in-loop evaluator, not search depth or temperature, is what determines creativity scores in iterative LLM recipe search.

desk verdict A transparent pilot with a plausible but unproven central claim: the in-loop evaluator seems to matter more than iteration or temperature, but the evidence rests on unvalidated LLM evaluation and a scorer/judge overlap. read the letter →

arxiv 2608.07243 v1 pith:L7ASQBB6 submitted 2026-08-07 cs.AI cs.CLcs.NE

classification cs.AIcs.CLcs.NE
keywords iterativegenerationLLMcreativityFunSearchin-loopevaluatorTTCTevaluationrecipetemperatureselectionscorer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This pilot study asks whether iterative generation-and-evaluation search improves the creativity of LLM-generated recipes, and which design factor matters most. It adapts the FunSearch program-search algorithm, originally built for objective tasks, to the 2024 Pillsbury Bake-Off: recipes are generated under a competition skeleton, an in-loop scorer keeps the best, and the survivors seed the next round. Across two experiments manipulating iteration count, generator temperature, and in-loop scorer size, the paper finds that iterative search can reach human-comparable creativity scores but extra iterations alone do not help. The in-loop evaluator is the decisive factor: a smaller scorer yields significantly higher creativity, fluency, flexibility, and elaboration scores than a larger one, while lower temperature mainly suppresses originality. The paper concludes that evaluator design is a first-order design variable in subjective creative search.

What carries the argument

The load-bearing mechanism is the FunSearch loop adapted to a subjective domain. Semi-isolated islands preserve different recipe lineages; best-shot prompting rebuilds prompts from high-scoring candidates; and an in-loop selection scorer applies a weighted Pillsbury-style rubric (recipe content 70%, story 30%) to decide which candidates survive into the next generation. The final creativity analysis is kept separate from the search objective, using four post hoc LLM evaluators with TTCT-derived definitions of fluency, flexibility, originality, and elaboration. The controlled contrast that carries the argument is the in-loop scorer's model size, since the generator stays fixed: the same search loop is run with a smaller and a larger evaluator, and the smaller one wins on most dimensions.

What would settle it

Have a panel of human cooks rate, blind, the same retained recipes from the small-scorer and large-scorer conditions; the central claim fails if average human creativity ratings do not favor the small-scorer condition, or if the official Bake-Off winner is not rated at or near the top of the human reference set.

Watch

Extended reading notes

Core claim

The central claim is that when creativity is judged subjectively, the in-loop evaluator shapes how much iterative search improves outputs more than search depth or sampling randomness does. The paper supports this by adapting FunSearch to recipe generation: multi-island search keeps separate lineages, high-scoring candidates seed later prompts, and an in-loop selection scorer applies a weighted Pillsbury-style rubric (70% recipe, 30% story) to admit or discard candidates. Final creativity is assessed separately by four LLM evaluators using TTCT-derived product-level dimensions: fluency, flexibility, originality, and elaboration. In the main comparison, the smaller 8B selection scorer produced significantly higher final scores on creativity, fluency, flexibility, and elaboration than the larger 17B scorer, whereas moving from 5 to 30 iterations produced no clear monotonic gain and lowering generator temperature only reduced originality. The paper interprets this as evidence that the evaluative ecology, not the amount of generation, determines whether iterative novelty accumulates in subjective creative domains.

Load-bearing premise

The load-bearing premise is that the automated judges used to score creativity are a valid stand-in for human taste in recipes; if they mostly reward rubric compliance or their own preferences, the reported advantage of the smaller in-loop scorer would not carry over to human judgment.

Editorial extensions

If this is right

  • Adding more FunSearch iterations to a recipe-generation loop does not by itself raise final creativity scores; the near-one-shot baseline and the 5, 15, and 30 iteration conditions land at similar levels.
  • Switching the in-loop selection scorer from a larger to a smaller model reliably raises final creativity, fluency, flexibility, and elaboration scores under TTCT-style LLM evaluation.
  • Lower generator temperature mainly costs originality; it does not improve any other TTCT dimension at a significant level.
  • Iterative search with a well-chosen evaluator can produce recipe artifacts whose LLM-judged creativity is comparable to a human reference set from the same competition.
  • Creative LLM systems should treat the diversity, architecture, and incentive structure of in-loop evaluators as a central design variable rather than relying on more generation cycles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether the evaluator-size effect survives human judging: because the paper's own benchmark shows only partial agreement with the official competition outcome, a human panel could plausibly reverse the smaller-scorer advantage.
  • The paper's framing suggests varying the in-loop evaluator during search, for example rotating several scorers or changing rubrics between islands, as a way to escape convergence on conservative recipes, but this extension is not tested here.
  • Because the larger in-loop scorer also served as one of the final evaluators, a cleaner replication with fully disjoint generator, scorer, and final judge models is needed to rule out self-preference as the source of the size effect.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper adapts the FunSearch evolutionary generation loop to a subjective creative domain—recipe generation for the 2024 Pillsbury Bake-Off—and evaluates the resulting artifacts with four TTCT-style LLM evaluators. Two experiments manipulate iteration count, generator temperature, and in-loop selection-scorer model size. The authors report that iterative generation can reach creativity scores comparable to a human reference set, that additional iterations do not improve scores, that the smaller Meta-8B in-loop scorer yields significantly higher scores than the Meta-17B scorer on most TTCT dimensions, and that lower temperature mainly reduces originality. They conclude that evaluator design is a first-order design variable in subjective creative search.

Significance. If the main finding holds, it would be a valuable reframing for computational creativity: the design of the in-loop evaluator, rather than additional search depth or sampling stochasticity, may determine the creativity of iteratively generated artifacts. The paper has real strengths: it transfers a well-known program-search method to a non-objective domain, provides detailed appendices with the recipe skeleton, in-loop rubric, evaluation prompt structure, and full regression output, and explicitly acknowledges its main limitation—that Meta-17B serves as both the in-loop scorer and one of the final evaluators. The use of four LLM evaluators and a partial human benchmark is also a reasonable pilot-level design. However, the statistical support is weak and the evaluator-overlap issue is load-bearing, so the contribution is best assessed as a promising pilot rather than a fully established result.

major comments (4)
  1. [Study Design and Evaluation; Limitations and Future Work; Appendix D, Table 1] The central negative LargeEvaluator coefficient in Appendix D, Table 1 is confounded by the fact that Meta-17B is both the in-loop selection scorer and one of the four final TTCT evaluators. The paper acknowledges this overlap in the Limitations section, but it does not address it quantitatively. Because the outcome is the average of four LLM evaluators, one of which is identical to the manipulated scorer, the observed 8B-over-17B difference could partly reflect self-preference or rubric alignment rather than perceived creativity. I request a re-analysis that excludes the Meta-17B evaluator from the outcome variable, or a small human-rating study on a subset of the generated recipes, before the evaluator-design claim is made.
  2. [Appendix D, Table 1; Results and Interpretation] The regression evidence is statistically fragile. Adjusted R2 values range from 0.006 to 0.037, meaning the manipulated factors explain almost none of the variance in the outcomes, and the five outcome models together involve fifteen significance tests with no multiple-comparison correction. The originality model, which is the basis for the claim that temperature mainly reduces originality, has adjusted R2 = 0.006. The paper should report standardized effect sizes, confidence intervals, and either corrected p-values or a pre-specified analysis plan before claiming that evaluator size 'matters most.' It should also include iteration count and scorer size in a single comparative model if the conclusion is to be that evaluator design matters more than search depth.
  3. [Results and Interpretation, Experiment 1] Experiment 1 reports mean final creativity scores of 3.921, 3.835, and 3.927 for 5, 15, and 30 iterations, respectively, but does not provide error bars, confidence intervals, or inferential tests for these means. The claim that additional iterations do not improve creativity, and that iterative search reaches scores comparable to the human benchmark, is therefore not supported by the statistics reported. The one-iteration baseline comparison (in-loop score dropping from 4.71 to 4.00, final creativity roughly 4.1) is also given only as point estimates in the text. I recommend reporting per-condition uncertainty and formal comparisons, and ideally placing Experiment 1 and Experiment 2 in a single model that allows a direct test of the relative importance of iteration count and evaluator size.
  4. [Study Design and Evaluation; Limitations and Future Work] The benchmark check validates only the in-loop rubric against the human competition, and even there the agreement is partial: the human winning recipe is placed 'near the top' rather than at rank one, and the stories are absent from the comparison. The post hoc TTCT-style evaluators are not validated against human creativity judgments of the generated recipes. Since the paper's central comparative claim ('evaluator design matters more than search depth') is expressed in terms of creativity scores from these LLM evaluators, this validation gap is load-bearing. The authors explicitly concede that the paper can say more about behavior under LLM evaluation than about human judgment; I would like that qualification to be reflected in the abstract and conclusion, or the missing human ratings to be supplied.
minor comments (3)
  1. [Figure 3] The caption mentions a red cross marking the winning Pillsbury recipe, but the figure would be easier to read with explicit panel labels and significance annotations; error bars are shown as one standard deviation, but no statistical tests are indicated on the figure itself.
  2. [Results and Interpretation, Experiment 1] The one-iteration baseline values (in-loop weighted score 4.71 versus 4.00, final creativity roughly 4.1) appear only in the text; a small table or figure would improve reproducibility and make the comparison easier to verify.
  3. [References] The reference to Taylor et al. (2024) concerns clinical tasks and is not an obvious citation for the model-size effect in creative generation; a more directly relevant scaling-law or model-capability reference would strengthen the motivation in the Introduction.

Circularity Check

1 steps flagged · score 4.0 of 10

Central scorer-size effect is partially circular: the in-loop Meta-17B scorer is also one of the four final TTCT evaluators, so the outcome measure overlaps with the manipulated variable.

  1. other [Limitations and Future Work; Study Design and Evaluation; Appendix D Table 1]
    "Meta-17B also served as both the in-loop scorer and as one of the final evaluators, introducing methodological overlap. Averaging across four models softens but does not remove this dependence between optimization and outcome assessment."

    The headline regression result (Appendix D Table 1) estimates LargeEvaluator = -0.161 on overall creativity, with the outcome defined as the average of four TTCT-style LLM evaluators. One of those four evaluators is Meta-17B, the same model that acts as the in-loop selection scorer when LargeEvaluator=1. Thus the final creativity score for the large-scorer condition is partly produced by the very model whose size is the manipulated factor; the negative coefficient can arise from self-consistency between search-time and evaluation-time preferences rather than from a genuine effect on independently judged creativity. The paper concedes this overlap, so the central 'evaluator design matters more than search depth' claim is not fully independently identified.

full rationale

The derivation chain is otherwise largely self-contained: the FunSearch adaptation, the in-loop rubric, and the TTCT prompting structure are described in enough detail to be reproduced, and the comparison against the Pillsbury reference set provides an external, if partial, benchmark. The main circular content is the overlap between the in-loop scorer and one of the four final evaluators, which the authors explicitly acknowledge in Limitations. Because three other evaluators and a partial human benchmark check are also used, the central claim retains independent content and the overlap is softened rather than fatal. The minor self-citation (Choudhury, Vanneste, and Zohrehvand 2024) is used only for a chain-of-thought prompt-design detail and is not load-bearing. On balance the paper is not definitionally circular, but the key quantitative comparison is partially self-referential, warranting a score of 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central comparison depends on hand-chosen rubric weights and skeleton constraints, plus the assumption that LLM judges capture creativity. These are disclosed, but they are load-bearing and would need sensitivity checks.

free parameters (4)
  • in-loop rubric weights = recipe 70%, story 30%
    Hand-set in Appendix B; determines which candidates survive and therefore shapes all downstream creativity scores.
  • search topology constants = 7 islands, batch size 5
    Fixed for Experiment 2 in Study Design and Evaluation; not varied, so interactions with scorer size are unmeasured.
  • recipe skeleton constraints = at most 10 ingredients, 1 Pillsbury ingredient, prep under 30 minutes, at most 2000 character instructions, at most…
    Appendix A defines the candidate space; these thresholds are arbitrary operationalizations of the Bake-Off rules and exclude recipes outside them.
  • temperature levels = 0.5, 1.0, 1.5
    Chosen as experimental conditions informed by prior work; the spacing is a design choice, not derived from theory.
assumptions (4)
  • domain assumption TTCT-style dimensions can be applied to recipe products as measures of creativity.
    Invoked in Study Design and Evaluation, following prior work (Chakrabarty et al. 2024), to justify the outcome metric.
  • domain assumption LLM-based TTCT evaluation is a useful proxy for human creativity judgment.
    Used for both in-loop selection and final scoring; authors acknowledge partial agreement with human benchmark in Limitations.
  • domain assumption The 30 human recipes from the 2024 Bake-Off are a meaningful human creativity reference set.
    Used in Study Design and Evaluation as 'partial calibration, not a fully matched contest replication'.
  • standard math Dummy-coded OLS regression is appropriate for 1-5 Likert score outcomes.
    Appendix D reports OLS models without addressing ordinality; this is a standard but debatable choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models." pith.science (2026). https://pith.science/paper/L7ASQBB6

@misc{pith2026260807243,
  author       = {Pith},
  title        = {Pith review of: Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7ASQBB6}},
  note         = {Machine review of arXiv:2608.07243}
}
read the original abstract

Generative models are often evaluated through singular artifacts, whereas human creativity typically emerges through iterative generation, appraisal, and refinement. This pilot study examines whether iterative search improves LLM creativity by adapting FunSearch to recipe generation for the 2024 Pillsbury Bake-Off and evaluating outputs against human benchmarks using TTCT-based LLM evaluation. Across two experiments, we test iteration count, generator temperature, and in-loop selection-scorer model size. Results show that iterative generation-selection can produce recipes with creativity scores comparable to human benchmarks, but additional iterations alone do not improve creativity. The in-loop evaluator matters most: a smaller selection scorer yields significantly higher scores across most TTCT dimensions, while temperature has limited effects except for originality. These findings suggest that evaluator design is a first-order design variable in subjective creative search.

Figures

Figures reproduced from arXiv: 2608.07243 by the authors.

Figure 1
Figure 1. Conceptual framework illustrating how the FunSearch algorithm is integrated with TTCT-derived evaluation to assess [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example recipe generated by the LLM after [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Average creativity scores across TTCT dimensions for FunSearch-generated recipes under each experimental condi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

  1. [1]

    Al Rabeyah, A.; G \'o es, F.; Volpe, M.; and Medeiros, T. 2025. Do LLMs agree on the creativity evaluation of alternative uses? In Proceedings of the International Conference on Computational Creativity (ICCC'25)

  2. [2]

    Bae, M., and Kim, H. 2024. Collective critics for creative story generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 18784--18819

  3. [3]

    Boden, M. A. 2004. The Creative Mind: Myths and Mechanisms . Routledge, 2nd edition

  4. [4]

    Chakrabarty, T.; Laban, P.; Agarwal, D.; Muresan, S.; and Wu, S. 2024. Art or artifice? Large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems

  5. [5]

    Chen, W.-L.; Wei, Z.; Zhu, X.; Feng, S.; and Meng, Y. 2025. Do LLM evaluators prefer themselves for a reason? arXiv

  6. [6]

    Choudhury, P.; Vanneste, B.; and Zohrehvand, A. 2024. The Wade test: Generative AI and CEO communication. Harvard Business School Working Paper

  7. [7]

    Csikszentmihalyi, M. 1988. Society, culture, and person: A systems view of creativity. In Sternberg, R. J., ed., The Nature of Creativity: Contemporary Psychological Perspectives . Cambridge University Press. 325--339

  8. [8]

    Dub\' e , J., and Xu, A. 2026. Large language models and creative content design: A case study of email marketing at wine access. Quantitative Marketing and Economics

Show all 30 references
  1. [9]

    Franceschelli, G., and Musolesi, M. 2024. Creative beam search: LLM-as-a-Judge for improving response generation. In Proceedings of the 15th International Conference on Computational Creativity (ICCC'24)

  2. [10]

    F.; Volpe, M.; Sawicki, P.; Grzes, M.; and Watson, J

    Goes, L. F.; Volpe, M.; Sawicki, P.; Grzes, M.; and Watson, J. 2023. Pushing GPT 's creativity to its limits: Alternative uses and torrance tests. In Proceedings of the International Conference on Computational Creativity (ICCC'23)

  3. [11]

    Gong, Y.-J.; Tang, K.; Li, J.-Y.; and Yao, X. 2015. Distributed evolutionary algorithms and their models: A survey of the state-of-the-art. Applied Soft Computing 34:286--300

  4. [12]

    Haase, J., and Hanel, P. H. P. 2023. Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity. Journal of Creativity 33(3):100066

  5. [13]

    Harada, T., and Alba, E. 2020. Parallel genetic algorithms: A useful survey. ACM Computing Surveys 53(4)

  6. [14]

    Hubert, K.; Awa, K.; and Zabelina, D. 2024. The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks. Scientific Reports 14:3440

  7. [15]

    G.; and Clarke, C

    Lamb, C.; Brown, D. G.; and Clarke, C. L. A. 2018. Evaluating computational creativity: An interdisciplinary tutorial. ACM Computing Surveys 51(2):28:1--28:34

  8. [16]

    P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-refine: Iterative refinement with self-feedback. arXiv

  9. [17]

    S.; Dayan, P.; and Stevenson, C

    Nath, S. S.; Dayan, P.; and Stevenson, C. 2024. Characterising the creative process in humans and large language models. In Proceedings of the International Conference on Computational Creativity (ICCC'24)

  10. [18]

    Pandhare, H. 2024. Evaluating large language models: Frameworks and methodologies for AI / ML system testing. International Journal of Scientific Research and Management

  11. [19]

    Peeperkorn, M.; Kouwenhoven, T.; Brown, D.; and Jordanous, A. 2024. Is temperature the creativity parameter of large language models? In Proceedings of the International Conference on Computational Creativity (ICCC'24)

  12. [20]

    Perez Y Perez, R., and Sharples, M. 2001. Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental and Theoretical Artificial Intelligence 13(2):119--139

  13. [21]

    Romera-Paredes, B.; Barekatain, M.; Novikov, A.; Balog, M.; Kumar, M.; Dupont, E.; Ruiz, F.; Ellenberg, J.; Wang, P.; Fawzi, O.; Kohli, P.; and Fawzi, A. 2024. Mathematical discoveries from program search with large language models. Nature

  14. [22]

    Saunders, R. 2019. Multi-agent based models of social creativity. In Readings in Computational Creativity . Springer

  15. [23]

    Sawyer, R. 2021. The iterative and improvisational nature of the creative process. Journal of Creativity 31:100002

  16. [24]

    Stevenson, C.; Smal, I.; Baas, M.; Grasman, R.; and van der Maas , H. 2022. Putting GPT -3's creativity to the (alternative uses) test. In Proceedings of the International Conference on Computational Creativity (ICCC'22)

  17. [25]

    Taylor, N.; Ghose, U.; Rohanian, O.; Nouriborji, M.; Kormilitzin, A.; Clifton, D.; and Nevado-Holgado, A. 2024. Efficiency at scale: Investigating the performance of diminutive language models in clinical tasks. Artificial Intelligence in Medicine

  18. [26]

    Torrance, E. P. 1966. Torrance Tests of Creative Thinking: Norms-Technical Manual . Princeton, NJ: Personnel Press

  19. [27]

    Wataoka, K.; Takahashi, T.; and Ri, R. 2024. Self-preference bias in LLM -as-a-judge. arXiv

  20. [28]

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems

  21. [29]

    Zhao, Y.; Zhang, R.; Li, W.; Huang, D.; Guo, J.; Peng, S.; Hao, Y.; Wen, Y.; Hu, X.; Du, Z.; Guo, Q.; Li, L.; and Chen, Y. 2024. Assessing and understanding creativity in large language models. arXiv

  22. [30]

    Zhou, E., and Lee, D. 2024. Generative artificial intelligence, human creativity, and art. PNAS Nexus

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.