{"id":"422e1dc3-5811-4908-bf51-f8cc0b33bc74","arxiv_id":"2411.14193","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A genetic-improvement search over ComfyUI workflow JSONs, guided by ImageReward, raises median reward scores by about 50% and wins about 90% of human preference comparisons.","lead":"ComfyGI automatically tweaks the settings and prompts in ComfyUI image-generation workflows, using a hill-climbing search guided by an AI quality score, until the generated images look better and match the description more closely. A human study found people preferred the automatically improved images about 90% of the time, so the tool could save designers hours of manual prompt tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 50% ImageReward gain and 90% human-preference rate are measured against a randomly initialized workflow, and Section 4.3 shows checkpoint selection alone dominates early gains; the causal attribution to hill-climbing search is not yet established.","rationale":"The reader correctly identifies the ImageReward double-use (optimizer objective equals headline metric) and the random-baseline concern, and appropriately credits the human study as independent evidence. My added emphasis is on causal attribution: the paper's own Figure 10 shows the checkpoint operator delivering the largest first-generation improvements, and the experimental setup randomizes the starting checkpoint. This makes the most load-bearing question not 'does ImageReward misrank?' (the human study partly answers that) but 'would any reasonable fixed workflow already show most of the gain without hill-climbing?' The human evaluation and per-category results are real strengths and rule out a purely self-fulfilling ImageReward story for the random baseline, but they do not isolate the contribution of the search process. A cheap no-search control would settle this. Since the paper is internally consistent and reports a well-run human study, I keep the verdict CONDITIONAL rather than moving to REJECT: the requested control could confirm the search's contribution, or it could show that checkpoint selection is the real driver, which would require reframing the central claim.","tokens_in":16531,"tokens_out":2385,"duration_ms":22082,"concrete_test":"Run a no-search control: for each of the 42 prompts and each of the 10 seeds, generate one image with each of the 9 checkpoint models at the workflow's default KSampler settings, score all with ImageReward, and record the best score. Also include a second control that applies only a single LLM prompt rewrite plus the best single-generation mutation. Compare these control distributions against ComfyGI's optimized scores. If the checkpoint-selection control already recovers most of the 50% median gain, the improvement is attributable to greedy model selection rather than hill-climbing search, and the headline should be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that ComfyGI's hill-climbing search improves text-to-image quality, giving a 50% median ImageReward gain and 90% human preference. The load-bearing weakness is causal attribution. In Section 4.1 and Appendix A, each of the 10 runs uses a random checkpoint model and a random seed in the initial workflow; Section 4.3 reports that the checkpoint mutation alone contributes over 1.75 points of improvement in the first generation. The baseline 'initial workflow' is therefore not a user-tuned or even default workflow, but an unlucky random draw from a pool of nine models. The observed gain could be driven largely by replacing a poor random checkpoint with a better-suited pre-trained model, rather than by iterative patch construction. Moreover, the search objective (ImageReward, Section 3.1) is the same metric used for the headline 50% improvement claim (Section 4.1). The human study is genuinely independent, but it compares only random-initial versus optimized images, so it inherits the same baseline-attribution problem. Thus the paper establishes that optimized workflows beat a random default, but not that the hill-climbing search itself is what delivers the gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ComfyGI, a method that automatically improves ComfyUI text-to-image generation workflows by applying genetic-improvement style hill climbing to the workflow's JSON representation. Mutations change the checkpoint model, KSampler settings, and prompts (via word-level operations or LLM-based rewriting), with each candidate workflow scored by the ImageReward model. In experiments on 42 ImagenHub prompts with 10 runs each, the median ImageReward score of optimized workflows is reported to improve by about 50% over the initial workflow, and a human study with 100 annotators shows the optimized images are preferred in roughly 90% of pairwise comparisons (Wilcoxon p<0.0001, Gwet's AC1=0.6346). The paper also analyzes per-mutation contributions and convergence behavior.","tokens_in":16962,"tokens_out":4634,"duration_ms":48427,"significance":"If the results hold, ComfyGI provides a practical, fully automatic workflow optimizer for text-to-image generation, building on genetic improvement and LLM-based mutation operators. The human evaluation is a notable strength: it uses a sizable annotator pool, attention checks, and inter-rater reliability reporting, giving independent evidence that the optimized images are indeed preferred. The method is simple, extensible, and the authors have released the project code. However, the scientific significance is currently limited by the absence of control experiments that isolate the contribution of the hill-climbing search from other factors, and by the reliance on ImageReward as both the fitness function and the headline evaluation metric.","major_comments":[{"comment":"The causal attribution of the reported gains to the hill-climbing search is not established. The experiments initialize each of the 10 runs with a random checkpoint model and a random seed (Appendix A, Figure 13), and Section 4.3 and Figure 10 show that the checkpoint mutation alone produces an average improvement of over 1.75 points in the first generation. A large part of the ~50% median ImageReward improvement may therefore simply reflect replacing an unlucky random checkpoint with a model that is better suited to the prompt, rather than the iterative patch construction that is the paper's central claim. To support the claim that ComfyGI's search is effective, the authors should compare against control conditions: (a) a random-search baseline that applies the same mutation operators with the same mutation budget but without greedy selection; (b) a checkpoint-only baseline that, for each prompt, selects the best checkpoint by evaluating all available models while keeping the remaining settings fixed; and (c) runs starting from a fixed, sensible default workflow (e.g., Stable Diffusion 1.5 with default KSampler settings) rather than a random draw. These controls would separate the effect of the search mechanism from the effect of chance checkpoint initialization.","section":"Section 4.1 / Appendix A"},{"comment":"The headline 50% ImageReward improvement is not independent evidence of quality improvement because ImageReward is exactly the objective being optimized by the search. The human evaluation is a genuinely independent check, but it only compares the random-initial workflow with the fully optimized workflow, and thus inherits the same baseline-attribution problem. The paper should report quantitative agreement between ImageReward scores and human preferences (e.g., correlation or accuracy on the pairwise comparisons) to validate the proxy in this setting, and should discuss the selection-artifact issue explicitly. Without such evidence, a reader cannot tell whether the ImageReward gain is a meaningful reflection of the qualities the authors claim to optimize or merely an artifact of hill climbing on a smooth score.","section":"Section 3.1 / Section 4.1"},{"comment":"The human evaluation does not distinguish between the contribution of the hill-climbing search and the contribution of simply choosing a better checkpoint from the predefined set. The paper should add a human-evaluation condition comparing the fully optimized workflow against a strong fixed baseline, such as the best single checkpoint selected per prompt on a separate validation run, or against the output of the random-search control suggested above. This would show whether the human-preference advantage is due to the search process or due to model selection alone. Without this comparison, the claim that ComfyGI's 'genetic improvement' techniques are responsible for the ~90% win rate is not fully supported.","section":"Section 4.2 / Section 4.3"}],"minor_comments":[{"comment":"The statistical test is reported only as 'Wilcoxon signed-rank test with p<0.0001'; the authors should specify the unit of analysis (e.g., per-prompt median win rate, per-participant preference counts, or per-image-pair choices), whether the test is two-sided, and how tied responses were handled.","section":"Section 4.2"},{"comment":"The y-axis in Figure 6 is labeled 'Average improvement' but the precise definition (average over what? improvement over the initial score?) is not given in the main text. Please define the quantity and report confidence intervals for the improvement curves.","section":"Figure 6 and Section 4.1"},{"comment":"The KSampler mutation ranges (steps, CFG, denoise, seed range) are only listed in Appendix A. Since the mutation operators are central to the method, these settings should be stated in the main text or the reader should be directed to Table 1 at first mention.","section":"Section 3.1 / Table 1"},{"comment":"The priming tells participants to prefer the prompt-aligned image even if it is less aesthetically pleasing; this could introduce a bias toward alignment over aesthetics, which is a stated goal but should be acknowledged as a potential limitation of the human study.","section":"Appendix D"},{"comment":"The impact statement is only two sentences. Given the potential for misuse of automated image generation (e.g., deepfakes, deceptive content), a fuller discussion of risks and mitigations would strengthen the paper.","section":"Section 5 / Impact Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a well-executed demonstration that a heuristic search over workflow JSON can produce images that humans prefer, with a solid human evaluation. The core weakness is causal attribution: the random initialization and the dominant checkpoint mutation mean the reported gains may be largely due to model selection, not the hill-climbing search. The editorial decision should hinge on whether the authors can provide control experiments (random search, checkpoint-only, and a fixed-default initialization). If they do, the paper is potentially a good fit for a journal focused on search-based software engineering or applied machine learning. The authors should also be encouraged to tone down the abstract's implication that the GI search itself is responsible for the improvements until such evidence is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: ComfyGI is real, usable engineering. It is the first GI-style search over ComfyUI JSON workflows, with mutation operators for checkpoints, sampler settings, and prompts including LLM rewrites. The human evaluation is well run: 100 annotators, attention checks, randomized side presentation, Gwet AC1 0.63, about 90% preference for optimized images. That is genuine evidence that the optimized workflows produce images people prefer over the starting images.\n\nThe load-bearing weakness is the baseline. Every run starts from a random checkpoint and a random seed. Section 4.3 shows the checkpoint mutation alone delivers over 1.75 average ImageReward points in generation one—most of the headline 50% gain. So the paper shows optimized beats random, not that hill-climbing search itself matters. Simple competing explanations—try all nine checkpoints, or random search over checkpoints and seeds, or use ImageReward to pick the best seed—are not tested. The 50% improvement is measured with ImageReward, the objective being maximized, so that number is largely a selection artifact; the human study is independent, but it inherits the same baseline problem when it compares optimized images to the same random initial ones.\n\nThe paper is honest about the checkpoint effect, and the appendix is unusually complete: JSON workflow, full model list, prompt templates, demographics. The mutation operator analysis is a useful contribution, even though it underlines the attribution problem.\n\nThis is an incremental applied advance, but a useful one for practitioners and for the GI community. It deserves peer review, conditional on adding the missing baselines and at least a few runs starting from a user-tuned, non-random workflow. If the hill-climbing search still holds up, it is a solid venue paper; if not, the checkpoint-selection result alone could still be a shorter paper. I would not desk reject it.","headline":"A solid applied GI-for-image-workflows paper with a strong human eval, but the headline gains are mostly checkpoint selection against a random baseline; needs baseline comparisons before the search itself gets credit.","tokens_in":17308,"tokens_out":2941,"would_cite":true,"duration_ms":27816,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ComfyGI shows that an automated hill-climbing search over the JSON of a ComfyUI workflow can improve text-to-image quality without human involvement, with median ImageReward scores rising by about 50% and human raters preferring the…","keywords":["ComfyGI","genetic improvement","image generation","text-to-image","hill climbing","ImageReward","workflow optimization","diffusion models"],"falsifier":"A direct test would compare ComfyGI's optimized images against images produced by the same workflow after a human expert tunes it, using the same human evaluation protocol; if the ComfyGI images do not win at a significantly higher rate, the claim that the hill-climbing search is an effective automatic improvement method fails.","tokens_in":16357,"feed_emoji":"🖼️","tokens_out":5979,"duration_ms":47819,"temperature":0.7,"pith_summary":"The paper introduces ComfyGI, a system that automatically improves text-to-image generation workflows by applying small mutations to the JSON representation of a ComfyUI workflow and keeping changes that raise an automated quality score. The central claim is that this search, guided only by the ImageReward model, produces images that better match the prompt and are more aesthetically pleasing, with median ImageReward scores rising by about 50% and human raters preferring the optimized images in roughly 90% of pairwise comparisons. If true, this means effective image-generation tuning does not require human judgment in the loop, and the same search idea could be extended to new workflow components and quality metrics.","feed_headline":"Automated workflow tuning wins human favor 90% of the time","feed_subtitle":"A genetic-improvement search over ComfyUI workflows lifts median ImageReward scores by about 50%, no human in the loop.","key_machinery":"The central mechanism is the hill-climbing patch search over the workflow's JSON representation. Each mutation operator targets one module: checkpoint (replacing the diffusion model), ksampler (changing seed, steps, CFG, sampler, scheduler, denoise), prompt word and prompt statement (editing prompt text with copy, switch, remove, add, replace operations), and prompt llm (rewriting the prompt with a randomly chosen LLM). The ImageReward model is the objective function that scores every generated image; the search accepts a mutation only if it raises the best score, and terminates when no further improvement is found. The resulting patch is applied to the original workflow to generate the final image.","core_discovery":"ComfyGI's central discovery is that a hill-climbing genetic-improvement search over the JSON of a ComfyUI text-to-image workflow can substantially improve output quality. Starting from a workflow with a randomly selected checkpoint and seed, the method repeatedly mutates the checkpoint, sampler settings, and prompt (using word-level edits and LLM rewrites), generates an image for each mutant, and accepts the single best mutation per generation as part of a patch. The patch is applied to the workflow to produce the final image. Across 42 prompts from 14 categories and 10 runs per prompt, the median ImageReward score improves by about 50% relative to the initial image, and all prompt categories improve. A human study with 100 annotators confirms the direction: the optimized image is preferred in about 90% of pairwise comparisons, with high inter-rater reliability (Gwet's AC1 = 0.6346).","pith_inferences":["Because the search optimizes ImageReward, the resulting images will reflect that model's notion of quality; swapping in a different reward model would steer the search toward different properties, such as photorealism or safety.","The 90% human-preference figure was obtained against randomly initialized workflows; on an already well-tuned workflow the expected gain is smaller, so the method is best seen as an automated tuner rather than a universal quality booster.","The category-dependent checkpoint choices suggest that a learned model-routing rule could pick the diffusion model before the search even starts, saving computation.","Extending the mutation set to newer modules such as ControlNet, LoRA, or IP-Adapter would likely open further improvements, but each requires a bespoke mutation operator."],"forward_implications":["Practitioners can improve their ComfyUI workflows automatically without manual prompt engineering or parameter tuning.","The improvement appears across all 14 prompt categories tested, including misspellings, rare words, counting, and text rendering, suggesting the method is not limited to easy prompts.","Changing the checkpoint model produces the largest early gains, indicating that automated model selection is a key lever for generation quality.","Most runs converge within three generations, so the search is fast enough for routine use on a single prompt.","Because workflows are stored as JSON patches, the same search mechanism can be applied to any ComfyUI workflow with additional modules."],"supporting_citations":[{"why":"Provides the ImageReward model that scores every generated image and guides the hill-climbing search.","marker":"(Xu et al., 2024)"},{"why":"Supplies the ImagenHub benchmark with 14 prompt categories from which the 42 test prompts are sampled.","marker":"(Ku et al., 2024)"},{"why":"Defines genetic improvement, the search-based methodology ComfyGI adapts to image-generation workflows.","marker":"(Petke et al., 2017)"},{"why":"Introduces the latent diffusion architecture used by the Stable Diffusion checkpoint models the workflow can switch between.","marker":"(Rombach et al., 2022)"},{"why":"Documents the sensitivity of text-to-image outputs to prompts and hyperparameter settings, motivating the search over workflows.","marker":"(Wang et al., 2022)"}],"fun_headline_variants":["ComfyGI auto-tunes ComfyUI workflows, no human needed","Genetic improvement lifts image quality 50% in benchmark","Human evaluators pick ComfyGI images 9 of 10 times","Automatic workflow tweaks yield 50% better ImageReward","ComfyGI: hands-free image workflow improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search is guided entirely by ImageReward's score, so the method assumes that ImageReward is a valid and smooth proxy for the human preferences the paper claims to improve: prompt alignment and aesthetics. If ImageReward misranks images, the hill climber optimizes the wrong objective.","fun_headline_variants_meta":{"raw":{"variants":["ComfyGI auto-tunes ComfyUI workflows, no human needed","Genetic improvement lifts image quality 50% in benchmark","Human evaluators pick ComfyGI images 9 of 10 times","Automatic workflow tweaks yield 50% better ImageReward","ComfyGI: hands-free image workflow improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3158,"prompt_tokens":866,"completion_tokens":2292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":2206}},"tokens_in":482,"tokens_out":2292,"duration_ms":15218,"temperature":1.0,"reasoning_tokens":2206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:25:51.031638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would compare ComfyGI's optimized images against images produced by the same workflow after a human expert tunes it, using the same human evaluation protocol; if the ComfyGI images do not win at a significantly higher rate, the claim that the hill-climbing search is an effective automatic improvement method fails.","supporting_citations":[{"cited_title":"O., Harman, M., Langdon, W","cited_arxiv_id":null,"evidence_quote":"Defines genetic improvement, the search-based methodology ComfyGI adapts to image-generation workflows."}],"review_version":1}