{"id":"d4b71d1a-7714-459e-a888-8a830cbd14b7","arxiv_id":"2506.09790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 7B reasoning model trained with supervised fine-tuning and reinforcement learning generates ComfyUI workflows from text instructions and outperforms GPT-4o and Claude-based baselines on the authors' tests.","lead":"ComfyUI-R1 is a 7B-parameter AI model trained to turn a plain-language instruction into a ready-to-run ComfyUI workflow, using step-by-step reasoning and a two-stage training recipe. It reports higher format validity and workflow-quality scores than prior systems built on GPT-4o or Claude, which matters for the millions of ComfyUI users who currently need expert knowledge to assemble image and video pipelines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ComfyBench head-to-head is not controlled: baseline pass rates are borrowed from the original paper and ComfyUI-R1 uses its own retrieval over a KB never checked for overlap with ComfyBench; the '0.67 vs 0.56' claim is not yet established.","rationale":"The central claim has two evidentiary pillars: the main test set with a candidate-node protocol (Table 1) and the end-to-end ComfyBench comparison (Table 3). The reader focused on the first pillar, arguing that giving the model a candidate set containing the gold nodes makes the task easier than real use. That is a fair external-validity concern, but the paper explicitly scopes the main evaluation as measuring reasoning over pre-retrieved nodes (Sec. 4.1), so it does not by itself invalidate the claimed comparison among methods. The second pillar is more load-bearing because it is the only evaluation that includes retrieval, and it is the one the abstract's 'high pass rate' and 'surpassing prior SOTA' rely on. The baseline numbers being borrowed from the original paper is an uncontrolled comparison: different retrieval, different agent loop, and potentially different execution checks. In addition, the absence of an overlap analysis between the scraped KB and ComfyBench creates a concrete risk of gold-node leakage through retrieval, which would directly inflate ComfyUI-R1's pass rate. Neither issue is evidence of misconduct; both are missing controls that a revision can supply. I also note a secondary reward-design weakness (Eq. 5 measures recall-minus-1 rather than precision, and Eq. 6 gives 0.75 of max reward to any valid but fully incorrect node selection), but this affects training efficiency rather than the validity of the measured F1, so it is not the primary concern. Given that the vulnerability is testable and fixable rather than a demonstrated failure, the reader's CONDITIONAL verdict remains appropriate; no change is needed, but the condition should explicitly require the matched-protocol ComfyBench rerun and the overlap check.","tokens_in":16394,"tokens_out":7065,"duration_ms":78785,"concrete_test":"Run a matched-protocol ComfyBench evaluation: execute the published ComfyAgent with the same execution environment and either (a) no candidate retrieval for both systems or (b) the identical top-3-workflow retrieval used for ComfyUI-R1, and compare pass rates. In the same run, compute the graph-level overlap (e.g., node-set Jaccard or graph edit distance) between the 3,917-workflow KB and the ComfyBench test workflows. If ComfyAgent's pass rate under matched conditions remains 0.56 and overlap is negligible, the concern is resolved; if ComfyAgent improves under matched retrieval or overlap is non-trivial, the reported 0.67 margin is not attributable to the model alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest end-to-end evidence for the central claim is Table 3: ComfyUI-R1's 0.67 pass rate on ComfyBench versus 0.56 for ComfyAgent. However, the paper states that 'the numbers of GPT-4o methods are taken from the original paper' (Table 3 note) rather than reproduced under identical conditions. ComfyUI-R1 is evaluated with its own retrieval protocol (top-3 workflows from a 3,917-workflow KB aggregated via text-embedding-3-small into V_cand), while ComfyAgent's published 0.56 was produced with a different retrieval/agent pipeline, possibly in a different execution environment. The 11-point margin therefore conflates the trained generator with retrieval and evaluation differences. Moreover, the workflow KB was scraped from ComfyUI community sites (Sec. 3.1), and ComfyBench tasks are themselves ComfyUI workflows; the paper reports no overlap or near-duplicate check between the 3,917-workflow training KB and the ComfyBench test set. If ComfyBench workflows (or close variants) appear in the KB, the embedding-based retrieval can leak gold nodes into V_cand, inflating the pass rate. This directly threatens the abstract's 'high pass rate ... significantly surpassing prior state-of-the-art' claim. The candidate-node protocol on the main test set (Sec. 3.2) is disclosed in Sec. 4.1 as intentionally evaluating reasoning over pre-retrieved nodes, so it is a scoped limitation rather than the load-bearing flaw.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ComfyUI-R1, a 7B-parameter reasoning model for generating ComfyUI workflows from natural-language instructions. The authors collect a knowledge base of 3,917 workflows and 7,238 nodes, synthesize long chain-of-thought training data (node selection, design rationale, code representation), fine-tune Qwen2.5-Coder-7B-Instruct on this data, and then apply GRPO with a rule-metric hybrid reward combining format validity, DAG structure, node fidelity, and node-selection accuracy. Evaluation on a self-built 600-sample test set reports 97% format validity, node-level F1 0.62, and graph-level F1 0.51, outperforming few-shot and CoT baselines with GPT-4o and Claude models. An end-to-end experiment on ComfyBench reports a 0.67 pass rate versus 0.56 for ComfyAgent, leading the authors to claim that their 7B model significantly surpasses prior state-of-the-art methods built on larger closed-source models.","tokens_in":16735,"tokens_out":5968,"duration_ms":61582,"significance":"If the results hold, the paper would be a useful demonstration that a 7B open model can outperform much larger closed models on structured workflow generation through SFT followed by reinforcement learning with a custom reward. The comparison of code-based versus JSON-based workflow representations, the detailed knowledge-base construction, and the integration into ComfyUI-Copilot are practical contributions. The reasoning-data generation and reward design are described clearly enough to be replicated, modulo the use of proprietary LLMs to produce rationales and descriptions. However, the strongest end-to-end claim rests on a comparison that is not currently controlled, and the node-selection reward is recall-only, so the evidence for the headline superiority is weaker than the text suggests.","major_comments":[{"comment":"The 0.67 versus 0.56 pass-rate comparison is not a controlled head-to-head. The ComfyAgent and GPT-4o numbers are taken from the original paper, while ComfyUI-R1 is evaluated with its own retrieval pipeline (top-3 workflows from a 3,917-workflow KB embedded with text-embedding-3-small) and an execution harness that is not shown to be identical to the one used for the published baselines. Because the KB is scraped from ComfyUI community sites (Sec. 3.1) and ComfyBench tasks are themselves ComfyUI workflows, the paper must quantify overlap or near-duplicates between the KB and the ComfyBench test set; if gold workflows leak into retrieval, Vcand can contain the correct nodes and inflate the pass rate. I ask for an overlap analysis, a re-run of ComfyAgent under the same retrieval and execution conditions, or an ablation that isolates the generator from the retrieval component.","section":"Sec. 4.5, Table 3"},{"comment":"The node-selection reward R_correct = |Vp∩Vg|/|Vg| - 1 is recall-only. It reaches its maximum of 0 whenever all gold nodes are included, regardless of how many additional valid but non-gold nodes are selected; R_fidelity only penalizes nodes outside Vcand and mismatches between the selected_nodes block and the workflow block, not extra in-candidate nodes. Thus the reward does not penalize a significant class of over-generation, which is also consistent with the case study's observation that ComfyUI-R1 produces more nodes than ComfyAgent. I recommend adding a precision or cardinality term and, at minimum, reporting reward variants in the ablation study.","section":"Sec. 3.3, Eq. (5)"},{"comment":"The main test set is evaluated with an oracle candidate set that always contains the gold node set, which the paper acknowledges as an evaluation of reasoning over pre-retrieved nodes. This is a fair across-model protocol, but it means the high absolute F1 scores and the format-validity rate do not by themselves establish end-to-end usability; the only end-to-end evidence is the ComfyBench experiment, whose control issues are raised above. The paper should either add end-to-end results on its own test set or clearly state in the abstract and conclusion that the superiority claims are for the pre-retrieved-node setting.","section":"Sec. 4.1, Table 1"}],"minor_comments":[{"comment":"The displayed KL divergence is not the standard KL(πθ||πref); as written it places expectations under πref and appears to be the reverse KL with a -1 offset. Please correct the equation or clarify the convention used.","section":"Eq. (9)"},{"comment":"The split statement is numerically inconsistent: 3,717 workflows split into training and 200 test should yield 3,517 training workflows, and with three generated samples per workflow (Qwen-Max, Claude 3.5, GPT-4o) the expected counts are 10,551 training and 600 test samples, not 11,140. Please clarify the sample-generation process.","section":"Sec. 3.2"},{"comment":"The implementation details report the main hyperparameters, but the paper does not state the exact RL training data size, the inference prompt template, or whether the model and training code will be released. Please provide a repository link or state the release plan for reproducibility.","section":"Sec. 4.3"},{"comment":"The qualitative claims that ComfyUI-R1's outputs 'adhere' to style and 'seamlessly combine' images are subjective; adding a small quantitative metric, such as CLIP similarity of generated images to the instruction, would strengthen the case study.","section":"Sec. 4.6, Fig. 3-5"}],"recommendation":"major_revision","confidential_remarks":"The main uncertainty is whether the end-to-end claim can survive a controlled comparison; the manuscript's own disclosure that baseline numbers are borrowed is a serious gap for a venue paper, but it is fixable with additional experiments and an overlap analysis. The reward issue is also fixable within the scope of a revision. I do not see a fundamental unsoundness in the central training pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short take: ComfyUI-R1 is a serious, clearly written application of RL post-training to a real creative tool, and the two-stage recipe—CoT SFT for cold start then GRPO with a rule-metric hybrid reward—is the right shape for this problem. The main test-set results are internally consistent and the ablations are informative. But the headline ComfyBench number (0.67 vs 0.56) is not a controlled comparison as reported, and the node-accuracy reward is recall-only, which shows up in the precision numbers.\n\nWhat's new: first RL-trained reasoning model for ComfyUI workflow generation, with a code-based workflow representation and a reward that combines format validity, DAG structure, node fidelity, and node-selection accuracy. That reward design is the most interesting part. The paper is honest that the main evaluation is over pre-retrieved candidate nodes, which is a reasonable scoping choice. The SFT+GRPO gain over SFT-only, and code over JSON, are clean ablations.\n\nWhere it's soft: (1) ComfyBench numbers for GPT-4o baselines are taken from the original paper, while ComfyUI-R1 uses its own retrieval over a 3,917-workflow KB with no overlap check against ComfyBench tasks. The 11-point margin therefore conflates generator, retrieval, and environment. This needs to be fixed: rerun baselines under the same retrieval protocol, or at minimum report overlap stats. (2) The node-accuracy reward in Eq. (5) only rewards recall; it doesn't penalize extra valid nodes chosen from the candidate set. Table 1 reflects that: ComfyUI-R1's node precision is 0.67, below Claude 3.5 Sonnet's 0.70, while its recall is much higher. If the intended use allows a human to prune extra nodes, that's fine, but the paper should say so. (3) No code, data, or model weights are released, which limits reproducibility. The citation pattern is fair, covering ComfyBench, WorFEval, and the RL post-training literature.\n\nThese are all addressable rather than fatal. The scoped main test set is a limitation but is disclosed and internally consistent.\n\nThis paper is for the creative-AI tooling community and for people working on RL with structured outputs. A serious referee should engage with it; I'd send it to review, but with a request for code/data, a controlled ComfyBench comparison, and a discussion of the precision-recall tradeoff in the reward.","headline":"A solid RL-for-workflow application with honest scoping on the main test set, but the ComfyBench comparison is uncontrolled and the node reward ignores precision; deserves peer review with revisions.","tokens_in":17261,"tokens_out":5985,"would_cite":false,"duration_ms":55170,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7-billion-parameter model writes executable ComfyUI workflows at a 97 percent validity rate.","keywords":["workflow generation","ComfyUI","chain-of-thought reasoning","reinforcement learning","GRPO","node selection","multimodal content creation","reasoning model"],"falsifier":"Give the model the same 600 test instructions but remove the candidate-node menu entirely, or perturb Vcand so that some gold nodes are missing, and recompute format validity, node-level F1, and execution pass rate; if these collapse toward the few-shot baselines, the claimed reasoning advantage depends on the candidate-menu protocol.","tokens_in":16223,"feed_emoji":"🎨","tokens_out":7543,"duration_ms":68337,"temperature":0.7,"pith_summary":"The paper's thesis is that automated workflow generation for ComfyUI, turning a natural-language instruction into an executable graph of image- and video-generation nodes, does not require a powerful closed API; a 7B open-weights model can be post-trained to do it better. ComfyUI-R1 starts from Qwen2.5-Coder-7B and is first fine-tuned on long chain-of-thought traces that select nodes, state a design principle, and emit a Python-like code representation of the workflow. A reinforcement-learning stage with GRPO then shapes the reasoning, using a veto-based hybrid reward that penalizes format errors, invalid directed acyclic graphs, and hallucinated nodes, while rewarding correct node selection. The trained model reaches 97% format validity, node-level F1 of 0.62 and graph-level F1 of 0.51 on the authors' test set, above every GPT-4o and Claude prompting baseline, and raises ComfyBench pass rate to 0.67 from ComfyAgent's 0.56. If these results hold, they suggest that reasoning-model training can transfer from math to structured creative tasks with only a well-shaped reward.","feed_headline":"7B model writes ComfyUI workflows at 97% validity","feed_subtitle":"Trained with chain-of-thought and reinforcement learning, the open 7B model beats GPT-4o and Claude.","key_machinery":"The load-bearing machinery is two-stage post-training with a structured action space. Stage one is supervised fine-tuning on distilled long chain-of-thought data: each example asks the model to output a node-selection block, a design-principle rationale, and a workflow expressed as Python-like function calls in topological order, reversibly convertible to ComfyUI JSON. Stage two is Group Relative Policy Optimization (GRPO), a reinforcement-learning method that samples a group of outputs for the same input and assigns each an advantage from its reward relative to the group. The reward used in training is a veto-based hybrid: format reward checks that all required tags parse, structure reward checks that the workflow forms a valid directed acyclic graph, node-fidelity reward penalizes nodes outside the candidate set or inconsistent between the selection block and the code, and only if all three pass is a precision-style node-selection reward applied; any single veto sets total reward to -1. This design is what lets the model maximize reasoning quality while hard constraints remain non-negotiable.","core_discovery":"The central claim, stated on the authors' terms, is that ComfyUI-R1 is the first large reasoning model for automated workflow generation and that its two-stage recipe, cold-start CoT supervised fine-tuning followed by GRPO reinforcement learning with a fine-grained rule-metric hybrid reward, is what enables a 7B parameter model to outperform prior state-of-the-art systems built on GPT-4o and Claude. On the in-house test set, format validity reaches 97% versus 41% for the untrained backbone, while node-level F1 is 0.62 and graph-level F1 is 0.51, both above all baselines. The ablation shows that the RL stage improves format validity from 95% to 97%, and that representing workflows as Python-like code outperforms representing them as JSON. On ComfyBench, which tests end-to-end retrieval and generation without a supplied candidate node set, the model achieves a 0.67 pass rate, an absolute 11% gain over the GPT-4o-based ComfyAgent.","pith_inferences":["The main test set's candidate-node protocol, which injects the gold nodes plus a small random decoy set, isolates node selection from retrieval; a reader should treat the ComfyBench number as the more realistic estimate of deployed performance, and even 0.67 likely degrades as the node space grows.","The same SFT-plus-GRPO recipe with a code-level DAG action space should transfer to other low-code creative tools, such as Blender geometry nodes or video-editing graphs, where the hard constraints are format, graph validity, and node fidelity.","A cheap testable extension is to vary the number of random candidates in the protocol: if performance holds when decoys are many or when gold nodes are partly withheld, retrieval is not the bottleneck; if it collapses, retrieval becomes the priority."],"forward_implications":["A 7B open-weights model can beat prompt-only systems powered by GPT-4o and Claude 3.5/3.7 on a structured generation task, so workflow-generation capability is trainable rather than dependent on API-scale models.","With 97% format validity, the model's outputs can almost always be parsed into executable ComfyUI graphs, making workflow generation usable as a component in a copilot rather than a suggestion tool.","The code-representation advantage over JSON indicates that a structured, semantically rich action space helps the model both plan and conform to graph constraints.","The veto-based hybrid reward provides a transferable recipe: enforce hard constraints as binary gates and let reinforcement learning optimize only the soft quality metric inside those gates.","The 0.67 ComfyBench pass rate means a majority of generated workflows actually execute on the ComfyUI server, which is the strongest evidence that the benefit reaches real retrieval-plus-generation use."],"supporting_citations":[{"why":"Defines the ComfyUI platform whose graph workflows and JSON/canvas format are the target of generation.","marker":"[comfyanonymous 2023]"},{"why":"Supplies ComfyBench, the end-to-end benchmark, and the ComfyAgent baseline whose 0.56 pass rate is beaten.","marker":"[Xue et al. 2024]"},{"why":"Introduces GRPO, the reinforcement-learning algorithm used in the RL stage.","marker":"[Shao et al. 2024]"},{"why":"Demonstrates that RL can incentivize reasoning in LLMs, the motivation for two-stage SFT plus RL.","marker":"[DeepSeek-AI 2025]"},{"why":"Is the Qwen2.5-Coder technical report for the 7B backbone model.","marker":"[Hui et al. 2024]"},{"why":"Supplies WorFEval's LIS/MCIS-based evaluation metrics for node- and graph-level precision, recall, and F1.","marker":"[Qiao et al. 2025]"},{"why":"Provides the JSON workflow representation baseline used in the ablation and the ComfyGPT multi-agent context.","marker":"[Huang et al. 2025]"}],"fun_headline_variants":["7B reasoning model writes ComfyUI workflows at 97% validity","ComfyUI-R1: 7B model beats GPT-4o and Claude on workflows","First reasoning model to auto-generate ComfyUI workflows","Two-stage training makes 7B model a ComfyUI workflow expert","Code-based workflows in ComfyUI-R1 lift pass rate 11% over GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation protocol, which hands the model a candidate node list that already contains every correct node plus a small random set of decoys, fairly represents how users will actually get node suggestions; if retrieval errors or the full-size node space make the real candidate list less helpful, the reported gains shrink.","fun_headline_variants_meta":{"raw":{"variants":["7B reasoning model writes ComfyUI workflows at 97% validity","ComfyUI-R1: 7B model beats GPT-4o and Claude on workflows","First reasoning model to auto-generate ComfyUI workflows","Two-stage training makes 7B model a ComfyUI workflow expert","Code-based workflows in ComfyUI-R1 lift pass rate 11% over GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1685,"prompt_tokens":1012,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":628,"tokens_out":673,"duration_ms":7377,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:40:35.006815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the model the same 600 test instructions but remove the candidate-node menu entirely, or perturb Vcand so that some gold nodes are missing, and recompute format validity, node-level F1, and execution pass rate; if these collapse toward the few-shot baselines, the claimed reasoning advantage depends on the candidate-menu protocol.","supporting_citations":[],"review_version":1}