{"id":"0bc3b2e9-814a-472c-bd04-fa26b16a5736","arxiv_id":"2506.05010","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-powered multi-agent Copilot retrieves and constructs ComfyUI workflows, reporting at least 88.5% recall on its own test set and 85.9% online acceptance of proposed workflows.","lead":"ComfyUI-Copilot is a large-language-model plugin for ComfyUI that recommends nodes and models and builds image/video generation workflows from a user's text request. The authors report offline recall above 88% and online acceptance of 85.9% for proposed workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline recall is measured on instructions rewritten from the system's own knowledge base, so the 88.5% figure may not transfer to real user requests.","rationale":"The reader's weakest assumption is precisely that the offline evaluation uses test instructions rewritten from the system's own knowledge base, with the original workflow as the correct answer. This is the same load-bearing concern I identified. The paper's headline numbers (88.5% recall) are the main quantitative evidence for the system's usefulness, and the evaluation's construction makes them an upper bound on an easy paraphrase-retrieval task rather than a measure of real-world performance. The online acceptance metrics, while valuable, lack detail on measurement, making them unverifiable. The most direct way to settle the concern is to test on external, real user queries with human ground truth. If that test reproduces the high recall, the central claim stands; if not, the paper's performance claims are overstated. Given that the reader already assigned CONDITIONAL and my concern does not point to a fundamental architectural flaw, the verdict should remain CONDITIONAL: the paper is acceptable only if the authors provide external validation or clearly condition their claims on the KB-rewritten test distribution.","tokens_in":786,"tokens_out":689,"duration_ms":58144,"concrete_test":"Sample 200 real user queries from the deployed system's logs (or from ComfyUI GitHub issues), have two expert ComfyUI users independently mark the correct workflow/node (or all acceptable ones), run ComfyUI-Copilot on these queries, and measure top-3 recall. If recall falls more than 10 percentage points below 88.5%, the reported numbers overstate real-world accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on Table 1: top-3 recall of 88.5%+ for nodes and workflows. These numbers come from 130 workflow and 104 node instructions that the authors created by rewriting the usage descriptions of entries already in the KB (Sec. 4). The retrieval pipeline searches exactly those KB descriptions, so the test measures paraphrase detection on in-distribution text, not functional understanding of novel requests. Real user queries from the deployed system (85K queries) likely mention specific models, visual attributes, or tasks in ways not represented by the KB's usage descriptions, so out-of-distribution recall may be substantially lower. The online acceptance rates (65.4% nodes, 85.9% workflows) could partially counterbalance this, but no methodology is given for how acceptance was recorded, what denominator was used, or how self-selection was handled, so they cannot be independently checked. Thus the claim that the system 'accurately recommends nodes and constructs workflows' is supported mainly by a self-referential benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ComfyUI-Copilot, an LLM-powered plugin for ComfyUI that provides automated workflow construction, node and model recommendation, and ComfyUI-related question answering. The system uses a hierarchical multi-agent framework with a central assistant agent and specialized worker agents, backed by curated knowledge bases of 7K nodes, 62K models, and 9K workflows. The authors report offline top-3 recall rates above 88.5% for nodes and workflows, online user acceptance rates of 65.4% for recommended nodes and 85.9% for proposed workflows, and a fine-tuned Qwen2.5-Coder-7B that achieves node-selection F1 of 0.95. The paper claims this is the first open-source ComfyUI assistant plugin and that it lowers entry barriers for beginners while enhancing efficiency for experienced users.","tokens_in":10396,"tokens_out":4240,"duration_ms":50312,"significance":"If the reported results hold, ComfyUI-Copilot is a practically valuable open-source tool: it addresses a real user pain point, ships with a large and continuously updated knowledge base, and demonstrates a workable multi-agent retrieval and generation pipeline. The paper's strengths include the detailed system description, the automatic documentation generation pipeline, the release of code and a demo, and the deployment evidence of 85K queries from 19K users across 22 countries. However, the central quantitative claims rest on a self-constructed offline test set whose instructions are rewritten from the system's own KB descriptions, and on online acceptance numbers without stated methodology. For a system paper, this is a meaningful gap but one that can be addressed with additional evaluation, so the contribution is potentially acceptable after substantial revision.","major_comments":[{"comment":"The offline test set is built from 130 workflow and 104 node instructions created by rewriting the usage descriptions of the system's own KB entries. Because the retrieval pipeline searches exactly those KB descriptions, the reported recall rates (88.5-90.0%) largely measure the system's ability to match a paraphrase of its own data. This does not establish the claimed 'accurate recommendation' for out-of-distribution or ambiguous real-world prompts, so the central conclusion is only weakly supported.","section":"Sec. 4, Table 1"},{"comment":"The manuscript reports acceptance rates of 65.4% for recommended nodes and 85.9% for workflows, but it does not define the acceptance event (e.g., clicking 'Accept' vs. manually using the suggestion), the denominator, the collection period, the treatment of queries for which no recommendation was made, or the effect of user self-selection. Without this methodology, these numbers cannot be interpreted or reproduced, and they should not be presented as independent validation.","section":"Sec. 4, online user feedback"},{"comment":"The retrieval pipeline has several free parameters—the 0.7/0.3 fusion weights, the top-30 candidate pool, the top-3 re-ranking, and the popularity factors—but no sensitivity analysis is provided. The reported recall rates have no confidence intervals or variance across the test instructions. The paper should report error bars and vary these parameters to show the result is not brittle.","section":"Sec. 3.2, Eq. (1); Sec. 4"},{"comment":"The node-selection F1 metric is computed against a single golden workflow, yet many valid workflows can satisfy the same user instruction. This conflates semantic equivalence with exact node overlap, and the F1=0.95 for the fine-tuned model is reported on only 100 examples without confidence intervals. The authors should either use multiple references, provide human equivalence judgments, or explicitly acknowledge the limitation and show that the metric is stable.","section":"Appendix B, Table 2"}],"minor_comments":[{"comment":"The introduction states that the knowledge bases are 'continuously expanded and updated daily,' while Section 3.1 says they are 'continuously expanded weekly.' These timelines should be reconciled.","section":"Sec. 1 vs. Sec. 3.1"},{"comment":"The URL for the GTE-Rerank model appears broken or malformed in the manuscript ('https://huggingface.co/Alibaba-NLP/ gte-multilingual-reranker-base'); it should be a single, clickable link.","section":"Sec. 3.2, footnote 1"},{"comment":"ComfyBench (Xue et al., 2024) is cited in the related work but is not used for comparison. Given that it is a public benchmark for LLM-based ComfyUI agents, the authors should at least explain why it is not applicable to their evaluation.","section":"Sec. 2 and Sec. 4"},{"comment":"The error analysis of unsuccessful workflow cases is described as showing that suggested workflows 'often still fulfill the user's intended functions,' but no numbers are given; quantifying this would strengthen the qualitative claim.","section":"Sec. 4"},{"comment":"The phrase 'the first open-source project for a ComfyUI assistant plugin' is a strong novelty claim; it would be safer to state the novelty relative to the specific capabilities or to cite a more systematic survey of existing plugins.","section":"Sec. 1 and Sec. 5"},{"comment":"The caption of Figure 1 describes 'one-click deployment,' but the mechanism by which a suggested workflow is loaded and any missing nodes installed should be stated more precisely in the text to avoid ambiguity about user effort.","section":"Sec. 3.3 and Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is better framed as a system demonstration than a novel research method. The core engineering contribution—a large curated KB plus a multi-agent retrieval/generation pipeline—is real, but the evaluation currently leans on a self-referential offline set and under-specified online metrics. If the authors add an external benchmark, provide a clear acceptance methodology with confidence intervals, and address the parameter sensitivity, the paper would be suitable for publication. I would not reject outright because the tool is open-sourced and the deployment data suggest practical utility, but the current quantitative claims are not yet credible as reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid engineering paper about a genuinely useful tool. It combines known pieces—hierarchical agents, retrieval, reranking, fine-tuned code LLM—into an open-source plugin that people actually use. That's real. The system extends beyond the text-to-image focus of prior work, and the GitHub traction suggests it fills a gap.\n\nThe best part is that they shipped it. The paper includes a fine-tuning experiment (Qwen2.5-Coder-7B) with pass rates and node-level F1, and the code and data appear to be available. That alone makes this worth reading for anyone building similar assistants.\n\nThe soft spots are in the evaluation. The offline recall numbers (88.5%+) come from instructions that are rewrites of usage descriptions in the same KB the system retrieves from. That measures paraphrase detection, not functional understanding of novel requests. The online acceptance rates (65.4% nodes, 85.9% workflows) could be meaningful, but the paper gives no methodology: how acceptances are counted, what the denominator is, or how self-selection is handled. So the headline numbers are engineering signals, not verified results.\n\nHaving said that, the fine-tuning experiment is more credible: it uses 100 held-out workflows, reports pass rates and node-level precision/recall/F1, and shows the open model competing with Claude. That part holds up.\n\nThe citation pattern is fine. They cite ComfyGen, ComfyBench, and other relevant work. The claims about being first are modest enough.\n\nOverall, this paper deserves a serious referee. It's not a breakthrough, but it's a real system with real usage and a plausible evaluation that could be strengthened. I'd recommend acceptance with revisions, mainly asking for a clearer online evaluation methodology and ideally a comparison or acknowledgment of ComfyBench as an external benchmark. The reader's CONDITIONAL verdict is fair.","headline":"A useful open-source ComfyUI assistant with a plausible architecture and real deployment, but the headline recall numbers come from a test set built from the system's own knowledge base.","tokens_in":10980,"tokens_out":1687,"would_cite":false,"duration_ms":19436,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ComfyUI-Copilot, an LLM-powered multi-agent plugin, claims to automate ComfyUI workflow construction, achieving over 88.5% recall for node and workflow recommendations and online acceptance rates of 65.4% for nodes and 85.9% for workflows.","keywords":["ComfyUI","large language model","multi-agent system","workflow generation","node recommendation","knowledge base","text-to-image generation","AI art"],"falsifier":"Run the plugin on a set of user requests collected from ComfyUI community forums and GitHub issues that were written before the system existed, where the correct workflow is not in the system's knowledge base, and check whether node and workflow acceptance remains near the reported 65.4% and 85.9%.","tokens_in":10020,"feed_emoji":"🎨","tokens_out":8780,"duration_ms":82555,"temperature":0.7,"pith_summary":"This paper presents ComfyUI-Copilot, a plugin that uses a hierarchy of language-model agents to turn plain-language instructions into ready-to-run ComfyUI workflows. The central claim is that the system can accurately recommend nodes and models and assemble complete workflows, with offline recall above 88.5% for both nodes and workflows and online acceptance of 65.4% for node suggestions and 85.9% for workflow suggestions. A curated knowledge base of 7K nodes, 62K models, and 9K workflows feeds a three-stage retrieve-and-rerank pipeline, and workflow generation also works by writing code. The authors argue this lowers the entry barrier for beginners while adding debugging and parameter-search features for experienced users. If correct, the system extends automated workflow construction beyond text-to-image to conditional image and video generation tasks.","feed_headline":"AI assistant auto-builds ComfyUI workflows, 85.9% accepted","feed_subtitle":"Multi-agent plugin turns plain instructions into ready-to-run workflows, with node recall above 88.5 percent.","key_machinery":"The load-bearing mechanism is the hierarchical multi-agent framework: an LLM-based assistant agent that either answers directly or delegates to worker agents, supported by three knowledge bases covering nodes, models, and workflows. All recommendations pass through a three-stage coarse-to-fine pipeline: intent expansion by an LLM or large multimodal model, hybrid scoring that combines a semantic embedding score (weight 0.7) with a lexical overlap score (weight 0.3) to shortlist 30 candidates, and a reranker plus popularity weighting to select the top 3. For workflow generation, ComfyUI graphs are converted between JSON and Python-like code, so the LLM can write a workflow as code with retrieved exemplars. The knowledge bases are themselves produced by an automated pipeline that runs node repositories in a sandbox, extracts metadata, embeds code with BGE-M3, and generates documentation with an LLM.","core_discovery":"The discovery is a system design: a central assistant agent that plans and delegates, three specialized worker agents (workflow generation, node recommendation, model recommendation), and knowledge bases refreshed from community sources. When a user gives an instruction, the system expands the intent with an LLM, scores candidates by a weighted hybrid of semantic and lexical similarity, reranks the top candidates, and presents the top three workflows or nodes for one-click acceptance. In offline tests on instructions rewritten from the system's own knowledge-base usage descriptions, recall of the top three exceeds 88.5% for both workflows and nodes with GPT-4o and DeepSeek-V3. Online, users accepted 85.9% of proposed workflows and 65.4% of recommended nodes. The paper also reports that a fine-tuned 7B open-source code model reaches a 0.95 node-selection F1, comparable to Claude-3.7-Sonnet, although generated workflow pass rates remain improvable.","pith_inferences":["Editorially, an ablation that removes the reranker stage would isolate how much of the reported 88.5% recall comes from the hybrid scoring alone, a comparison the paper does not report.","The dependency-aware recommendation logic, such as pairing LoRA models with compatible diffusion checkpoints, suggests a general principle for node-based toolkits: recommending compatible subgraphs may matter more than recommending individual components, a pattern other modular creative tools could adopt.","If given access to the system telemetry, one could test whether the 65.4% node acceptance rate is driven mainly by unfamiliar users accepting default suggestions; sorting acceptances by user experience level would show whether the assistant helps novices more than experts."],"forward_implications":["A user could state an image or video task in natural language and load a working workflow onto the canvas in one click, without manually wiring nodes or installing dependencies.","The system's coverage of conditional image and video generation tasks means automated workflow construction extends beyond the text-to-image setting of prior work.","Because a fine-tuned 7B open-source model matches a much larger closed-source model on node-selection F1, the workflow-generation backend could be run locally or at lower cost.","The 85.9% online acceptance of proposed workflows suggests that retrieval from a large curated knowledge base is a reliable default, with generative construction as a supplement."],"supporting_citations":[{"why":"Defines the ComfyUI platform and its JSON workflow format that the plugin targets.","marker":"comfyanonymous, 2023"},{"why":"Supplies the JSON-to-code conversion convention and the benchmark framing for LLM-based ComfyUI agents.","marker":"Xue et al., 2024"},{"why":"Prior prompt-adaptive workflow generation for text-to-image that the paper extends to broader multimodal tasks.","marker":"Gal et al., 2024"},{"why":"Prior automatic workflow improvement work that motivates the debugging and optimization features.","marker":"Sobania et al., 2024"},{"why":"The Qwen2.5-Coder-7B model that the paper fine-tunes for workflow generation.","marker":"Hui et al., 2024"},{"why":"The fine-tuning framework used to train the workflow-generation model.","marker":"Zheng et al., 2024"},{"why":"BGE-M3, the embedding model used for code retrieval in automatic node documentation generation.","marker":"Chen et al., 2024a"}],"fun_headline_variants":["Meet ComfyUI-Copilot: AI agent that builds workflows for you","LLM assistant with multi-agent design automates ComfyUI workflows","ComfyUI-Copilot: one-click workflows from plain instructions","AI copilot for ComfyUI: node recall over 88.5%, 85.9% accepted","Beginner-friendly AI plugin crafts ComfyUI workflows automatically"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The offline evaluation assumes that instructions formed by rewriting usage descriptions from the system's own knowledge base, with the original workflow as the ground truth, are representative of real user requests, so the 88.5% recall may not hold for unseen or ambiguously phrased prompts.","fun_headline_variants_meta":{"raw":{"variants":["Meet ComfyUI-Copilot: AI agent that builds workflows for you","LLM assistant with multi-agent design automates ComfyUI workflows","ComfyUI-Copilot: one-click workflows from plain instructions","AI copilot for ComfyUI: node recall over 88.5%, 85.9% accepted","Beginner-friendly AI plugin crafts ComfyUI workflows automatically"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3511,"prompt_tokens":947,"completion_tokens":2564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2464}},"tokens_in":563,"tokens_out":2564,"duration_ms":19994,"temperature":1.0,"reasoning_tokens":2464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:27:35.773906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the plugin on a set of user requests collected from ComfyUI community forums and GitHub issues that were written before the system existed, where the correct workflow is not in the system's knowledge base, and check whether node and workflow acceptance remains near the reported 65.4% and 85.9%.","supporting_citations":[],"review_version":1}