Pith. sign in

REVIEW 4 major objections 7 minor 46 references

By distilling expert workflows into strategies and pseudo-code, a knowledge-centric LLM agent generates executable ComfyUI graphs at 86.9% Pass, versus 36.4% for the strongest prior baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:07 UTC pith:SP7ZUBBV

load-bearing objection Genuinely interesting pipeline for ComfyUI workflow generation, but the headline Pass/Resolve numbers rest on a circular LLM-inferred evaluation that needs major revision before they can be taken at face value. the 4 major comments →

arxiv 2607.15845 v2 pith:SP7ZUBBV submitted 2026-07-17 cs.AI

Knowledge-Centric Agents for Workflow Generation in ComfyUI

Zhendong Li , Lei Sun , Ruibo Ming , He Zhang , Danda Pani Paudel , Luc Van Gool , Jinjin Gu This is my paper
classification cs.AI
keywords workflow generationComfyUIknowledge inversionsupervised fine-tuningLLM agentshierarchical reasoninggraph generationvisual programming
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that generating ComfyUI workflows—node graphs that orchestrate image-generation pipelines—requires more than translating text to JSON; it requires the kind of experiential, multi-level design knowledge that human experts use. To supply that knowledge, the authors propose a pipeline that first inverts unannotated real-world workflows into a hierarchy (full pseudo-code → skeleton pseudo-code → strategy reasoning → high-level strategy → inferred task instruction), then fine-tunes a language model to reason from task to strategy and strategy to executable structure, and finally runs the reverse process at inference to synthesize workflows with self-refinement. The central claim is that this knowledge-centric approach yields workflows with greater node diversity, structural coherence, and execution success than existing LLM prompting, retrieval, and template-based methods. If correct, it suggests that the bottleneck in automated workflow design is not syntax but the recovery and reuse of implicit design knowledge.

Core claim

The paper's core discovery is that unannotated ComfyUI workflows contain recoverable design knowledge that can be made explicit through a five-level inversion ladder: raw graphs are regularized into full pseudo-code, stripped of parameters to form skeleton pseudo-code, interpreted into strategy-reasoning traces, summarized into high-level strategy descriptions, and finally aligned to inferred task instructions. This inverted knowledge is then injected via supervised fine-tuning into two models—a strategy planner and a pseudo-code generator—while parameter filling and graph reconstruction are handled by rules. At inference, the agent runs the inversion in reverse, optionally refining its own

What carries the argument

The central mechanism is the hierarchical knowledge representation distilled from raw workflows: Full Pseudo-code (complete module names, connections, parameters), Skeleton Pseudo-code (topology with parameters removed), Strategy Reasoning traces (explanations of why the graph is arranged as it is), and High-level Strategy descriptions (the expert's processing plan). The paper's innovation is the symmetry between inversion (workflow → knowledge) and inference (knowledge → workflow), with the two SFT-trained models—High-Level Structure Planner and Pseudo-code Generator—carrying the reasoning steps that cannot be rule-encoded. This machinery turns workflow generation from a brittle direct text

Load-bearing premise

The load-bearing premise is that the task instructions, strategies, and reasoning traces used for training, which are inferred by a language model from raw workflows, faithfully represent what human experts actually intended; if those inferred labels are wrong, the model learns synthetic patterns and the reported task-consistency scores are inflated.

What would settle it

Run an ablation that replaces the language-model-inferred task instructions and strategies with human expert annotations on the same training corpus and test set; if Pass/Resolve or TaskCons changes materially (e.g., drops more than a few points), the inferred labels are not faithful. Alternatively, invite ComfyUI power users to rate the 'Resolve' outputs and the inferred workflow rationale against the original user intent for the real-world cases.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Users could give natural-language instructions to ComfyUI and receive executable, task-specific workflows directly, without manual node wiring.
  • The reported gains on FlowBench and ComfyBench indicate the distilled knowledge transfers to out-of-distribution tasks, not just memorized training workflows.
  • Only two learned transitions (task→strategy, strategy→skeleton) require fine-tuning; parameter filling and graph reconstruction stay rule-based, so the approach is sample-efficient and easier to update than end-to-end generators.
  • The auxiliary strategy-reasoning objective regularizes the model and improves interpretability and generalization, suggesting that reasoning traces are a useful supervision signal beyond the main task.
  • The comparison with Qwen3-32B Q2P shows that multi-level supervision with a smaller 14B model can beat a larger end-to-end model, implying that structural supervision is more important than scale for structured graph generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same inversion ladder to other node-based visual programming tools or to scientific workflow systems, where executable graphs exist but expert annotations are rare.
  • Because the paper's gain over the direct end-to-end 32B baseline is large, the hierarchical decomposition itself—not the backbone model—appears to be the decisive factor; this predicts that scaling the backbone further while keeping the hierarchy would yield diminishing returns compared to enriching the distilled knowledge.
  • A testable extension: augmenting the inversion step with human-verified annotations for a small subset of high-value workflows could correct label bias and improve rare-node substitution, which the paper identifies as a limitation.
  • The framework implies that workflow generation benefits from explicit reasoning traces, so combining the SFT-injected knowledge with reinforcement-style optimization on execution feedback (e.g., maximizing Pass rate) might close the remaining gap to human-designed workflows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a knowledge-centric framework for ComfyUI workflow generation. It first performs "knowledge inversion" on a large crawl of real workflows, converting raw graphs into hierarchical representations (full pseudo-code, skeleton pseudo-code, strategy reasoning traces, high-level strategies, and inferred task instructions). It then injects this knowledge via supervised fine-tuning of Qwen3-14B on task→strategy and strategy→pseudo-code pairs, with an auxiliary strategy-reasoning objective. At inference, the model reverses the pipeline: strategy planning, pseudo-code generation, parameter filling, and rule-based reconstruction, with self-refinement at two stages. Experiments compare against GPT-5-based zero-shot, few-shot, CoT, CoT-SC, RAG, and ComfyAgent baselines on an internal 30-task test set, a 20-task ComfyBench subset, and a 30-task FlowBench subset, reporting large gains on VND, NodeComp, LinkComp, TaskCons, Pass, and Resolve.

Significance. The core idea is well motivated: instead of direct text-to-JSON generation, the paper explicitly models multiple abstraction levels of workflow design knowledge, which is a plausible path toward more robust and interpretable workflow generation. The paper ships a clear methodology, a relatively large workflow collection, evaluations on two external benchmarks in addition to the internal set, a dual-LLM-judge consistency check for TaskCons, and qualitative case studies that suggest meaningful structural differences from template-based baselines. If the results hold, the work could provide a useful foundation for knowledge-driven agentic workflow generation. However, the current evaluation has several load-bearing problems — most notably a circularity between LLM-inferred training task labels and the internal test queries, an internal contradiction about whether LLM judges are used, and a suspicious exact identity between two purportedly distinct structural metrics — that prevent full confidence in the headline numbers without additional evidence.

major comments (4)
  1. [§3.2 and §4.1 (Tables 1–2)] The in-house evaluation exhibits a circularity in task supervision. The last bullet of §3.2 states that task instructions for unannotated workflows are produced by "decoding plausible task descriptions" via LLMs. §4.1 then says the 30 test workflows "and their corresponding task queries" come from the same curated pool. Thus the SFT model is trained to reverse the LLM's inversion, and it is evaluated on tasks generated by that same inversion process. TaskCons and the Resolve component of Table 2 therefore measure agreement with a synthetic instruction distribution, not with genuine user requests. This is load-bearing for the abstract's "higher execution success rates" claim. Please provide human-written instructions for at least the 30 internal test workflows (or a separate human-annotated test set), and report agreement between LLM-inferred and human instructions. Alternatively, use a b
  2. [§4.2 and Table 5] There is a direct internal contradiction about LLM judges. §4.2 first defines TaskCons as scored by "a GPT-based evaluator," but two paragraphs later states "we do not rely on LLM-based judges. Instead, all workflows are executed manually in ComfyUI and the outputs are assessed through human inspection." Table 5 then reports TaskCons from both GPT-5 and Gemini-2.5. If TaskCons is an LLM score, the "no LLM judges" statement is false; if it is human-only, the numbers in Table 5 are not TaskCons as used in Tables 1–4. In addition, the manual Pass/Resolve protocol is described in one sentence with no rubric, no blinding, and no inter-rater agreement. Please clarify which metrics used which evaluation mode, provide the human-evaluation rubric, and report inter-annotator agreement or at least a second annotator on a subset.
  3. [Tables 1, 3, 4] NodeComp and LinkComp are reported as exactly identical for every baseline on every benchmark: Table 1 lists 36.7/36.7, 33.3/33.3, 36.7/36.7, 56.7/56.7, 56.7/56.7; Table 3 lists 35/35, 25/25, 20/20, 60/60, 65/65; Table 4 lists 46.67/46.67, 43.33/43.33, 40/40, 73.33/73.33, 53.33/53.33. The definitions in §4.2 are conceptually distinct: NodeComp measures valid instantiated nodes that are linked, while LinkComp measures connection slot correctness. Exact equality across all baselines and benchmarks strongly suggests the two metrics are implemented as the same quantity or one is derived from the other. Please report the exact computation, provide an example where the two metrics can diverge, and correct the tables. Without this, the "more accurate connectivity" claim is unsupported.
  4. [§4.1, Tables 1–4] No uncertainty quantification or reproducibility artifacts are provided. All headline results are based on 30 internal tasks and 20–30 external tasks, with no standard deviations, repeated sampling, or significance tests. Since LLM generation is stochastic and SFT involves randomness, the reader cannot judge whether the reported margins (e.g., 86.9% vs. 36.4% Pass in Table 2) are robust to seed or temperature. Moreover, no code, data, or evaluation scripts are released, and no contamination analysis is reported between the external ComfyBench/FlowBench subsets and the 10k+ crawled workflows. Please report results over multiple runs, provide release plans or artifacts, and document an overlap analysis between external test tasks and the training corpus.
minor comments (7)
  1. [Figure 1 caption] The caption contains a duplicated phrase: "During inference, During inference, the agent executes..." Please fix.
  2. [§4.1 vs. Supplementary §C.1] The main text says 882 workflows were used for training and 30 for testing (total 912), but Supplementary §C.1 reports 879 training items. This inconsistency should be reconciled.
  3. [Tables 2 and 4] Table 2 uses "Resolve" while Table 4 uses "Solve" for the same concept. Please unify the terminology.
  4. [§4.3 FlowBench paragraph] The sentence "Our approach consistently outperforms all all the other methods" contains a typo ("all all").
  5. [§4.1] ComfyBench is described as "the only open-source baseline currently available," yet it is also used as an evaluation benchmark. Please clarify which specific methods are being compared and whether ComfyAgent is a distinct baseline or part of the ComfyBench suite.
  6. [§4.1 and §3.2] The structural deduplication threshold (10%), complexity bounds (6 and 90 nodes), and the GPT-4o semantic-equivalence threshold are given without sensitivity analysis. A brief discussion of their effect on the curated dataset would help.
  7. [Table 6] The Qwen3-32B Q2P end-to-end baseline is not described in enough detail. Please provide its training data, prompt format, and any hyperparameter settings so the comparison is reproducible.

Circularity Check

2 steps flagged

In-house test re-uses LLM-inverted task queries and LLM judges, making the headline comparison partially circular despite external benchmarks.

specific steps
  1. self definitional [§3.2 'To task instruction and inputs'; §4.1 'Workflow Collection'; §4.2 'Evaluation Metrics']
    "Since most collected workflows lack explicit user prompts, we further infer the corresponding task instruction and input specifications required for training. This is achieved by aligning the generated high-level strategy with retrieved exemplars and by decoding plausible task descriptions consistent with the structural and semantic context of the workflow. … Among them, 882 workflows were used for training, while 30 workflows and their corresponding task queries were reserved for testing."

    The test queries are not independent user instructions: they are produced by the same LLM-based inversion (§3.2) that generates the training pairs from raw workflows, and the test workflows are the ground-truth targets from which the queries were inverted. SFT teaches task→strategy→pseudo-code on these inferred labels; at test the model receives another inverted label and is scored against the source workflow. The in-house TaskCons/Resolve comparison therefore measures how well the model reverses the inversion on a held-out slice of the same synthetic label distribution, not generalization to independently stated intent. This is a fitted input called a prediction; only FlowBench/ComfyBench supply non-inverted task inputs.

  2. other [§4.2 'Evaluation Metrics' vs. §4.3 'LLM Judge Bias Analysis' (Table 5)]
    "To ensure stable and unbiased evaluation, we do not rely on LLM-based judges. Instead, all workflows are executed manually in ComfyUI and the outputs are assessed through human inspection. … Table 5: Task consistency (%) evaluated by two LLM judges (GPT-5 and Gemini-2.5)."

    The paper claims human-only evaluation, but Table 5 and §4.3 report TaskCons scores from GPT-5 and Gemini-2.5. Since the task instructions themselves are LLM-inverted (§3.2) and the judge is an LLM, the headline TaskCons metric is an LLM judging whether an LLM-inferred task was satisfied by an LLM-generated workflow; it does not provide the independent human grounding claimed in §4.2. This is a load-bearing evaluation step because TaskCons is one of the four headline metrics.

full rationale

The paper's central in-house comparison is partially circular by construction: the task instructions used for both training and testing are inferred from the raw workflows via the same LLM-inversion pipeline (§3.2). The model is therefore trained to reverse that inversion, and the test 'queries' are derived from the very workflows that serve as ground truth. This does not reduce the whole paper to circularity, because FlowBench and ComfyBench provide externally sourced task inputs and the Pass metric is checked by actual ComfyUI execution. However, those external sets are small (30 and 20 tasks), no overlap/contamination analysis with the 10k crawled workflows is reported, and the TaskCons metric is explicitly assigned by LLM judges despite the text claiming otherwise. I find no load-bearing self-citation: the cited prior works do not appear to be by the present authors, and no uniqueness theorem or ansatz is smuggled in via self-citation. The score of 6 reflects the partial circularity of the main in-house evaluation while acknowledging the independent grounding provided by the external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or formal entities. Its load-bearing assumptions are about data quality, the fidelity of LLM-generated supervision, and the validity of the evaluation protocols. These assumptions are substantial because the entire approach depends on AI-generated labels and judgments.

free parameters (3)
  • Structural deduplication threshold (10%) = 10% differing nodes
    Chosen by hand in §4.1 to decide when two workflows are structurally similar; directly shapes the training and evaluation data.
  • Workflow complexity bounds (6 and 90 nodes) = 6–90
    Workflows with fewer than six or more than ninety nodes are excluded; affects dataset composition and generalization claims.
  • GPT-4o semantic equivalence threshold = not specified
    Used in §4.1 to merge near-duplicate workflows; no operational threshold or human agreement is reported.
axioms (5)
  • domain assumption The cleaned collection of 912 internet workflows, after GPT-based deduplication and filtering, represents high-quality expert design knowledge.
    All training and evaluation labels are derived from this pool; if the curation is flawed, the whole pipeline inherits the flaw (§4.1).
  • domain assumption LLM-generated pseudo-codes, skeletons, strategies, and task instructions faithfully reconstruct the original workflow author's intent.
    The SFT targets in §3.2 are produced by LLM interpretation rather than by human annotation, so learning depends on these inferred labels being correct.
  • domain assumption Qwen3-14B fine-tuned on the two SFT stages acquires generalizable workflow-design knowledge rather than memorizing the 882 training workflows.
    The system's success on the 30 held-out tasks and small external benchmarks is interpreted as generalization.
  • domain assumption The GPT-5/Gemini TaskCons judges and the undocumented 'human inspection' accurately measure task fulfillment.
    Task consistency is scored by LLM judges in §4.2, and execution outcomes are claimed to be manually assessed, but no protocol, rubric, or inter-annotator agreement is given.
  • domain assumption Rule-based reconstruction from full pseudo-code yields executable ComfyUI workflows without semantic errors.
    The final conversion is deterministic and not learned; its correctness is assumed in the Pass/Resolve metrics (§3.4).

pith-pipeline@v1.3.0-alltime-deepseek · 16186 in / 9512 out tokens · 100814 ms · 2026-08-01T22:07:51.586320+00:00 · methodology

0 comments
read the original abstract

Workflow generation in visual creation systems such as ComfyUI demands not only syntactic accuracy but also expert-level reasoning over modular compositions. Existing large language model (LLM) approaches often treat this as a direct text-to-JSON generation task, struggling with structural brittleness and lacking the experiential knowledge required for effective design. We argue that successful workflow generation requires modeling knowledge itself, including its structure, hierarchy, and reasoning dynamics. To this end, we propose a knowledge-centric framework that learns to invert, inject, and infer with knowledge across multiple abstraction levels. We first perform knowledge inversion to distill hierarchical representations, ranging from full pseudo-codes and skeletons to high-level strategies, from large collections of real-world workflows. We then conduct knowledge injection through supervised fine-tuning, teaching the model to reason from task descriptions to strategies and from strategies to executable structures. During inference, the model performs reversible reasoning to synthesize executable workflows, augmented by self-refinement for structural coherence. Extensive experiments demonstrate that our method produces workflows with richer node diversity, more coherent structures, and higher execution success rates than existing systems, establishing a new foundation for knowledge-driven, agentic workflow generation.

Figures

Figures reproduced from arXiv: 2607.15845 by Danda Pani Paudel, He Zhang, Jinjin Gu, Lei Sun, Luc Van Gool, Ruibo Ming, Zhendong Li.

Figure 1
Figure 1. Figure 1: Given a user task instruction, our agent learns to construct and execute a com￾plete ComfyUI workflow. Through knowledge inversion, it distills pseudo-code struc￾tures and workflow strategies from large collections of real workflows. During inference, During inference, the agent executes the generated workflow to produce a high-quality visual result. The example shows how the agent interprets a stylistic e… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our framework. The system consists of two symmetric phases. (1) Knowledge Inversion distills hierarchical representations from real ComfyUI workflows. (2) Knowledge Inference reverses this process to generate workflows from instructions via a strategy planner, pseudo-code generator, and rule-based reconstruction. controls to introduce, and how controls interact. This high-level strategy require… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison with baseline methods. Given a user instruction, our method ac￾curately adheres to the task description. these indicators assess both syntactic validity and the functional effectiveness of workflow generation. To ensure stable and unbiased evaluation, we do not rely on LLM-based judges. Instead, all workflows are executed manually in ComfyUI and the outputs are assessed through human inspection.… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of generated workflow structures. – Few-shot: Extends the zero-shot setting by providing several in-context exemplars, allowing the model to better understand expected formats and improve generation accuracy [3] [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task type distribution in the training dataset. Left: count of workflows con￾taining each task type (workflow-level distribution, showing the number of workflows with each task type). Right: proportion of each task type among all task type oc￾currences (occurrence-level distribution, accounting for multi-label classification where each workflow may contain multiple task types). text_to_image painting upsca… view at source ↗
Figure 6
Figure 6. Figure 6: Task type distribution in the test dataset. Left: count of workflows containing each task type (workflow-level distribution, 30 workflows total). Right: proportion of each task type among all task type occurrences (occurrence-level distribution, account￾ing for multi-label classification where each workflow may contain multiple task types) [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 20 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    In: Proceed- ings of the AAAI Conference on Artificial Intelligence

    Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Giani- nazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al.: Graph of thoughts: Solving elaborate problems with large language models. In: Proceed- ings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 17682–17690 (2024)

    2024

  3. [3]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    1901

  4. [4]

    arXiv preprint arXiv:2309.17288 (2023)

    Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B.F., Fu, J., Shi, Y.: Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288 (2023)

    Pith/arXiv arXiv 2023

  5. [5]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Chen, G.H., Chen, S., Liu, Z., Jiang, F., Wang, B.: Humans or llms as the judge? a study on judgement bias. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 8301–8327 (2024)

    2024

  6. [6]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Chen, J., Zhu, X., Wang, Y., Liu, T., Chen, X., Chen, Y., Leong, C.T., Ke, Y., Liu, J., Yuan, Y., et al.: Symbolic representation for any-to-any generative tasks. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 27816–27826 (2025)

    2025

  7. [7]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  8. [8]

    com / comfyanonymous / ComfyUI (2023)

    comfyanonymous: Comfyui.https : / / github . com / comfyanonymous / ComfyUI (2023)

    2023

  9. [9]

    Cursor: Cursor: Ai-powered code editor.https://www.cursor.com/en

  10. [10]

    In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186 (2019)

    2019

  11. [11]

    In: Ran- zato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. In: Ran- zato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 8780–8794. Curran Asso- ciates, Inc.(2021),https://proceedings.neurips.cc/paper_files/paper/2021/ file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf

    2021

  12. [12]

    In: International workshop on agent theories, architectures, and languages

    Franklin, S., Graesser, A.: Is it an agent, or just a program?: A taxonomy for autonomous agents. In: International workshop on agent theories, architectures, and languages. pp. 21–35. Springer (1996)

    1996

  13. [13]

    arXiv preprint arXiv:2410.01731 (2024) Knowledge-Centric Agents for Workflow Generation 17

    Gal, R., Haviv, A., Alaluf, Y., Bermano, A.H., Cohen-Or, D., Chechik, G.: Com- fygen: Prompt-adaptive workflows for text-to-image generation. arXiv preprint arXiv:2410.01731 (2024) Knowledge-Centric Agents for Workflow Generation 17

    Pith/arXiv arXiv 2024

  14. [14]

    arXiv preprint arXiv:2505.17908 (2025)

    Guo, L., Xu, X., Wang, L., Lin, J., Zhou, J., Zhang, Z., Su, B., Chen, Y.C.: Com- fymind: Toward general-purpose generation via tree-based planning and reactive feedback. arXiv preprint arXiv:2505.17908 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    (eds.) Advances in Neu- ral Information Processing Systems

    Ho,J.,Jain,A.,Abbeel,P.:Denoisingdiffusionprobabilisticmodels.In:Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neu- ral Information Processing Systems. vol. 33, pp. 6840–6851. Curran Associates, Inc. (2020),https://proceedings.neurips.cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf

    2020

  16. [16]

    arXiv preprint arXiv:2503.17671 (2025)

    Huang, O., Ma, Y., Zhao, Z., Wu, M., Ji, J., Zhang, R., Hu, Z., Sun, X., Ji, R.: Comfygpt: A self-optimizing multi-agent system for comprehensive comfyui workflow generation. arXiv preprint arXiv:2503.17671 (2025)

    arXiv 2025

  17. [17]

    Science Robotics 4(26), eaau5872 (2019)

    Hwangbo,J.,Lee,J.,Dosovitskiy,A.,Bellicoso,D.,Tsounis,V.,Koltun,V.,Hutter, M.: Learning agile and dynamic motor skills for legged robots. Science Robotics 4(26), eaau5872 (2019)

    2019

  18. [18]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition

    Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept cus- tomization of text-to-image diffusion. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 1931–1941 (2023)

    1931

  19. [19]

    In: NeurIPS (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: NeurIPS (2020)

    2020

  20. [20]

    Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., Gao, J., et al.: Multimodal foundationmodels:Fromspecialiststogeneral-purposeassistants.Foundationsand Trends®in Computer Graphics and Vision16(1-2), 1–214 (2024)

    2024

  21. [21]

    arXiv preprint arXiv:2306.12624 (2023)

    Li, T., Ku, M., Wei, C., Chen, W.: Dreamedit: Subject-driven image editing. arXiv preprint arXiv:2306.12624 (2023)

    Pith/arXiv arXiv 2023

  22. [22]

    org/abs/2503.23383

    Li, X., Zou, H., Liu, P.: Torl: Scaling tool-integrated rl (2025),https://arxiv. org/abs/2503.23383

    Pith/arXiv arXiv 2025

  23. [23]

    Briefings in bioinformatics23(6), bbac409 (2022)

    Luo,R.,Sun,L.,Xia,Y.,Qin,T.,Zhang,S.,Poon,H.,Liu,T.Y.:Biogpt:generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics23(6), bbac409 (2022)

    2022

  24. [24]

    Qian, C., Acikgoz, E.C., He, Q., Wang, H., Chen, X., Hakkani-Tür, D., Tur, G., Ji, H.: Toolrl: Reward is all tool learning needs (2025),https://arxiv.org/abs/ 2504.13958

    Pith/arXiv arXiv 2025

  25. [25]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22500–22510 (2023)

    2023

  26. [26]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y., Guo, D.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models (2024),https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  27. [27]

    Advances in Neural Information Processing Systems36(2024)

    Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems36(2024)

    2024

  28. [28]

    Advances in Neural Information Processing Systems36(2024)

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Lan- guage agents with verbal reinforcement learning. Advances in Neural Information Processing Systems36(2024)

    2024

  29. [29]

    Science362(6419), 1140–1144 (2018) 18 Z

    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science362(6419), 1140–1144 (2018) 18 Z. Li et al

    2018

  30. [30]

    Nature620(7972), 172–180 (2023)

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clinical knowledge. Nature620(7972), 172–180 (2023)

    2023

  31. [31]

    arXiv preprint arXiv:2411.14193 (2024)

    Sobania, D., Briesch, M., Rothlauf, F.: Comfygi: Automatic improvement of image generation workflows. arXiv preprint arXiv:2411.14193 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    arXiv preprint arXiv:2211.09085 (2022)

    Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., Stojnic, R.: Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 (2022)

    Pith/arXiv arXiv 2022

  33. [33]

    arXiv preprint arXiv:2312.11805 (2023)

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    Pith/arXiv arXiv 2023

  34. [34]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    Pith/arXiv arXiv 2023

  35. [35]

    arXiv preprint arXiv:2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    Pith/arXiv arXiv 2022

  36. [36]

    Advances in neural information processing systems35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  37. [37]

    arXiv preprint arXiv:2303.04671 (2023)

    Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N.: Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 (2023)

    Pith/arXiv arXiv 2023

  38. [38]

    arXiv preprint arXiv:2506.09790 (2025)

    Xu, Z., Wang, Y., Yang, X., Wang, L., Luo, W., Zhang, K., Hu, B., Zhang, M.: Comfyui-r1: Exploring reasoning models for workflow generation. arXiv preprint arXiv:2506.09790 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    arXiv preprint arXiv:2506.05010 (2025)

    Xu, Z., Yang, X., Wang, Y., Hu, Q., Wu, Z., Wang, L., Luo, W., Zhang, K., Hu, B., Zhang, M.: Comfyui-copilot: An intelligent assistant for automated workflow development. arXiv preprint arXiv:2506.05010 (2025)

    Pith/arXiv arXiv 2025

  40. [40]

    In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence

    Xue, X., Lu, Z., Huang, D., Wang, Z., Ouyang, W., Bai, L.: Comfybench: Bench- marking llm-based agents in comfyui for autonomously designing collaborative ai systems. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence. pp. 24614–24624 (2025)

    2025

  41. [41]

    arXiv preprint arXiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  42. [42]

    Advances in Neural Information Processing Systems36(2024)

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems36(2024)

    2024

  43. [43]

    differentiation

    Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., Zou, J.: Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496 (2024)

    Pith/arXiv arXiv 2024

  44. [44]

    In: CVPR (2023)

    Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: CVPR (2023)

    2023

  45. [45]

    In: Forty-first International Conference on Machine Learning (2024)

    Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., Schmidhuber, J.: Gptswarm: Language agents as optimizable graphs. In: Forty-first International Conference on Machine Learning (2024)

    2024

  46. [46]

    Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al.: Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934 (2024) Knowledge-Centric Agents for Workflow Generation 19 Knowledge-Centric Agents for Workflow Generation in ComfyUI Supplementary Materials This su...

    Pith/arXiv arXiv 2024