Pith. sign in

REVIEW 4 major objections 6 minor 56 references

Symbolic Representation for Any-to-Any Generative Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes that any multimodal generative task can be expressed as a symbolic flow of functions, parameters, and topology, and that a pre-trained language model can infer that flow from a natural language instruction with no…

desk verdict A practical workflow-synthesis system with solid ablations and competitive numbers, but the 'any-to-any' claim outruns the evidence and the formalism is a restatement of prior work. read the letter →

arxiv 2504.17261 v1 pith:KODBMWVJ submitted 2025-04-24 cs.LG cs.AI

Jiaqi Chen , Xiaoye Zhu , Yue Wang , Tianyang Liu , Xinhui Chen , Ying Chen , Chak Tou Leong , Yifei Ke
show 4 more authors
Joseph Liu Yiwen Yuan Julian McAuley Li-jia Li
This is my paper · ORCID
classification cs.LGcs.AI
keywords symbolictaskrepresentationany-to-anygenerationtraining-freeinferenceworkflowsynthesismultimodalgenerativemodelslargelanguageprogramfunction-parameter-topology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that 'any-to-any' generative tasks—changing, blending, or producing content across text, image, video, audio, and 3D—need not be solved by training a single neural model. Instead, any task can be written down as a symbolic flow $S = \{(f_i, \phi_{f_i}, D_i)\}$ of atomic functions, their parameters, and the topology that connects them. A pre-trained language model can read a natural language instruction plus a list of available functions and emit this flow directly, then a workflow executor runs it. On 120 tasks across 12 categories and a 200-workflow benchmark, the paper reports that this training-free route matches or beats task-specific unified models, with the extra ability to edit and interrupt the generation. If true, the practical bottleneck moves from training data and model scale to the completeness of the function library and the reliability of language-model mapping.

What carries the argument

The load-bearing object is the symbolic flow $S = \{(f_i, \phi_{f_i}, D_i)\}$, built from the triple $\Omega(t) := (F, \Phi, T)$. Functions are atomic modal operations such as encode, condition, blend, or denoise; parameters are per-function settings like blending strength or a text prompt; topology is the directed graph of data connections between outputs and inputs. The paper treats this triple as a complete description of any generative task, and the inference problem as the problem of writing this program from natural language.

What would settle it

Take a task that requires an operation missing from the executor's function list but expressible in natural language; if the inference engine cannot substitute available functions and the workflow fails after its refinement budget, the claimed 'any task' coverage is falsified.

Watch

Extended reading notes

Core claim

Formally, the paper defines a generative task as $\Omega(t) := (F, \Phi, T)$, where $F$ is a set of atomic computational functions $f_i : I_i \times \phi_i \to O_i$, $\Phi$ collects the parameter choices for every function, and $T$ is a set of directed data-flow edges $d_k = (f_j, y_j) \to (f_i, x_i)$ from a specific output of one function to a specific input of another. These combine into a symbolic flow $S = \{(f_i, \phi_{f_i}, D_i) \mid f_i \in F\}$, one entry per function with its parameters and all incoming connections. The inference engine is a frozen pre-trained language model that performs three steps—component inference $\psi_1 : (X, s, C) \to (F, \Phi)$, topology construction $\psi_2 : (X, s, C, F, \Phi) \to T$, and iterative refinement against compilation or execution errors—so that the full mapping is $\mathcal{M} : (X, s, C) \to \Omega(t)$. Because the representation is explicit, users can edit the generated program directly, swap functions, or stop it mid-execution, properties the paper demonstrates along with competitive or better output quality in user studies.

Load-bearing premise

The claim rests on the executor's library of available functions containing every operation the tasks require, and on the language model mapping instructions to correct combinations of those functions reliably.

Editorial extensions

If this is right

  • Task composition becomes program synthesis: new any-to-any behaviors are created by selecting and connecting existing functions, not by collecting paired training data.
  • Generative systems become editable and interruptible: a user can change a function, alter a parameter, or stop and resume execution at any node.
  • The same frozen language model can span tasks across image, video, audio, and 3D, since no weights are retrained per task.
  • On the paper's 120-task suite and the 200-workflow benchmark, this approach reaches a 43 percent overall resolve rate, about 10 points above the best agentic baseline, while using a simpler design.
  • Simple symbolic inference beats complex multi-agent planning on straightforward tasks, and the paper notes that combining symbolic flows with agent-based planning may help on harder workflows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the function library ever becomes complete enough, the ceiling on generative competence is set by the executor's atomic functions, not by the language model or by training data volume.
  • A direct testable extension is measuring how Pass@1 degrades when the inference language model is swapped for smaller or weaker models; the paper fixes one inference model, so the robustness of the mapping to model scale is left open.
  • The three syntax styles suggest a design axis for the DSL: making the syntax more readable raised format errors, while stricter structure caused topology gaps, so a hybrid syntax could combine the strengths and is a natural next experiment.
  • Because refinement is feedback-driven, the framework could in principle consume executor error messages as a signal to repair its own flows, which is exactly the loop the paper uses but only with a bounded iteration count.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes A-LANGUAGE, a symbolic representation for any-to-any generative tasks in which each task is a triple (F, Φ, T) of functions, parameters, and topology, and an executable symbolic flow S = {(f_i, φ_f_i, D_i)}. A pretrained language model (GPT-4o) is used as a training-free inference engine that maps natural-language instructions, inputs, and constraints to such a symbolic flow; iterative refinement repairs compilation and execution failures, and ComfyUI serves as the executor. The evaluation covers a self-constructed suite of 120 tasks in 12 categories plus the ComfyBench benchmark of 200 workflows, comparing against GenAgent, HuggingGPT, ComfyAgent, LWM, Unified-IO, Show-o, Seed-X, and Gen-3. The main quantitative results are a 0.98/0.87 compilation/execution pass rate on the 120-suite (Table 5), a 43% resolve rate on ComfyBench (Table 3), and user-study win rates against several unified models (Figure 6). The paper also includes an ablation of the two-stage inference and refinement design (Table 4) and a comparison of three syntax styles (Figure 7).

Significance. If taken as a demonstration on a defined executor and task distribution, the paper is a useful contribution: it packages workflow synthesis under a clean symbolic abstraction, shows that a two-stage inference plus refinement design is effective relative to agentic baselines on ComfyBench, and provides a concrete, open-source implementation. The breadth of the evaluation (120 tasks plus ComfyBench), the explicit ablations, and the reproducible executor setup are strengths. However, the title and abstract claim 'arbitrary multimodal tasks' and 'any-to-any' representability, and this claim is not established by the paper: representability is constrained by the finite function library in C, no completeness or coverage argument is given, and the paper's own execution pass rates (0.87 on the 120-suite, 43% on ComfyBench) show that many instructions do not lead to executable flows. The empirical comparison is also weakened by the small annotator pool and by unclear treatment of failures. The work is best read as an empirical recipe with promising results on a finite, self-selected task set, not as a proof of universal any-to-any representation.

major comments (4)
  1. [Abstract; Sec. 3.1; Sec. 4] The claim that A-LANGUAGE is 'capable of representing arbitrary multimodal tasks' is not supported by the formalism or the experiments. The inference map M : (X,s,C) → Ω(t) in Sec. 4 is constrained to functions available in C, and the paper provides no completeness, Turing-universality, or coverage argument for the ComfyUI function library. Table 5 reports an execution pass rate of only 0.87 on the authors' own 120-task suite, and Table 3 reports a 43% resolve rate on ComfyBench, which directly bounds the fraction of instructions that become executable flows. Please either scope the claims to the evaluated task distribution and executor, or provide a formal argument (e.g., a reduction from a known universal model of computation) that the function library can express any operation required by the claimed task class.
  2. [Sec. 5.1; Sec. 5.2; Table 2; Figure 6] The user study uses only five annotators, and the reported win rates and average ranks are presented without inter-annotator agreement, confidence intervals, or significance tests. Moreover, many baseline cells in Table 2 are marked '✗' or '—', and the paper does not state how a task on which Ours fails to execute (13% of the 120-suite per Table 5) is treated in the ranking. This makes the headline numbers (e.g., 94% win rate vs. Show-o, 100% vs. Unified-IO on Text-to-Audio) difficult to interpret. Please report success-aware aggregation, specify the treatment of failed executions, and provide agreement metrics or a larger annotator pool.
  3. [Sec. 4; Sec. 5.1] The inference pipeline is not specified enough for replication. The refinement operator R, the two-stage decomposition ψ1 and ψ2, and the retrieval mechanism are described only at a high level; the prompt templates, the contents of the 16-program retrieval database, the exact constraint set C, and the full list of available ComfyUI functions are not included in the submitted text (the paper refers to an appendix that is absent from the provided manuscript). Without these details, the central claim that a pretrained LM reliably maps natural-language instructions to executable flows cannot be independently checked.
  4. [Table 4] The ablation in Table 4 does not include a row with neither the two-stage design nor refinement, so it is not possible to quantify the individual contribution of each component relative to a common baseline. The table also does not state whether the improvement from 28.50% to 41.00% is statistically significant given the reported error margins in Table 3. Please add the missing baseline and, if possible, variance estimates for each ablation condition.
minor comments (6)
  1. [Abstract] The abstract says 'over 12 diverse multimodal generative tasks,' while the evaluation uses 120 tasks in 12 categories; please make this wording unambiguous.
  2. [Figure 7; Figure 8] The captions for Figure 7 and Figure 8 do not define the Pass@1 aggregation for each syntax style or the six error categories shown in Figure 8; adding these definitions would improve interpretability.
  3. [Table 2] Table 2 uses both '✗' and '—' without a legend; please clarify whether these denote inapplicability, failure, or tasks not benchmarked for that method.
  4. [References [49] and [50]] References [49] and [50] appear to share the same arXiv identifier (arXiv:2409.01392); please disambiguate the GenAgent and ComfyBench citations.
  5. [Throughout] The text contains typos and formatting inconsistencies, including 'agaentic' in Sec. 1 and the nonstandard spacing in 'A-L ANGUAGE' throughout; a copyedit pass is needed.
  6. [Sec. 5.1; Sec. 5.2] Since the paper claims efficiency advantages, it would be helpful to report the per-task inference time and execution cost of the proposed method alongside the baselines, rather than only stating the hardware setup.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the symbolic-flow construction is a formal definition, and the validation is empirical against external and self-constructed benchmarks.

full rationale

The paper's central construction, Omega(t) = (F, Phi, T), is introduced as a formal representation rather than as a derived prediction, and the inference engine M: (X, s, C) -> Omega(t) is implemented using a pretrained language model and validated through execution pass rates, user studies, and the external ComfyBench benchmark. No parameter is fitted to a subset of the evaluation outcomes and then renamed as a prediction, and no load-bearing claim is justified by a self-citation or by an imported uniqueness theorem. The 'arbitrary multimodal tasks' wording in the abstract is broader than what the finite ComfyUI function library and the reported 0.87 execution pass rate can fully establish, but that is an evidentiary and coverage limitation rather than a circular reduction: the paper never equates the expressiveness of its representation with the function library by construction, and it explicitly reports the failure rate in Table 5. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on the availability and decomposability of functions, which the paper assumes. The free parameters are hyperparameters of the inference engine, not fitted to the outcome.

free parameters (3)
  • number of retrieved reference programs = 3
    RAG retrieves three most relevant programs from a curated database of 16 reference programs. This k is chosen by hand and affects inference quality.
  • reference program database = 16 programs
    A curated database of 16 reference programs is used for in-context learning. The content and selection are hand-crafted and not detailed in the paper.
  • max refinement iterations = not specified
    An iteration cap is mentioned in Section 4 but its value is not reported; it influences the resolve rate and is chosen by hand.
assumptions (3)
  • domain assumption Generative tasks can be decomposed into atomic functions with typed inputs/outputs and parameters, and composed through dataflow.
    The entire A-LANGUAGE formalism relies on this decomposition, asserted in Section 3.1 without proof.
  • domain assumption The pre-trained LM (GPT-4o) can correctly infer functions, parameters, and topology from natural language instructions when provided constraints.
    The inference engine's validity is empirical, but the paper assumes reliability; pass rates of 0.87 execution bound this.
  • ad hoc to paper The function library (ComfyUI) contains all necessary atomic functions for the target task suite.
    The 'arbitrary' claim is only as strong as the available function set; the paper does not prove coverage for arbitrary tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Symbolic Representation for Any-to-Any Generative Tasks." pith.science (2026). https://pith.science/paper/KODBMWVJ

@misc{pith2026250417261,
  author       = {Pith},
  title        = {Pith review of: Symbolic Representation for Any-to-Any Generative Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KODBMWVJ}},
  note         = {Machine review of arXiv:2504.17261}
}
read the original abstract

We propose a symbolic generative task description language and a corresponding inference engine capable of representing arbitrary multimodal tasks as structured symbolic flows. Unlike conventional generative models that rely on large-scale training and implicit neural representations to learn cross-modal mappings, often at high computational cost and with limited flexibility, our framework introduces an explicit symbolic representation comprising three core primitives: functions, parameters, and topological logic. Leveraging a pre-trained language model, our inference engine maps natural language instructions directly to symbolic workflows in a training-free manner. Our framework successfully performs over 12 diverse multimodal generative tasks, demonstrating strong performance and flexibility without the need for task-specific tuning. Experiments show that our method not only matches or outperforms existing state-of-the-art unified models in content quality, but also offers greater efficiency, editability, and interruptibility. We believe that symbolic task representations provide a cost-effective and extensible foundation for advancing the capabilities of generative AI.

Figures

Figures reproduced from arXiv: 2504.17261 by the authors.

Figure 1
Figure 1. A symbolic representation for Any-to-Any generative tasks. (a) We develop a training-free inference engine that transforms natural language task descriptions into executable symbolic flow comprising functions, parameters, and the topology. (b) The symbolic flow allows executing generative tasks as programs. Example task is mentioned in the first sentence of Sec. 1. (c) Both functions and parameters can be easily mod… view at source ↗
Figure 2
Figure 2. The Any-to-Any generative model. Our model demonstrates the capability to handle any-to-any generative tasks across various modalities, including text, images, videos, audio, and 3D content. It supports flexible transformations such as converting image to video, generating 3D models from images, or synthesizing audio from textual prompts. Formally, any-to-any generative tasks refer to generating outputs in any desir… view at source ↗
Figure 3
Figure 3. Syntax comparison. We implement our symbolic repre￾sentation using three different styles of domain-specific languages (DSLs). (a) The declarative syntax registers all components into the workflow. (b) The dataflow syntax emphasizes the direction of data flow. (c) The pseudo-natural syntax mimics human language expression. Pseudo-natural syntax Pseudo-natural syntax [9] aims to bridge formal representations with mor… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Inferring symbolic flow with pre-trained language model (LM). Beginning with (a) a natural language task description and key functions and parameters, we leverage LM to infer (b) a comprehensive set of functions and parameters. We then integrate (a) and (b) to deduce t…
Figure 5
Figure 5. Figure 5: Demonstration of the inference and execution. The inference framework translates a natural language task description into an executable symbolic representation. This symbolic representation is then compiled and executed through a workflow executor to perform the desire…
Figure 6
Figure 6. Figure 6: Comparison of our win rates with the state-of-the-art unified multimodal models on our task suite [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison of syntax styles. Metric: Pass@1 (↑). See Appendix for details [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparative error distribution for dataflow, declar￾ative, and pseudo-natural syntax styles, illustrating six types of errors occur when testing on the 120 generative tasks. Initial Modified (a) Switch model to <Zero123> (b) Change the color of table workflow.add_node(…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 39 canonical work pages

  1. [1]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text

    Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206–24221, 2021. 3

    work page 2021

  2. [2]

    Neural module networks, 2017

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks, 2017. 3

    work page 2017

  3. [3]

    The claude 3 model family: Opus, sonnet, haiku,

    Anthropic. The claude 3 model family: Opus, sonnet, haiku,

  4. [4]

    Im- proving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jian- feng Wang, Linjie Li, † LongOuyang, † JuntangZhuang, † JoyceLee, † YufeiGuo, † WesamManassra, † PrafullaDhari- wal, † CaseyChu, † YunxinJiao, and Aditya Ramesh. Im- proving image generation with better captions. 3

    work page

  5. [5]

    Audiolm: a language modeling approach to audio generation, 2023

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation, 2023. 3

    work page 2023

  6. [6]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. NeurIPS, 2020. 3

    2020

  7. [7]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. 2023

    work page 2023

  8. [8]

    DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, Shan- huang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y . K. Li, Wenfeng Liang, Fangyun Lin, A. ...

    work page

Show all 56 references
  1. [9]

    Natural language is a programming lan- guage: Applying natural language processing to software de- velopment

    Michael D Ernst. Natural language is a programming lan- guage: Applying natural language processing to software de- velopment. In 2nd Summit on Advances in Programming Languages (SNAPL 2017). Schloss-Dagstuhl-Leibniz Zen- trum für Informatik, 2017. 5

    2017

  2. [10]

    Clip2video: Mastering video-text retrieval via image clip

    Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097, 2021. 3

    2021 arXiv

  3. [11]

    Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation. arXiv preprint arXiv:2404.14396, 2024. 7

    2024 arXiv

  4. [12]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In CVPR, pages 14953–14962, 2023. 2, 3, 7

    2023

  5. [13]

    Toolkengpt: Augmenting frozen language models with mas- sive tools via tool embeddings, 2024

    Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with mas- sive tools via tool embeddings, 2024. 2

    2024

  6. [14]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers, 2022. 3

    2022

  7. [15]

    Learning to reason: End- to-end module networks for visual question answering

    Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End- to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804–813, 2017. 3

    2017

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guil- laume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-...

    2024

  9. [17]

    Inferring and executing programs for visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference on computer vision, pages 2989–2998, 2017. 3

    2017

  10. [18]

    Audiogen: Textually guided audio gen- eration, 2023

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taig- man, and Yossi Adi. Audiogen: Textually guided audio gen- eration, 2023. 3

    2023

  11. [19]

    Align before fuse: Vision and language representation learning with momentum distillation

    Junnan Li, Ramprasaath R Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021

    2021

  12. [20]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022. 3

    2022

  13. [21]

    Starcoder: may the source be with you!, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muen- nighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, 9 Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier De- haene, Mishig Davaadorj, Joel Lamy-Poirier, João Mont...

    2023

  14. [22]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,...

    2020

  15. [23]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024. 7

    2024

  16. [24]

    Starcoder 2 and the stack v2: The next generation,

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Fed- erico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, In- draneil Pau...

  17. [25]

    Unified-io: A unified model for vision, language, and multi-modal tasks, 2022

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks, 2022. 2, 3

    2022

  18. [26]

    Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action, 2023

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action, 2023. 2, 3, 7

    2023

  19. [27]

    Chameleon: Plug-and-play compositional reasoning with large language models

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  20. [28]

    4m: Massively multimodal masked modeling, 2023

    David Mizrahi, Roman Bachmann, O ˘guzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling, 2023. 3

    2023

  21. [29]

    Chatgpt: Optimizing language models for dialogue

    OpenAI. Chatgpt: Optimizing language models for dialogue. http : / / web . archive . org / web / 20230109000707/https://openai.com/blog/ chatgpt/, 2022. 3

    2022

  22. [30]

    Gpt-4 technical report, 2024

    OpenAI et al. Gpt-4 technical report, 2024

    2024

  23. [31]

    Gpt-4o system card, 2024

    OpenAI et al. Gpt-4o system card, 2024. 2, 3

    2024

  24. [32]

    Barron, and Ben Milden- hall

    Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion, 2022. 3

    2022

  25. [33]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789,

    arXiv

  26. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  27. [35]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022

    2022

  28. [36]

    High-resolution image syn- thesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image syn- thesis with latent diffusion models, 2022

    2022

  29. [37]

    Runway. Gen-3. https://runwayml.com/blog/ introducing-gen-3-alpha/ , 2024. 3, 7

    2024

  30. [38]

    Toolformer: Language models can teach themselves to use tools, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. 2

    2023

  31. [39]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. 2, 3, 7

    2023

  32. [40]

    Codi-2: In-context, in- terleaved, and interactive any-to-any generation, 2023

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context, in- terleaved, and interactive any-to-any generation, 2023. 2, 3

    2023

  33. [41]

    Any-to-any generation via composable diffu- sion, 2023

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffu- sion, 2023. 2, 3

    2023

  34. [42]

    Any-to-any generation via composable diffu- sion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffu- sion. Advances in Neural Information Processing Systems, 36, 2024. 2, 3

    2024

  35. [43]

    Llama: Open and efficient foundation lan- guage models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation lan-...

    2023

  36. [44]

    Llama 2: Open foundation and fine- tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fer- nandes, Jere...

    2023

  37. [45]

    A neurocognitive perspective on lan- guage: The declarative/procedural model

    Michael T Ullman. A neurocognitive perspective on lan- guage: The declarative/procedural model. Nature reviews neuroscience, 2(10):717–726, 2001. 4

    2001

  38. [46]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learn- ing framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learn- ing framework. In International Conference on Machine Learni...

    2022

  39. [47]

    Image as a foreign language: Beit pretraining for all vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXiv:2208.10442, 2022. 3

    2022 arXiv

  40. [48]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 7

    2024 arXiv

  41. [50]

    Comfybench: Benchmarking llm- based agents in comfyui for autonomously designing collab- orative ai systems

    Xiangyuan Xue, Zeyu Lu, Di Huang, Zidong Wang, Wanli Ouyang, and Lei Bai. Comfybench: Benchmarking llm- based agents in comfyui for autonomously designing collab- orative ai systems. arXiv preprint arXiv:2409.01392, 2024. 7

    2024

  42. [51]

    Video- text pre-training with learned regions

    Rui Yan, Mike Zheng Shou, Yixiao Ge, Alex Jinpeng Wang, Xudong Lin, Guanyu Cai, and Jinhui Tang. Video- text pre-training with learned regions. arXiv preprint arXiv:2112.01194, 2021. 3

    2021 arXiv

  43. [52]

    Vision-language pre-training with triple contrastive learning

    Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15...

    2022

  44. [53]

    Merlot reserve: Neu- ral script knowledge through vision and language and sound

    Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yan- peng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neu- ral script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Compu...

    2022

  45. [54]

    Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

    Yan Zeng, Xinsong Zhang, and Hang Li. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. In International Conference on Machine Learning, pages 25994–26009. PMLR, 2022. 3

    2022

  46. [55]

    Anygpt: Unified multimodal llm with discrete sequence modeling, 2024

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu- gang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling, 2024. 2, 3

    2024

  47. [56]

    Speechgpt: Empow- ering large language models with intrinsic cross-modal con- versational abilities, 2023

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empow- ering large language models with intrinsic cross-modal con- versational abilities, 2023. 3 11

    2023

  48. [2024]

    Corpus ID: 268232499. 3

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.