Pith. sign in

REVIEW 4 major objections 6 minor 43 references

RoboEgo System Card: An Omnimodal Model with Native Full Duplexity

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RoboEgo claims an omnimodal 7B model can watch, think, listen, speak, and act simultaneously with an 80 ms duplex granularity.

desk verdict A plausible text-first full-duplex architecture with an honest limitations section, but the headline responsiveness claim rests on a five-annotator, single-baseline comparison and an unmeasured 80 ms figure. read the letter →

arxiv 2506.01934 v1 pith:WALQQLTT submitted 2025-06-02 cs.AI

Yiqun Yao , Xiang Li , Xin Jiang , Xuezhi Fang , Naitong Yu , Aixin Sun , Yequan Wang This is my paper
classification cs.AI
keywords omnimodalmodelfullduplexitystreamingspeechdialoguetext-firststreamorganizationvisual-language-actionembodiedAIreal-timeresponsespeech-textalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents RoboEgo, a 7-billion-parameter model system that aims to combine omnimodal processing—handling text, vision, audio, and embodied actions together—with native full duplexity, meaning the model can receive input and produce output at the same time rather than alternating turns. Its central claim is that full-duplex behavior does not require word-level audio-text alignment or time-division multiplexing: by writing the full textual monologue into the stream ahead of the corresponding speech and padding with a special wait token, a single autoregressive backbone can decode text, audio, and actions from one shared hidden state. The stated result is a theoretical response granularity of 80 ms, with superior responsiveness and speech naturalness in streaming visually grounded conversations while keeping content quality comparable to leading semi-duplex omnimodal systems. A reader would care because real-time embodied assistants need exactly this simultaneous perception-and-response ability.

What carries the argument

The carrier of the argument is the text-first stream organization. Where token-level alignment splits each sentence into word fragments synchronized to audio frames, RoboEgo writes the whole textual monologue before the speech audio, delayed by a learned `spk_delay`, and pads the text channel with a special $\langle wait \rangle$ token until speech completes. This keeps the textual channel a coherent, forward-progressing monologue, preserves the pretrained autoregressive language modeling objective, and reduces annotation to sentence-level transcripts. The second piece is the shared hidden state: one 7-billion-parameter autoregressive backbone produces $h_t$ over merged input streams, and lightweight audio, text, and action decoders read $h_t$ directly, so outputs stay synchronized without re-aggregating $O(N^2)$ context as in talker-style architectures. Together these two choices are what the paper credits for the 80 ms theoretical duplex granularity.

What would settle it

Run a controlled streaming test in which a speaker interrupts RoboEgo at a random mid-sentence point and measure the time from interruption onset to a relevant response over many trials; if the median latency exceeds 200 ms, or if the model only reacts after the user finishes the whole utterance, the native full-duplexity claim is refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that native full duplexity is achievable by a stream-organization choice rather than a new attention mechanism. For each utterance, RoboEgo places the semantic token at time step $t$, starts the full textual monologue at $t - spk\_delay$, and fills the remaining text-channel time with a $\langle wait \rangle$ token until the speech audio finishes. The backbone then computes a shared hidden state $h_t$ from the merged visual, speaking, listening, and text streams, and three lightweight decoders emit audio, text, and action tokens from that same state as $i^{\mathrm{aud/txt/act}}_t = \mathrm{Dec}^{\mathrm{aud/txt/act}}(h_t)$. The paper reports that this text-first scheme preserves autoregressive language modeling, needs only sentence-level transcripts, avoids the cascading alignment errors of token-level alignment, and lets the model respond in a full-duplex manner through ordinary supervised fine-tuning. In evaluations, RoboEgo matches or approaches specialized and semi-duplex baselines on visual, ASR, and TTS benchmarks, and in head-to-head streaming chat it is rated higher on naturalness, responsiveness, and robustness while remaining comparable on helpfulness.

Load-bearing premise

The load-bearing premise is that placing the full text of an utterance before its speech, padded with special wait tokens, teaches the model to genuinely think, listen, and speak at once; if the wait tokens instead make the model stall, or if the shared hidden state cannot drive all decoders simultaneously, the 80 ms full-duplex claim fails.

Editorial extensions

If this is right

  • If the central claim holds, streaming spoken assistants can reach sub-100 ms duplex response without token-level alignment or time-sliced turn-taking.
  • The same backbone can drive speech, text, and embodied actions from one shared state, so adding a modality means adding a lightweight decoder rather than restructuring the context window.
  • Training for full-duplex behavior reduces to supervised fine-tuning on sentence-level, visually grounded dialogues, which is cheaper to annotate than word-level aligned data at scale.
  • Omnimodal systems built this way can match semi-duplex rivals on content quality while winning on responsiveness and naturalness, a combination the paper says native full-duplex systems previously could not achieve.
  • Deployment on real robots becomes practical: the reported embodied tasks, locomotion and a collaborative describing game, succeed at 96.5% and 97.2%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run an ablation that isolates the text-first stream from its data-cost advantage; comparing it directly against token-level alignment under identical sentence-level data would show whether the responsiveness gain is architectural or mostly from cheaper training data.
  • The mechanism is modality-agnostic: if text-first streams work for speech, the same delay-plus-wait-token pattern could schedule other asymmetric outputs, such as planning an action sequence before describing it.
  • A practical ceiling worth testing is whether the $\langle wait \rangle$ token teaches the model to stall; measuring response latency as the wait-token fraction grows would reveal whether 80 ms is a true floor or a best case.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript reports RoboEgo (FLM-Ego), a 7B-parameter omnimodal model that processes text, audio, vision, and actions in parallel, claiming native full duplexity with a theoretical response granularity of 80 ms. The architecture uses a text-first stream organization with <wait> padding instead of Moshi-style token-level alignment, and training proceeds through post-training and supervised fine-tuning. Evaluations cover visual understanding, ASR, TTS, streaming dialogue, and embodied robot tasks; the headline result is that RoboEgo matches Qwen2.5-Omni on helpfulness while scoring higher on naturalness, responsiveness, and robustness.

Significance. If the claims hold, RoboEgo would be a meaningful step toward omnimodal full-duplex systems: the text-first stream organization is a concrete, low-annotation alternative to token-level alignment, and the unified decoder from a shared hidden state is an interesting architectural choice. The paper also goes beyond chat by testing embodied action generation on a robot, which is rare. The empirical support, however, is currently too thin to establish the central duplexity and UX claims: the only head-to-head evidence is a five-annotator comparison against a single baseline, and no direct latency or barge-in measurement is reported. The non-circular use of external benchmarks and the explicit limitation statement about TDM for visually intensive tasks are strengths.

major comments (4)
  1. [Section 4.3, Table 4] The human evaluation supporting the central 'superior responsiveness' claim uses five annotators, one baseline (Qwen2.5-Omni), and reports no confidence intervals, no inter-annotator agreement, and no significance test; the responsiveness advantage (8.8 vs 8.1) is therefore not established. The paper should report a direct, measured full-duplex latency (e.g., time from user barge-in to assistant response change) on identical hardware for both systems, rather than relying solely on subjective ratings.
  2. [Section 2.3, Figure 3] The text-first stream organization is the load-bearing innovation behind the full-duplex claim, but no ablation compares it with token-level alignment (Moshi) or with alternative padding schemes on the same backbone, and no analysis shows that the <wait> tokens do not train the model to stall or ignore late input. I ask for an ablation measuring throughput, WER, and interruption responsiveness under text-first vs token-level alignment, plus a check of whether the model can revise a planned response when a user interrupts during the <wait> region.
  3. [Section 2.2, Eq. (4)] The claim that the shared hidden state h_t is 'sufficiently informative' to simultaneously drive text, audio, and action decoders is asserted without evidence; no experiment measures cross-modal consistency or decoding conflict. Please provide a quantitative test, e.g., comparing joint decoding against separate per-modality decoding on matched content, and report the effect on content quality and alignment.
  4. [Section 5, Table 5] The conclusion admits that visually intensive tasks still require TDM, which is in tension with the abstract's 'watches, thinks, listens, speaks, and acts simultaneously'; at minimum the central claim should be scoped to the audio-text-action channels. In addition, Table 5 reports embodied action accuracy without any baseline, task definition, or trial count, so the claim of a first demonstration is not empirically supported.
minor comments (6)
  1. [Abstract and Section 1] The 'theoretical duplex latency of 80 ms' is never defined or measured; please specify whether this is a frame parameter or an end-to-end metric and report a measured value.
  2. [Table 1] RoboEgo (SFT) scores 50.0 on MMStar versus 63.9 for Qwen2.5-VL-7B and 63.1 versus 68.5 on RealWorldQA; the text saying the model 'matches the performance' of Qwen2.5-VL-7B is inaccurate and should be revised.
  3. [Tables 2 and 3] There is a typo in 'GLM-4-V oice', and the LlamaQuestions entry for GLM-4-Voice appears as '50(64.7)' without explanation.
  4. [Figure 3] The notation c_i, P, EP, and wait tokens is used without a legend in the figure itself; please add a caption explaining each token type.
  5. [Section 2.3] The parameter spk_delay is introduced but its value or range is never reported; please state the value used in training and any sensitivity analysis.
  6. [Overall] No code, model weights, or detailed training-data composition are provided, which limits reproducibility and makes it hard to verify the claimed 'large-scale data scaling' in Section 2.3.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the central empirical claims rest on external benchmarks and an external head-to-head baseline; the only self-referential element is the L3+ embodied-AGI framing imported from the authors' own survey (ref [33]), which is minor and non-load-bearing.

  1. self citation load bearing [Section 2, Model Architecture (closing paragraph), and Section 1; refs [33] and [38]]
    "When stream-level visual embeddings are employed, RoboEgo realizes a complete prototype instantiation of a model architecture that satisfies the requirements for L3+ embodied AI [33]."

    Ref [33] is a survey authored by this paper's own authors (Y. Wang and A. Sun), and it supplies the 'L3+ embodied AGI' requirement set that RoboEgo is then declared to satisfy ('satisfies the requirements for L3+ embodied AI [33]'). The significance claim is therefore self-referential: the goal taxonomy and the assertion that the architecture meets it both originate in the authors' prior work, without an external standard for what 'L3+' requires. Ref [38] (another paper by two of the authors) appears only in a scaling-motivation sentence ('This limitation becomes increasingly restrictive as omnimodal foundation models continue to scale up [17, 38, 19]').

full rationale

The claimed derivation chain is not circular in its core. Empirical results are measured against external references rather than against quantities derived from the model itself: vision (MMStar, RealWorldQA, OCRBench), audio understanding (Fleurs-zh, WenetSpeech-net, LibriSpeech-clean, LlamaQuestions), audio generation (Seed-TTS WER/SIM), embodied tasks (LEJU Kuavo success rates), and real-time chatting (human head-to-head vs. Qwen2.5-Omni with five annotators). No parameter is fitted to a subset and then renamed a prediction. The 80 ms figure is explicitly 'theoretical' and is a frame-level design parameter, not an end-to-end measured latency; the text-first stream organization (Sec 2.3) is not ablated against Moshi-style token alignment, and the sufficiency of h_t for simultaneous decoding (Eqs. 1, 4) is asserted rather than demonstrated. These are verification gaps and correctness risks, which the review rules explicitly separate from circularity. Per the review rule to flag limitations, Section 5 concedes 'current stream-based visual encoders still fall short in reliability for full-duplex, all-modality processing, necessitating the use of time-division multiplexing (TDM) as a temporary workaround,' which qualifies the blanket 'watches, thinks, listens, speaks, and acts simultaneously' claim but does not make any derivation self-referential. The only self-referential element is the motivational L3+ embodied-AGI framing imported from the authors' own survey (ref [33]), with ref [38] cited in a scaling remark; neither is load-bearing for the benchmark or evaluation results. Honest finding: one minor non-load-bearing self-citation, score 2.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The ledger captures the design choices and empirical assumptions the full-duplex claim rests on. The only explicit hand-tuned parameter is the speech delay offset spk_delay. The larger burden is three empirical domain assumptions about the shared hidden state, the text-first token layout, and the sufficiency of SFT for duplex behavior, none of which is tested in isolation.

free parameters (1)
  • spk_delay = not reported
    Introduced in Section 2.3 as the time offset between the textual monologue and speech start; chosen to mimic human cognition, but no value, tuning procedure, or sensitivity analysis is given. The full-duplex behavior depends on this choice, so it functions as a tuned hyperparameter.
assumptions (4)
  • domain assumption The hidden state h_t produced by the backbone is sufficient to decode text, audio, and action tokens simultaneously (Eq. 4, Section 2.2).
    The architecture relies on a single shared hidden state at each time step driving all decoders; no experiment isolates whether this causes modality conflict or degradation.
  • domain assumption The 'text-first' stream layout with wait padding preserves autoregressive language modeling and avoids cascading alignment errors from word-level timestamps (Section 2.3).
    This is an empirical claim with no ablation or quantitative comparison against Moshi-style token-level alignment.
  • domain assumption Standard supervised learning on the described data is enough for the model to learn full-duplex turn-taking and interruption handling without specialized training mechanisms (Section 3.2).
    The paper states this as a design benefit but provides no control experiment, for example training on turn-based data and measuring duplex behavior.
  • domain assumption Human ratings from five annotators on a single, unspecified test set are representative of real-world user experience (Section 4.3).
    No inter-annotator agreement, number of conversations, or statistical significance is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RoboEgo System Card: An Omnimodal Model with Native Full Duplexity." pith.science (2026). https://pith.science/paper/WALQQLTT

@misc{pith2026250601934,
  author       = {Pith},
  title        = {Pith review of: RoboEgo System Card: An Omnimodal Model with Native Full Duplexity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WALQQLTT}},
  note         = {Machine review of arXiv:2506.01934}
}
read the original abstract

Humans naturally process real-world multimodal information in a full-duplex manner. In artificial intelligence, replicating this capability is essential for advancing model development and deployment, particularly in embodied contexts. The development of multimodal models faces two primary challenges: (1) effectively handling more than three modalities-such as vision, audio, and text; and (2) delivering full-duplex responses to rapidly evolving human instructions. To facilitate research on models that support both omnimodal processing and full duplexity, we present RoboEgo (alias: FLM-Ego), a unified model system designed to address both challenges. RoboEgo incorporates a backbone architecture and algorithms that natively support full duplexity, achieving a theoretical duplex latency of 80 ms. In streaming visually grounded conversations under real-world conditions, RoboEgo exhibits superior responsiveness and speech naturalness, while maintaining comparable content qualities to state-of-the-art semi-duplex omnimodal models-a feat previously considered unattainable by native full-duplex systems.

Figures

Figures reproduced from arXiv: 2506.01934 by the authors.

Figure 1
Figure 1. The “Telepathy Challenge” game (心有灵犀): a real demo for RoboEgo’s omnimodal full-duplex capabilities. arXiv:2506.01934v1 [cs.AI] 2 Jun 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the architecture of RoboEgo [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Stream organization for text and audio in RoboEgo [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 7 canonical work pages

  1. [1]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Singh Chap- lot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Théophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timo- thée Lacroix, Guillaume Lample, Diego de Las Casas, Thibaut Lavril, Teven Le Scao, Andy 8 System...

    arXiv

  2. [2]

    Seed-tts: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, J...

    arXiv 2024

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Audiolm: a language modeling approach to audio generation

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523–2533, 2023

    2023

  5. [5]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash...

    arXiv 2023

  6. [6]

    Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712, 2023

    arXiv 2023

  7. [7]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

  8. [8]

    Minmo: A multimodal large language model for seamless voice interaction

    Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282, 2025

    arXiv 2025

Show all 43 references
  1. [9]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  2. [10]

    Fleurs: Few-shot learning evaluation of universal representations of speech

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446, 2022

    2022 arXiv

  3. [11]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  4. [12]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. CoRR, abs/2410.00037, 2024

    2024 arXiv

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  6. [14]

    How far are we from agi: Are llms all we need? arXiv preprint arXiv:2405.10313, 2024

    Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are we from agi: Are llms all we need? arXiv preprint arXiv:2405.10313, 2024

    2024 arXiv

  7. [15]

    Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning

    Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, et al. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321, 2024

    2024 arXiv

  8. [16]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  9. [17]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  10. [18]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  11. [19]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya K...

    2024 arXiv

  12. [20]

    Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: A...

    2024

  13. [21]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  14. [22]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informatio...

    2023

  16. [24]

    Aligning cyber space with physical world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06886, 2024

    2024 arXiv

  17. [25]

    Introducing meta llama 3: The most capable openly available llm to date

    Meta. Introducing meta llama 3: The most capable openly available llm to date. https: //ai.meta.com/blog/meta-llama-3/, 2024

    2024

  18. [26]

    Spoken question answering and speech continuation using spectrogram-powered llm

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered llm. arXiv preprint arXiv:2305.15...

    2023 arXiv

  19. [27]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023

    2023 arXiv

  20. [28]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 5206–5210. IEEE, 2015

    2015

  21. [29]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  22. [30]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processi...

    2024

  23. [31]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  24. [32]

    A full-duplex speech dialogue scheme based on large language models

    Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Yuanjun Xiong, and Wei Xia. A full-duplex speech dialogue scheme based on large language models. CoRR, abs/2405.19487, 2024

    2024 arXiv

  25. [33]

    Toward embodied agi: A review of embodied ai and the road ahead

    Yequan Wang and Aixin Sun. Toward embodied agi: A review of embodied ai and the road ahead. arXiv preprint arXiv:2505.14235, 2025

    2025 arXiv

  26. [34]

    Realworldqa, 2024

    X.AI. Realworldqa, 2024

    2024

  27. [35]

    Mini-omni: Language models can hear, talk while thinking in streaming

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024. 11 System Card of FLM-EgoREFERENCES

    2024 arXiv

  28. [36]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  29. [37]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  30. [38]

    Research without re-search: Maximal update parametrization yields accurate loss prediction across scales

    Yiqun Yao and Yequan Wang. Research without re-search: Maximal update parametrization yields accurate loss prediction across scales. CoRR, abs/2304.06875, 2023

    2023 arXiv

  31. [39]

    Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multimodal live streaming on your phone, 2025

    Chongyi Wang Yuan Yao, Tianyu Yu et al. Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multimodal live streaming on your phone, 2025. https://openbmb.notion.site/MiniCPM-o-2-6-A-GPT-4o-Level -MLLM-for-Vision-Speech-and-Multimodal-Live-Streaming-on-Your-Phone

    2025

  32. [40]

    Scaling speech-text pre-training with synthetic interleaved data

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. CoRR, abs/2411.17607, 2024

    2024 arXiv

  33. [41]

    Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition

    Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xi- aoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech an...

    2022

  34. [42]

    Omniflatten: An end-to-end gpt model for seamless voice conversation

    Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan, Zhihao Du, et al. Omniflatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint arXiv:2410.17799, 2024

    arXiv

  35. [43]

    Beyond the turn-based game: Enabling real-time conversations with duplex models

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Xu Han, Zihang Xu, Yuanwei Xu, Weilin Zhao, Maosong Sun, and Zhiyuan Liu. Beyond the turn-based game: Enabling real-time conversations with duplex models. arXiv preprint arXiv:2406.15718, 2024. 12

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.