Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Auras speeds embodied AI agents 2.54x on average while keeping accuracy at 102.7% of sequential execution.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Auras, a perception-generation disaggregation framework with a public context buffer and asynchronous pipeline executor, raises embodied-agent throughput by 2.54x on average without losing accuracy (102.7%).

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Good throughput engineering; the accuracy claim needs sharper definitions before it means what it says. the 3 major comments →

arxiv 2509.09560 v1 pith:7YBO6BSO submitted 2025-09-11 cs.AI cs.LG

Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution

Shulai Zhang , Ao Xu , Quan Chen , Han Zhao , Weihao Cui , Ningxin Zheng , Haibin Lin , Xin Liu
show 1 more author
Minyi Guo
This is my paper
classification cs.AI cs.LG
keywords embodied AIinference frameworkpipeline parallelismperception-generation disaggregationpublic contextauto-regressive policydiffusion policyreal-time robot control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the closed-loop sequential perception-generation loop, which guarantees observation-response bijection, is the main bottleneck preventing embodied AI agents from reaching high 'thinking' frequencies. Auras breaks this loop by disaggregating perception from generation and running them as an asynchronous pipeline that shares a public context buffer, so generation always computes on the freshest perception output. Across six policies (auto-regressive and diffusion-based), Auras reports 2.54x average throughput improvement—3.05x for auto-regressive, 2.28x for diffusion—while retaining 102.7% of the original sequential accuracy, all without fine-tuning model weights. If this holds, embodied agents can achieve actuator-rate control on the same consumer GPU, addressing a key practical gap for real-world robotics.

Core claim

Auras establishes that an embodied policy does not need to be executed as a single sequential perception-generation request. By dissecting the compute graph, Auras identifies the volatile variables (the perception output for diffusion models; the perception output plus the generated action-token sequence for auto-regressive models) and promotes them to a shared public context buffer. Generation stages fetch that buffer asynchronously, so actions are computed on the latest environment state instead of stale observations. For auto-regressive models, causal masking lets concurrent decode steps be merged into a single prefill over the public context, cutting redundant computation. An asynchronou

What carries the argument

The central mechanism is the public context buffer: a shared memory region updated by the perception stage each frame and fetched asynchronously by generation stages. The fetch_offset controls how fresh the context is (0 for diffusion policies, -1 for auto-regressive policies). For auto-regressive models, causal masking allows a single merged prefill to replace multiple smaller ones, since the hidden state of the i-th token depends only on preceding tokens. The asynchronous pipeline executor defines frames of concurrent perception and generation stages, with pipeline degrees pp_perception and pp_generation, and a hierarchical tuner searches these along with a skewness weight alpha that biase

Load-bearing premise

The load-bearing premise is that a policy trained for closed-loop sequential execution remains accurate when its generation is recomputed on a continuously refreshed public context (for auto-regressive models, including the generated action-token sequence) without any fine-tuning; the paper offers simulation evidence but no formal argument.

What would settle it

Run an auto-regressive policy on a task where the correct action depends on the previous action (e.g., a multi-step assembly) with fetch_offset=-1; if success rate drops substantially below the sequential baseline, the fresh-context assumption fails. Concretely, compare Pick-Coke-Can task-completion frames under Auras versus sequential execution while artificially delaying perception updates by one frame.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Auras raises OpenVLA's thinking frequency from 6Hz to 17Hz on an RTX 4090 without any fine-tuning.
  • For auto-regressive policies, decode steps from parallel requests merge into one prefill; speedup grows with output token count, reaching 2.63x when the generation module is scaled 4x.
  • For diffusion policies, skewing more denoising steps to later pipeline stages (alpha=1) improves accuracy by 23.98% at a modest throughput drop (11.84 to 10.25 FPS).
  • Pipeline parallelism with public context outperforms unstructured multi-stream parallelism, which only achieves 1.11x for auto-regressive models versus Auras' 2.20x-3.29x.
  • The framework works across six policies and two GPUs, with speedups of 1.32x-3.48x on RTX 4090 and 1.18x-3.08x on RTX 3090.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If fresh-context reasoning transfers to physical robots, closed-loop policies could run at actuator-rate control on the same model, potentially removing the need for a separate 'System 1' fast controller architecture.
  • The causal-masking merge generalizes beyond vision-language-action models: any causal transformer policy could fuse concurrent generation steps, which may benefit streaming LLM agents in real-time settings.
  • A key open question is whether a single freshest frame always suffices; tasks with partial observability or long-horizon dependencies may require the public context to carry more history, which the paper does not test.
  • The 20.5% frame reduction on Pick-Coke-Can suggests the public action-token context acts as a lightweight short-term memory; this could be developed into an explicit memory mechanism for embodied policies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Auras, an inference framework for embodied AI agents that disaggregates perception and generation, connects them through a shared public-context buffer, and executes the two stages as asynchronous pipelines on a single GPU. The authors claim that this improves inference throughput by 2.54x on average compared with sequential closed-loop execution (3.05x for auto-regressive models and 2.28x for diffusion-based models) while preserving agent accuracy at 102.7% of the sequential baseline. The method includes a hierarchical tuner that selects pipeline degrees and a skewness parameter, and the evaluation covers OpenVLA, RT2*, and several Diffusion Policy variants in simulation.

Significance. If the accuracy claim is reliable, the paper is practically significant: it demonstrates that a closed-loop sequential embodied policy can be reorganized into concurrent perception/generation pipelines without retraining, substantially increasing the 'thinking' frequency on commodity GPUs. The throughput measurements are direct and credible, covering a useful range of models, GPUs, and scaling scenarios. The public-context idea is interesting, and the paper is among the first, to my knowledge, to apply single-GPU pipeline parallelism at the perception/generation granularity for embodied agents. However, the headline accuracy claim currently rests on an underspecified public-context semantics for auto-regressive policies and on a tuning procedure that searches the same benchmarks on which accuracy is reported. The central conclusion is therefore not yet established, despite the encouraging throughput results.

major comments (3)
  1. [§6.2, Figure 10(a)] The central accuracy claim of 102.7% of sequential is reported as a single point with no error bars, no number of evaluation episodes, and no seed variance. The Simpler and robomimic benchmarks are stochastic, so this is not a deterministic comparison. More importantly, the hierarchical tuner (§4.3.2) chooses pp_perception, pp_generation, alpha, and fetch_offset by grid search, and §6.6 states that the boxed configurations are then forwarded to the simulation platform to test accuracy. Because the same simulation benchmarks are used for tuning and for the reported accuracy, 102.7% is a post-search best, not an unbiased estimate of the deployed policy. Please report per-configuration accuracy with variance and either use held-out tasks for tuning or demonstrate insensitivity of accuracy to the chosen configuration.
  2. [§4.2.2 and §4.2.3] The public-context update rule for auto-regressive models is not well-defined. The text says X_A is 'updated by the concurrent requests' and that each request 'leverages a prefill from [X_V, X_L, X_A^{1→i}] to update X_A^{i+1}', but it never specifies how tokens from multiple concurrent requests with different iteration counts are appended or merged, whether X_A is reset or filtered when X_V is refreshed, or what the initial X_A is. This is load-bearing because the next-token distribution of an auto-regressive policy depends on the exact prefix. The causal-masking argument in §4.2.3 only justifies that a hidden state is prefix-determined within one fixed token sequence; it does not define the merged sequence when requests interleave. Without these semantics, the 102.7% accuracy number is not reproducible as the accuracy of a well-defined policy.
  3. [§4.2.4 and Figure 8] The comparison against the sequential baseline is internally inconsistent. The text explains the sequential OpenVLA failure by saying 'the original sequential computation does not incorporate the X_A into computation', but §3.1 and Eq. (1) define auto-regressive generation as conditioning on previously generated action tokens X_A. If the baseline omits X_A, it is not the standard OpenVLA policy; if it includes X_A, then the gain in Figure 8 conflates adding action-history feedback with the proposed fresh-perception mechanism. The §6.2 accuracy comparison needs an exact definition of the baseline and an ablation in which the sequential policy is given the same action history, so that the 102.7% claim can be attributed to Auras rather than to an asymmetric comparison.
minor comments (4)
  1. [§4.3.2] The grid-search ranges for pp_perception, pp_generation, alpha, and the upper bound L are not specified. Please give the exact search space and selection criterion for reproducibility.
  2. [Figures 5, 12, 13] These tuning curves and accuracy points are shown without error bars or run counts. State how many episodes/seeds each accuracy point uses so that the apparent non-monotonic trends can be assessed.
  3. [Table 2 and §6.6] TinyVLA is marked as 'No' in the Open-sourced column but is used as a case study in §6.6. Clarify whether the model was obtained under a different release, or whether this is only a profiling study.
  4. [§4.2.4] The claim that Auras 'reduce[s] the number of frames by 20.5%' is stated only for successful tasks. Report the number of successful tasks and how the percentage is computed.

Circularity Check

0 steps flagged

No significant circularity: the reported throughput and accuracy are empirical results, not reductions of outputs to fitted constants or self-citations.

full rationale

The paper's central claims (2.54x throughput improvement and 102.7% of sequential accuracy) are empirical measurements of a system whose pipeline parameters (pp_perception, pp_generation, fetch_offset, alpha) are selected via a grid search on the same simulation benchmarks where accuracy is reported. This introduces a selection-bias or overfitting risk, but it is not circular in the definitional sense: the accuracy numbers are not derived from the fitted parameters by construction; they are measured outcomes of running the simulator. The computation merging for auto-regressive models relies on the legitimate causal-masking property of transformer architectures, which the paper explicitly states and which does not depend on the paper's own prior work. There are no load-bearing self-citations, uniqueness theorems imported from the authors, or ansatz smuggled in via citation. The public-context design is presented as a system design choice rather than a prediction that reduces to its inputs. Thus, while the evaluation methodology may warrant caution regarding generalization, there is no circular derivation in the paper.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The public context buffer is a software design component, not a newly postulated physical entity; it has no independent empirical handle beyond the system's own measurements. The free parameters are system tuning knobs fit to the evaluation benchmarks.

free parameters (5)
  • pp_perception (pipeline degree of perception) = grid-searched per model (values not listed)
    Number of stages into which the perception module is split; chosen to maximize throughput/accuracy in the evaluation simulator.
  • pp_generation (pipeline degree of generation) = grid-searched per model (values not listed)
    Number of stages for the generation phase; controls both concurrency and data freshness.
  • fetch_offset = 0 for diffusion, -1 for auto-regressive
    Frame offset at which generation fetches public context; chosen by hand per algorithm type in §4.3.1.
  • skewness weight alpha = alpha=1 used for diffusion in Fig. 13
    Skews the distribution of compute among generation pipeline stages to favor fresher context; tuned for accuracy/throughput tradeoff in §6.4.
  • upper bound L for request frames = not specified
    Limit on the number of frames a request may span, used to prune the grid search in §4.3.2.
axioms (5)
  • standard math Causal masking in transformer decoders allows computing hidden states for earlier token positions from a longer prefill (prefix property).
    Used to justify merged generation for auto-regressive models in §4.2.3; standard property of causal attention.
  • domain assumption A policy trained for closed-loop sequential execution remains accurate when conditioned on a continuously updated public context, without fine-tuning.
    Stated in §4.2.2/§4.2.4 where concurrent requests share public context; no weight modifications are made.
  • domain assumption The most recent observation embedding is sufficient conditioning for generation; older observations can be discarded without harming accuracy.
    Underlies the public-context design and the fetch_offset mechanism in §4.3.1.
  • domain assumption Simulation environments used for tuning and evaluation (Simpler, Push-T, robomimic) are faithful predictors of relative agent accuracy.
    All accuracy results are measured in simulation; no real-robot validation is presented.
  • domain assumption CUDA graphs and multiple CUDA streams on a single consumer GPU can execute perception and generation stages concurrently without correctness issues.
    Implementation choice in §5; interference is claimed to be controlled but is not formally guaranteed.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution." pith.science (2026). https://pith.science/paper/7YBO6BSO

@misc{pith2026250909560,
  author       = {Pith},
  title        = {Pith review of: Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7YBO6BSO}},
  note         = {Machine review of arXiv:2509.09560}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Embodied AI systems operate in dynamic environments, requiring seamless integration of perception and generation modules to process high-frequency input and output demands. Traditional sequential computation patterns, while effective in ensuring accuracy, face significant limitations in achieving the necessary "thinking" frequency for real-world applications. In this work, we present Auras, an algorithm-system co-designed inference framework to optimize the inference frequency of embodied AI agents. Auras disaggregates the perception and generation and provides controlled pipeline parallelism for them to achieve high and stable throughput. Faced with the data staleness problem that appears when the parallelism is increased, Auras establishes a public context for perception and generation to share, thereby promising the accuracy of embodied agents. Experimental results show that Auras improves throughput by 2.54x on average while achieving 102.7% of the original accuracy, demonstrating its efficacy in overcoming the constraints of sequential computation and providing high throughput.

Figures

Figures reproduced from arXiv: 2509.09560 by Ao Xu, Haibin Lin, Han Zhao, Minyi Guo, Ningxin Zheng, Quan Chen, Shulai Zhang, Weihao Cui, Xin Liu.

Figure 1
Figure 1. Figure 1: The general workflow of an embodied AI agent. nature [27, 69], where multiple rounds of generation are employed to achieve high-quality outputs [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: GPU utilization of robotic agents within a single request. The utilization is the SM Warp Occupancy profiled using Nsight system [2]. The profiled models are Diffusion Policy [15], OpenVLA [35] and TinyVLA [74] respectively. the generated response of each observation is fixed to be 7. Some mobile manipulation robots [32, 70] have a degree of freedom of 5. There also exist other embodied agents that generat… view at source ↗
Figure 4
Figure 4. Figure 4: Concurrent execution of requests with pipeline degree = 3. The agent’s computed output 𝐴𝑡 is based on the staled data 𝑂𝑡−3. 79.2%, higher than Diffusion Policy. But we still observe the inefficiency within the decode iterations (lower than 75%). We also observe the resource requirements of perception and generation are different. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: The systematic workflow of Auras. Auras first modifies the compute pattern of embodied AI algorithms to achieve satisfying agent accuracy when the par￾allelism increases. Given an embodied AI algorithm, Auras takes a thorough analysis of its compute characteristics and disaggregates the perception and generation modules within. A public context is extracted from the original compute graph to share between … view at source ↗
Figure 7
Figure 7. Figure 7: The compute graphs of the generation module. With sequential computation, the variables that request R1 use are consistent within the request, but are considered volatile when requests are concurrent ((a) and (b)). With the public context, generation steps from parallel requests (R1 and R2) can share the same latest public context for computation ((c) and (d)). instead of supporting only one request at a t… view at source ↗
Figure 8
Figure 8. Figure 8: Behaviors of OpenVLA on the Pick-Coke-Can task. (a) The original sequential execution; (b) Execution with integrating the latest public context into the computation. sequential execution in the Pick-Coke-Can scenario. Note that we do not modify any model weights through fine￾tuning. 4.3 Asynchronous Pipeline Executor Since we have disaggregated the perception and generation, we must instruct the data flow … view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of the asynchronous pipeline execu￾tor. (a) Different patterns of accessing the public context buffer. (b) A configuration of pipelines for perception and generation. The complete compute process of the request R1 is marked red. In the generation phase of R1, the four generation stages compute on different public contexts (with different depths of color) because they are in different frames. T… view at source ↗
Figure 10
Figure 10. Figure 10: The accuracy and throughput of embodied robotic agents. Accuracy: The S1-S3 represents the average success rate in three scenarios: Pick-Coke-Can, Move-Near and Open-Drawer for OpenVLA, and Sweep-wo-Exceeding, Scene-Understanding and Visual-Manipulation for RT2∗ . For Diffusion Policy models, S1-S3 represents the result on models that perform 40, 100, 160 diffusion steps on the Push-T task, respectively. … view at source ↗
Figure 11
Figure 11. Figure 11: Throughput of agents under various parallelisms. while Auras improves the throughput by 2.20×–3.29×. For diffusion models, the throughput increases with the number of worker streams because of the comparably small model sizes. However, the gain diminishes as the parallelism degree reaches 16. Auras easily surpasses the maximum achiev￾able throughput by an additional 1.07×–1.19×. We conclude that the benef… view at source ↗
Figure 13
Figure 13. Figure 13: Accuracy and throughput with different partition￾ing skewness weight for generation (pp_generation = 5) on the Push-T task. non-monotonic relationship between pipeline parameters and accuracy holds for other auto-regressive models in our experiment, necessitating the complete grid search to find the optimal numbers of pipeline stages. Fine-tuning the skewness of partition distribution [PITH_FULL_IMAGE:fi… view at source ↗
Figure 12
Figure 12. Figure 12: Throughput and accuracy tradeoff. The bench￾mark used in diffusion policy is Push-T and the benchmark used in the auto-regressive model is Pick-Coke-Can. Merged computation within frames. An additional op￾timization that helps Auras to beat unstructured parallelism is the merging of computation based on the public context. It is applied in auto-regressive models. In transformer-based generative models tha… view at source ↗
Figure 15
Figure 15. Figure 15: Speedup of Auras with different numbers of iteration steps in the generation phase. 56.7 FPS 62.5 FPS fetch_offset=0 fetch_offset=-1 [PITH_FULL_IMAGE:figures/full_fig_p012_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Throughput of Auras on TinyVLA with different configurations. DP model. This is because when generation shrinks, the per￾ception ratio expands, thereby reducing the opportunity for efficient overlapping across iterative generations. 6.6 A Case Study in Auras The open-sourced models listed in [PITH_FULL_IMAGE:figures/full_fig_p012_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Models

    cs.RO 2026-03 unverdicted novelty 7.0

    KERV integrates kinematic Kalman Filter predictions with speculative decoding in VLA models to achieve 27-37% faster inference while maintaining nearly the same task success rates.

Reference graph

Works this paper leans on

92 extracted references · 31 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Nvidia CUDA Stream Management.https://docs.nvidia.com/ cuda/cuda-runtime-api/group__CUDART__STREAM.html

    2012. Nvidia CUDA Stream Management.https://docs.nvidia.com/ cuda/cuda-runtime-api/group__CUDART__STREAM.html

    2012

  2. [2]

    Nvidia Nsight Systems.https://developer.nvidia.com/nsight- systems

    2018. Nvidia Nsight Systems.https://developer.nvidia.com/nsight- systems

    2018

  3. [3]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL]https://arxiv.org/abs/2307.09288

    Pith/arXiv arXiv 2023

  4. [4]

    Helix: A Vision-Language-Action Model for Generalist Hu- manoid Control.https://www.figure.ai/news/helix

    2025. Helix: A Vision-Language-Action Model for Generalist Hu- manoid Control.https://www.figure.ai/news/helix

    2025

  5. [5]

    NVIDIA Isaac GR00T.https://developer.nvidia.com/isaac/gr00t

    2025. NVIDIA Isaac GR00T.https://developer.nvidia.com/isaac/gr00t

    2025

  6. [6]

    Gulavani, and Ramachandran Ramjee

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwa- tra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. (2023).https://www.microsoft.com/en- us/research/publication/sarathi-efficient-llm-inference-by- piggybacking-decodes-with-chunked-prefills/

    2023

  7. [7]

    João Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. 2023. CIRCLE: Capture in Rich Contextual Environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 21211–21221

    2023

  8. [8]

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. 2024. RT-H: Action Hierarchies using Language. In https://arxiv.org/abs/2403.01823

    Pith/arXiv arXiv 2024

  9. [9]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Her- bie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al . 2023. Pythia: A suite for analyzing large language models across training and scaling. InInternational Conference on Machine Learning. PMLR, 2397–2430

    2023

  10. [10]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. 2024. 𝜋0: A Vision-Language-Action Flow Model for General Robot Control

    2024

  11. [11]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Haus- man, Alex Herzog, Jasmine Hsu, et al. 2022. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817(2022)

    Pith/arXiv arXiv 2022

  12. [12]

    Branden Butler, Sixing Yu, Arya Mazaheri, and Ali Jannesari. 2024. PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation. InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–19. doi:10.1109/ SC41406.2024.00046

    Pith/arXiv arXiv 2024

  13. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  14. [14]

    Xi Chen and Xiao Wang. 2022. PaLI: Scaling Language-Image Learning in 100+ Languages. InConference on Neural Information Processing Systems (NeurIPS)

    2022

  15. [15]

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Ben- jamin Burchfiel, Russ Tedrake, and Shuran Song. 2023. Diffusion policy: Visuomotor policy learning via action diffusion.The Interna- tional Journal of Robotics Research(2023), 02783649241273668

    2023

  16. [16]

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, and Chunhua Shen. 2024. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv:2402.03766 [cs.CV]https://arxiv. org/abs/2402.03766

    Pith/arXiv arXiv 2024

  17. [17]

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied Question Answering. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1–10. doi:10.1109/CVPR.2018.00008

    arXiv 2018

  18. [18]

    Sudeep Dasari, Oier Mees, Sebastian Zhao, Mohan Kumar Srirama, and Sergey Levine. 2024. The Ingredients for Robotic Diffusion Trans- formers. arXiv:2410.10088 [cs.RO]https://arxiv.org/abs/2410.10088

    Pith/arXiv arXiv 2024

  19. [19]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

  20. [20]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2020

  21. [21]

    Hugging Face. 2023. Loading Big Models into Memory.https: //huggingface.co/docs/accelerate/usage_guides/big_modeling.https: //huggingface.co/docs/accelerate/usage_guides/big_modeling

    2023

  22. [22]

    Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. 2024. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism.arXiv preprint arXiv:2411.01738(2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Google. 2023. Bard, an experiment by Google.https://bard.google. com/

    2023

  24. [24]

    Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. 2022. Stochastic trajectory prediction via motion indeterminacy diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17113–17122

    2022

  25. [25]

    Ozgur Guldogan, Jackson Kunde, Kangwook Lee, and Ramtin Pedarsani. 2024. Multi-Bin Batching for Increasing LLM Inference Throughput. arXiv:2412.04504 [cs.CL]https://arxiv.org/abs/2412. 13 Shulai Zhang, Ao Xu, Quan Chen, Han Zhao, Weihao Cui, Ningxin Zheng, Haibin Lin, Xin Liu, and Minyi Guo 04504

    Pith/arXiv arXiv 2024

  26. [26]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition. 770–778

    2016

  27. [27]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. InProceedings of the 34th International Confer- ence on Neural Information Processing Systems (NIPS ’20). Article 574, 12 pages

    2020

  28. [28]

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool.ArXivabs/2406.17565 (2024). https://api.semanticscholar.org/CorpusID:270711067

    Pith/arXiv arXiv 2024

  29. [29]

    Abbeel, Deepak Pathak, and Igor Mordatch

    Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch

  30. [30]

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei- Fei. 2024. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652 (2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Xia, Ted Xiao, Harris Chan, Jacky Liang, Peter R

    Wenlong Huang, F. Xia, Ted Xiao, Harris Chan, Jacky Liang, Peter R. Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Cheb- otar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022. Inner Monologue: Embodied Reasoning through Planning with Language Models.ArXiv abs/2207.05608 (2022).https...

    Pith/arXiv arXiv 2022

  32. [32]

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. 2023. VIMA: General Robot Manipulation with Multimodal Prompts. InFortieth International Conference on Machine Learning

    2023

  33. [33]

    Shiyu Jin, Jinxuan Xu, Yutian Lei, and Liangjun Zhang. 2024. Rea- soning Grasping via Multimodal Large Language Model.ArXiv abs/2402.06798 (2024).https://api.semanticscholar.org/CorpusID: 267627619

    Pith/arXiv arXiv 2024

  34. [34]

    Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. 2023. DALL- E-Bot: Introducing Web-Scale Diffusion Models to Robotics.IEEE Robotics and Automation Letters8, 7 (2023), 3956–3963. doi:10.1109/ LRA.2023.3272516

    arXiv 2023

  35. [35]

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pan- nag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn

  36. [36]

    Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. InProceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, Melbourne, Aus- tralia, 66–75. doi:10....

    doi:10.18653/v1/p18-1007 2018

  37. [37]

    Gonzalez, Haotong Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention.Proceedings of the 29th Symposium on Operating Systems Principles(2023).https://api.semanticscholar.org/ CorpusID:261697361

    2023

  38. [38]

    Kailin Li, Jingbo Wang, Lixin Yang, Cewu Lu, and Bo Dai. 2024. Sem- Grasp: Semantic Grasp Generation via Language Aligned Discretiza- tion. InComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part II(Milan, Italy). Springer-Verlag, Berlin, Heidelberg, 109–127. doi:10.1007/978-3-031- 72627-9_7

    doi:10.1007/978-3-031- 2024

  39. [39]

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. 2024. Distrifusion: Distributed parallel inference for high-resolution diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7183–7193

    2024

  40. [40]

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. 2024. Evaluating Real-World Robot Manipulation Policies in Simulation.arXiv preprint arXiv:2405.05941(2024)

    Pith/arXiv arXiv 2024

  41. [41]

    Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burg- ert, Mu Cai, Yong Jae Lee, et al. 2024. LLaRA: Supercharging Robot Learning Data for Vision-Language Policy.CoRR(2024)

    2024

  42. [42]

    Meta LLaMA. 2023. LLaMA 3.https://github.com/meta-llama/llama3. https://github.com/meta-llama/llama3

    2023

  43. [43]

    Ruilong Ma, Jingyu Wang, Qi Qi, Xiang Yang, Haifeng Sun, Zirui Zhuang, and Jianxin Liao. 2023. Poster: PipeLLM: Pipeline LLM Infer- ence on Heterogeneous Devices with Sequence Slicing. InProceedings of the ACM SIGCOMM 2023 Conference(New York, NY, USA)(ACM SIGCOMM ’23). Association for Computing Machinery, New York, NY, USA, 1126–1128. doi:10.1145/3603269.3610856

    arXiv 2023

  44. [44]

    Xiao Ma, Sumit Patidar, Iain Haughton, and Stephen James. 2024. Hi- erarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 18081–18090

    2024

  45. [45]

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sri- ram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Olek- sandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, a...

    2024

  46. [46]

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. 2021. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. InarXiv preprint arXiv:2108.03298

    Pith/arXiv arXiv 2021

  47. [47]

    Oier Mees, Dibya Ghosh, Karl Pertsch, Kevin Black, Homer Rich Walke, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, et al. 2024. Octo: An Open-Source Generalist Robot Policy. InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024

    2024

  48. [48]

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2024. EmbodiedGPT: vision-language pre-training via embodied chain of thought. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY...

    2024

  49. [49]

    Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved Denoising Diffusion Probabilistic Models. InProceedings of the 38th International Conference on Machine Learning. 8162–8171.https: //proceedings.mlr.press/v139/nichol21a.html

    2021

  50. [50]

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Mad- dukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al . 2023. Open x- embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864(2023)

    Pith/arXiv arXiv 2023

  51. [51]

    OpenAI. 2022. Introducing ChatGPT.https://openai.com/blog/chatgpt

    2022

  52. [52]

    OpenAI. 2023. GPT-4 Technical Report.https://openai.com/index/gpt- 4-research/ 14

    2023

  53. [53]

    Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatu...

  54. [54]

    Weiguang Pang, Xiantong Luo, Kailun Chen, Dong Ji, Lei Qiao, and Wang Yi. 2023. Efficient CUDA stream management for multi-DNN real-time inference on embedded GPUs.Journal of Systems Architecture 139 (2023), 102888

    2023

  55. [55]

    Bhrij Patel, Vishnu Sashank Dorbala, Amrit Singh Bedi, and Di- nesh Manocha. 2024. Multi-LLM QA with Embodied Exploration. arXiv:2406.10918 [cs.LG]https://arxiv.org/abs/2406.10918

    Pith/arXiv arXiv 2024

  56. [56]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118–132. doi:10.1109/ISCA59077.2024.00019

    arXiv 2024

  57. [57]

    Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Per- ception, Decision, and Reaction.arXiv preprint arXiv:2501.03218 (2025)

    Pith/arXiv arXiv 2025

  58. [58]

    Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik

  59. [59]

    Sai Ramana Kiran Pinnama Raju, Rishabh Singh, Manoj Velmurugan, and Nitin J. Sanket. 2025. EdgeFlowNet: 100FPS@1W Dense Optical Flow for Tiny Mobile Robots.IEEE Robotics and Automation Letters 10, 1 (2025), 128–135. doi:10.1109/LRA.2024.3496336

    arXiv 2025

  60. [60]

    P. Ramarao. 2018. CUDA 10 Features Revealed: Turing, CUDA Graphs, and More. (Sept. 2018).https://devblogs.nvidia.com/cuda-10-features- revealed/

    2018

  61. [61]

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. InMed- ical Image Computing and Computer-Assisted Intervention – MICCAI

    2015

  62. [62]

    Sanket, Chethan M

    Nitin J. Sanket, Chethan M. Parameshwara, Chahat Deep Singh, Ash- win V. Kuruttukulam, Cornelia Fermuller, Davide Scaramuzza, and Yiannis Aloimonos. 2019. EVDodge: Embodied AI For High-Speed Dodging On A Quadrotor Using Event Cameras

    2019

  63. [63]

    Humanoid locomotion as next token prediction.arXiv preprint arXiv:2402.19469(2024)

    Pith/arXiv arXiv 2024

  64. [64]

    Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5149–5152. doi:10.1109/ICASSP.2012. 6289079

    doi:10.1109/icassp.2012 2012

  65. [65]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. InProceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Katrin Erk and Noah A. Smith (Eds.). Association for Computational Linguistics, Berlin, Germany, 1715–1725. doi:10.18653/v1...

    doi:10.18653/v1/p16-1162 2016

  66. [66]

    Bokui (William) Shen, Ge Yang, Alan Yu, Jan Rang Wong, Leslie Pack Kaelbling, and Phillip Isola. 2023. Distilled Feature Fields Enable Few- Shot Language-Guided Manipulation. InConference on Robot Learning. https://api.semanticscholar.org/CorpusID:260926035

    2023

  67. [67]

    Weiss, Niru Maheswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilib- rium thermodynamics. InProceedings of the 32nd International Con- ference on International Conference on Machine Learning - Volume 37 (Lille, France)(ICML’15). JMLR.org, 2256–2265

    2015

  68. [68]

    P. Schmid. 2023. Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers.https://www.philschmid.de/fine-tune- flan-t5-deepspeed.https://www.philschmid.de/fine-tune-flan-t5- deepspeedOnline

    2023

  69. [69]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InProceedings of the 31st International Con- ference on Neural Information Processing Systems (NIPS’17). 6000–6010

    2017

  70. [70]

    Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. 2023. Bridgedata v2: A dataset for robot learning at scale. InConference on Robot Learning. PMLR, 1723–1736

    2023

  71. [71]

    Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. 2024. PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Dif- fusion Transformer Models.arXiv preprint arXiv:2405.14430(2024)

    Pith/arXiv arXiv 2024

  72. [72]

    P. Wang. 2023. X-transformers.https://github.com/lucidrains/x- transformers.https://github.com/lucidrains/x-transformersGitHub repository

    2023

  73. [73]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL]https://arxiv. org/abs/2302.13971

    Pith/arXiv arXiv 2023

  74. [74]

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, et al. 2024. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation.arXiv preprint arXiv:2409.12514(2024)

    Pith/arXiv arXiv 2024

  75. [75]

    Zhou Xian, Nikolaos Gkanatsios, Theophile Gervet, Tsung-Wei Ke, and Katerina Fragkiadaki. 2023. ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation. In7th Annual Conference on Robot Learning.https://openreview.net/forum? id=W0zgY2mBTA8

    2023

  76. [76]

    Yecheng Xiang and Hyoseung Kim. 2019. Pipelined data-parallel CPU/GPU scheduling for multi-DNN real-time inference. In2019 IEEE Real-Time Systems Symposium (RTSS). IEEE, 392–405

    2019

  77. [77]

    Yida Yin, Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roei Herzig. 2024. In-Context Learning Enables Robot Action Prediction in LLMs.arXiv preprint arXiv:2410.12782(2024)

    Pith/arXiv arXiv 2024

  78. [78]

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, Yitao Liang, and Team CraftJarvis. 2024. Describe, explain, plan and select: interactive planning with large language models enables open- world multi-task agents. InProceedings of the 37th International Con- ference on Neural Information Processing Systems (NIPS ’23). Article 1480, 37 pages

    2024

  79. [79]

    Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Marcin Choroman- ski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. 2023. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. InThe Eleventh International Conference on Learning Represen...

    2023

  80. [80]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. arXiv:2303.15343 [cs.CV]https://arxiv.org/abs/2303.15343 15 Shulai Zhang, Ao Xu, Quan Chen, Han Zhao, Weihao Cui, Ningxin Zheng, Haibin Lin, Xin Liu, and Minyi Guo

    Pith/arXiv arXiv 2023

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.