Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an 8B multimodal language model can drive a quadruped robot at the full 50Hz controller frequency—without pruning or distilling the model—by emitting compressed action-chunk codes that a small fixed decoder expands…

desk verdict A plausible and clearly explained method for cutting MLLM latency in quadruped control, but the headline 50Hz and 65% numbers overstate what the experiments actually show. read the letter →

arxiv 2412.15576 v5 pith:6VYWRINO submitted 2024-12-20 cs.RO cs.CV

Xinyang Tong , Pengxiang Ding , Yiguo Fan , Donglin Wang , Wenjie Zhang , Can Cui , Mingyang Sun , Han Zhao
show 4 more authors
Hongyin Zhang Yonghao Dang Siteng Huang Shangke Lyu
This is my paper · ORCID
classification cs.ROcs.CV
keywords quadrupedrobotlearningvision-language-actionmodelmultimodallargelanguageactionchunkdiscretizationresidualvectorquantizationlatency-freeinferencereal-timecontrolQUAR-VLA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to remove the inference-latency barrier that keeps large multimodal language models (MLLMs) from acting as real-time quadruped policies. Its claim is that the bottleneck is the action representation, not the model size: tokenizing every action dimension into 256 bins forces the MLLM to emit many tokens per frame and to plan only one step ahead. QUART-Online compresses whole chunks of continuous 11-dimensional action commands into a few discrete codes with Action Chunk Discretization (ACD), has the MLLM predict those codes, and lets a small fixed decoder reconstruct the full multi-step trajectory. With a ten-step chunk the system runs at the controller's 50Hz, and the paper reports a 65% average success-rate improvement over QUART, concentrated in tasks with unseen visual elements and language instructions.

What carries the argument

The central object is Action Chunk Discretization (ACD), a two-stage pipeline. First, a 1D-convolutional temporal encoder maps an N-step chunk of the 11 high-level command dimensions plus a termination signal into a latent sequence; a residual quantizer with Nq=2 codebook layers (each with K=512 D-dimensional vectors) turns it into compact discrete codes; and a fixed decoder expands the codes back into the continuous action chunk. Second, the frequency-matching identity fl = lac × fm fixes the chunk length lac so that the MLLM's inference rate fm multiplied by the chunk length equals the low-level controller's rate fl, which in the paper's experiments is 10 × 5Hz = 50Hz.

What would settle it

Take the fixed ACD decoder and run it on action trajectories predicted by the fine-tuned MLLM for QUARD's unseen-language instructions, computing MAE against the demonstrated ground truth. If the error is close to the 0.012 reported for in-distribution chunk-10 data, the compression claim holds; if it is several times larger, the decoder does not generalize to the novel conditions that carry the reported 65% success gain.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its own terms, is that discretizing actions in a learned latent space—rather than in the raw per-frame 256-bin space—makes an 8B MLLM controllable at full frequency without losing the foundation model's generalization. A temporal convolutional encoder, a two-layer residual vector quantizer (K=512 codes per layer, Nq=2), and a fixed decoder cut the number of tokens the MLLM must emit by roughly 2.5x, and predicting N=10 action steps at once adds a further 10x frequency multiplier, moving QUART-Online from QUART's 2Hz to 50Hz. On the QUARD benchmark the success rate rises with chunk length—0.28 average at length 1, 0.47 at length 5, and 0.68 at length 10, against 0.37-0.52 for QUART over the same conditions. The authors attribute the improvement to compressed action tokens carrying semantic meaning that can be fine-tuned jointly with language and vision without disturbing the model's existing distribution.

Load-bearing premise

The load-bearing premise is that two layers of 512 residual code vectors can faithfully reconstruct every action trajectory the system will need, including trajectories for unseen objects and instructions never present when the codebook and decoder were trained.

Editorial extensions

If this is right

  • At chunk length 10 the MLLM emits only a handful of code tokens per chunk instead of one 256-bin token per action dimension per frame, a roughly 2.5x reduction in output tokens before the 10x chunk multiplier.
  • Success rates on the QUARD benchmark rise monotonically with chunk length, so planning over longer temporal horizons improves both reconstruction fidelity and decision quality.
  • The recipe transfers to any MLLM robot system: choose action chunk length so that chunk length times model inference frequency equals controller frequency, and the large model becomes zero-latency.
  • Because the MLLM weights are untouched, the foundation model's emergent abilities—understanding novel commands, generalizing to unseen objects—are preserved, which the paper argues explains the large gains on unseen conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Table III validates ACD reconstruction on only 15% held-out in-distribution action data, so the fixed decoder's fidelity on trajectories arising from the unseen visual and language conditions—where the reported gains are largest—is assumed rather than measured.
  • Editorial inference: The 65% gain fuses two variables, the action representation and the control frequency; an ablation that feeds the same MLLM continuous raw chunks at 50Hz would separate the representation benefit from the pure frequency benefit.
  • Editorial inference: The paper's own limitations state that outputs remain high-level commands needing a low-level controller and that only simple terrains were tested; applying ACD to joint-level output would test whether the compression idea extends to higher-dimensional action spaces.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes QUART-Online, an extension of the QUART quadruped vision-language-action model, in which continuous action commands are compressed via residual vector quantization into discrete codebook codes that an MLLM outputs, and a fixed decoder reconstructs multi-step action chunks. The authors report that this design enables real-time inference at 50Hz and improves average task success by 65% over QUART on the QUARD benchmark. The paper also claims that parameter-reduction techniques degrade the MLLM's generalization, while action chunk discretization avoids this by preserving the model's semantic capacities.

Significance. If the claims were fully supported, ACD would be a practical contribution: it would allow an 8B multimodal model to generate high-frequency action outputs without parameter pruning or architecture changes, using a compact discrete action space. The paper reports a substantial empirical gain and includes a real-robot illustration. However, the central claims are currently undermined by a conflation between action output frequency and MLLM inference frequency, by a missing ablation that separates chunk-length effects from latency effects, and by several inconsistencies in the reported losses and evaluations. The core idea is interesting and potentially publishable, but the evidence as presented does not yet establish the headline result.

major comments (5)
  1. [Section III-D, Eq. (10); Table II] The paper's headline claim of 'real-time inference at 50Hz' is not supported by its own equations. Equation (10) fixes the relationship fl = lac × fm; for the reported operating point (lac=10, fl=50Hz), the MLLM inference frequency fm is 5Hz, not 50Hz. Table II confirms that QUART-Online-1, -5, and -10 all share the same 5Hz MLLM update rate; only the action decoder's output rate changes with chunk length. The phrase 'inference at 50Hz' therefore conflates action execution rate with perception-decision frequency. Please revise the claim to distinguish '50Hz action output' from '5Hz MLLM inference', or provide a different definition of 'inference' that is consistent with Eq. (10).
  2. [Table II and abstract] The reported '65% improvement' over QUART is confounded with action chunk length. All QUART-Online variants share the same 5Hz MLLM inference rate, yet unseen-task success increases sharply with chunk length: the Uv average rises from 0.28 (chunk 1) to 0.47 (chunk 5) to 0.68 (chunk 10), and the Ul average from 0.57 to 0.62 to 0.79. QUART-Online-1's average success (0.425) is actually slightly below QUART's (0.445). Thus the gain attributed to 'latency-free 50Hz operation' is, on the present evidence, driven by the longer action chunk itself, not by reduced inference latency. A raw-action chunk-10 baseline and an ablation that holds chunk length fixed while varying the MLLM update rate are essential to separate these effects.
  3. [Section III-C, Eq. (8)] The text states that 'we leverage a combination of reconstruction loss and adversarial loss for ACD', but Eq. (8) and the implementation details contain only Lrec + Lcom. No adversarial loss term appears, and no generator/discriminator architecture, adversarial training schedule, or loss weight is described. Either add the adversarial loss term and its training details, or remove the claim from the paper.
  4. [Section IV-B, Table III] Reconstruction fidelity is evaluated only on the 15% held-out portion of the in-distribution action dataset. The success-rate improvements in Table II are concentrated in the unseen-visual (Uv) and unseen-language (Ul) columns, yet the paper provides no evidence that the fixed decoder reconstructs action chunks under these out-of-distribution conditions. Since the decoder is frozen during Stage 2, its ability to generalize to novel visual and language contexts is load-bearing for the reported success gains; Table III does not measure it.
  5. [Section IV-A and Table II] The paper reports 50 experiments per task but gives no variance, confidence intervals, or statistical tests. For binomial success rates near 0.5, the standard error over 50 trials is approximately 0.07, so many pairwise differences in Table II (for example, between QUART and QUART-Online-1 on several tasks) are not statistically distinguishable. In addition, the statement 'We have addressed previous bugs in the QUART evaluation environment' is undocumented. Because QUART is the primary baseline, an unspecified change to the evaluation setup could affect every comparison. Please specify the bugs, report confidence intervals, or release the evaluation code.
minor comments (4)
  1. [Section IV-A, Implementation Details] The sentence 'The quantization network features a codebook embedding layer with 512 dimensions and 512 quantizers (Nq = 2)' is ambiguous; it presumably means a codebook with 512 entries per layer and Nq=2 residual layers. Please rephrase to state the number of codebook entries and the number of quantization layers explicitly.
  2. [Section III-D] The term 'latency-free' overstates the system's behavior: Eq. (10) describes frequency matching between the action output and the low-level controller, but the MLLM still introduces a 200ms decision interval at the reported operating point. Consider using 'frequency-matched' or 'output-frequency-synchronized' instead.
  3. [Figure 2 caption] The caption claims a '2.5x' token reduction and a '10x' chunk-based speed-up, but does not account for the overhead of the action encoder/decoder. Please state the measured end-to-end latency, including MLLM plus decoder, to justify the speed-up claims.
  4. [Section IV-A, Evaluation] The manuscript does not state whether the reported results are averaged over multiple training runs or over fixed seeds. A brief statement about seed handling and the number of evaluation episodes per task would improve reproducibility.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild self-definitional issue: the 50Hz 'real-time inference' claim is the controller frequency fl defined by Eq. (10), not the MLLM inference rate; otherwise no load-bearing circularity.

  1. self definitional [Abstract; Section III-D, Eq. (10); Table II (QUART-Online-10 50HZ)]
    "achieving real-time inference at 50Hz in sync with the underlying controller frequency ... This is accomplished when the action chunk length lac multiplied by the frequency of MLLM inference fm matches the frequency of low-level control fl ... fl = lac × fm. (10)"

    The headline 'real-time inference at 50Hz' is not an independently measured inference rate: Eq. (10) defines the output/controller frequency as fl = lac × fm. At the reported 50Hz operating point, lac = 10 and fm = 5Hz, so 50Hz is the decoder/controller rate obtained by multiplying the 5Hz MLLM inference rate by the chunk length. QUART-Online-1, -5, and -10 all run at the same 5Hz MLLM command-update rate, so the 'latency-free' property is true by construction once the chunk length is chosen as fl/fm. The 50Hz claim is thus a restatement of the chosen chunk length rather than a measured inference result.

full rationale

The paper's main contribution is empirical: ACD compresses actions into discrete codes, an MLLM is fine-tuned to emit those codes, and a fixed decoder expands them into trajectories. Success rates are measured against external baselines (VLA-CLIP, VLA-VC-1) as well as the prior QUART model, and the ACD reconstruction quality is evaluated on a held-out 15% split of action data, so the central performance claim is not derived from the inputs. There is no load-bearing uniqueness theorem, no ansatz smuggled in solely by self-citation, and no fitted parameter renamed as a prediction. The one definitional reduction is the '50Hz real-time inference' label: Eq. (10) defines output frequency as chunk length times MLLM frequency, so at lac = 10 the 50Hz figure is the controller/decoder rate, not the MLLM inference rate. The abstract's '65% improvement' is also confounded with chunk length, because QUART-Online-1, -5, and -10 share the same 5Hz decision rate and the comparison with QUART changes both the action representation and the chunk length; however, this is an experimental-confounding and attribution concern, not a logical circularity. The manuscript's own stated limitations (actions still require low-level control; not tested in more complex terrains) further show that the authors do not claim a derivation where the conclusion is presupposed. Overall circularity is low: one self-definitional labeling issue, with the empirical substance intact.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four purchased assumptions: an external low-level controller that can execute high-level commands at 50Hz, a benchmark environment whose undocumented bug fixes keep prior baselines comparable, a stage-2 fine-tuning objective that aligns compressed codes with language and vision, and a codebook learned on in-distribution action data that generalizes to unseen instructions and objects. The hand-chosen chunk length, codebook size, quantizer layers, and lookahead steps are free parameters that shape the reported 50Hz operation and success rates. No new physical entity is introduced; the latent action codes are a learned representation without an independent falsifiable handle.

free parameters (4)
  • Action chunk length lac = 10 (also 1 and 5 in ablations)
    Equation (10) sets controller frequency fl = lac * fm; the latency-free 50Hz claim and the best success rates use lac=10. The choice is a design parameter, not derived.
  • Codebook size K and latent dimension D = 512 and 512
    Section IV-A; capacity of the discrete action code space is hand-set and directly bounds how much action information survives compression.
  • Residual quantization layers Nq = 2
    Section IV-A; compression depth is hand-set; it affects reconstruction fidelity and the number of code indices the MLLM must predict.
  • Lookahead action steps N and conv layers = N=5 or 10, 3 conv layers at N=5
    Section IV-A; the number of future frames encoded per chunk determines temporal context and reconstruction metrics in Table III.
assumptions (4)
  • domain assumption Low-level controller [39] reliably translates high-level commands into joint-level motion at 50Hz.
    The method outputs high-level commands (Eq. 1), not joint angles; Limitations admits actions still require a low-level controller. If the controller is unavailable or inaccurate, the 50Hz latency-free result does not transfer.
  • domain assumption QUARD benchmark and the updated simulation environment are valid and comparable across methods.
    Section IV-A says previous bugs in the QUART evaluation environment were addressed without specifying the fixes; baseline and reported numbers are assumed to be comparable.
  • domain assumption Stage-2 cross-entropy fine-tuning maps compressed action codes into the MLLM semantic space without degrading generalization to unseen instructions and objects.
    Section III-D; the paper claims a unified semantic space but provides no probing or ablation showing the codes carry semantic meaning; gains may instead come from execution frequency.
  • domain assumption Action codebook learned on 85% of QUARD action data generalizes to out-of-distribution trajectories encountered at test time.
    Section IV-B evaluates reconstruction only on 15% held-out action data; success on unseen visual and language tasks assumes the decoder and codebook cover novel command distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning." pith.science (2026). https://pith.science/paper/6VYWRINO

@misc{pith2026241215576,
  author       = {Pith},
  title        = {Pith review of: QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VYWRINO}},
  note         = {Machine review of arXiv:2412.15576}
}
read the original abstract

This paper addresses the inherent inference latency challenges associated with deploying multimodal large language models (MLLM) in quadruped vision-language-action (QUAR-VLA) tasks. Our investigation reveals that conventional parameter reduction techniques ultimately impair the performance of the language foundation model during the action instruction tuning phase, making them unsuitable for this purpose. We introduce a novel latency-free quadruped MLLM model, dubbed QUART-Online, designed to enhance inference efficiency without degrading the performance of the language foundation model. By incorporating Action Chunk Discretization (ACD), we compress the original action representation space, mapping continuous action values onto a smaller set of discrete representative vectors while preserving critical information. Subsequently, we fine-tune the MLLM to integrate vision, language, and compressed actions into a unified semantic space. Experimental results demonstrate that QUART-Online operates in tandem with the existing MLLM system, achieving real-time inference in sync with the underlying controller frequency, significantly boosting the success rate across various tasks by 65%. Our project page is https://quart-online.github.io.

Figures

Figures reproduced from arXiv: 2412.15576 by the authors.

Figure 1
Figure 1. Overview of QUART-Online: With the implementation of action chunk discretization, QUART-Online enhances the existing MLLM system, which was previously operating at a low frequency, enabling more precise actions to be executed in real-time at a frequency of 50Hz. TABLE I: Success rate and inference speed on the unseen multi￾task benchmark with various parameter reduction methods on QUART. Here, "P" signifies the impl… view at source ↗
Figure 2
Figure 2. Comparison of QUART and QUART-Online. QUART-Online enhances the inference process by employing two key strategies: 1) it accelerates MLLM inference by generating a reduced number of tokens in the latent space as opposed to the raw space (2.5x); 2) it introduces an action chunk mechanism during the action decoding phase, facilitating higher-frequency inference via multi-step predictions (10x). By integrating these tw… view at source ↗
Figure 3
Figure 3. Overall framework of QUART-Online. while efficiently managing computational complexity: Aq =Quantize(Ah,B). (5) Dequantization. The dequantize process is the inverse operation of quantization. Dequantization commences at the apex with layer BNq and descends to the foundational layer B1. Utilizing indices from Aq, it extracts code vectors successively from each layer, aggregating them through a process of accumulatio… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The top half of the comparison highlights the QUART method’s latency-induced collision with the red bar (red highlight). The [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Experiments in the real world. locomotion scenarios. Ensuring that our efficient inference model can adapt to various complex terrains and further tapping into its potential will be the focus of our next phase of work. VI. CONCLUSION This paper addresses the challenge …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver

    cs.RO 2025-08 unverdicted novelty 4.0 of 10

    Adding a reconstruction target that redraws the object region makes a vision-language-action model focus its attention on the right object and manipulate more precisely.

Reference graph

Works this paper leans on

41 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Saytap: Language to quadrupedal locomotion,

    Y . Tang, W. Y u, J. Tan, H. Zen, A. Faust, and T. Harada, “Saytap: Language to quadrupedal locomotion,”arXiv preprint arXiv:2306.07580, 2023

    arXiv 2023

  2. [2]

    Vinl: Visual navigation and locomotion over obstacles,

    S. Kareer, N. Y okoyama, D. Batra, S. Ha, and J. Truong, “Vinl: Visual navigation and locomotion over obstacles,” 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2018–2024, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:253117178

    work page 2023

  3. [3]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,

    H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P . Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 807–11 814, 2022

    work page 2022

  4. [4]

    Quar-vla: Vision-language-action model for quadruped robots,

    P . Ding, H. Zhao, Z. Wang, Z. Wei, S. Lyu, and D. Wang, “Quar-vla: Vision-language-action model for quadruped robots,” arXiv preprint arXiv:2312.14457, 2023

    arXiv 2023

  5. [5]

    Barkour: Benchmarking animal-level agility with quadruped robots,

    K. Caluwaerts, A. Iscen, J. C. Kew, W . Y u, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Saliceti, V . Zhuanget al., “Barkour: Benchmarking animal-level agility with quadruped robots,”arXiv preprint arXiv:2305.14654, 2023

    arXiv 2023

  6. [6]

    Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers,

    R. Y ang, M. Zhang, N. Hansen, H. Xu, and X. Wang, “Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers,”ArXiv, vol. abs/2107.03996, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:235765481

    arXiv 2021

  7. [7]

    Learning advanced locomotion for quadrupedal robots: A distributed multi-agent reinforcement learning framework with riemannian motion policies,

    Y . Wang, R. Sagawa, and Y . Y oshiyasu, “Learning advanced locomotion for quadrupedal robots: A distributed multi-agent reinforcement learning framework with riemannian motion policies,”Robotics, vol. 13, no. 6, 2024. [Online]. Available: https://www.mdpi.com/2218-6581/13/6/86

    work page 2024

  8. [8]

    The surprising effectiveness of representation learning for visual imitation,

    J. Pari, N. M. Shafiullah, S. P . Arunachalam, and L. Pinto, “The surprising effectiveness of representation learning for visual imitation,” 2021

    work page 2021

Show all 41 references
  1. [9]

    Learning agent-aware affordances for closed-loop interaction with articulated objects,

    G. Schiavi, P . Wulkop, G. Rizzi, L. Ott, R. Siegwart, and J. J. Chung, “Learning agent-aware affordances for closed-loop interaction with articulated objects,” in2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 5916–5922

    2023

  2. [10]

    Affordances from human videos as a versatile representation for robotics,

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” 2023

    2023

  3. [11]

    Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks,

    M. A. Lee, Y . Zhu, K. Srinivasan, P . Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks,” in 2019 IEEE International Conference on Robotics and Automation (ICRA...

    2019 arXiv

  4. [12]

    Hydra: Hybrid robot actions for imitation learning,

    S. Belkhale, Y . Cui, and D. Sadigh, “Hydra: Hybrid robot actions for imitation learning,” inProceedings of the 7th Conference on Robot Learning (CoRL), 2023

    2023

  5. [13]

    Watch and match: Supercharging imitation with regularized optimal transport,

    S. Haldar, V . Mathur, D. Y arats, and L. Pinto, “Watch and match: Supercharging imitation with regularized optimal transport,”CoRL, 2022

    2022

  6. [14]

    Learning robust perceptive locomotion for quadrupedal robots in the wild,

    T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, 2022. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.abk2822

    2022 doi

  7. [15]

    Learning visual quadrupedal loco-manipulation from demonstrations,

    Z. He, K. Lei, Y . Ze, K. Sreenath, Z. Li, and H. Xu, “Learning visual quadrupedal loco-manipulation from demonstrations,” 2024. [Online]. Available: https://arxiv.org/abs/2403.20328

    2024 arXiv

  8. [16]

    Genloco: Generalized locomotion controllers for quadrupedal robots,

    G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Y ue, Z. Song, L. Y ang, Y . Liu, K. Sreenath, and S. Levine, “Genloco: Generalized locomotion controllers for quadrupedal robots,” 2022. [Online]. Available: https://arxiv.org/abs/2209.05309

    2022 arXiv

  9. [17]

    Learning whole-body manipulation for quadrupedal robot,

    S. Jeon, M. Jung, S. Choi, B. Kim, and J. Hwangbo, “Learning whole-body manipulation for quadrupedal robot,”IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 699–706, 2024

    2024

  10. [18]

    Long-horizon locomotion and manipulation on a quadrupedal robot with large language models,

    Y . Ouyang, J. Li, Y . Li, Z. Li, C. Y u, K. Sreenath, and Y . Wu, “Long-horizon locomotion and manipulation on a quadrupedal robot with large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2404.05291

    2024 arXiv

  11. [19]

    Lanmp: A multifaceted mobile manipulation benchmark for robots,

    A. Jaafar, S. S. Raman, Y . Wei, S. E. Juliani, A. Wernerfelt, I. Idrees, J. X. Liu, and S. Tellex, “Lanmp: A multifaceted mobile manipulation benchmark for robots,” inRSS 2024 W orkshop: Data Generation for Robotics

    2024

  12. [20]

    Learning multiple gaits within latent space for quadruped robots,

    J. Wu, Y . Xue, and C. Qi, “Learning multiple gaits within latent space for quadruped robots,” 2023. [Online]. Available: https://arxiv.org/abs/2308.03014

    2023 arXiv

  13. [21]

    Quadrupedgpt: Towards a versatile quadruped agent in open-ended worlds,

    Y . Wang, Y . Mei, S. Zheng, and Q. Jin, “Quadrupedgpt: Towards a versatile quadruped agent in open-ended worlds,”arXiv preprint arXiv:2406.16578, 2024

    2024 arXiv

  14. [22]

    Germ: A generalist robotic model with mixture-of-experts for quadruped robot,

    W. Song, H. Zhao, P . Ding, C. Cui, S. Lyu, Y . Fan, and D. Wang, “Germ: A generalist robotic model with mixture-of-experts for quadruped robot,” arXiv e-prints, pp. arXiv–2403, 2024

    2024

  15. [23]

    Efficient large language models: A survey,

    Z. Wan, X. Wang, C. Liu, S. Alam, Y . Zheng, Z. Qu, S. Y an, Y . Zhu, Q. Zhang, M. Chowdhuryet al., “Efficient large language models: A survey,” arXiv preprint arXiv:2312.03863, vol. 1, 2023

    2023 arXiv

  16. [24]

    Dynamic neural networks: A survey,

    Y . Han, G. Huang, S. Song, L. Y ang, H. Wang, and Y . Wang, “Dynamic neural networks: A survey,”IEEE Transactions on P attern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7436–7456, 2021

    2021

  17. [25]

    Llava-gemma: Accelerating multimodal foundation models with a compact language model,

    M. Hinck, M. L. Olson, D. Cobbley, S.-Y . Tseng, and V . Lal, “Llava-gemma: Accelerating multimodal foundation models with a compact language model,”arXiv preprint arXiv:2404.01331, 2024

    2024 arXiv

  18. [26]

    Moe-llava: Mixture of experts for large vision-language models,

    B. Lin, Z. Tang, Y . Y e, J. Cui, B. Zhu, P . Jin, J. Zhang, M. Ning, and L. Y uan, “Moe-llava: Mixture of experts for large vision-language models,”arXiv preprint arXiv:2401.15947, 2024

    2024 arXiv

  19. [27]

    Cobra: Extending mamba to multi-modal large language model for efficient inference,

    H. Zhao, M. Zhang, W. Zhao, P . Ding, S. Huang, and D. Wang, “Cobra: Extending mamba to multi-modal large language model for efficient inference,”arXiv preprint arXiv:2403.14520, 2024

    2024 arXiv

  20. [28]

    Linearizing large language models,

    J. Mercat, I. V asiljevic, S. Keh, K. Arora, A. Dave, A. Gaidon, and T. Kollar, “Linearizing large language models,”arXiv preprint arXiv:2405.06640, 2024

    2024 arXiv

  21. [29]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. M...

    2023

  22. [30]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P . Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. ...

    2023

  23. [31]

    Rt-h: Action hierarchies using language,

    S. Belkhale, T. Ding, T. Xiao, P . Sermanet, Q. Vuong, J. Tompson, Y . Chebotar, D. Dwibedi, and D. Sadigh, “Rt-h: Action hierarchies using language,” 2024. [Online]. Available: https://arxiv.org/abs/2403.01823

    2024 arXiv

  24. [32]

    Openvla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P . Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P . Liang, and C. Finn, “Openvla: An open-source vision-language-action model,” 2024. [Onli...

    2024 arXiv

  25. [33]

    Unleashing large-scale video generative pre-training for visual robot manipulation,

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong, “Unleashing large-scale video generative pre-training for visual robot manipulation,” 2023. [Online]. Available: https://arxiv.org/abs/2312.13139

    2023 arXiv

  26. [34]

    Vision-language foundation models as effective robot imitators,

    X. Li, M. Liu, H. Zhang, C. Y u, J. Xu, H. Wu, C. Cheang, Y . Jing, W . Zhang, H. Liu, H. Li, and T. Kong, “Vision-language foundation models as effective robot imitators,” 2024. [Online]. Available: https://arxiv.org/abs/2311.01378

    2024 arXiv

  27. [35]

    3d diffuser actor: Policy diffusion with 3d scene representations,

    T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki, “3d diffuser actor: Policy diffusion with 3d scene representations,” 2024. [Online]. Available: https://arxiv.org/abs/2402.10885

    2024 arXiv

  28. [36]

    Grounding multimodal large language models in actions,

    A. Szot, B. Mazoure, H. Agrawal, D. Hjelm, Z. Kira, and A. Toshev, “Grounding multimodal large language models in actions,” 2024. [Online]. Available: https://arxiv.org/abs/2406.07904

    2024 arXiv

  29. [37]

    Introducing our multimodal models,

    R. Bavishi, E. Elsen, C. Hawthorne, M. Nye, A. Odena, A. Somani, and S. Ta¸ sırlar, “Introducing our multimodal models,” 2023. [Online]. Available: https://www.adept.ai/blog/fuyu-8b

    2023

  30. [38]

    Isaac gym: High performance gpu-based physics simulation for robot learning,

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021

    2021

  31. [39]

    Walk these ways: Tuning robot control for generalization with multiplicity of behavior,

    G. B. Margolis and P . Agrawal, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” inProceedings of The 6th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. P...

    2023

  32. [40]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W . Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P . Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021

    2021

  33. [41]

    Where are we in the search for an artificial visual cortex for embodied intelligence?

    A. Majumdar, K. Y adav, S. Arnaud, Y . J. Ma, C. Chen, S. Silwal, A. Jain, V .-P . Berges, P . Abbeel, J. Malik, D. Batra, Y . Lin, O. Maksymets, A. Rajeswaran, and F. Meier, “Where are we in the search for an artificial visual cortex for embodied intelligence?” 2023

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.