REVIEW 4 major objections 5 minor 8 cited by
The paper claims that a hierarchical mixture-of-experts action module, with action-space experts at the input/output boundaries and balancing experts at adjacent layers, converts the negative transfer caused by heterogeneous robot data into
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 18:18 UTC pith:N7MKYG4N
load-bearing objection Genuinely new two-level MoE action module that moves CALVIN/LIBERO numbers and earns a serious referee, but the central positive-transfer claim is under-specified and the paper has a few fixable reporting inconsistencies. the 4 major comments →
HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the main obstacle to training one vision-language-action policy on diverse robot data is not the data itself but the way a shared dense action module mixes it. When action spaces differ—joint angles versus end-effector deltas, single-arm versus dual-arm—the authors show that co-training degrades performance, with a baseline dropping from 3.81 to 3.55 on a long-horizon benchmark. Their HiMoE-VLA replaces the dense module with a hierarchical mixture of experts: Action-Space MoE layers at the input and output boundaries specialize computation for each action space; Heterogeneity-Balancing MoE layers in adjacent layers absorb residual variation in embodiments, sensors,
What carries the argument
Hierarchical Mixture-of-Experts (HiMoE) action module: a layered action network in which boundary layers use Action-Space MoE to specialize per action space, adjacent layers use Heterogeneity-Balancing MoE to spread broader variation across experts, and middle dense transformer blocks consolidate shared representations. Routing uses top-4 selection among 32 experts, with a contrastive Action-Space Regularization aligning experts routed to the same action-space token and a load-balancing regularization aligning expected gating scores with realized routing frequencies. Training uses a flow-matching objective, and all states and actions are projected into a fixed 24-dimensional vector with vali
Load-bearing premise
Everything rests on the fixed 24-dimensional unified vector: if a robot's true action space cannot be squeezed into 8 end-effector plus 16 joint-angle numbers without losing information, the experts will route on a distorted input and the transfer gains may vanish.
What would settle it
Co-train HiMoE-VLA on a robot with an action space that exceeds 24 dimensions—for example a 7-DoF arm with a multi-finger hand, where joint commands need more than 16 entries—and compare against a baseline with a dedicated head for that robot. If the unified-vector version does not beat the dedicated head (or at least match the positive-transfer pattern reported for 24-dim spaces), the load-bearing premise is the fixed representation, not the MoE hierarchy.
If this is right
- If correct, one policy can be pre-trained on a large mixture of single-arm and dual-arm, joint-space and end-effector-space demonstrations without needing dataset-specific heads or manual embodiment labels.
- Co-training on heterogeneous action spaces becomes a source of improvement rather than a cause of degradation, at least within the range of the fixed 24-dimensional action/state representation.
- The hierarchy gives a concrete recipe: specialize at the boundaries, balance in adjacent layers, share in the middle; ablations show removing either MoE stage or either regularization erodes the gain.
- Performance gains on long-horizon tasks and unseen objects suggest the design supports generalization, not just fitting to the training distribution.
Where Pith is reading between the lines
- One extension the paper leaves implicit: the same boundary-specialize/middle-share pattern could be applied to other multi-domain sequence models, where the contrastive regularization would act as an automatic classifier of action or control modes.
- The fixed 24-dimensional vector suggests a scaling path: as new embodiments are added, the vector must grow and the validity masks must be re-learned; an implicit, testable prediction is that the number of experts should grow with the number of distinct action spaces rather than with dataset size.
- The load-balancing objective could be replaced by an adaptive capacity-allocation rule that lets the router discover how many action spaces actually exist, removing the need to pre-specify a fixed top-k.
- A direct test of the representation's ceiling would be to feed the same hierarchy with a learned per-embodiment embedding instead of the fixed vector; if that variant matches or exceeds HiMoE, the unified vector, not the MoE structure, is the operative limit.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HiMoE-VLA, a 4B vision-language-action model whose action module is organized as a hierarchy of mixture-of-experts layers: Action-Space MoE layers at the boundaries specialize computation for different action spaces (e.g., joint-angle vs. end-effector control), Heterogeneity-Balancing MoE layers in adjacent layers handle broader embodiment/sensor/scene variability, and dense Transformer blocks in the middle integrate shared representations. Two auxiliary losses are introduced: a contrastive Action-Space Regularization and a load-balancing Heterogeneity-Balancing Regularization. The model is pretrained on OXE and Aloha data, then fine-tuned and evaluated on CALVIN, LIBERO, and real xArm7 and ALOHA robots; the paper reports state-of-the-art results on these benchmarks and claims that, under heterogeneous co-training, HiMoE turns negative transfer into positive transfer. Code and models are publicly released.
Significance. If the reported results are correct, the paper makes a practically valuable contribution: it offers an architectural treatment of action-space and embodiment heterogeneity in VLAs, with extensive benchmark coverage, public code and models, and ablations of the main design choices. The layered AS-MoE/HB-MoE idea is clean and the routing visualizations are a useful sanity check. However, the paper's strongest conceptual claim, that heterogeneous co-training becomes positively transferable, rests on an under-specified comparison in Table 7, and the empirical sections generally omit uncertainty estimates. As presented, the evidence is suggestive but not yet sufficient for the strong conclusions drawn in the abstract and Section 4.3.
major comments (4)
- [§4.3, Table 7] The headline claim that the full HiMoE model 'not only avoids negative transfer but also improves when trained with heterogeneous data' is not established by Table 7 as presented. The columns 'D (Joint)' and 'ABC (EEF) + D (Joint)' do not state the evaluation domain. If the reported sums are CALVIN-D success (the CALVIN metric from Table 1), then the +0.186 gain for Full HiMoE (3.826 → 4.012) only shows that adding ABC data helps performance on D; there is no ABC-only baseline and no co-trained ABC success rate, so the claim of positive transfer on the ABC side is unverified. If the sums are intended as an aggregate over ABC and D, then the D-only baseline is not matched to the two-environment co-trained column. Please report ABC-side success for D-only, ABC-only, and co-trained models, and include multiple seeds.
- [§4.2–§4.3, Tables 3–9] No variance, confidence intervals, or significance tests are reported anywhere. Real-world evaluations use 15–18 trials per condition (Appendix A.2), and several headline differences are comparable to plausible binomial noise, e.g., Table 4 overall 63.7% vs. 54.2% for π0, or Table 5 single-arm Sum. 67.6% vs. 55.9% for π0. Simulation results also appear to be single-seed. Adding standard errors, per-seed results, or a significance statement is necessary before 'consistently surpasses' and 'performance boost' can be accepted as quantitative claims.
- [§4 Implementation Details vs. Appendix B and Table 9] The final Mixture-of-Experts configuration is ambiguous. Section 4 states N=32 experts with top-K=4 and reports that 'very high routing widths (K=8) cause instability.' Appendix B, however, states 'We set the number of experts to N=32 with a top-k routing of 8,' and refers to a table that shows this configuration outperforms alternatives. Table 9, the actual N/K ablation, contains no K=8 row at all. Since N and K are core architectural hyperparameters and the reported SOTA results depend on this choice, the paper must state the exact configuration used for the main results and make the ablation table consistent.
- [§4 Pre-training Dataset; Appendix A.1] The pretraining data sizes are internally inconsistent. Section 4 reports OXE 22.5M frames + Aloha 1.6M frames = 24.1M frames, while Appendix A.1 states that the Aloha datasets contribute 36.28M frames. Table 10 lists the Aloha mixture ratio as 10.4%, which matches neither 1.6/24.1 ≈ 6.6% nor 36.28/(22.5+36.28) ≈ 61.7%. These numbers must be corrected because the pretraining mixture is a central part of the experimental setup and the current text is not reproducible.
minor comments (5)
- [Abstract; §4.1; Table 1; Table 6(b)] The CALVIN headline number is reported as 3.98 in the abstract, 3.94 in the text, 3.967 in Table 1, and 4.012 in Table 6(b). Please align these numbers and clarify which evaluation condition each refers to.
- [§3.3, Eq. (4)] The AS-Reg objective is under-specified: 'pairs of experts (i,j) assigned to the same action-space token' needs a formal definition of how i and j are selected for each token and how 'assigned' relates to the routing decisions. As written, the positive-pair selection is ambiguous.
- [§4.3, Table 6(a) text] The text says that removing pretrained weights (w/o pretrain) leads to a 'more notable drop' than removing MoE re-initialization (w/o init), but the sums are 3.826 and 3.827, respectively. Either report the per-length numbers that support the qualitative claim or rephrase.
- [Appendix B] The sentence 'As shown in Table 6, this configuration consistently outperforms...' appears to refer to Table 9, since Table 6 is the ablation of initialization and heterogeneity-handling methods, not the N/K sweep.
- [Figure 1] Figure 1 labels the end-effector action space as '6D EEF Poses,' while the text and Appendix A.2 state an 8-dimensional end-effector vector (6D pose plus presumably gripper dimensions). Please make the notation consistent.
Circularity Check
No significant circularity: the paper's claims are empirical, benchmarked against external baselines, and no prediction reduces to a fitted value or self-citation by construction.
full rationale
The paper's central claim is empirical: a hierarchical Mixture-of-Experts action module (AS-MoE at boundaries, HB-MoE adjacent, with contrastive and load-balancing regularizations) improves heterogeneous co-training and avoids negative transfer. The auxiliary losses in Eqs. (4)-(7) do enforce expert specialization and balanced routing by construction, but that is their stated training mechanism, not a result being passed off as a prediction. The claimed gains are assessed against external baselines and ablations on CALVIN, LIBERO, xArm7, and ALOHA (Tables 1-9), and the code is public. The fixed 24-dimensional unified action/state vector with zero-padding and validity masks is an input representation, not an output derived from itself. Hyperparameters (N=32, K=4, λ_AS=0.002, λ_HB=0.001) were selected through ablations on CALVIN, which is a form of target-benchmark tuning that can inflate results but is not circular in the paper's equations—no fitted value is renamed as a prediction. The paper's self-citations, such as CogACT (Li et al., 2024), appear as related work or baselines and are not load-bearing for the central architecture. Appendix D's stated limitations concern feature redundancy and model scale, not a conceded circularity. Concerns about Table 7—the under-specified evaluation domain, missing ABC-side success and variance, and possible single-seed comparison—are evidence-quality or correctness issues, not evidence that the derivation reduces to its inputs. No step meets the quoted-reduction standard required for a circularity finding.
Axiom & Free-Parameter Ledger
free parameters (2)
- Number of experts N and top-k K =
N=32, K=4 (main text); App. B says K=8
- Regularization coefficients lambda_AS and lambda_HB =
lambda_AS=0.002, lambda_HB=0.001
axioms (5)
- domain assumption A fixed 24-dimensional state-action vector with zero-padding and validity masks is a sufficient encoding for all heterogeneous robots in the training set.
- domain assumption The contrastive AS-Reg objective with cosine similarity on expert scores induces the intended action-space specialization.
- domain assumption The load-balancing HB-Reg objective (fi*Pi) aligns empirical routing frequencies with expected probabilities in a way that improves abstraction.
- standard math Flow-matching (Lipman et al., 2022) with Beta-distributed timestep sampling is a valid action-generation objective.
- domain assumption The OXE+ALOHA pretraining mixture is representative of general heterogeneous robot data beyond the evaluation benchmarks.
read the original abstract
Generalist vision--language--action (VLA) policies are typically trained on heterogeneous mixtures of robot demonstrations spanning diverse embodiments, action spaces, and observation configurations. Modeling such heterogeneity with a shared dense action module can induce negative transfer, particularly when action spaces or visual observations differ across data sources. We address this issue with HiMoE-VLA, a VLA framework built around a Hierarchical Mixture-of-Experts (HiMoE) action module. HiMoE uses Action-Space MoE layers at the input/output boundaries to specialize computation for distinct action spaces, Heterogeneity-Balancing MoE layers in neighboring layers to provide balanced capacity for residual variation in observations, scenes, and embodiments, and dense Transformer blocks in the middle to integrate shared representations. Two auxiliary objectives further guide this hierarchy: a contrastive Action-Space Regularization objective for boundary specialization and a load-balancing objective for stable expert utilization. HiMoE-VLA reaches 3.98 on CALVIN, 98.0\% on LIBERO, and 75.0\% and 63.7\% average success on real xArm7 and ALOHA tasks; under controlled heterogeneous co-training, it turns the negative transfer observed in strong baselines into positive transfer. The code and models are publicly available at https://github.com/ZhiyingDu/HiMoE-VLA.
Figures
Forward citations
Cited by 8 Pith papers
-
Route by Kinematics, Act by Observation: Kinematics-Supervised Expert Routing in MoE-Augmented VLA
Supervising MoE routers with offline kinematic trajectory clusters, then routing from vision-language alone, lifts VLA manipulation success by roughly 20%+ over dense and implicit-MoE baselines.
-
PAMAE: Phase-Aware-MoE Action Experts Towards Reliable Flow-Matching Vision-Language-Action Policies
PAMAE adds a phase-aware router and expert mixture to flow-matching VLA models, yielding up to 9.2% higher task success on multi-stage manipulation simulations via two-stage training.
-
UniFS: Unified Fast-to-Slow Hierarchical Architecture for Vision-Language-Action Models
UniFS achieves 98.3% success on LIBERO with 2.1x lower latency than prior fast-slow VLA models by stratifying VLM layer update frequencies, inverting latent interactions, and applying multi-level supervision.
-
SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
SARM2 presents RM, a multi-task stage-aware reward model achieving 80% lower value-estimation MSE, which when used in SPIRAL boosts manipulation task success from ~50% to near-perfect on several benchmarks.
-
TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulation
TORL-VLA couples a tactile wrench-aware VLA policy with a lightweight online RL module and an intervention-censored critic to improve success and efficiency on contact-rich robotic tasks.
-
HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulation
HEX is a new framework with humanoid-aligned state representation, mixture-of-experts proprioceptive predictor, history tokens, and residual-gated fusion that achieves state-of-the-art success and generalization on re...
-
HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulation
HEX introduces a state-centric framework with humanoid-aligned representations and mixture-of-experts proprioceptive prediction for coordinated whole-body control on bipedal humanoids.
-
From Foundation to Application: Improving VLA Models in Practice
LingBot-VLA 2.0 combines 60k hours of multi-embodiment pretraining data, an expanded whole-body action space, and dual-query distillation from depth and video teachers to improve VLA performance on GM-100 and long-hor...
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...
-
[3]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...
-
[4]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.