REVIEW 6 major objections 6 minor 1 cited by
Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning
T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that splitting embodied continual learning into high-level instruction and low-level action layers, and routing each input through clustered task embeddings to SVD-protected LoRA experts, cuts catastrophic forgetting to…
desk verdict The HEC setups are a useful new benchmark configuration for embodied continual learning, but the SOTA claims don't hold up: the accuracy metric is never defined and the key hyperparameters were selected on the same test setup that produces the headline numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is Task-aware MoILE, built from three parts. Cross-modal task clustering (CTC) updates $M$ cluster centroids online and assigns each sample the centroid vector of its nearest cluster, using that vector as a task embedding. A two-level mixture-of-experts then routes hidden states through a token-level router (top-$K$ among $N_1$ LoRA experts) and a task-level router (top-1 among $N_2$ LoRA experts conditioned on the task embedding). The forgetting defense is Incremental LoRA: after each task, LoRA matrices are decomposed as $W = U\Sigma V^{\top}$, the top-$p$ singular components are kept frozen, and the residual is trained while an extra loss pulls the leading singular values of the combined update toward the frozen ones and an orthogonal loss keeps residual and principal subspaces disjoint.
What would settle it
Measure the purity of the cross-modal clusters against the true behavior or environment labels on a held-out set: if the cluster assignment of new inputs is near chance, or if the task-level router's top-1 expert choice disagrees with the ground-truth task type on a substantial fraction of samples, the reported accuracy and forgetting figures would not be reproducible. A simpler check is to run the Hybrid setup with M set to the true number of categories (7) and see whether accuracy improves; if it does not, the router is not actually exploiting cluster separation.
Extended reading notes
Core claim
The central claim is that hierarchical continual learning for embodied agents, learning high-level planning instructions and low-level actions sequentially and interleaved, is made tractable by a task-aware mixture of incremental LoRA experts. Cross-modal task clustering maps each input's image-text embedding to a cluster centroid that stands in for the hidden task ID; a task-level router then selects one of two level-specific experts while a token-level router picks among six token experts per layer. Prior-task LoRA weights are decomposed with SVD; the top singular components are frozen as principal knowledge, and the residual is trained under a singular-value-matching loss and an orthogonality loss so new learning cannot overwrite old knowledge. As evidence, the paper reports that on the Hybrid Hierarchical setup its method raises average accuracy to 53.58 from 44.92 for the best baseline and cuts the forgetting measure to 13.51 from 36.04, with consistent gains on the four non-hybrid setups.
Load-bearing premise
The whole routing scheme depends on the online clustering of image-text embeddings producing a stable centroid that actually separates task types, yet the number of clusters is fixed at four while some setups contain up to seven behavior categories and the paper never measures cluster quality.
Editorial extensions
If this is right
- On the Hybrid Hierarchical setup (LB → HB → LE → HE orders), the method reports average accuracy 53.58 and forgetting 13.51, versus 44.92 and 36.04 for InfLoRA, the strongest baseline the paper compares.
- On Low-level Action Behavior and Low-level Action Environment setups, the method reports average accuracy 67.91 and 64.88, roughly 2.3 and 2.2 points above the second-best baselines, with forgetting measures 3.37 and 3.79.
- On High-level Instruction Behavior and Environment setups, average accuracy reaches 55.58 and 57.69 while forgetting drops to 3.12 and 1.76.
- Ablations attribute the largest forgetting increase to removing Incremental LoRA (FM from 13.51 to 38.68), and the largest accuracy drop to removing the task-level router (AA from 53.58 to 47.87).
- Routing visualizations show the task-level router assigns separate experts to high-level and low-level tasks, which is the intended hierarchical separation.
Reading between the lines
- The paper fixes the cluster count at $M=4$ even though behavior sets contain up to seven categories; a natural test is whether adaptive cluster counts or a measure of cluster purity would change the routing accuracy and the reported numbers.
- The SVD-based principal/residual split is not specific to embodied tasks; it could be applied to any continual instruction-tuning setting where LoRA experts accumulate, such as multi-domain dialogue or tool use.
- Because HEC is built only on ALFRED, running the same method on other instruction-following or navigation benchmarks would show whether the hierarchical forgetting gains transfer.
- The two-level decomposition suggests evaluating instruction-level and action-level forgetting separately; the paper's aggregate FM may hide asymmetric forgetting between planning and execution layers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes the Hierarchical Embodied Continual Learning (HEC) setups, which decouple embodied instruction following into high-level instruction generation and low-level action execution, and defines five incremental learning scenarios. The authors introduce Task-aware MoILE, which uses cross-modal task clustering (CTC) to produce task embeddings, a token-level router with top-K LoRA experts, a task-level router with top-1 expert selection, and an incremental LoRA mechanism that freezes singular-value principal components and trains residual components under singular-value and orthogonality regularizers. Experiments are conducted on the ALFRED benchmark with LLAMA 2-7B and CLIP, reporting improvements in average accuracy and forgetting over EWC, CAMA, MoELoRA, O-LoRA, and InfLoRA across all HEC setups.
Significance. The HEC setups address a real gap: existing embodied continual learning benchmarks focus on low-level action sequences and ignore higher-level task planning. The proposed hierarchy is a useful conceptual contribution, and the method combines several plausible ideas (task-ID-free clustering, expert routing, and SVD-based parameter freezing) that could be relevant for LLM-driven embodied agents. The paper provides extensive empirical comparisons, ablations, and analysis of routing and embeddings. However, the state-of-the-art claim is not yet verifiable: the evaluation uses the same setup for hyperparameter selection and final reporting, the mathematical formulation of two key losses does not match the textual claims, and no variance estimates are provided. These issues are fixable but currently prevent acceptance.
major comments (6)
- [Appendix B (Tables 5, 7) and Section 4.2 (Table 3)] The hyperparameter configuration (M=4, N1=6, N2=2, K=2, λ1=1.0, λ2=0.5) is selected on the Hybrid Hierarchical Incremental Learning setup itself, as shown in Appendix B Tables 5 and 7, and the same setup is then used for the headline results in Table 3. No held-out validation split or nested cross-validation is described. The reported AA 53.58 and FM 13.51 in Table 3 are therefore selected values, not predicted ones, and the 8.66-point AA gain over InfLoRA may reflect test-set tuning rather than method efficacy. Please re-run the evaluation with a validation split for hyperparameter selection and report the results on a truly held-out set.
- [Section 3.3, Eq. (12)] The text states that the singular value loss 'minimizes the difference between the top p singular values' of ˙W and Wp, but Eq. (12) actually computes the absolute difference of the square roots of the sums of squares, i.e., the difference of the Frobenius norms of the top-p subspaces. This does not force the individual singular values to match; any redistribution among the top p values with the same norm is unpenalized. Either modify the loss to Σ|˙σ_i − σ_i| or weaken the claim accordingly.
- [Section 3.3, Eqs. (13)-(14)] The orthogonal loss is claimed to enforce orthogonality between the residual LoRA parameters W and the principal components Wp, but O = A^T A_p penalizes only the column-space correlation between the A matrices. Since W = B A and Wp = Bp Ap, the inner product ⟨W, Wp⟩ involves A A_p^T (or the full matrices), not A^T A_p. As written, the loss does not ensure that W and Wp are orthogonal; it constrains only part of the factorization. Please correct the loss to use A A_p^T or the full matrices, or revise the claim.
- [Section 4.1, Tables 1-3] All experimental tables report single runs for each of the three task orders, with no standard deviations or significance tests across seeds. Several of the claimed improvements are small (e.g., Table 2, HB: AA 55.58 vs 54.50 for InfLoRA), and could easily be within run-to-run variation. Please report means and standard deviations over at least three seeds per order, and perform a significance test where appropriate.
- [Section 4.1, Baseline and Appendix B] The baselines are not capacity-matched. Task-aware MoILE uses 8 LoRA experts per layer (N1=6 token-level plus N2=2 task-level), while the compared methods—MoELoRA, O-LoRA, and InfLoRA—use a different and likely smaller set of adapters. The accuracy gains could therefore be due to larger parameter capacity rather than the proposed routing or regularization mechanisms. Please report the total number of trainable parameters for each method and include capacity-matched baselines.
- [Section 3.1, Appendix C (Figure 7)] The cluster count M is fixed at 4, while the HB and LB setups contain 7 behavior categories. The paper does not quantify cluster quality (e.g., purity, NMI) or the stability of the cluster centroids over the course of continual learning. Because the task-level router selects experts based on the cluster centroid, the routing decisions—and hence the reported accuracy and forgetting numbers—depend on the clusters reliably separating task types. Please report cluster-quality metrics and analyze how M affects routing accuracy.
minor comments (6)
- [Section 4.1, Eq. (16)] The notation A_{T,t} in Eq. (16) is ambiguous, and the subscript order is inconsistent with Eq. (17) where A_{i,t} and A_{T,t} appear. Please define both indices clearly and use a consistent convention.
- [Section 4.1] The paper states that the agent is 'evaluate[d] in unseen environments' but does not specify which ALFRED split (e.g., valid-unseen or test) is used. Please clarify the evaluation split.
- [Section 3.2, Eqs. (4) and (6)] G1(x) and G2(e) are defined as sets of expert indices, yet Eq. (4) uses G1(x)_i and G2(e)_i as multipliers. Please specify how the selected expert outputs are aggregated (e.g., equal weights, normalized probabilities, or averaging).
- [References] The baseline 'MoELoRA (Chen et al., 2024)' appears to cite the 'Coin: A benchmark of continual instruction tuning' paper, which is not the MoELoRA method described in the text. Please verify and correct the citation.
- [Section 3.3] The symbol ˙W for the merged LoRA matrix is nonstandard and easily confused with a derivative; please use a clearer notation such as W_merge.
- [Section 7 (Limitations)] The Limitations section appropriately acknowledges that experiments are limited to ALFRED and rely on high-quality data; these limitations should be reflected in the abstract and conclusion, which currently state 'state-of-the-art results' without qualification.
Circularity Check
The Hybrid Hierarchical (HH) SOTA numbers are the same numbers selected by a hyperparameter search on that exact HH setup, so the headline comparison is a selected value rather than an independent prediction.
-
fitted input called prediction
[Appendix B, Table 5 (hyperparameter search) vs. Section 4.2, Table 3 (reported Hybrid results)]
"Additionally, we conducted experiments with different hyperparameter settings to evaluate the model's performance. The results summarized in Table 5 indicate that the current hyperparameter setting achieves optimal performance."
Table 5 searches N1, N2, K, lambda1, lambda2, and M on the Hybrid Hierarchical Incremental Learning setup and reports AA/FM for that same setup; the row (N1=6, N2=2, K=2, lambda1=1.0, lambda2=0.5, M=4) yields AA 53.58 and FM 13.51. Table 3 then reports exactly these values as 'Ours' for the Hybrid setup, and Section 4.2 celebrates an 8.66-point AA gain and a 22.53-point FM reduction over InfLoRA. Since the configuration was chosen by maximizing performance on the same test setup whose results are presented as the headline, the reported advantage is a selected value rather than an independently predicted one. No held-out validation split or nested cross-validation is described, and the same test-set tuning also fixes the clustering hyperparameter M (Tables 5 and 7).
full rationale
The paper's method is not analytically derived from its own output, and the four non-hybrid setups (LB, LE, HB, HE) are evaluated on the external ALFRED benchmark, so those comparisons carry independent content. The one concrete circular step is the Hybrid Hierarchical (HH) headline: Appendix B reports a hyperparameter search over N1, N2, K, lambda1, lambda2, and M on the exact HH setup and selects the row with AA 53.58 / FM 13.51; Section 4.2 then presents those same numbers as 'Ours' in Table 3 and claims an 8.66-point AA gain and 22.53-point FM reduction. Because the configuration was chosen by maximizing performance on the same test setup whose results are celebrated, the HH SOTA claim reduces to a test-set selection effect rather than a prediction. No held-out validation or nested cross-validation is described. The paper is otherwise self-contained: there is no load-bearing self-citation chain and no definitional identity between the method's components and its metrics. Eq. 16 defines AA with an undefined A_{T,t}, which blocks verification but is a rigor issue rather than circularity. Overall, the central claim partially reduces to selection, so the score is 6.
Assumptions & free parameters
free parameters (8)
- Number of clusters M =
4
- Number of token-level LoRA experts N1 =
6
- Number of task-level LoRA experts N2 =
2
- Top-K token selection K =
2
- Singular value loss weight lambda1 =
1.0
- Orthogonal loss weight lambda2 =
0.5
- LoRA rank r =
not reported
- Learning rate, optimizer, epochs =
not reported
assumptions (5)
- domain assumption SVD of a LoRA weight matrix yields a small set of principal components that capture task-specific knowledge
- domain assumption Online cluster centers (Eq. 1-2) converge to stable task partitions without task IDs
- domain assumption The ALFRED dataset and AI2-THOR simulator are a valid testbed for embodied continual learning at both instruction and action levels
- domain assumption CLIP visual-text embeddings for the same task type cluster together
- domain assumption Freezing principal components across sequential tasks does not interfere with learning new residual components
Cite this review
Pith. "Pith review of Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning." pith.science (2026). https://pith.science/paper/4CEIQVC7
@misc{pith2026250604595,
author = {Pith},
title = {Pith review of: Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/4CEIQVC7}},
note = {Machine review of arXiv:2506.04595}
}
read the original abstract
Previous continual learning setups for embodied intelligence focused on executing low-level actions based on human commands, neglecting the ability to learn high-level planning and multi-level knowledge. To address these issues, we propose the Hierarchical Embodied Continual Learning Setups (HEC) that divide the agent's continual learning process into two layers: high-level instructions and low-level actions, and define five embodied continual learning sub-setups. Building on these setups, we introduce the Task-aware Mixture of Incremental LoRA Experts (Task-aware MoILE) method. This approach achieves task recognition by clustering visual-text embeddings and uses both a task-level router and a token-level router to select the appropriate LoRA experts. To effectively address the issue of catastrophic forgetting, we apply Singular Value Decomposition (SVD) to the LoRA parameters obtained from prior tasks, preserving key components while orthogonally training the remaining parts. The experimental results show that our method stands out in reducing the forgetting of old tasks compared to other methods, effectively supporting agents in retaining prior knowledge while continuously learning new tasks.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
Reference graph
Works this paper leans on
-
[3]
MOV ABLE→ COOL → PICK&PLACE → HEAT → CLEAN → EXAMINE → PICK2&PLACE A.2 High-level Instruction Environment Incremental Learning In the dataset, four unique environments are in- cluded: KITCHENS, LIVINGROOMS, BED- ROOMS, and BATHROOMS. In the high-level In- struction Environment Incremental Learning setup, the model is tasked with acquiring the ability to p...
-
[4]
Large language models as generalizable poli- cies for embodied tasks. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023. Orthogonal subspace learning for lan- guage model...
arXiv 2024
-
[10]
We trained the model using the same sequences as described above in A.1
BATHROOMS → LIVINGROOMS → KITCHENS→BEDROOMS A.3 Low-level Action Behavior Incremental Learning In the Low-level Action Behavior Incremental Learning setup, the model is required to acquire low-level control capabilities across diverse behav- ior types. We trained the model using the same sequences as described above in A.1
-
[11]
EXAMINE → HEAT → PICK2&PLACE → COOL → PICK&PLACE → CLEAN → MOV ABLE
-
[12]
PICK2&PLACE → CLEAN → MOV ABLE → PICK&PLACE → HEAT → EXAMINE →COOL
-
[13]
We trained the model using the same sequences as described above in A.2
MOV ABLE→ COOL → PICK&PLACE → HEAT → CLEAN → EXAMINE → PICK2&PLACE A.4 Low-level Action Environment Incremental Learning In the Low-level Action Environment Incremental Learning setup, the model is required to acquire low-level control capabilities across diverse envi- ronmental contexts. We trained the model using the same sequences as described above in A.2
-
[14]
BEDROOMS → BATHROOMS → LIVIN- GROOMS→KITCHENS
-
[15]
KITCHENS → BEDROOMS → LIVIN- GROOMS→BATHROOMS
Show all 15 references
-
[16]
BATHROOMS → LIVINGROOMS → KITCHENS→BEDROOMS A.5 Hybrid Hierarchical Incremental Learning In our experimental setup, we also designed three distinct sequences comprising four incremental learning modes, resulting in the following orders for model training
-
[17]
Low-level Action Behavior → High-level In- struction Behavior → Low-level Action En- vironment→High-level Instruction Environ- ment
-
[18]
High-level Instruction Behavior → Low-level Action Behavior → High-level Instruction En- vironment → Low-level Action Environment
-
[19]
The hyperparameters are set as follows
Low-level Action Behavior → Low-level Ac- tion Environment → High-level Instruction Behavior → High-level Instruction Environ- ment B Details of Model Training In this section, we provide the detailed hyperparam- eter settings used for training the proposed model. The hyperpar...
-
[2014]
In2nd International Con- ference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings
Online continual learning for interactive in- struction following agents. In2nd International Con- ference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings. OpenReview.net. James Kirkpatrick, Razvan Pascanu, Neil Rabino...
2014
-
[2018]
Lifelong inverse reinforcement learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 4507–4518. Amit Parekh, Nikolas Vitsakis, Alessandro S...
2018
-
[2024]
Zhao Mandi, Shreeya Jain, and Shuran Song
InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 23638–23647. Zhao Mandi, Shreeya Jain, and Shuran Song. 2024. Roco: Dialectic multi-robot collaboration with large language models. InIEEE ...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.