Pith. sign in

REVIEW 3 major objections 5 minor 27 references

This paper establishes that exhaustive instruction-tuple enumeration is unnecessary: a structured subset covering pairwise dependencies can match full-task performance in sequential robot manipulation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 01:45 UTC pith:JOK3LIAX

load-bearing objection Strong empirical paper on data coverage for compositional generalization; the theory is correctly derived but does not prove the pairwise-coverage claim — the headline result is an empirical regularity that deserves follow-up. the 3 major comments →

arxiv 2607.29687 v1 pith:JOK3LIAX submitted 2026-07-31 cs.RO

Diagnosing Compositional Generalization in Sequential Robot Tasks

classification cs.RO
keywords compositional generalizationinstruction-space coveragesequential robot manipulationimitation learninggeneralization gaporthogonal arraysdata collectionout-of-distribution generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Sequential robot policies must handle novel combinations of familiar instruction components, but collecting demonstrations for every possible tuple is combinatorially expensive. This paper claims that exhaustive enumeration is unnecessary: the generalization gap can be decomposed into marginal instruction shift, instruction-compositional shift, and context–action shift, and only the dependencies that actually steer the action distribution must be covered. In experiments, an orthogonal task set covering all instruction pairs, using only one quarter of the full task space, matched the out-of-distribution success of full-task training. The paper also shows that sparse training failures come from weak instruction steering, not missing low-level skills: finetuning with one demonstration per task raised OOD success from 0.4% to 54.7%. For semantically dependent factors, joint coverage of the dependent pairs is required, not mere factor diversity. The authors caution that the experiments are limited to a small set of manipulation tasks with a frozen vision encoder, leaving broader applicability as future work.

Core claim

The paper's central assertion is that the compositional generalization gap in instruction-conditioned policies is governed by three additive L1 discrepancy terms (Proposition 4.1): a marginal term for the active subtask, a compositional term for how familiar subtasks are recombined, and a context–action term for behavior conditioned on the full instruction. Because the compositional term measures mismatch in the conditional distribution of surrounding instructions given the active one, controlling it does not require full Cartesian coverage. Empirically, the number of observed instruction pairs in training (seen-pair count) predicts OOD success, and a 16-task orthogonal design that covers al

What carries the argument

The load-bearing mechanism is the three-term generalization-gap bound of Proposition 4.1, written for any fixed stage i as |Δq(θ)| ≤ M(||q(li)-p(li)||1 + E[||q(l-i|li)-p(l-i|li)||1] + E[||q(z,a|l)-p(z,a|l)||1]). The middle term, instruction-compositional shift, is what sparse training leaves uncontrolled; the experiments operationalize it through the seen-pair count, the number of factor pairs observed together in the training set. The orthogonal task set is the combinatorial device that maximizes pair coverage with a small number of tuples. The stage-wise modular policy bound (Corollary 4.2) shows how removing inactive instructions from the policy input eliminates the explicit compositional

Load-bearing premise

The central claim rests on the assumption that the action-relevant dependencies among instruction factors are at most pairwise—that training data covering every pair of factors also controls the conditional distribution of leftover factor combinations—so a task whose correct action depends on a genuine three-way interaction would defeat the coverage principle.

What would settle it

Construct a sequential task where the correct action is a unique function of a three-way combination (e.g., pick object o, place in container c, press button b, with the requirement that o, c, and b are all distinct or that a specific (o,c,b) triple maps to a unique reward). Train on an orthogonal set covering all factor pairs—so every pair appears—but none of the full triples, then evaluate on held-out triples. If success on those triples is low while a model trained on the full Cartesian space succeeds, the pairwise-coverage claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Robot data collection can replace exhaustive task enumeration with orthogonal or covering designs that hit every pair of instruction factors, at significant cost savings (e.g., 16 tasks instead of 64).
  • Sparse pretrained policies retain reusable subtask skills; a tiny amount of broad-coverage finetuning (one demonstration per task) can recover most out-of-distribution success, so new data should prioritize breadth of instruction pairs.
  • When instruction factors semantically constrain each other, the training set must cover the dependent pairs jointly, or the policy will learn spurious correlations (e.g., same-container violations).
  • A stage-wise modular policy reduces, but does not eliminate, the need for coverage, because inactive instructions can still reshape the context distribution; a dependency-preserving planner or dataset is still required.
  • The seen-pair count offers a simple, pre-training diagnostic: a training set with high pairwise coverage should generalize better than one with same marginal coverage but low pair coverage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same pairwise-coverage principle could be used to audit large pretraining corpora for robotics: databases collected without attention to higher-order dependencies may silently fail on recombinations, and the seen-pair count could flag which instruction pairs are undertrained.
  • If the pairwise assumption generalizes, then a model trained on orthogonal, pair-covered data might be a stronger starting checkpoint for finetuning on novel tasks than a model trained on more, but more redundant, demonstrations.
  • The decomposition is not specific to imitation learning; it could diagnose compositional failures in offline RL or learned planners, where the context–action shift term would capture distribution shift in the visited state-action distribution.
  • One testable extension: measure whether seen-pair count predicts OOD success on instruction spaces with 4+ factors; if the trend continues, the principle would support covering arrays of strength 2 rather than strength 3+, though the paper leaves that to future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies compositional generalization in sequential robot manipulation from an instruction-space coverage perspective. It derives an upper bound (Proposition 4.1 / A.1) that decomposes the compositional generalization gap into marginal instruction shift, instruction-compositional shift, and context–action shift, and a corollary for stage-wise modular policies. The authors then present simulated robomimic experiments on three task families (PP, PPP, 2S-PP). The central empirical claim is that full Cartesian enumeration of instruction tuples is unnecessary: a structured subset covering action-relevant dependencies — in particular a 16-task orthogonal array for the 64-task PPP space, one quarter of the full space — can reach OOD success close to full-task training. Additional findings concern finetuning from sparse diagonal pretraining (one demonstration per task raises OOD success from 0.4% to 54.7%) and the need for dependent-pair coverage in semantically dependent tasks. The paper also proposes a data-collection principle: prioritize dependency coverage over exhaustive task expansion.

Significance. If the claims are ultimately supported, the work would be a valuable contribution to data-efficient imitation learning and compositional generalization in robotics. Proposition 4.1 is a correct triangle-inequality identity and is presented with a clean proof; Corollary 4.2 is also correctly derived. The experimental program is unusually thorough for a short paper: 55 training-set configurations for PPP, multiple fixed-budget comparisons, finetuning ablations, and failure-case Sankey diagrams. The paper also makes a falsifiable prediction — that pairwise coverage of instruction factors is sufficient for tasks whose action-relevant dependencies are at most pairwise — and provides a website with supplementary material. The main risk is that the theoretical framework is used to explain an empirical regularity it does not actually imply, as detailed in the major comments. The empirical findings remain interesting, but the theory-driven interpretation needs substantial revision before the paper can be accepted.

major comments (3)
  1. [Section 5.2, Eq. (5)] The claim that 'pairwise coverage mitigates the instruction-compositional discrepancy q(l_{-i}|l_i)-p(l_{-i}|l_i)' is not supported by Proposition 4.1. In PPP with n=3 and a strength-2 orthogonal array of 16 tasks, for any fixed l_i the training conditional p(l_{-i}|l_i) has support only 4 points, while the test conditional q(l_{-i}|l_i) is uniform over 16 points. The L1 distance is 1.5, which is large, not small. Therefore the second term in the bound (5) is not controlled by pairwise coverage; the bound is vacuous in exactly the regime claimed to support it. The observed success of the orthogonal design is better explained by the stage-wise structure of the task — stage 2 depends on (o,c) and stage 3 on (c,b), with no action depending on all three factors jointly. The paper should either (i) reframe the one-quarter claim as an empirical coverage principle and provide a sharper formaliz
  2. [Section 5.2, Figures 2c and 8b] The 'seen pair count' analysis is post-hoc and in-sample. The same PPP configurations are used to discover the correlation between seen-pair count and OOD success, and the orthogonal set is deliberately constructed to maximize pairwise coverage. Thus the experiment does not independently validate the coverage principle; the design choice and the metric are on the same data. Additionally, the correlation is reported descriptively without statistical measures (e.g., error bars, permutation tests, or a hold-out task family). The one-quarter result therefore remains an observational finding on two simulated task families, not a consequence of Proposition 4.1. I recommend either pre-specifying the orthogonal design before measuring performance, or validating the principle on held-out task families with different interaction orders and reporting the associated uncertainty.
  3. [Section 5.3, Figures 5 and 10] The interpretation 'sparse pretraining learns reusable skills but not steering' is reasonable, but it is supported only indirectly. The finetuning experiments show that one demonstration per task improves OOD success, yet Appendix D (Figure 10) also reports that finetuning can degrade OOD performance when the pretrained checkpoint is already strong, e.g., the orthogonal-set pretraining. This nuance should be integrated into the main text; otherwise the narrative overstates the universality of the finetuning benefit. The paper should acknowledge that the steering-vs-skills dichotomy is a conceptual interpretation, not a direct measurement.
minor comments (5)
  1. [References] References [12] and [13] are identical (Lake and Baroni, ICML 2018); one should be removed or replaced with a distinct citation.
  2. [Acknowledgments] The Acknowledgments section contains a placeholder text ('If a paper is accepted...'). In a submitted manuscript this should be replaced with actual acknowledgments or omitted entirely.
  3. [Numbering] The main text uses Proposition 4.1 / Corollary 4.2 while the appendix and several references use Proposition A.1 / Corollary A.2. Unify the numbering to avoid reader confusion.
  4. [Section 5.4 / Appendix E] The transition from the general 2S-PP definition (where the full instruction has four components o1, c1, o2, c2) to the experimental instantiation with 36 or 48 tasks is abrupt. The relation between the hidden container c1, the observed instruction l=(o1,o2,c2), and the same-container violation rate should be stated more clearly in the main text.
  5. [Figures 3 and 11] In the fixed-budget comparison, confounds between total demonstration count and per-task demonstration count are inherent to the design, but the captions should explicitly state the per-task allocation for each curve. This is especially important in Figure 11, where the random baseline uses one demonstration per sampled task while the orthogonal set uses two demonstrations per task in the 32-demonstration case.

Circularity Check

0 steps flagged

No significant circularity; the generalization-gap bound is self-contained and the one-quarter claim is empirical, not derived by construction.

full rationale

The paper's theoretical core, Proposition 4.1 (restated as Proposition A.1), is a direct triangle-inequality decomposition of the compositional generalization gap; it introduces no fitted parameters and no equation reduces to its own inputs. Corollary A.2 follows by the same type of exact decomposition under the modular-policy assumption. The experiments do not fit constants to the theory, and the paper does not claim to derive the 78.2% OOD number from the bound. Section 5.2 explicitly says the pairwise-coverage observation 'motivates' the orthogonal design, and then reports 'Experiments show' the result—this is empirical validation of a design principle, not a circular derivation. The citations to orthogonal-array theory are external and not self-referential; the self-citations (e.g., Interleave-VLA) are contextual and not load-bearing. The main weakness is that the paper asserts pairwise coverage 'supports Proposition 4.1' in the sense of controlling the instruction-compositional discrepancy, but the proposition itself does not logically imply that pairwise coverage bounds that term for the tested regime; this is an interpretive overreach and a correctness risk, not a circularity. Section 6 acknowledges the restriction to a small set of tasks, which limits generality but does not make the derivation circular.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no novel physical or architectural entities. The main unstated burden is the assumption that pairwise instruction coverage captures the action-relevant dependencies, and that success rate reflects the theoretical loss bound.

axioms (4)
  • standard math Loss is measurable and uniformly bounded on supp(p) ∪ supp(q), i.e., 0 ≤ Lθ ≤ M.
    Stated in Proposition 4.1; a standard bounded-loss assumption to make the generalization gap finite.
  • domain assumption A stage indicator σ(z) exists that identifies the active subtask for the stage-wise modular policy.
    Needed for Corollary 4.2 and the modular policy analysis; in practice a high-level oracle must provide this indicator.
  • domain assumption Pairwise coverage of instruction factors is sufficient to control the instruction-compositional shift.
    Section 5.2 uses 'seen pair count' as the explanatory statistic and designs the orthogonal set to maximize pairwise coverage. No proof shows that pairwise coverage controls the conditional distribution mismatch for arbitrary higher-order dependencies; this is an empirical regularity assumed for the tasks.
  • ad hoc to paper Success rate is a valid monotone proxy for the bounded loss in the theory.
    The theory bounds expected loss, but all headline metrics are success rates. The paper does not establish a monotone relationship between the loss bound and binary task success, so the theoretical results are only loosely connected to the empirical claims.

pith-pipeline@v1.3.0-daily-deepseek · 13721 in / 12251 out tokens · 131980 ms · 2026-08-03T01:45:09.016506+00:00 · methodology

0 comments
read the original abstract

Sequential robot manipulation requires policies to execute novel combinations of familiar instruction components. However, collecting demonstrations for all possible instruction tuples is combinatorially expensive, while sparsely covered datasets often fail under out-of-distribution recombination. This paper studies compositional generalization through the lens of instruction-space coverage. We decompose the generalization gap into three sources: \textit{marginal instruction shift}, \textit{instruction-compositional shift}, and \textit{context--action shift}. This decomposition allows us to diagnose when sparse training coverage is sufficient, and what structure the training set must preserve for reliable action prediction. Our results show that exhaustive tuple enumeration is unnecessary: a structured subset, as small as one quarter of the full task space, can recover strong out-of-distribution performance when it covers action-relevant dependencies. We further find that sparse training often fails due to instruction steering rather than missing low-level skills; finetuning only one demonstration per task improves OOD success from \(0.4\%\) to \(54.7\%\). For semantically dependent tasks, effective coverage must capture relational structure rather than only factor diversity. These findings suggest that efficient robot data collection should prioritize dependency coverage in instruction space over exhaustive task expansion. More results are available in the supplementary material. Project website: https://yixiaowang7.github.io/Diagnosing_Compositional_Generalization_Robot_Page/.

Figures

Figures reproduced from arXiv: 2607.29687 by Boyuan Liang, Cheng-En Wu, Guojian Zhan, Lingfeng Sun, Masayoshi Tomizuka, Pengcheng Wang, Xiang Ji, Yixiao Wang.

Figure 1
Figure 1. Figure 1: Illustration of Pick-Place-Press and Dependent Two-Stage Pick-and-Place tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Full tuple coverage is unnecessary. Higher seen pair count predicts stronger OOD success. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Representative examples showing that OOD tasks with higher seen-pair coverage tend to [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Orthogonal set is more effective than full-set [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Finetuning from sparse diagonal pretraining. Small amounts of broad-coverage data [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance and same-container violations in 2S-PP. Broader dependent-pair coverage [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Policy network structure. Then we have q(z, a, li) − p(z, a, li) = q(li) − p(li)  q(z, a | li) + p(li) [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Additional experiments on PPP task. 0 64 128 256 512 Number of finetuning demos 0 20 40 60 80 100 Success rate (%) no FT | finetuned 6 tasks in diag_03 (pretraining set) ID success rate full-FT orthogonal-FT trained from scratch 0 64 128 256 512 Number of finetuning demos no FT | finetuned 58 held-out tasks (not in diag_03) OOD success rate [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Finetuning from sparse pretraining (diag 03, B = 6). Small amounts of broad-coverage data substantially improve OOD success, suggesting that sparse pretraining learns reusable skills but not reliable instruction steering, which can be acquired with limited additional data. D Q2: What Is Missing When Sparse Coverage Fails? [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Finetuning from different pre￾trained model (B = 4, 6, 8, 16). Only one demonstration per task can significantly im￾prove OOD success. However, finetuning can also suffer from overfitting when the pre￾trained model is already strong. We pretrained the model for 100k steps and then fine￾tuned it for only 5k steps, using a batch size of 64 for both stages. In addition to pretraining on the minimal diagonal … view at source ↗
Figure 11
Figure 11. Figure 11: The orthogonal set achieves superior performance in the extremely data-limited setting. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Failure case analysis on 2S-PP. Numbers in the figure means the number of rollouts. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance and same-container violations in 2S-PP (48 task set). Broader dependent-pair [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Failure case analysis on 2S-PP (48 task set). Numbers in the figure means the number of [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Success rate at B = 8 . 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Seen pair count at B = 8 . 19 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 10 linked inside Pith

  1. [2]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

  2. [3]

    G. R. Team, A. Abdolmaleki, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, A. Balakr- ishna, N. Batchelor, A. Bewley, J. Bingham, et al. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer.arXiv preprint arXiv:2510.03342, 2025

  3. [4]

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

  4. [5]

    C. Fan, X. Jia, Y . Sun, Y . Wang, J. Wei, Z. Gong, X. Zhao, M. Tomizuka, X. Yang, J. Yan, et al. Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions.arXiv preprint arXiv:2505.02152, 2025

  5. [6]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

  6. [7]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

  7. [8]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

  8. [9]

    Pertsch, K

    K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

  9. [10]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

  10. [11]

    Jiang, A

    Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei-Fei, A. Anandkumar, Y . Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts, 2023. URL https: //arxiv.org/abs/2210.03094. 9

  11. [13]

    Lake and M

    B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. InInternational conference on machine learning, pages 2873–2882. PMLR, 2018

  12. [14]

    Yagcioglu, O

    S. Yagcioglu, O. B. Ince, A. Erdem, E. Erdem, D. Elliott, and D. Yuret. Sequential compositional generalization in multimodal models.arXiv preprint arXiv:2404.12013, 2024

  13. [15]

    Haresh, D

    S. Haresh, D. Dijkman, A. Bhattacharyya, and R. Memisevic. Clevrskills: Compositional language and visual reasoning in robotics.Advances in Neural Information Processing Systems, 37:38235–38266, 2024

  14. [16]

    Y . Chen, Z. Chen, N. T. Chan, J. Chen, J. Yin, J. Shi, Y . Gao, Y .-L. Li, and J. Huo. Robohiman: A hierarchical evaluation paradigm for compositional generalization in long-horizon manipulation. arXiv preprint arXiv:2510.13149, 2025

  15. [17]

    Mandlekar, S

    A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023

  16. [18]

    J. Gao, A. Xie, T. Xiao, C. Finn, and D. Sadigh. Efficient data collection for robotic manipulation via compositional generalization.arXiv preprint arXiv:2403.05110, 2024

  17. [19]

    R. N. Kacker, E. S. Lagergren, and J. J. Filliben. Taguchi’s orthogonal arrays are classical designs of experiments.Journal of research of the National Institute of Standards and Technology, 96 (5):577, 1991

  18. [20]

    A. S. Hedayat, N. J. A. Sloane, and J. Stufken.Orthogonal arrays: theory and applications. Springer Science & Business Media, 2012

  19. [21]

    D. R. Kuhn, R. N. Kacker, and Y . Lei.Introduction to combinatorial testing. CRC press, 2013

  20. [22]

    Bacon, J

    P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. InProceedings of the AAAI conference on artificial intelligence, volume 31, 2017

  21. [23]

    Nachum, S

    O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018

  22. [24]

    C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´erez. Integrated task and motion planning.Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

  23. [25]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

  24. [26]

    L. X. Shi, M. R. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, A. Li-Bell, et al. Hi robot: Open-ended instruction following with hierarchical vision-language- action models. InForty-second International Conference on Machine Learning

  25. [27]

    Mandlekar, D

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation. InConference on Robot Learning (CoRL), 2021

  26. [28]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pages 1–31, 2024. 10

  27. [29]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023. 11 A Theoretical Analysis Proposition A.1(General upper bound on the compositional generalization gap).Assume the loss is measurable an...