pith. sign in

arxiv: 2607.05377 · v1 · pith:FX44TQ2A · submitted 2026-07-06 · cs.RO · cs.AI· cs.CV

Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

Jiaqi Peng , Xiqian Yu , Delin Feng , Yuqiang Yang , Wenzhe Cai , Jing Xiong , Ganlin Yang , Jinliang Zheng
show 5 more authors
Jiafei Cao Xueyuan Wei Jiangmiao Pang Yuan Shen Tai Wang
This is my paper

pith:FX44TQ2Areviewed 2026-07-07 13:44 UTCmodel glm-5.2open to challenge →

classification cs.RO cs.AIcs.CV
keywords cortexdataframeworklong-horizonmanipulationplanningagentaligned
0
0 comments X

The pith

Dual-system robot hits 65% on 14-step chemistry tasks

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Cortex argues that the reason today's Vision-Language-Action (VLA) models fail at long-horizon manipulation — tasks like running a multi-stage chemistry experiment or washing a beaker — is not a lack of motor skill but a lack of procedural memory. A reactive policy that maps only the current camera frame to actions cannot tell which stage of a multi-step procedure it is in, so it loops or jumps ahead. Cortex solves this by splitting the job into two asynchronously coupled systems: a high-level Vision-Language Model (VLM) that tracks progress in text-based semantic memory and dispatches one subtask at a time, and a low-level VLA that executes that subtask reactively. The central mechanism is a bidirectionally aligned subtask interface: the planner is constrained to output only subtasks drawn from 32 canonical skill primitives with strict language templates, and the executor is conditioned on those standardized commands rather than on the full task instruction. This interface is trained on over 4,000 hours of auto-annotated video and 30 hours of simulation data, with an event-balanced sampling strategy that oversamples the ambiguous moments around subtask boundaries so the planner learns when to hold the current subtask versus when to advance. A deployment harness normalizes the planner's language to executor-compatible commands and applies timeout-driven resets to escape perceptual deadlocks. The paper claims that this architecture, when paired with a fine-tuned VLA executor, can zero-shot complete 14-step real-world chemistry procedures at 65% success — something end-to-end VLA fine-tuning alone cannot achieve (0% success) — because the executor never has to infer global progress from raw observations.

Core claim

The paper's central claim is that the bottleneck in long-horizon robotic manipulation is temporal-semantic ambiguity at subtask transitions, not motor execution capability. By constraining a VLM planner to a 32-skill vocabulary with actively updated text memory, conditioning the VLA executor on single subtask commands, and training with event-balanced sampling that densifies supervision around transition boundaries, the system resolves the ambiguity that causes monolithic VLAs to loop or skip stages. The proof point is that this planner can be paired with a VLA fine-tuned on generic subtask-to-action data and then complete entirely unseen 14-step real-world chemistry tasks at 65% success, a

What carries the argument

The bidirectionally aligned subtask interface: 32 canonical skill primitives with strict language templates, an actively updated text-based semantic memory log, event-balanced sampling with asymmetric temporal margins around subtask boundaries, and a deployment harness that normalizes planner output to executor-compatible commands and applies timeout-driven kinematic resets.

If this is right

  • If the bidirectional alignment thesis holds, the path to longer-horizon robot autonomy runs through better planner-executor interfaces and transition-focused training data, not through scaling monolithic VLA models alone.
  • The 32-skill vocabulary could become a standard API layer: any VLM that outputs these primitives could swap between different VLA executors or robot embodiments without retraining the planner.
  • Event-balanced sampling suggests that data curation for robotic foundation models should prioritize transition frames over steady-state execution frames — a shift from volume-driven to event-driven data scaling.
  • The timeout-driven reset mechanism implies that perceptual deadlocks, not motor failures, may be the dominant failure mode in real-world long-horizon deployment, and that controlled physical perturbation is a viable recovery strategy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The zero-shot claim likely applies to the planning level but not the motor-execution level: the VLA executor was fine-tuned on ~10 hours of subtask-to-action data from the same platform with the same object types, so its motor patterns may be specialized to that embodiment and scene distribution. The paper does not make this distinction explicit.
  • If text-based memory is the bottleneck for spatial precision (as the authors acknowledge in their limitations), then tasks requiring pixel-level object-instance disambiguation — such as distinguishing two visually identical beakers — would expose a ceiling that the current architecture cannot break without adding visual memory retrieval.
  • The 32-skill vocabulary may be sufficient for tabletop chemistry and kitchen tasks but could be too coarse for domains requiring continuous force-controlled skills (e.g., deburring, suturing, assembly with tight tolerances), where the discrete subtask abstraction breaks down.
  • The event-balanced sampling result — that fewer transition-focused samples outperform more steady-state samples — suggests a general principle for data-efficient robot learning that could extend beyond the dual-system paradigm to any sequential decision-making model trained on demonstration data.

Load-bearing premise

The zero-shot transfer claim depends on the VLA executor being a general reactive policy, but the executor was fine-tuned on about 10 hours of subtask-to-action data collected on the same robot platform with the same object types, so its motor execution may be implicitly specialized to that scene distribution rather than truly zero-shot.

What would settle it

Run Cortex's VLM planner paired with a VLA executor fine-tuned on a genuinely different embodiment or object domain (e.g., a different arm morphology handling objects never seen in the executor's training data). If the 65% success rate on chemistry tasks drops to near zero, the zero-shot claim is specific to the planning layer and the executor is not generalizing.

Figures

Figures reproduced from arXiv: 2607.05377 by Delin Feng, Ganlin Yang, Jiafei Cao, Jiangmiao Pang, Jiaqi Peng, Jing Xiong, Jinliang Zheng, Tai Wang, Wenzhe Cai, Xiqian Yu, Xueyuan Wei, Yuan Shen, Yuqiang Yang.

Figure 1
Figure 1. Figure 1: Compared to previous works, Cortex is an embodied agent framework aligning the high [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Cortex framework. The cognitive orchestrator dynamically updates se [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Long-horizon metadata construction and interface standardization. Our data generation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Event-balanced Sampling. Training trajectories are strategically divided into boundary [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Harness Engineering. The deployment harness acts as a lightweight arbitration layer that [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Success rates on RoboTwin benchmark. RoboTwin Evaluation. As illustrated in Fig￾ure 6, monolithic VLAs suffer performance degradation as the task horizon scales, suc￾cumbing to premature task completion due to temporal ambiguity. Conversely, Cortex utiliz￾ing π0.5 and maintains an exceptionally high success rate (88.00%) on long-horizon splits. We attribute this robustness directly to our pro￾posed ambigui… view at source ↗
Figure 7
Figure 7. Figure 7: Zero-shot Real-world Deployment in Multi-stage Chemistry Tasks. A 14-step continuous [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Annotation-free subtask boundary inference on [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Annotation-free boundary inference for a representative [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Representative raw head-view frames extracted from the 15 tasks in the evaluation suite. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: State history exposes high-frequency progress in a representative [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Training prompt template for System-2. The prompt contains task-level language, input [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Multi-view rollout of the oven heating experiment. From top to bottom, the rows show [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt-mode comparison for trash disposal. Each frame is an action-switching frame, [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Detailed subtask prediction and execution process of Beaker Washing task [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Performance comparison across different baselines in the beaker washing task. [PITH_FULL_IMAGE:figures/full_fig_p031_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Local execution failure during stopper grasping. Cortex keeps the current subtask and [PITH_FULL_IMAGE:figures/full_fig_p032_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: End-to-end failure cases in the chemical liquid stirring task. [PITH_FULL_IMAGE:figures/full_fig_p033_18.png] view at source ↗
read the original abstract

While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex's efficacy, e.g., it outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin. Notably, Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA-a capability infeasible through VLA fine-tuning alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. The paper introduces Cortex, a dual-system framework for long-horizon robotic manipulation that bidirectionally aligns a VLM planner (System-2) with a VLA executor (System-1) via a structured subtask interface. The framework standardizes manipulation into 32 canonical skill primitives, constructs training data through automatic annotation of 4k+ hours of video and 30 hours of simulation, and introduces an event-balanced sampling strategy for subtask transitions. Evaluations include open-loop VLM planning (Table 1), closed-loop simulation benchmarks (LIBERO-Long 95.5%, RoboTwin 86.8%), and real-world chemistry/kitchen tasks (Table 3: 65%/55% success). The central claim is that Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks infeasible through VLA fine-tuning alone.

Significance. The paper makes several solid contributions: (1) a scalable automatic annotation pipeline using dynamic programming for subtask boundary inference (Appendix A.1), (2) the event-balanced sampling strategy with a controlled ablation (Table 5) showing improved sample efficiency, (3) strong simulation results on LIBERO-Long and RoboTwin with component-wise ablations, and (4) real-world deployment on complex 14-step chemistry tasks. The harness engineering and timeout-driven recovery mechanism are practical contributions. However, the central real-world claim rests on a comparison whose experimental controls are insufficiently specified, which weakens the contribution's load-bearing evidence.

major comments (3)
  1. §4.3, Table 3: The central real-world claim — that Cortex enables zero-shot task completion 'infeasible through VLA fine-tuning alone' — rests on the 0% vs 65% comparison between end-to-end baselines (π0.5, π_mem) and Cortex. However, the experimental setup confounds two variables. Appendix A.5.2 states that π_sub_mem was fine-tuned on 'about 10 hours of automatically segmented subtask-to-action data collected in the target robot setup.' The paper states π_mem has 'the same short-memory design' as π_sub_mem but does not explicitly confirm whether π_mem and π0.5 were trained on the same 10 hours of same-platform data. If they were not, the 0% vs 65% gap reflects both (a) the Cortex planning interface and (b) same-platform motor adaptation. Even if π_mem was trained on the same data, the comparison still conflates subtask-level conditioning with the full Cortex planning loop. Without an ab
  2. §4.3 and Abstract: The 'zero-shot' label is applied to the full system but Appendix A.5.2 reveals that System-1 (π_sub_mem) was fine-tuned on same-platform, same-object-type data (beakers, funnels, flasks). The system is zero-shot at the planning level but not at the motor-execution level. This distinction is not made explicit in the main text. The abstract's claim of 'zero-shot completion of unseen real-world long-horizon tasks' overstates the generalization: the executor has potentially memorized scene-specific motor patterns. The authors should clarify that 'zero-shot' refers specifically to the VLM planner's deployment without task-specific planning fine-tuning, and that the VLA executor was fine-tuned on same-platform data.
  3. §4.1, Table 1: The LLM-as-a-Judge evaluation uses Qwen-3.5-9B to score a system (Cortex) built on Qwen3-VL-8B. This creates a same-family bias risk: the judge may systematically favor outputs from the Qwen family over those from GPT-5 or Gemini. The paper does not discuss this potential bias or report any robustness check (e.g., cross-model judge agreement, human evaluation on a subset). This undermines the open-loop evaluation's validity as evidence for the interface design. The authors should either (a) use a judge from a different model family, (b) report human-annotated scores on a subset for calibration, or (c) explicitly acknowledge this limitation and justify why same-family judging is acceptable here.
minor comments (7)
  1. §3.2: The 32 canonical skill primitives are listed in Appendix A.3 but the main text does not explain how this vocabulary was derived — whether through clustering of existing datasets, manual task analysis, or coverage analysis. A brief justification for the choice of 32 and the specific skills would strengthen the design.
  2. Table 1: The scoring scale (out of 5 for subtask/memory, out of 10 for total) and the discrete judge ratings {0, 0.4, 0.9, 1.0} make interpretation difficult. The mapping from discrete ratings to the reported scale should be stated explicitly in the main text or table caption, not only in the appendix.
  3. §3.3: The temporal margins ε₁ and ε₂ are described qualitatively (ε₂ > ε₁, ~1 second total) but concrete values are only given in Appendix A.2 (ε = 0.5s for RoboCerebra, 1.5s for Galaxea). The main text should include concrete values.
  4. Figure 6: The RoboTwin results are presented as a bar chart with numerical values embedded. A table format (as in Table 8 in the appendix) would be easier to read and reference. Currently, cross-referencing between Figure 6 and Table 8 is cumbersome.
  5. §4.2: The paper states Cortex uses 'π0.5 and maintains an exceptionally high success rate (88.00%)' — the sentence structure is slightly broken here and should be revised for clarity.
  6. Appendix A.5.2, Table 10: The weight decay is listed as 1×10⁻², which differs from the simulation configuration in Table 7 (1×10⁻¹⁰). This large discrepancy should be explained or corrected.
  7. The paper references GPT-5.4 [42] and Gemini-3.1-Pro [14] with future dates (2026). These appear to be preview/pre-release models. The exact versions and access dates should be noted for reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The three major comments identify legitimate gaps in our experimental controls and terminology. We address each below and commit to concrete revisions in all three cases: (1) clarifying and strengthening the baseline comparison for the real-world claim, (2) correcting the 'zero-shot' terminology to distinguish planner-level from executor-level generalization, and (3) adding robustness checks for the LLM-as-a-Judge evaluation including cross-model judge agreement and human-annotated calibration on a subset.

read point-by-point responses

  1. Referee: [MAJOR-1] The central real-world claim (0% vs 65%) confounds Cortex's planning interface with same-platform motor adaptation, because it is unclear whether pi_mem and pi0.5 were trained on the same 10 hours of same-platform data as pi_sub_mem. Even if they were, the comparison conflates subtask-level conditioning with the full Cortex planning loop.

    Authors: The referee is correct that the main text does not explicitly state the training data configuration for the end-to-end baselines. To clarify: pi0.5 is the pretrained foundation model without any same-platform fine-tuning, while pi_mem was fine-tuned on the same 10 hours of same-platform subtask-to-action data as pi_sub_mem, with the only architectural difference being that pi_mem receives the full long-horizon task instruction rather than per-subtask commands. We agree this should have been stated explicitly in the main text and will add it. Regarding the deeper concern about confounding subtask-level conditioning with the full Cortex planning loop: we acknowledge that the 0% vs 65% comparison does not isolate the VLM planner's contribution from the subtask-conditioning benefit alone. The Human+pi_sub_mem row (75% success) partially addresses this by showing that even with a human providing optimal subtask commands, the executor alone achieves 75%, while Cortex's autonomous planning achieves 65%, demonstrating the planning loop closes most of the gap. However, we agree a cleaner ablation, pi_sub_mem with a non-Cortex planner (e.g., GPT-5 as planner), would further isolate the contribution. We will add this experiment and also revise the claim language to specify that the comparison is between end-to-end monolithic VLA execution and the full Cortex dual-system, not between subtask-conditioning alone and the full system. revision: partial

  2. Referee: [MAJOR-2] The 'zero-shot' label overstates generalization because System-1 (pi_sub_mem) was fine-tuned on same-platform, same-object-type data. The system is zero-shot at the planning level but not at the motor-execution level.

    Authors: This is a fair and accurate criticism. The abstract and main text use 'zero-shot' in a way that conflates two distinct generalization claims: the VLM planner operates zero-shot (no task-specific planning fine-tuning for the real-world chemistry tasks), while the VLA executor was fine-tuned on same-platform data with overlapping object types. We will revise the abstract and Section 4.3 to explicitly state that 'zero-shot' refers specifically to the VLM planner's deployment without task-specific planning fine-tuning, and that the VLA executor was fine-tuned on same-platform motor data. The revised phrasing will clarify that the contribution is the zero-shot transfer of the planning interface to unseen task compositions, not zero-shot motor execution. We appreciate the referee catching this overstatement. revision: yes

  3. Referee: [MAJOR-3] The LLM-as-a-Judge evaluation uses Qwen-3.5-9B to score a system (Cortex) built on Qwen3-VL-8B, creating same-family bias risk. No robustness check is reported.

    Authors: The referee raises a valid methodological concern. Using a Qwen-family judge to evaluate a Qwen-family system does introduce potential same-family bias. We will address this in revision through two additions: (1) We will report human-annotated scores on a randomly sampled subset (approximately 100 samples across all three evaluation buckets) to calibrate the automated judge. (2) We will add a cross-model judge agreement check using a non-Qwen model (e.g., GPT-5) as a secondary judge on the same subset, reporting inter-judge agreement. If the human and cross-model scores are consistent with the Qwen-3.5-9B judge, this would indicate the same-family bias is not materially affecting conclusions. If discrepancies emerge, we will report them transparently. We will also add an explicit limitation paragraph acknowledging the same-family judge risk. We note that the judge evaluates structured JSON outputs (subtask text and memory text) against ground-truth references using a discrete rubric, which is less susceptible to stylistic preference bias than free-form generation judging, but we agree this does not fully eliminate the concern. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the derivation chain is self-contained against external benchmarks.

full rationale

The paper's central claims rest on external benchmarks (LIBERO-Long, RoboTwin) and real-world task evaluations that are not constructed by the authors. The LLM-as-a-Judge evaluation (Table 1) uses Qwen-3.5-9B to score a system built on Qwen3-VL-8B — a same-family evaluator that introduces potential evaluation bias, but the judge scores are not fed back as inputs to the system's design or used to define any claimed output, so this is a correctness risk rather than circularity. The data annotation pipeline uses Qwen3-VL-235B to label data that fine-tunes Qwen3-VL-8B, which is intra-family distillation but not circular. The Table 3 comparison (Cortex 65% vs end-to-end 0%) has a potential experimental confound — whether π_mem was trained on the same 10 hours of data as π_sub_mem is not explicitly confirmed — but this is an experimental design concern, not a step where a prediction reduces to its inputs by construction. The self-citation to [23] (Wei et al., sharing authors) is cited as related work for navigation and is not load-bearing for any derivation. No step in the paper's chain exhibits X defined in terms of Y and then Y 'derived' from X.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The framework introduces 6 hand-tuned parameter sets (skill vocabulary size, temporal margins, sampling ratio, duration priors, cost weights, instruction mixture ratios), 5 domain/ad-hoc assumptions (skill decomposability, text memory sufficiency, boundary detectability, same-family judge unbiasedness, executor generality), and 3 invented mechanisms (skill vocabulary, event-balanced phases, kinematic reset). The skill vocabulary and event-balanced sampling have independent empirical validation; the kinematic reset does not. Several key parameters (λ_s, λ_v, λ_d, λ_m, ρ_k) are used in equations but their values are not reported, limiting reproducibility of the annotation pipeline.

free parameters (6)
  • 32 canonical skill primitives = 32 skills (Pick, Place, Pour, etc.)
    The skill vocabulary is manually designed (Appendix A.3) to cover manipulation primitives. The choice of 32 is not derived from theory but selected to cover observed task requirements.
  • Temporal margins ε₁, ε₂ = ε₂ > ε₁, total ~1 second; ε=0.5s for RoboCerebra, ε=1.5s for Galaxea
    Section 3.3 states these are 'empirically set' and 'adaptively customized across heterogeneous datasets.' The asymmetry and per-dataset values are hand-tuned.
  • Event-balanced sampling ratio = ~76% intra-task, ~24% boundary (ratio 2.23:1)
    Table 5 shows this ratio was selected by ablation. The target 'near equal sampling ratio' is approximate and dataset-dependent.
  • Duration prior ρ_k = Not specified numerically
    Appendix A.1, Eq. 8: expected relative duration of each subtask used as regularization. Values not reported.
  • Cost weights λ_s, λ_v, λ_d, λ_m = Not specified numerically
    Appendix A.1, Eqs. 4-9: modality balancing, duration prior weight, and motion penalty weight in the dynamic programming boundary inference. Values not reported in the paper.
  • Training mixture instruction ratios = 42.5% procedural, 20.0% subtask lists, 37.5% coarse goals
    Appendix A.2: the distribution of instruction paradigms in the training data is manually set.
axioms (5)
  • domain assumption Long-horizon manipulation tasks can be decomposed into sequences from a fixed vocabulary of 32 skill primitives.
    Section 3.2 and Appendix A.3. This is the foundational assumption of the interface design. No proof of completeness is given; the vocabulary is empirically motivated.
  • domain assumption Text-based semantic memory is a sufficient temporal bridge for subtask sequencing in manipulation.
    Section 3.1, Eq. for M(t). The paper acknowledges this as a limitation in Section 6: 'Text-based memory discards spatial coordinates and visual nuances.'
  • domain assumption Subtask boundaries in video data can be reliably detected by multimodal feature matching and dynamic programming.
    Appendix A.1. The annotation pipeline assumes monotone segmentation with Gaussian prototype distributions. No quantitative validation of boundary detection accuracy is reported.
  • ad hoc to paper An LLM from the same model family as the system backbone can serve as an unbiased judge of system outputs.
    Appendix A.3: Qwen-3.5-9B is used as judge for a system built on Qwen3-VL-8B. The paper does not address potential evaluator bias.
  • ad hoc to paper The VLA executor fine-tuned on subtask-segmented data from the target platform constitutes a 'general' reactive policy.
    Appendix A.5.2: π_sub_mem is trained on ~10 hours of same-platform data. The zero-shot claim in the main text implicitly assumes this executor generalizes, but the training data specificity is only revealed in the appendix.
invented entities (3)
  • 32 canonical skill primitives vocabulary independent evidence
    purpose: Constrains VLM output space to executable commands for VLA
    The vocabulary is tested against external benchmarks (LIBERO-Long, RoboTwin) and real-world tasks. Its coverage is empirically validated, not just postulated. However, no formal proof of completeness is provided.
  • Event-balanced sampling phases (boundary transition, intra-task execution, final tail) independent evidence
    purpose: Training data construction to address temporal ambiguity at subtask boundaries
    Table 5 ablation on Galaxea dataset with leave-one-episode-out validation shows the sampling strategy improves performance with fewer total samples, providing falsifiable evidence.
  • Timeout-driven kinematic reset mechanism no independent evidence
    purpose: Resolves perceptual deadlocks during real-world inference by introducing controlled physical perturbations
    Described in Appendix A.5.3 and demonstrated qualitatively in Figs. 7, 17. No quantitative ablation isolating this mechanism's contribution is provided.

pith-pipeline@v1.1.0-glm · 28652 in / 4123 out tokens · 435319 ms · 2026-07-07T13:44:03.747885+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages · 25 internal anchors

  1. [1]

    GPT-4o System Card

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024

    work page 2024

  3. [3]

    GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

    J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Torne, K

    M. Torne, K. Pertsch, H. Walke, K. Vedder, S. Nair, B. Ichter, A. Z. Ren, H. Wang, J. Tang, K. Stachowicz, et al. Mem: Multi-scale embodied memory for vision language action models. arXiv preprint arXiv:2603.03596, 2026

    work page arXiv 2026

  6. [6]

    Long-Horizon Manipulation via Trace-Conditioned VLA Planning

    I. Liu, A.-C. Cheng, R. Yan, G. Chen, R.-Z. Qiu, X. Zou, S. Yi, H. Yin, X. Wang, and S. Liu. Long-horizon manipulation via trace-conditioned vla planning.arXiv preprint arXiv:2604.21924, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  7. [7]

    H. Shi, B. Xie, Y . Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang. Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic ma- nipulation.arXiv preprint arXiv:2508.19236, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    H. Li, S. Yang, Y . Chen, Y . Tian, X. Yang, X. Chen, H. Wang, T. Wang, F. Zhao, D. Lin, et al. Cronusvla: Transferring latent motion across time for multi-frame prediction in manipulation. arXiv e-prints, pages arXiv–2506, 2025

    work page 2025

  9. [9]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  10. [10]

    Inner Monologue: Embodied Reasoning through Planning with Language Models

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models.arXiv preprint arXiv:2207.05608, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  11. [11]

    Sridhar, J

    A. Sridhar, J. Pan, S. Sharma, and C. Finn. Memer: Scaling up memory for robot control via experience retrieval.arXiv preprint arXiv:2510.20328, 2025

    work page arXiv 2025

  12. [12]

    T. Chen, Y . Wang, M. Li, Y . Qin, H. Shi, Z. Li, Y . Hu, Y . Zhang, K. Wang, Y . Chen, et al. Rm- bench: Memory-dependent robotic manipulation benchmark with insights into policy design. arXiv preprint arXiv:2603.01229, 2026

    work page arXiv 2026

  13. [13]

    OpenAI GPT-5 System Card

    OpenAI. Gpt-5 system card, 2025. URLhttps://arxiv.org/abs/2601.03267. Released: August 7, 2025; arXiv published: August 13, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    DeepMind

    G. DeepMind. Gemini 3.1 pro (preview) model card, 2 2026. URLhttps://storage. googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card. pdf. Preview release, accessed: 2026-05-26

    work page 2026

  15. [15]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 10

    work page 2023

  16. [16]

    PaLM-E: An Embodied Multimodal Language Model

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    X. Chen, Y . Chen, Y . Fu, N. Gao, J. Jia, W. Jin, H. Li, Y . Mu, J. Pang, Y . Qiao, et al. Internvla- m1: A spatially guided vision-language-action framework for generalist robot policy.arXiv preprint arXiv:2510.13778, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    D. Driess, K. Black, N. Brown, J. Darpinian, K. Dhabalia, A. Esmail, M. Equi, C. Finn, N. Fu- sai, et al.π 0.5: A vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International conference on robotics and automation (ICRA), pages 9493–9500. IEEE, 2023

    work page 2023

  21. [21]

    Q. Bu, H. Li, L. Chen, J. Cai, J. Zeng, H. Cui, M. Yao, and Y . Qiao. Towards synergistic, gen- eralized, and efficient dual-system for robotic manipulation.arXiv preprint arXiv:2410.08001, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers

    J. Zhang, Y . Guo, X. Chen, Y .-J. Wang, Y . Hu, C. Shi, and J. Chen. Hirt: Enhancing robotic control with hierarchical robot transformers.arXiv preprint arXiv:2410.05273, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    M. Wei, C. Wan, J. Peng, X. Yu, Y . Yang, D. Feng, W. Cai, C. Zhu, T. Wang, J. Pang, et al. Ground slow, move fast: A dual-system foundation model for generalizable vision- and-language navigation.arXiv preprint arXiv:2512.08186, 2025

    work page arXiv 2025

  24. [24]

    L. P. Kaelbling and T. Lozano-P ´erez. Hierarchical task and motion planning in the now. In 2011 IEEE international conference on robotics and automation, pages 1470–1477. IEEE, 2011

    work page 2011

  25. [25]

    C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´erez. Integrated task and motion planning.Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

    work page 2021

  26. [26]

    R. Li, Y . Zhou, Y . Zhu, K. Chen, J. Wang, S. Wang, K. Hu, M. Yu, B. Jiang, Z. Su, et al. Roboclaw: An agentic framework for scalable long-horizon robotic tasks.arXiv preprint arXiv:2603.11558, 2026

    work page arXiv 2026

  27. [27]

    P. Xu, J. Zheng, and Y . Mu. Roboagent: Chaining basic capabilities for embodied task plan- ning.arXiv preprint arXiv:2604.07774, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  28. [28]

    Srivastava, E

    S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In2014 IEEE international conference on robotics and automation (ICRA), pages 639–646. IEEE, 2014

    work page 2014

  29. [29]

    K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese. Scene memory transformer for embodied agents in long-horizon tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 538–547, 2019

    work page 2019

  30. [30]

    Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks

    Y . Zhang, J. Shu, Y . Ma, X. Lin, S. Wu, and J. Sang. Memory as action: Autonomous context curation for long-horizon agentic tasks.arXiv preprint arXiv:2510.12635, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    Q. Bu, J. Cai, L. Chen, X. Cui, Y . Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    A. Team. Agibot world 2026, 2026. URLhttps://huggingface.co/datasets/ agibot-world/AgiBotWorld2026. Open-source real-world embodied AI dataset

    work page 2026

  33. [33]

    Galaxea Open-World Dataset and G0 Dual-System VLA Model

    T. Jiang, T. Yuan, Y . Liu, C. Lu, J. Cui, X. Liu, S. Cheng, J. Gao, H. Xu, and H. Zhao. Galaxea open-world dataset and g0 dual-system vla model.arXiv preprint arXiv:2509.00576, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Mart ´ın-Mart´ın, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. InConference on Robot Learning, pages 80–93. PMLR, 2023

    work page 2023

  35. [35]

    S. Han, B. Qiu, Y . Liao, S. Huang, C. Gao, S. Yan, and S. Liu. Robocerebra: A large-scale benchmark for long-horizon robotic manipulation evaluation.Advances in Neural Information Processing Systems, 38, 2026

    work page 2026

  36. [36]

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y . Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y . Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    T. Chen, Z. Chen, B. Chen, Z. Cai, Y . Liu, Z. Li, Q. Liang, X. Lin, Y . Ge, Z. Gu, et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation.arXiv preprint arXiv:2506.18088, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Qwen 3.5: Open multimodal models, February 2026

    Qwen Team. Qwen 3.5: Open multimodal models, February 2026. URLhttps://qwen.ai/ blog?id=qwen3.5. Qwen 3.5-9B, accessed: 2026-05-26

    work page 2026

  39. [39]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [41]

    H. Tan, E. Zhou, Z. Li, Y . Xu, Y . Ji, X. Chen, C. Chi, P. Wang, H. Jia, Y . Ao, et al. Robobrain 2.5: Depth in sight, time in mind.arXiv preprint arXiv:2601.14352, 2026

    work page arXiv 2026

  42. [42]

    Gpt-5.4 thinking system card, 3 2026

    OpenAI. Gpt-5.4 thinking system card, 3 2026. URLhttps://deploymentsafety. openai.com/gpt-5-4-thinking/gpt-5-4-thinking.pdf. Official safety and model documentation

    work page 2026

  43. [43]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. Rdt-1b: a diffu- sion foundation model for bimanual manipulation. InInternational Conference on Learning Representations, volume 2025, pages 29982–30009, 2025

    work page 2025

  45. [45]

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.arXiv preprint arXiv:2403.03954, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model

    J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y . Feng, Y . Zheng, J. Zou, Y . Chen, J. Zeng, et al. X- vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. arXiv preprint arXiv:2510.10274, 2025. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [47]

    current skill

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025. 13 A Appendix This appendix provides supplementary implementation details, evaluation protocols, and qualitative analyses for Corte...

    work page 2025

  48. [48]

    current skill

    subtask 2 ... K. subtask K Candidate Atomic Skills: [skill 1, ..., skill 32], Current Observation Images (in order): Image 1 (view 1): [image 1] ... Image N (view N): [image N] Choose exactly one atomic skill from the candidate list when the task is in progress; if the task is already completed, set current skill to null and current subtask to task comple...

    work page 2048