REVIEW 1 major objections 6 minor 38 references
Half the robot demos, half the training, better policy
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · glm-5.2
2026-07-08 05:35 UTC pith:43YIADF6
load-bearing objection SIEVE selects 50% of VLA training data via primitive-composition structure and matches or beats full-data training — but all results are single-seed, and the margins are small enough to be noise. the 1 major comments →
SIEVE: Structure-Aware Data Selection for Imitation Learning with VLA Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes that selecting training demonstrations based on their coverage of reusable primitive-transition composition patterns yields better imitation learning policies than training on the full dataset, while using only half the data and half the training steps. The key mechanism is a three-stage pipeline: trajectory segmentation at gripper state boundaries produces visuo-motor primitives via clustering; a structural exposure objective with logarithmic diminishing returns allocates selection budgets across composition patterns to maximize coverage of diverse primitives and their transition interfaces; and medoid selection within each pattern bucket picks stable, central realizex
What carries the argument
The central machinery is the composition-pattern representation of trajectories. Each trajectory is segmented at end-effector grasp/release state flips into visuo-motor segments, which are encoded with a pretrained video encoder (V-JEPA2), dimensionality-reduced via PCA, and clustered via Mini-Batch K-Means into a vocabulary of K primitives (where K is auto-selected by maximizing a reuse-discriminability criterion). Each trajectory then becomes an ordered sequence of primitive IDs called a composition pattern, plus the transition interfaces between adjacent primitives. A greedy structural exposure objective allocates selection budget across composition-pattern buckets, weighting each prim
Load-bearing premise
The method segments trajectories at end-effector state flips (gripper grasp/release) under the assumption that these physically grounded boundaries correctly divide demonstrations into reusable behavioral primitives. If meaningful behavior transitions occur without gripper state changes, or if gripper flips happen mid-behavior, the entire primitive vocabulary and downstream composition analysis would be built on an invalid structural prior. The paper does not test alternative
What would settle it
A task domain where meaningful behavioral transitions do not align with gripper grasp/release state changes. For example, tasks dominated by continuous contact manipulation (pushing, sliding, wiping) where the gripper state rarely or never flips, or tasks where a single grasp-release cycle contains multiple distinct behavioral phases. In such domains, the segmentation would produce primitives that do not correspond to reusable behavioral units, and the structural exposure allocation would optimize over a degenerate composition-pattern space, likely yielding no improvement over random selection
If this is right
- Robot learning datasets could be pre-filtered for compositional diversity at collection time, reducing storage and annotation costs before training begins.
- The primitive vocabulary discovered by SIEVE could serve as an interpretable diagnostic tool: composition-pattern distributions reveal which behavioral structures are over- or under-represented in a dataset.
- If gripper-state segmentation generalizes to dexterous hands and multi-finger manipulators, the same pipeline could apply to increasingly complex embodiments without architectural changes.
- The structural exposure objective could be inverted for active data collection: instead of selecting from an existing pool, it could guide which new demonstrations to collect to maximize compositional coverage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SIEVE, a structure-aware data selection method for imitation learning with VLA models. SIEVE segments trajectories at gripper state flips, clusters segment representations (from V-JEPA2) into reusable primitives, allocates a selection budget across composition patterns by maximizing a structural exposure objective with diminishing returns, and selects medoid trajectories within each pattern bucket. The method is evaluated on three datasets (Bridge-V2, Fractal, GR00T-X-Sim) and two VLA models (Qwen3-VL-4B-GR00T, Qwen3-VL-4B-OFT), showing that SIEVE with 50% data and 50% training steps can match or exceed full-data training. The structural exposure objective (Eq. 11) and K-selection criterion (Eq. 6) are derived from internal principles (MDL, reuse-discriminability) without fitting to downstream task performance, which is a strength. The ablation study (Table 4) supports each component. The central concern is that all results appear to be from single-seed runs with no confidence intervals or standard deviations, making it difficult to assess whether the reported margins (1.4–4.5 points) reflect systematic improvement over training noise.
Significance. The paper addresses a practically important problem: selecting compact, informative subsets from large robot demonstration datasets for VLA training. The methodological design is principled: the structural exposure objective (Eq. 11) is derived from MDL considerations without fitting to downstream task performance, the K-selection criterion (Eq. 6) balances reuse and discriminability internally, and the greedy budget allocation is well-specified. The ablation study (Table 4) isolates the contribution of transition exposure, primitive exposure, and medoid selection. The composition-pattern redistribution analysis (Figure 3) provides qualitative insight into how SIEVE reshapes the selected subset. Code is publicly available. These are genuine strengths. However, the significance of the empirical contribution is tempered by the absence of statistical significance testing, which is needed to determine whether the headline claim—that SIEVE surpasses full-data training—holds reliably.
major comments (1)
- Tables 1–3 and 6–7: All success rates are reported as single numbers with no confidence intervals, standard deviations, or multiple seeds. The headline claim is that SIEVE surpasses full-data training with 50% data and 50% steps, but the margins are small enough to fall within typical VLA training noise: 4.5 points on Bridge-V2 (56.3 vs 51.8), 1.4 points on Fractal (76.4 vs 75.0), and 2.1 points on GR00T-X-Sim (54.8 vs 52.7). On GR00T-X-Sim, Random selection also beats Full-Training (53.5 vs 52.7), and SIEVE's margin over Full-Training is only 1.3 points larger than Random's. Without multiple seeds, it is impossible to determine whether these gains reflect a systematic advantage of SIEVE's structural exposure allocation or run-to-run variance. This is the most load-bearing issue because it directly determines whether the headline claim is true. At minimum, the authors should report 3+种子,
minor comments (6)
- Primitive Discovery, Trajectory Segmentation paragraph: the choice of gripper state flips as segmentation boundaries is a key design decision, but no sensitivity analysis or comparison with alternative segmentation strategies is provided. A brief discussion of failure modes (e.g., tasks where behavior transitions do not coincide with gripper state changes) would strengthen the paper. This is a presentation gap, not a correctness issue, as the method is still well-specified.
- Eq. (6): the K-selection criterion (1-J)log R combines a Jaccard-based discriminability measure and a reuse measure. The choice of this specific functional form is not justified beyond intuition. A brief note on why this form (vs alternatives) or a reference to prior work using similar criteria would help readers.
- Table 1: the 'Training Steps' column header is ambiguous. It mixes absolute step counts (25K, 35K, 50K) with percentage annotations (50%, 70%, 100%). Consider standardizing the notation.
- Figure 3: the x-axis labels showing composition patterns as bracketed lists (e.g., [8,8,8]) are difficult to read at normal scale. Consider providing a zoomed inset or a table summarizing the top patterns.
- The paper does not report computational overhead of SIEVE's selection pipeline (clustering, greedy allocation, medoid selection) relative to training cost. A brief note on wall-clock time for the selection step would be useful for practitioners.
- References include several 2026 arXiv preprints (e.g., Xu et al. 2026, Wu et al. 2026, Lian et al. 2026). Ensure these are properly cited and accessible.
Simulated Author's Rebuttal
We thank the referee for the careful and constructive review. The referee correctly identifies that our headline claim—SIEVE surpassing full-data training—rests on single-seed results without confidence intervals, and that the reported margins (1.4–4.5 points) are small enough to potentially fall within VLA training noise. We agree this is the most load-bearing issue and will address it by running multiple seeds for the key comparisons. Below we respond point by point.
read point-by-point responses
-
Referee: Tables 1–3 and 6–7: All success rates are reported as single numbers with no confidence intervals, standard deviations, or multiple seeds. The headline claim is that SIEVE surpasses full-data training with 50% data and 50% steps, but the margins are small enough to fall within typical VLA training noise: 4.5 points on Bridge-V2 (56.3 vs 51.8), 1.4 points on Fractal (76.4 vs 75.0), and 2.1 points on GR00T-X-Sim (54.8 vs 52.7). On GR00T-X-Sim, Random selection also beats Full-Training (53.5 vs 52.7), and SIEVE's margin over Full-Training is only 1.3 points larger than Random's. Without multiple seeds, it is impossible to determine whether these gains reflect a systematic advantage of SIEVE's structural exposure allocation or run-to-run variance. This is the most load-bearing issue because it directly determines whether the headline claim is true. At minimum, the authors should report 3+种子.
Authors: The referee is correct. Our current results are from single-seed runs, and we agree that the reported margins—particularly on Fractal (1.4 points) and GR00T-X-Sim (2.1 points)—are small enough that they cannot be distinguished from training noise without replication. We also agree that the GR00T-X-Sim setting is especially concerning in this regard, since Random selection already exceeds Full-Training there, making it difficult to attribute SIEVE's advantage to its structural exposure allocation rather than to the general effect of training on a smaller subset in a relatively in-domain simulated setting. We will address this as follows: (1) We will re-run the key comparisons—Full-Training, Random, and SIEVE—at 3 seeds for the 50% selection budget on all three datasets (Bridge-V2, Fractal, GR00T-X-Sim) using Qwen3-VL-4B-GR00T, and report mean ± standard deviation in all main tables. We will also run at least 3 seeds for the Qwen3-VL-4B-OFT comparison on Bridge-V2. (2) We will add a brief statistical analysis (e.g., paired comparisons or bootstrap confidence intervals on the per-task success rates) to assess whether SIEVE's advantage over Full-Training and Random is statistically significant. (3) If, after replication, the margin on a particular dataset (e.g., Fractal or GR00T-X-Sim) is not statistically significant, we will state this honestly and adjust the headline claim accordingly—e.g., 'SIEVE matches or exceeds full-data training' rather than claiming a consistent surpass. We note that the current evidence does provide some reason to expect systematic improvement: SIEVE outperforms Random on every individual task across all three datasets and both models (Tables 1–3, 6–7), and the ablation study (Table 4) shows that each component contributes in the expected方向. This revision: no
Circularity Check
No significant circularity: SIEVE's derivation chain is self-contained, with parameters selected by internal heuristics rather than downstream task performance.
full rationale
The paper's core derivation chain — primitive discovery (Eq. 2-6), structural exposure allocation (Eq. 11-15), and learning-friendly trajectory selection (Eq. 17) — is constructed from first principles (MDL, reuse-discriminability, diminishing returns) without fitting any parameter to downstream task success rates. The K-selection criterion (Eq. 6) optimizes an internal clustering score on a 10% subset, not the target metric. The structural exposure objective (Eq. 11) is a greedy optimization over composition patterns with no fitted parameters. Self-citations (Wu et al. 2026 ScalSelect, Lian et al. 2026 IntentVLA) appear in related work and are not load-bearing for the method's derivation. The segmentation boundary choice (gripper state flips) is a design assumption, not a circular definition. No step in the derivation reduces to its own inputs by construction.
Axiom & Free-Parameter Ledger
free parameters (5)
- Number of frames sampled per segment =
8
- PCA dimensionality =
256
- State transition persistence threshold =
5
- Number of clusters K =
Dataset-dependent (auto-selected)
- Selection budget =
50% or 70%
axioms (5)
- domain assumption End-effector state flips (gripper grasp/release) are physically grounded interaction boundaries that correctly segment trajectories into reusable primitives.
- domain assumption V-JEPA2 pretrained video encoder produces representations that capture semantically meaningful behavior content for robot manipulation segments.
- domain assumption Trajectories near the center of a composition-pattern bucket provide more stable and consistent supervision for behavior cloning than outliers.
- standard math The MDL principle implies that useful training data is rich in extractable, reusable structure (non-random regularities a bounded learner can internalize).
- domain assumption Mini-Batch K-Means is an appropriate clustering algorithm for discovering behavior primitives in segment representation space.
invented entities (3)
-
Visuo-motor primitives
independent evidence
-
Transition interfaces
independent evidence
-
Composition patterns
independent evidence
read the original abstract
Vision-Language-Action (VLA) models are typically trained by imitation learning on large-scale robot demonstration datasets, but more data does not necessarily yield better policies due to redundancy, noise, and uneven coverage. Existing data selection methods often assess demonstrations at either the trajectory or state-action level, missing the reusable structures that compose long-horizon behaviors. In this paper, we propose SIEVE, a structure-aware data selection method for VLA imitation learning. SIEVE views demonstrations as compositions of reusable primitives and transition interfaces. It first discovers visuo-motor primitives from segmented trajectories, then allocates selection budgets to composition patterns by maximizing reuse-aware structural exposure under diminishing returns. Finally, it selects medoid trajectories within each composition-pattern bucket to retain central, stable, and imitation-friendly demonstrations. Experiments across multiple datasets, benchmarks, and VLA models show that SIEVE consistently outperforms competitive data selection baselines. Notably, SIEVE can surpass full-data training while using only 50% of demonstrations and 50% of training steps, suggesting that reusable structure, captured through primitives and transitions, is an important signal for efficient VLA imitation learning.
Figures
Reference graph
Works this paper leans on
-
[1]
Conference on Robot Learning , pages=
Rt-2: Vision-language-action models transfer web knowledge to robotic control , author=. Conference on Robot Learning , pages=. 2023 , organization=
work page 2023
-
[2]
2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=
Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0 , author=. 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2024 , organization=
work page 2024
-
[3]
OpenVLA: An Open-Source Vision-Language-Action Model
Openvla: An open-source vision-language-action model , author=. arXiv preprint arXiv:2406.09246 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[4]
The International Journal of Robotics Research , volume=
Diffusion policy: Visuomotor policy learning via action diffusion , author=. The International Journal of Robotics Research , volume=. 2025 , publisher=
work page 2025
-
[5]
$\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
_0 : A Vision-Language-Action Flow Model for General Robot Control , author=. arXiv preprint arXiv:2410.24164 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[6]
Intelligence, Physical and Black, Kevin and Brown, Noah and Darpinian, James and Dhabalia, Karan and Driess, Danny and Esmail, Adnan and Equi, Michael and Finn, Chelsea and Fusai, Niccolo and others , journal=. _
-
[7]
Intelligence, Physical and Ai, Bo and Amin, Ali and Aniceto, Raichelle and Balakrishna, Ashwin and Balke, Greg and Black, Kevin and Bokinsky, George and Cao, Shihao and Charbonnier, Thomas and others , journal=. _
-
[8]
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
Fine-tuning vision-language-action models: Optimizing speed and success , author=. arXiv preprint arXiv:2502.19645 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[9]
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
Gr00t n1: An open foundation model for generalist humanoid robots , author=. arXiv preprint arXiv:2503.14734 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[10]
Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation , author=. arXiv preprint arXiv:2508.06426 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[11]
IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation
IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation , author=. arXiv preprint arXiv:2605.14712 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[12]
ACM Computing Surveys (CSUR) , volume=
Imitation learning: A survey of learning methods , author=. ACM Computing Surveys (CSUR) , volume=. 2017 , publisher=
work page 2017
-
[13]
Efficient reductions for imitation learning , author=. Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages=. 2010 , organization=
work page 2010
-
[14]
arXiv preprint arXiv:2510.18137 , year=
Quality Over Quantity: Curating Contact-Based Robot Datasets Improves Learning , author=. arXiv preprint arXiv:2510.18137 , year=
-
[15]
Advances in neural information processing systems , volume=
Data quality in imitation learning , author=. Advances in neural information processing systems , volume=
-
[16]
International Conference on Learning Representations , volume=
Data scaling laws in imitation learning for robotic manipulation , author=. International Conference on Learning Representations , volume=
-
[17]
arXiv preprint arXiv:2511.09119 , year=
Data Assessment for Embodied Intelligence , author=. arXiv preprint arXiv:2511.09119 , year=
-
[18]
Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning
Re-mix: Optimizing data mixtures for large scale imitation learning , author=. arXiv preprint arXiv:2408.14037 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[19]
SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning
Scizor: A self-supervised approach to data curation for large-scale imitation learning , author=. arXiv preprint arXiv:2505.22626 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[20]
Robot Data Curation with Mutual Information Estimators
Robot data curation with mutual information estimators , author=. arXiv preprint arXiv:2502.08623 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[21]
DataMIL: Selecting Data for Robot Imitation Learning with Datamodels
Datamil: Selecting data for robot imitation learning with datamodels , author=. arXiv preprint arXiv:2505.09603 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[22]
Curating Demonstrations using Online Experience
Curating demonstrations using online experience , author=. arXiv preprint arXiv:2503.03707 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[23]
FrameSkip: Learning from Fewer but More Informative Frames in VLA Training
FrameSkip: Learning from Fewer but More Informative Frames in VLA Training , author=. arXiv preprint arXiv:2605.13757 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[24]
arXiv preprint arXiv:2606.16208 , year=
ATHENA: Accelerated Multi-Task Heterogeneous Influence Functions for Robot Data Curation , author=. arXiv preprint arXiv:2606.16208 , year=
-
[25]
arXiv preprint arXiv:2602.11636 , year=
ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuning , author=. arXiv preprint arXiv:2602.11636 , year=
-
[26]
Synthetic Data for Multimodal Large Language Models: A Lifecycle-Oriented Survey , author=. 2026 , publisher=
work page 2026
-
[27]
Encyclopedia of statistical sciences , volume=
Minimum description length principle , author=. Encyclopedia of statistical sciences , volume=. 2004 , publisher=
work page 2004
-
[28]
The minimum description length principle , author=. 2007 , publisher=
work page 2007
-
[29]
IEEE transactions on information theory , volume=
The minimum description length principle in coding and modeling , author=. IEEE transactions on information theory , volume=. 1998 , publisher=
work page 1998
-
[30]
arXiv preprint arXiv:2601.03220 , year=
From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence , author=. arXiv preprint arXiv:2601.03220 , year=
-
[31]
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
V-jepa2: Self-supervised video models enable understanding, prediction and planning , author=. arXiv preprint arXiv:2506.09985 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[32]
Proceedings of the 19th international conference on World wide web , pages=
Web-scale k-means clustering , author=. Proceedings of the 19th international conference on World wide web , pages=
-
[33]
Conference on Robot Learning , pages=
Bridgedata v2: A dataset for robot learning at scale , author=. Conference on Robot Learning , pages=. 2023 , organization=
work page 2023
-
[34]
Evaluating Real-World Robot Manipulation Policies in Simulation
Evaluating real-world robot manipulation policies in simulation , author=. arXiv preprint arXiv:2405.05941 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[35]
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
Robocasa: Large-scale simulation of everyday tasks for generalist robots , author=. arXiv preprint arXiv:2406.02523 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[36]
Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[37]
arXiv preprint arXiv:2512.16793 , year=
Physbrain: Human egocentric data as a bridge from vision language models to physical intelligence , author=. arXiv preprint arXiv:2512.16793 , year=
-
[38]
Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
Large vlm-based vision-language-action models for robotic manipulation: A survey , author=. arXiv preprint arXiv:2508.13073 , year=
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.