{"id":"960192e7-a0f3-4b3f-93b5-2f31ac155569","arxiv_id":"2505.22626","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SCIZOR filters suboptimal and redundant state-action pairs from robot demonstrations without human labels, improving imitation-learning policy success rates by about 15% on average.","lead":"SCIZOR is a self-supervised pipeline that cleans large robot imitation datasets by deleting low-quality state-action pairs using a task-progress predictor and a state-action deduplicator. It reports average success-rate gains of about 15% over training on unfiltered data across several manipulation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 15.4% average mixes in-sample threshold-tuned benchmarks (RoboMimic, OXE Magic) with one held-out (Sirius-Fleet); OXE gain is within seed noise, and a truly out-of-sample evaluation is missing.","rationale":"The paper's central empirical claim is that SCIZOR improves policy success and that the average gain is 15.4%. For that claim to be trustworthy, the measured gains must be out-of-sample estimates of the method's effect. The strongest evidence is Sirius-Fleet: thresholds fixed a priori, real robot, +32.9 points over no curation. I find this genuinely supportive. However, the two simulation benchmarks that are included in the headline average are not out-of-sample: Appendix A.1 shows the unified thresholds were chosen by running the sub-methods on RoboMimic and OXE Magic, varying deletion ratios, and picking the thresholds that produce the best success rate on those same benchmarks. This is a direct form of test-set tuning for two of the three components of the 15.4% average. The OXE Magic result also uses only two seeds and reports no variance for the no-deletion baseline, so the +8.1 point gain is not established as statistically reliable. The linear-progress limitation is real and the authors disclose it, but it does not by itself undermine the measured improvements on the tested tasks; it narrows the scope of the method. The threshold-tuning issue is more load-bearing because it questions whether the headline number reflects genuine generalization. The concrete test is to run the pipeline on a new benchmark with frozen thresholds, or to perform nested tuning on splits, and compare the gain to the reported range. If the frozen-threshold gain is much smaller, the 15.4% average should be presented as a tuned-benchmark result, not as a general claim.","tokens_in":17478,"tokens_out":9570,"duration_ms":104320,"concrete_test":"Run SCIZOR on a previously untouched benchmark (e.g., DROID or a held-out OXE subset) with the frozen thresholds (epsilon_s=0.58, epsilon_d=0.99) and no threshold re-tuning, and compare success rate against training on the full dataset. If the improvement is not comparable to the reported +8 to +33 point range, the headline 15.4% average overstates out-of-sample benefit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not a flaw in the method's mechanics but in the evidence linking it to the headline number. In Appendix A.1 the unified thresholds (epsilon_s=0.58, epsilon_d=0.99) are selected by running SCIZOR with only one sub-method at a time on exactly the two benchmarks that contribute to the 15.4% average and picking the deletion ratio (and hence threshold) that maximizes downstream policy success rate. Thus the reported gains of +5.4 points on RoboMimic and +8.1 points on OXE Magic are in-sample estimates; the threshold selection procedure used the test performance of those benchmarks. The only fully held-out estimate is Sirius-Fleet (+32.9 points), which is a single real-robot environment, and even that result is averaged over only 10 trials per seed. The average improvement of 15.4% therefore mixes tuned and untuned benchmarks and is not an unbiased estimate of the method's benefit on new data. The OXE Magic result in particular uses only two seeds with no reported baseline variance, so the +8.1 point gain could be within noise. A reader cannot currently distinguish the method's true out-of-sample benefit from tuning artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SCIZOR proposes a self-supervised, transition-level data curation method for imitation learning. It combines two modules: a suboptimal-transition filter that trains a temporal-distance classifier to predict task progress from pairs of frames and removes chunks whose predicted progress is below the elapsed time, and a state-action deduplication module that clusters joint state-action embeddings with K-means and removes near-duplicate chunks. The method is evaluated on RoboMimic, Sirius-Fleet, and three variants of the Open X-Embodiment dataset (Magic Soup, RT-X, RT-1), using BC, BC-Transformer, and Octo policies. The authors report absolute success-rate gains of 5.4% on RoboMimic, 8.1% on OXE Magic, and 32.9% on Sirius-Fleet over training on the full datasets, an average improvement of 15.4%, and state that SCIZOR outperforms uniform deletion, DemInf, and Re-Mix in several settings.","tokens_in":17598,"tokens_out":6697,"duration_ms":79752,"significance":"If the reported results hold out of sample, SCIZOR would be a useful contribution: it is the first transition-level, annotation-free curation framework for large-scale imitation learning, and the idea of using temporal-distance classification as a self-supervised quality signal is simple and potentially scalable. The paper also includes reasonable ablations, comparisons against trajectory- and dataset-level baselines, qualitative analyses of the removed transitions, and a candid limitations section. However, the headline average mixes in-sample threshold-tuned benchmarks with a single held-out real-robot dataset, and the OXE results are based on only two seeds without reported variance; as a result, the strength of the central empirical claim is currently uncertain.","major_comments":[{"comment":"The unified thresholds epsilon_s=0.58 and epsilon_d=0.99 are selected by running the suboptimal-only and deduplication-only variants on exactly RoboMimic and OXE Magic with deletion ratios of 10%, 20%, and 30%, and choosing the thresholds that maximize downstream policy success on those same benchmarks. Consequently, the gains of +5.4 points on RoboMimic and +8.1 points on OXE Magic reported in Figure 3 are in-sample estimates, and they are included in the headline 15.4% average. The only fully held-out estimate is Sirius-Fleet (+32.9 points), which is a single real-robot setting with 10 trials per seed. Please provide a genuinely out-of-sample evaluation, for example by fixing thresholds on one benchmark and evaluating on untouched benchmarks, or by nested cross-validation, and report tuned and untuned gains separately.","section":"Appendix A.1, Section 4.2, Figure 3"},{"comment":"The OXE Magic and OXE RT-X results are based on two random seeds, and the main figure and Table 9 report averaged success rates without per-seed values or confidence intervals. With two seeds and no baseline variance, the +8.1 point OXE Magic gain and the RT-X comparison against Re-Mix could be within noise. Please report per-seed results and standard errors, or increase the number of seeds for these key comparisons.","section":"Section 4.1, Appendix B.2, Table 9"},{"comment":"The comparison with the trajectory-level baseline DemInf weakens the claim that transition-level curation is generally superior. On RoboMimic Can, DemInf achieves 88.9% versus SCIZOR's 84.0%, and on Square 41.4% versus 40.8%; the text acknowledges this in RQ2 but still concludes that fine-grained curation offers advantages. The evidence supports a transition-level advantage only on datasets with complex intra-trajectory quality distributions such as Sirius-Fleet. Please temper the claim accordingly and, ideally, report a statistical comparison across all benchmarks rather than relying on a few point estimates.","section":"Section 4.2 (RQ2), Table 7"},{"comment":"The suboptimality score V_{i:i+T}=T - T_p treats elapsed time as the expected task progress by construction, and the progress predictor is trained on the same demonstrations it later filters. As the Limitations section acknowledges, this fails for tasks with intentional pauses, stirring, or waiting; moreover, if slow or repetitive behavior is common in a dataset, the predictor will absorb that behavior as normal and will not flag it. This is a load-bearing assumption of the proposed detector, not merely a future extension. Please state the intended operating regime explicitly (e.g., datasets where the majority of transitions are proficient and progress is roughly monotonic) and, if possible, include a controlled experiment that varies the fraction of suboptimal demonstrations to show how the method degrades.","section":"Section 3.2, Section 6 (Limitation)"}],"minor_comments":[{"comment":"There are several typos and spacing errors, for example 'SCIZORis as elf-supervised datac uration method' in the Introduction and inconsistent spelling of the DemInf baseline ('Deminf', 'DemoInf', 'DemInf') across the text, tables, and figures.","section":"Introduction, Section 2"},{"comment":"The legend lists five methods ('No Deletion', 'Uniform', 'DemInf', 'Re-Mix', 'Ours'), but not every panel appears to contain all five methods; please clarify which bars correspond to which baseline on each dataset, or use separate legends per panel.","section":"Figure 3"},{"comment":"The notation T is used both for the chunk duration in the deduplication module and for the elapsed time in the progress predictor, and Appendix A.4 states that all experiments use a constant 2-second interval. Please unify the notation and define T's units consistently.","section":"Appendix A.4, Section 3.3"},{"comment":"The description of how the predicted progress bin is converted into the scalar T_p is not explicit; Appendix A.5 describes the bin edges but not the conversion used to compute V_{i:i+T}. Please state this conversion and the value of alpha used in the final score.","section":"Section 3.2, Equations (2)-(4)"},{"comment":"No code or trained curation masks are released, and the paper does not mention a data-release plan. For a data-curation method, releasing the curation masks or the code that produces them would substantially aid reproducibility and would also allow the out-of-sample threshold issue to be examined by other groups.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is attractive and the experimental setup is ambitious, but the threshold-tuning procedure in Appendix A.1 directly affects the headline average, and the two-seed OXE results, together with the lack of released code or data, leave the paper's central claim weaker than the abstract suggests. I would encourage the editor to request the per-seed results and an out-of-sample threshold evaluation before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SCIZOR makes a genuine contribution: it is the first self-supervised, transition-level curation framework for imitation learning, filtering individual state-action pairs rather than whole trajectories or domains. The combination of a temporal-distance progress classifier with joint state-action deduplication is new in this setting, and the evaluation is extensive: RoboMimic, three OXE mixtures, and a real-robot fleet. The strongest evidence is Sirius-Fleet, where success rises from 46.7% to 79.6% using thresholds applied without tuning. The ablations and qualitative error analysis are also useful; the authors identify meaningful suboptimal modes and show most recovery behaviors are preserved.\n\nThe soft spots are real but not disqualifying. The headline 15.4% average improvement mixes RoboMimic and OXE Magic, on which the unified thresholds were tuned (Appendix A.1), with Sirius-Fleet, the only fully held-out benchmark. The OXE Magic gain is small and rests on two seeds with no reported variance for the no-deletion baseline, so I would not put much weight on it. The RoboMimic comparison is honestly won by DemInf, and the authors say so. The linear task progress assumption is a genuine limitation for tasks with pauses or stirring; the authors acknowledge it in Section 6, and it means the method can conflate \"behind schedule\" with \"suboptimal\" when the dataset's own statistics are skewed. No code or data is released, so the numbers are hard to verify independently. The deduplication threshold of 0.99 deletes almost nothing on RoboMimic (0.3%), so that component is effectively inert there.\n\nNone of this sinks the central idea. The self-supervision is plausible, the Sirius-Fleet result is a strong existence proof, and the authors are transparent about the main weakness in their own progress measure. This paper deserves a serious referee. If I were handling it, I would require code and data release, a headline number that separates tuned from untuned thresholds, and either additional OXE seeds or a threshold-selection protocol that is genuinely out-of-sample. The paper targets researchers who train large-scale imitation policies and need practical, unlabeled curation; they will get real value from both the framework and the honest comparison against velocity and VLM baselines.","headline":"SCIZOR's transition-level curation idea is real and the Sirius-Fleet result is promising, but the headline average is inflated by in-sample threshold tuning; the paper deserves a serious referee once artifacts and out-of-sample thresholds are addressed.","tokens_in":18280,"tokens_out":2743,"would_cite":true,"duration_ms":30883,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCIZOR, a self-supervised curation framework, filters suboptimal and redundant state-action pairs from robot demonstrations and improves imitation learning success by 15.4% on average, with no human labels.","keywords":["imitation learning","data curation","self-supervised learning","state-action pair filtering","task progress prediction","semantic deduplication","vision-language-action models","robot learning"],"falsifier":"Run SCIZOR's progress predictor on a task that requires deliberate waiting or repeated motion, such as stirring food for a set duration, and check whether those productive transitions are scored as suboptimal and deleted; if policies trained on the curated set fall below the no-curation baseline on that task, the linear-progress assumption is the cause.","tokens_in":1906,"feed_emoji":"🤖","tokens_out":4736,"duration_ms":117622,"temperature":0.7,"pith_summary":"The paper claims that the quality of a robot imitation learning dataset can be curated automatically, at the level of individual state-action pairs, without any human annotation. It introduces SCIZOR, a self-supervised pipeline that removes two kinds of harmful data: suboptimal transitions, identified by a learned task-progress predictor, and redundant transitions, identified by deduplication on joint state-action features. The claim is that this fine-grained, label-free filtering consistently improves policy success, by an average of 15.4% across RoboMimic, Sirius-Fleet, and Open-X-Embodiment benchmarks, and that it beats trajectory-level and dataset-level curation baselines. If true, this means expensive manual quality labeling can be replaced by a scalable, annotation-free step when building large robot training sets.","feed_headline":"Self-supervised data curation lifts robot policy success by 15.4%","feed_subtitle":"SCIZOR removes suboptimal and redundant state-action pairs without human labels, beating full-dataset training.","key_machinery":"The load-bearing object is the self-supervised task-progress predictor: given the first and last frames of a two-second chunk, a frozen DINO-V2 encoder plus a transformer classification head predicts the temporal bin of the task progress $T_p$, and the suboptimality score is the gap $V = T - T_p$ between real elapsed time and predicted progress. This turns the assumption that progress grows steadily with time into a measurable per-transition score: scores are spread back over the transitions in each chunk, temporally discounted so evidence of future suboptimality penalizes earlier transitions, and mixed with the trajectory-level mean so whole bad demonstrations are caught. The second mechanism is state-action deduplication, which extends semantic deduplication to sequential decision making by concatenating a Cosmos video embedding of each chunk with its delta end-effector actions, clustering with K-means, and deleting chunks whose maximum intra-cluster cosine similarity exceeds a threshold. A single threshold pair ($\\epsilon_s = 0.58$, $\\epsilon_d = 0.99$) found on two simulation datasets transfers directly to the real-world Sirius-Fleet data.","core_discovery":"SCIZOR is the first self-supervised, transition-level data curation framework for large-scale imitation learning. It treats a demonstration as a sequence of state-action pairs and scores each pair by two complementary criteria: a progress-prediction module, trained with temporal distance classification on unlabeled video, estimates how many seconds of task progress a short sub-trajectory represents, and when predicted progress falls short of the elapsed time the transitions are flagged suboptimal and removed; a deduplication module encodes visual frames with a pretrained video encoder, concatenates the executed actions, clusters the joint features, and deletes chunks that are near-duplicates of other chunks. Across RoboMimic, Sirius-Fleet, and three Open-X-Embodiment mixtures, policies trained on the curated data improve by an average of 15.4% over policies trained on the full datasets, and the full system outperforms uniform random deletion by 16.1% on average. The method scales to the one-million-trajectory Open-X dataset and benefits both small behavior-cloning policies and a 27M-parameter Octo vision-language-action model.","pith_inferences":["The progress-gap score $V = T - T_p$ is not limited to curation: the same signal could serve as a dense reward surrogate for reinforcement fine-tuning, or as an online trigger for requesting human intervention, since it requires only unlabeled video and elapsed time.","The linear-progress assumption limits the method to monotonic tasks; a hierarchical or multi-timescale progress model, which the paper names as future work, could plausibly extend the same pipeline to tasks with phases such as waiting or stirring.","Transition-level curation is orthogonal to dataset-level mixture weighting, so SCIZOR could be composed with methods like Re-Mix: first filter within each dataset, then reweight across datasets, potentially compounding the gains.","The joint state-action deduplication idea transfers to other domains with demonstration libraries, such as surgical robotics or autonomous driving logs, where visual similarity alone would wrongly merge distinct behaviors."],"forward_implications":["Fine-grained curation at the transition level beats coarser filtering: SCIZOR outperforms DemInf by 19.2% on Sirius-Fleet and Re-Mix by 3.5% on average, while trajectory-level filtering stays competitive only when whole trajectories are uniformly good or bad.","Curation can scale to web-scale robot data: the pipeline runs on the one-million-trajectory Open-X dataset and improves the 27M-parameter Octo vision-language-action model, not just small behavior-cloning policies.","The two filtering signals are complementary: removing only suboptimal transitions or only duplicates each helps, but neither alone reaches the full system's performance on any of the three benchmarks.","Deleting the right data beats deleting the same amount of data arbitrarily: SCIZOR beats uniform random deletion by 16.1% on average, so the gains come from selection, not from reduced dataset size.","Training on less data can yield more robust behavior: on Sirius-Fleet, failure modes such as book-caddy collisions disappeared in policies trained on the curated dataset, and the policies recovered more quickly from errors."],"supporting_citations":[{"why":"Supplies the temporal-distance classification objective that trains the task-progress predictor.","marker":"[16]"},{"why":"The semantic deduplication method that SCIZOR extends from images to joint state-action representations.","marker":"[7]"},{"why":"The RoboMimic benchmark and its behavior-cloning baseline, one of the three evaluation settings.","marker":"[1]"},{"why":"The Octo generalist policy and its 'Magic Soup' mixture weighting used for the large-scale Open-X evaluation.","marker":"[13]"},{"why":"The Open-X-Embodiment dataset, the one-million-trajectory collection that defines the scalability target.","marker":"[15]"},{"why":"DemInf, the trajectory-level mutual-information curation method that fine-grained filtering must beat.","marker":"[6]"},{"why":"Re-Mix, the dataset-level mixture-optimization baseline used in the OXE RT-X comparison.","marker":"[5]"},{"why":"The Sirius-Fleet real-world dataset of policy rollouts with human interventions, the real-robot evaluation setting.","marker":"[48]"},{"why":"The Cosmos video encoder that produces the visual features combined with actions for deduplication.","marker":"[46]"}],"fun_headline_variants":["Self-supervised pruning of bad data boosts robot imitation by 15.4%","SCIZOR automatically curates robot data, lifting success 15.4%","Filtering poor and redundant actions improves robot policies 15.4%","No human labels: SCIZOR cleans imitation data, success up 15.4%","SCIZOR's self-supervised curation raises imitation success 15.4%"],"cache_read_input_tokens":20224,"weakest_assumption_plain":"The method assumes both that task progress grows linearly with elapsed time, so that elapsed time can stand in for expected progress, and that most demonstrations are good enough to learn from, so suboptimal behavior can be detected as a deviation from the majority norm.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised pruning of bad data boosts robot imitation by 15.4%","SCIZOR automatically curates robot data, lifting success 15.4%","Filtering poor and redundant actions improves robot policies 15.4%","No human labels: SCIZOR cleans imitation data, success up 15.4%","SCIZOR's self-supervised curation raises imitation success 15.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000912,"raw_usage":{"total_tokens":3951,"prompt_tokens":1009,"completion_tokens":2942,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2836}},"tokens_in":625,"tokens_out":2942,"duration_ms":24046,"temperature":1.0,"reasoning_tokens":2836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:02:53.497548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SCIZOR's progress predictor on a task that requires deliberate waiting or repeated motion, such as stirring food for a set duration, and check whether those productive transitions are scored as suboptimal and deleted; if policies trained on the curated set fall below the no-curation baseline on that task, the linear-progress assumption is the cause.","supporting_citations":[{"cited_title":"Aytar, T","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal-distance classification objective that trains the task-progress predictor."}],"review_version":1}