{"id":"4c542373-8588-40b2-9e8f-08da5b08618c","arxiv_id":"2608.07895","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MMPF detects and corrects instruction-trajectory mismatches in robot demonstration datasets using local neighborhood voting, global prototype similarity, and entropy-weighted multimodal fusion.","lead":"Robot training data can contain demonstrations that are behaviorally correct but paired with the wrong language instruction, silently corrupting vision-language-action policies. This paper introduces MMPF, a training-free auditor that combines camera and proprioception evidence to flag and relabel such mismatches, reporting gains on LIBERO and two real-robot datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LIBERO instance-dependent noise is generated from the same Cosmos-Embed1 trajectory embeddings that MMPF votes over in Eq. (2), so Table I may overstate MMPF's detection and correction advantage; real-robot uniform-noise results mitigate but do not settle this.","rationale":"I read the paper in good faith as an empirical methods contribution: MMPF is claimed to be the strongest ITM auditor on LIBERO and real-robot benchmarks, with downstream benefits. The real-robot experiments, which use uniform label noise and genuine camera neglect, are independent support and should not be discounted. However, the LIBERO instance-dependent noise protocol in Section IV.A.c constructs corrupted labels from the same Cosmos-Embed1 trajectory geometry that MMPF uses in Eq. (2). Because a flipped episode is defined as one whose nearest neighbors include a different-task demonstration, and the replacement label is taken from that neighbor, the benchmark instantiates the exact local-neighborhood signature MMPF is designed to detect. This is not evidence of misconduct; it is an evaluation-design confound. It plausibly explains the 100% correction accuracy and near-ceiling AUROC on LIBERO. The proposed test—generate mismatches using a different representation or uniform flips—would settle whether MMPF's advantage is real or an artifact of the noise generator. Since this is unresolved but testable, the existing CONDITIONAL verdict is appropriate; I do not move it. The reader's weakest assumption concerned embedding quality; my concern is related but distinct, focusing on the evaluation protocol rather than on whether the embedding geometry holds in deployment.","tokens_in":13230,"tokens_out":5331,"duration_ms":70453,"concrete_test":"Re-run the LIBERO instance-dependent experiments with the same MMPF embeddings but generate mismatches independently of Cosmos-Embed1 space, for example using uniform random task flips, flips based on a held-out action-space distance, or flips generated with a different pretrained video encoder. Compare MMPF against baselines on AUROC, AUPRC, F1, and correction accuracy. If MMPF's margin over baselines collapses, or its correction accuracy drops well below 100%, Table I is confounded by the noise-construction protocol; if the margin persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is the construction of the synthetic LIBERO ITMs, not the embedding-quality assumption itself. Section IV.A.c generates instance-dependent noise by selecting episodes whose nearest neighbors in trajectory-embedding space include different-task demonstrations, then replacing the label with that nearest different-task neighbor's label. This is exactly the signal MMPF's local evidence in Eq. (2) keys on: a mismatched episode is, by construction, adjacent to an episode carrying the replacement label in the same Cosmos-Embed1 space. Consequently, Table I's near-ceiling AUROC/AUPRC values and the 100% correction accuracy partly measure how well the noise generator instantiates MMPF's kNN assumption, not how well MMPF detects naturally occurring ITMs. The uniform-noise real-robot results are independent support and should be credited, but the abstract's first benchmark claim ('Across LIBERO benchmarks with injected instruction mismatches ... strongest') is the part that may be inflated. A competing causal story—MMPF wins because the test set was constructed from its own decision rule—cannot be ruled out from the paper as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies post-hoc auditing of Instruction–Trajectory Mismatches (ITMs) in multimodal robot demonstration datasets. It proposes MMPF, a classifier-free auditor that derives a per-modality task-label distribution from local kNN agreement and global prototype similarity, scores each modality by predictive entropy, and fuses the modalities through a reliability-weighted product of experts. The method is evaluated on LIBERO with two kinds of injected synthetic noise, on two real-robot datasets with 30% uniform noise, and in downstream policy-learning experiments with smolvla and pi0.5, including a counterfactual grounding evaluation. The central claims are that MMPF achieves the strongest overall ITM detection and label-correction accuracy and that auditing improves downstream policy learning where language is needed to disambiguate the task.","tokens_in":13352,"tokens_out":4114,"duration_ms":50466,"significance":"If the claims hold, the paper makes a useful applied contribution: it formalizes ITM auditing for robot demonstrations, provides a post-embedding training-free detector that requires no clean-label supervision, and demonstrates a practical filtering-versus-relabeling trade-off on real hardware. The method is a sensible integration of established ingredients (weighted kNN voting, prototype distances, product-of-experts fusion), and the paper is honest about relying on embedding quality. The most valuable parts are the real-robot auditing results and the counterfactual grounding evaluation, which directly probe instruction grounding rather than only clean-environment success. However, the headline LIBERO claims are weakened by the fact that the synthetic instance-dependent noise is generated from the same trajectory-embedding geometry that MMPF's kNN evidence uses, and several central quantitative comparisons lack uncertainty estimates. These issues are fixable and do not invalidate the approach, but they currently prevent accepting the strongest comparative claims without qualification.","major_comments":[{"comment":"The instance-dependent LIBERO noise construction is aligned with MMPF's own decision rule. Section IV.A.c states that episodes are corrupted when their nearest neighbors in trajectory-embedding space include demonstrations from different tasks, and that the replacement label is taken from the nearest different-task neighbor. This is exactly the signal that MMPF's local evidence in Eq. (2) keys on, using the same Cosmos-Embed1 embeddings. The near-ceiling AUROC/AUPRC values and the 100% correction accuracy in Table I therefore partly measure how faithfully the noise generator instantiates MMPF's kNN assumption, not how well MMPF detects naturally occurring ITMs. The real-robot uniform-noise results are independent and supportive, but they do not settle the magnitude of the LIBERO advantage. Please add an experimental condition in which synthetic noise is generated without using the auditing embeddings—for example, uniform label noise, noise based on task semantics, or noise injected before embedding extraction—and report the same metrics for that condition.","section":"IV.A.c and Table I"},{"comment":"The main auditing comparisons are reported as point estimates without uncertainty. Table I has no error bars or significance tests for any of the eight LIBERO settings, and Tables II, V, and VI likewise report single numbers, despite Table III appropriately reporting 95% Student-t intervals. Because the headline claim is 'strongest overall ITM detection and label correction accuracy,' the reader needs to know whether the observed margins over Confident Learning, Retrieval, and LEMoN are stable under dataset construction or seed variation. Please provide confidence intervals or bootstrap/variance estimates for at least the principal detection and correction metrics in Tables I, II, and V, and perform a significance test for the comparisons that support the abstract's strongest-claim wording.","section":"Tables I, II, V, VI"},{"comment":"The real-robot downstream policy claims are not statistically supported at the reported confidence level. Under filtering-based cleaning on Table, MMPF's 78.8% [68.6,86.3] overlaps the unfiltered baseline's 73.8% [63.2,82.1]; under correction-based cleaning on Bottle-mug, MMPF's 77.5% [62.5,87.7] overlaps the unfiltered baseline's 62.5% [47.0,75.8]. The text says MMPF 'obtains the highest observed point estimate' and 'attains an overall better success rate,' which is fair as a point-estimate statement, but the conclusion should be tempered or backed by additional rollouts, seeds, or a paired comparison. As written, the downstream advantage over no filtering is suggestive rather than established.","section":"Table VII and Section IV.G"},{"comment":"The correction-accuracy metric is conditional on true-positive detection, which makes the reported 100% values harder to interpret. CorrAcc is defined as |{i in I_TP : y_i_hat = y_i}| / |I_TP|, so it measures the reliability of proposed corrections only among episodes already correctly flagged as ITMs. In settings where recall is substantially below precision, a high CorrAcc can coexist with a large number of uncorrected mismatches. The claim that MMPF has 'highly reliable' correction labels is therefore not the same as a claim that MMPF corrects most ITMs in the dataset. Please also report, for each condition, the absolute fraction of all injected ITMs that are both detected and correctly relabeled, or provide a full confusion matrix for the LIBERO results.","section":"IV.A.e and Table I"}],"minor_comments":[{"comment":"The abstract and conclusion call MMPF 'training-free,' but the robot-state encoder in Section IV.A.b is trained (albeit transductively and without task labels). Please consistently say 'training-free after embedding extraction' to avoid misleading readers.","section":"Abstract and Section III"},{"comment":"The clustered noise setting also selects clusters in the same Cosmos-Embed1 embedding space used for auditing; while less directly aligned with the kNN vote than the instance-dependent setting, it still deserves a sentence acknowledging that this shared geometry may favor methods built on the same embedding.","section":"IV.A.c"},{"comment":"The description of the Retrieval baseline is ambiguous: it is said to use 'CLIP-style scoring to the same pretrained trajectory embeddings used by MMPF' and also 'ridge regression to align the video and text embedding spaces.' Please clarify whether the baseline is genuine CLIP scoring or a learned linear alignment, since this affects whether it is a strong comparator.","section":"IV.A.d"},{"comment":"CF-Object is described as an independent re-implementation of the benchmark in [29] because the original is not public. Please state explicitly how the re-implementation was validated for equivalence to the original counterfactual protocol, since the grounding claims in Table IV depend on this variant.","section":"IV.A.a"},{"comment":"The notation would benefit from a short worked example or a table summarizing the values of K, lambda, sigma_m, tau_m, epsilon, and the reliability threshold, since the reader must otherwise infer these from the text in Section IV.A.b and Table VI.","section":"Equations (1)-(6)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the RA-L applied rubric, and the real-robot component is a genuine strength. My main concern is that the synthetic LIBERO evaluation is constructed from the same embedding geometry that MMPF votes over, so the abstract's first benchmark claim is not yet established as a statement about natural ITM detection. The other load-bearing issue is statistical: several central tables report only point estimates, and the real-robot downstream intervals overlap. Both are addressable with additional experiments or reanalysis, so I recommend major revision rather than rejection. I would also gently note to the editor that the novelty is mainly in the problem formulation and the integration of known components, which is acceptable for a letters venue but should be represented accurately in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical methods paper. MMPF — a training-free combination of kNN voting, global prototypes, and entropy-weighted product-of-experts — is a sensible, well-motivated tool for auditing instruction-trajectory mismatches in robot demonstration datasets. The problem is real and the paper is honestly written. The main thing you should know: the synthetic LIBERO results are partly circular, but the real-robot experiments and downstream analysis give independent support.\n\nWhat's new and good: the paper formalizes ITM auditing as a distinct curation task, and the integration of established components is competently done. The reliability weighting is a nice touch: per-episode entropy gating makes sense for sensor corruption. The evaluation is extensive — several benchmark suites, real robot data, ablations, downstream policy learning, and a counterfactual grounding analysis. The authors are transparent about hyperparameters: K, lambda, sigma, and tau are fixed or set from unsupervised geometry, and the robot-state encoder is trained without task labels, so there is no fudging to the auditing target.\n\nSoft spots, in order of size. First, the instance-dependent noise in LIBERO is generated from the same Cosmos-Embed1 embeddings MMPF uses. Section IV.A.c flips labels to the nearest different-task neighbor's label for episodes that are already boundary-near in that embedding space. That is exactly the local signal MMPF's Eq. (2) keys on. So the near-ceiling AUROC and 100% correction accuracy in Table I partly measure how well the noise generator instantiates the method's assumptions, not how well it detects naturally occurring mismatches. This matters because the abstract's first benchmark claim rests on those numbers. It is not fatal: the real-robot results use uniform noise and still show a large advantage, and the downstream improvements on LIBERO-Goal/Spatial and CF-Object are consistent with better instruction grounding.\n\nSecond, the main auditing tables (I, II, V, VI) have no error bars and no multiple noise seeds. The policy tables have intervals, but the core detection metrics are point estimates, which makes the absence of code or data worse for reproducibility. Third, the real-robot policy comparisons are underpowered: 10 rollouts per condition, and some Wilson intervals in Table VII overlap between MMPF and CL. The point estimates favor MMPF, and the correction accuracy gap is large, but the sample sizes are small. Fourth, CF-Object is an independent re-implementation of a not-yet-public benchmark; the grounding conclusions should be treated as provisional until the original benchmark is released.\n\nWho this is for: robot learning practitioners building instruction-conditioned VLA datasets. It deserves a serious referee, mainly to push on the noise-generation setup and demand error bars or code. I'd send it out.","headline":"A useful, clearly-written data-curation method for ITM auditing, with real-robot support, but the synthetic LIBERO results are partly inflated by the noise generator's use of the same embeddings.","tokens_in":14005,"tokens_out":2952,"would_cite":true,"duration_ms":33655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that instruction–trajectory mismatches in multimodal robot demonstration datasets can be detected and corrected post-hoc by a training-free fusion of per-modality neighborhood and prototype evidence, improving downstream…","keywords":["instruction-trajectory mismatch","robot demonstration auditing","multimodal fusion","product-of-experts","label error detection","data curation","learning from demonstration","vision-language-action models"],"falsifier":"On a dataset where proprioception and motor trajectories are nearly identical across two tasks that differ only by object identity, inject 30% label flips and run MMPF: if correction accuracy remains near 100% while proprioception-only evidence is at chance, the method does not need per-modality task geometry; if it collapses, the embedding-geometry assumption is the operative load-bearing factor.","tokens_in":12925,"feed_emoji":"🤖","tokens_out":6863,"duration_ms":64321,"temperature":0.7,"pith_summary":"Robot demonstrations used to train vision-language-action policies can be behaviorally correct yet paired with the wrong language instruction, a failure mode the paper calls an instruction–trajectory mismatch (ITM). The paper proposes Multimodal Probabilistic Fusion (MMPF), a training-free auditor that, for each sensing modality, builds a task-label distribution from local neighborhood agreement and global prototype similarity, then fuses the modalities with reliability-weighted product of experts. On LIBERO benchmarks with injected mismatches and on two real-robot datasets, MMPF achieves the strongest ITM detection and label-correction accuracy among the compared baselines. The paper also shows that auditing improves downstream policy learning most when language is needed to disambiguate the task, and that relabeling can beat filtering in small real-robot datasets.","feed_headline":"Auditor catches and fixes mismatched robot demo labels","feed_subtitle":"Fusing camera and robot-state evidence flags and fixes wrong instruction labels, improving downstream policy success.","key_machinery":"The central object is the reliability-weighted product of experts over unimodal task-label distributions. For modality $m$ and episode $i$, MMPF forms a distribution $p_m(y|i)$ by taking the geometric mixture of an RBF-weighted kNN vote (Eq. 2) and an exponential distance to per-task prototypes (Eq. 3). Predictive entropy converts each unimodal posterior into a reliability weight $r_{i,m}=1-H(p_m)/\\log C$, and the final posterior is $\\log p(y|i)\\propto \\sum_m r_{i,m}\\log p_m(y|i)$. This mechanism does the work: a modality that is occluded, noisy, or task-ambiguous produces a diffuse posterior, gets downweighted, and is prevented from dragging the joint prediction; agreement across modalities sharpens the final label. The same reliability scores can be thresholded to flag sensor-corrupted episodes independently of label errors.","core_discovery":"The paper claims that instruction–trajectory mismatches can be found and corrected without training a classifier or invoking a VLM, by exploiting the geometry of pretrained trajectory embeddings. Each modality (head camera, wrist camera, proprioception) votes on the true task using an RBF-weighted k-nearest-neighbor histogram and a distance-to-prototype distribution; the two are combined by a geometric mixture, and the resulting distributions are fused by a product of experts weighted by per-episode predictive entropy. Across the eight LIBERO settings with instance-dependent and clustered noise, MMPF reaches 88.9–100.0 F1, near-perfect AUROC/AUPRC, and 100.0% correction accuracy among detected true ITMs. On real-robot bottle-mug and table datasets it attains 91.8 and 94.3 F1 with 97.9% and 94.9% correction accuracy. Downstream, MMPF-curated training raises LIBERO-Goal success from 58.7 to 77.5 and LIBERO-Spatial from 70.3 to 76.7, and on a counterfactual object-grounding benchmark it increases faithful grasp rate from 28.3% to 41.4% while cutting biased grasps from 40.3% to 14.3%.","pith_inferences":["Editorial: The framework's contribution is best read as an auditing head on top of whatever trajectory embeddings exist, so its accuracy in practice will ride on the next generation of proprioceptive and video foundation models rather than on the fusion itself.","Editorial: The reliability score could be used online during data collection to flag uncertain or sensor-corrupted episodes in real time, turning a post-hoc auditor into a collection-time filter.","Editorial: The finite-task-label assumption could be relaxed by replacing hard classes with soft neighborhoods in instruction-embedding space (as the paper itself notes), which would let the same fusion audit open-vocabulary paraphrases; a concrete test is to apply MMPF to a dataset with naturally occurring instruction paraphrases and compare corrections against human audit.","Editorial: A natural stress test is to inject mismatches with adversarial similarity, choosing wrong labels whose trajectories are closest in embedding space; the paper's synthetic instance-dependent noise approximates this, but an explicitly adversarial version would bound the method's robustness."],"forward_implications":["ITM auditing becomes a distinct stage in robot data curation, complementary to filtering failed rollouts or redundant trajectories.","Relabeling flagged demonstrations can outperform dropping them when trajectories remain valid, especially in small real-robot datasets.","The per-modality reliability score doubles as a sensor-corruption detector, so one auditing pass surfaces both label errors and broken observations.","Counterfactual grounding metrics reveal downstream benefits of auditing that clean-environment success alone hides.","Because the audit stage is training-free, the method transfers to any dataset that has pretrained trajectory embeddings, including future foundation-model encoders."],"supporting_citations":[{"why":"Supplies the LIBERO benchmark suites (head camera, wrist camera, proprioception) on which ITM detection metrics are measured.","marker":"[28]"},{"why":"Provides the Cosmos-Embed1 trajectory embeddings shared by MMPF and all baselines for visual modalities.","marker":"[30]"},{"why":"Confident Learning is the classifier-confidence baseline whose label-error detection MMPF must beat.","marker":"[21]"},{"why":"LEMoN is the multimodal nearest-neighbor label-error baseline that MMPF compares against.","marker":"[23]"},{"why":"DIAL underlies the retrieval-style instruction–trajectory consistency baseline used for comparison.","marker":"[13]"},{"why":"smolvla is the policy trained from scratch on simulated datasets after auditing, used for downstream success evaluation.","marker":"[31]"},{"why":"pi0.5 is the policy trained from scratch on real-robot datasets after auditing, used for downstream success evaluation.","marker":"[7]"},{"why":"Defines the counterfactual evaluation paradigm on which CF-Object grounds MMPF's instruction-following benefit.","marker":"[29]"}],"fun_headline_variants":["No-training auditor catches wrong robot demo labels","Fusing camera and robot data finds mismatched instructions","Wrong labels in robot demos? MMPF finds and fixes them","Audit corrects robot label errors, boosting policy success","Training-free method detects and fixes robot label mismatches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"MMPF assumes that, within each modality, demonstrations of the same task are geometrically close and demonstrations of different tasks are far apart, so that nearest neighbors and prototypes reveal the true task rather than viewpoint, object color, or robot-state noise; the paper states the same dependence in its conclusion.","fun_headline_variants_meta":{"raw":{"variants":["No-training auditor catches wrong robot demo labels","Fusing camera and robot data finds mismatched instructions","Wrong labels in robot demos? MMPF finds and fixes them","Audit corrects robot label errors, boosting policy success","Training-free method detects and fixes robot label mismatches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1310,"prompt_tokens":998,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":614,"tokens_out":312,"duration_ms":3933,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:42:30.167201+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset where proprioception and motor trajectories are nearly identical across two tasks that differ only by object identity, inject 30% label flips and run MMPF: if correction accuracy remains near 100% while proprioception-only evidence is at chance, the method does not need per-modality task geometry; if it collapses, the embedding-geometry assumption is the operative load-bearing factor.","supporting_citations":[{"cited_title":"Libero: Benchmarking knowledge transfer for lifelong robot learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the LIBERO benchmark suites (head camera, wrist camera, proprioception) on which ITM detection metrics are measured."}],"review_version":1}