{"id":"ac151dcd-c7fe-4199-ae56-ff008ebf600a","arxiv_id":"2505.09577","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VTLA, a vision-tactile-language-action model trained purely in simulation, achieves over 90% success on peg insertion in simulation and 95% in the real world.","lead":"Researchers built a robot control model that combines camera images, fingertip touch data, and language instructions to guide insertion of pegs into holes. The model trained only in simulation and still completed most real-world insertions, which matters because touch and language together are useful for hard-to-see assembly tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own TLA baseline collapses from 80–92% simulation success to 30–40% real-world success, showing the simulated tactile images are unvalidated; VTLA's 95% real-world result may therefore be driven by vision alone, leaving the central vision-tactile fusion claim unsupported.","rationale":"The reader's weakest assumption is exactly the sim-to-real fidelity of the simulated tactile images. The paper's self-reported TLA result is the strongest available evidence on that assumption: a policy trained on the same simulated tactile data loses more than half its success rate when transferred to the real robot. That is not an external critique; it is an internal inconsistency between the claim that tactile fusion drives VTLA's success and the demonstrated failure of tactile-only transfer. The central headline number, 95% real-world success, might survive even if tactile is useless, but the paper's contribution as a vision-tactile-language-action model would not: it would reduce to a VLA with an unused input channel. The real-world VTLA-versus-VLA gap is small and based on 20 trials, so it cannot rescue the tactile claim. The only qualitative evidence for tactile value, the poor-lighting comparison in Appendix C, lacks success rates and trial counts, making it anecdotal. Other concerns, such as the lack of confidence intervals, code release, and the modest DPO improvement, are real but secondary; they affect the strength of the evidence, not the identity of the claimed contribution. The correct disposition remains CONDITIONAL, but the acceptance condition should specifically require a real-world tactile ablation. My read does not change the reader's verdict, hence UNCHANGED.","tokens_in":13663,"tokens_out":4488,"duration_ms":46280,"concrete_test":"Run the same real-world 0.6 mm clearance protocol used for Table 7 (20 trials per condition, same initial misalignment distribution) with VTLA under three input ablations: (a) full vision+tactile; (b) tactile tokens replaced by blank or zero images, leaving vision intact; and (c) vision obscured (e.g., darkened as in Appendix C) with tactile intact. If (a) and (b) have statistically indistinguishable success rates and steps, the tactile channel is not load-bearing and the fusion claim fails. If (c) does not yield reliable insertion, the sim-to-real tactile transfer is not functional. Report per-condition trial counts and confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: a sim-trained policy reaches 95% real-world insertion success, and this success results from effective vision-tactile-language integration. The second part is load-bearing. The self-built visuotactile simulator's tactile images are never quantitatively compared with real GelStereo 2.0 outputs, and the paper itself provides a direct falsification probe: TLA, trained on the same simulated tactile data, achieves 80–92% success in simulation but only 30–40% in the real world (Table 7). That sim-to-real collapse is strong evidence that the simulated tactile images are not faithful. Under this condition, VTLA's real-world performance can be explained by the wrist-camera vision stream alone. The real-world VTLA-versus-VLA comparison (Table 7) is within the noise of 20-trial runs (95 vs 90 on triangle; 100 vs 100 on pentagon; mean steps 3.94 vs 4.06 and 1.85 vs 2.3), so it does not establish a tactile benefit. The only evidence offered for tactile contribution is the qualitative poor-lighting demo in Appendix C, which reports no success rates or trial counts. Consequently, the paper's central claim that VTLA 'effectively integrat[es] visual and tactile inputs' and that VGTE tactile tokens improve performance is not currently supported: the headline result may be a VLA result with a non-functional tactile channel.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VTLA, a vision-tactile-language-action model for peg-in-hole insertion, trained exclusively on simulated data from a self-built visuotactile simulator in Isaac Gym. The contributions are a 28,000-sample vision-tactile-action-instruction dataset, a Vision-Guided Temporally Enhanced Tokens (VGTE) design for temporal fusion, and a Direct Preference Optimization (DPO) stage intended to provide regression-like supervision. The authors evaluate in simulation against a diffusion policy, VLA, and TLA baselines, and report real-world experiments on a UR3 robot with GelStereo 2.0 sensors, claiming 95% real-world success and over 90% success on unseen peg shapes. The central claim is that a policy trained only on simulated data can achieve this performance through effective integration of visual and tactile inputs via language grounding.","tokens_in":13955,"tokens_out":8053,"duration_ms":72899,"significance":"If the claims are substantiated, the paper would be a valuable demonstration of sim-to-real transfer for multimodal insertion policies and would support the case for tactile-language grounding in VLA models. The authors have made a serious effort: the simulation dataset is large, the real-world setup mirrors the simulation, and the evaluations cover multiple peg shapes and clearances. The DPO ablation and domain randomization details are useful. However, the central claim requires better evidence; in particular, the tactile channel's contribution is not isolated, the TLA baseline's sim-to-real collapse suggests the simulated tactile images are not validated, and the reported real-world differences are within sampling noise. The contribution is therefore conditionally significant: it would be strong after additional experiments and analysis.","major_comments":[{"comment":"The simulated tactile channel is never quantitatively validated against real GelStereo 2.0 outputs, yet the central claim that VTLA 'effectively integrates visual and tactile inputs' depends on the tactile channel being informative in the real world. The paper's own TLA baseline, trained on the same simulated tactile data, achieves 80–92% success in simulation (Tables 2 and 3) but only 30–40% in the real world (Table 7). This sim-to-real collapse is direct evidence that the simulated tactile images are not faithful. The authors should provide a quantitative comparison of simulated and real tactile images (e.g., image statistics or a sensor transfer evaluation) or a real-world ablation in which the tactile stream is removed from VTLA under identical conditions. Without this, the 95% real-world success rate can be explained by the wrist-camera vision stream alone, and the paper does not support its central fusion claim.","section":"§3.1, Appendix A, Table 7"},{"comment":"The real-world comparison between VTLA and VLA does not establish a tactile benefit. With 20 trials per condition, the success-rate differences are 95 vs 90 on the triangle and 100 vs 100 on the pentagon, and the mean step differences are 0.12 and 0.45 steps. No confidence intervals, standard deviations, or significance tests are reported. The claim that VTLA achieves 'better insertion efficiency' is therefore not supported by the data. The qualitative poor-lighting demonstration in Appendix C reports no success rates or trial counts and cannot substitute for a quantitative comparison. Additional trials or a paired statistical test are needed, or the manuscript should be revised to state the evidence as suggestive rather than conclusive.","section":"§4.4, Table 7, Appendix C"},{"comment":"The DPO ablation does not show improvement on in-distribution data: ID GCR is 47.5 both with and without DPO, and the L1 differences are at the third decimal. The OOD GCR gain is 4.4 percentage points (from 27.0 to 31.4). The statement that 'preference learning with DPO significantly improves performance on both ID and OOD data' is therefore overstated, and the conclusion that DPO 'mitigates overfitting to ground-truth actions' is not directly evidenced. Moreover, the preference labels are generated by ranking the SFT model's own outputs by distance to the ground-truth action, so the DPO stage is essentially a ranking loss on regression residuals; the paper should acknowledge this and show that the improvement is not simply due to additional training iterations on the same data.","section":"§4.3, Table 4"},{"comment":"The Goal Convergence Rate (GCR) is defined as 'the percentage of actions that are all correct in the x, y, and rz directions,' but no tolerance for 'correct' is specified. GCR is the primary metric in Tables 1 and 4 and underpins the comparison of VTLA with baselines and ablations, so an unspecified tolerance makes the metric uninterpretable. Please specify the exact tolerance(s) used and report sensitivity of GCR to the tolerance value.","section":"§4.1, Tables 1 and 4"},{"comment":"Vision-Guided Temporally Enhanced Tokens (VGTE) are presented as the first technical contribution, but they are never isolated in the experiments. The only ablation is for DPO (Table 4); the comparison of VTLA versus VLA in Table 1 conflates the addition of tactile input with the VGTE token design. An ablation of VTLA with tactile tokens fed without the proposed temporal enhancement and without the vision-guided ordering is needed to support the claim that VGTE improves performance. Without this, the contribution of VGTE is not demonstrated.","section":"§3.2, Tables 1 and 4"},{"comment":"The abstract claims 'over 90% success rates on unseen peg shapes.' In simulation, the OOD triangle peg achieves 88% success (Table 3), so the claim is inaccurate for the simulation results. If the claim is intended to refer only to the real-world experiments, the abstract should make that explicit, since the simulation results are the primary quantitative evaluation of generalization.","section":"Abstract, §4.2, Table 3"}],"minor_comments":[{"comment":"The text says the tactile sensing program runs at '20 PFS'; this should be '20 FPS' (frames per second).","section":"Appendix B"},{"comment":"The VGTE description is high-level; please provide a precise definition or pseudocode for how temporal fusion is applied before the ViT and how the vision-guided ordering is realized.","section":"§3.2"},{"comment":"For the real-world evaluation, the reported success rates and mean steps are percentages and averages over 20 trials; please report per-trial variance or standard error, and consider binomial confidence intervals for success rates.","section":"§4.1 and §4.4"},{"comment":"The tactile image sequence is described as 'arranged in a 2×2 grid,' but the temporal order is not explained; clarify whether the grid is a 2×2 spatial arrangement of the four frames or a 2×2 tiling for input.","section":"Figure 2"},{"comment":"Since TLA is the authors' prior work and serves as a baseline, the paper should explicitly state the architectural differences between VTLA and TLA (e.g., addition of vision, VGTE, DPO) and what new evidence is provided beyond that prior work.","section":"Related Work [36]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core contribution hinges on demonstrating a tactile benefit, but as written the evidence is not yet sufficient. The DPO preference data are generated from the model's own outputs ranked by distance to ground-truth actions, so the 'preference learning' framing deserves scrutiny. The authors' prior TLA work is a natural baseline, and the paper should more clearly differentiate the new architectural and experimental contributions from that prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things about this paper. First, the sim-to-real result itself is real: a Qwen2-VL model fine-tuned on simulated vision-tactile data hits 95-100% success on real peg insertion across shapes. Second, the paper's claim that tactile fusion drives this result is not supported by their own data. The TLA baseline collapses from 80-92% in simulation to 30-40% in the real world, which strongly suggests the simulated tactile images are not faithful to the GelStereo 2.0 output. On the real robot, VTLA vs VLA is 95 vs 90 on triangle and 100 vs 100 on pentagon—within noise at 20 trials. So the tactile channel may be a no-op in practice.\n\nThat said, there are real contributions. The integration of vision, tactile, and language via a VLM is reasonably novel, the VGTE token ordering is a sensible design, and the DPO stage for action tokens is a neat trick. The simulation experiments are fairly comprehensive: multiple shapes, clearances, and ablations. The dataset construction and domain randomization are described in enough detail to reproduce, though no code or data are released.\n\nSoft spots, in order of severity:\n\n1. The real-world evidence for tactile benefit is weak. No confidence intervals, no statistical tests, 20 trials per condition. The poor-lighting appendix (C) is qualitative—no rates or counts. The stress-test point about TLA's collapse is a real falsification probe: the sim tactile images are never compared to real ones, and the only tactile domain randomization is color jittering.\n\n2. The abstract overstates the numbers. Table 3 shows 88% on triangle OOD, not \"over 90.\" Table 6 title claims \"100% on OOD shapes\" while triangle and hexagon are 95. The conclusion says \"exceeding 90%\" which is also false for the sim triangle. These are careless but fixable.\n\n3. The DPO benefit is small and the preference labels come from the model's own outputs ranked by distance to ground truth—circular in spirit, though the final eval is independent. OOD GCR goes from 27.0 to 31.4 (1k), no ID change; the 16% claim is relative, and the GCR tolerance is a free parameter.\n\n4. No artifact release. No code, no model weights, no dataset. That matters more for a benchmark claim.\n\nWho is this for? People working on tactile VLA and sim-to-real for contact-rich manipulation. It's a reasonable starting point and the failure modes are instructive. But as a benchmark result it needs stronger evidence and more honest reporting.\n\nMy recommendation: send it to peer review with the expectation of major revision. The core idea is defensible, the sim results are consistent, and the real-world gap analysis is exactly what referees should push on. If the authors can release data/code, report uncertainties, and either get the tactile advantage above noise or reframe the contribution as a VLA result with a tactile side channel, it could be a solid paper. As-is, it does not support the central fusion claim.","headline":"Sim-to-real insertion works, but the tactile fusion claim is not backed by the real-world numbers; the abstract overstates results.","tokens_in":14494,"tokens_out":5460,"would_cite":false,"duration_ms":47060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model trained on purely simulated vision-tactile data reaches 95% real-world insertion success and generalizes to unseen peg shapes.","keywords":["vision-tactile-language-action model","peg-in-hole insertion","direct preference optimization","simulation-to-real transfer","tactile sensing","contact-rich manipulation","language-conditioned robotic manipulation","domain randomization"],"falsifier":"Run the tactile-only TLA model on the same real 0.6 mm square-peg setup used in Table 7: if its real-world success rate stays near the reported 30-40% while simulation success is 80-90%, the tactile sim-to-real transfer is the weak link. Alternatively, compare VTLA with a vision-only VLA on the real 0.6 mm task under normal lighting; if the vision-only model matches VTLA's 95%, the tactile modality is not load-bearing.","tokens_in":13455,"feed_emoji":"🤖","tokens_out":6897,"duration_ms":66276,"temperature":0.7,"pith_summary":"The paper introduces VTLA, a model that combines a wrist-camera image, sequences from two fingertip tactile sensors, and a text instruction to predict a small corrective action ($\\Delta x$, $\\Delta y$, $\\Delta r_z$) for peg-in-hole insertion. The central claim is that this policy, trained entirely on 28,000 simulated samples with domain randomization and then refined by direct preference optimization, transfers to a real robot: 95% success at 0.6 mm clearance on a square peg, 100% on unseen round and pentagon pegs, and higher insertion efficiency than a vision-only baseline. The work argues that tactile input adds a real margin over vision alone in contact-rich assembly, and that preference learning gives a regression-like training signal that better matches continuous control than plain next-token prediction.","feed_headline":"Sim-only training hits 95% success on real robot insertion","feed_subtitle":"Vision plus touch beats vision-only and touch-only baselines in tight-clearance peg assembly.","key_machinery":"The load-bearing mechanism is the Vision-Tactile-Language-Action token stream. Tactile frames from both fingertips are arranged in a $2\\times2$ grid and encoded by a vision transformer; the wrist camera image is placed after the tactile tokens, closer to the action output, to counter the recency bias of language models and emphasize visual cues. These tokens, together with the text instruction, are fed into a pretrained vision-language model that predicts the action tokens. The second mechanism is Direct Preference Optimization (DPO): candidate actions are generated by sampling from the fine-tuned model, labeled chosen or rejected by their closeness to the ground-truth action, and the model is optimized to prefer the closer one, giving a regression-like supervision signal.","core_discovery":"On its own terms, the paper claims that a single autoregressive model can serve as a contact-rich manipulation policy: given a language instruction, a wrist image, and tactile frame sequences, it emits the next corrective action for insertion. The discovery is that this works when trained only in simulation. In the toughest real-world test, a square peg with 0.6 mm clearance, the model trained solely on simulated data succeeds 95% of the time across 20 trials, requires on average 4.31 attempts, and reaches 95-100% success on unseen peg shapes at the same clearance. The authors attribute the margin over vision-only and tactile-only alternatives to the specific token ordering and temporal encoding of tactile frames, and to direct preference optimization as a preference-based, regression-like loss.","pith_inferences":["The paper's tactile augmentation is limited to color jittering; a learned or physics-based sim-to-real tactile translation could make the tactile channel more informative and is a natural next step.","Because the vision-only baseline also achieves over 90% success in several settings, the specific value added by touch is most visible in tight-clearance and poor-lighting cases; quantifying that margin would be valuable.","The reported DPO saturation with preference dataset size hints that diversity of preference pairs matters more than quantity.","The same instruction-tuned delta-action framework may extend to other contact-rich tasks such as screwing, connector mating, and cable insertion without architectural changes."],"forward_implications":["Contact-rich assembly policies can be learned from simulation alone, avoiding expensive real demonstration collection.","Adding a tactile channel to a vision-language-action model improves both success rate and efficiency at tight clearances.","Preference learning with DPO improves out-of-distribution generalization over plain next-token supervision.","Language-conditioned action prediction works for continuous low-level control when actions are framed as corrective deltas.","Assembly success holds across multiple peg shapes and clearances, suggesting the learned skill is not memorized for one geometry."],"supporting_citations":[{"why":"Supplies the pretrained vision-language model and vision encoder that VTLA fine-tunes.","marker":"[55]"},{"why":"Provides the Direct Preference Optimization objective used for preference learning.","marker":"[37]"},{"why":"Defines the TLA baseline, task setup, and data alignment that VTLA extends with visual input.","marker":"[36]"},{"why":"The diffusion policy baseline that VTLA is compared against in simulation.","marker":"[58]"},{"why":"Describes the tactile sensor used in the real-world evaluation.","marker":"[59]"},{"why":"Supports the decision to freeze the vision encoder and modality adapter during supervised fine-tuning.","marker":"[57]"}],"fun_headline_variants":["95% real peg insertion from sim-only training","Vision-tactile model nails 95% real insertion","Sim-trained VTLA achieves 95% on real peg insertion","Cross-modal model: 95% success on tight clearance","Preference learning boosts sim-to-real insertion to 95%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The unstated premise carrying the whole result is that the simulated tactile images are close enough to real sensor output that a policy trained only on simulated touch still behaves correctly on real hardware; if that gap is large, the tactile channel may be contributing little and the reported success could come mostly from vision.","fun_headline_variants_meta":{"raw":{"variants":["95% real peg insertion from sim-only training","Vision-tactile model nails 95% real insertion","Sim-trained VTLA achieves 95% on real peg insertion","Cross-modal model: 95% success on tight clearance","Preference learning boosts sim-to-real insertion to 95%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2946,"prompt_tokens":919,"completion_tokens":2027,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1946}},"tokens_in":535,"tokens_out":2027,"duration_ms":14191,"temperature":1.0,"reasoning_tokens":1946,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:27:53.695474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the tactile-only TLA model on the same real 0.6 mm square-peg setup used in Table 7: if its real-world success rate stays near the reported 30-40% while simulation success is 80-90%, the tactile sim-to-real transfer is the weak link. Alternatively, compare VTLA with a vision-only VLA on the real 0.6 mm task under normal lighting; if the vision-only model matches VTLA's 95%, the tactile modality is not load-bearing.","supporting_citations":[{"cited_title":"Rafailov, A","cited_arxiv_id":null,"evidence_quote":"Provides the Direct Preference Optimization objective used for preference learning."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"Describes the tactile sensor used in the real-world evaluation."}],"review_version":1}