Pith. sign in

REVIEW 4 major objections 9 minor 34 references

A world model that jointly imagines vision, contact, force geometry, and slip plans contact-rich robot actions more accurately than vision alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 19:37 UTC pith:L7OC4W4Q

load-bearing objection Solid systems paper: hierarchical touch + contact-gated fusion actually moves LPIPS and real-robot CEM success; the hard gate under counterfactual actions is the unmeasured risk, not a shown failure. the 4 major comments →

arxiv 2607.24267 v1 pith:L7OC4W4Q submitted 2026-07-27 cs.RO

FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning

Wenxuan Ma , Chaofan Zhang , Chao Xue , Yinghao Cai , Guocai Yao , Shaowei Cui , Shuo Wang This is my paper
classification cs.RO
keywords visuo-tactile world modelhierarchical contact predictioncontact-gated attentionslip predictionCEM planningcontact-rich manipulationlatent dynamics3D tactile point clouds
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robots that plan by imagining futures usually only simulate how things look, so their rollouts can look right while violating contact physics. FeelWorld is a shared latent dynamics model that also predicts three hierarchical tactile states—whether contact has occurred, a 3D tactile latent encoding force-related deformation, and whether slip is starting—and uses those states both to ground the imagined future and to score candidate actions. A contact gate keeps free-space tactile noise from corrupting the visual stream and only turns on joint visuo-tactile fusion and tactile planning costs once contact is predicted. On chip grasping, fruit grasping, and USB insertion, this cuts short-horizon visual error, keeps long rollouts far more stable, and raises zero-shot model-based planning success to about 82 percent on average. The practical claim is that structured, selectively fused touch is what makes imagined contact futures physically usable.

Core claim

FeelWorld shows that jointly predicting future visual latents together with hierarchical tactile states—contact, a force-related 3D tactile latent, and slip—under a contact-gated asymmetric attention mechanism yields imagined futures that are both visually sharper and physically more consistent than vision-only or naive visuo-tactile baselines, and that those predicted contact and slip states enable contact-aware CEM planning with substantially higher zero-shot success on contact-rich tasks.

What carries the argument

Contact-gated asymmetric attention: tactile tokens always ground themselves in vision, but visual tokens query tactile features only when a predicted binary contact gate is open, so free-space sensor noise cannot degrade appearance dynamics while contact-phase dynamics become jointly visuo-tactile.

Load-bearing premise

A hard on/off contact threshold is enough to decide when touch is pure noise versus when it should fully reshape both the visual prediction and the planner’s cost—so gating mistakes do not cascade into bad fusion or unreachable objectives.

What would settle it

Run the same chip, fruit, and USB tasks with systematically delayed or advanced contact labels (or sensors that chatter near onset) and check whether 10-step and 80-step LPIPS, contact/slip F1, and CEM success collapse once the gate is wrong at transition frames.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • World models for contact-rich work should predict contact, local force geometry, and slip as separate supervised levels rather than one undifferentiated tactile target.
  • Free-space tactile readings should be gated out of visual dynamics and planning costs until contact is predicted.
  • Contact-aware CEM can switch from vision-only approach costs to joint visuo-tactile goals after predicted contact, avoiding unreachable pre-contact tactile objectives.
  • Autoregressive rollout training with context noise reduces compounding error enough for multi-second imagined horizons to stay usable for planning.
  • Zero-shot planning success on fragile grasp and occluded insertion can roughly double relative to a strong visual-only latent world model when hierarchical touch is added this way.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same contact gate could be reused as a curriculum signal for policy learning: train approach with vision only, then unlock tactile losses only after the model’s own contact head fires.
  • If slip F1 remains the weakest link under class imbalance, denser temporal tactile context or multi-finger slip heads may be the next bottleneck for insertion under occlusion.
  • Hard binary gating may need a soft or uncertainty-aware version for deformable or multi-contact scenes where “contact” is graded rather than instantaneous.
  • Pairing the world model with a fast policy evaluator, as the authors sketch, would test whether hierarchical tactile rollouts still help when CEM’s sample budget is no longer affordable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. The paper presents FeelWorld, an action-conditioned visuo-tactile world model for contact-rich manipulation. Tactile state is decomposed hierarchically into a binary contact state, a 3D tactile latent (encoded by a frozen FG-CLTP encoder over tactile point clouds), and a slip state, all predicted by a shared latent dynamics transformer with explicit per-level supervision. A contact-gated asymmetric attention mechanism lets tactile tokens always attend to visual tokens while visual tokens attend to tactile tokens only when a hard binary gate g_t = I[ĉ_t ≥ 0.5] is open, protecting visual prediction from free-space tactile noise. Training combines teacher forcing with R-step autoregressive rollout and Gaussian context-noise injection. At inference, a contact-aware CEM planner scores candidate action sequences with a visual goal-distance term over the full horizon plus tactile-goal and slip-penalty terms that activate only after predicted contact. On chip grasping, fruit grasping, and USB insertion (Imeta-Y1 robot, 200 training / 40 test trajectories per task, 6 fps), the model improves 10-step LPIPS from 0.084 (visual-only V-JEPA 2) to 0.058, stays 61% below the visual baseline after an 80-step rollout, reaches 98.1%/83.4% F1 on contact/slip, and achieves 81.7% average zero-shot planning success vs 49.2% visual-only over 40 real-robot trials per task.

Significance. If the results hold, this is a solid contribution to visuo-tactile world modeling. Strengths worth naming explicitly: (i) real-robot zero-shot CEM planning evaluated over 40 trials per task on three physically distinct contact-rich tasks, including USB insertion under occlusion where the gap over the visual baseline is largest (75.0% vs 37.5%); (ii) a component-wise ablation in Table I that separates tactile input, hierarchical supervision, and gating, so the design claims are not merely asserted against external baselines; (iii) long-horizon autoregressive evaluation out to 80 steps (Fig. 5) rather than only short-horizon teacher-forced metrics; (iv) honest inclusion of ~30% failed trajectories in the evaluation set; and (v) concrete, falsifiable prediction metrics (contact/slip F1, onset error) tied to sensor-API ground truth rather than self-referential targets. The hierarchical contact/tactile-latent/slip decomposition and the gated fusion idea are simple and transferable. The main limit on significance is that the planning attribution currently rests on an untested assumption about gate behavior under counterfactual actions; if the requested controls confirm the story, the 32.

major comments (4)
  1. [Sec. III-E, Eq. (15), Algorithm 1; Table III; Table IV] The planning attribution rests on the gate activating at the correct time under counterfactual actions, but gate accuracy is only measured open-loop. Table III (contact acc. 98.6%, onset error 0.44 frames) is computed 'over all test frames', i.e., rollouts along logged trajectories with the executed actions. The CEM planner, however, evaluates N=400 sampled action sequences per cycle (Algorithm 1, lines 3–14), most of which are off-distribution relative to the 200 training trajectories per task, and the gate g_k = I[ĉ_{t+k} ≥ 0.5] in Eq. (15) then decides when the tactile-goal and slip costs switch on. A mistimed onset mis-scores candidates asymmetrically (late onset defers tactile costs, so contact-seeking sequences are under-valued; early onset activates slip penalties on free-space motions). The 81.7% vs 49.2% gap in Table IV is attributed to correct contact-stage decomposition, but n
  2. [Sec. III-C, Eq. (12); Sec. III-E, Eq. (15); Fig. 6(b)] The hard threshold and binary switch are never ablated as design choices. Table I ablates only gate on/off; there is no sweep of the 0.5 threshold, and no soft-gate variant (g_k = ĉ_{t+k} in Eq. (15), or multiplicative gating by ĉ_t in Eq. (12)) is compared. A soft gate is the obvious alternative — it degrades gracefully under calibration error exactly where the hard gate is most brittle — and the manuscript's own Fig. 6(b) shows the predicted contact probability hovering near the 0.5 threshold around transitions, the regime where a hard switch is maximally sensitive to small probability errors. Given that the gate's sharpness is a central design claim ('contact-gated asymmetric attention', 'contact-aware CEM'), at least one soft-gate row in Table I and Table IV (or a threshold sensitivity curve) is needed to show the binary choice is justified rather than incidental.
  3. [Table IV, Sec. IV-D] Real-robot success rates are reported without any statistical treatment. With 40 trials per task, the binomial standard error at 75% success is ~6.8pp, so the USB result (75.0% gated vs 50.0% naive) is likely but not certainly significant, and the chip gap (82.5% vs 47.5%) is safer. Since the planning claims are the paper's headline contribution and rest on 120 total real-robot trials, Table IV should include Wilson confidence intervals or a significance test (e.g., two-proportion z-test or Fisher exact) per task. Relatedly, please state whether the same goal configurations / initial conditions were used across the three planners, and whether trial order was randomized, to rule out systematic drift effects (object wear, sensor drift) confounding the comparison.
  4. [Sec. III-C vs. Sec. III-D, Eq. (12)-(14)] Train/inference mismatch of the fusion pathway is only partially stress-tested. During teacher forcing, g_t uses ground-truth contact labels (Sec. III-C: 'subsequent gates also use ground-truth contact labels'), so the visual pathway is trained under clean gating; the predicted gate is exercised only during R=4-step rollout training and inference. The 10-step and 80-step evaluations (Table I, Fig. 5) do use the predicted gate, which mitigates this concern for prediction quality, but the CEM planning horizon H_p=6 with replanning every 2 steps means the gate under its own predictions is the operating regime that matters most — and it is exactly the regime covered only indirectly. Please clarify explicitly in Sec. III-C/III-D whether gradient flows through g_t when it is derived from the predicted ĉ_t during rollout training (the indicator is non-differentiable; presumably the gate is deta
minor comments (9)
  1. [Fig. 2] Fig. 2 contains leftover placeholder text '在此处键入公式' ('type formula here') inside the planning-cost box. This must be removed; it also suggests the figure should be proofread for other rendering artifacts.
  2. [Sec. III-E, Eq. (15)] h_goal in Eq. (15) is never defined operationally. Since tactile goals are 'physically unreachable before contact', please state how h_goal is obtained (a goal tactile observation from a reference successful trajectory? the tactile reading at the goal image's timestep?) and how D_τ behaves when the goal tactile latent was recorded on a different object instance.
  3. [Table III] Table III: the slip offset error is reported as '–' without explanation (contact offset is 0.69 frames). Either report it or state why it is undefined/not meaningful for slip.
  4. [Table III, Sec. IV-A.1] The onset/offset errors are in frames at 6 fps (0.44 frames ≈ 73 ms). Since contact onset is sub-frame at the 30 Hz recording rate, please clarify how ground-truth onset times were localized (sensor API timestamps at 30 Hz mapped to 6 fps frames?) and whether the 0.44-frame figure is limited by label quantization rather than model precision.
  5. [Table I] Table I: the 'FeelWorld (w/o hierarchical)' row shows Chamfer 0.072 but '–' for both F1 columns. Please state precisely which heads/losses this ablation removes (contact and slip heads only, keeping L_tis?), since 'hierarchical' is used loosely in the text.
  6. [Fig. 5] Fig. 5 caption should state explicitly that rollouts are open-loop along the executed actions of test trajectories; readers may otherwise assume counterfactual rollouts. The same clarification applies to Table I.
  7. [Sec. IV-A.1] Reproducibility: the paper would be strengthened by a statement on code/data release. The dataset (200+40 trajectories × 3 tasks on a specific Imeta-Y1/DM-Tac W2 platform) is not publicly benchmarked, so release matters more than usual for follow-up comparison.
  8. [Eq. (10), Eq. (13), Sec. IV-A.2] Loss weights λ_c = λ_s = 0.03 vs λ_τ = 0.3 differ by an order of magnitude; a one-line justification (gradient magnitudes? a small sweep?) would help. Similarly σ = 0.007 for context noise injection (Eq. 13) is asserted to 'approximate the prediction error distribution' — a brief empirical check would support this.
  9. [Sec. IV-D] 'Zero-shot planning' (Sec. IV-D) is used in the DINO-WM sense (no task-specific policy training); a short clarifying phrase would prevent misreading, since the world model itself is trained per-task on task data.

Circularity Check

0 steps flagged

No circularity: empirical world-model paper with external encoder targets, sensor labels, and real-robot planning metrics.

full rationale

FeelWorld is a standard supervised dynamics + CEM planning systems paper. Visual and tactile latents are trained against stop-gradient frozen encoder targets (V-JEPA 2, FG-CLTP); contact and slip heads are supervised by sensor-API labels with BCE/focal loss; LPIPS/PSNR/SSIM/FVD and zero-shot planning success are measured on held-out trajectories and 40 real-robot trials per task. None of these quantities is equal by construction to a fitted input. Use of the authors' prior FG-CLTP encoder is a frozen feature dependency, not a tautology: planning success and long-horizon LPIPS remain external measurements. The contact gate g_t = I[ĉ ≥ 0.5] is a design choice evaluated by ablation and robot success, not a self-definitional identity. No uniqueness theorem, ansatz smuggled via self-citation, or renamed known result carries the central claim. Evaluation gaps (e.g., gate accuracy under counterfactual CEM actions) are correctness/risk issues, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 3 invented entities

Load-bearing content is mostly engineering assumptions and fitted training/planning knobs on top of frozen pretrained encoders and sensor-provided labels. The scientific bet is that hierarchical tactile factorization plus hard contact gating is the right inductive bias; that bias is not derived from physics first principles but postulated and tested empirically.

free parameters (6)
  • λ_τ, λ_c, λ_s, λ_roll = 0.3, 0.03, 0.03, 0.5
    Loss mixture weights balancing visual L1, tactile latent L1, contact BCE, slip focal, and rollout terms; chosen for training stability and reported as fixed values.
  • rollout horizon R and context noise σ = R=4, σ=0.007
    Controls multi-step AR training and injected Gaussian context noise meant to match inference error; hand-set robustness knobs.
  • slip focal-loss α, γ = α=0.95 (positives), γ=2.0
    Class imbalance handling for rare slip (0.93% positives); directly affects slip F1.
  • contact gate threshold 0.5 = 0.5
    Hard threshold converting predicted contact probability into fusion/planning gate; discrete design choice.
  • CEM planner hyperparameters and cost weights w_τ, w_s = N=400, K=8, I=8, H_p=6; w_τ,w_s task-set (numeric values not fully tabulated)
    Candidate count, elites, iterations, horizon, execute-2 receding horizon, and tactile/slip cost weights shape reported success rates.
  • data rate and context window = 6 fps, 9 frames
    30 Hz logs downsampled to 6 fps; 9-frame context (1.5 s) fixes temporal scale of dynamics learning.
axioms (6)
  • domain assumption Frozen V-JEPA 2 visual latents and frozen FG-CLTP tactile latents are adequate state representations for dynamics and planning without fine-tuning the encoders.
    Sec. III-A/B and IV-A: only dynamics predictor and heads are trained; all prediction losses use stop-gradient encoder targets.
  • domain assumption Tactile signals outside contact are irrelevant sensor noise that should not modulate visual tokens or planning cost.
    Stated motivation for contact-gated asymmetric attention (Intro, Sec. III-C) and contact-aware CEM (Sec. III-E).
  • ad hoc to paper Contact, 3D force-related tactile latent, and slip form a sufficient hierarchical decomposition of touch for dynamics and cost design.
    Central modeling choice in Sec. III-B; order contact→latent→slip is imposed rather than derived.
  • domain assumption Official sensor API contact/slip labels are reliable ground truth for supervision and metrics.
    Sec. IV-A.1: labels from sensor API; slip rarity and F1 depend on this labeling process.
  • domain assumption L1 distances in visual/tactile latent spaces plus predicted slip probability are valid CEM task costs toward demonstrated goals.
    Eq. (15) and Algorithm 1 define planning success optimization without a separately validated value function.
  • standard math Standard transformer attention, BCE/focal classification, and CEM sampling behave as usual optimization tools.
    Background ML machinery used throughout Sec. III without novel theory claims.
invented entities (3)
  • FeelWorld hierarchical tactile state tuple τ_t=(c_t,h_t,s_t) with shared latent dynamics readout no independent evidence
    purpose: Structure touch prediction and supervision into contact, force geometry latent, and slip for fusion and planning.
    Introduced as the paper’s central representation; evaluated via F1/Chamfer/planning but not independently standardized outside this work.
  • Contact-gated asymmetric attention (tactile→vision always; vision→tactile only if gate open) independent evidence
    purpose: Prevent free-space tactile noise from degrading visual dynamics while allowing joint prediction in contact.
    Eqs. (11)–(12); ablations in Table I support utility but the mechanism is a paper-specific architectural postulate.
  • Contact-aware CEM objective with hard future contact gate on tactile/slip costs independent evidence
    purpose: Avoid optimizing unreachable tactile goals before contact while retaining visual progress.
    Eq. (15), Algorithm 1; success-rate gains are the external handle.

pith-pipeline@v1.2.0-grok45-kimik3 · 19352 in / 4320 out tokens · 90703 ms · 2026-07-31T19:37:08.169367+00:00 · methodology

0 comments
read the original abstract

Humans plan physical interactions by imagining the possible outcomes of candidate actions. However, existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing imagined futures that appear visually plausible but violate physical dynamics. We introduce FeelWorld, a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. FeelWorld organizes these states hierarchically as contact state, a 3D tactile latent that encodes force-related information, and slip state. These states are jointly predicted by a shared latent dynamics model with explicit supervision. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, we introduce a contact-gated asymmetric attention mechanism that maintains a visual-only prediction pathway before contact and enables joint visuo-tactile dynamics prediction during contact. The model is further trained with autoregressive rollouts and context noise injection to improve robustness to compounding errors. The predicted contact and slip states also support contact-aware CEM planning. Experiments on chip grasping, fruit grasping, and USB insertion show that FeelWorld reduces 10-step LPIPS from 0.084 to 0.058 and maintains an LPIPS that is 61% lower than that of the visual baseline after an 80-step autoregressive rollout. FeelWorld also achieves an average zero-shot planning success rate of 81.7%, providing an effective approach for incorporating tactile sensing into world models.

Figures

Figures reproduced from arXiv: 2607.24267 by Chaofan Zhang, Chao Xue, Guocai Yao, Shaowei Cui, Shuo Wang, Wenxuan Ma, Yinghao Cai.

Figure 1
Figure 1. Figure 1: Overview of FeelWorld. Left: multi-view visual observations, 3D tactile point clouds, and robot actions are encoded and fused to jointly predict future visual observations and hierarchical tactile states, including contact, 3D tactile latents, and slip. Right: contact-aware CEM planning consistently outperforms vision-only and naive visuo-tactile baselines across all tasks. tactile signals as a monolithic … view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of FeelWorld. Visual observations, tactile observations, and actions are encoded into latent tokens. Hierarchical tactile modeling decomposes touch into explicitly supervised contact states, 3D tactile latents, and slip states. Contact-gated asymmetric attention preserves visual prediction before contact and enables joint visuo-tactile prediction during contact. Autoregressive rollout training… view at source ↗
Figure 3
Figure 3. Figure 3: Robot platform. The Imeta-Y1 robot is equipped with three RGB cameras and DM tactile sensors. FeelWorld uses the left external camera to observe the global state and the wrist-mounted camera to provide local observations. The DM tactile sensor captures 3D tactile point clouds and contact states. and compliance; and USB insertion, which requires precise contact reasoning under visual occlusion. For each tas… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative visualizations of FeelWorld. The model predicts future visual observations together with three tactile states: contact state, 3D tactile latent, and slip state. These tactile states change according to the interaction context and robot action. evaluated over 40 real-robot trials for each task. B. Visual Prediction Evaluation We evaluate the visual prediction quality of FeelWorld against several… view at source ↗
Figure 5
Figure 5. Figure 5: Long-horizon autoregressive visual prediction. At step 80, FeelWorld improves PSNR and SSIM over Vision Only by 6.70 dB and 0.115, while reducing LPIPS and MAE by 61% and 62%. The shaded region marks the long-horizon regime. 2 visual backbone and train only a lightweight contact￾prediction head. The resulting contact-onset error is 1.17 frames, compared with only 0.44 frames when tactile features from Feel… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of contact-gated prediction during a USB insertion episode. (a) Multi-stage visual rollout. (b) Ground-truth and predicted contact states. (c) Ground-truth and predicted slip states. (d) Gated visuo￾tactile cross-attention for the external and wrist camera views. (e) Hard contact-gate activation over time. that the fused tactile latent zˆ τ effectively captures contact state, force-related in… view at source ↗
Figure 7
Figure 7. Figure 7: CEM planning curves over a 50-step execution. The plots show total loss (left) and tactile loss (right). At each replanning cycle, CEM optimizes a 6-step imagined horizon and executes the first two actions. The dotted line and gray band mark the median and interquartile range of contact-gate activation. TABLE IV ZERO-SHOT CEM PLANNING SUCCESS RATES. Planner Chip Fruit USB Visual-only CEM (w/o Tactile) 40.0… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 5 linked inside Pith

  1. [1]

    Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,

    M. A. Lee, Y . Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,”IEEE Transactions on Robotics, vol. 36, no. 3, pp. 582–596, 2020

    2020

  2. [2]

    Tactile robotics: An outlook,

    S. Luo, N. F. Lepora, W. Yuan, K. Althoefer, G. Cheng, and R. Dahiya, “Tactile robotics: An outlook,”IEEE Transactions on Robotics, vol. 41, pp. 5564–5583, 2025

    2025

  3. [3]

    Dream to control: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” inInternational Conference on Learning Representations (ICLR), 2020

    2020

  4. [4]

    Mastering diverse control tasks through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse control tasks through world models,”Nature, vol. 640, pp. 647–653, 2025

    2025

  5. [5]

    Self-supervised learning from images with a joint-embedding predictive architecture,

    M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y . LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15 619–15 629

    2023

  6. [6]

    V-JEPA 2: Self-supervised video models enable understanding, pre- diction and planning,

    M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V . Khali- dov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y . Li, X. Ma, S. Chandar, F. Meier, Y . LeCun, M. Rabbat, and N. Ballas, “V-JEPA 2: Self-supervis...

    Pith/arXiv arXiv 2025

  7. [7]

    Dream-Tac: A unified tactile world action model for contact-rich robot manipulation,

    Y . Lou, Y . Ye, Y . Fu, J. Cen, X. Chi, Y . Lyu, P. Jia, S. Han, Z. Lu, and S. Zhang, “Dream-Tac: A unified tactile world action model for contact-rich robot manipulation,”arXiv preprint arXiv:2606.08737, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    VT-W AM: Visual-tactile world action model for contact-rich manipulation,

    S. Tian, Y . Zheng, Y . Zheng, S. Gu, Y . Zang, Y . Qin, W. Li, H. Li, W. Ding, and D. Zhao, “VT-W AM: Visual-tactile world action model for contact-rich manipulation,”arXiv preprint arXiv:2607.02503, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Tactile-W AM: Touch-aware world action model with tactile asymmetric attention,

    S. Wu, L. You, J. Zhu, Y . Liu, C. Zhang, J. Liu, W. Wang, Q. Li, J. Li, and H. Zhao, “Tactile-W AM: Touch-aware world action model with tactile asymmetric attention,”arXiv preprint arXiv:2606.26663, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Learning to feel the future: DreamTacVLA for contact-rich manipulation,

    G. Ye, Z. Zhang, X. Zhao, S. Wu, H. Lu, S. Lu, and H. Liu, “Learning to feel the future: DreamTacVLA for contact-rich manipulation,”arXiv preprint arXiv:2512.23864, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Ctrl-World: A controllable generative world model for robot manipulation,

    Y . Guo, L. X. Shi, J. Chen, and C. Finn, “Ctrl-World: A controllable generative world model for robot manipulation,” inThe F ourteenth International Conference on Learning Representations, 2026

    2026

  12. [12]

    Interactive world simulator for robot policy training and evaluation,

    Y . Wang, R. Syed, F. Wu, M. Zhang, A. Onol, J. Barreiros, H. Nayyeri, T. Dear, H. Zhang, and Y . Li, “Interactive world simulator for robot policy training and evaluation,” inProceedings of Robotics: Science and Systems, 2026

    2026

  13. [13]

    Learning interactive real-world simulators,

    S. Yang, Y . Du, S. Ghasemipour, J. Tompson, L. Kaelbling, D. Schuur- mans, and P. Abbeel, “Learning interactive real-world simulators,” in International Conference on Learning Representations (ICLR), 2024

    2024

  14. [14]

    Robo- Dreamer: Learning compositional world models for robot imagina- tion,

    S. Zhou, Y . Du, J. Chen, Y . Li, D.-Y . Yeung, and C. Gan, “Robo- Dreamer: Learning compositional world models for robot imagina- tion,” inProceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 235. PMLR, 2024, pp. 61 885–61 896

    2024

  15. [15]

    Visuo-tactile world models,

    C. Higuera, S. Arnaud, B. Boots, M. Mukadam, F. R. Hogan, and F. Meier, “Visuo-tactile world models,”arXiv preprint arXiv:2602.06001, 2026

    arXiv 2026

  16. [16]

    ContactWorld: What matters in vision- tactile world models for contact-rich manipulation,

    Z. Zhang, P. Zhou, K. Zhang, A. Desai, T. Amosa, D. Soleymanzadeh, J. Lei, M. Zheng, and Y . She, “ContactWorld: What matters in vision- tactile world models for contact-rich manipulation,”arXiv preprint arXiv:2606.13877, 2026

    arXiv 2026

  17. [17]

    GelSight: High-resolution robot tactile sensors for estimating geometry and force,

    W. Yuan, S. Dong, and E. H. Adelson, “GelSight: High-resolution robot tactile sensors for estimating geometry and force,”Sensors, vol. 17, no. 12, p. 2762, 2017

    2017

  18. [18]

    DIGIT: A novel design for a low-cost compact high- resolution tactile sensor with application to in-hand manipulation,

    M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V . R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, D. Jayaraman, and R. Calandra, “DIGIT: A novel design for a low-cost compact high- resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 3838–3845, 2020

    2020

  19. [19]

    Sparsh: Self-supervised touch representations for vision-based tactile sensing,

    C. Higuera, A. Sharma, C. K. Bodduluri, T. Fan, P. Lancaster, M. Kalakrishnan, M. Kaess, B. Boots, M. Lambeta, T. Wu, and M. Mukadam, “Sparsh: Self-supervised touch representations for vision-based tactile sensing,” inProceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 270. PMLR, 2025, pp. 885–915

    2025

  20. [20]

    Self-supervised visuo-tactile pretraining to locate and follow garment features,

    J. Kerr, H. Huang, A. Wilcox, R. I. Hoque, J. Ichnowski, R. Calandra, and K. Goldberg, “Self-supervised visuo-tactile pretraining to locate and follow garment features,” inProceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023

    2023

  21. [21]

    A touch, vision, and language dataset for multimodal alignment,

    L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, J. Ortiz, M. Mukadam, M. Lambeta, R. Calandra, and K. Goldberg, “A touch, vision, and language dataset for multimodal alignment,” inProceed- ings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 235. PMLR, 2024, pp. 14 080–14 101

    2024

  22. [22]

    Binding touch to everything: Learning unified multimodal tactile representations,

    F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y . Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, and A. Wong, “Binding touch to everything: Learning unified multimodal tactile representations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 26 340–26 353

    2024

  23. [23]

    Touch100k: A large-scale touch- language-vision dataset for touch-centric multimodal representation,

    N. Cheng, J. Xu, C. Guan, J. Gao, W. Wang, Y . Li, F. Meng, J. Zhou, B. Fang, and W. Han, “Touch100k: A large-scale touch- language-vision dataset for touch-centric multimodal representation,” Information Fusion, vol. 124, p. 103305, 2025

    2025

  24. [24]

    AnyTouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors,

    R. Feng, J. Hu, W. Xia, T. Gao, A. Shen, Y . Sun, B. Fang, and D. Hu, “AnyTouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  25. [25]

    CLTP: Contrastive language-tactile pre-training for 3D contact geometry understanding,

    W. Ma, X. Cao, Y . Zhang, C. Zhang, S. Yang, P. Hao, B. Fang, Y . Cai, S. Cui, and S. Wang, “CLTP: Contrastive language-tactile pre-training for 3D contact geometry understanding,”Biomimetic Intelligence and Robotics, p. 100324, 2026

    2026

  26. [26]

    FG- CLTP: Fine-grained contrastive language tactile pretraining for robotic manipulation,

    W. Ma, C. Zhang, Y . Cai, G. Yao, S. Cui, and S. Wang, “FG- CLTP: Fine-grained contrastive language tactile pretraining for robotic manipulation,”IEEE Robotics and Automation Letters, 2026

    2026

  27. [27]

    3D-ViTac: Learning fine-grained manipulation with visuo-tactile sensing,

    B. Huang, Y . Wang, X. Yang, Y . Luo, and Y . Li, “3D-ViTac: Learning fine-grained manipulation with visuo-tactile sensing,” inProceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 270. PMLR, 2025, pp. 2557–2578

    2025

  28. [28]

    Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation,

    H. Xue, J. Ren, W. Chen, G. Zhang, Y . Fang, G. Gu, H. Xu, and C. Lu, “Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation,” inProceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025

    2025

  29. [29]

    TaF-VLA: Tactile-force alignment in vision-language-action mod- els for force-aware manipulation,

    Y . Huang, P. Lin, W. Li, D. Li, J. Li, J. Jiang, C. Xiao, and Z. Jiao, “TaF-VLA: Tactile-force alignment in vision-language-action mod- els for force-aware manipulation,”arXiv preprint arXiv:2601.20321, 2026

    arXiv 2026

  30. [30]

    VTLA: Vision-tactile-language-action model with preference learning for insertion manipulation,

    C. Zhang, P. Hao, X. Cao, X. Hao, S. Cui, and S. Wang, “VTLA: Vision-tactile-language-action model with preference learning for insertion manipulation,”Biomimetic Intelligence and Robotics, p. 100333, 2026

    2026

  31. [31]

    OmniVTLA: Vision-tactile-language-action models with semantic-aligned tactile sensing,

    Z. Cheng, Y . Zhang, A. Tang, K. Wang, W. Zhang, H. Li, H. Zhang, and L. Song, “OmniVTLA: Vision-tactile-language-action models with semantic-aligned tactile sensing,”IEEE Robotics and Automation Letters, 2026

    2026

  32. [32]

    OmniVTA: Visuo-tactile world modeling for contact-rich robotic manipulation,

    Y . Zheng, S. Gu, W. Li, Y . Zheng, Y . Zang, S. Tian, X. Li, R. Wu, C. Hao, C. Gao, S. Liu, H. Li, Y . Chen, S. Yan, and W. Ding, “OmniVTA: Visuo-tactile world modeling for contact-rich robotic manipulation,”arXiv preprint arXiv:2603.19201, 2026

    arXiv 2026

  33. [33]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988

    2017

  34. [34]

    DINO-WM: World models on pre-trained visual features enable zero-shot planning,

    G. Zhou, H. Pan, Y . Lecun, and L. Pinto, “DINO-WM: World models on pre-trained visual features enable zero-shot planning,” inProceed- ings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 79 115–79 135

    2025