Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Surformer v2 claims that classifying vision and touch independently and merging only the final decisions reaches 97.4% accuracy on Touch and Go at 0.0239 ms per inference — a ~30x speed-up over Surformer v1.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Surformer v2 combines an ImageNet-pretrained CNN for vision with a handcrafted-feature transformer for touch, fuses their outputs via learned weights, and reports 97.4% accuracy with 0.0239 ms inference on Touch and Go.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Inference-time claim is implausible and the split likely leaks, so the efficiency and accuracy results don't hold up. the 4 major comments →

arxiv 2509.04658 v1 pith:PCU4P7YY submitted 2025-09-04 cs.RO

Surformer v2: A Multimodal Classifier for Surface Understanding from Touch and Vision

Manish Kansana , Sindhuja Penchala , Shahram Rahimi , Noorbakhsh Amiri Golilarz This is my paper
classification cs.RO
keywords surface material classificationmultimodal tactile perceptionlate fusiondecision-level fusionvision and touchtransformer encoderrobotic manipulationreal-time inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Surformer v2 sets out to make surface-material classification from vision and touch fast enough for real-time robotics while keeping accuracy competitive. Its design principle is late fusion: a CNN (EfficientNetV2-S) classifies the wide-angle camera view on its own, a small encoder-only transformer classifies handcrafted features from GelSight tactile images on its own, and the two final class-logit vectors are combined by a learnable weighted sum. On the Touch and Go benchmark the model reports 97.4% accuracy at 0.0239 ms per inference, roughly a 30-fold speed-up over the authors' earlier Surformer v1 in exchange for about two points of accuracy. The authors argue that because the two input types differ so much in viewpoint and scale, classifying them independently and merging only the decisions is simpler, faster, and more reliable than trying to fuse their features.

Core claim

The central claim is that decision-level fusion beats feature-level fusion for the Touch and Go benchmark once end-to-end compute is counted. The vision branch runs ImageNet-pretrained EfficientNetV2-S with early layers frozen, producing vision logits; the tactile branch converts grayscale GelSight images into a normalized 7-dimensional handcrafted feature vector, projects it to 64 dimensions, and runs it through a single-layer transformer encoder with four attention heads, producing tactile logits. A softmax-normalized learnable weighted sum merges the two logits, and training minimizes a composite loss — the fused cross-entropy plus each branch's cross-entropy weighted by 0.3 — so both mod

What carries the argument

The load-bearing object is the late-fusion weighted logit layer flanked by two independent branch classifiers and a multi-objective loss. The fusion layer holds two softmax-normalized scalar weights, learned by backpropagation, that linearly combine the vision and tactile logits before a final softmax; because each branch produces its own logits, the model never mixes feature maps. The tactile branch's handcrafted-feature pipeline — grayscale conversion, 7-dim feature extraction with dataset-level normalization, 64-dim projection, single-layer four-head transformer encoder — is the mechanism that carries the touch stream, and the composite loss L_total = L_main + 0.3·L_vision + 0.3·L_tactile

Load-bearing premise

The 97.4% accuracy rests on an 80-20 stratified split of individual images; the paper does not state that entire physical surfaces were kept in one split, so images of the same concrete, wood, or grass surface may appear in both training and testing, which would inflate the number.

What would settle it

Re-run the same training with the Touch and Go samples split by physical surface rather than by image — hold out whole surfaces (e.g., all photos and touches of one brick wall) and measure accuracy on them. If accuracy stays near 97.4%, the result generalizes across surfaces; if it drops materially, the published number reflects image-level familiarity with the same surfaces rather than learned material categories.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At 0.0239 ms per sample, including handcrafted feature extraction, the architecture is fast enough to run inside a manipulator control loop, which is the paper's stated real-time target.
  • If one sensor fails or is occluded, the system still produces an answer because each branch already emits class logits on its own — the robustness the authors claim for late fusion.
  • The 30x speed-up over Surformer v1 with only a ~2-point accuracy drop frames the design as a latency-aware trade, not a pure accuracy win — something the authors explicitly argue for.
  • Because timing is measured end-to-end with feature extraction inside the model, the efficiency numbers are meant to transfer to deployment conditions better than v1's, which excluded extraction time.
  • The learnable fusion weights can in principle adapt per data context and surface class, letting the model lean on vision for some materials and touch for others.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Not verified in the paper: the 80-20 stratified split is described at the image level, not the surface level. If images of the same physical patch of concrete or wood appear in both train and test, the reported 97.4% would overstate generalization to new surfaces; an object-level split would test this directly.
  • The tactile stream still leans on a handcrafted 7-dim feature vector, so 'feature extraction within the model' is only fully true for the vision branch; swapping those handcrafted features for learned ones would reveal how much the transformer itself contributes.
  • Since both CNN baselines already hit 100% on Touch and Go, the benchmark looks close to saturated; the late-fusion advantage is more likely to show up under occlusion, noise, or sensor dropout than on this clean benchmark.
  • The learned fusion weights are reported but never analyzed; inspecting which weight each surface class drives would turn the interpretability claim into a concrete, testable statement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Surformer v2, a late-fusion multimodal classifier for surface material classification from vision and tactile inputs. The vision branch is an EfficientNetV2-S CNN and the tactile branch is a transformer encoder fed by seven handcrafted grayscale-image features; the two branches' logits are combined via a learnable weighted sum with auxiliary per-modality losses. On the Touch and Go dataset, the authors report 97.4% accuracy and an inference time of 0.0239 ms, claiming a ~30x speed-up over Surformer v1 while retaining real-time suitability for robotics.

Significance. If the reported efficiency were credible, a late-fusion design that keeps the two heterogeneous modalities independent would be a useful contribution to visuo-tactile surface classification. The paper also honestly reports a small accuracy drop relative to its baselines, and the architecture is described clearly enough to reproduce. However, the central real-time claim rests entirely on an inference-time figure that lacks any measurement protocol and is physically implausible for the stated model. The accuracy comparison is also presented without variance or significance testing, and the data-split description does not rule out surface-level leakage. As it stands, the paper's headline claims are not supported.

major comments (4)
  1. [Table II / Section III-B] The inference time of 0.0239 ms for Surformer v2 is load-bearing for the claimed ~30x speed-up and real-time suitability, but no hardware, batch size, precision, warm-up, repetition count, or measurement procedure is reported. A 20.7M-parameter model containing EfficientNetV2-S on 224×224 inputs cannot plausibly run in ~24 µs on any standard CPU or GPU at batch 1; this suggests a unit error, an unfair comparison, or a measurement artifact. The paper also does not explain how a model with 30x more parameters than Surformer v1 can be 30x faster. This figure must be re-measured and documented, or the central efficiency claim must be withdrawn.
  2. [Tables I and II / Section III-B] All accuracy, precision, recall, and F1 results are reported as single numbers with no error bars, seeds, or significance tests. Differences such as 97.4% vs. 99.4% or 100% may be within training noise, especially given the small per-class sample counts implied by the confusion matrix description (200 true samples per class). The paper should report statistics over multiple runs and, if possible, confidence intervals or a significance test.
  3. [Section III-A] The 80-20 stratified train/test split is not defined at the surface-instance or object level. The Touch and Go dataset contains multiple images and tactile readings of the same physical surface; if images from the same surface appear in both training and test splits, the reported accuracy is inflated and the comparison to baselines is invalid. The authors must clarify whether the split was performed at the image level or grouped by physical surface/object, and justify the choice.
  4. [Section II-B and Abstract/Introduction] The paper claims that Surformer v2 'performs feature extraction within the model itself' and contrasts this with Surformer v1's handcrafted features. However, the tactile branch still begins with a fixed, handcrafted 7-dimensional feature vector (grayscale conversion plus handcrafted features). Only the vision branch learns features end-to-end. This contradiction affects contribution 3 and should be corrected or the tactile feature extraction should be made learnable.
minor comments (5)
  1. [Throughout] There are typographical inconsistencies: 'Efficient V-Net' vs. 'EfficientNetV2-S', 'Weighed Fusion Layer' vs. 'Weighted Fusion Layer', 'Multi-model CNN' vs. 'Multimodal CNN'. Please standardize terminology.
  2. [References] Reference [12] (ViTacFormer) has the wrong arXiv ID (it points to 2210.00121, which is reference [11]). Reference [4] (DenseNet) links to an unrelated 2020 arXiv abstract. These need correction.
  3. [Section II-A] The phrase 'Only the final 20 parameters are unfrozen' is likely a typo; presumably the final layers are unfrozen. Please clarify the exact fine-tuning scheme.
  4. [Section IV / Contributions] The claim that Surformer v2 'accounts for the complete inference pipeline, including feature extraction time' is not backed by any description of how end-to-end latency was measured (e.g., data loading, preprocessing, resizing). Specify the protocol or soften the claim.
  5. [General] No code or trained models are released, and no training details such as number of epochs, batch size, optimizer settings, or data augmentation are given. Providing these would significantly improve reproducibility.

Circularity Check

1 steps flagged

No derivation-level circularity in the accuracy result; the ~30x speed-up claim over v1 is load-bearing on a self-citation.

specific steps
  1. self citation load bearing [Section III-B (Table II) and Section IV; setup inherited in Section III-A]
    "Compared to Surformer v1, Surformer v2 offers a ∼ 30x speed-up in inference with only a marginal drop in accuracy (from 99.4% to 97.4%)."

    The 30x speed-up is the ratio of the authors' new v2 timing (0.0239 ms) to the v1 timing (0.7271 ms) taken from the authors' own prior paper [1], with v1 accuracy 99.4% also from that self-citation. Section III-A states the preprocessing and v1/Multimodal-CNN training setups 'follow those established in our prior work,' so the comparison against v1 is entirely inherited from the same group's unverified measurements. The headline efficiency trade-off is therefore a self-referential ratio rather than an independently reproduced result. The Touch-and-Go accuracy itself is measured externally and is not circular.

full rationale

The paper's supervised classification accuracy (97.4%, AUC, confusion matrix) is an empirical result measured on the external Touch and Go dataset; no equation-level derivation reduces a prediction to a fitted parameter. The main circularity-adjacent step is the ~30x speed-up claim, which compares a self-reported v2 inference time (0.0239 ms) with a v1 inference time imported from the authors' own prior paper [1]. Because the v1 baseline, preprocessing, and hyperparameters are inherited from [1], the efficiency comparison is load-bearing on a self-citation and lacks independent timing methodology. This lowers confidence in the speed-up claim but does not make the central accuracy result circular. The absence of an object-level split is a data-validity concern, not a circularity, and the implausible 0.0239 ms figure is a measurement/correctness risk that the circularity score does not penalize further.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests mainly on dataset assumptions (leakage-free split, label quality), the sufficiency of the seven handcrafted tactile features, and the validity of the self-reported timing measurement. All are unverified in the paper.

free parameters (3)
  • Fusion weights (vision, tactile) = softmax-normalized learned scalars (not reported)
    Learned during training to weight vision and tactile logits in the weighted fusion layer (Section II-C).
  • Feature normalization mean and std = computed from up to 1000 training examples (not reported)
    Dataset-specific statistics used to normalize the 7 handcrafted tactile features (Section II-B.1).
  • Auxiliary loss weights = 0.3 for both vision and tactile auxiliary losses
    Hand-chosen in composite loss L_total = L_main + 0.3 L_vision + 0.3 L_tactile (Section III-A).
axioms (4)
  • domain assumption Touch and Go labels are correct and representative of surface materials.
    The evaluation treats benchmark labels as ground truth (Section III).
  • domain assumption Images from the same physical surface do not appear in both train and test despite image-level stratified split.
    Section III-A states an 80-20 stratified split but never describes object-level partitioning.
  • domain assumption The 7 handcrafted tactile features (never enumerated) are sufficient to represent tactile texture for classification.
    Section II-B relies on a normalized 7-dimensional handcrafted feature vector as the sole tactile representation.
  • domain assumption ImageNet pretrained features transfer to surface classification.
    Vision branch uses ImageNet-initialized EfficientNetV2-S with selective fine-tuning (Section II-A).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Surformer v2: A Multimodal Classifier for Surface Understanding from Touch and Vision." pith.science (2026). https://pith.science/paper/PCU4P7YY

@misc{pith2026250904658,
  author       = {Pith},
  title        = {Pith review of: Surformer v2: A Multimodal Classifier for Surface Understanding from Touch and Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCU4P7YY}},
  note         = {Machine review of arXiv:2509.04658}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal surface material classification plays a critical role in advancing tactile perception for robotic manipulation and interaction. In this paper, we present Surformer v2, an enhanced multi-modal classification architecture designed to integrate visual and tactile sensory streams through a late(decision level) fusion mechanism. Building on our earlier Surformer v1 framework [1], which employed handcrafted feature extraction followed by mid-level fusion architecture with multi-head cross-attention layers, Surformer v2 integrates the feature extraction process within the model itself and shifts to late fusion. The vision branch leverages a CNN-based classifier(Efficient V-Net), while the tactile branch employs an encoder-only transformer model, allowing each modality to extract modality-specific features optimized for classification. Rather than merging feature maps, the model performs decision-level fusion by combining the output logits using a learnable weighted sum, enabling adaptive emphasis on each modality depending on data context and training dynamics. We evaluate Surformer v2 on the Touch and Go dataset [2], a multi-modal benchmark comprising surface images and corresponding tactile sensor readings. Our results demonstrate that Surformer v2 performs well, maintaining competitive inference speed, suitable for real-time robotic applications. These findings underscore the effectiveness of decision-level fusion and transformer-based tactile modeling for enhancing surface understanding in multi-modal robotic perception.

Figures

Figures reproduced from arXiv: 2509.04658 by Manish Kansana, Noorbakhsh Amiri Golilarz, Shahram Rahimi, Sindhuja Penchala.

Figure 1
Figure 1. Figure 1: Architecture of Surformer v2 for multi-modal surface classification from tactile and visual data. The model accepts two input modalities: (1) vision [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance evaluation of Surformer v2 on the Touch and Go dataset. The confusion matrix (left) illustrates the model’s classification accuracy across [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms

    cs.RO 2026-05 unverdicted novelty 4.0

    A survey proposing a hierarchical taxonomy for multimodal tactile fusion datasets and methods across perception, generation, and interaction in embodied intelligence.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features

    M. Kansana, E. Hossain, S. Rahimi, and N. A. Golilarz, “Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features,” arXiv preprint arXiv:2508.06566v1, Aug. 2025. Available: https://doi.org/10.48550/arXiv.2508.06566 1, 4

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.06566 2025

  2. [2]

    Touch and go: Learning from human-collected vision and touch,

    F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. Owens, “Touch and go: Learning from human-collected vision and touch,” arXiv preprint arXiv:2211.12498, 2022. https://arxiv.org/abs/2211.12498v1 1, 2, 4

    Pith/arXiv arXiv 2022

  3. [3]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 770–778, 2016. https://ieeexplore.ieee.org/document/7780459 1

    arXiv 2016

  4. [4]

    Convolutional Networks with Dense Connectivity

    G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , pp. 4700–4708, 2017. https://arxiv.org/abs/2001.02394 1

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  5. [5]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE , vol. 86, no. 11, pp. 2278–2324, 2002. https://ieeexplore.ieee.org/document/726791 1

    work page 2002

  6. [6]

    EfficientNet: Rethinking model scaling for convo- lutional neural networks,

    M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convo- lutional neural networks,” in Proc. Int. Conf. Mach. Learn. (ICML) , pp. 6105–6114, 2019. https://arxiv.org/abs/1905.11946v5 1

    Pith/arXiv arXiv 2019

  7. [7]

    Majority voting: Material classification by tactile sensing using surface textures,

    N. Jamali and C. Sammut, “Majority voting: Material classification by tactile sensing using surface textures,” IEEE Trans. Robot. , vol. 27, no. 3, pp. 508–521, 2010. 1

    work page 2010

  8. [8]

    Tactile-data classification of contact materials using computational intelligence,

    S. Decherchi, P. Gastaldo, R. S. Dahiya, M. Valle, and R. Zunino, “Tactile-data classification of contact materials using computational intelligence,” IEEE Trans. Robot. , vol. 27, pp. 635–639, 2011. https://ieeexplore.ieee.org/document/5756488 1

    work page arXiv 2011

  9. [9]

    Tactile-data classification of contact materials using principal component analysis and self-organizing maps,

    S. Decherchi, P. Gastaldo, R. S. Dahiya, and M. Valle, “Tactile-data classification of contact materials using principal component analysis and self-organizing maps,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), pp. 1712–1717, 2005. 1

    work page 2005

  10. [10]

    Estimating perceptual attributes of haptic textures using visuo-tactile data,

    M. I. Awan and S. Jeon, “Estimating perceptual attributes of haptic textures using visuo-tactile data,” IEEE Access , vol. 13, pp. 109931– 109945, 2025. https://ieeexplore.ieee.org/document/11045379 1

    work page arXiv 2025

  11. [11]

    Visuo-tactile trans- formers for manipulation,

    Y . Chen, A. Sipos, M. Merwe, and N. Fazeli, “Visuo-tactile trans- formers for manipulation,” arXiv preprint arXiv:2210.00121 , 2022. https://arxiv.org/abs/2210.00121 2

    Pith/arXiv arXiv 2022

  12. [12]

    Vi- TacFormer: Learning cross-modal representation for visuo-tactile dexterous manipulation,

    L. Heng, H. Geng, K. Zhang, P. Abbeel, and J. Malik, “Vi- TacFormer: Learning cross-modal representation for visuo-tactile dexterous manipulation,” arXiv preprint arXiv:2506.15953 , 2025. https://arxiv.org/abs/2210.00121 2

    Pith/arXiv arXiv 2025

  13. [13]

    GelFusion: Enhancing robotic manipulation under visual constraints via visuotactile fusion,

    S. Jiang, S. Zhao, Y . Fan, and P. Yin, “GelFusion: Enhancing robotic manipulation under visual constraints via visuotactile fusion,” arXiv preprint arXiv:2505.07455, 2025. https://arxiv.org/abs/2505.07455 2

    Pith/arXiv arXiv 2025

  14. [14]

    Efficient visual-tactile transformer with token reorganization for robotic slip detection,

    L. Chaowen, H. Wu, and J. Shen, “Efficient visual-tactile transformer with token reorganization for robotic slip detection,” J. Braz. Soc. Mech. Sci. Eng. , vol. 47, no. 6, 2025. https://link.springer.com/article/10.1007/s40430-025-05587-w 2

    work page doi:10.1007/s40430-025-05587-w 2025

  15. [15]

    EfficientNetV2: Smaller models and faster training,

    M. Tan and Q. V . Le, “EfficientNetV2: Smaller models and faster training,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2021. https://proceedings.mlr.press/v139/tan21a.html 2

    work page 2021

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.