Pith. sign in

REVIEW 4 major objections 5 minor 111 references

SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SimCMF shows that SAM's frozen RGB patch embedding, paired with a small cross-modal adapter, can be fine-tuned to segment images from thermal, depth, polarization, HHA, and near-infrared sensors, raising average mIoU from 22.15% to 53.88%.

desk verdict A useful empirical recipe for adapting SAM to non-RGB sensors, but the headline numbers rest on an unvalidated instance-level metric that needs scrutiny before the quantitative claims can be trusted. read the letter →

arxiv 2411.18669 v1 pith:UQ6UQK75 submitted 2024-11-27 cs.CV

Chenyang Lei , Liyi Chen , Jun Cen , Xiao Chen , Zhen Lei , Felix Heide , Qifeng Chen , Zhaoxiang Zhang This is my paper · ORCID
classification cs.CV
keywords cross-modalfine-tuningSegmentAnythingModelparameter-efficientmodalityalignmentimagesegmentationpolarizationimagingdepththermal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SimCMF asks whether vision foundation models trained on natural RGB images can be reused for other imaging sensors that lack large datasets. It answers yes: prepend a small cross-modal adapter to the frozen patch embedding of SAM, then fine-tune with parameter-efficient methods. Across five modalities (thermal, polarization, depth, HHA, NIR), segmentation mIoU rises from 22.15% to 53.88% on average versus training from scratch, and the method beats existing RGB-X fusion baselines without using any RGB channel-order prior. The paper also constructs a new benchmark, AIMS, so that cross-modal transfer can be evaluated consistently. If the claim holds, data-scarce sensor fields can inherit interactive segmentation ability from RGB foundation models at low compute.

What carries the argument

The load-bearing component is the cross-modal alignment adapter: a stack of two convolutional layers with 3×3 kernels, 64 channels, and ReLU nonlinearity (except the last layer) that projects an arbitrary C-channel sensor input into the dimensional space of SAM's patch embedding. The pretrained patch embedding is kept frozen, a choice the paper shows is crucial for training stability and final performance. The adapter resolves channel-dimension misalignment while letting the frozen RGB-trained features carry the semantic load, and the backbone is then tuned with LoRA or an MLP adapter.

What would settle it

Run the released SimCMF code on a sixth modality whose geometry and statistics are far from natural RGB—for example raw event-camera frames, ultrasound, or X-ray attenuation—using the same training protocol; if instance-level mIoU does not exceed a from-scratch ViT-B trained on that modality, the paper's central claim that frozen SAM patch embeddings transfer to any imaging modality would be falsified for that sensor.

Watch

Extended reading notes

Core claim

The paper demonstrates that a vision foundation model pretrained on natural RGB images can be adapted to arbitrary imaging modalities by inserting a lightweight cross-modal alignment module before the model's frozen patch embedding. With SAM as the backbone, this raises instance-level segmentation mIoU from 22.15% to 53.88% on average across thermal, polarization, depth, HHA, and near-infrared images, compared with training the same architecture from scratch. It further shows that parameter-efficient fine-tuning (LoRA and MLP adapters) matches full fine-tuning while updating only about 4% of the backbone parameters, and that on pseudo-new modalities (RGB-T, RGB-D, RGB-HHA, RGB-NIR) SimCMF outperforms dedicated fusion baselines such as ViPT and CMX without exploiting which channels come from RGB.

Load-bearing premise

The entire transfer rests on the empirical claim that SAM's frozen RGB patch embedding, after a small learned projection, remains informative for inputs from physically different sensors; the paper only tests five modalities, so a sensor whose local structure RGB features cannot represent would break the method.

Editorial extensions

If this is right

  • Any imaging modality with modest labeled data can obtain SAM-level interactive segmentation by adding a small adapter, without retraining the backbone.
  • Parameter-efficient fine-tuning suffices, cutting trainable parameters to roughly 4% of full fine-tuning while matching accuracy and lowering hardware requirements.
  • The design recipe (frozen pretrained patch embedding, 3×3 kernels, two layers, 64 dims) provides a plug-in rule for future cross-modal adapters.
  • The AIMS benchmark offers a shared evaluation protocol (instance-level mIoU extracted from semantic labels, center-click prompts) for comparing cross-modal transfer methods.
  • On pseudo-new modalities, SimCMF outperforms dedicated RGB-X fusion baselines without using the RGB channel-prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The frozen RGB patch embedding may be acting less as an RGB-specific encoder and more as a generic local-texture encoder, predicting that the adapter will also transfer to unseen modalities such as event cameras, ultrasound, or X-ray, provided those sensors share local-contrast structure with natural images.
  • The 3×3 kernel result suggests that a small spatial context is needed to merge multi-channel sensor signals, implying that pure per-pixel channel projection is insufficient and that early fusion should preserve local spatial correlation.
  • Because performance keeps rising with more fine-tuning data (Figure 5b), a testable extension is that SimCMF's edge over from-scratch training will shrink but persist as modality datasets scale.
  • The channel-shuffling experiment hints that the adapter learns to rediscover RGB-like features from arbitrary channel order, so it is worth testing whether the same adapter works on spectrograms or range-Doppler images, which are imaging-like but not natural images.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SimCMF, a cross-modal fine-tuning strategy that prepends a small convolutional adapter to a frozen pretrained SAM patch embedding, enabling SAM to accept inputs from non-RGB imaging modalities such as thermal, polarization, depth, HHA, and near-infrared. The authors construct a new benchmark, AIMS, from existing datasets, and report that SimCMF improves segmentation mIoU from 22.15% to 53.88% on average across five modalities compared with training from scratch, while also outperforming ViPT and CMX on pseudo new modalities (RGB-X). The paper includes an ablation study of adapter components, a comparison of full fine-tuning, LoRA, MLP adapter, and prompt tuning, and an analysis of learning rate and training data size effects.

Significance. If the reported results hold, the paper makes a practically useful contribution: it shows that a small, modality-agnostic adapter can transfer a frozen RGB foundation model to sensors with very different physical signal representations, and it documents a systematic exploration of adapter design and parameter-efficient fine-tuning strategies. The release of code and the construction of a multi-modality segmentation benchmark are also assets. However, the central quantitative claims are expressed through a nonstandard evaluation protocol, and the absence of statistical robustness measures currently limits confidence in the headline numbers.

major comments (4)
  1. [Supplement §11, Figure 8] The headline result, Table 1, is measured with an instance-level mIoU protocol in which semantic labels are decomposed into connected components and IoU is averaged over instances rather than semantic classes. This protocol is never validated against a standard semantic segmentation metric. On datasets like NYUv2 with 40 semantic classes, connected-component decomposition can produce many small or noisy instances, and per-instance averaging may be dominated by easy small regions, inflating absolute mIoU. Because the abstract and Table 1 present 22.15% to 53.88% as 'segmentation performance (mIoU)', the paper should report standard semantic mIoU per semantic category as well, or otherwise justify that the instance-level protocol is equivalent in difficulty. As it stands, the headline absolute numbers are not interpretable as standard semantic segmentation quality.
  2. [§4.1 and Supplement §8] All reported numbers are single runs with a learning rate sweep in which the best performance per model and per method is reported. No multiple seeds, error bars, or statistical significance tests are provided. This is load-bearing because several comparisons in Table 3 (e.g., LoRA 56.44 vs. MLP Adapter 57.25 on Thermal, and LoRA 45.09 vs. FFT 43.98 on Depth) are separated by margins smaller than typical run-to-run variation for this kind of fine-tuning. Without variance estimates, the claims that SimCMF 'consistently outperforms' baselines and that one PEFT method is superior to another are not established. Please report mean and standard deviation over at least three seeds, or otherwise justify that the reported margins are stable.
  3. [§4.1, 'AIMS dataset construction'] The benchmark construction is not sufficiently specified for reproducibility. The text lists dataset names and coarse image counts, but does not state the exact train/validation split used for each dataset, the protocol for merging RGBP-Glass and ZJU-RGBP, the conversion of NYUv2's semantic labels to instance labels, or the preprocessing applied to each modality besides resizing to 1024×1024. Since all tables depend on this benchmark, the missing split and label-conversion details prevent independent verification of every reported number. The supplement should provide a per-dataset data card with exact splits, label mappings, and any filtering steps.
  4. [Table 2 and Table 6] The claim that SimCMF 'consistently outperforms other baselines' is only substantiated for pseudo new modalities against reproduced ViPT and CMX baselines. For the five primary modalities in Table 1, the only comparison is training from scratch; no existing cross-modal segmentation method or adapter-based SAM variant is used as a reference on those datasets. Adding at least one established modality-specific or cross-modal baseline, or reporting standard semantic mIoU on the same datasets, would help calibrate whether the improvements are meaningful in absolute terms rather than solely relative to a deliberately weak baseline.
minor comments (5)
  1. [§4.3] The sentence 'we set the number of convolution layers to 1,3,5 to explore this question' appears to be a typo: the surrounding text and Figure 4 indicate the kernel size is varied, not the number of layers.
  2. [§4.3 and Supplement §10] There are typographical errors, including 'bacobone' in §4.3, 'prestrained' in Supplement §10, and 'Finetuing' in Table 4's caption. These should be corrected.
  3. [Qualitative figures and §4.2] The qualitative figures label a column as 'SAM Zero-shot' for depth, thermal, polarization, HHA, and NIR, but the main text only defines the zero-shot baseline for pseudo new modalities with available RGB input. It should be stated explicitly whether the zero-shot predictions for the five primary modalities use a paired RGB reference image or the raw modality image as input.
  4. [Figure 4] The bars in Figure 4 are annotated with numeric mIoU values that are not all repeated in the text; please ensure the figure and text report the same numbers, or clarify the reading of the figure.
  5. [Section 7] The 'Technical Report' section, which simply points to a previous arXiv version, is unusual for a formal paper and should either be integrated into the related work or removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SimCMF is an empirical recipe whose headline numbers depend on a self-defined instance-level metric, but no claim reduces by construction to its inputs or to self-citation.

full rationale

The paper contains no mathematical derivation chain whose conclusion is equivalent to its premises. The central claims are empirical: a small convolutional adapter before a frozen SAM patch embedding improves measured mIoU across five modalities, and the method outperforms ViPT and CMX on four pseudo-new modalities. These claims are supported by experiments on public datasets, not by a theorem that assumes the conclusion. The benchmark is constructed by the authors, and the supplement defines the evaluation metric as instance-level IoU after splitting semantic labels into connected components ('Instead of average IoU over semantic categories, we take the average IoU of all instances as the mIoU results'). That is a nonstandard and potentially unvalidated metric choice, and it is a legitimate correctness and interpretability concern, but it is applied uniformly to all compared methods. It does not make the reported improvement true by definition, because the model still has to learn to predict those instance masks; the metric does not encode the method's output. The adapter configuration (l=2, k=3, dims=64) was selected through experiments on polarization data and then transferred to other modalities. That is a hyperparameter-selection and generalizability limitation, not a fitted-input-called-prediction circularity, because the gains on depth, thermal, HHA, and NIR are empirical outcomes that are not constructed from those fitted values. The paper contains several self-citations (e.g., the previous technical report, prior polarization work by the authors), but none is load-bearing: no uniqueness theorem is imported from the authors' prior work, and no design choice is justified solely by a self-citation. The strongest quantitative claim is therefore an empirical finding under a self-defined protocol, and any weakness lies in the validity and reproducibility of that protocol, not in a circular derivation. Accordingly, no circular step meeting the quoted-evidence standard can be identified, and the score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method is an empirical recipe. Its main free parameters are the adapter architecture and learning rate schedule, chosen by validation on one modality. The core axioms are the transferability of SAM's frozen embedding and the validity of the constructed evaluation protocol. No new physical entities are introduced.

free parameters (3)
  • Cross-modal adapter layers, kernel size, and channel dimension = l=2, k=3, d=64
    Selected by sweeping on polarization datasets (RGBP-Glass, ZJU-RGBP); see Figure 4 and Table 5 in supplementary. These values are then applied to all other modalities.
  • Learning rate per model and modality = swept from 3e-6 to 3e-3, best reported
    The paper reports the peak mIoU after sweeping learning rates (Section 4.1 and Table 5), which is a selection procedure that can inflate results if the test set is used for model selection.
  • LoRA rank and adapter dimensions for PEFT = not specified; balanced to ~4% of full parameters
    Trainable parameters are balanced to 3.9 to 4.4 M (Table 4), but the exact rank or bottleneck size is not given, making exact replication difficult.
assumptions (4)
  • domain assumption A frozen pretrained patch embedding of SAM can be reused for any imaging modality after a small input adapter.
    This is the core modeling assumption, validated only on five modalities (Section 4.2, Figure 4). It is not derived and may fail for modalities with very different statistics (e.g., event cameras, hyperspectral).
  • domain assumption Segmentation labels can be converted from semantic to instance level by splitting disconnected components, and mIoU over these instances is a meaningful metric.
    Section 11 in the supplementary describes this post-processing. This can introduce errors for touching objects of the same class and is a non-standard evaluation protocol.
  • domain assumption The center-point click prompt protocol of SAM transfers to these modalities.
    They follow SAM's interactive setting (Section 4.1), but there is no evidence that a single center click is a good prompt for all these sensor types.
  • domain assumption Datasets are used as provided; no leakage exists between RGB reference and modality images in pseudo new modality experiments.
    They shuffle channels to avoid using RGB prior, but the pseudo new modality experiments still use the same underlying images as the RGB reference, which may make zero-shot an unfair baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality." pith.science (2026). https://pith.science/paper/UQ6UQK75

@misc{pith2026241118669,
  author       = {Pith},
  title        = {Pith review of: SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQ6UQK75}},
  note         = {Machine review of arXiv:2411.18669}
}
read the original abstract

Foundation models like ChatGPT and Sora that are trained on a huge scale of data have made a revolutionary social impact. However, it is extremely challenging for sensors in many different fields to collect similar scales of natural images to train strong foundation models. To this end, this work presents a simple and effective framework, SimCMF, to study an important problem: cross-modal fine-tuning from vision foundation models trained on natural RGB images to other imaging modalities of different physical properties (e.g., polarization). In SimCMF, we conduct a thorough analysis of different basic components from the most naive design and ultimately propose a novel cross-modal alignment module to address the modality misalignment problem. We apply SimCMF to a representative vision foundation model Segment Anything Model (SAM) to support any evaluated new imaging modality. Given the absence of relevant benchmarks, we construct a benchmark for performance evaluation. Our experiments confirm the intriguing potential of transferring vision foundation models in enhancing other sensors' performance. SimCMF can improve the segmentation performance (mIoU) from 22.15% to 53.88% on average for evaluated modalities and consistently outperforms other baselines. The code is available at https://github.com/mt-cly/SimCMF

Figures

Figures reproduced from arXiv: 2411.18669 by the authors.

Figure 1
Figure 1. Transferability Across Modalities. a, the number of natural images is significantly larger than images in other modalit￾ies in different areas, including medical imaging, thermal images, depth images, and polarization images. b, natural images can train vision foundation models, which can be applied to achieve strong performance on different downstream tasks. c, it is very challen￾ging for other modalities to benefi… view at source ↗
Figure 2
Figure 2. SimCMF Conceptual Overview. SimCMF receives new modality x as input and pass it through a cross-modal alignment module to obtain an embedding. The embedding matches the dimension of a pretrained foundation model backbone, and then we obtain the output y. The input and foundation are designed in a generic formulation for different input modalities and foundation models. In this work, we select SAM as a representative… view at source ↗
Figure 3
Figure 3. Qualitative Results. We transfer the segment any￾thing ability of SAM to different modalities, including segment￾ation from depth, thermal, polarization, HHA, and NIR images. The proposed method significantly improves segmentation quality compared to SAM zero-shot and training from scratch. chitecture without pretrained weights and train the network only with the new modality data. While we understand it is quite ch… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: The Effect of Learning Rate and Training Data Size. The models are evaluated on the polarization modality. a. the full fine-tuning and parameter efficient tuning achieve peak performance in different learning rates. b. increasing the scale of training data brings consi…
Figure 6
Figure 6. Figure 6: The Effect of the Configuration of our cross-modal alignment module, evaluated on Polarization modality. Based on the above results, we set the k, d, n to 3, 64, and 2, respectively, considering the trade-off of performance of efficiency. 43.81 67.66 66.24 72.14 68.76 …
Figure 7
Figure 7. Figure 7: The Training Curves for SimCMF and Baselines. SimCMF achieves the best performance. Method Params Finetuning methods RGB-T RGB-D RGB-HHA RGB-NIR CMX* [96] 403.8M Full finetuning 44.91 36.41 37.33 34.75 ViPT* [102] 94.5M Prompt tuning 75.93 48.89 49.50 51.90 SimCMF 94.4…
Figure 8
Figure 8. Figure 8: The Illustration of Segmentation Generation Pipeline in Our Benchmark. The semantic-level segmentation ground truth is split into instance-level segmentation ground truth. Train from scratch SAM Zero-shot Ours Ground truth Input with prompt [PITH_FULL_IMAGE:figures/fu…
Figure 9
Figure 9. Figure 9: Additional Qualitative Results in Depth Modality. Our approach can perform better than zero-shot and training from scratch. provide semantic labels. Hence, to align with the output of SAM, we perform post-processing to convert the semantic labels to instance labels by …
Figure 11
Figure 11. Figure 11: Additional Qualitative Results in Polarization Mod￾ality. Our approach can perform better than zero-shot and training from scratch. Train from scratch SAM Zero-shot Ours Ground truth Input with prompt [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Additional Qualitative Results in NIR Modality. Our approach can perform better than zero-shot and training from scratch. As shown in the figure, the performance of training from Train from scratch SAM Zero-shot Ours Ground truth Input with prompt [PITH_FULL_IMAGE:fi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

111 extracted references · 59 canonical work pages

  1. [1]

    Sequential modeling enables scalable learning for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re- cognition, pages 22861–22872, 2024. 1

    2024

  2. [2]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022. 3

    2022

  3. [3]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. 1, 3

    arXiv 2021

  4. [4]

    Multi-spectral sift for scene category recognition

    Matthew Brown and Sabine S ¨usstrunk. Multi-spectral sift for scene category recognition. In CVPR 2011, pages 177–

    2011

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Ad- vances in neural information processing systems, 33:1877– 1901, 2020. 1, 3

    1901

  6. [6]

    Pretrainable geometric graph neural network for antibody affinity maturation

    Huiyu Cai, Zuobai Zhang, Mingkai Wang, Bozitao Zhong, Quanxiao Li, Yuxuan Zhong, Yanling Wu, Tianlei Ying, and Jian Tang. Pretrainable geometric graph neural network for antibody affinity maturation. Nature Communications, 15(1):7785, 2024. 1

    2024

  7. [7]

    Sad: Segment any rgbd

    Jun Cen, Yizheng Wu, Kewei Wang, Xingyi Li, Jingkang Yang, Yixuan Pei, Lingdong Kong, Ziwei Liu, and Qifeng Chen. Sad: Segment any rgbd. arXiv preprint arXiv:2305.14207, 2023. 4

    arXiv 2023

  8. [8]

    Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 646–650. IEEE, 2022. 3

    2022

Show all 111 references
  1. [9]

    Domain ad- aptation for semantic segmentation with maximum squares loss

    Minghao Chen, Hongyang Xue, and Deng Cai. Domain ad- aptation for semantic segmentation with maximum squares loss. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 2090–2099, 2019. 3

    2019

  2. [10]

    Adaptformer: Ad- apting vision transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Ad- apting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems , 35: 16664–16678, 2022. 3, 4

    2022

  3. [11]

    A simple framework for contrastive learn- ing of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learn- ing of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020. 2, 4, 7, 14

    2020

  4. [12]

    Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more

    Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more. arXiv preprint arXiv:2304.09148, 2023. 4

    2023 arXiv

  5. [13]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 2

    2003 arXiv

  6. [14]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2

    2009

  7. [15]

    Lift: Language- interfaced fine-tuning for non-language machine learning tasks

    Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Mi- chael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language- interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35:1176...

    2022

  8. [16]

    Hyperspectral image super-resolution via non-negative structured sparse repres- entation

    Weisheng Dong, Fazuo Fu, Guangming Shi, Xun Cao, Jinjian Wu, Guangyu Li, and Xin Li. Hyperspectral image super-resolution via non-negative structured sparse repres- entation. IEEE Transactions on Image Processing , 25(5): 2337–2352, 2016. 1

    2016

  9. [17]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In International C...

    2021

  10. [18]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichten- hofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vis- ion (ICCV), pages 6824–6835, 2021. 3, 4, 7

    2021

  11. [19]

    Event-based vision: A survey

    Guillermo Gallego, Tobi Delbr ¨uck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, J ¨org Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE trans- actions on pattern analysis and machine intelligence , 44...

    2020

  12. [20]

    Low-latency auto- motive vision with event cameras

    Daniel Gehrig and Davide Scaramuzza. Low-latency auto- motive vision with event cameras. Nature, 629(8014): 1034–1040, 2024. 1

    2024

  13. [21]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 3

    2023

  14. [22]

    Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778,

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778,

    arXiv

  15. [23]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doer- sch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

    2020

  16. [24]

    Pct: Point cloud transformer

    Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai- Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media , 7:187– 199, 2021. 3

    2021

  17. [25]

    Learning rich features from rgb-d im- ages for object detection and segmentation

    Saurabh Gupta, Ross Girshick, Pablo Arbel ´aez, and Jitendra Malik. Learning rich features from rgb-d im- ages for object detection and segmentation. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13, pa...

    2014

  18. [26]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021. 14

    2021 arXiv

  19. [27]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9729–9738, 2020. 3, 14

    2020

  20. [28]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 16000–16009, 2022. 3

    2022

  21. [29]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 2, 3, 14

    2019

  22. [30]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2022. 2, 3, 4, 7, 8, 14

    2022

  23. [31]

    Polarization structured light 3d depth image sensor for scenes with reflective sur- faces

    Xuanlun Huang, Chenyang Wu, Xiaolan Xu, Baishun Wang, Sui Zhang, Chihchiang Shen, Chiennan Yu, Jiaxing Wang, Nan Chi, Shaohua Yu, et al. Polarization structured light 3d depth image sensor for scenes with reflective sur- faces. Nature Communications, 14(1):6855, 2023. 1

    2023

  24. [32]

    Glass segmentation with rgb-thermal image pairs

    Dong Huo, Jian Wang, Yiming Qian, and Yee-Hong Yang. Glass segmentation with rgb-thermal image pairs. IEEE Transactions on Image Processing, 32:1911–1926, 2023. 5

    1911

  25. [33]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 3

    2021

  26. [34]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. In International Conference on Learning...

    2022

  27. [35]

    Sam struggles in concealed scenes–empirical study on” segment anything”

    Ge-Peng Ji, Deng-Ping Fan, Peng Xu, Ming-Ming Cheng, Bowen Zhou, and Luc Van Gool. Sam struggles in concealed scenes–empirical study on” segment anything”. arXiv preprint arXiv:2304.06022, 2023. 4

    2023 arXiv

  28. [36]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Com- puter Vision, pages 709–727. Springer, 2022. 3, 8, 14

    2022

  29. [37]

    A multi-modal pre-training transformer for univer- sal transfer learning in metal–organic frameworks

    Yeonghun Kang, Hyunsoo Park, Berend Smit, and Jihan Kim. A multi-modal pre-training transformer for univer- sal transfer learning in metal–organic frameworks. Nature Machine Intelligence, 5(3):309–318, 2023. 3

    2023

  30. [38]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of naacL-HLT, page 2, 2019. 1, 3

    2019

  31. [39]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015–4026, 2023. 1,...

    2023

  32. [40]

    Decou- plenet: Decoupled network for domain adaptive semantic segmentation

    Xin Lai, Zhuotao Tian, Xiaogang Xu, Yingcong Chen, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Decou- plenet: Decoupled network for domain adaptive semantic segmentation. In European Conference on Computer Vis- ion, pages 369–387. Springer, 2022. 3

    2022

  33. [41]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 8

    2024

  34. [42]

    Polarized reflection re- moval with perfect alignment in the wild

    Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen. Polarized reflection re- moval with perfect alignment in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1750–1758, 2020. 1, 8

    2020

  35. [43]

    Shape from polarization for com- plex scenes in the wild

    Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, and Qifeng Chen. Shape from polarization for com- plex scenes in the wild. In Proceedings of the ieee/cvf con- ference on computer vision and pattern recognition , pages 12632–12641, 2022. 1

    2022

  36. [44]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 3, 4

    2024 arXiv

  37. [45]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 7

    2023

  38. [46]

    Prefix-tuning: Optim- izing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optim- izing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021. 3

    2021 arXiv

  39. [47]

    Heterogen- eous domain adaptation: An unsupervised approach

    Feng Liu, Guangquan Zhang, and Jie Lu. Heterogen- eous domain adaptation: An unsupervised approach. IEEE transactions on neural networks and learning systems , 31 (12):5588–5602, 2020. 3

    2020

  40. [48]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural inform- ation processing systems, 36, 2024. 3, 7

    2024

  41. [49]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 3

    2021

  42. [50]

    Frozen pretrained transformers as universal compu- tation engines

    Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mord- atch. Frozen pretrained transformers as universal compu- tation engines. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7628–7636, 2022. 2, 3, 7

    2022

  43. [51]

    Transferring knowledge fragments for learning dis- tance metric from a heterogeneous domain

    Yong Luo, Yonggang Wen, Tongliang Liu, and Dacheng Tao. Transferring knowledge fragments for learning dis- tance metric from a heterogeneous domain. IEEE transac- tions on pattern analysis and machine intelligence , 41(4): 1013–1026, 2018. 3

    2018

  44. [52]

    Segment anything in medical im- ages

    Jun Ma and Bo Wang. Segment anything in medical im- ages. arXiv preprint arXiv:2304.12306, 2023. 2, 3, 4

    2023 arXiv

  45. [53]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 1

    2024

  46. [54]

    Mul- timodal tactile sensing fused with vision for dexterous ro- botic housekeeping

    Qian Mao, Zijian Liao, Jinfeng Yuan, and Rong Zhu. Mul- timodal tactile sensing fused with vision for dexterous ro- botic housekeeping. Nature Communications, 15(1):6871,

  47. [55]

    Glass segmentation using intensity and spectral polarization cues

    Haiyang Mei, Bo Dong, Wen Dong, Jiaxi Yang, Seung- Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, and Xin Yang. Glass segmentation using intensity and spectral polarization cues. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , page...

    2022

  48. [56]

    Scaling deep learning for materials discovery

    Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624 (7990):80–85, 2023. 1

    2023

  49. [57]

    4m: Massively multimodal masked modeling

    David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36, 2024. 4

    2024

  50. [58]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 5, 15

    2012

  51. [59]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. Advances in neural information processing systems , 35...

    2022

  52. [60]

    Foundation model for cancer imaging biomark- ers

    Suraj Pai, Dennis Bontempi, Ibrahim Hadzic, Vasco Prudente, Mateo Soka ˇc, Tafadzwa L Chaunzwa, Simon Bernatz, Ahmed Hosny, Raymond H Mak, Nicolai J Birk- bak, et al. Foundation model for cancer imaging biomark- ers. Nature machine intelligence, 6(3):354–367, 2024. 1

    2024

  53. [61]

    Unsupervised intra-domain adaptation for semantic segmentation through self-supervision

    Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3764–3773, 2020. 3

    2020

  54. [62]

    Transfer learning for metal–organic frameworks

    Jie Pan. Transfer learning for metal–organic frameworks. Nature Computational Science, 3(4):280–280, 2023. 3

    2023

  55. [63]

    A survey on transfer learning

    Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engin- eering, 22(10):1345–1359, 2009. 1

    2009

  56. [64]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International conference on machine learning...

    2021

  57. [65]

    Transfer learning with kernel methods

    Adityanarayanan Radhakrishnan, Max Ruiz Luyten, Neha Prasad, and Caroline Uhler. Transfer learning with kernel methods. Nature Communications, 14(1):5570, 2023. 3

    2023

  58. [66]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 4

    2021

  59. [67]

    Cross-modal fine-tuning: Align then refine

    Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. In Interna- tional Conference on Machine Learning , pages 31030– 31056. PMLR, 2023. 2, 3, 7

    2023

  60. [68]

    Depth estimation from camera im- age and mmwave radar point cloud

    Akash Deep Singh, Yunhao Ba, Ankur Sarker, Howard Zhang, Achuta Kadambi, Stefano Soatto, Mani Srivast- ava, and Alex Wong. Depth estimation from camera im- age and mmwave radar point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re- cognition,...

    2023

  61. [69]

    Rna secondary structure prediction using an en- semble of two-dimensional deep neural networks and trans- fer learning

    Jaswinder Singh, Jack Hanson, Kuldip Paliwal, and Yaoqi Zhou. Rna secondary structure prediction using an en- semble of two-dimensional deep neural networks and trans- fer learning. Nature communications, 10(1):5407, 2019. 3

    2019

  62. [70]

    Rtfnet: Rgb- thermal fusion network for semantic segmentation of urban scenes

    Yuxiang Sun, Weixun Zuo, and Ming Liu. Rtfnet: Rgb- thermal fusion network for semantic segmentation of urban scenes. IEEE Robotics and Automation Letters, 4(3):2576– 2583, 2019. 7

    2019

  63. [71]

    Seeing far in the dark with patterned flash

    Zhanghao Sun, Jian Wang, Yicheng Wu, and Shree Nayar. Seeing far in the dark with patterned flash. In European Conference on Computer Vision, pages 709–727. Springer,

  64. [72]

    Dabs: A domain-agnostic benchmark for self-supervised learning.Advances in neural information processing systems, 2021

    Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah Goodman. Dabs: A domain-agnostic benchmark for self-supervised learning.Advances in neural information processing systems, 2021. 3

    2021

  65. [73]

    Can sam segment anything? when sam meets camouflaged object detection

    Lv Tang, Haoke Xiao, and Bo Li. Can sam segment anything? when sam meets camouflaged object detection. arXiv preprint arXiv:2304.04709, 2023. 4

    2023 arXiv

  66. [74]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 ,

    arXiv

  67. [75]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  68. [76]

    Neural nano-optics for high-quality thin lens ima- ging

    Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar, and Felix Heide. Neural nano-optics for high-quality thin lens ima- ging. Nature communications, 12(1):6493, 2021. 1

    2021

  69. [77]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 3

    2017

  70. [78]

    Audio transformers: Transformer architectures for large scale audio understand- ing

    Prateek Verma and Jonathan Berger. Audio transformers: Transformer architectures for large scale audio understand- ing. adieu convolutions. arXiv preprint arXiv:2105.00335,

    arXiv

  71. [79]

    Reprogramming pretrained language models for protein sequence represent- ation learning

    Ria Vinod, Pin-Yu Chen, and Payel Das. Reprogramming pretrained language models for protein sequence represent- ation learning. arXiv preprint arXiv:2301.02120, 2023. 2, 3

    2023 arXiv

  72. [80]

    Advent: Adversarial entropy min- imization for domain adaptation in semantic segmentation

    Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick P´erez. Advent: Adversarial entropy min- imization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2517–2526, 2019. 3

    2019

  73. [81]

    Predicting fault slip via transfer learning

    Kun Wang, Christopher W Johnson, Kane C Bennett, and Paul A Johnson. Predicting fault slip via transfer learning. Nature communications, 12(1):7319, 2021. 3

    2021

  74. [82]

    Uncertainty-aware clustering for unsupervised domain adaptive object re- identification

    Pengfei Wang, Changxing Ding, Wentao Tan, Mingming Gong, Kui Jia, and Dacheng Tao. Uncertainty-aware clustering for unsupervised domain adaptive object re- identification. IEEE Transactions on Multimedia, 2022. 3

    2022

  75. [83]

    Sub-surface thermal measurement in additive man- ufacturing via machine learning-enabled high-resolution fiber optic sensing

    Rongxuan Wang, Ruixuan Wang, Chaoran Dou, Shuo Yang, Raghav Gnanasambandam, Anbo Wang, and Zhenyu Kong. Sub-surface thermal measurement in additive man- ufacturing via machine learning-enabled high-resolution fiber optic sensing. Nature Communications, 15(1):7568,

  76. [84]

    Internimage: Exploring large-scale vision foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2023

  77. [85]

    Internimage: Exploring large-scale vision foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vis- ion...

    2023

  78. [86]

    Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition

    Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in Neural Information Processing Systems, 34:11960–11973,

  79. [87]

    Randomized quantization: A generic aug- mentation for data agnostic self-supervised learning

    Huimin Wu, Chenyang Lei, Xiao Sun, Peng-Shuai Wang, Qifeng Chen, Kwang-Ting Cheng, Stephen Lin, and Zhirong Wu. Randomized quantization: A generic aug- mentation for data agnostic self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vis...

    2023

  80. [88]

    Medical sam adapter: Adapting segment anything model for med- ical image segmentation

    Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel. Medical sam adapter: Adapting segment anything model for med- ical image segmentation. arXiv preprint arXiv:2304.12620,

    arXiv

  81. [89]

    Point transformer v2: Grouped vector attention and partition-based pooling

    Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. Advances in Neural Information Processing Systems, 35:33330–33342, 2022. 3

    2022

  82. [90]

    Polarization- driven semantic segmentation via efficient attention- bridged fusion

    Kaite Xiang, Kailun Yang, and Kaiwei Wang. Polarization- driven semantic segmentation via efficient attention- bridged fusion. Optics Express, 29(4):4802–4820, 2021. 5, 15

    2021

  83. [91]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663, 2022. 3

    2022

  84. [92]

    Video-rate hyperspectral camera based on a cmos-compatible random array of fabry–p ´erot filters

    Motoki Yako, Yoshikazu Yamaoka, Takayuki Kiyohara, Chikai Hosokawa, Akihiro Noda, Klaas Tack, Nick Spooren, Taku Hirasawa, and Atsushi Ishikawa. Video-rate hyperspectral camera based on a cmos-compatible random array of fabry–p ´erot filters. Nature Photonics, 17(3):218– 223, 2023. 1

    2023

  85. [93]

    A vision chip with complement- ary pathways for open-world sensing

    Zheyu Yang, Taoyi Wang, Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, et al. A vision chip with complement- ary pathways for open-world sensing. Nature, 629(8014): 1027–1033, 2024. 1

    2024

  86. [94]

    Superanimal pretrained pose estimation models for behavioral analysis

    Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Steffen Schneider, Maxime Vidal, Tian Qiu, Alexander Mathis, and Mackenzie Weygandt Mathis. Superanimal pretrained pose estimation models for behavioral analysis. Nature Commu- nications, 15(1):5165, 2024. 1

    2024

  87. [95]

    Taskonomy: Disentangling task transfer learning

    Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, 2018. 2

    2018

  88. [96]

    Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers

    Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE Transactions on Intelligent Transportation Systems ,

  89. [97]

    A generalist vision–language foundation model for diverse biomedical tasks

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine, pages 1–13, 2024. 3

    2024

  90. [98]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 8

    2023

  91. [99]

    Meta- transformer: A unified framework for multimodal learning

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hong- sheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta- transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023. 3

    2023 arXiv

  92. [100]

    Point transformer

    Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 16259–16268, 2021. 3

    2021

  93. [101]

    Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4

    Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xi- uying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari, et al. Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4. Nature Communications, 15(1):5649, 2024. 1

    2024

  94. [102]

    Visual prompt multi-modal tracking

    Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-modal tracking. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9516–9526, 2023. 2, 6, 15

    2023

  95. [103]

    Generalized decoding for pixel, image, and language

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 151...

    2023

  96. [104]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. In NeurIPS, 2023. 4

    2023

  97. [105]

    Unsupervised domain adaptation for semantic segmenta- tion via class-balanced self-training

    Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmenta- tion via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV) , pages 289–305, 2018. 3 SimCMF: A Simple Cross-modal Fine-tuning...

    2018

  98. [106]

    The foundation model SAM with ViT-B [17] as backbone contains 93.7M para- meters from the image encoder, prompt encoder, and mask decoder

    Additional Training Details We report the effect of different finetuning strategies on trainable parameters in Table 4. The foundation model SAM with ViT-B [17] as backbone contains 93.7M para- meters from the image encoder, prompt encoder, and mask decoder. Full finetuning ma...

  99. [108]

    As shown in Figure 6, SimCMF stack then convolutional layers withk kernel size and dimension d

    Additional Controlled Experiments We provide the study of the hyper-parameter setting of Sim- CMF by applying it to the Polarization modality. As shown in Figure 6, SimCMF stack then convolutional layers withk kernel size and dimension d. SimCMF achieves best 72.7% mIoU by set...

  100. [109]

    One can observe the training from scratch only achieves 25.43% mIoU, signific- antly worse than other methods using prestrained weight as initialization

    Additional Comparisons We report the training curve of SimCMF and baselines on the Polarization dataset in Figure 7. One can observe the training from scratch only achieves 25.43% mIoU, signific- antly worse than other methods using prestrained weight as initialization. To tac...

  101. [110]

    Additional Benchmark Details To study the problem of cross-modality transfer learning of SAM, we construct a new benchmark by collecting im- age segmentation datasets from different modalities, as de- scribed in the main paper. However, the segmentation labels of SAM are insta...

  102. [111]

    For the SAM zero-shot performance, we use the provided RGB reference as the input

    Additional Qualitative Results We provide further qualitative visualizations in Figure 9 to Figure 13. For the SAM zero-shot performance, we use the provided RGB reference as the input. We present the res- ults on diverse image modalities for better understanding. Train from s...

  103. [1024]

    to meet the requirements of SAM

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.