{"id":"e0271be6-6794-41b9-8824-b2ae6cdaaf2d","arxiv_id":"2411.18669","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A small convolutional adapter plus a frozen patch embedding lets SAM segment depth, thermal, polarization, HHA, and NIR images far better than training from scratch, with parameter-efficient fine-tuning matching full fine-tuning.","lead":"This paper introduces SimCMF, a method for adapting the Segment Anything Model (SAM) to images from sensors that do not capture ordinary RGB color, such as depth, thermal, polarization, and near-infrared cameras. The authors report large gains in segmentation accuracy compared to training a model from scratch, and argue that pretrained vision models can be reused across imaging modalities with a small trainable adapter.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 22.15→53.88 mIoU headline rests on an unvalidated instance-level connected-component metric; if that metric is re-expressed as standard semantic mIoU, the central quantitative claim may not survive.","rationale":"The stress-test pass looked for the weakest link in the argument. The strongest claim is the 22.15-to-53.88 mIoU improvement, which is the paper's headline evidence for cross-modal transfer. That number depends entirely on a custom evaluation protocol: semantic labels are split into connected components and mIoU is averaged over instances. This protocol is not validated against standard semantic segmentation metrics, and it may systematically favor models that produce compact masks around clicks. The reader's verdict already identifies the evaluation protocol as a concern, so my objection is partial agreement rather than a new finding. The proposed check is feasible because the original semantic labels exist; re-running the same trained models with semantic mIoU would settle whether the improvement is metric-specific. I did not find a more fundamental flaw in the architecture or training itself. The conditional verdict remains appropriate, with the added requirement that the authors validate or replace the instance-level protocol before the quantitative claims are accepted.","tokens_in":20038,"tokens_out":9622,"duration_ms":95085,"concrete_test":"Recompute Table 1 with standard semantic mIoU on the original semantic labels, for at least the datasets that provide them: NYUv2 (depth and HHA) with its 40-class protocol, ZJU-RGBP, and the two glass datasets. For each clicked instance, assign the predicted mask to the semantic class of the ground-truth connected component, aggregate IoU per class, and average over classes. If the SimCMF-vs-from-scratch gap shrinks to a few points or reverses on any modality, the central quantitative claim is an artifact of the connected-component instance metric rather than a real segmentation improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—SimCMF raises mIoU from 22.15% to 53.88% across five modalities—is measured with a nonstandard protocol described only in the supplement (§11): semantic labels are decomposed into instance masks via connected components, and mIoU is averaged over instances rather than semantic categories. This protocol is never validated against a standard segmentation metric. On datasets such as NYUv2 (40 semantic classes), connected-component decomposition can create many small, noisy instances; per-instance averaging can then be dominated by easy, small objects, and a model that outputs a compact blob around the center click can accumulate high IoU without demonstrating semantic segmentation quality. The binary glass datasets become a collection of glass regions, which may or may not correspond to the user-meaningful instances the benchmark intends. The paper also does not report dataset splits or label-conversion details for every source dataset, so the reported numbers are not independently reproducible. Because all methods are evaluated under the same protocol, the relative comparisons are internally consistent; but the headline absolute numbers and the words 'segmentation performance (mIoU)' are defined by an unvalidated metric. If this metric inflates scores unevenly across modalities, the claim that SimCMF improves segmentation on average across evaluated modalities is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SimCMF, a cross-modal fine-tuning strategy that prepends a small convolutional adapter to a frozen pretrained SAM patch embedding, enabling SAM to accept inputs from non-RGB imaging modalities such as thermal, polarization, depth, HHA, and near-infrared. The authors construct a new benchmark, AIMS, from existing datasets, and report that SimCMF improves segmentation mIoU from 22.15% to 53.88% on average across five modalities compared with training from scratch, while also outperforming ViPT and CMX on pseudo new modalities (RGB-X). The paper includes an ablation study of adapter components, a comparison of full fine-tuning, LoRA, MLP adapter, and prompt tuning, and an analysis of learning rate and training data size effects.","tokens_in":20258,"tokens_out":3273,"duration_ms":33821,"significance":"If the reported results hold, the paper makes a practically useful contribution: it shows that a small, modality-agnostic adapter can transfer a frozen RGB foundation model to sensors with very different physical signal representations, and it documents a systematic exploration of adapter design and parameter-efficient fine-tuning strategies. The release of code and the construction of a multi-modality segmentation benchmark are also assets. However, the central quantitative claims are expressed through a nonstandard evaluation protocol, and the absence of statistical robustness measures currently limits confidence in the headline numbers.","major_comments":[{"comment":"The headline result, Table 1, is measured with an instance-level mIoU protocol in which semantic labels are decomposed into connected components and IoU is averaged over instances rather than semantic classes. This protocol is never validated against a standard semantic segmentation metric. On datasets like NYUv2 with 40 semantic classes, connected-component decomposition can produce many small or noisy instances, and per-instance averaging may be dominated by easy small regions, inflating absolute mIoU. Because the abstract and Table 1 present 22.15% to 53.88% as 'segmentation performance (mIoU)', the paper should report standard semantic mIoU per semantic category as well, or otherwise justify that the instance-level protocol is equivalent in difficulty. As it stands, the headline absolute numbers are not interpretable as standard semantic segmentation quality.","section":"Supplement §11, Figure 8"},{"comment":"All reported numbers are single runs with a learning rate sweep in which the best performance per model and per method is reported. No multiple seeds, error bars, or statistical significance tests are provided. This is load-bearing because several comparisons in Table 3 (e.g., LoRA 56.44 vs. MLP Adapter 57.25 on Thermal, and LoRA 45.09 vs. FFT 43.98 on Depth) are separated by margins smaller than typical run-to-run variation for this kind of fine-tuning. Without variance estimates, the claims that SimCMF 'consistently outperforms' baselines and that one PEFT method is superior to another are not established. Please report mean and standard deviation over at least three seeds, or otherwise justify that the reported margins are stable.","section":"§4.1 and Supplement §8"},{"comment":"The benchmark construction is not sufficiently specified for reproducibility. The text lists dataset names and coarse image counts, but does not state the exact train/validation split used for each dataset, the protocol for merging RGBP-Glass and ZJU-RGBP, the conversion of NYUv2's semantic labels to instance labels, or the preprocessing applied to each modality besides resizing to 1024×1024. Since all tables depend on this benchmark, the missing split and label-conversion details prevent independent verification of every reported number. The supplement should provide a per-dataset data card with exact splits, label mappings, and any filtering steps.","section":"§4.1, 'AIMS dataset construction'"},{"comment":"The claim that SimCMF 'consistently outperforms other baselines' is only substantiated for pseudo new modalities against reproduced ViPT and CMX baselines. For the five primary modalities in Table 1, the only comparison is training from scratch; no existing cross-modal segmentation method or adapter-based SAM variant is used as a reference on those datasets. Adding at least one established modality-specific or cross-modal baseline, or reporting standard semantic mIoU on the same datasets, would help calibrate whether the improvements are meaningful in absolute terms rather than solely relative to a deliberately weak baseline.","section":"Table 2 and Table 6"}],"minor_comments":[{"comment":"The sentence 'we set the number of convolution layers to 1,3,5 to explore this question' appears to be a typo: the surrounding text and Figure 4 indicate the kernel size is varied, not the number of layers.","section":"§4.3"},{"comment":"There are typographical errors, including 'bacobone' in §4.3, 'prestrained' in Supplement §10, and 'Finetuing' in Table 4's caption. These should be corrected.","section":"§4.3 and Supplement §10"},{"comment":"The qualitative figures label a column as 'SAM Zero-shot' for depth, thermal, polarization, HHA, and NIR, but the main text only defines the zero-shot baseline for pseudo new modalities with available RGB input. It should be stated explicitly whether the zero-shot predictions for the five primary modalities use a paired RGB reference image or the raw modality image as input.","section":"Qualitative figures and §4.2"},{"comment":"The bars in Figure 4 are annotated with numeric mIoU values that are not all repeated in the text; please ensure the figure and text report the same numbers, or clarify the reading of the figure.","section":"Figure 4"},{"comment":"The 'Technical Report' section, which simply points to a previous arXiv version, is unusual for a formal paper and should either be integrated into the related work or removed.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The main risk I see is the evaluation protocol: the headline numbers are computed with an instance-level connected-component metric that is not validated. If the authors can supply standard semantic mIoU results and error bars over multiple seeds, I would view the central claim as much stronger. I would also encourage the editor to consider whether the benchmark is documented in enough detail to serve as a reusable resource, since the paper presents it as a contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look if you work on transferring foundation models to new sensors. The core recipe is simple and plausible: freeze the SAM patch embedding, prepend a small two-layer 3x3 conv adapter, and fine-tune with LoRA or an MLP adapter. The systematic ablations on adapter design are the real contribution, and the consistent gains over training-from-scratch and over ViPT/CMX on pseudo RGB-X inputs are credible evidence that the approach has value. Code and hyperparameters are provided, which is more than many papers do.\n\nThe soft spots are real, and the biggest one is the evaluation metric. The headline 22.15-to-53.88 mIoU improvement is computed with a nonstandard protocol: semantic labels are split into connected components, and mIoU is averaged over instances, not semantic classes. That protocol is described only in the supplement and is never validated against standard semantic mIoU. For datasets like NYUv2 with 40 classes, connected components can fragment labels into many small masks, making per-instance averaging easy to game by predicting small blobs around the click. So the absolute numbers are hard to interpret, and the claim that SimCMF improves segmentation by that margin is not established. The relative comparisons are internally consistent because all methods face the same metric, but the paper sells the absolute improvement in the abstract and that is a problem.\n\nOther weaknesses are more standard: no error bars or multiple seeds, learning-rate sweeps with peak reporting, and some dataset splits and label-conversion details are under-specified. The training-from-scratch baseline is deliberately weak, though the pseudo-modality comparisons against ViPT and CMX partially address that concern. The \"any modality\" claim rests on five sensors, which is a reasonable start but not a universal proof.\n\nWho is the paper for? Researchers adapting SAM or similar ViTs to depth, thermal, polarization, or NIR data will find the recipe useful and the ablations informative. The metric issue means the exact numbers should not be quoted without caveats, but the method itself is simple and likely to transfer.\n\nThis deserves a serious referee. The right outcome is probably a major revision where the authors validate the instance-level protocol against standard semantic mIoU, add seeds or error bars, and report standard metrics where the datasets allow them. I would not desk-reject it.","headline":"A useful empirical recipe for adapting SAM to non-RGB sensors, but the headline numbers rest on an unvalidated instance-level metric that needs scrutiny before the quantitative claims can be trusted.","tokens_in":20837,"tokens_out":1919,"would_cite":true,"duration_ms":19737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SimCMF shows that SAM's frozen RGB patch embedding, paired with a small cross-modal adapter, can be fine-tuned to segment images from thermal, depth, polarization, HHA, and near-infrared sensors, raising average mIoU from 22.15% to 53.88%.","keywords":["cross-modal fine-tuning","Segment Anything Model","parameter-efficient fine-tuning","modality alignment","image segmentation","polarization imaging","depth imaging","thermal imaging"],"falsifier":"Run the released SimCMF code on a sixth modality whose geometry and statistics are far from natural RGB—for example raw event-camera frames, ultrasound, or X-ray attenuation—using the same training protocol; if instance-level mIoU does not exceed a from-scratch ViT-B trained on that modality, the paper's central claim that frozen SAM patch embeddings transfer to any imaging modality would be falsified for that sensor.","tokens_in":19808,"feed_emoji":"📡","tokens_out":4023,"duration_ms":36455,"temperature":0.7,"pith_summary":"SimCMF asks whether vision foundation models trained on natural RGB images can be reused for other imaging sensors that lack large datasets. It answers yes: prepend a small cross-modal adapter to the frozen patch embedding of SAM, then fine-tune with parameter-efficient methods. Across five modalities (thermal, polarization, depth, HHA, NIR), segmentation mIoU rises from 22.15% to 53.88% on average versus training from scratch, and the method beats existing RGB-X fusion baselines without using any RGB channel-order prior. The paper also constructs a new benchmark, AIMS, so that cross-modal transfer can be evaluated consistently. If the claim holds, data-scarce sensor fields can inherit interactive segmentation ability from RGB foundation models at low compute.","feed_headline":"Frozen SAM adapters lift five sensors from 22% to 54% mIoU","feed_subtitle":"A small cross-modal adapter lets SAM's RGB-pretrained features segment thermal, depth, polarization, HHA, and NIR images.","key_machinery":"The load-bearing component is the cross-modal alignment adapter: a stack of two convolutional layers with 3×3 kernels, 64 channels, and ReLU nonlinearity (except the last layer) that projects an arbitrary C-channel sensor input into the dimensional space of SAM's patch embedding. The pretrained patch embedding is kept frozen, a choice the paper shows is crucial for training stability and final performance. The adapter resolves channel-dimension misalignment while letting the frozen RGB-trained features carry the semantic load, and the backbone is then tuned with LoRA or an MLP adapter.","core_discovery":"The paper demonstrates that a vision foundation model pretrained on natural RGB images can be adapted to arbitrary imaging modalities by inserting a lightweight cross-modal alignment module before the model's frozen patch embedding. With SAM as the backbone, this raises instance-level segmentation mIoU from 22.15% to 53.88% on average across thermal, polarization, depth, HHA, and near-infrared images, compared with training the same architecture from scratch. It further shows that parameter-efficient fine-tuning (LoRA and MLP adapters) matches full fine-tuning while updating only about 4% of the backbone parameters, and that on pseudo-new modalities (RGB-T, RGB-D, RGB-HHA, RGB-NIR) SimCMF outperforms dedicated fusion baselines such as ViPT and CMX without exploiting which channels come from RGB.","pith_inferences":["The frozen RGB patch embedding may be acting less as an RGB-specific encoder and more as a generic local-texture encoder, predicting that the adapter will also transfer to unseen modalities such as event cameras, ultrasound, or X-ray, provided those sensors share local-contrast structure with natural images.","The 3×3 kernel result suggests that a small spatial context is needed to merge multi-channel sensor signals, implying that pure per-pixel channel projection is insufficient and that early fusion should preserve local spatial correlation.","Because performance keeps rising with more fine-tuning data (Figure 5b), a testable extension is that SimCMF's edge over from-scratch training will shrink but persist as modality datasets scale.","The channel-shuffling experiment hints that the adapter learns to rediscover RGB-like features from arbitrary channel order, so it is worth testing whether the same adapter works on spectrograms or range-Doppler images, which are imaging-like but not natural images."],"forward_implications":["Any imaging modality with modest labeled data can obtain SAM-level interactive segmentation by adding a small adapter, without retraining the backbone.","Parameter-efficient fine-tuning suffices, cutting trainable parameters to roughly 4% of full fine-tuning while matching accuracy and lowering hardware requirements.","The design recipe (frozen pretrained patch embedding, 3×3 kernels, two layers, 64 dims) provides a plug-in rule for future cross-modal adapters.","The AIMS benchmark offers a shared evaluation protocol (instance-level mIoU extracted from semantic labels, center-click prompts) for comparing cross-modal transfer methods.","On pseudo-new modalities, SimCMF outperforms dedicated RGB-X fusion baselines without using the RGB channel-prior."],"supporting_citations":[{"why":"Supplies the pretrained Segment Anything Model that SimCMF transfers to new modalities.","marker":"[39]"},{"why":"Provides the observation that freezing the patch embedding improves training stability, which motivates the frozen patch embedding design.","marker":"[18]"},{"why":"Defines LoRA, the parameter-efficient fine-tuning method used to train the adapted foundation model.","marker":"[30]"},{"why":"Describes ViPT, a prompt-tuning baseline for RGB-X referencing that SimCMF must outperform on pseudo-new modalities.","marker":"[102]"},{"why":"Describes CMX, an RGB-X fusion baseline that SimCMF is compared against in the pseudo-new modality experiments.","marker":"[96]"},{"why":"Supplies an alternative channel-alignment strategy (MedicalSAM) that the paper analyzes and compares in terms of FLOPs and accuracy.","marker":"[88]"},{"why":"Provides the polarization dataset (RGBP-Glass) used for the cross-modal alignment module ablations.","marker":"[55]"},{"why":"Provides the NYUv2 depth dataset used as one of the five evaluated modalities in the benchmark.","marker":"[58]"}],"fun_headline_variants":["Adapter lifts SAM from RGB to 5 sensors: 22% to 54% mIoU","Tiny adapter makes SAM work on thermal, depth, NIR images","Cross-modal fine-tuning: SAM adapters boost mIoU 2.4x","Vision foundation models adapt to any sensor with 4% params","SAM sees beyond RGB: adapter raises mIoU 22→54%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire transfer rests on the empirical claim that SAM's frozen RGB patch embedding, after a small learned projection, remains informative for inputs from physically different sensors; the paper only tests five modalities, so a sensor whose local structure RGB features cannot represent would break the method.","fun_headline_variants_meta":{"raw":{"variants":["Adapter lifts SAM from RGB to 5 sensors: 22% to 54% mIoU","Tiny adapter makes SAM work on thermal, depth, NIR images","Cross-modal fine-tuning: SAM adapters boost mIoU 2.4x","Vision foundation models adapt to any sensor with 4% params","SAM sees beyond RGB: adapter raises mIoU 22→54%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1594,"prompt_tokens":946,"completion_tokens":648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":562,"tokens_out":648,"duration_ms":6031,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:07:51.585212+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released SimCMF code on a sixth modality whose geometry and statistics are far from natural RGB—for example raw event-camera frames, ultrasound, or X-ray attenuation—using the same training protocol; if instance-level mIoU does not exceed a from-scratch ViT-B trained on that modality, the paper's central claim that frozen SAM patch embeddings transfer to any imaging modality would be falsified for that sensor.","supporting_citations":[{"cited_title":"Segment anything","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Segment Anything Model that SimCMF transfers to new modalities."},{"cited_title":"Visual prompt multi-modal tracking","cited_arxiv_id":null,"evidence_quote":"Describes ViPT, a prompt-tuning baseline for RGB-X referencing that SimCMF must outperform on pseudo-new modalities."},{"cited_title":"Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers","cited_arxiv_id":null,"evidence_quote":"Describes CMX, an RGB-X fusion baseline that SimCMF is compared against in the pseudo-new modality experiments."},{"cited_title":"Glass segmentation using intensity and spectral polarization cues","cited_arxiv_id":null,"evidence_quote":"Provides the polarization dataset (RGBP-Glass) used for the cross-modal alignment module ablations."},{"cited_title":"Indoor segmentation and support inference from rgbd images","cited_arxiv_id":null,"evidence_quote":"Provides the NYUv2 depth dataset used as one of the five evaluated modalities in the benchmark."}],"review_version":1}