{"id":"5ffd3734-f993-462c-80d0-b6355f3f1267","arxiv_id":"2412.04220","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MLE-SAM adapts SAM2 with modality-specific LoRA experts and a routing mechanism, reporting state-of-the-art multi-modal segmentation results on DELIVER, MUSES, and MCubeS, although the comparisons are confounded by different backbone sizes.","lead":"This paper adapts SAM2, a powerful image segmentation model, to work with multiple sensor inputs like RGB, depth, LiDAR, and event cameras for semantic segmentation. The authors report large accuracy gains on three driving datasets, especially when some sensors are missing or noisy, but the comparisons may be unfair because the proposed model uses a stronger backbone than its competitors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLE-SAM's gains are confounded with backbone: it uses SAM2/Hiera-B+ while baselines use MiT-B0/ViT-B, no SAM2-LoRA control is reported, and the 32.15% gain is vs CMNeXt, which the paper says needs RGB. Matched-backbone controls are needed before the SOTA claim holds.","rationale":"The paper proposes MLE-SAM: a frozen SAM2/Hiera-B+ encoder, per-modality LoRA, a top-k MoE router, and a dual-path decoder with an auxiliary head. The central claim is that this design is state-of-the-art on multi-modal segmentation and robust to missing modalities. For that to be supported, the reported mIoU gains must be attributable to the proposed modules rather than to a more capable base encoder. The paper does not provide that attribution. Section IV-A states that backbones were \"standardized\" to MiT-B0 (CMNeXt), ViT-B (CWSAM, SAM-LoRA), and Hiera-B+ (MLE-SAM); those are not comparable controls. The only same-backbone ablation (Table V) varies the router and auxiliary head within Hiera-B+, but never includes a plain SAM2-LoRA baseline without the router and auxiliary head. Hence Tables I-III can be read as a SAM2/Hiera-B+ advantage, not a MoE-LoRA advantage. The 32.15% missing-modality improvement in Table VII is against CMNeXt, which the paper's own text says depends on the RGB/Frame modality; the matched-comparison margin over SAM-LoRA is 4.99 points. The reader's REJECT verdict is therefore appropriate, and I see no reason to change it. I credit the authors for a fairly detailed architecture description and for including ablations and robustness studies, and I am not accusing any misconduct. The concern is purely that the experiment design has not isolated the contribution. Secondary issues, including unrelated citations such as reference [55] on the Hubble tension, absent code, and no error bars, reinforce but do not drive the verdict.","tokens_in":26039,"tokens_out":11411,"duration_ms":115916,"concrete_test":"Add the missing control to Tables I, II, and VII on DELIVER and MUSES: a SAM2/Hiera-B+ model with per-modality LoRA (rank 32) and simple averaging of the modality-specific logits, keeping all training hyperparameters from Table IX and no MoE router or auxiliary head. If this control's mIoU is within about 1 point of MLE-SAM on the full-modality and missing-modality rows, the paper's gains are attributable to the base encoder/SAM2 rather than to the proposed MoE-LoRA routing and dual head. If the control is much lower, the proposed components are validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison is not controlled for backbone. Section IV-A says backbones were \"standardized\" as MiT-B0 for CMNeXt, ViT-B for CWSAM and SAM-LoRA, and Hiera-B+ for MLE-SAM. These are different encoders, and Hiera-B+ is the hierarchical backbone of SAM2, not the SAM ViT used by the two SAM baselines. Consequently, Table I's RGB-only gap of 3.39 mIoU over SAM-LoRA, Table II's 8.04 gap on Frame-only, and the large MCubeS gaps over CMNeXt are all compatible with a backbone-capacity explanation. The ablations in Table V isolate the router and auxiliary head only on top of the same Hiera-B+ base; they never compare against a plain SAM2-LoRA model with Hiera-B+, per-modality LoRA, and simple averaging. Without that control, the contributions of the MoE router and dual-head design are not separated from the choice of base encoder. The missing-modality headline is similarly fragile: the 32.15% improvement in Table VII is the gap between MLE-SAM and CMNeXt, yet the paper itself states that CMNeXt relies on the RGB/Frame modality and collapses to 3.50 mIoU when Frame-only is presented after F-L-E training; the same-backbone comparison to SAM-LoRA is only 4.99 points. Therefore neither the SOTA claim nor the robustness claim is demonstrated as submitted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MLE-SAM, a method for adapting the SAM2 model to multi-modal semantic segmentation by inserting modality-specific LoRA modules into the frozen image encoder, combining their outputs through a top-k Mixture-of-Experts routing mechanism, and adding an auxiliary segmentation head for multi-scale fusion. The method is evaluated on DELIVER, MUSES, and MCubeS across various modality combinations, including missing-modality and noise-robustness settings, and the authors report consistent mIoU improvements over CMNeXt, CWSAM, and a self-built SAM-LoRA baseline.","tokens_in":26311,"tokens_out":3988,"duration_ms":39576,"significance":"If the reported gains survive a controlled comparison, the paper would make a useful contribution: it demonstrates a parameter-efficient way to adapt a large frozen foundation model to heterogeneous sensor modalities, with a concrete routing mechanism and a dual-head decoder. The experiments cover three benchmarks and several adverse-condition settings, and the method description is sufficiently detailed to be reproduced. The main value is therefore conditional on resolving the backbone-capability confound described below and on providing a same-backbone control that isolates the proposed routing and decoder components.","major_comments":[{"comment":"The headline comparisons are not controlled for backbone. The paper states in Section IV-A that CMNeXt uses MiT-B0, CWSAM and SAM-LoRA use ViT-B, and MLE-SAM uses Hiera-B+. Because Hiera-B+ is a different, likely stronger encoder than ViT-B, the reported gaps (e.g., 3.39 mIoU over SAM-LoRA on RGB-only in Table I, 8.04 on Frame-only in Table II, and 0.43 on MCubeS RGB-A-D-NIR in Table III) may be due to backbone capacity rather than to the proposed MoE-LoRA or auxiliary head. A same-backbone comparison is required, for instance MLE-SAM with a ViT-B encoder or SAM-LoRA with a Hiera-B+ encoder, before the claim that MLE-SAM 'significantly outperforms' prior methods is supported.","section":"Section IV-A, Tables I-III"},{"comment":"The ablation does not isolate the contribution of the MoE router against a plain per-modality LoRA adaptation on the same backbone. Table V varies integrated features, weighted features, and the auxiliary head, but it does not include a baseline consisting of the frozen Hiera-B+ encoder with per-modality LoRA and simple mask averaging, without the modified SAM2 mask decoder described in Eq. (11). The first row of Table V still uses the modified decoder and the integrated-feature averaging of Eq. (7), so it is not a standard SAM2-LoRA control. Without this control, the specific benefit of the routing mechanism in Eqs. (8)-(10) is not separated from the backbone and decoder changes.","section":"Section IV-C, Table V"},{"comment":"The abstract's 32.15% missing-modality improvement is misleading because it is measured against CMNeXt, which the paper itself states depends on the Frame/RGB modality and drops to 3.50 mIoU when tested on Frame-only after F-L-E training (Table VII). The gain over the more appropriate same-family baseline SAM-LoRA on the same configuration is 4.99 points, not 32.15. The robustness claim should be reframed relative to baselines that also support training without RGB, and the current wording overstates the magnitude of the benefit.","section":"Section IV-D, Table VII"},{"comment":"No repeated runs or error bars are reported anywhere. Some margins are small (0.43 mIoU on MCubeS RGB-A-D-NIR in Table III, 0.77 mean mIoU over SAM-LoRA on R-D-E-L in Table VI), and without at least three seeds with mean and standard deviation, it is impossible to determine whether these differences are statistically meaningful. The paper should add seed-variability reporting or explicitly discuss the magnitude of run-to-run variance in these benchmarks.","section":"Tables I-VIII"}],"minor_comments":[{"comment":"Several references appear unrelated to the cited claims: ref. [55] is titled 'Impact of the hubble tension on the r-n s contour', ref. [58] is about anomaly detection for sequential decision sequences, ref. [59] is about orbital dynamics identification, and ref. [60] is about autoformalisation in Lean. These seem to be placeholder or mismatched entries and should be corrected.","section":"References"},{"comment":"The notation in Eq. (10) is ambiguous: 'Top-k(w_1^i,...,w_M^i) ⊙ Y_i^m' suggests that the same top-k mask multiplies every modality feature, but the intent is presumably to zero out or reweight only the selected modalities. Please clarify how the top-k selection is applied across the modality index m.","section":"Eq. (10)"},{"comment":"The loss weights w0 and w1 in Eq. (14) are not listed in the training configuration table. Please specify their values or state that they were set to 1.0.","section":"Appendix A, Table IX"},{"comment":"The statement that this is 'the first attempt to adapt SAM for multi-modal semantic segmentation' is not consistent with the paper's own citation of SAM-Event-Adapter (ref. [50]), which adapts SAM to event-RGB semantic segmentation. Please soften the novelty claim to account for prior adaptation work.","section":"Introduction, Section II-B"},{"comment":"The qualitative figures are difficult to read at the printed resolution; consider enlarging the crops and marking the semantic classes clearly, since the text refers to specific objects such as pedestrians and bus stations.","section":"Figures 5-7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically interesting but the experimental validation as written does not support the central SOTA claim because of the backbone mismatch and the absence of a plain same-backbone LoRA control. The issues are fixable with additional experiments, so I recommend major revision rather than rejection. I would also flag to the editor that the reference list contains several evidently unrelated entries, which suggests the manuscript may have been assembled from a template and not carefully proofread; this should be corrected before any production stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the architecture is sensible and the paper is readable, but the evaluation design makes the headline numbers uninterpretable. All baselines run on smaller backbones (MiT-B0, ViT-B) while MLE-SAM uses Hiera-B+ from SAM2. The paper calls this 'standardized' in Section IV-A, which it is not. The gains in Tables I-III and the 32.15% missing-modality claim are all compatible with a backbone-capacity explanation.\n\nWhat is actually new: per-modality LoRA experts inside a frozen SAM2 encoder with a softmax top-k router over spatially averaged features, plus a dual-head (adapted SAM decoder + auxiliary MLP head). I don't know of that exact combination in the multi-modal segmentation literature, and the method section is clear. The experiments are broad: three datasets, many modality combos, missing-modality testing, noise stress. The trainable parameter count (20.79M) is respectable.\n\nThe soft spots are serious. The central comparison lacks a same-backbone control. The ablation in Table V isolates the router and auxiliary head on top of the same Hiera-B+ base, but never against a plain SAM2-LoRA with simple averaging; so the router's contribution is not separated from the base encoder. On MCubeS the margins over CWSAM are 0.43-1.87 mIoU, within noise. There are no error bars or repeated runs. The appendix gives LoRA rank and learning rates but not the top-k value k or the loss weights w0, w1. The related-work section has a citation problem: refs [55], [58], [59], and [60] are unrelated to routing or LoRA (one is about the Hubble tension, another about Koopman theory). That needs cleaning, not necessarily an integrity charge, but it weakens trust. The 'first attempt' claim is also overstated given the existence of e.g. SAM-Event-Adapter (ref [50]).\n\nBottom line: this is a plausible recipe and a reasonable empirical study once the backbones are matched. The current version does not support the SOTA or robustness claims as written. I would send it to peer review with the expectation of major revisions, not desk-reject it: the method is coherent and the problem is timely.\n\nWho is this for? Researchers adapting SAM/SAM2 to multi-sensor driving or robotics perception. They can borrow the LoRA-fusion recipe, but they should not cite the mIoU numbers as evidence until a controlled comparison exists.","headline":"Plausible LoRA-MoE recipe for adapting SAM2 to multi-modal segmentation, but every headline comparison is tainted by a backbone mismatch, so the SOTA claim is unproven.","tokens_in":26922,"tokens_out":4169,"would_cite":false,"duration_ms":39296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By freezing SAM2's image encoder and training only per-modality LoRA experts with a top-k routing layer, the paper claims MLE-SAM reaches state-of-the-art multi-modal semantic segmentation and gains up to 32.15% over prior methods when…","keywords":["multi-modal semantic segmentation","Segment Anything Model","parameter-efficient fine-tuning","low-rank adaptation (LoRA)","mixture of experts","feature fusion","missing modality robustness","autonomous driving"],"falsifier":"Run CMNeXt, CWSAM, and SAM-LoRA with the same Hiera-B+ backbone and the same 1024x1024 training recipe on DELIVER, MUSES, and MCubeS; if their mIoU rises to within about one point of MLE-SAM, the claimed gains come mostly from the backbone, not the MoE-LoRA routing.","tokens_in":25747,"feed_emoji":"🚗","tokens_out":7049,"duration_ms":61019,"temperature":0.7,"pith_summary":"This paper tries to establish that the Segment Anything Model's frozen image encoder can be adapted to multi-modal semantic segmentation (RGB, depth, LiDAR, event, and polarization) by training only small low-rank LoRA experts plus a routing layer. The authors propose MLE-SAM, which assigns each input modality its own LoRA updates inside SAM2's attention while keeping the pre-trained weights frozen, then blends the resulting features with a top-k mixture-of-experts weighting and a dual-pathway mask decoder. They test it on DELIVER, MUSES, and MCubeS and report that it outperforms prior multi-modal segmenters, with particularly large gains when modalities are missing or noisy, such as 32.15% improvement on MUSES under missing-modality testing. If true, this would let foundation-model segmentation be reused across sensor suites without full fine-tuning.","feed_headline":"LoRA expert mix lets frozen SAM2 segment any multi-sensor input","feed_subtitle":"Training only ~20.8M extra parameters, MLE-SAM tops DELIVER, MUSES, and MCubeS and survives missing sensors.","key_machinery":"The mechanism is a Mixture of LoRA Experts (MoE-LoRA) layered on the frozen SAM2 image encoder. Each modality gets its own low-rank update to the attention query and value projections (Eqs. 4-5), so a tiny parameter set adapts the backbone to that sensor. The FPN-style neck produces three feature pyramids per modality: the semantic feature map, the fine-grained pyramid, and the intermediate-resolution pyramid, which are averaged across modalities to form $\\mathbf{Y}$, then re-weighted by a softmax top-k router that selects the most relevant modality features at each scale to form $\\hat{\\mathbf{Y}}$. The two streams are combined and fed into a dual-pathway decoder: SAM2's augmented mask decoder refines logits hierarchically, while an auxiliary MLP-based head fuses all scales; both heads are supervised with OhemCrossEntropy (Eq. 14).","core_discovery":"The central claim is that SAM2, a model trained for RGB image and video segmentation, can be converted into a strong multi-modal semantic segmenter without updating its weights. The paper argues that inserting modality-specific LoRA matrices into the query and value projections of SAM2's Hiera backbone, together with an equal-weight averaged feature stream, a top-k routing that re-weights the most informative modality features, and an auxiliary segmentation head that fuses multi-scale features, jointly deliver state-of-the-art mIoU on three benchmarks. Evidence includes mIoU of 64.08% on DELIVER with RGB-Depth-Event-LiDAR, 75.42% on MUSES with Frame-LiDAR, and 51.02% on MCubeS with all four material-sensing modalities. The robustness experiments, in which models are trained on all modalities but tested with subsets or with injected Gaussian/random noise, are the basis for the reported 14.13% and 32.15% gains over existing methods.","pith_inferences":["Going beyond the paper: the comparisons give MLE-SAM a larger backbone (Hiera-B+) than the baselines (MiT-B0, ViT-B), so the cleanest test of the MoE-LoRA contribution is a same-backbone rerun; the MCubeS margin over CWSAM is only 0.43%, suggesting the advantage may shrink considerably.","Going beyond the paper: the routing weights are trained only on the segmentation loss of the full model, so an untested prediction is that the router's top-k choices should align with sensor quality; one could verify by correlating router weights with per-modality noise levels.","Going beyond the paper: the near-catastrophic sparse-modality test scores suggest the model learns RGB/depth priors rather than truly sensor-agnostic features; training with random modality dropout would test whether the reported missing-modality robustness comes from the router or from dense-modality dominance.","Going beyond the paper: since only a small parameter set is trained, the same frozen-backbone plus MoE-LoRA recipe should transfer to other foundation encoders or to video/3D perception, provided the same multi-scale fusion heads are added."],"forward_implications":["Training only LoRA layers and routing (about 20.8M parameters for four modalities) means a foundation model can serve new sensor configurations without full fine-tuning.","If the reported numbers hold, missing or degraded sensors become far less catastrophic: the framework claims 14.13% higher mean mIoU on DELIVER and 32.15% higher on MUSES than prior methods in partial-input tests.","The fusion recipe is additive: combining the uniform-average features, the top-k weighted features, and the auxiliary head raises DELIVER mIoU from 58.35% (weighted-only) to 64.08% with the same full model.","Dense modalities (RGB, depth, frame) carry most of the performance, while sparse modalities (event, LiDAR) mainly help in adverse conditions; the paper's own tables show sparse-only inputs score near 1-5% mIoU.","Because SAM2's weights stay frozen, the approach is compatible with the model's continued use for its original zero-shot segmentation role after the LoRA experts are removed."],"supporting_citations":[{"why":"Supplies the base Segment Anything Model whose frozen weights and generalization the paper builds on.","marker":"[3]"},{"why":"Introduces SAM2, the specific frozen encoder and mask decoder that MLE-SAM customizes.","marker":"[8]"},{"why":"Provides the MUSES multi-sensor dataset and prior baselines used for missing-modality and noise robustness comparisons.","marker":"[10]"},{"why":"Provides the DELIVER dataset and the CMNeXt model that is the main state-of-the-art baseline to beat.","marker":"[13]"},{"why":"Provides the MCubeS material-segmentation benchmark with RGB, NIR, and polarization modalities.","marker":"[14]"},{"why":"CWSAM is the principal SAM-based semantic-segmentation baseline and the prior adapter design the paper contrasts with.","marker":"[48]"},{"why":"LoRA is the parameter-efficient fine-tuning technique that the per-modality experts are built from.","marker":"[52]"},{"why":"Hiera is the hierarchical backbone inside SAM2 that the LoRA layers modify.","marker":"[62]"}],"fun_headline_variants":["Frozen SAM2 + MoE-LoRA masters multi-sensor segmentation","Multi-modal SAM via mixture of LoRA experts","LoRA experts adapt SAM to depth, event, LiDAR","SAM2 goes multi-modal with top-k LoRA routing","Missing sensor? MoE-LoRA SAM still leads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that comparing MLE-SAM (with the larger Hiera-B+ backbone) against baselines with smaller backbones (MiT-B0 and ViT-B) is a fair test; if the baselines were given the same backbone, the reported mIoU advantages, some as small as 0.43%, could shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Frozen SAM2 + MoE-LoRA masters multi-sensor segmentation","Multi-modal SAM via mixture of LoRA experts","LoRA experts adapt SAM to depth, event, LiDAR","SAM2 goes multi-modal with top-k LoRA routing","Missing sensor? MoE-LoRA SAM still leads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":3126,"prompt_tokens":1001,"completion_tokens":2125,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":2041}},"tokens_in":617,"tokens_out":2125,"duration_ms":17012,"temperature":1.0,"reasoning_tokens":2041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:38:26.233381+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CMNeXt, CWSAM, and SAM-LoRA with the same Hiera-B+ backbone and the same 1024x1024 training recipe on DELIVER, MUSES, and MCubeS; if their mIoU rises to within about one point of MLE-SAM, the claimed gains come mostly from the backbone, not the MoE-LoRA routing.","supporting_citations":[],"review_version":1}