{"id":"c0952ef5-82f1-4b21-9363-9a4ce6dfeb39","arxiv_id":"2505.01950","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SARTM fine-tunes SAM2 with LoRA and distills CLIP text knowledge to improve RGB-thermal semantic segmentation, reporting top mIoU on PST900, MFNet, and FMB.","lead":"This paper adapts the Segment Anything Model 2 to RGB-thermal semantic segmentation using LoRA fine-tuning and CLIP-based language distillation, reporting strong mIoU on three benchmarks. The main value for a generalist is as an incremental engineering recipe, but the experimental tables contain inconsistencies that need correction.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table III's SARTM row on FMB is identical to SegMiF's per-class IoUs while claiming a higher mIoU; without corrected or reproducible data, the FMB SOTA claim is unsupported.","rationale":"The reader's verdict is CONDITIONAL, and my concern does not move it: the paper still needs corrected tables, released code, and direct comparisons before the central claim can be accepted. However, I identify a different load-bearing weak point than the reader's stated weakest assumption. The reader focused on whether the language-aided distillation mechanism is semantically meaningful, which is an important novelty and reproducibility concern. But even if that mechanism were replaced by ordinary regularization, the empirical SOTA claim could in principle still hold. The FMB table duplication is more directly load-bearing because it undermines one of the three benchmark pillars used to support the headline 'significantly outperforms state-of-the-art approaches across a variety of conditions.' If Table III is unreliable, the breadth of the central claim shrinks to two datasets. I also note the ill-defined Eq. 13 and the unspecified normalization in Eq. 14 as compounding reproducibility problems: the method as written cannot be reimplemented, so the ablation evidence for the language mechanism (Table VI) cannot be checked. I credit the paper for reporting a plausible PST900 improvement over HAPNet and for including ablations, but no code, checkpoints, or error bars are provided, so independent verification is currently impossible. My agreement with the reader is partial because the reader did explicitly flag the FMB duplication in the rationale even though it was not selected as the weakest assumption.","tokens_in":18542,"tokens_out":6744,"duration_ms":71689,"concrete_test":"Obtain the SARTM and SegMiF predictions or released checkpoints and recompute all 15 per-class IoUs on the FMB test set. Verify whether SARTM's reported eight class IoUs and mIoU of 61.57 are reproducible from actual predictions, and whether SegMiF's row is genuinely different. If the rows remain identical, the table contains a transcription error and must be corrected. Also inspect the training code for Eq. 13 to determine what loss term L_cr actually computes and whether it matches the text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SARTM significantly outperforms SOTA across RGB-T benchmarks. Table III is the sole quantitative evidence for FMB, and its SARTM row lists per-class IoUs (78.3, 65.4, 47.3, 43.1, 74.8, 82.0, 85.0, 49.8 for Car, Person, Truck, T-Lamp, T-Sign, Building, Vegetation, Pole) that are identical to the SegMiF row, while the mIoU differs by 6.77 points (61.57 vs 54.8). If the table reports the full class set, the printed mIoU cannot be derived from the printed per-class values: the mean of the eight listed values is 65.71, not 61.57. If seven of FMB's fifteen classes are omitted, the table must state that, and the exact duplication of all eight reported class IoUs across two independent methods remains unexplained and effectively impossible for genuinely independent results. This makes the FMB contribution to the headline claim unreliable: either the per-class values, the mIoU, or the row assignment is wrong. A secondary but related reproducibility concern is Eq. 13, L_cr = KL(f, y), where f is a visual feature tensor and y is ground-truth labels; KL divergence is not defined between these objects, so the stated loss cannot be implemented as written, and Eq. 14 applies KL to cosine-similarity matrices without the normalization needed to treat them as distributions. These issues do not by themselves disprove the PST900 or MFNet results, but they mean the claimed cross-benchmark superiority is not currently supported by trustworthy published evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SARTM, a SAM2-based framework for RGB-thermal semantic segmentation. The SAM2 image encoder is frozen and adapted with modality-specific LoRA layers; three feature hierarchies (SFM, FFP, IFP) are fused; a dual-path decoder with an auxiliary segmentation head produces predictions; and CLIP text embeddings of class names are used in a cross-modal knowledge distillation loss. Experiments on PST900, MFNet, and FMB report state-of-the-art mIoUs of 89.88, 60.03, and 61.57, respectively, and the paper claims significant improvement over prior RGB-T methods.","tokens_in":18901,"tokens_out":3582,"duration_ms":35742,"significance":"If the reported results are correct, the paper offers a practical and timely recipe for adapting SAM2 to multimodal semantic segmentation through parameter-efficient LoRA fine-tuning, and the idea of injecting CLIP-derived language guidance into RGB-T feature fusion is worth investigating. The manuscript includes useful ablations of LoRA rank, loss weights, and component removal, as well as t-SNE and similarity-matrix visualizations. However, the central cross-benchmark claim is not currently trustworthy: the FMB table contains an apparent numerical inconsistency and an unexplained duplication of another method's per-class scores, the MFNet text contradicts its own table, and the main language-aided distillation loss is not formally defined. These issues are load-bearing because the headline claim rests on all three benchmark results and on the language-guidance mechanism.","major_comments":[{"comment":"Table III reports per-class IoUs for only eight classes, while the FMB dataset has 15 categories. The SARTM row lists per-class values (78.3, 65.4, 47.3, 43.1, 74.8, 82.0, 85.0, 49.8) that are identical to the SegMiF row, yet SARTM's mIoU is 61.57 while SegMiF's is 54.8. The mean of the eight listed values is 65.71, not 61.57. If the table intentionally shows only a subset of classes, that must be stated and the reported mIoU must be derived from a defined class set; if the table shows the full set, the mIoU is arithmetically wrong. In either case, identical per-class numbers for two independently developed methods cannot be accepted without explanation. The FMB portion of the abstract's 'significantly outperforms' claim is therefore unsupported as printed.","section":"Table III"},{"comment":"The text states that on the MFNet dataset 'our method slightly underperforms compared to the best-performing method in terms of mIoU,' but Table II shows SARTM with the highest mIoU (60.03) against CMX (59.7) and all other listed methods. This direct contradiction between the text and the table should be resolved, since the paper's characterization of its own results is part of the evidence for the claimed cross-benchmark superiority.","section":"Section IV.B, Table II"},{"comment":"The language-aided distillation loss is not well-defined as written. Eq. (13) sets Lcr = KL(f, y), where f is a feature tensor and y is ground-truth labels; KL divergence is not defined between such objects, so this term cannot be implemented as stated. Eq. (14) applies KL to cosine-similarity matrices without converting them into normalized probability distributions, and the dimensions of Te and the feature set {f0,...,fK} are not specified. Since this loss is the entire mechanism for the claimed language guidance, the ablation in Table V cannot be interpreted until the loss is properly defined.","section":"Section III.E, Eqs. (13)-(14)"}],"minor_comments":[{"comment":"The heading 'Ovearll Training Objectives' contains a typo; it should be 'Overall Training Objectives.'","section":"Section III.F"},{"comment":"The text says LoRA ranks 2, 4, 16, 32, and 64 were tested, but the table lists 4, 8, 16, 32, and 64, with rank 2 missing and rank 8 appearing instead. The text and table should be aligned.","section":"Table IV"},{"comment":"The description of the second pathway contains garbled and duplicated phrases, e.g., 'ierarchical feature integration framework that aggregates multi-scale features into a unified high-resolution embedding' and 'Finlly,fpn merge.' The paragraph needs careful rewriting for clarity.","section":"Section III.D"},{"comment":"The qualitative figure captions are inconsistently matched to the text: for example, the text describing Figure 5 mentions CMNext and CAINet while the caption references CAINet and MMSFormer, and the text in Section IV.C has typos such as 'ocjects' and 'metods.'","section":"Figures 4-6"},{"comment":"The caption contains repeated typos: 'Epoach' should be 'Epoch.'","section":"Figure 8"},{"comment":"The loss-weight ablation reports values for w1, w2, and w3, but the mapping of these symbols to the weights in Eq. (15) is not stated, and the table does not specify how w0 is set. The optimal mIoU of 60.03 is reported with w3 = 100, but the table's rows do not show a single joint configuration, so the reader cannot verify how the combination was obtained.","section":"Table VII"}],"recommendation":"major_revision","confidential_remarks":"The Table III duplication of SegMiF's per-class numbers with a different mIoU is the most serious issue and should be resolved before the paper is considered further. If the FMB numbers cannot be reproduced from training logs or a corrected protocol, the FMB claim should be removed or substantially weakened. The same applies to the MFNet text/table contradiction. I would also recommend that the authors be asked to provide code or detailed implementation of Eqs. (13)-(14), since the method's novelty depends on a loss that is currently undefined."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: SARTM has a plausible PST900 story, but the FMB table is not credible, and the paper overclaims novelty relative to its own references.\n\nWhat's actually new: the combination of LoRA-fine-tuned SAM2 with an FPN decoder, an auxiliary head, and a CLIP-based self-similarity distillation for RGB-T segmentation. The PST900 result (89.88 mIoU, +0.88 over HAPNet) is strong, and the ablations on LoRA rank, loss weights, and components are more thorough than you usually see. If the PST900 numbers reproduce, that is a useful engineering result.\n\nThe soft spots are real. First, Table III on FMB appears to have a duplicated row: SARTM's per-class IoUs are identical to SegMiF's (78.3, 65.4, 47.3, 43.1, 74.8, 82.0, 85.0, 49.8), yet the mIoU is listed as 61.57 versus SegMiF's 54.8. The mean of those eight values is 65.71, not 61.57. Either the table is wrong, the mIoU is wrong, or something is seriously off. This is load-bearing, because the paper claims SOTA on FMB, and this table is the sole evidence. Second, the MFNet text says the method \"slightly underperforms\" the best method, but Table II shows SARTM with the highest mIoU (60.03 vs CMX's 59.7). That's a text-table contradiction, minor but careless. Third, Eq. 13 defines L_cr = KL(f,y) with f a feature tensor and y ground-truth labels; KL is not defined between those objects. Eq. 14 runs into the same issue unless the similarity matrices are properly normalized. Finally, the paper claims to be the first to apply SAM/SAM2 to RGB-T with language guidance, while citing refs [31] and [32] that already do this. It does not compare against them.\n\nNone of this disproves the PST900 or MFNet results, but it means the cross-benchmark superiority claim is not supported by trustworthy published evidence. The paper needs corrected tables, a proper definition of the distillation loss, and direct comparison with the most similar cited works.\n\nWho is this for: people working on RGB-T segmentation or on adapting foundation models to new modalities. It deserves peer review only after the authors clean up the tables and equations. I'd send it back for major revision, or desk-reject the current version with an invitation to resubmit once the data is fixed.\n\nBest.","headline":"A promising PST900 recipe, undermined by an FMB table that duplicates SegMiF's per-class numbers with an incompatible mIoU, and an overclaim of novelty relative to its own references.","tokens_in":19462,"tokens_out":3229,"would_cite":false,"duration_ms":28989,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims SARTM, a SAM2-based RGB-thermal segmentation framework, sets new state-of-the-art results on three benchmarks by combining LoRA fine-tuning with CLIP-guided knowledge distillation.","keywords":["RGB-thermal semantic segmentation","SAM2","Segment Anything Model","LoRA","CLIP","knowledge distillation","cross-modal fusion","multi-scale feature fusion"],"falsifier":"Train SARTM on PST900 and FMB with the CLIP text embeddings replaced by random fixed vectors of the same dimension, keeping every other component and loss weight unchanged; if the mIoU drop is the same as the reported drop when the language loss is removed, about 3.6 and 2.7 points, then the language content itself is not doing the work. A second check is to inspect the implementation of Equation 13: as written it compares pooled visual features to ground-truth labels, so if that is literally what is computed, the language term is partly label supervision rather than distillation from text.","tokens_in":18332,"feed_emoji":"🌡️","tokens_out":8345,"duration_ms":76383,"temperature":0.7,"pith_summary":"SARTM is a framework for RGB-thermal semantic segmentation built on SAM2. It freezes SAM2's image encoder, adds modality-specific LoRA layers so the backbone can adapt to thermal input without losing its pretrained knowledge, and adds a dual-path decoder with an auxiliary segmentation head for multi-scale fusion. The paper's central claim is that language helps: CLIP text embeddings of class names are distilled into the visual features by aligning their self-similarity matrices, which the authors say resolves cross-modal ambiguity and improves category discrimination. On the three standard benchmarks the framework reports state-of-the-art mean intersection-over-union scores of 89.88 on PST900, 60.03 on MFNet, and 61.57 on FMB. If true, this would matter because RGB-thermal segmentation is a practical route to reliable scene understanding in low-light, foggy, and overexposed conditions, and adapting a general segmentation foundation model with small trainable components would be cheaper than training a specialised network from scratch.","feed_headline":"SAM2 tops three RGB-thermal benchmarks with language guidance","feed_subtitle":"SARTM distills CLIP class-name knowledge into SAM2, reaching 89.88 mIoU on PST900 and 61.57 on FMB","key_machinery":"The load-bearing object is the Cross-Modal Knowledge Distillation (CMKD) module. It performs Mask Average Pooling on the fused features, using ground-truth labels as class masks, to obtain one vector per category; then it computes the cosine self-similarity matrix of those vectors and of CLIP text embeddings for the class names, and minimizes the KL divergence between the two matrices. This relational transfer is what the paper says injects semantic understanding into the fused RGB-thermal representation. The rest of the machinery supports it: modality-specific LoRA layers adapt the frozen SAM2 Hiera encoder, and a dual-path decoder combines the semantic feature map, fine-grained feature pyramid, and intermediate-resolution feature pyramid through SAM2's mask decoder plus an auxiliary FPN-style head.","core_discovery":"On the paper's own terms, the discovery is that a frozen SAM2 can be turned into a top RGB-thermal semantic segmenter by three coordinated changes: low-rank (LoRA) adaptation of the image encoder per modality, a redesigned dual-path segmentation head that fuses semantic, fine-grained, and intermediate-resolution feature pyramids, and a Cross-Modal Knowledge Distillation module that uses CLIP's text encoder as a teacher. The teacher's class-name embeddings are compared with mask-pooled visual features through their cosine self-similarity matrices, and a KL divergence (Equation 14) pushes the student to reproduce the semantic relations among categories. The authors report that this combination achieves the best published mIoU on PST900 and FMB and competitive state-of-the-art on MFNet, with the language term contributing about 3.6 and 2.7 mIoU points on PST900 and FMB respectively in the ablation.","pith_inferences":["Inference: the language mechanism is not cleanly isolated by the paper, since the only evidence is an ablation that removes the loss; a reviewer-friendly experiment would replace CLIP embeddings with random fixed vectors of the same dimension and check whether the mIoU drop persists.","Inference: the self-similarity alignment in Equation 14 is essentially a class-relation regularizer, so if it works, a purely visual prototype-consistency loss without any text encoder should reproduce most of the gain, which would be cheaper and work for class names unseen by CLIP.","Inference: the approach as described uses only static class-name embeddings, not richer language; extending the teacher to attribute phrases or scene captions is a natural test of whether semantic language content, rather than label statistics, is what helps."],"forward_implications":["SAM2 can be reused for RGB-thermal segmentation with only LoRA matrices and segmentation heads trainable, keeping the frozen image encoder, so deployment cost stays well below training a specialised RGB-T network from scratch.","The language-guided relational distillation should improve category separation specifically in low-light and overexposed scenes, where RGB and thermal evidence conflict, matching the PST900 qualitative results.","Because the LoRA layers are modality-specific, the same architecture should transfer to other RGB-X inputs such as depth, event streams, or near-infrared with little modification.","The reported state-of-the-art results on PST900 and FMB, and the narrow gap on MFNet, imply that foundation-model adaptation is now a competitive recipe for RGB-T semantic segmentation."],"supporting_citations":[{"why":"Supplies the SAM2 image encoder, mask decoder, and frozen-backbone architecture that SARTM adapts.","marker":"[14]"},{"why":"Defines the Segment Anything Model whose strong RGB segmentation is the starting point the paper claims to preserve.","marker":"[13]"},{"why":"Provides the CLIP text encoder that generates class-name embeddings used as the language teacher in distillation.","marker":"[22]"},{"why":"Defines the MFNet dataset and the dual-stream baseline that SARTM compares against.","marker":"[6]"},{"why":"Provides the PST900 dataset and the two-stage segmentation baseline.","marker":"[8]"},{"why":"Provides the FMB dataset and the fusion/segmentation benchmark used in experiments.","marker":"[7]"},{"why":"HAPNet, the strongest prior PST900 baseline whose 89.0 mIoU SARTM improves to 89.88.","marker":"[68]"},{"why":"MRFS, a strong prior baseline on PST900 and the strongest listed FMB baseline at 61.1 mIoU.","marker":"[30]"},{"why":"CMX, the strongest listed MFNet baseline at 59.7 mIoU that SARTM edges out with 60.03.","marker":"[71]"}],"fun_headline_variants":["Language distillation turns SAM2 into RGB-T SOTA","SAM2 with CLIP text sweeps RGB-T benchmarks","LoRA and CLIP adapt SAM2 to RGB-thermal segmentation","Language guidance lifts SAM2 to top RGB-thermal results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's central premise is that CLIP's text embeddings for class names, when aligned to mask-pooled visual features through the similarity-matching loss, actually deliver semantic knowledge that resolves RGB-thermal ambiguity; the evidence for this is an ablation that only turns the language loss off, so the premise is asserted rather than demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Language distillation turns SAM2 into RGB-T SOTA","SAM2 with CLIP text sweeps RGB-T benchmarks","LoRA and CLIP adapt SAM2 to RGB-thermal segmentation","Language guidance lifts SAM2 to top RGB-thermal results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002162,"raw_usage":{"total_tokens":8415,"prompt_tokens":1011,"completion_tokens":7404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":7336}},"tokens_in":627,"tokens_out":7404,"duration_ms":59650,"temperature":1.0,"reasoning_tokens":7336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:05:13.823110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SARTM on PST900 and FMB with the CLIP text embeddings replaced by random fixed vectors of the same dimension, keeping every other component and loss weight unchanged; if the mIoU drop is the same as the reported drop when the language loss is removed, about 3.6 and 2.7 points, then the language content itself is not doing the work. A second check is to inspect the implementation of Equation 13: as written it compares pooled visual features to ground-truth labels, so if that is literally what is computed, the language term is partly label supervision rather than distillation from text.","supporting_citations":[{"cited_title":"Segment anything,","cited_arxiv_id":null,"evidence_quote":"Defines the Segment Anything Model whose strong RGB segmentation is the starting point the paper claims to preserve."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP text encoder that generates class-name embeddings used as the language teacher in distillation."},{"cited_title":"Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,","cited_arxiv_id":null,"evidence_quote":"Defines the MFNet dataset and the dual-stream baseline that SARTM compares against."},{"cited_title":"Pst900: Rgb-thermal calibration, dataset and segmentation network,","cited_arxiv_id":null,"evidence_quote":"Provides the PST900 dataset and the two-stage segmentation baseline."},{"cited_title":"Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the FMB dataset and the fusion/segmentation benchmark used in experiments."},{"cited_title":"Mrfs: Mutually rein- forcing image fusion and segmentation,","cited_arxiv_id":null,"evidence_quote":"MRFS, a strong prior baseline on PST900 and the strongest listed FMB baseline at 61.1 mIoU."},{"cited_title":"Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers,","cited_arxiv_id":null,"evidence_quote":"CMX, the strongest listed MFNet baseline at 59.7 mIoU that SARTM edges out with 60.03."}],"review_version":1}