{"id":"e66d8441-d14f-47a6-a8bc-fab140365105","arxiv_id":"2506.17869","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CM-SSM reports 74.6% mIoU on the CART RGB-thermal benchmark and 85.9% on PST900, using 12.59M parameters and running at 114 FPS on an RTX 4090.","lead":"This paper describes a lightweight neural network that combines color and thermal camera images to label outdoor scenes in real time for field robots. It builds on state space models, which scan pixels efficiently, and reports accuracy close to larger Transformer-based systems at a fraction of the compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II's PST900 mIoU (85.9%) is inconsistent with the average of the four listed per-class IoUs (82.4%), undermining the claimed state-of-the-art on PST900 as reported.","rationale":"The reader's verdict is CONDITIONAL and its rationale already lists the PST900 metric discrepancy among the issues to fix. I agree that the CART split-comparability question is a real concern, but I judge the PST900 mIoU inconsistency to be the single most load-bearing problem because it is an internal arithmetic contradiction directly affecting a headline result. The CART split issue is an unverified comparability assumption; the PST900 discrepancy is visible in the paper itself. If the four listed per-class IoUs are correct, then the reported 85.9% mIoU is not the class-average mIoU as defined, and the claimed best performance on PST900 collapses or requires an undocumented metric change. This does not overturn the paper's CART result, which is internally consistent (the ten CART per-class IoUs average to 74.6%), so the overall verdict remains CONDITIONAL: the paper should be accepted only after the authors correct the PST900 mIoU, clarify the evaluation metric, and ideally verify the CART split comparability. The Eq. (10) inconsistency (Fi = CBR1x1(Cat(Ri, Ti, L)) omits GR and GT despite the text saying they are combined) is also worth fixing, but it is a reproducibility typo rather than a direct challenge to the empirical claim. My concern aligns with the reader's overall conditional posture, but with a different primary emphasis.","tokens_in":10556,"tokens_out":6117,"duration_ms":64828,"concrete_test":"Recompute the class-average mIoU from Table II for CM-SSM: (80.5 + 85.4 + 85.7 + 78.1) / 4 = 82.4%. If the reported 85.9% cannot be reproduced under the stated mIoU definition, request the authors' exact evaluation script or metric definition (including whether background is included and whether classes are weighted) and recompute all Table II entries under that same definition; if the corrected value drops below MDNet's 83.0%, the PST900 state-of-the-art claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim includes a state-of-the-art 85.9% mIoU on PST900 (Sec. IV-C2, Table II, abstract). This number is not supported by the paper's own table: the four per-class IoUs for CM-SSM in Table II are 80.5, 85.4, 85.7, and 78.1, whose unweighted mean is 82.4%, not 85.9%. The paper defines mIoU as mean intersection over union (Sec. IV-A) with no mention of frequency weighting or inclusion of a background class. If the standard unweighted four-class average is intended, the reported mIoU is arithmetically wrong; if a different definition is intended (e.g., a five-class mean with background), the table is incomplete and the comparison with all baselines in Table II is not on a consistent footing. Since PST900 is the evidence for the paper's generalizability claim, the discrepancy is load-bearing: under the four-class average, CM-SSM's 82.4% would rank below MDNet's 83.0% in Table II, so the claimed 'best performance' on PST900 would not hold. This is an internal inconsistency, not a matter of external consensus, and it must be resolved before the central accuracy claims can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CM-SSM, a real-time RGB-thermal semantic segmentation architecture based on cross-modal state space modeling. The method consists of two modules: CM-SS2D, which builds a cross-modal visual sequence by interleaving RGB and thermal features and computes each modality's hidden state from the other modality's previous state, and CM-SSA, which combines the resulting global associations with local convolutional features. The authors report state-of-the-art results on CART (74.6% mIoU) and PST900 (85.9% mIoU) with 12.59M parameters and 114 FPS on an RTX 4090, and include ablation studies supporting the contribution of CM-SS2D and CM-SSA. The paper also provides code publicly.","tokens_in":10801,"tokens_out":4133,"duration_ms":44837,"significance":"If the reported results are correct, the paper offers a useful practical contribution: an efficient, linear-complexity alternative to Transformer-based cross-modal fusion for resource-constrained field robots. The architecture is plausible, and the ablation study compares against several fusion strategies, which is a strength. The public code and reproducible experimental setup are also positive. However, the central empirical claims are currently undermined by arithmetic inconsistencies in the PST900 table, an apparent mismatch between the text and Eq. (10), and ambiguity about whether reported CART splits match those of the cited baselines. These issues must be resolved before the state-of-the-art claims can be accepted.","major_comments":[{"comment":"Eq. (10) states that F_i = CBR_1x1(Cat(R_i, T_i, L)), but the surrounding text says that the module combines G_R, G_T, and L. As written, the equation drops the global association outputs G_R and G_T entirely, meaning the CM-SS2D branch would not contribute to the final fused feature. This contradicts the ablation study in Table IV, where removing CM-SS2D decreases mIoU by 1.3%. Please correct the equation or the text, and specify which formulation is actually implemented.","section":"§III-C, Eq. (10)"},{"comment":"The reported PST900 mIoU of 85.9% is inconsistent with the four per-class IoUs listed for CM-SSM in Table II (80.5, 85.4, 85.7, 78.1). The unweighted mean of these four values is 82.4%. Since Sec. IV-A defines mIoU as the mean intersection over union and does not mention frequency weighting or an additional background class, the 85.9% figure is arithmetically unsupported. If a different averaging scheme is used, the table and metric definition must be clarified, and all baseline comparisons must use the same scheme. Under the standard four-class mean, CM-SSM would rank below MDNet (83.0%) in Table II, which would directly affect the generalizability claim.","section":"§IV-C2, Table II"},{"comment":"The cross-modal sequence is defined as f_1_c = {r_1, t_1, r_2, t_2, ..., r_k, t_k, ...}. In this ordering, the state immediately preceding t_k is rbar_k, not rbar_{k-1}. Eq. (5) nevertheless uses rbar_{k-1} to compute tbar_k. This means the thermal hidden state does not incorporate the RGB hidden state at the same position, which is inconsistent with the described scanning order and with the stated idea of deriving one modality's state from the other at each step. Please correct the indexing or explicitly define a different intended ordering.","section":"§III-B, Eqs. (4)-(5)"},{"comment":"The paper states that CART \"was randomly partitioned into train/val/test set at a 6:1:1 ratio,\" but it does not specify whether this random partition is identical to the official CART split or to the splits used by the baseline methods in Table I. If the split differs, the comparison may not be meaningful. Please state whether the official splits were used, and if a custom random split was used, provide the split indices or explain how the comparison remains fair.","section":"§IV-A"}],"minor_comments":[{"comment":"The text says that ⊗ denotes addition, but in context Eq. (8) uses ⊗ to combine a gated feature with the input, which should be element-wise multiplication. Please clarify the notation.","section":"§III-C, Eq. (8)"},{"comment":"There are several typos, including \"dateset\" (Sec. IV-A), \"filed robots\" (Sec. V), and \"corss-modal\" (Sec. V). A careful proofread would improve presentation.","section":"Throughout"},{"comment":"The contribution list includes specific parameter counts and FPS values; please ensure these are consistent with Table III and with the corresponding numbers stated in the abstract and conclusion, which they currently are.","section":"§I, Contributions"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the efficiency results are attractive, but the PST900 mIoU arithmetic and the Eq. (10) mismatch are exactly the kind of issues that could indicate reporting errors in the experimental section. I would ask the authors to provide the corrected tables and to clarify the exact architecture used in the released code before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the PST900 SOTA claim doesn't survive the paper's own table. The reported 85.9 mIoU is not the mean of the four per-class IoUs listed (80.5, 85.4, 85.7, 78.1), which average to 82.4. That puts CM-SSM below MDNet's 83.0 in Table II, so the generalizability claim is unsupported as written. The authors need to either correct the number or state a different mIoU definition (e.g., including a background class or frequency weighting).\n\nWhat's genuinely new: the CM-SS2D interleaved RGB-thermal scan with reciprocal hidden-state recursion (Eqs. 4-5) is a real twist on Coupled Mamba and Sigma; prior work couples chains or generates parameters from the other modality, but not this exact cross-modal hidden-state construction. The linear-complexity argument is sound, and the efficiency numbers (10.34 GFLOPs, 12.59M params, 114 FPS) are the paper's strongest asset. On CART, 74.6 mIoU vs. CMX's 74.0 with one-sixth the FLOPs is a credible efficiency story, and the ablation showing CM-SS2D adds 1.3 mIoU supports the design.\n\nSoft spots, in order of severity. First, the PST900 inconsistency is load-bearing because the abstract and Sec. IV-C2 both advertise 85.9 as SOTA. Under the table's own numbers, CM-SSM would rank second at 82.4, not first. Second, Eq. (10) concatenates R_i, T_i, L while the text says it combines G_R, G_T, and L; the global branch disappears from the equation. That looks like a typo, but as written it's confusing and should be fixed. Third, the paper says CART was randomly split 6:1:1 but doesn't confirm this matches the official benchmark splits used by the cited baselines. If the splits differ, Table I collapses. That's a standard sentence to add.\n\nI'm not convinced by the claim that the novelty over Coupled Mamba is experimentally demonstrated; there's no direct comparison. But that's a minor point for an empirical paper.\n\nCode is promised, which helps, but I didn't run it.\n\nWho this is for: people building real-time RGB-T segmentation for field robots. The efficiency-accuracy trade-off is practically relevant. The paper deserves a serious referee, but the PST900 number and Eq. (10) need fixing before acceptance. Send it out.","headline":"The PST900 SOTA claim is contradicted by the paper's own table; the architecture has merit but needs a corrected revision.","tokens_in":11371,"tokens_out":3651,"would_cite":false,"duration_ms":35281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A cross-modal state space model fuses RGB and thermal images in linear time, reaching state-of-the-art segmentation accuracy on wild-scene benchmarks.","keywords":["RGB-thermal semantic segmentation","state space model","cross-modal fusion","linear complexity","real-time segmentation","selective scan","wild scene understanding","EfficientViT"],"falsifier":"Rerun CM-SSM and the leading baselines on the official CART and PST900 split definitions; if CM-SSM's mIoU falls below CMX's 74.0% on the same split, the state-of-the-art claim fails.","tokens_in":10355,"feed_emoji":"🌡️","tokens_out":2937,"duration_ms":28172,"temperature":0.7,"pith_summary":"The paper sets out to show that global cross-modal fusion for RGB-thermal semantic segmentation can be done in linear time, not quadratic, by building a state space model whose hidden state for each modality is computed from the other modality's previous state. It introduces CM-SS2D, which interleaves the two modalities into one scan sequence, and CM-SSA, which combines the resulting global associations with local convolutional features. On the CART and PST900 benchmarks the resulting model, CM-SSM, reports state-of-the-art mean intersection-over-union of 74.6% and 85.9% while using 12.59M parameters and running at 114 frames per second on an RTX 4090 GPU. A sympathetic reader would take the central claim to be that constructing the hidden state across modalities is the right way to adapt state space models to multi-modal fusion.","feed_headline":"RGB-thermal fusion goes linear-time with cross-modal state space","feed_subtitle":"Hidden states passed between modalities hit 74.6% mIoU on CART, 85.9% on PST900, and 114 FPS.","key_machinery":"The central object is the cross-modal hidden state recurrence, implemented in the CM-SS2D module. First, RGB and thermal features are interleaved into a 'RGB-thermal-RGB' visual sequence across four scanning directions. Then the hidden state of each modality is derived from the other modality's previous hidden state through the discretized state space equations, giving each hidden state a blend of cross-modal and global context. The CM-SSA module then merges the global association from CM-SS2D with a convolutional local association using gated residual connections and a 1x1 convolution.","core_discovery":"CM-SSM establishes that the pivotal design for cross-modal state space fusion is to construct the hidden state across modalities, rather than merely using multi-modal information to generate SSM parameters. At each scan position k, the RGB hidden state $\\bar{r}_k$ is computed from the thermal hidden state $\\bar{t}_{k-1}$ and the current RGB input, and symmetrically $\\bar{t}_k = \\bar{A}\\bar{r}_{k-1} + \\bar{B}t_k$. This couples the two modalities' recurrent memories so that each hidden state carries both the other modality's information at the same spatial position and global context accumulated along the scan. Because the recurrence processes a sequence of length $2HW$ with a fixed per-step cost, the entire fusion module has linear computational complexity with respect to image resolution.","pith_inferences":["The same cross-modal state construction could plausibly extend to RGB-depth or RGB-event fusion, where the complementary cues have different spatial statistics and noise profiles.","The linear-complexity claim should be stress-tested at higher input resolutions, since actual wall-clock speed depends on the scan implementation and memory access patterns.","The paper does not analyze behavior when one modality is degraded or missing; a testable extension would corrupt thermal input and measure how much the cross-modal recurrence degrades relative to convolutional fusion.","A deeper investigation might isolate whether the improvement over FusionMamba-style parameter-sharing comes from the hidden-state coupling itself or from the residual gating in CM-SSA."],"forward_implications":["If the central claim is correct, quadratic-complexity transformer-based fusion can be replaced by linear-complexity state space fusion for high-resolution multi-modal inputs without sacrificing accuracy.","The cross-modal hidden-state exchange gives a template for fusing more than two modalities by chaining state updates.","Real-time deployment on resource-constrained robots becomes more feasible: 114 FPS on an RTX 4090 with 12.59M parameters.","Existing SSM fusion baselines that use multi-modal information only to generate parameters are subsumed by a design that also carries information through the hidden state.","The reported gains on small object classes such as vehicles and persons on CART suggest the mechanism especially helps spatially compact targets."],"supporting_citations":[{"why":"Provides the SS2D module and discretization procedure that CM-SS2D extends to cross-modal scanning.","marker":"[15]"},{"why":"Supplies the selective state space model and the linear-time recurrence that the method builds on.","marker":"[23]"},{"why":"Introduces the CART dataset used for the main benchmark and defines its train/val/test partitioning.","marker":"[1]"},{"why":"Introduces the PST900 dataset used to demonstrate generalization.","marker":"[21]"},{"why":"Provides the EfficientViT-B1 backbone that the encoder is built on with ImageNet-1K pretraining.","marker":"[25]"},{"why":"The CMX fusion strategy is used as a comparison baseline and as the 'FFM' ablation variant.","marker":"[8]"},{"why":"The Cross Mamba and Concat Mamba blocks are the ablation baseline 'CroMB+ConMB' that the paper must beat.","marker":"[19]"},{"why":"Provides the MDNet method and the 'MDFusion' ablation variant that uses SS2D without cross-modal hidden states.","marker":"[16]"}],"fun_headline_variants":["Cross-modal state space fuses RGB-thermal in linear time","Real-time RGB-thermal segmentation via cross-modal SSM","Linear-complexity fusion: RGB-thermal semantic segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported state-of-the-art comparisons assume that the train/val/test splits used here match the official splits used by every baseline paper, because CART is described only as randomly partitioned 6:1:1.","fun_headline_variants_meta":{"raw":{"variants":["Cross-modal state space fuses RGB-thermal in linear time","Real-time RGB-thermal segmentation via cross-modal SSM","Linear-complexity fusion: RGB-thermal semantic segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1210,"prompt_tokens":939,"completion_tokens":271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":217}},"tokens_in":555,"tokens_out":271,"duration_ms":3276,"temperature":1.0,"reasoning_tokens":217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:24:17.146378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun CM-SSM and the leading baselines on the official CART and PST900 split definitions; if CM-SSM's mIoU falls below CMX's 74.0% on the same split, the state-of-the-art claim fails.","supporting_citations":[{"cited_title":"Vmamba: Visual State Space Model,","cited_arxiv_id":null,"evidence_quote":"Provides the SS2D module and discretization procedure that CM-SS2D extends to cross-modal scanning."},{"cited_title":"Caltech aerial rgb-thermal dataset in the wild,","cited_arxiv_id":null,"evidence_quote":"Introduces the CART dataset used for the main benchmark and defines its train/val/test partitioning."},{"cited_title":"Pst900: Rgb-thermal calibration, dataset and segmenta- tion network,","cited_arxiv_id":null,"evidence_quote":"Introduces the PST900 dataset used to demonstrate generalization."},{"cited_title":"Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the EfficientViT-B1 backbone that the encoder is built on with ImageNet-1K pretraining."},{"cited_title":"CMX: Cross-modal fusion for RGB-X semantic segmentation with transform- ers,","cited_arxiv_id":null,"evidence_quote":"The CMX fusion strategy is used as a comparison baseline and as the 'FFM' ablation variant."},{"cited_title":"MDNet: Mamba-effective diffusion- distillation network for RGB-thermal urban dense prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the MDNet method and the 'MDFusion' ablation variant that uses SS2D without cross-modal hidden states."}],"review_version":1}