Pith. sign in

REVIEW 9 cited by

Fusion-Mamba for Cross-modality Object Detection

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.09146 v1 pith:KTJUMLYU submitted 2024-04-14 cs.CV cs.AI

Wenhao Dong , Haodong Zhu , Shaohui Lin , Xiaoyan Luo , Yunhang Shen , Xuhui Liu , Juan Zhang , Guodong Guo
show 1 more author
Baochang Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords fusioncross-modalitydifferentfeaturesspacestatecross-modaldetection
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Cross-modality fusing complementary information from different modalities effectively improves object detection performance, making it more useful and robust for a wider range of applications. Existing fusion strategies combine different types of images or merge different backbone features through elaborated neural network modules. However, these methods neglect that modality disparities affect cross-modality fusion performance, as different modalities with different camera focal lengths, placements, and angles are hardly fused. In this paper, we investigate cross-modality fusion by associating cross-modal features in a hidden state space based on an improved Mamba with a gating mechanism. We design a Fusion-Mamba block (FMB) to map cross-modal features into a hidden state space for interaction, thereby reducing disparities between cross-modal features and enhancing the representation consistency of fused features. FMB contains two modules: the State Space Channel Swapping (SSCS) module facilitates shallow feature fusion, and the Dual State Space Fusion (DSSF) enables deep fusion in a hidden state space. Through extensive experiments on public datasets, our proposed approach outperforms the state-of-the-art methods on $m$AP with 5.9% on $M^3FD$ and 4.9% on FLIR-Aligned datasets, demonstrating superior object detection performance. To the best of our knowledge, this is the first work to explore the potential of Mamba for cross-modal fusion and establish a new baseline for cross-modality object detection.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SMamba: Sparse Mamba for Event-based Object Detection

    cs.CV 2025-01 conditional novelty 6.0 of 10

    SMamba prunes uninformative event tokens using a spatiotemporal continuity score, then scans the survivors with Mamba blocks to match or beat prior event detectors at lower compute.

  2. A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects

    cs.MM 2025-06 conditional novelty 5.0 of 10

    A task-agnostic survey of multi-sensor fusion perception methods for embodied AI, covering multi-modal, multi-agent, time-series, and multimodal large language model fusion.

  3. MV-GMN: State Space Model for Multi-View Action Recognition

    cs.CV 2025-01 conditional novelty 5.0 of 10

    MV-GMN, a state-space model with graph convolution, reports state-of-the-art accuracies on NTU RGB+D and PKU-MMD action recognition benchmarks.

  4. Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Merged-token retraining recovers Vision Mamba accuracy after token reduction, within minutes and with up to 1.5x faster inference.

  5. AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment

    cs.CV 2024-12 conditional novelty 5.0 of 10

    AlignMamba fuses audio, video, and language by matching tokens to a language anchor and enforcing distribution similarity, reporting small accuracy gains with large efficiency gains on MOSI and MOSEI.

  6. FTMoMamba: Motion Generation with Frequency and Text State Space Models

    cs.CV 2024-11 conditional novelty 5.0 of 10

    FTMoMamba achieves FID 0.181 on HumanML3D by injecting frequency-domain features into the state transition matrix and text features into the output matrix of a Mamba-based diffusion denoiser.

  7. SpectMamba: Integrating Frequency and State Space Models for Enhanced Medical Image Detection

    cs.CV 2025-09 conditional novelty 4.0 of 10

    A Mamba-based detector with frequency attention and Hilbert curve scanning edges out several baselines on pneumonia, brain tumor, and fracture detection.

  8. LASFNet: A Lightweight Attention-Guided Self-Modulation Feature Fusion Network for Multimodal Object Detection

    cs.CV 2025-06 conditional novelty 4.0 of 10

    LASFNet fuses RGB and infrared features in one lightweight stage with attention modules, reporting similar or better detection accuracy than heavier multimodal detectors.

  9. Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition

    cs.CV 2025-05 conditional novelty 4.0 of 10

    MCO-E Net fuses event and RGB eye data via a jointly optimized Mamba and a heterogeneous MoE, achieving 91.3% WAR and 91.9% UAR on the SEE dataset.

Pith tools