Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion

Sangyun Chung; Taeheon Kim; Yong Man Ro; Youngjoon Yu

REVIEW 4 major objections 6 minor 31 references

Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes a language-driven cross-modal alignment method that makes multispectral pedestrian detection work on raw, uncalibrated RGB-thermal camera pairs.

desk verdict Interesting matching idea for heavily misaligned RGB-thermal data, but the central AP claims are unsupported because the paper never defines how correspondences become detection boxes and scores. read the letter →

arxiv 2411.17995 v1 pith:GARCWZUM submitted 2024-11-27 cs.CV

Taeheon Kim , Sangyun Chung , Youngjoon Yu , Yong Man Ro This is my paper · ORCID

classification cs.CV

keywords multispectralpedestriandetectionRGB-thermalmisalignmentlargevision-languagemodelscross-modalalignmentminimumspanningtreeerrorratelatefusionsemanticmatching

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles the practical problem of multispectral pedestrian detection when the RGB and thermal cameras are not calibrated, so the same person appears at different positions in the two images. Existing fusion methods assume the two views are largely overlapping and break down under heavy misalignment. The proposed framework first builds a minimum spanning tree of detected pedestrians in each modality to capture their relative spatial layout, then uses large vision-language models to describe each person's appearance, and finally prompts an LLM to fuse both cues and output cross-modal matches. On two heavily misaligned test sets, the method raises detection AP over the ProbEn baseline and reduces a new alignment error rate from over 80 percent to under 10 percent on the FLIR set. If the results hold, accurate multispectral detection becomes possible on raw data from uncalibrated cameras, removing the need for beam splitters, checkerboards, or image registration.

What carries the argument

The method has four load-bearing components. Positional graphs: each modality's single-modal detections (from Co-DETR) become nodes, and Kruskal's algorithm builds a minimum spanning tree connecting each person to its nearest neighbours, so the graph encodes relative spatial layout rather than absolute coordinates. Appearance embeddings: each detected person is zoomed and cropped, and an LVLM (e.g., GPT-4, Gemini, Claude 2) produces a structured textual description of facing, clothing, accessories, and action; this text is attached to the corresponding graph node. Multi-LLM debate: several LVLMs propose descriptions and a judge model (GPT-4) synthesizes a consensus, mitigating hallucination. LLM prediction: the final prompt supplies both the MST structure and the appearance texts and asks the LLM to output a cross-modal matching, with a rationale. The paper also introduces the alignment error rate (AER), defined as the fraction of RGB-thermal person pairs that are mismatched, as a new evaluation metric for heavily misaligned data.

What would settle it

Take a set of RGB-thermal pairs with intentionally varied camera baselines or zoom factors, run the proposed method, and measure AER; if the alignment error climbs steeply as the nearest-neighbor structure diverges, the MST-preservation assumption is falsified. Alternatively, replace the LVLM appearance descriptions with random textual attributes and keep the positional graphs; if the matching performance stays the same, the contribution of the language-driven appearance cue is not causal.

Watch

Extended reading notes

Core claim

The central claim is that heavy RGB-thermal misalignment can be resolved at the semantic level rather than the geometric level: instead of aligning pixels or feature maps, the method aligns identities. It detects pedestrians independently in each modality, encodes the spatial configuration of each modality as a minimum spanning tree whose edges are inter-person distances, and augments each node with LVLM-generated textual descriptions of appearance and action. A large language model then reads both graphs and matches nodes across modalities using positional and appearance agreement. The paper reports that this approach outperforms the probabilistic ensembling baseline ProbEn on both a modified FLIR ADAS set and a newly collected set of challenging uncalibrated pairs, and that adding a multi-LLM debate before the final judgement further improves matching, achieving an alignment error rate of 8.8 percent on the FLIR set.

Load-bearing premise

The approach assumes that the relative positions and distances of pedestrians are roughly preserved between the two misaligned views, so the nearest-neighbor connection structure of people in one image resembles the other; if the cameras have very different fields of view or viewing angles, this structural similarity breaks down and the matching degrades.

Editorial extensions

If this is right

Multispectral pedestrian detectors can be deployed directly on raw, uncalibrated camera feeds, since the proposed method requires no beam splitters, checkerboard calibration, or image registration preprocessing.
Fusion no longer depends on overlapping regions of interest between modalities; the method matches identities via spatial layout and appearance, so it can handle cases where a person appears at very different locations in RGB and thermal.
The alignment error rate (AER) provides a concrete, repeatable measure of cross-modal matching quality that can be used to compare future methods on heavily misaligned data.
The framework's ability to correct misalignment at inference time suggests it could be extended to other multispectral tasks, such as tracking or re-identification, without retraining the fusion pipeline per camera pair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The success of the method hinges on the MST structure being roughly preserved across views; a natural stress test is to measure AER as the field-of-view difference or baseline between cameras grows, since the paper does not quantify how much positional distortion the graph matching tolerates.
Because the matching is performed by an LLM on textual descriptions, the approach inherits the cost, latency, and hallucination risk of LVLMs; replacing the debate with a cheaper single-model or learned matcher could be a practical extension the authors do not explore.
The same semantic-alignment recipe could apply to other modalities whose images share physical layout but not appearance, such as visible-thermal re-identification, or even visible-depth matching, where positional graphs and language descriptions could serve as the common representation.
The paper's AER metric counts only matched person pairs; extending it to penalize false matches or missed detections might change the ranking of methods when detectors are imperfect.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper claims large gains on heavily misaligned RGB-thermal pairs: 74.9 vs 61.6 AP on a FLIR subset, 85.5 vs 75.6 on their collected set. If true, that would remove calibration hardware and preprocessing for multispectral detection, a real practical advance. The problem is genuine; raw uncalibrated multispectral data are common and standard detectors fail on them.

What's actually new: they construct positional MST graphs from per-modality detections, use LVLMs to describe each person's appearance, run a multi-LLM debate, and prompt an LLM to output RGB-to-thermal correspondences. This combination is not in the literature I know, and it's a plausible way to handle misalignment without calibration. The AER metric is a reasonable idea for measuring correspondence error, and collecting a hundred hard pairs is useful.

But there is a load-bearing evaluation problem. The method's final output is explicitly "the matching results" (Section 3.3), i.e., a set of correspondences. AP, though, is computed from detection boxes with scores. The paper never says how correspondences become boxes and scores, which ground-truth set is used, or how matched versus unmatched instances are handled. Without that step, the AP numbers in Table 1 are uninterpretable. They could come from RGB-only detections, from filtering the ground truth to matched pairs, or from some other rule. That is not a minor omission; it undercuts the central quantitative claim. ProbEn is a fusion detector, so comparing a matching-only pipeline to it on AP is not well-posed.

Other soft spots: two private 100-pair datasets, no error bars or significance tests, no comparison to image registration or other misalignment-robust baselines, and no code or data. The MST assumption that relative pedestrian positions are approximately preserved across modalities is not stress-tested; it likely breaks under sufficiently different FOVs. AER is not validated against any independent measure. Self-citation to the authors' prior MSCoTDet is present but not the core issue.

What's good: the problem is important, the approach is original, and the qualitative examples show plausible matches. But the paper as written is an idea with illustrative results, not a sound empirical demonstration. A reader working on LVLM grounding or cross-modal matching might get useful prompts; for multispectral detection, the numbers shouldn't be trusted.

Recommendation: I would not accept this in current form, but it deserves a serious referee if the authors can define the fusion-to-detection step and run a proper evaluation. Clarify the AP computation, add baselines and error bars, and release code and data.

Referee Report

4 major / 6 minor

Summary. The paper addresses heavily misaligned multispectral pedestrian detection by proposing a language-driven cross-modal alignment fusion framework. The method first runs a single-modal detector (Co-DETR) on RGB and thermal images, constructs Minimum Spanning Trees from detected pedestrian coordinates, collects appearance descriptions of each detected person using multiple LVLMs with a debate-and-judge scheme, and finally prompts an LLM to output RGB-to-thermal person correspondences. The correspondences are then used to fuse modalities for detection. The authors evaluate on a modified FLIR set and a newly collected set, each with 100 challenging pairs, comparing against ProbEn in terms of AP and a newly proposed Alignment Error Rate (AER). The paper reports substantial improvements (e.g., FLIR AP 74.9 vs 61.6; AER 8.8 vs 82.3) when using LLM debate.

Significance. If the results are reproducible, the paper would demonstrate that uncalibrated, heavily misaligned multispectral data can be handled without geometric preprocessing, which would be practically valuable and is a novel use of LVLMs. The paper introduces AER, a concrete first attempt to quantify misalignment in this setting, and provides detailed prompt texts and a clear pipeline diagram that help reproduction. However, the significance is currently limited by the evaluation: only two private 100-pair datasets, a single baseline, no statistical validation, and no public code or data, so the quantitative claims are not independently checkable. The conceptual contribution is clear and the qualitative examples are illustrative, but the evidence as presented does not yet substantiate the central claim of AP improvement.

major comments (4)

[Section 3.3, Table 1] The final output of the proposed pipeline is stated to be "the matching results" (RGB-to-thermal instance correspondences), but the paper never specifies how these correspondences are converted into fused detection bounding boxes and confidence scores. AP is computed on a ranked set of box detections with scores; without this conversion rule, the AP numbers in Table 1 are not well-defined. The improvement over ProbEn could in principle come from evaluating only matched ground-truth instances or from taking single-modality detections as the final boxes, which would make the comparison unfair. The authors must define the fusion-to-detection step and state exactly which detections and scores enter the AP computation.
[Section 4.1, Table 1] The quantitative evaluation relies on two 100-pair datasets that are not publicly released: one "modified from public FLIR ADAS" with no description of the modification protocol, and one newly collected. No error bars, standard deviations, or significance tests are reported, and ProbEn is the only baseline. As a result, the claimed 13.3-point AP gain on FLIR cannot be distinguished from dataset-selection effects or run-to-run variance. The authors should report repeated runs, include at least one registration-based or calibration-based preprocessing baseline, and release the datasets or detail the collection and modification procedure.
[Eq. (1), Section 4.2] AER is defined as the fraction of mismatched RGB-T person pairs, but the paper does not specify how ground-truth pairs are established, how pairs with missing detections are counted, or how false positive detections are treated. Since the detector outputs are inputs to the matching step, an evaluation of matching accuracy must control for detection quality. The metric is also not validated against any measure of geometric misalignment, so its meaning as a measure of "misalignment severity" is unclear. The paper should specify the annotation protocol and report AER with detection-quality controls (e.g., recall and precision of the box detections).
[Section 3.1] The MST-based positional-matching assumption that relative positions and distances of pedestrians are approximately preserved across modalities is not tested. In heavily misaligned cameras with different fields of view or vantage points, the nearest-neighbor structure of pedestrian configurations can change, causing correct correspondences to violate the MST topology. The authors should provide an analysis or ablation showing how much spatial distortion (rotation, scale, or perspective) the method tolerates before matching degrades, or explicitly discuss this limitation as a boundary condition of the method's validity.

minor comments (6)

[Section 1, Fig. 1] The terms "well-aligned," "weakly-aligned," and "unaligned" are used throughout but never defined operationally; the authors should state a quantitative or at least a qualitative criterion for these categories, and fix the subject-verb agreement in the caption ("does not overlap" should be "do not overlap").
[Section 4.1] The phrase "modified from public FLIR ADAS" is vague; the exact selection and modification procedure should be described, including how the 100 pairs were chosen and whether they are a subset of the original test set, so that readers can assess the difficulty of the constructed benchmark.
[Table 1] The "LLM Debate" column is ambiguous because the baseline rows do not have a corresponding debate-ablated variant; the table should be restructured to show a clean ablation of the debate component and use a clearer notation for whether the debate module is enabled.
[Section 3.2] The paper claims the LVLM descriptions include clothing, accessories, hairstyle, and other attributes, but the qualitative example in Fig. 2 lists only facing, clothes, and acting; it is unclear which attributes are actually used in the final matching prompt and whether the judge aggregates all of them.
[Section 2, Ref. [15]] The paper mentions the authors' prior MSCoTDet work [15] but does not compare with it or explain how the proposed method differs from it; adding a sentence that distinguishes the two would help position the contribution and avoid the appearance of an incomplete related-work discussion.
[Section 4.2] The notation "RGB-T person pairs" in Eq. (1) is undefined; the authors should clarify whether these are ground-truth identity pairs, matched detections, or something else, and how the denominator is computed when the two modalities contain different numbers of detected persons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is an empirical matching pipeline with no fitted-parameter or self-citation-derived predictions.

full rationale

The paper does not derive any quantity from another quantity by construction. The proposed pipeline takes per-modality detections from Co-DETR, constructs MST positional graphs, obtains LVLM appearance descriptions, and uses an LLM to output RGB-to-thermal correspondences. There are no trainable parameters fitted to the test sets, and no prediction is defined in terms of the method's own outputs beyond the normal sense in which any accuracy metric is computed from a system's output. The self-citations ([14], [15]) are related work on multispectral detection and are not load-bearing; the method does not invoke a claimed uniqueness theorem or prior result to force its design. The newly proposed AER metric is essentially matching accuracy and is therefore a natural, if somewhat self-referential, evaluation of a matching system, but this is not circularity as defined here. The skeptic's concern that the paper does not specify how correspondences are converted to detection boxes for AP is an experimental-reporting gap and a correctness risk, not a circular step, because no equation in the paper reduces the reported AP to the method's inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method introduces no fitted free parameters and no new physical entities. It relies on four domain assumptions: spatial arrangement consistency, reliability of LVLM descriptions, generalization of the pretrained detector, and the matching ability of the LLM. These are all empirical premises with limited validation.

assumptions (4)

domain assumption Relative positions and distances of people are similar across the two misaligned modalities.
Stated in Section 3.1: 'Since both types of images are based on the physical arrangement of space, the relative positions and distances of people will appear similarly.' The method depends on this for MST graph matching.
domain assumption LVLM-generated appearance descriptions are accurate enough to distinguish individuals, and the debate scheme resolves hallucinations.
The method relies on GPT-4, Gemini, and Claude 2 to describe each person, with no quantitative validation of description accuracy on the test data; hallucination is acknowledged but only mitigated by debate.
domain assumption The Co-DETR detector pretrained on FLIR ADAS generalizes to the new uncalibrated data and produces consistent detections across both modalities.
Single-modal detection is performed with Co-DETR trained on FLIR ADAS (Section 3.1); no fine-tuning or domain adaptation for the new unaligned data is described.
domain assumption The LLM matching prompt yields correct correspondences given the graphs and descriptions.
The final matching step is performed by prompting an LLM, and its reliability is only measured indirectly through the reported AP and AER on the two small datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion." pith.science (2026). https://pith.science/paper/GARCWZUM

@misc{pith2026241117995,
  author       = {Pith},
  title        = {Pith review of: Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GARCWZUM}},
  note         = {Machine review of arXiv:2411.17995}
}

read the original abstract

Multispectral pedestrian detection is a crucial component in various critical applications. However, a significant challenge arises due to the misalignment between these modalities, particularly under real-world conditions where data often appear heavily misaligned. Conventional methods developed on well-aligned or minimally misaligned datasets fail to address these discrepancies adequately. This paper introduces a new framework for multispectral pedestrian detection designed specifically to handle heavily misaligned datasets without the need for costly and complex traditional pre-processing calibration. By leveraging Large-scale Vision-Language Models (LVLM) for cross-modal semantic alignment, our approach seeks to enhance detection accuracy by aligning semantic information across the RGB and thermal domains. This method not only simplifies the operational requirements but also extends the practical usability of multispectral detection technologies in practical applications.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 29 canonical work pages

[1]

Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion

INTRODUCTION Multispectral pedestrian detection uses both RGB and in- frared (thermal) images for pedestrian detection [1, 2, 3, 4, 5, 6, 7]. Compared to relying on just one modality detec- tion [8, 9, 10, 11, 12], utilizing both modalities provide dis- tinct advantages on pedestrian detection [13, 14, 15]. How- ever, positional misalignments between the ...

work page Pith review arXiv 2024
[2]

To solve the misalignment problems, camera calibration tech- niques and image registration algorithms have been devel- oped

RELA TED WORK Misalignments in Multispectral Pedestrian Detection. To solve the misalignment problems, camera calibration tech- niques and image registration algorithms have been devel- oped. They pre-process the raw data to be spatially aligned. Camera calibration techniques physically align the two image domains using special devices such as beam splitt...

work page
[3]

front"} Clothes: {clothes: [

PROPOSED METHOD 3.1. Constructing Positional Graphs Humans can analyze the scenes and extract semantic contexts useful for pedestrian detection using global and local contexts of the scene. Specifically, humans can recognize pedestrians at a glance by their positional location relative to surrounding objects or landmarks. Such positional relations can be ...

work page
[4]

Implementation Details We conduct experiments on two different heavily misaligned RGB-thermal multispectral dataset

EXPERIMENTS 4.1. Implementation Details We conduct experiments on two different heavily misaligned RGB-thermal multispectral dataset. One is modified from public FLIR ADAS dataset, and the other is our collected dataset. Each dataset contains a hundred challenging mis- alignment RGB-thermal unaligned pairs respectively. For the comparison, we use the base...

work page
[5]

CONCLUSION This paper introduces an innovative framework for mul- tispectral pedestrian detection, specifically addressing the challenges of heavy misalignment between RGB and thermal images in practical applications. Unlike conventional meth- ods that rely on costly and complex pre-processing to align images, our approach utilizes semantic alignment thro...

work page
[6]

Multispectral pedestrian de- tection: Benchmark dataset and baseline,

Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon, “Multispectral pedestrian de- tection: Benchmark dataset and baseline,” in Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, 2015, pp. 1037–1045

work page 2015
[7]

Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection,

Jung Uk Kim, Sungjune Park, and Yong Man Ro, “Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 3, pp. 1510–1523, 2021

work page 2021
[8]

Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning

Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro, “Investigat- ing vulnerability to adversarial examples on multi- modal data fusion in deep learning,” arXiv preprint arXiv:2005.10987, 2020

work page Pith review arXiv 2005

Show all 31 references

[9]

Mul- tispectral invisible coating: laminated visible-thermal physical attack against multispectral object detectors us- ing transparent low-e films,

Taeheon Kim, Youngjoon Yu, and Yong Man Ro, “Mul- tispectral invisible coating: laminated visible-thermal physical attack against multispectral object detectors us- ing transparent low-e films,” inProceedings of the AAAI Conference on Artificial Intelligence, 2023, vol. 37, pp...

2023
[10]

Towards robust train- ing of multi-sensor data fusion network against adver- sarial examples in semantic segmentation,

Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro, “Towards robust train- ing of multi-sensor data fusion network against adver- sarial examples in semantic segmentation,” in ICASSP 2021-2021 IEEE International Conference on Acous- tics, Speech and Sig...

2021
[11]

Robust multispectral pedestrian de- tection via spectral position-free feature mapping,

Sungjune Park, Jung Uk Kim, Jin Mo Song, and Yong Man Ro, “Robust multispectral pedestrian de- tection via spectral position-free feature mapping,” in 2023 IEEE International Conference on Image Process- ing (ICIP). IEEE, 2023, pp. 1795–1799

2023
[12]

Robust multispectral pedestrian detection via uncertainty-aware cross-modal learning,

Sungjune Park, Jung Uk Kim, Yeon Gyun Kim, Sang- Keun Moon, and Yong Man Ro, “Robust multispectral pedestrian detection via uncertainty-aware cross-modal learning,” in MultiMedia Modeling: 27th International Conference, MMM 2021, Prague, Czech Republic, June 22–24, 2021, Proce...

2021
[13]

Detrs with collaborative hybrid assignments training,

Zhuofan Zong, Guanglu Song, and Yu Liu, “Detrs with collaborative hybrid assignments training,” in Proceed- ings of the IEEE/CVF international conference on com- puter vision, 2023, pp. 6748–6758

2023
[14]

Defending person detection against adversarial patch attack by using universal defensive frame,

Youngjoon Yu, Hong Joo Lee, Hakmin Lee, and Yong Man Ro, “Defending person detection against adversarial patch attack by using universal defensive frame,” IEEE Transactions on Image Processing , vol. 31, pp. 6976–6990, 2022

2022
[15]

De- fending physical adversarial attack on object detection via adversarial patch-feature energy,

Taeheon Kim, Youngjoon Yu, and Yong Man Ro, “De- fending physical adversarial attack on object detection via adversarial patch-feature energy,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 1905–1913

2022
[16]

Inte- grating language-derived appearance elements with vi- sual cues in pedestrian detection,

Sungjune Park, Hyunjun Kim, and Yong Man Ro, “Inte- grating language-derived appearance elements with vi- sual cues in pedestrian detection,” IEEE Transactions on Circuits and Systems for Video Technology, 2024

2024
[17]

Robust pedestrian detection via constructing versatile pedestrian knowledge bank,

Sungjune Park, Hyunjun Kim, and Yong Man Ro, “Robust pedestrian detection via constructing versatile pedestrian knowledge bank,” Pattern Recognition, p. 110539, 2024

2024
[18]

Map: Multispectral adversarial patch to attack person detec- tion,

Taeheon Kim, Hong Joo Lee, and Yong Man Ro, “Map: Multispectral adversarial patch to attack person detec- tion,” in ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 4853–4857

2022
[19]

Causal mode multiplexer: A novel framework for unbiased multispectral pedestrian detec- tion,

Taeheon Kim, Sebin Shin, Youngjoon Yu, Hak Gu Kim, and Yong Man Ro, “Causal mode multiplexer: A novel framework for unbiased multispectral pedestrian detec- tion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26784–26793

2024
[20]

Mscotdet: Language-driven multi-modal fusion for improved multispectral pedestrian detection,

Taeheon Kim, Sangyun Chung, Damin Yeom, Youngjoon Yu, Hak Gu Kim, and Yong Man Ro, “Mscotdet: Language-driven multi-modal fusion for improved multispectral pedestrian detection,” arXiv preprint arXiv:2403.15209, 2024

2024 arXiv
[21]

A survey of image registration techniques,

Lisa Gottesfeld Brown, “A survey of image registration techniques,” ACM computing surveys (CSUR), vol. 24, no. 4, pp. 325–376, 1992

1992
[22]

Re- mote sensing image registration techniques: A survey,

Suma Dawn, Vikas Saxena, and Bhudev Sharma, “Re- mote sensing image registration techniques: A survey,” in Image and Signal Processing: 4th International Con- ference, ICISP 2010, Trois-Rivi`eres, QC, Canada, June 30-July 2, 2010. Proceedings 4 . Springer, 2010, pp. 103–112

2010
[23]

A survey of medical image registration,

JB Antoine Maintz and Max A Viergever, “A survey of medical image registration,” Medical image analysis, vol. 2, no. 1, pp. 1–36, 1998

1998
[24]

An iterative integrated frame- work for thermal–visible image registration, sensor fu- sion, and people tracking for video surveillance applica- tions,

Atousa Torabi, Guillaume Mass ´e, and Guillaume- Alexandre Bilodeau, “An iterative integrated frame- work for thermal–visible image registration, sensor fu- sion, and people tracking for video surveillance applica- tions,” Computer Vision and Image Understanding, vol. 116, no....

2012
[25]

Attentive alignment network for multispectral pedestrian detection,

Nuo Chen, Jin Xie, Jing Nie, Jiale Cao, Zhuang Shao, and Yanwei Pang, “Attentive alignment network for multispectral pedestrian detection,” in Proceedings of the 31st ACM international conference on multimedia , 2023, pp. 3787–3795

2023
[26]

Weakly aligned cross-modal learning for multispectral pedestrian detection,

Lu Zhang, Xiangyu Zhu, Xiangyu Chen, Xu Yang, Zhen Lei, and Zhiyong Liu, “Weakly aligned cross-modal learning for multispectral pedestrian detection,” in Pro- ceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5127–5137

2019
[27]

Illumination-aware faster r-cnn for robust multispectral pedestrian detection,

Chengyang Li, Dan Song, Ruofeng Tong, and Min Tang, “Illumination-aware faster r-cnn for robust multispectral pedestrian detection,” Pattern Recognition, vol. 85, pp. 161–171, 2019

2019
[28]

Improving multispectral pedestrian detection by addressing modal- ity imbalance problems,

Kailai Zhou, Linsen Chen, and Xun Cao, “Improving multispectral pedestrian detection by addressing modal- ity imbalance problems,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part XVIII 16. Springer, 2020, pp. 787–803

2020
[29]

Multimodal object de- tection via probabilistic ensembling,

Yi-Ting Chen, Jinghao Shi, Zelin Ye, Christoph Mertz, Deva Ramanan, and Shu Kong, “Multimodal object de- tection via probabilistic ensembling,” in European Con- ference on Computer Vision . Springer, 2022, pp. 139– 158

2022
[30]

Gpt-4 technical report,

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

2023 arXiv
[31]

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Fi- rat, Julian Schrittwieser, et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXi...

2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion

INTRODUCTION Multispectral pedestrian detection uses both RGB and in- frared (thermal) images for pedestrian detection [1, 2, 3, 4, 5, 6, 7]. Compared to relying on just one modality detec- tion [8, 9, 10, 11, 12], utilizing both modalities provide dis- tinct advantages on pedestrian detection [13, 14, 15]. How- ever, positional misalignments between the ...

work page Pith review arXiv 2024

[2] [2]

To solve the misalignment problems, camera calibration tech- niques and image registration algorithms have been devel- oped

RELA TED WORK Misalignments in Multispectral Pedestrian Detection. To solve the misalignment problems, camera calibration tech- niques and image registration algorithms have been devel- oped. They pre-process the raw data to be spatially aligned. Camera calibration techniques physically align the two image domains using special devices such as beam splitt...

work page

[3] [3]

front"} Clothes: {clothes: [

PROPOSED METHOD 3.1. Constructing Positional Graphs Humans can analyze the scenes and extract semantic contexts useful for pedestrian detection using global and local contexts of the scene. Specifically, humans can recognize pedestrians at a glance by their positional location relative to surrounding objects or landmarks. Such positional relations can be ...

work page

[4] [4]

Implementation Details We conduct experiments on two different heavily misaligned RGB-thermal multispectral dataset

EXPERIMENTS 4.1. Implementation Details We conduct experiments on two different heavily misaligned RGB-thermal multispectral dataset. One is modified from public FLIR ADAS dataset, and the other is our collected dataset. Each dataset contains a hundred challenging mis- alignment RGB-thermal unaligned pairs respectively. For the comparison, we use the base...

work page

[5] [5]

CONCLUSION This paper introduces an innovative framework for mul- tispectral pedestrian detection, specifically addressing the challenges of heavy misalignment between RGB and thermal images in practical applications. Unlike conventional meth- ods that rely on costly and complex pre-processing to align images, our approach utilizes semantic alignment thro...

work page

[6] [6]

Multispectral pedestrian de- tection: Benchmark dataset and baseline,

Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon, “Multispectral pedestrian de- tection: Benchmark dataset and baseline,” in Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, 2015, pp. 1037–1045

work page 2015

[7] [7]

Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection,

Jung Uk Kim, Sungjune Park, and Yong Man Ro, “Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 3, pp. 1510–1523, 2021

work page 2021

[8] [8]

Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning

Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro, “Investigat- ing vulnerability to adversarial examples on multi- modal data fusion in deep learning,” arXiv preprint arXiv:2005.10987, 2020

work page Pith review arXiv 2005