{"id":"2771436f-8720-48c7-b8b8-d86bab5b3a95","arxiv_id":"2411.17995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A language-driven system uses positional graphs and LVLM appearance descriptions with a debate scheme to match pedestrians across heavily misaligned RGB and thermal images, reporting big gains over ProbEn on two 100-pair datasets.","lead":"This paper proposes a system for pedestrian detection on misaligned RGB and thermal camera pairs, using large vision-language models to match people across modalities instead of calibrating the cameras. It reports large accuracy gains over a single baseline on two private 100-pair datasets, but the evaluation is thin and no code or data is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AP gain may be an artifact of an unspecified fusion-to-detection step; as written the pipeline outputs only instance correspondences, not detection boxes with scores.","rationale":"The reader's identified weakest assumption, that MST structure is approximately preserved across misaligned modalities, is technically legitimate: a perspective change need not preserve Euclidean nearest-neighbor structure, and the projection of non-coplanar pedestrian centers is not a homography. However, the paper's own experiments use two roughly co-located cameras, so the method may be adequate in the tested regime; the paper simply provides no tolerance analysis. The more fundamental gap is upstream: the paper claims a detection AP but defines and evaluates a matching algorithm. Section 3.3 ends with matching results, and no subsequent module is described that merges matched boxes into a single detection set with confidence scores. AP cannot be computed from correspondences alone; one needs boxes and scores. The paper's Table 1 reports AP with no definition of how it is derived, making the central claim unverifiable from the manuscript as written. It is possible the authors do use matched RGB boxes with some score aggregation, but that rule is absent. It is also possible that AP is computed from RGB detections only, in which case the comparison to ProbEn is misleading. The proposed concrete test would settle this. I do not see evidence of intentional misrepresentation; the issue is missing specification and missing artifacts. Therefore I would move the reader's verdict from CONDITIONAL to UNVERDICTED: the central claim may be true, but the evidence as written does not yet support it. If the authors supply the fusion rule and code, or an RGB-only ablation confirms a real fusion benefit, then the conditional acceptance could be restored.","tokens_in":7650,"tokens_out":4967,"duration_ms":50565,"concrete_test":"Ask the authors to specify the exact inference-time fusion rule and release code/data or detailed pseudocode: for each matched pair, which box(es) and score(s) are emitted as the final detection? Then reproduce Table 1 under that rule on the same 100 FLIR-challenging pairs. As a minimal ablation, compute AP using only the RGB Co-DETR detections (no thermal, no matching) and only the thermal detections on the same pairs. If RGB-only AP equals or exceeds the reported 'Ours' AP within a small margin (e.g., less than 1 AP point), the headline improvement is not evidence for cross-modal fusion. Also report AP computed over all ground-truth boxes, not only those that were matched.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central quantitative claim is the AP improvement in Table 1 (74.9 vs 61.6 on FLIR; 85.5 vs 75.6 on the collected set). But Sections 3.1-3.3 define a matching procedure whose final output is explicitly 'the matching results' (Section 3.3), i.e., a set of RGB-to-thermal instance correspondences. The paper never states how these correspondences are converted into the fused detection bounding boxes and confidence scores on which AP is computed. AP requires a ranked list of boxes with scores over a fixed ground-truth set; AER only requires pairwise correspondence accuracy. Without the conversion rule, the reported AP could be obtained by taking RGB-only Co-DETR detections as the final output (so thermal contributes nothing to AP), or by evaluating only matched pairs, which would inflate AP. The baseline ProbEn, by contrast, is an actual late-fusion detector; comparing it against a matching-only pipeline on an AP metric is not well-posed. This is more load-bearing than the MST-structure robustness issue because even a perfect matcher cannot justify a multispectral detection AP claim if AP is computed from a single modality or from a filtered subset of ground-truth boxes.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses heavily misaligned multispectral pedestrian detection by proposing a language-driven cross-modal alignment fusion framework. The method first runs a single-modal detector (Co-DETR) on RGB and thermal images, constructs Minimum Spanning Trees from detected pedestrian coordinates, collects appearance descriptions of each detected person using multiple LVLMs with a debate-and-judge scheme, and finally prompts an LLM to output RGB-to-thermal person correspondences. The correspondences are then used to fuse modalities for detection. The authors evaluate on a modified FLIR set and a newly collected set, each with 100 challenging pairs, comparing against ProbEn in terms of AP and a newly proposed Alignment Error Rate (AER). The paper reports substantial improvements (e.g., FLIR AP 74.9 vs 61.6; AER 8.8 vs 82.3) when using LLM debate.","tokens_in":7993,"tokens_out":4129,"duration_ms":36237,"significance":"If the results are reproducible, the paper would demonstrate that uncalibrated, heavily misaligned multispectral data can be handled without geometric preprocessing, which would be practically valuable and is a novel use of LVLMs. The paper introduces AER, a concrete first attempt to quantify misalignment in this setting, and provides detailed prompt texts and a clear pipeline diagram that help reproduction. However, the significance is currently limited by the evaluation: only two private 100-pair datasets, a single baseline, no statistical validation, and no public code or data, so the quantitative claims are not independently checkable. The conceptual contribution is clear and the qualitative examples are illustrative, but the evidence as presented does not yet substantiate the central claim of AP improvement.","major_comments":[{"comment":"The final output of the proposed pipeline is stated to be \"the matching results\" (RGB-to-thermal instance correspondences), but the paper never specifies how these correspondences are converted into fused detection bounding boxes and confidence scores. AP is computed on a ranked set of box detections with scores; without this conversion rule, the AP numbers in Table 1 are not well-defined. The improvement over ProbEn could in principle come from evaluating only matched ground-truth instances or from taking single-modality detections as the final boxes, which would make the comparison unfair. The authors must define the fusion-to-detection step and state exactly which detections and scores enter the AP computation.","section":"Section 3.3, Table 1"},{"comment":"The quantitative evaluation relies on two 100-pair datasets that are not publicly released: one \"modified from public FLIR ADAS\" with no description of the modification protocol, and one newly collected. No error bars, standard deviations, or significance tests are reported, and ProbEn is the only baseline. As a result, the claimed 13.3-point AP gain on FLIR cannot be distinguished from dataset-selection effects or run-to-run variance. The authors should report repeated runs, include at least one registration-based or calibration-based preprocessing baseline, and release the datasets or detail the collection and modification procedure.","section":"Section 4.1, Table 1"},{"comment":"AER is defined as the fraction of mismatched RGB-T person pairs, but the paper does not specify how ground-truth pairs are established, how pairs with missing detections are counted, or how false positive detections are treated. Since the detector outputs are inputs to the matching step, an evaluation of matching accuracy must control for detection quality. The metric is also not validated against any measure of geometric misalignment, so its meaning as a measure of \"misalignment severity\" is unclear. The paper should specify the annotation protocol and report AER with detection-quality controls (e.g., recall and precision of the box detections).","section":"Eq. (1), Section 4.2"},{"comment":"The MST-based positional-matching assumption that relative positions and distances of pedestrians are approximately preserved across modalities is not tested. In heavily misaligned cameras with different fields of view or vantage points, the nearest-neighbor structure of pedestrian configurations can change, causing correct correspondences to violate the MST topology. The authors should provide an analysis or ablation showing how much spatial distortion (rotation, scale, or perspective) the method tolerates before matching degrades, or explicitly discuss this limitation as a boundary condition of the method's validity.","section":"Section 3.1"}],"minor_comments":[{"comment":"The terms \"well-aligned,\" \"weakly-aligned,\" and \"unaligned\" are used throughout but never defined operationally; the authors should state a quantitative or at least a qualitative criterion for these categories, and fix the subject-verb agreement in the caption (\"does not overlap\" should be \"do not overlap\").","section":"Section 1, Fig. 1"},{"comment":"The phrase \"modified from public FLIR ADAS\" is vague; the exact selection and modification procedure should be described, including how the 100 pairs were chosen and whether they are a subset of the original test set, so that readers can assess the difficulty of the constructed benchmark.","section":"Section 4.1"},{"comment":"The \"LLM Debate\" column is ambiguous because the baseline rows do not have a corresponding debate-ablated variant; the table should be restructured to show a clean ablation of the debate component and use a clearer notation for whether the debate module is enabled.","section":"Table 1"},{"comment":"The paper claims the LVLM descriptions include clothing, accessories, hairstyle, and other attributes, but the qualitative example in Fig. 2 lists only facing, clothes, and acting; it is unclear which attributes are actually used in the final matching prompt and whether the judge aggregates all of them.","section":"Section 3.2"},{"comment":"The paper mentions the authors' prior MSCoTDet work [15] but does not compare with it or explain how the proposed method differs from it; adding a sentence that distinguishes the two would help position the contribution and avoid the appearance of an incomplete related-work discussion.","section":"Section 2, Ref. [15]"},{"comment":"The notation \"RGB-T person pairs\" in Eq. (1) is undefined; the authors should clarify whether these are ground-truth identity pairs, matched detections, or something else, and how the denominator is computed when the two modalities contain different numbers of detected persons.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads like a short-conference or workshop paper rather than a full journal article. The load-bearing issue is Section 3.3/Table 1: the paper never defines how matching correspondences become the detections on which AP is computed, so the central quantitative claim is not verifiable as written. The evaluation is also under-powered: two private 100-pair sets, one baseline, and no statistical controls. I would not reject the idea itself, since the language-driven approach is interesting and the qualitative examples are suggestive, but the authors need to specify the fusion-to-detection step, release or fully describe the datasets, and provide a more thorough comparison before this can be accepted. I also noted a pattern of heavy self-citation to the group's own prior work without comparative evaluation, which should be monitored for positioning but does not by itself affect the verdict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper claims large gains on heavily misaligned RGB-thermal pairs: 74.9 vs 61.6 AP on a FLIR subset, 85.5 vs 75.6 on their collected set. If true, that would remove calibration hardware and preprocessing for multispectral detection, a real practical advance. The problem is genuine; raw uncalibrated multispectral data are common and standard detectors fail on them.\n\nWhat's actually new: they construct positional MST graphs from per-modality detections, use LVLMs to describe each person's appearance, run a multi-LLM debate, and prompt an LLM to output RGB-to-thermal correspondences. This combination is not in the literature I know, and it's a plausible way to handle misalignment without calibration. The AER metric is a reasonable idea for measuring correspondence error, and collecting a hundred hard pairs is useful.\n\nBut there is a load-bearing evaluation problem. The method's final output is explicitly \"the matching results\" (Section 3.3), i.e., a set of correspondences. AP, though, is computed from detection boxes with scores. The paper never says how correspondences become boxes and scores, which ground-truth set is used, or how matched versus unmatched instances are handled. Without that step, the AP numbers in Table 1 are uninterpretable. They could come from RGB-only detections, from filtering the ground truth to matched pairs, or from some other rule. That is not a minor omission; it undercuts the central quantitative claim. ProbEn is a fusion detector, so comparing a matching-only pipeline to it on AP is not well-posed.\n\nOther soft spots: two private 100-pair datasets, no error bars or significance tests, no comparison to image registration or other misalignment-robust baselines, and no code or data. The MST assumption that relative pedestrian positions are approximately preserved across modalities is not stress-tested; it likely breaks under sufficiently different FOVs. AER is not validated against any independent measure. Self-citation to the authors' prior MSCoTDet is present but not the core issue.\n\nWhat's good: the problem is important, the approach is original, and the qualitative examples show plausible matches. But the paper as written is an idea with illustrative results, not a sound empirical demonstration. A reader working on LVLM grounding or cross-modal matching might get useful prompts; for multispectral detection, the numbers shouldn't be trusted.\n\nRecommendation: I would not accept this in current form, but it deserves a serious referee if the authors can define the fusion-to-detection step and run a proper evaluation. Clarify the AP computation, add baselines and error bars, and release code and data.","headline":"Interesting matching idea for heavily misaligned RGB-thermal data, but the central AP claims are unsupported because the paper never defines how correspondences become detection boxes and scores.","tokens_in":8389,"tokens_out":3471,"would_cite":false,"duration_ms":31332,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a language-driven cross-modal alignment method that makes multispectral pedestrian detection work on raw, uncalibrated RGB-thermal camera pairs.","keywords":["multispectral pedestrian detection","RGB-thermal misalignment","large vision-language models","cross-modal alignment","minimum spanning tree","alignment error rate","late fusion","semantic matching"],"falsifier":"Take a set of RGB-thermal pairs with intentionally varied camera baselines or zoom factors, run the proposed method, and measure AER; if the alignment error climbs steeply as the nearest-neighbor structure diverges, the MST-preservation assumption is falsified. Alternatively, replace the LVLM appearance descriptions with random textual attributes and keep the positional graphs; if the matching performance stays the same, the contribution of the language-driven appearance cue is not causal.","tokens_in":7451,"feed_emoji":"🚶","tokens_out":6229,"duration_ms":52363,"temperature":0.7,"pith_summary":"This paper tackles the practical problem of multispectral pedestrian detection when the RGB and thermal cameras are not calibrated, so the same person appears at different positions in the two images. Existing fusion methods assume the two views are largely overlapping and break down under heavy misalignment. The proposed framework first builds a minimum spanning tree of detected pedestrians in each modality to capture their relative spatial layout, then uses large vision-language models to describe each person's appearance, and finally prompts an LLM to fuse both cues and output cross-modal matches. On two heavily misaligned test sets, the method raises detection AP over the ProbEn baseline and reduces a new alignment error rate from over 80 percent to under 10 percent on the FLIR set. If the results hold, accurate multispectral detection becomes possible on raw data from uncalibrated cameras, removing the need for beam splitters, checkerboards, or image registration.","feed_headline":"Language-driven fusion cuts RGB-thermal alignment error to under 9%","feed_subtitle":"Spatial graphs plus language descriptions match people across uncalibrated RGB and thermal cameras, no calibration hardware needed.","key_machinery":"The method has four load-bearing components. Positional graphs: each modality's single-modal detections (from Co-DETR) become nodes, and Kruskal's algorithm builds a minimum spanning tree connecting each person to its nearest neighbours, so the graph encodes relative spatial layout rather than absolute coordinates. Appearance embeddings: each detected person is zoomed and cropped, and an LVLM (e.g., GPT-4, Gemini, Claude 2) produces a structured textual description of facing, clothing, accessories, and action; this text is attached to the corresponding graph node. Multi-LLM debate: several LVLMs propose descriptions and a judge model (GPT-4) synthesizes a consensus, mitigating hallucination. LLM prediction: the final prompt supplies both the MST structure and the appearance texts and asks the LLM to output a cross-modal matching, with a rationale. The paper also introduces the alignment error rate (AER), defined as the fraction of RGB-thermal person pairs that are mismatched, as a new evaluation metric for heavily misaligned data.","core_discovery":"The central claim is that heavy RGB-thermal misalignment can be resolved at the semantic level rather than the geometric level: instead of aligning pixels or feature maps, the method aligns identities. It detects pedestrians independently in each modality, encodes the spatial configuration of each modality as a minimum spanning tree whose edges are inter-person distances, and augments each node with LVLM-generated textual descriptions of appearance and action. A large language model then reads both graphs and matches nodes across modalities using positional and appearance agreement. The paper reports that this approach outperforms the probabilistic ensembling baseline ProbEn on both a modified FLIR ADAS set and a newly collected set of challenging uncalibrated pairs, and that adding a multi-LLM debate before the final judgement further improves matching, achieving an alignment error rate of 8.8 percent on the FLIR set.","pith_inferences":["The success of the method hinges on the MST structure being roughly preserved across views; a natural stress test is to measure AER as the field-of-view difference or baseline between cameras grows, since the paper does not quantify how much positional distortion the graph matching tolerates.","Because the matching is performed by an LLM on textual descriptions, the approach inherits the cost, latency, and hallucination risk of LVLMs; replacing the debate with a cheaper single-model or learned matcher could be a practical extension the authors do not explore.","The same semantic-alignment recipe could apply to other modalities whose images share physical layout but not appearance, such as visible-thermal re-identification, or even visible-depth matching, where positional graphs and language descriptions could serve as the common representation.","The paper's AER metric counts only matched person pairs; extending it to penalize false matches or missed detections might change the ranking of methods when detectors are imperfect."],"forward_implications":["Multispectral pedestrian detectors can be deployed directly on raw, uncalibrated camera feeds, since the proposed method requires no beam splitters, checkerboard calibration, or image registration preprocessing.","Fusion no longer depends on overlapping regions of interest between modalities; the method matches identities via spatial layout and appearance, so it can handle cases where a person appears at very different locations in RGB and thermal.","The alignment error rate (AER) provides a concrete, repeatable measure of cross-modal matching quality that can be used to compare future methods on heavily misaligned data.","The framework's ability to correct misalignment at inference time suggests it could be extended to other multispectral tasks, such as tracking or re-identification, without retraining the fusion pipeline per camera pair."],"supporting_citations":[{"why":"Supplies the baseline ProbEn late-fusion model that the proposed method must outperform on heavily misaligned data.","marker":"[24]"},{"why":"Supplies the Co-DETR detector used to obtain single-modal pedestrian bounding boxes in each modality for graph construction.","marker":"[8]"},{"why":"Supplies GPT-4, used both as one of the LVLMs generating appearance descriptions and as the judge model that synthesizes the debate.","marker":"[25]"},{"why":"Supplies Gemini, another LVLM used in the multi-model debate to cross-validate appearance descriptions.","marker":"[26]"},{"why":"Represents prior feature-level alignment work for weakly aligned data, which motivates the need for a method that handles heavy misalignment.","marker":"[21]"}],"fun_headline_variants":["Language models align RGB-thermal without calibration","AI matches pedestrians across unaligned infrared and RGB","LVLM-driven fusion cuts misalignment to under 9%","Semantic alignment replaces geometric calibration for multispectral detection","Text descriptions fuse RGB and thermal cameras for accurate detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that the relative positions and distances of pedestrians are roughly preserved between the two misaligned views, so the nearest-neighbor connection structure of people in one image resembles the other; if the cameras have very different fields of view or viewing angles, this structural similarity breaks down and the matching degrades.","fun_headline_variants_meta":{"raw":{"variants":["Language models align RGB-thermal without calibration","AI matches pedestrians across unaligned infrared and RGB","LVLM-driven fusion cuts misalignment to under 9%","Semantic alignment replaces geometric calibration for multispectral detection","Text descriptions fuse RGB and thermal cameras for accurate detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2858,"prompt_tokens":848,"completion_tokens":2010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1934}},"tokens_in":464,"tokens_out":2010,"duration_ms":14255,"temperature":1.0,"reasoning_tokens":1934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:36:36.635433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of RGB-thermal pairs with intentionally varied camera baselines or zoom factors, run the proposed method, and measure AER; if the alignment error climbs steeply as the nearest-neighbor structure diverges, the MST-preservation assumption is falsified. Alternatively, replace the LVLM appearance descriptions with random textual attributes and keep the positional graphs; if the matching performance stays the same, the contribution of the language-driven appearance cue is not causal.","supporting_citations":[{"cited_title":"An iterative integrated frame- work for thermal–visible image registration, sensor fu- sion, and people tracking for video surveillance applica- tions,","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline ProbEn late-fusion model that the proposed method must outperform on heavily misaligned data."},{"cited_title":"Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning","cited_arxiv_id":"2005.10987","evidence_quote":"Supplies the Co-DETR detector used to obtain single-modal pedestrian bounding boxes in each modality for graph construction."},{"cited_title":"Attentive alignment network for multispectral pedestrian detection,","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4, used both as one of the LVLMs generating appearance descriptions and as the judge model that synthesizes the debate."},{"cited_title":"Weakly aligned cross-modal learning for multispectral pedestrian detection,","cited_arxiv_id":null,"evidence_quote":"Supplies Gemini, another LVLM used in the multi-model debate to cross-validate appearance descriptions."},{"cited_title":"A survey of image registration techniques,","cited_arxiv_id":null,"evidence_quote":"Represents prior feature-level alignment work for weakly aligned data, which motivates the need for a method that handles heavy misalignment."}],"review_version":1}