{"id":"a39d0b4d-9a9e-4d30-849a-a17b8610d300","arxiv_id":"2412.00622","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ModPrompt adapts vision-language object detectors to infrared and depth data with an input-dependent visual prompt and a decoupled text-embedding residual, without updating the frozen detector.","lead":"This paper introduces ModPrompt, a visual prompt method that translates infrared and depth images into pseudo-RGB inputs so frozen vision-language object detectors can detect objects in these modalities. The method keeps the detector's original zero-shot capabilities while approaching full fine-tuning performance on some benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing MPDR-only control leaves the visual-prompt contribution unattributed, and ModPrompt numbers differ across Tables 1, 2, and 4 for the same YOLO-World LLVIP setting.","rationale":"I read the central claim as: a frozen vision-language detector can be adapted by a learned input translation plus MPDR, with gains near full fine-tuning on infrared while preserving zero-shot knowledge. This is plausible, and the paper provides broad benchmarking, ablations, and code. However, the evidence for the visual-prompt mechanism specifically is incomplete because MPDR is always trained jointly, and Table 2 shows MPDR substantially boosts even weak visual prompts. The absent 'MPDR-only' control is directly load-bearing: if the text residual alone recovers most of the gain, then the encoder-decoder visual prompt is not the source of improvement. The cross-table inconsistencies (92.80 vs 95.63 vs 96.60) make it difficult to know which configuration supports the knowledge-preservation claim. These issues do not falsify the approach, but they make the attribution conditional on a missing experiment and on reproducible reporting. The reader's weakest assumption about whether the detection loss provides enough gradient signal is related but not identical; my concern is about separating the contributions of the two trainable components once that gradient signal is assumed to be usable.","tokens_in":19909,"tokens_out":8071,"duration_ms":80628,"concrete_test":"Run the released code with h(x) fixed to zero (identity visual prompt), train the MPDR text residual with the same detection loss, optimizer, epochs, and class vocabulary, and evaluate YOLO-World and Grounding DINO on LLVIP, FLIR, and NYUv2. Report AP50/AP75/AP; if MPDR-only reaches within 2 AP of ModPrompt+MPDR on any benchmark, the visual prompt is not the driver. Separately, rerun the YOLO-World-S LLVIP configuration and state which variant (MB or RES, MPDR on/off, model version) yields 92.80, 95.63, and 96.60; if the three numbers cannot be reproduced under documented configurations, the reported magnitudes are unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central attribution claim is that the learned input translation hϑ (Eq. 3), not the text residual, is what adapts the frozen detector. This is not yet established because MPDR is always trained jointly with the visual prompt, and Table 2 shows that adding MPDR to weak visual-prompt baselines produces large gains (YOLO-World LLVIP AP50: Fixed 70.30→86.60, Random 60.13→86.43, Padding 79.87→83.57). If MPDR alone, with the original zero-shot image (h(x)=0), reaches comparable AP, then the encoder-decoder visual prompt contributes little. No such row is reported. The inconsistency of reported ModPrompt numbers compounds this: Table 1 lists 92.80 AP50 for YOLO-World on LLVIP, Table 2 lists 96.60 with MPDR, and Table 4 reports 95.63 for 'ModPrompt' without a configuration label, while §4.2 repeats 92.80. Table 4's average 73.77 is computed from 95.63, not from 92.80, so the claim of preserved zero-shot knowledge with +7.32 rests on a number that appears in no other table. Without reconciling these numbers or providing the missing control, the quantitative headline cannot be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ModPrompt, an encoder-decoder visual prompt that transforms input images from a target modality (infrared or depth) into a pseudo-RGB representation fed to a frozen vision-language object detector (YOLO-World or Grounding DINO), trained end-to-end with the detection loss. It additionally introduces MPDR, a learnable residual added to precomputed text embeddings, intended to preserve the detector's zero-shot text knowledge while adapting to the new modality. Experiments on LLVIP, FLIR, and NYUv2 compare ModPrompt against zero-shot, head/full fine-tuning, and several pixel-level visual prompt baselines, and report that ModPrompt achieves performance \"comparable to full fine-tuning\" while preserving zero-shot capability. The paper also compares against prior modality-translation methods HalluciDet and ModTr.","tokens_in":20240,"tokens_out":3557,"duration_ms":31783,"significance":"If the reported results hold, the paper addresses a meaningful gap: adapting modern vision-language object detectors to new visual modalities without losing their open-vocabulary zero-shot abilities. The approach is detector-agnostic, is evaluated on two detector families and three datasets, and the authors release code, which are concrete strengths. The notion of an input-dependent, detection-loss-driven visual prompt for VLMs is a reasonable and potentially useful contribution. However, the current manuscript contains numerical inconsistencies across tables, an omitted control that prevents attributing gains to the visual prompt, and an under-specified text-residual component; these issues currently block a reliable assessment of the central claims.","major_comments":[{"comment":"The headline result for YOLO-World on LLVIP is not consistently reported: Table 1 gives ModPrompt AP50 92.80, Table 2 gives 96.60, and Table 4 gives 95.63. Table 4's average 73.77 and the +7.32 gain over zero-shot are computed from 95.63, a value that appears in no other table. §4.2 repeats 92.80. This inconsistency directly affects the abstract's claim of \"performance comparable to full fine-tuning,\" and the NYUv2 results in Table 1 (AP 24.93 vs FT 33.57) show a gap of 8.64 AP, which is not \"comparable\" in the usual sense. The authors must reconcile these numbers and qualify the claim accordingly.","section":"§4.2, Tables 1, 2, and 4"},{"comment":"The contribution of the learned encoder-decoder prompt hϑ from Eq. (3) is not isolated. Table 2 shows that adding MPDR to weak visual-prompt baselines produces large gains (e.g., YOLO-World LLVIP AP50: Fixed 70.30→86.60, Random 60.13→86.43, Padding 79.87→83.57). Since MPDR is trained jointly with ModPrompt in all reported configurations, the observed gains may be largely due to the text residual rather than the visual translation that the paper's central claim emphasizes. A control experiment with MPDR alone applied to the original zero-shot image (i.e., hϑ(x)=0) is necessary to attribute the improvement to the visual prompt component.","section":"§4.2, Table 2"},{"comment":"The MPDR objective is under-specified. Eq. (4) defines the final cost as Cmp(ϑ) + Ctp(ϕ) with Ctp defined in the supplementary material on text inputs, but §3.4 states that instead of the online text cost, the method works on the text-embedding space. The actual training objective for the residual added to precomputed embeddings is never written down, nor is the zero-masking deactivation operation defined formally. Without this, the reader cannot determine what exactly is optimized, how the residual is scaled, or why zero-masking recovers the original embedding. A precise formulation is needed for reproducibility.","section":"§3.4, Eq. (4)"},{"comment":"The input-domain constraint is unclear. The text says the decoder's last layer is constrained to produce values in [0,1], but the prompted input is x + hϑ(x). If x is an image with pixel values in [0,1], the sum can exceed 1; if x is in [0,255], the sum is even further from the expected input range. No normalization or clipping is specified. Since the frozen detector was trained on RGB images, the distribution of x + hϑ(x) is load-bearing for the method's validity. Please specify how the summed input is normalized or why the out-of-range values are unproblematic.","section":"§3.2, Eq. (3)"}],"minor_comments":[{"comment":"The abbreviation is inconsistent: the paragraph \"MPDR Knowledge Preservation mechanism\" twice writes \"MDPR\" instead of \"MPDR.\"","section":"§4.3"},{"comment":"The text says \"Figure 3 shows the visualization of ModPrompt on YOLO-World,\" but Figure 3 in the main text is the text-prompt tuning schematic; the qualitative detection examples appear to be Figure 5. The reference should be corrected.","section":"§4.4"},{"comment":"The ablation section says \"we tested various patch sizes for each of the visual prompt methods and reported the performance in Table 8,\" but the main-text table with these variations is Table 3; Table 8 is in the supplementary material. This cross-reference is confusing.","section":"§4.3"},{"comment":"The sentence reporting the direct-embedding adaptation without MPDR states that performance was \"close to MDPR\" and zero-shot knowledge was lost, but gives no numbers. Since this directly motivates the MPDR design, a quantitative comparison belongs in the main text or a clearly referenced supplementary table.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The numerical inconsistencies across Tables 1, 2, and 4 are the most serious issue; they must be reconciled before the paper can be evaluated. The missing MPDR-only control is also essential for the paper's attribution claim. The paper's zero-shot preservation on COCO in Table 4 is partly by construction because the detector is frozen; this should not be oversold as a learned property of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the combination: an input-dependent encoder-decoder visual prompt trained with detection loss, plus a decoupled text-embedding residual, applied to frozen vision-language detectors for infrared and depth. I have not seen that exact package before. The paper also does something useful on the empirical side: it benchmarks a family of pixel-level prompts across two detectors and three datasets, and it reports FLIR results in the supplement rather than hiding them. The zero-shot preservation claim is legitimate by construction: the detector weights are frozen, so COCO performance cannot move. That is a design property, not a circular argument.\n\nNow the soft spots, in proportion. First, the abstract overstates the depth results. On NYUv2, YOLO-World AP is 24.93 versus 33.57 for full fine-tuning, and Grounding DINO is 14.13 versus 35.77. That is not \"comparable to full fine-tuning.\" The claim should be restricted to the infrared benchmarks, or at least carefully qualified.\n\nSecond, the MPDR component is under-specified and its contribution is unattributed. Equation 4 references a text-prompt cost that is then replaced by an embedding-space procedure in Section 3.4, and the formal definition in the supplement (Eq. 5) does not match what the main text describes. More importantly, Table 2 shows that adding MPDR to weak visual prompts produces very large gains (YOLO-World LLVIP AP50: Fixed 70.30 to 86.60, Random 60.13 to 86.43). Without a row for MPDR alone on the original zero-shot image, the reader cannot tell whether the encoder-decoder visual prompt is doing the heavy lifting or whether the text residual alone accounts for most of the improvement. That missing control is load-bearing and should be supplied.\n\nThird, the numbers are internally inconsistent. Table 1 lists YOLO-World LLVIP AP50 as 92.80; Table 2 lists 96.60 with MPDR; Table 4 reports 95.63 for ModPrompt without a configuration label; and the text in Section 4.2 repeats 92.80. Table 4's average is computed from 95.63, not from 92.80. The same method name should not refer to different configurations across tables without explicit labels. This is fixable, but it has to be fixed before the quantitative headline can be assessed.\n\nThe limitations section is honest about small objects, duplicated boxes, and the need for target labels. The code is promised and the approach is reproducible in principle. This is not a fundamentally flawed idea; it is an empirical paper that is currently too sloppy in its reporting and missing a key ablation.\n\nRecommendation: send it to peer review, but with a major-revision expectation. The referees should ask for the MPDR-only control, a cleaned-up set of configuration labels, and a rewritten abstract that says where the method is actually comparable to full fine-tuning. If those land, this becomes a solid empirical contribution for people adapting frozen VLM detectors to non-RGB modalities.","headline":"A plausible parameter-efficient modality-adaptation method for frozen vision-language detectors, but the reported numbers are inconsistent and the attribution of the gains is not yet established.","tokens_in":20726,"tokens_out":2170,"would_cite":false,"duration_ms":23156,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ModPrompt claims that a small image-conditioned encoder-decoder, added to the input of a frozen vision-language detector, can translate infrared and depth images into a representation the detector handles well, reaching near…","keywords":["visual prompt","modality adaptation","open-vocabulary object detection","infrared detection","depth detection","zero-shot knowledge preservation","encoder-decoder prompt","text embedding residual"],"falsifier":"Train ModPrompt on a target dataset with the ground-truth box labels randomly shuffled or replaced by constant boxes; if the reported gains over zero-shot persist, the improvement is not coming from detection-supervised translation, and the central mechanism fails. Alternatively, evaluate the adapted detector on the original COCO zero-shot set with MPDR zero-masked; any drop below the untrained zero-shot numbers would contradict the knowledge-preservation claim.","tokens_in":1845,"feed_emoji":"🖼️","tokens_out":2777,"duration_ms":69929,"temperature":0.7,"pith_summary":"Open-vocabulary object detectors trained on ordinary RGB images lose accuracy when tested on infrared or depth images, and fine-tuning them on the new modality erases their ability to detect the original categories. This paper proposes ModPrompt, a way to adapt such detectors by learning a small image-to-image translator that is added to each input image while every weight of the detector stays frozen. The translator is trained only by the detector's own detection loss, so no reconstruction or alignment between modalities is needed. On infrared benchmarks the results approach full fine-tuning, and on depth data the gains are large, while the detector's zero-shot performance on RGB is preserved exactly. A companion text-side residual, called MPDR, tunes the class embeddings without overwriting the originals, which can be restored by zero-masking at test time.","feed_headline":"Learned image prompt adapts frozen detectors to infrared and depth","feed_subtitle":"A small input translator gives near full-fine-tuning detection accuracy while keeping zero-shot knowledge.","key_machinery":"The central object is the input-conditioned visual prompt $h_\\vartheta(x)$: an encoder-decoder network shaped like a U-Net whose output is constrained to three channels in $[0,1]$, trained solely by the frozen detector's detection loss once it is added to the image. The key identity is the additive combination $x + h_\\vartheta(x)$ used as the detector input, with no auxiliary reconstruction or alignment loss. The second piece is MPDR, a learnable residual vector added to precomputed text embeddings for the target classes; it is trained together with $h_\\vartheta$ but kept separate from the frozen embeddings so the original zero-shot text knowledge survives and can be restored by zero-masking. Static prompts fail, the paper argues, because they apply the same translation to every image; conditioning the prompt on the input is what lets the translation handle large modality shifts.","core_discovery":"The paper claims that the right place to adapt a vision-language detector to a new visual modality is the input image itself: a U-Net-style encoder-decoder $h_\\vartheta$ reads the target-modality image $x$ and produces a pseudo-RGB prompt that is added to $x$, and the frozen detector $f_\\theta$ is then scored against the ground-truth boxes with the ordinary detection loss $L_{\\mathrm{det}}(f_\\theta(x + h_\\vartheta(x)), Y)$. Because the prompt is conditioned on each input image rather than being a fixed patch, it can translate infrared or depth content into a form the RGB-pre-trained detector can interpret. The paper further claims that adapting the text branch as a decoupled residual (MPDR) added to the frozen class embeddings improves the target-modality results and, unlike direct embedding tuning, does not destroy the original zero-shot knowledge, since zero-masking the residual restores the original embeddings. Empirically the paper reports that on the LLVIP infrared dataset with YOLO-World, ModPrompt reaches 92.80 AP50 against 81.00 for zero-shot and 97.43 for full fine-tuning, and on NYUv2 depth it raises AP50 from 4.80 to 37.17, with corresponding gains for Grounding DINO.","pith_inferences":["Beyond the paper: the same detection-loss-only recipe should extend to other large modality gaps, such as radar or medical imaging, because Equation 3 does not depend on infrared or depth specifically.","Beyond the paper: because the prompt is additive and the detector stays frozen, ModPrompt could be combined with test-time adaptation or unsupervised losses to reduce the need for target labels, an extension the authors list as future work.","Beyond the paper: the near-full-fine-tuning result on LLVIP suggests that RGB pre-training already encodes enough structure for infrared pedestrian detection, so even lighter prompt backbones may suffice, consistent with the MobileNet-ResNet ablation in the paper.","Beyond the paper: MPDR-style decoupled residual tuning could be ported to other vision-language tasks, such as classification or retrieval, whenever per-domain adaptation must not overwrite the original embeddings."],"forward_implications":["With only about 3.08 million trainable parameters, ModPrompt on YOLO-World reaches LLVIP infrared AP50 of 92.80, up from 81.00 zero-shot and within about 4.6 points of full fine-tuning at 97.43, while keeping COCO zero-shot AP50 at 51.90.","On NYUv2 depth, ModPrompt raises YOLO-World AP50 from 4.80 to 37.17, a large improvement over zero-shot, though still below the full-fine-tuning level of 49.90.","MPDR adds gains for most of the tested visual prompt strategies while preserving the original knowledge, and zero-masking it at inference returns the detector to its exact zero-shot text embeddings.","ModPrompt is backbone-agnostic: it works on both the CNN-based YOLO-World and the transformer-based Grounding DINO with the same additive prompt recipe.","Static visual prompts such as fixed, random, padding, and weight-map patches can perform worse than zero-shot on large modality shifts, which supports the paper's claim that input-conditioned prompts are necessary for modality adaptation."],"supporting_citations":[{"why":"Supplies the YOLO-World open-vocabulary detector that ModPrompt adapts and benchmarks.","marker":"[6]"},{"why":"Supplies the Grounding DINO detector used as the second adaptation target.","marker":"[29]"},{"why":"Supplies the U-Net encoder-decoder structure that ModPrompt turns into an image-conditioned input translator.","marker":"[40]"},{"why":"Supplies the conditional-prompt idea of adapting through a small network, which ModPrompt transfers from text to the visual input.","marker":"[53]"},{"why":"Supplies the task-residual mechanism that MPDR adapts for text embeddings of detectors.","marker":"[47]"},{"why":"Represents the fixed linear visual-prompt baseline that fails on large modality shifts.","marker":"[1]"},{"why":"Supplies visual prompt tuning baselines and the parameter-efficient adaptation framing.","marker":"[19]"},{"why":"Provides a prior detection-loss-guided modality translator that ModPrompt outperforms on infrared benchmarks.","marker":"[33]"},{"why":"Provides a prior modality translation method that preserves pre-trained knowledge and that ModPrompt extends to vision-language detectors.","marker":"[31]"}],"fun_headline_variants":["Image-conditional prompt adapts VL detectors to IR and depth","ModPrompt: input translator boosts zero-shot to near fine-tune","U-Net prompt swaps modality without losing zero-shot knowledge","Frozen detectors see IR and depth via learned pixel translation","Conditional visual prompt keeps zero-shot while adding modal ability"],"cache_read_input_tokens":22912,"weakest_assumption_plain":"The load-bearing premise is that the frozen detector's detection loss, computed on the summed image, sends a strong enough training signal to teach the prompt network to translate the new modality, even though no reconstruction or alignment loss is used and the summed image is not guaranteed to look like an ordinary RGB image.","fun_headline_variants_meta":{"raw":{"variants":["Image-conditional prompt adapts VL detectors to IR and depth","ModPrompt: input translator boosts zero-shot to near fine-tune","U-Net prompt swaps modality without losing zero-shot knowledge","Frozen detectors see IR and depth via learned pixel translation","Conditional visual prompt keeps zero-shot while adding modal ability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1659,"prompt_tokens":1048,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":527}},"tokens_in":664,"tokens_out":611,"duration_ms":9429,"temperature":1.0,"reasoning_tokens":527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:09:57.239955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ModPrompt on a target dataset with the ground-truth box labels randomly shuffled or replaced by constant boxes; if the reported gains over zero-shot persist, the improvement is not coming from detection-supervised translation, and the central mechanism fails. Alternatively, evaluate the adapted detector on the original COCO zero-shot set with MPDR zero-masked; any drop below the untrained zero-shot numbers would contradict the knowledge-preservation claim.","supporting_citations":[{"cited_title":"Yolo-world: Real-time open-vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO-World open-vocabulary detector that ModPrompt adapts and benchmarks."},{"cited_title":"Task residual for tuning vision-language models","cited_arxiv_id":null,"evidence_quote":"Supplies the task-residual mechanism that MPDR adapts for text embeddings of detectors."},{"cited_title":"Hallucidet: Hallucinating rgb modality for person de- tection through privileged information","cited_arxiv_id":null,"evidence_quote":"Provides a prior detection-loss-guided modality translator that ModPrompt outperforms on infrared benchmarks."},{"cited_title":"Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge","cited_arxiv_id":"2404.01492","evidence_quote":"Provides a prior modality translation method that preserves pre-trained knowledge and that ModPrompt extends to vision-language detectors."}],"review_version":1}