MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality
read the original abstract
Recently, prompt learning has garnered considerable attention for its success in various Vision-Language (VL) tasks. However, existing prompt-based models are primarily focused on studying prompt generation and prompt strategies with complete modality settings, which does not accurately reflect real-world scenarios where partial modality information may be missing. In this paper, we present the first comprehensive investigation into prompt learning behavior when modalities are incomplete, revealing the high sensitivity of prompt-based models to missing modalities. To this end, we propose a novel Multi-step Adaptive Prompt Learning (MuAP) framework, aiming to generate multimodal prompts and perform multi-step prompt tuning, which adaptively learns knowledge by iteratively aligning modalities. Specifically, we generate multimodal prompts for each modality and devise prompt strategies to integrate them into the Transformer model. Subsequently, we sequentially perform prompt tuning from single-stage and alignment-stage, allowing each modality-prompt to be autonomously and adaptively learned, thereby mitigating the imbalance issue caused by only textual prompts that are learnable in previous works. Extensive experiments demonstrate the effectiveness of our MuAP and this model achieves significant improvements compared to the state-of-the-art on all benchmark datasets
This paper has not been read by Pith yet.
Forward citations
Cited by 10 Pith papers
-
Resolving Ambiguity in Composed Image Retrieval via Calibrated Interaction
Composed image retrieval is reframed as calibrated intent resolution under uncertainty via conformal prediction sets and expected-information-gain clarification, with new AmbiCIR benchmark showing matched single-turn ...
-
RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses
RelWitness introduces relation witnesses from visual and geometric cues to learn open-vocabulary 3D scene graphs under incomplete supervision using a positive-unlabeled objective.
-
Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?
C²R improves robust accuracy in distilled datasets by 2.8% on average by coupling an attack-aware margin-based curriculum with a class-balanced contrastive robustness objective.
-
ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection
ScriptHOI decomposes HOI phrases into state slots and uses script coverage, conflict, interval partial-label learning, and counterfactual contrast to improve rare and unseen interaction detection while cutting afforda...
-
Show, Don't Ask: Generative Visual Disambiguation for Composed Image Retrieval with Turn-Valid Coverage
CLARA achieves turn-valid conformal coverage in ambiguous composed image retrieval by replacing text clarification with user selection among snapped real-image prototypes and reweighting calibration accordingly.
-
RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses
RelWitness uses concrete visual-geometric cues to verify and learn from missing relation labels in open-vocabulary 3D scene graph generation.
-
RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses
RelWitness introduces relation witnesses as observable visual-geometric cues to classify unannotated relations and enable positive-unlabeled learning for open-vocabulary 3D scene graph generation.
-
Mind Your Margin and Boundary: Are Your Distilled Datasets Truly Robust?
C²R framework for robust dataset distillation prioritizes small-margin adversaries via a derived perturbation score and widens class boundaries with contrastive loss, yielding 2.8% average robust accuracy gains on CIF...
-
ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection
ScriptHOI decomposes HOI phrases into state slots, uses slot-wise script coverage and conflict matching, and applies interval partial-label learning to improve rare and unseen interaction detection.
-
ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection
ScriptHOI improves rare and unseen HOI recognition by decomposing phrases into state slots, using visual tokenization and slot-wise matching for script coverage and conflict to calibrate predictions and constrain trai...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.