MSLA is the first physically deployable attack that uses adversarial lighting to break semantic alignment in VLMs such as CLIP, LLaVA, and BLIP, causing classification failures and hallucinations in real scenes.
super hub Mixed citations
Microsoft COCO Captions: Data Collection and Evaluation Server
Mixed citation behavior. Most common role is background (53%).
108
Pith papers citing it
1,634
external citations · Pith
Background
53% of classified citations
abstract
In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.
hub tools
citation-role summary
background 15
dataset 15
citation-polarity summary
claims ledger
- abstract In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.
- dataset sertion scores indicate that the highlighted regions have a stronger causal effect on the model's responses. We conduct experiments using Qwen2.5-VL-3B as the VLM, and compare our method with several baselines, in- cluding CAM [97], Grad-CAM [57], raw attention, atten- tion rollout [1], ATTN-LRP [2], and TAM [40]. The eval- uation is performed on three datasets, namely the COCO Caption dataset [13], GranDf [54], and OpenPSG [98]. For each dataset, we sample 1k images for evaluation. The re- sult
- dataset This approach allows us to benefit from readily-available image instruction data, creating a system capable of handling both images and videos with shared spatial perception and reasoning capacity. Stage1: Alignment. To strike a balance between training convergency and efficiency we introduce 25M vision-text pairs for one epoch of fine-tuning, The data consists 10M video-text pairs from WebVid-10M, and 15M image-text pairs from COCO Caption [ 6], Visual Genome [ 17], SBU Captions [31], CC3M [35]
- dataset High-Quality Bilingual Dataset Pre-training Dataset. The pre-training dataset utilized in our InternVL 1.5 encompasses a diverse range of pub- licly accessible sources. We provide an overview of these datasets in Table 1a. These datasets span multi- 4 task ratio dataset Laion-EN (en) [93], Laion-ZH (zh) [93], COYO (zh) [10],Captioning 53.9% GRIT (zh) [90], COCO (en) [17], TextCaps (en) [99] Objects365 (en&zh) [97], GRIT (en&zh) [90],Detection 5.2% All-Seeing (en&zh) [119] Wukong-OCR (zh) [29], L
- dataset [92] Mustafa Shukor, Maxime Oquab, Ishan Misra, and Enrico Fini. Scaling laws for native multimodal models. arXiv preprint arXiv:2504.07951, 2025. [93] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll'ar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.ArXiv, abs/1504.00325, 2015. [94] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for au
- dataset Inthedistance,treescanbeseen,addingatouchofnaturetothisman-madesetting.Theimageisasnapshotofeverydaylifeatatrainstation,capturingbothitsroutineoperationsanditsinherentcharm. (a) Comparison of Captions' Quality(b) Comparison of Performance Figure 1. (a) We showcase a comparison between the caption in our proposed ShareGPT4V dataset and those utilized by recent large multi-modal models (LMMs). Unlike COCO-Caption [7] involves brief human-made captions on the main subject. LLaV A-Instruct [31] comb
- dataset modeling for long-term temporal reasoning [186], and audio language modeling based on Whisper-like or speech- oriented frameworks [187]. Systems such as Flamingo-style 4 TABLE I Comparative evaluation on standard open-end zero-shot VideoQA benchmarks. This table shows the accuracy scores for all models on MSVD-QA, MSRVTT-QA, and ActivityNet-QA. Model MSVD-QA [170] MSRVTT-QA [171] ActivityNet-QA [172] FAVOR [173] 67.8 59.3 / Dolphin [174] 72.7 62.6 49.1 OneLLM [175] 56.5 53.8 / Video-SALMONN [176
authors
co-cited works
representative citing papers
Molmo VLMs trained on newly collected PixMo open datasets achieve state-of-the-art performance among open-weight models and surpass multiple proprietary VLMs including Claude 3.5 Sonnet and Gemini 1.5 Pro.
The paper introduces #PraCegoVer, the first large-scale image captioning dataset in Portuguese sourced from Instagram posts with single user-generated captions per image.
MTLA is a training-free, post-hoc confidence score for multimodal LLM localization that restricts attention aggregation to the model's own predicted region and tokens, substantially improving hallucination detection and re-ranking across image, video, and audio.
GRIP uses contrastive training on LMM feedback to retrieve beneficial in-context examples for multimodal tasks, outperforming similarity-based methods and transferring across models including GPT-4o.
Introduces a benchmark dataset with automated pipeline for evaluating VLMs on dynamic in-context human preferences, distinct from static benchmarks.
TwinQuant learns quantization-friendly subspaces for 4-bit LLM weights via manifold optimization and a fused kernel, preserving near-FP16 accuracy with up to 1.8x speedup on LLaMA3 and Qwen3 models.
Introduces Synergistic Faithfulness metric based on Shapley Interaction Index to evaluate cross-modal synergy in VLM explainers, revealing over-reliance on visual salience in existing methods.
M-ORE decouples text and visual update statistics in MLLMs and applies recursive low-rank edits in an orthogonal subspace to reduce cross-modal conflict and long-horizon interference.
OxyEcomBench is a unified multimodal benchmark covering 6 capability areas and 29 tasks with authentic e-commerce data to measure how well foundation models handle real platform, merchant, and customer challenges.
The paper proves statistical consistency of contrastive loss to optimal ranking via an AUC criterion and derives generalization bounds O(1/m + 1/sqrt(n)) for supervised and O(1/sqrt(m) + 1/sqrt(n)) for self-supervised CRL that explain benefits of large negative sets.
Hierarchical confidence calibration and LoCLIP adaptation improve pseudo-label quality for open-vocabulary object detection, achieving new state-of-the-art results on COCO and LVIS benchmarks.
GaLa uses hypergraph representations of objects and a TriView encoder with contrastive learning to improve vision-language models on procedural planning benchmarks.
Instruction-tuned vision-language model PaveGPT, trained on a large unified pavement dataset, achieves substantial gains over general models in comprehensive, standard-compliant pavement condition assessment.
DetailVerifyBench supplies 1,000 images and densely annotated long captions to evaluate precise hallucination localization in multimodal large language models.
BLS approximates per-sample loss importance via EMA of batch losses, enabling simple and effective dynamic pruning of 20-50% samples losslessly across many datasets and models.
Popular LLM activation steering methods are shown to act as proportional controllers; a PID steering framework is proposed that improves robustness and outperforms baselines in experiments across model families.
HyperCap is the first large-scale hyperspectral captioning dataset built from four benchmark HSI datasets using hybrid automated-manual annotations, with evaluations showing classification gains for vision-language models.
DBAC is a new directional metric for bias amplification in image captions that is less sensitive to sentence encoders and more accurate than LIC, validated on COCO gender and race attributes.
S^2-Bench is a new one-to-many benchmark for natural language-driven molecule generation with three tasks, and OpenMolIns is an instruction dataset enabling Llama3.1-8B to outperform GPT-4o and Claude-3.5 on it.
Adversarial hubs can be generated to be retrieved as top-1 for over 84% of test queries in text-to-image retrieval, far exceeding natural hubs.
Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.
ANCHOR dataset exposes T2I model weaknesses on multi-subject abstractive captions; SAFE uses LLMs for subject extraction and embedding enhancement to improve consistency.
VideoChat integrates video models and LLMs via a learnable interface for chat-based spatiotemporal and causal video reasoning, trained on a new video-centric instruction dataset.
citing papers explorer
-
Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks
MSLA is the first physically deployable attack that uses adversarial lighting to break semantic alignment in VLMs such as CLIP, LLaVA, and BLIP, causing classification failures and hallucinations in real scenes.
-
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo VLMs trained on newly collected PixMo open datasets achieve state-of-the-art performance among open-weight models and surpass multiple proprietary VLMs including Claude 3.5 Sonnet and Gemini 1.5 Pro.
-
#PraCegoVer: A Large Dataset for Image Captioning in Portuguese
The paper introduces #PraCegoVer, the first large-scale image captioning dataset in Portuguese sourced from Instagram posts with single user-generated captions per image.
-
Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention
MTLA is a training-free, post-hoc confidence score for multimodal LLM localization that restricts attention aggregation to the model's own predicted region and tokens, substantially improving hallucination detection and re-ranking across image, video, and audio.
-
GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
GRIP uses contrastive training on LMM feedback to retrieve beneficial in-context examples for multimodal tasks, outperforming similarity-based methods and transferring across models including GPT-4o.
-
A Dataset for Dynamic Human Preferences for Vision Language Models
Introduces a benchmark dataset with automated pipeline for evaluating VLMs on dynamic in-context human preferences, distinct from static benchmarks.
-
TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization
TwinQuant learns quantization-friendly subspaces for 4-bit LLM weights via manifold optimization and a fused kernel, preserving near-FP16 accuracy with up to 1.8x speedup on LLaMA3 and Qwen3 models.
-
Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
Introduces Synergistic Faithfulness metric based on Shapley Interaction Index to evaluate cross-modal synergy in VLM explainers, revealing over-reliance on visual salience in existing methods.
-
Modality-Decoupled Online Recursive Editing
M-ORE decouples text and visual update statistics in MLLMs and applies recursive low-rank edits in an orthogonal subspace to reduce cross-modal conflict and long-horizon interference.
-
OxyEcomBench: Benchmarking Multimodal Foundation Models across E-Commerce Ecosystems
OxyEcomBench is a unified multimodal benchmark covering 6 capability areas and 29 tasks with authentic e-commerce data to measure how well foundation models handle real platform, merchant, and customer challenges.
-
Statistical Consistency and Generalization of Contrastive Representation Learning
The paper proves statistical consistency of contrastive loss to optimal ranking via an AUC criterion and derives generalization bounds O(1/m + 1/sqrt(n)) for supervised and O(1/sqrt(m) + 1/sqrt(n)) for self-supervised CRL that explain benefits of large negative sets.
-
Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection
Hierarchical confidence calibration and LoCLIP adaptation improve pseudo-label quality for open-vocabulary object detection, achieving new state-of-the-art results on COCO and LVIS benchmarks.
-
GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning
GaLa uses hypergraph representations of objects and a TriView encoder with contrastive learning to improve vision-language models on procedural planning benchmarks.
-
Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment
Instruction-tuned vision-language model PaveGPT, trained on a large unified pavement dataset, achieves substantial gains over general models in comprehensive, standard-compliant pavement condition assessment.
-
DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions
DetailVerifyBench supplies 1,000 images and densely annotated long captions to evaluate precise hallucination localization in multimodal large language models.
-
Batch Loss Score for Dynamic Data Pruning
BLS approximates per-sample loss importance via EMA of batch losses, enabling simple and effective dynamic pruning of 20-50% samples losslessly across many datasets and models.
-
Activation Steering with a Feedback Controller
Popular LLM activation steering methods are shown to act as proportional controllers; a PID steering framework is proposed that improves robustness and outperforms baselines in experiments across model families.
-
HyperCap: Hyperspectral Land Cover Captioning Dataset for Vision Language Models
HyperCap is the first large-scale hyperspectral captioning dataset built from four benchmark HSI datasets using hybrid automated-manual annotations, with evaluations showing classification gains for vision-language models.
-
A Woman with a Knife or A Knife with a Woman? Measuring Directional Bias Amplification in Image Captions
DBAC is a new directional metric for bias amplification in image captions that is less sensitive to sentence encoders and more accurate than LIC, validated on COCO gender and race attributes.
-
Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation
S^2-Bench is a new one-to-many benchmark for natural language-driven molecule generation with three tasks, and OpenMolIns is an instruction dataset enabling Llama3.1-8B to outperform GPT-4o and Claude-3.5 on it.
-
Adversarial Hubness in Multi-Modal Retrieval
Adversarial hubs can be generated to be retrieved as top-1 for over 84% of test queries in text-to-image retrieval, far exceeding natural hubs.
-
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.
-
ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis
ANCHOR dataset exposes T2I model weaknesses on multi-subject abstractive captions; SAFE uses LLMs for subject extraction and embedding enhancement to improve consistency.
-
VideoChat: Chat-Centric Video Understanding
VideoChat integrates video models and LLMs via a learnable interface for chat-based spatiotemporal and causal video reasoning, trained on a new video-centric instruction dataset.
-
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI jointly scales a 4B-parameter vision transformer with language models on a new 10B multilingual image-text dataset to reach state-of-the-art results on vision-language tasks while keeping a simple modular design.
-
A Generalist Agent
Gato is a multi-modal, multi-task, multi-embodiment generalist policy using one transformer network to handle text, vision, games, and robotics tasks.
-
Flamingo: a Visual Language Model for Few-Shot Learning
Flamingo models reach new state-of-the-art few-shot results on image and video tasks by bridging frozen vision and language models with cross-attention layers trained on interleaved web-scale data.
-
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Socratic Models compose zero-shot multimodal reasoning by prompting pretrained language and vision models to exchange information and enable new capabilities without finetuning.
-
Gen4U: Unifying Video Generation and Understanding via Diffusion
Frozen video diffusion models, probed at optimal depth and noise levels, produce representations competitive with discriminative encoders across semantic and geometric video tasks in a single forward pass.
-
The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models
Safety-aligned T2I diffusion models exhibit semantic collapse in text embeddings causing TIFA drops; SAGE regularization restores structured utility while retaining safety.
-
Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots
A relative wrist translation bridging action with a vision-language-action model using interleaved tokens and attention masking transfers human manipulation skills to robots more effectively than 6DoF actions.
-
HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning
HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.
-
MIRCaps: A Large-Scale Mixed-Domain Dataset with Image-Level and Region-Level Captions for Fine-Grained Vision-Language Learning
MIRCaps supplies 141,364 images, 981,947 image-level captions, 1,742,264 region-level captions, and 1,391,779 bounding boxes to enable fine-grained vision-language learning across general and surveillance domains.
-
Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
Qwen-RobotWorld is a language-conditioned video world model using Double-Stream MMDiT, an 8.6M-frame embodied corpus, and progressive curriculum training that ranks first on EWMBench and DreamGen Bench.
-
On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study
Systematic experiments reveal that activation steering trades fluency for concept control, is less effective on instruction-tuned models, and that prompting/SFT excel at injection but not removal, with textual metrics correlating to LLM judges.
-
Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs
A new framework improves open-vocabulary object detection on novel categories by using scene graphs for relational modeling and distilling knowledge from image captions.
-
LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent for ANN index optimization outperforms Optuna TPE by 33.3% and VDTuner by 34.2% on the SIEVE metric for HICO-DET retrieval.
-
Zamba2-VL Technical Report
Zamba2-VL is a family of 1.2B–7B hybrid Mamba2-transformer vision-language models that match leading transformer VLMs on image, reasoning, OCR, grounding and counting benchmarks while delivering roughly 10x lower time-to-first-token.
-
InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
InfoAtlas is a pretrained neural model for zero-shot mutual information estimation that matches state-of-the-art accuracy with 100x speedup and handles varying dimensions via a single model.
-
PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding
PARCEL is a new visual tokenization architecture combining pool-anchored resampling with conditioned elastic queries to enhance performance-efficiency tradeoffs in LVLMs over prior matryoshka methods.
-
Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
VCF aligns CLIP image features to text embeddings via a lightweight aligner to enable dual image-text conditioning in Stable Diffusion at inference without concept-specific training.
-
Cambrian-P: Pose-Grounded Video Understanding
Adding per-frame camera-pose supervision to a video MLLM improves spatial and general video question answering by 2–6% and yields SOTA streaming pose estimates on ScanNet.
-
Towards Large Model Feature Coding
LaMoFCBench is a new benchmark covering 4 categories and 16 scenarios that exposes misalignment between mainstream feature codecs and the heterogeneous statistics of large-model activations.
-
DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers
DiRotQ uses PCA-based rotation-aware activation quantization combined with GPTQ to achieve better FID and PSNR in 4-bit diffusion transformers than prior methods like SVDQuant.
-
Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations
X-Shift is a grey-box attack that perturbs patch-level visual features in VLMs to shift explanation heatmaps without changing the predicted output.
-
DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Models
DarkLLM trains an LLM to generate language-driven adversarial perturbations that unify targeted, untargeted, segmentation, and multi-model attacks on foundation models.
-
Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
Gaze Attention groups visual embeddings into selectable regions and dynamically restricts attention to task-relevant ones, matching dense baselines with up to 90% fewer visual KV entries via added context tokens.
-
MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
MSD-Score introduces multi-scale distributional scoring on von Mises-Fisher mixtures to evaluate image captions without references and reports state-of-the-art correlation with human judgments.
-
Sentinel2Cap: A Human-Annotated Benchmark Dataset for Multimodal Remote Sensing Image Captioning
Sentinel2Cap provides human-annotated captions for multimodal Sentinel satellite images, with zero-shot tests showing RGB outperforming SAR and prompts helping performance.
-
EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure
EASE closes three residual anchors in federated multimodal unlearning using bilateral displacement, cosine-sine decomposition, and forget lock, achieving near-retrain performance on forget and retain data.