REVIEW 16 cited by
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large-scale pretrained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from the problems of low computational efficiency and information asymmetry brought by the long visual sequence in cross-modal alignment. To address these problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections, which creates inter-layer shortcuts that skip a certain number of layers for time-consuming full self-attention on the vision side. mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability when directly transferred to multiple video-language tasks.
Forward citations
Cited by 16 Pith papers
-
Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios
A new attention-based reasoning module combining iterative steps with parallel operation slots improves accuracy on multiple visual question answering benchmarks while staying lightweight and partially interpretable.
-
Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
Lavida-O introduces an elastic mixture-of-transformers architecture that brings high-resolution text-to-image generation, object grounding, and image editing into a single masked diffusion model, using planning and se...
-
INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
INTER is a training-free logit-correction method that adds Harsanyi interaction scores to selected keyword tokens, lowering hallucination on six LVLM benchmarks.
-
GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning
A multi-agent vision-language framework that extracts event, time, and location from public event images, evaluated with a new soft metric on VLM-augmented datasets.
-
DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?
Introduces DetailMaster, a 4,116-prompt benchmark with fine-grained evaluation of long-prompt text-to-image generation, finding that state-of-the-art models achieve only about 50% accuracy on attribute binding and spa...
-
Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation
MR-PLIP is a multi-resolution pathology vision-language model that aligns histology patches and generated text across 5x, 10x, 20x, and 40x magnifications and reports improved transfer to 26 downstream pathology benchmarks.
-
PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction
PuzzleGPT, a zero-shot expert pipeline, reports state-of-the-art scores on TARA and WikiTilo time and location prediction, though the evaluation uses different metrics for the proposed method and baselines.
-
Pix2Cap-COCO: Advancing Visual Comprehension via Pixel-Level Captioning
Pix2Cap-COCO provides 167,254 pixel-aligned instance captions for COCO images and introduces panoptic segmentation-captioning as a benchmark task.
-
Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time
A reformulation model trained on human caption edits improves image captioning at inference time, yielding strong results on German captions and style transfer.
-
Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps
Aligning cross-attention map similarity to text self-attention maps at test time improves semantic alignment in Stable Diffusion for prompts with multiple objects and attributes.
-
Multimodal Feature Fusion Network with Text Difference Enhancement for Remote Sensing Change Detection
MMChange fuses image features with VLM-generated text descriptions of bitemporal remote sensing images, reporting state-of-the-art IoU/F1 on LEVIR-CD, WHU-CD, and SYSU-CD.
-
Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning
Selftok encodes images as diffusion-time-indexed discrete tokens, enabling a pure autoregressive VLM and visual RL with strong GenEval and DPG scores, though its claim that spatial tokens cannot support RL is not proven.
-
ReFrame: Rectification Framework for Image Explaining Architectures
ReFrame wraps image captioning, VQA, and GPT-4 with a Mask R-CNN rectifier, reporting big gains on metrics defined against that same detector.
-
Explainability for Vision Foundation Models: A Survey
A structured review of 122 papers on explainability for vision foundation models, with a taxonomy and the finding that quantitative evaluation is rare (36%).
-
Visual question answering: from early developments to recent advances -- a survey
A survey that classifies VQA architectures by encoder, fusion, and decoder, reviews datasets and metrics, and discusses applications and future directions.
-
Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.
Discussion (0). Continue with ORCID to comment.