REVIEW 9 cited by
Benchmarking Spatial Relationships in Text-to-Image Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Spatial understanding is a fundamental aspect of computer vision and integral for human-level reasoning about images, making it an important component for grounded language understanding. While recent text-to-image synthesis (T2I) models have shown unprecedented improvements in photorealism, it is unclear whether they have reliable spatial understanding capabilities. We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image. To benchmark existing models, we introduce a dataset, $\mathrm{SR}_{2D}$, that contains sentences describing two or more objects and the spatial relationships between them. We construct an automated evaluation pipeline to recognize objects and their spatial relationships, and employ it in a large-scale evaluation of T2I models. Our experiments reveal a surprising finding that, although state-of-the-art T2I models exhibit high image quality, they are severely limited in their ability to generate multiple objects or the specified spatial relations between them. Our analyses demonstrate several biases and artifacts of T2I models such as the difficulty with generating multiple objects, a bias towards generating the first object mentioned, spatially inconsistent outputs for equivalent relationships, and a correlation between object co-occurrence and spatial understanding capabilities. We conduct a human study that shows the alignment between VISOR and human judgement about spatial understanding. We offer the $\mathrm{SR}_{2D}$ dataset and the VISOR metric to the community in support of T2I reasoning research.
Forward citations
Cited by 9 Pith papers
-
Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification
Rule-VLN is the first large-scale benchmark injecting 177 regulatory categories into an urban environment, and the proposed SNRM module equips pre-trained VLN agents with zero-shot semantic reasoning and detour planni...
-
Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text
Image-generation models become competitive on spatial benchmarks when answers are expressed as protocol-constrained pixels, while text-output VLMs retain an edge on compositional reasoning — with a parser-sensitivity ...
-
Transferability Between Understanding and Generation in Unified Multimodal Models
Cross-task capability transfer in UMMs is architecture-dependent and can be exploited by training understanding to improve generation while avoiding distribution shift.
-
Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation
D-GPTScore, which averages GPT-4o's per-aspect ratings of concept-customized images, correlates with human preference at 0.78 Pearson on the new CC-AlignBench, beating prior metrics.
-
Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas
Canvas3D lets users arrange objects in a 3D canvas generated from a text prompt, then feeds depth, skeleton, and lighting constraints to diffusion models to produce images that match the layout.
-
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
A cross-modal benchmark derived from MMVP shows VLMs and IGMs share several elementary visual-reasoning failure modes, with IGMs struggling most on explicit attribute-control prompts.
-
Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models
A probabilistic overlap measure for object positions yields a human-aligned spatial relationship metric and a training-free generation guidance method for text-to-image models.
-
Towards Evaluating Robustness of Prompt Adherence in Text to Image Models
New benchmark results show that Stable Diffusion 3.x and Janus Pro models struggle to place simple geometric shapes in the correct image quadrant, with best F1 scores around 0.41 to 0.5.
-
Re-Thinking the Automatic Evaluation of Image-Text Alignment in Text-to-Image Models
Current image-text alignment metrics, including CLIPScore and DSGScore, produce unstable model rankings under random seeds and are highly sensitive to tiny image perturbations.
Discussion (0). Sign in to comment.