Pith. sign in

REVIEW 3 major objections 6 minor 80 references

Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning open multimodal LLMs on a synthetic 240K-question camera-object dataset lifts their recognition accuracy 33.4 points above GPT-4o and Claude.

desk verdict A genuinely useful synthetic dataset and pipeline with a plausible central claim, but the exact 33.4% margin is not settled until the GPT-4o grader is calibrated against human judgments. read the letter →

arxiv 2507.08513 v2 pith:P26KD62L submitted 2025-07-11 cs.GR cs.CV

Liu He , Xiao Zeng , Yizhi Song , Albert Y. C. Chen , Lu Xia , Shashwat Verma , Sankalp Dayal , Min Sun
show 2 more authors
Cheng-Hao Kuo Daniel Aliaga
This is my paper
classification cs.GRcs.CV
keywords camera-objectrelationsmultimodallargelanguagemodelsvisualinstructiontuningsyntheticdatasetgenerationdiffusion3Dassetsobjectorientationcameraviewpoint
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the reason multimodal LLMs fail at camera-object relations—which way an object faces, whether the camera is above or below, whether the shot is close-up or long—is a dataset bottleneck, not a model-capacity limit. It builds a pipeline that renders 3D assets from arbitrary camera angles to obtain ground-truth depth, edges, and segmentations, then uses diffusion image generation to turn those priors into photorealistic training images that keep the exact camera-object relation. LLMs write diverse image descriptions and multiple-choice questions for each rendered relation. The result is Ultimate3D: 240K visual question-answer pairs with precise annotations. One-epoch fine-tuning of open models on this dataset, the paper reports, outperforms commercial models by an average of 33.4 percentage points on orientation, viewpoint, and shot-type recognition, across synthetic, real-photo, and cross-dataset evaluations.

What carries the argument

The load-bearing mechanism is the three-parameter camera-object relation $\beta = \{\varphi, \theta, D\}$: the object's azimuth orientation, camera elevation, and camera-object distance. A renderer turns any $\beta$ into depth, Canny edge, and segmentation priors; stacked ControlNets condition SDXL on depth and edge priors so the generated photorealistic image inherits the exact geometry; and an LLM turns $\beta$ into categorical labels (eight orientations, three viewpoints, three shot types) and writes the QA pairs. Because the same $\beta$ that grounded the image generation also grounds the text supervision, image and label cannot drift apart.

What would settle it

Have humans grade the same set of model responses on samples of the Ultimate3D and MMVP benchmarks, or force GPT-4o, Claude, and the fine-tuned models to answer in identical multiple-choice format; if the fine-tuned models' advantage over commercial models shrinks by a large amount or disappears under those conditions, the claimed 33.4% improvement is an artifact of grading format rather than camera-object understanding.

Watch

Extended reading notes

Core claim

The central claim is that camera-object relation recognition in multimodal LLMs is mainly a training-data problem: models trained on real-image datasets see mostly front-facing objects from near-horizontal viewpoints, so they fall back to a few stereotyped answers. The paper proposes a generation pipeline with ground-truth camera-object parameters $\beta = \{\varphi, \theta, D\}$ that can produce unlimited image-text pairs without this distribution bias. By rendering depth, Canny edges, and segmentation masks from Blender and feeding them as ControlNet conditions into SDXL, the pipeline produces photorealistic images that provably preserve the controlled camera-object relation; LLMs then convert $\beta$ into diverse multiple-choice QA pairs. Fine-tuned LLaVA-1.6-13B on Ultimate3D achieves 72.4% accuracy on the benchmark orientation task, 72.3% on viewpoint, and 94.8% on camera-shots, beating GPT-4o, GPT-4o-mini, Claude-3-Sonnet, and Claude-3.5-Sonnet by an average of 33.4 percentage points, with similar gains on the MMVP real-image benchmark.

Load-bearing premise

The reported gains rest on the assumption that GPT-4o grades all model answers fairly, even though the fine-tuned models are trained to output the exact option-letter format the grader accepts while commercial models answer freely, and no human-graded calibration is reported.

Editorial extensions

If this is right

  • One-epoch fine-tuning on Ultimate3D lifts open MLLMs such as LLaVA-1.5/1.6 (7B and 13B) and Llama-3.2-Vision-11B from near-random accuracy to above 70% on orientation and viewpoint and above 90% on shot type on the synthetic benchmark.
  • Gains transfer to real photos: on the Real benchmark set and the cross-dataset MMVP benchmark, fine-tuned LLaVA-1.6-13B outperforms commercial SOTA models by an average of 19.23 percentage points on MMVP.
  • Mixing Ultimate3D half-and-half with general visual instruction data keeps VQAv2 accuracy at 80.01% (compared with the 80.00 baseline) while improving camera-object accuracy from 29.70% to 74.72%, so fixing the bottleneck does not sacrifice general VQA performance.
  • The generation pipeline scales: given any 3D asset with a facing direction, it can produce synthetic image-text pairs for arbitrary $\beta$ without the front-view bias that dominates real image datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation section notes that numerical prediction of $\beta$ (continuous azimuth or elevation angles) is harder than categorical prediction; a natural extension is to use the same pipeline to generate regression-style supervision and test whether the dataset-bottleneck claim holds for continuous values.
  • If the claimed gains hold under human grading, an untested corollary is that reweighting real image datasets to match Ultimate3D's uniform distribution over $\beta$ might recover much of the improvement without synthetic data.
  • Because the evaluation uses GPT-4o as grader and the fine-tuned models are trained to emit terse option letters while commercial models answer freely, an independent check is to have humans grade the same outputs or force all models into the same multiple-choice response format; this would separate format effects from genuine camera-object understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a synthetic-data generation pipeline for camera-object relation QA, in which 3D assets are rendered under controlled camera-object parameters, photorealistic images are produced with ControlNet conditioned on depth and Canny priors, and GPT-4o generates both image descriptions and multiple-choice VQA instructions. The authors release Ultimate3D, containing 240K VQAs from 85K synthetic images plus 18K MEBOW human crops, together with a benchmark that has synthetic and real (Pascal3D+ and MEBOW) subsets plus an external 89-question MMVP subset. They report that one-epoch fine-tuning of open LLaVA-1.5/1.6-7B/13B and Llama-3.2-Vision-11B models yields large accuracy gains, culminating in the claim that fine-tuned LLaVA-1.6-13B outperforms commercial models (GPT-4o, Claude) by an average of 33.4% on camera-object relation recognition (Sec. 4.3, Table 1).

Significance. If the central claim is correct, the paper provides a scalable and precise way to repair a specific 3D-perception deficiency in MLLMs using synthetic data with exact geometric ground truth, and the dataset and benchmark would be a useful community asset. The paper has real strengths: the pipeline is reproducible in principle from open assets and models; the ablations of ControlNet priors and backbones (Sec. 4.5, Supp. Sec. 5) are informative; the user study reports high image-quality success (93.07%) with 94.4% inter-rater agreement (Sec. 4.6); and the MMVP real-image gains provide some evidence of cross-dataset transfer. The significance is tempered by two load-bearing evaluation concerns: the GPT-4o-as-judge protocol has no human calibration, and the synthetic benchmark shares its generation pipeline with the training set. Because both concerns are addressable through additional analysis rather than fundamental errors, the work is potentially acceptable, but the evaluation must be tightened first.

major comments (3)
  1. [Sec. 4.3; Supp. Secs. 2-3] All accuracy numbers in Table 1 are produced by a single GPT-4o judge that grades every model's response, including GPT-4o's own responses, and no human-graded calibration or inter-annotator agreement is reported. The fine-tuned models are explicitly trained (via the QA-generation system prompt in Supp. Sec. 2) to emit terse option-letter answers of the exact form the grading prompt (Supp. Sec. 3) is designed to accept, whereas GPT-4o and Claude provide free-form explanatory answers. If GPT-4o is biased toward the expected format or penalizes verbosity, every headline number, including the real-image Pascal3D+/MEBOW columns and the MMVP column, is inflated by an unknown amount. This directly affects the abstract's 33.4% claim and the claim in Sec. 4.3 that the fine-tuned model 'outperforms commercial SOTAs by average of 33.4%.' I ask the authors to add a human-graded subset of at least a few hundred responses per model with reported agreement, or to replace the judge with a rule-based exact-match scorer for the fine-tuned models and a semantically validated scorer for free-form responses.
  2. [Sec. 4.2 and Sec. 4.3 (Table 1)] The Synthetic Set of the Ultimate3D benchmark is generated with the same rendering back end, the same ControlNet prior configuration, and the same GPT-4o prompt templates as the training images; only the 3D asset identities do not overlap. Consequently, the large gains on the synthetic set may partly reflect the model learning the pipeline's visual signature (lighting, texture, distribution of ControlNet artifacts) rather than transferable recognition of object orientation, viewpoint, and shot type. The MMVP external set and the Pascal3D+/MEBOW real images provide independent evidence, but the abstract and Sec. 4.3 state the 33.4% figure on 'Ultimate3D benchmark' without separating synthetic and real columns. Please report the headline gains on the Real set alone, and provide an analysis of whether the synthetic-set gains persist across asset categories, backgrounds, and prompt variations that were not used in training.
  3. [Sec. 4.3 (Table 1)] The computation behind the 'average 33.4%' improvement is not defined. Table 1 has multiple sub-columns per task (Both/Syn/Real for Ultimate3D and MMVP), and neither the caption nor the text states whether the 33.4% is averaged over tasks, over sub-columns, over which commercial model baseline it is compared against (GPT-4o, GPT-4o-mini, or Claude), or whether it is the mean of per-category accuracy differences. Please give the exact aggregation formula and, because the MMVP subset contains only 89 questions, report standard errors or confidence intervals. Without this, the abstract's headline number cannot be independently verified from the table.
minor comments (6)
  1. [Table 1] The column headers in Table 1 are ambiguous: 'Both Syn Real' appears consecutively under both Ultimate3D and MMVP, and the reader must infer which sub-column belongs to which benchmark; a cleaner layout or a header with explicit benchmark names (e.g., 'Ultimate3D-Syn') would remove this confusion.
  2. [Sec. 4.2] The dataset composition is not exactly specified: 85K synthetic images times 3 QAs would give 255K VQAs, yet the paper reports 240K VQAs; the discrepancy (due to images with fewer than 3 QAs or the MEBOW portion) should be clarified in one sentence.
  3. [Sec. 3.1] The definition of camera-object distance D is given only as 'relative units of Blender,' with D=1 meaning fully covered, but the actual relationship to the object bounding box or image frame is not specified, which makes the shot-type cutoffs (1.25, 3.0) hard to interpret or reproduce.
  4. [Sec. 4.6 vs. Supp. Sec. 7] The number of user-study samples is stated as 225 in Sec. 4.6 but as 200 in Supp. Sec. 7; please reconcile these numbers.
  5. [Sec. 4.3] For Llama-3.2-Vision-11B, the baseline 'refuses to answer most questions,' and the reported 60.46% improvement is computed from a baseline near zero; the paper should state whether the baseline accuracy is computed over all questions or only answered ones, and how refusals are treated.
  6. [Figure 4 caption] The caption states that GPT-4o's object-orientation performance is 'similar to random guess,' but Table 1 shows GPT-4o at 43-52% on orientation, far above the 12.5% chance level for 8 options; please rephrase to avoid inconsistency with the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 33.4% gain is an empirical benchmark comparison, not a reduced or fitted quantity; the GPT-4o grading and benchmark-overlap concerns are validity risks, not circularity.

full rationale

The paper's central claim is that one-epoch fine-tuning on the Ultimate3D dataset raises camera-object relation accuracy over commercial SOTAs, with an average gain of 33.4%. This is an empirical benchmark result, not a quantity derived from the same equations or fitted parameters used to produce it. Ground-truth labels are generated from rendered 3D geometry (Sec. 3.1 and Sec. 3.4) and from external real-image annotations (Pascal3D+, MEBOW, MMVP), and the evaluated MLLMs are not part of the label-generation process. The 33.4% figure is an arithmetic average of the accuracy columns in Table 1, not a fitted parameter renamed as a prediction. No uniqueness theorem from prior work is imported to force a choice, and no ansatz is smuggled in via self-citation: MMVP [54] is an external benchmark, and MEBOW [58] is used only as a source of human-orientation labels. The remaining concerns noted in the review—GPT-4o acting as answer generator, grader, and competitor, and the synthetic benchmark sharing renderer/ControlNet/QA-template conventions with the training data—are validity and calibration risks rather than circularity, because grading is defined against a fixed ground-truth answer set and the real-image and MMVP gains provide independent support outside the training pipeline. I therefore find no circular step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a hand-set discretization of camera-object relations (8x3x3 bins plus D cutoffs), three automation assumptions that are not validated at scale (LLM text fidelity, ControlNet label preservation, facing-direction well-definedness), and the grader-neutrality assumption. No parameters are secretly fit to the evaluation benchmark, which is why the circularity burden stays low; the real issues are measurement fairness and unvalidated mass production of labels.

free parameters (4)
  • Camera-shot distance cutoffs = D = 1.25 and D = 3.0 (Blender units)
    Hand-set boundaries dividing close-up, medium-shot, and long-shot categories. They define the task's ground-truth labels, so all 240K training labels and camera-shot metrics inherit this arbitrary choice. Not fitted to MLLM accuracy.
  • ControlNet weights = depth 0.5, Canny 0.8
    Hand-chosen mixing coefficients for the stacked ControlNets; selected in Sec. 4.1 and validated by ablations in Sec. 4.5 and 4.6 rather than by a fitting procedure.
  • Diffusion sampling steps = T = 30
    Hand-set number of denoising steps for all generated images; robustness implications are discussed only through ablations on priors, not on T.
  • Orientation and viewpoint bin boundaries = 8 bins of pi/4 azimuth; 3 bins of pi/3 elevation
    Discretization of the continuous beta parameterization into categorical labels. Reasonable, but the 8x3x3 binning sets the granularity of what MLLMs are asked to learn, and objects near bin edges are labeled with coarse angular precision.
assumptions (5)
  • domain assumption The camera model beta = {phi, theta, D} with the camera facing the object centroid fully captures the camera-object relations needed for orientation, viewpoint, and shot questions.
    Invoked in Sec. 3.1 to define all ground-truth labels; assumes the object's facing direction is semantically meaningful and that no other relation (e.g., camera roll, object translation within the frame) is needed for the three tasks.
  • domain assumption A well-defined 'facing direction' exists for each of the 100 manually selected ImageNet synsets.
    Sec. 4.1 states synsets were selected for explicit facing orientation; the phi label (azimuth of the facing direction) is ill-posed for categories without an intrinsic front, and the selected set excludes them.
  • domain assumption GPT-4o-generated image descriptions and QA pairs are semantically faithful and error-free at scale.
    Sec. 3.2 and 3.4 use GPT-4o for all 85K image prompts and 240K QA texts; only the 3,600 benchmark QAs were manually reviewed (Sec. 4.2), leaving training QA quality unaudited.
  • domain assumption Depth and Canny ControlNet priors preserve the rendered beta in the generated images.
    Core to the pipeline (Eq. 2, Sec. 3.3). Partially supported by the user study (93.07% success on 225 pairs, Sec. 4.6) and ablations (Sec. 4.5), but no automated large-scale verification of label preservation on the 85K training images.
  • standard math The SDXL-1.0 and ControlNet machinery behaves as published.
    Eq. 1 and 2 invoke standard DDPM and ControlNet formulations from cited works; treated as background knowledge the paper relies on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation." pith.science (2026). https://pith.science/paper/P26KD62L

@misc{pith2026250708513,
  author       = {Pith},
  title        = {Pith review of: Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P26KD62L}},
  note         = {Machine review of arXiv:2507.08513}
}
read the original abstract

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a dataset of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 33.4% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.

Figures

Figures reproduced from arXiv: 2507.08513 by the authors.

Figure 1
Figure 1. Generating Synthetic Visual Instruction Dataset. Our framework uses open-sourced 3D assets to generate photo-realistic images with precisely controlled camera-object relation. Corresponding text instructions are also generated by Large Language Model (LLM). The generated Ultimate3D dataset (240K) and benchmark (8K) advance baseline MLLM models (LLaVA-1.6, Llama-3.2-Vision, etc.) to outperform commercial MLLMs (GPT-4… view at source ↗
Figure 2
Figure 2. Our Framework. Each shade box corresponds to a section in Sec. 3. Given open-sourced 3D assets, our approach leverages a 3D Renderer to generate 3D visual priors (Iβ) preserving ground truth camera-object relation (β). Meanwhile, LLMs take 3D asset category to generate diverse image descriptions (Timg) as conditional guidance. Both the 3D visual priors and diverse text prompts are used to generate synthetic images (… view at source ↗
Figure 3
Figure 3. 3D Visual Prior Rendering. With a general camera model, our method utilizes a 3D Renderer (e.g., Blender) to render multiple 3D visual priors given arbitrary camera-object relations. 3.1. 3D Visual Prior Rendering Our system renders several 3D visual priors including 2D depth images, Canny-edge images, and segmentation masks from arbitrary viewpoints. These 3D visual priors con￾tain the scene’s geometry and will be … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Qualitative Results. We show responses by GPT-4o, Claude-3-Sonnet, LLaVA-1.6-13B, and fine-tuned LLaVA-1.6-13B, on Ultimate3D benchmark. Each model is asked the questions (in gray boxes) regarding object orientation, camera viewpoint, and camera￾shots type, together wi…
Figure 5
Figure 5. Figure 5: Ablations on Image Generation Pipeline. Our frame￾work shows superior performance by using multi-ControlNet with SDXL backbones, and introducing both Canny edges and depth maps as visual priors. It consistently delivers better robustness and quality compared to text-on…
Figure 6
Figure 6. Figure 6: Limitations of Text-Image Aligning Metrics. We eval￾uate the ImageReward [60] (IR) between the text promopt regard￾ing object orientation with above 4 synthetic images. The image matches the correct camera-object relation text prompt (”front”) may reach the highest sco…
Figure 7
Figure 7. Figure 7: Failures of Pose Estimation Model. Enlighten by [40], we train a transformer-based pose estimation model on a 10-fold manner for generated images on each single category. The im￾ages with less prediction errors may indicate successful genera￾tion. However, the pose est…
Figure 8
Figure 8. Figure 8: Factors influencing Image Generation. The upper-left example shows that bottom-view camera-object relations can introduce physical anomalies, such as the tricycle appearing with its front lifted unnaturally. The upper-right example highlights that low-quality 3D assets…
Figure 9
Figure 9. Figure 9: User Study UI Page. Users will see a pair of side-by-side images. The left hand side is an RGB image generated by Blender, and the right hand side is the synthetic image generated by DMs using that RGB image as guidance prior. A vivid preserving of 3D geometry and stru…
Figure 10
Figure 10. Figure 10: Additional Qualitative Results. We show the evaluations of camera-object relation recognition capability of GPT-4o, Claude￾3-Sonnet, LLaVA-1.6-13B, and finetuned LLaVA-1.6-13B, on Ultimate3D benchmark. Each model is asked the questions (in gray boxes) regarding object…
Figure 11
Figure 11. Figure 11: Diversity of Ultimate3D. Our Ultimate3D dataset and benchmark cover 100 categories of objects, range diverse camera-object relation settings, and provide plausible image quality. (Each row shows images with the same orientation but in diverse subject and context.) 8 …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 39 canonical work pages

  1. [1]

    Claude v3.0, 2024

    Anthropic. Claude v3.0, 2024. https : / / www . anthropic.com/. 2

    work page 2024

  2. [2]

    Shapenet: An information-rich 3d model repository

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015. 5

    arXiv 2015

  3. [3]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14455–14465,

  4. [4]

    Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 2

    arXiv 2024

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 2

    2024

  6. [6]

    Spatial- rgpt: Grounded spatial reasoning in vision language model

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584, 2024. 2

    arXiv 2024

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2

    2023

  8. [8]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

Show all 80 references
  1. [9]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2023

  2. [10]

    Objaverse-xl: A universe of 10m+ 3d objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram V oleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Informa- tion Processing Systems, 36, 2024. 5

    2024

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2

    2010 arXiv

  4. [12]

    What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning

    Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jin- peng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen. What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning. arXiv preprint arXiv:2311.01487, 2023. 2

    2023 arXiv

  5. [13]

    Training on synthetic data beats real data in multimodal relation extrac- tion

    Zilin Du, Haoxin Li, Xu Guo, and Boyang Li. Training on synthetic data beats real data in multimodal relation extrac- tion. arXiv preprint arXiv:2312.03025, 2023. 2

    2023 arXiv

  6. [14]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390,

    arXiv

  7. [15]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  8. [16]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern re...

    2022

  9. [17]

    Regiongpt: Towards region understanding vision lan- guage model

    Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision lan- guage model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13796– 1380...

    2024

  10. [18]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 2

    2019

  11. [19]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  12. [20]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 3

    2022 arXiv

  13. [21]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  14. [22]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 3

    2017

  15. [23]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  16. [24]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 2

    2019

  17. [25]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognitio...

    2017

  18. [26]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  19. [27]

    Learning action and reasoning-centric image editing from videos and simulations

    Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jam- pani, Eva Portelance, Christopher Pal, and Siva Reddy. Learning action and reasoning-centric image editing from videos and simulations. arXiv preprint arXiv:2407.03471 ,

    arXiv

  20. [28]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 2

    2024 arXiv

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  22. [30]

    What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    arXiv

  23. [31]

    Stablellava: Enhanced visual instruction tun- ing with synthesized image-dialogue data

    Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yun- chao Wei. Stablellava: Enhanced visual instruction tun- ing with synthesized image-dialogue data. arXiv preprint arXiv:2308.10253, 2023. 2

    2023 arXiv

  24. [32]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 2

    2024

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  26. [34]

    Visual spa- tial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spa- tial reasoning. Transactions of the Association for Computa- tional Linguistics, 11:635–651, 2023. 2, 3, 4

    2023

  27. [35]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2, 6, 8

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 3, 4

    2024

  29. [37]

    Clevr-ref+: Diagnosing visual reasoning with referring ex- pressions

    Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr-ref+: Diagnosing visual reasoning with referring ex- pressions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4185–4194,

  30. [38]

    Synthvlm: High-efficiency and high-quality synthetic data for vision language models

    Zheng Liu, Hao Liang, Wentao Xiong, Qinhan Yu, Conghui He, Bin Cui, and Wentao Zhang. Synthvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint arXiv:2407.20756, 2024. 2

    2024 arXiv

  31. [39]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  32. [40]

    Generating images with 3d annotations using diffusion models

    Wufei Ma, Qihao Liu, Jiahao Wang, Angtian Wang, Xiaod- ing Yuan, Yi Zhang, Zihao Xiao, Guofeng Zhang, Beijia Lu, Ruxiao Duan, et al. Generating images with 3d annotations using diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. 3, 4, 8, 9

    2024

  33. [41]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 2

    2019

  34. [42]

    Llama 3.2 vision, 2024

    Meta. Llama 3.2 vision, 2024. https://ai.meta. 10 com/blog/llama-3-2-connect-2024-vision- edge-mobile-devices/. 2, 6

    2024

  35. [43]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–

    2019

  36. [44]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 3

    2023 arXiv

  37. [45]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 3

    2023 arXiv

  38. [46]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 3, 4, 5

    2023 arXiv

  39. [47]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3, 8

    2022

  40. [48]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 5

    2015

  41. [49]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  42. [50]

    Synth2: Boosting visual-language models with synthetic captions and image embeddings

    Sahand Sharifzadeh, Christos Kaplanis, Shreya Pathak, Dharshan Kumaran, Anastasija Ilic, Jovana Mitrovic, Charles Blundell, and Andrea Banino. Synth2: Boosting visual-language models with synthetic captions and image embeddings. arXiv preprint arXiv:2403.07750, 2024. 2

    2024 arXiv

  43. [51]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. 2

    2019

  44. [52]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3

    2010 arXiv

  45. [53]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2, 3

    2024 arXiv

  46. [54]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 2, 3, 5, 6

    2024

  47. [55]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  48. [56]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. 2

    2023 arXiv

  49. [57]

    Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting. In Proceedings of the IEEE/CVF c...

    2023

  50. [58]

    Mebow: Monocular estima- tion of body orientation in the wild

    Chenyan Wu, Yukun Chen, Jiajia Luo, Che-Chun Su, Anuja Dawane, Bikramjot Hanzra, Zhuo Deng, Bilan Liu, James Z Wang, and Cheng-hao Kuo. Mebow: Monocular estima- tion of body orientation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2020

  51. [59]

    Beyond pascal: A benchmark for 3d object detection in the wild

    Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2014. 5, 4

    2014

  52. [60]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. Advances in Neural Information Pro- cessing Systems, 36, 2024. 3

    2024

  53. [61]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 2

    2023 arXiv

  54. [62]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 2

    2014

  55. [63]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 2

    2016

  56. [64]

    When and why vision- language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936, 2022. 2

    2022 arXiv

  57. [65]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 3

    2024

  58. [66]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In 11 Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3, 4

    2023

  59. [67]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 2

    2023 arXiv

  60. [68]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2

    2024 arXiv

  61. [69]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302–321, 2019. 2 12 Advancing Multimodal LLMs by Large-Scale 3D ...

    2019

  62. [70]

    Corresponding section in main paper is Sec

    Image Description Generation We provide system prompts given to GPT-4oto generate context description for image generation. Corresponding section in main paper is Sec. 3.2. In the prompt, we provide one-shot example for better robustness of the text genera- tion. Specifically,...

  63. [71]

    front" also means

    Text Instruction Generation We provide the system prompt to generate QA pairs given camera-object relation as below. Corresponding section in main paper is Sec. 3.4. In the prompt, we provide few-shot example for better instructions for diversity of the generated text. SYSTEM_...

  64. [72]

    SYSTEM_PROMPT_FOR_GRADING_MLLM_RESPONSE =’You are a helpful and precise assistant for checking the quality of the answer

    LLM-based Response Grading We provide the system prompt for evaluation of the MLLM response to our camera-object relation multiple choice questions. SYSTEM_PROMPT_FOR_GRADING_MLLM_RESPONSE =’You are a helpful and precise assistant for checking the quality of the answer. You sh...

  65. [73]

    6, we show an example of using ImageReward [60] for the dataset curation by evaluating the alignment be- tween generated image and text prompts

    Metric Limitations for Evaluating Camera- Object Relation In Fig. 6, we show an example of using ImageReward [60] for the dataset curation by evaluating the alignment be- tween generated image and text prompts. Preliminary test indicates that text-image aligning metric is not ...

  66. [74]

    3, we perform quantitative comparisons on general image visual quality between different DM backbones: SD V1.5 and SDXL

    Ablation of DM Backbones In Tab. 3, we perform quantitative comparisons on general image visual quality between different DM backbones: SD V1.5 and SDXL. We randomly sample 1,000 generated im- ages (Isyn), each image corresponds to: (1) the RGB visual prior rendered using Blen...

  67. [75]

    Factors influencing Image Generation Several factors may lead to unreasonable artifacts in our im- age generation pipeline. In Fig. 8, the upper-left example shows that bottom-view camera-object relations can intro- duce physical anomalies, such as the tricycle appearing with ...

  68. [76]

    9 shows the UI page of our user study

    User Study Details The Fig. 9 shows the UI page of our user study. We ran- domly selected 200 images from the Ultimate3D dataset. Each image corresponds to an RGB prior generated by Blender. During the user study, users will see a pair of side-by-side images. One is an RGB ima...

  69. [77]

    10 to show more qualitative comparisons between fine- tuned LLaV A model to commercial SOTAs

    Additional Qualitative Results Based on Ultimate3D benchmark, we provide additional Fig. 10 to show more qualitative comparisons between fine- tuned LLaV A model to commercial SOTAs. The finetuned LLaV A model outperforms other models

  70. [78]

    11, we shows more examples of diversity on object categories, camera-object relation, and background con- texts

    Additional Visuals of Ultimate3D Dataset In Fig. 11, we shows more examples of diversity on object categories, camera-object relation, and background con- texts

  71. [79]

    Additional Discussions We provide some insights of using synthetic generated vi- sual instruction dataset for MLLM finetuning. For the task of camera-object relation recognition for 100 categories (as we collected in Ultimate3D dataset), a dataset compris- ing 100K to 1M VQAs ...

  72. [80]

    Algorithm of 3D visual instruction dataset generation pipeline Algorithm 1 Synthetic VQA generation Input: 3D asset A, asset category c, camera-object relation β. Parameter: Renderer R, DM-based image generator G, image decoder D, LLM text generator L, system prompt given to L...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.