Pith. sign in

REVIEW 4 major objections 5 minor 118 references

Feedback-Driven Vision-Language Alignment with Minimal Human Supervision

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SVP improves vision-language alignment without curated image-text pairs, using the model's own captions filtered through a pre-trained grounding model.

desk verdict SVP is a genuinely new and broadly evaluated method for VLM alignment without curated pairs; the empirical case is strong, but missing error bars and the unmeasured risk of detector-missed objects need addressing before I'd fully trust the numbers. read the letter →

arxiv 2501.04568 v2 pith:LCVXSKXL submitted 2025-01-08 cs.CV cs.AIcs.CLcs.LG

Giorgio Giannone , Ruoteng Li , Qianli Feng , Evgeny Perevodchikov , Rui Chen , Aleix Martinez This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords vision-languagemodelsvisualgroundinghallucinationreductionself-improvementimagecaptioningreferringexpressionsobjectrecallfeedback-drivenalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a vision-language model can be made to see more accurately without any curated image-text pairs or preference annotations. The proposed framework, SVP, uses a small set of plain images, asks the model to caption them, passes those captions through a pre-trained open-set grounding model to obtain object boxes, and then fine-tunes the model only on the grounded captions that score highest. Across ten benchmarks the authors report a 14% average gain in captioning, up to 12% higher object recall, and substantial hallucination reduction, with a 7-billion-parameter model reaching hallucination-avoidance scores comparable to models several times larger. The reason this matters is that alignment data is currently the bottleneck: if the claim holds, alignment becomes a self-supervised loop over images plus one frozen detector.

What carries the argument

The central object is the visual projection z, a latent 'generalized caption' bridging image and text. The mechanism is a three-stage loop: (1) prior sampling z_p ~ p_θ(z_p|c); (2) grounding feedback g ← g(z_p, c) from a pre-trained open-set detector; (3) guided sampling z ~ q(z|g,c) that re-prompts the VLM with textual boxes. A scoring function S(z) = log q(z|c,g) - log p_θ(z|c), or a weighted-difference variant, ranks samples so that only grounded captions that surprise or inform the base model are kept. The outer loop is a reweighted maximum-likelihood or off-policy gradient that selects the top-k samples per image and fine-tunes with low-rank adaptation. The grounding text is stripped away before adaptation, so the model never sees boxes at test time.

What would settle it

Sample a set of images where a stronger detector or human labelers show the grounding model makes systematic false negatives or false positives, and run SVP on those images alone. If hallucination scores get worse or object recall does not improve relative to the base model on images where the detector is wrong, the feedback signal is carrying detector error rather than latent alignment.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that grounding feedback can act as a training signal to elicit latent alignment already present in a VLM. The base model first samples a 'visual projection' z, a detailed caption treated as a latent variable. A grounding model g(z_p, c) converts that draft into detected objects and boxes, and the VLM is prompted again with this textual grounding to produce a guided caption z. The divergence between the guided distribution q and the prior p, measured by a log-ratio or weighted-difference score, identifies which samples carry new visual information; the top fraction becomes the fine-tuning set. Because the grounding boxes are used only inside this inner loop and never at inference or in the loss, the improvement in referring, captioning, and hallucination behavior is attributed to better vision-language alignment itself rather than to memorized boxes.

Load-bearing premise

The load-bearing premise is that the pre-trained grounding model returns accurate and sufficiently complete object and spatial information for natural images; if it misses, mislabels, or overshoots objects, those errors are baked into the selected grounded captions and then into the fine-tuned model.

Editorial extensions

If this is right

  • Captioning improves by roughly 14% on average across COCO2017, Flickr30k, and NoCaps, with larger gains on models that start weak.
  • Referring expression generation can more than double, letting a 7B model approach a 13B model without ever training on bounding boxes.
  • POPE hallucination F1 rises (for example, from 86.7 to 88.3 for a 7B model), matching or approaching models five times larger.
  • Object recall rises by up to 12%, while VQA and multitasking scores are roughly preserved.
  • The method needs only about 1,000 seed images and K=20 samples per image, replacing curated pair annotation with compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same sampling-scoring loop could in principle work with any differentiable 'world model' feedback channel, not just object grounding; depth or segmentation maps would be natural alternatives.
  • The approach implicitly assumes the detector's object vocabulary matches what a caption should mention, so on domains with rare or novel objects, the feedback would suppress valid descriptions the detector cannot ground.
  • Because grounding is applied only at data-generation time, future improvements in open-set detection should transfer directly to SVP-trained models without retraining them.
  • A testable extension is to apply SVP to video frames with a tracking-based grounding model to see whether spatial-temporal feedback improves temporal consistency of generated captions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SVP (Sampling-based Visual Projection), a three-stage procedure consisting of inner-loop sampling with grounding feedback from a pretrained detector (GroundingDINO), scoring and top-k selection of grounded captions, and outer-loop LoRA adaptation of the base VLM. The method requires no curated image-text pairs or preference annotations, only a small set of unlabeled images and self-captioning. The authors evaluate SVP on LLaVA and other VLM families across captioning, referring, VQA, multitasking, hallucination, and object-recall benchmarks, reporting consistent gains on captioning and referring, reduced POPE hallucinations, and improved object recall after up to three iterations. The paper also includes ablations of scoring mechanisms, iterations, sample size, and a comparison against DPO.

Significance. If the results hold, SVP offers a cheap, task-agnostic alignment method that could reduce dependence on human annotation and curated data, with potential applicability across VLM architectures. The empirical coverage is broad (10 benchmarks, multiple model scales, multiple text encoders), the qualitative comparisons are instructive, and the pseudocode and prompt templates facilitate reproduction. However, the theoretical derivation in Appendix E contains an unjustified step, and the empirical claims lack statistical support, so the significance can only be fully assessed after those issues are resolved.

major comments (4)
  1. [Appendix E, Eqs. (14)-(15)] The derivation drops the importance ratio pθ(zi|c)/q(zi|c,g) by stating that it is 'constant' for selected samples. This is not justified: selection is based on S(q,p) = log q - log p, so the retained samples are exactly those with large log-ratios, meaning pθ/q is systematically small and varies across the selected set. The claimed equivalence between the MLE gradient in Eq. (10) and the off-policy REINFORCE gradient in Eq. (15) is therefore not established. Because the 'principled formulation' in Section 3 and the Introduction rests on this equivalence, this step needs a rigorous justification or an explicit acknowledgment that Eq. (15) is a heuristic approximation.
  2. [Section 4 (Implementation Details), Fig. 7, Table 11, Algorithms 1-2] The paper is internally inconsistent about the selection ratio. The Implementation Details paragraph states 'selecting the top 10% using our scoring mechanisms', while Fig. 7's caption says 'selecting the top 20%' and Table 11 lists topk = 0.2 for LLaVA-1.5-13b, LLaVA-1.6-7b, and LLaVA-1.6-13b, but topk = 0.1 for LLaVA-OV. Since the selection ratio is a free parameter that affects the training set and results, the ratio used for each reported experiment must be stated unambiguously.
  3. [Section 4, Tables 1-5 and Figures 2-5] All results are single-run point estimates with no error bars, variance, or significance tests. Several improvements are small (e.g., Table 1: LLaVA-1.6-13b w/SVP all-split F1 is 87.30 vs. 86.24 for the base; Table 3: MMMU for 13b w/SVP(CVQ) is 35.66 vs. 35.22 for the base). Given the small seed-image count (C=1000) and the stochastic sampling procedure, the wording 'significant improvements' in the Abstract and Section 4 is not supported without multiple seeds with standard deviations or a statistical test over evaluation subsets.
  4. [Section 3, step (2) and Section 4 (Hallucination and Object Recall)] The training target is defined by the detector output: the grounded caption is generated from the box list produced by GroundingDINO, and the adapted model is trained to imitate the selected grounded captions. The paper does not measure how often GroundingDINO omits or mislabels objects that the base model correctly mentions, nor does it evaluate SVP on rare, small, or out-of-distribution objects. POPE's object categories overlap heavily with categories GroundingDINO detects well, so the reported object-recall and hallucination gains may not generalize to objects the detector misses. Since the detector is not used at final inference, the adapted model cannot recover such omissions. I recommend adding a detector-recall analysis on the seed images and reporting POPE-style recall split by object frequency or on a rare-object benchmark to substantiate the general claim.
minor comments (5)
  1. [Section 3, after Eq. (3)] The sentence 'When grounding provides no additional information, q(z|g, c) ≈ p(z|zp, c), and Eq. 3 reduces to 1' is unclear; presumably the ratio q/p reduces to 1, but this should be stated explicitly.
  2. [Algorithms 1 and 2] Algorithm 1 line 26 includes a normalization factor 1/|k(c)| in the loss, while Algorithm 2 line 26 omits it. The two objectives should be consistent, or the difference should be explained.
  3. [Table 8] The entries 'w/oiSVP' and 'w/iSVP' appear to be typographical errors for 'w/o iSVP' and 'w/ iSVP'.
  4. [Section 1, Introduction] The Introduction says grounding information is 'not shown during the fine-tuning process but is utilized during inference (Fig. 7.(iii))', but Section 4 later clarifies that grounding is used only during inner-loop sampling and not at final inference. This wording should be corrected to avoid confusion.
  5. [Abstract] The abstract's '14% average improvement in captioning tasks' and 'up to 12% increase in object recall' should specify the exact datasets and metrics over which the averages are computed, since gains vary substantially across benchmarks in Section 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SVP's core loop relies on an external grounding model and independent benchmarks, so the claimed improvements are not definitionally equivalent to the method's inputs.

full rationale

The derivation chain is self-contained relative to the external feedback signal: the base VLM samples z_p ~ p_theta(z|c); the pre-trained GroundingDINO model supplies g = g(z_p, c); grounded samples z_q ~ q(z|c,g) are generated; samples are selected by the log-ratio score S(z) = log q(z|c,g) - log p_theta(z|c); and the base VLM is fine-tuned to maximize log p_theta(z|c) on the selected samples. Each step is explicitly defined in Section 3 (Eqs. 3-7) and Algorithm 1, and no equation reduces the final objective to the score itself: the adaptation objective is maximum likelihood on selected z_q, not the score S. GroundingDINO is a fixed external model trained independently of p_theta and of the evaluation benchmarks; it is not fitted in this paper, and its detections are not defined in terms of the VLM's outputs. The evaluation uses standard external benchmarks (COCO2017, NoCaps, Flickr30k, RefCOCO variants, POPE, MMBench, MMMU, ScienceQA, GQA) with held-out or lite splits, so the reported captioning, referring, hallucination, and object-recall numbers are not computed from the training-time selection scores. The same VLM is indeed used both to generate candidate captions and to be adapted on the selected grounded captions, which is a self-training loop; however, self-training with external grounding feedback is not circular by construction, because the feedback g is not derived from p_theta or from the evaluation metric. The paper's own limitation statement ('depends on grounding model quality') correctly identifies a robustness risk, not a definitional circularity. The only self-citation, reference [67], appears in a motivational sentence about engineering applications and is not load-bearing for the method or the results. Therefore, no pattern of self-definitional reasoning, fitted-input-called-prediction, or self-citation chain is present, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. Its claims rest on the accuracy of the external grounding model, the validity of the self-captioning loop, and a hand-wavy importance-ratio approximation in the training objective derivation.

free parameters (5)
  • samples per image K = 20 (10 for LLaVA-OV)
    Chosen by hand; ablations show recall improves with more samples, but no systematic selection.
  • top-k ratio = 0.1 in Implementation Details, 0.2 in Fig 7 and Table 11
    Inconsistency in the paper; this threshold controls which samples are used for training.
  • seed image count C = 1000 (2000 for LLaVA-OV)
    Arbitrary; sample-size ablation shows performance grows with C, suggesting a free resource parameter.
  • number of iterations I = up to 3
    Stopped by hand; Fig 11 shows gains plateau after iteration 2.
  • LoRA rank and alpha = r=64-128, alpha=16-256
    Standard settings, not tuned per task.
assumptions (4)
  • domain assumption GroundingDINO provides accurate open-set detections and bounding boxes for natural images.
    The entire SVP feedback signal g(zp,c) is produced by this detector; if it is noisy or incomplete, the selected training samples inherit that noise. Section 3 and Section 4.
  • standard math The deterministic output distribution p(x|z,c)=δ(x-d(z,c)) holds for tokenized VLM outputs.
    Standard in tokenization-based models and stated in Section 2; used to justify the MLE training objective.
  • ad hoc to paper The importance ratio pθ(z|c)/q(z|c,g) is approximately constant for selected samples.
    Appendix E simplifies the off-policy importance-sampling gradient by 'considering the importance ratio constant'; without this, the MLE and RL derivations are not equivalent.
  • domain assumption Self-captioning of the base VLM produces captions detailed enough for the grounding model to extract useful boxes.
    If the base VLM's captions are too vague, the grounding feedback adds little; the paper provides examples but no systematic failure analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Feedback-Driven Vision-Language Alignment with Minimal Human Supervision." pith.science (2026). https://pith.science/paper/LCVXSKXL

@misc{pith2026250104568,
  author       = {Pith},
  title        = {Pith review of: Feedback-Driven Vision-Language Alignment with Minimal Human Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCVXSKXL}},
  note         = {Machine review of arXiv:2501.04568}
}
read the original abstract

Vision-language models (VLMs) have demonstrated remarkable potential in integrating visual and linguistic information, but their performance is often constrained by the need for extensive, high-quality image-text training data. Curation of these image-text pairs is both time-consuming and computationally expensive. To address this challenge, we introduce SVP (Sampling-based Visual Projection), a novel framework that enhances vision-language alignment without relying on manually curated text-image pairs or preference annotation. SVP leverages a small set of manually selected images, self-captioning and a pre-trained grounding model as a feedback mechanism to elicit latent information in VLMs. We evaluate our approach across six key areas: captioning, referring, visual question answering, multitasking, hallucination control, and object recall. Results demonstrate significant improvements, including a 14 % average improvement in captioning tasks, up to 12 % increase in object recall, and significantly reduced hallucinations, while maintaining question-answering capabilities. Using SVP, a small VLM achieves hallucination reductions similar to a model five times larger, while a VLM with initially poor referring capabilities more than doubles its performance, approaching parity with a model twice its size.

Figures

Figures reproduced from arXiv: 2501.04568 by the authors.

Figure 1
Figure 1. Improving Vision-Language Alignment. Vision-language models (VLMs) often produce descrip￾tions lacking specificity and accuracy, frequently halluci￾nating objects or missing important elements (left). Our Sampling-based Visual Projection (SVP) addresses these issues by leveraging self-captioning and grounding feed￾back. SVP enhances visual-language alignment without requiring human annotations, curated image-text pa… view at source ↗
Figure 2
Figure 2. Referring w/ Bounding Box (left) and Segmentation Mask (right). 70 75 80 85 90 95 100 105 CIDEr Score LLaVA +7.8LLaVA w/ SVP (C) +11.3LLaVA w/ SVP (CVQ) NoCaps - Captioning 70 80 90 100 110 CIDEr Score LLaVA +2.6LLaVA w/ SVP (C) +5.3LLaVA w/ SVP (CVQ) COCO2017 - Captioning 70 72 75 77 80 82 85 87 90 CIDEr Score LLaVA +3.4LLaVA w/ SVP (C) +6.6LLaVA w/ SVP (CVQ) Flickr30k - Captioning 50 60 70 80 90 100 CIDEr Score LL… view at source ↗
Figure 4
Figure 4. Object Recall and Hallucination Reduction [PITH_FULL_IMAGE:figures/full_fig_p002_4.png] view at source ↗
Figures from the paper (15 more)
Figure 6
Figure 6. Figure 6: Vision-Language Generative Model (left) and Vision-Language Grounding (right) Notation We use p(x|c) and p(z|c) to denote auto-regressive distributions, where c is the condition￾ing information (image and prompt), z is a visual projection using grounding feedback, and …
Figure 7
Figure 7. Figure 7: SVP Overview. The inner-loop (left) generates K samples per input C with and without grounding, then scores and ranks them, selecting the top 20% (right side). Instead of visually representing the grounding, we transform it into textual form and incorporate it into the…
Figure 8
Figure 8. Figure 8: Graphical Models for the sampling processes. Left: standard sampling. Center: hierarchical sampling. Right: hierarchical sampling with internal structure. Problem Formulation For a VLM with conditional model pθ(x|c), where c = (cv, ct) contains visual in￾put and option…
Figure 9
Figure 9. Figure 9: Visualization of prior and guided distribution [PITH_FULL_IMAGE:figures/full_fig_p005_9.png]
Figure 10
Figure 10. Figure 10: Referring Expression Generation on Ref￾COCO comparing base LLaVA-1.6-7b versus SVP (C) and SVP (CVQ) variants. CIDEr scores shown for detec￾tion (bbox) and segmentation (seg) on test/validation sets. SVP models outperform baseline without using bounding boxes. See App…
Figure 12
Figure 12. Figure 12: Sample Size. Halluci￾nations (F1) and object recall (R) results with I = 1 a single itera￾tion increasing the sample size C ∈ (0.1, 0.2, 0.5, 1, 5, 10)k. 0 10 20 30 Object Counting 0.00 0.02 0.04 0.06 0.08 0.10 Density q p [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 14
Figure 14. Figure 14: Input image from COCO2017 [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 17
Figure 17. Figure 17: Sampling VLMs with and without Grounding Feedback. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: FLUX-schnell [43] text to image generation using the original VLM response (left) and the response leveraging grounding (right) as input. We generated a single image without multiple attempts or selective filtering. The comparison clearly illustrates that the groundin…
Figure 19
Figure 19. Figure 19: FLUX-schnell [43] text to image generation using the base VLM response (left) and the response using iSVP (right) as input. We generated a single image without multiple attempts or selective filtering. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: FLUX-schnell [43] text to image generation using the base VLM response (left) and the response using iSVP (right) as input. We generated a single image without multiple attempts or selective filtering. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: SVP effectively boosts captioning performance and reduces hallucinations on benchmark tasks using LLaVA-1.6-7b as base model. The second iteration of SVP adaptation leads to significant improvements compared to the initial round, underscoring the value of this techniq…
Figure 22
Figure 22. Figure 22: Model size comparison using the F1 metric on the POPE dataset. SVP improves the base model and achieves better or comparable performance with models five times larger. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]
Figure 25
Figure 25. Figure 25: Distribution of groundable objects in generated caption sampling the base model pθ(z|c) and the grounded model q(z|c, g). Models adapted with SVP generate less groundable objects and have better object recall. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 27
Figure 27. Figure 27: Comparison between rationalization in STaR [108] and posterior sampling in SVP. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_27.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

118 extracted references · 21 canonical work pages

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Agrawal, K

    H. Agrawal, K. Desai, Y . Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019

    2019

  3. [3]

    Ahmadian, C

    A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  4. [4]

    Akyürek, D

    E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022

    arXiv 2022

  5. [5]

    M. L. Anderson. Neural reuse: A fundamental organizational principle of the brain. Behavioral and brain sciences, 33(4):245–266, 2010

    2010

  6. [6]

    Anthony, Z

    T. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017

    2017

  7. [7]

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  8. [8]

    Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

Show all 118 references
  1. [9]

    Banerjee and A

    S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  2. [10]

    L. W. Barsalou. Grounded cognition. Annu. Rev. Psychol., 59(1):617–645, 2008

    2008

  3. [11]

    Beyer, A

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024

    2024 arXiv

  4. [12]

    Bordes, R

    F. Bordes, R. Y . Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Mañas, Z. Lin, A. Mahmoud, B. Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247, 2024

    2024 arXiv

  5. [13]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B....

    2020

  6. [14]

    Carion, F

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020

    2020

  7. [15]

    Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  8. [16]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  9. [17]

    Chiang, Z

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    2023

  10. [18]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  11. [19]

    Collerton, J

    D. Collerton, J. Barnes, N. J. Diederich, R. Dudley, K. Friston, C. G. Goetz, J. G. Goldman, R. Jardri, J. Kulisevsky, S. J. Lewis, et al. Understanding visual hallucinations: A new synthesis. Neuroscience & Biobehavioral Reviews, 150:105208, 2023. 13

    2023

  12. [20]

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  13. [21]

    A. Dave, P. Dollár, D. Ramanan, A. Kirillov, and R. Girshick. Evaluating large-vocabulary object detectors: The devil is in the details. arXiv preprint arXiv:2102.01066, 2021

    2021 arXiv

  14. [22]

    H. Dong, W. Xiong, D. Goyal, Y . Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023

    2023 arXiv

  15. [23]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  16. [24]

    Favero, L

    A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303–14312, 2024

    2024

  17. [25]

    C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. 70:1126–1135, 2017

    2017

  18. [26]

    Girshick, J

    R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014

    2014

  19. [27]

    A. M. Glenberg and M. P. Kaschak. Grounding language in action. Psychonomic bulletin & review, 9(3):558–565, 2002

    2002

  20. [28]

    D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.08215, 2023

    2023 arXiv

  21. [29]

    Gulcehre, T

    C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ah- ern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  22. [30]

    Gupta, P

    A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019

    2019

  23. [31]

    Hattie and H

    J. Hattie and H. Timperley. The power of feedback. Review of educational research, 77(1):81–112, 2007

    2007

  24. [32]

    Y . He, F. Huang, X. Jiang, Y . Nie, M. Wang, J. Wang, and H. Chen. Foundation model for advancing healthcare: Challenges, opportunities, and future directions. arXiv preprint arXiv:2404.03264, 2024

    2024 arXiv

  25. [33]

    M. D. Hoffman, D. Phan, D. Dohan, S. Douglas, T. A. Le, A. Parisi, P. Sountsov, C. Sutton, S. Vikram, and R. A Saurous. Training chain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36, 2024

    2024

  26. [34]

    Hospedales, A

    T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey. ArXiv preprint, abs/2004.05439, 2020

    2004 arXiv

  27. [35]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  28. [36]

    D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  29. [37]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  30. [38]

    M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999

    1999

  31. [39]

    J. Kang, L. Karlinsky, H. Luo, Z. Wang, J. Hansen, J. Glass, D. Cox, R. Panda, R. Feris, and A. Ritter. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406.12034, 2024. 14

    2024 arXiv

  32. [40]

    J. Kang, H. Luo, Y . Zhu, J. Hansen, J. Glass, D. Cox, A. Ritter, R. Feris, and L. Karlinsky. Self- specialization: Uncovering latent expertise within large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 2681–2706, 2024

    2024

  33. [41]

    Kazemzadeh, V

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  34. [42]

    Kiefer and L

    M. Kiefer and L. W. Barsalou. Grounding the human conceptual system in perception, action, and internal states. 2013

    2013

  35. [43]

    B. F. Labs. Flux. https://github.com/black-forest-labs/flux , 2024

    2024

  36. [44]

    S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024

    2024

  37. [45]

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y . Li, Z. Liu, and C. Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  38. [46]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  39. [47]

    J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In ICML, 2022

    2022

  40. [48]

    L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y . Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022

    2022

  41. [49]

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  42. [50]

    Z. Li, F. Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. ArXiv preprint, abs/1707.09835, 2017

    2017 arXiv

  43. [51]

    C.-Y . Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  44. [52]

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024

    2024

  45. [53]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755....

    2014

  46. [54]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  47. [55]

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  48. [56]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  49. [57]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  50. [58]

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025

    2025

  51. [59]

    Madaan, N

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024. 15

    2024

  52. [60]

    Oquab, L

    M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717–1724, 2014

    2014

  53. [61]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    2022 arXiv

  54. [62]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  55. [63]

    J. W. Peirce. Understanding mid-level representations in visual processing. Journal of Vision, 15(7):5–5, 2015

    2015

  56. [64]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  57. [65]

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, Q. Ye, and F. Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations, 2024

    2024

  58. [66]

    Peters and S

    J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007

    2007

  59. [67]

    Picard, K

    C. Picard, K. M. Edwards, A. C. Doris, B. Man, G. Giannone, M. F. Alam, and F. Ahmed. From concept to manufacturing: Evaluating vision-language models for engineering design. arXiv preprint arXiv:2311.12668, 2023

    2023 arXiv

  60. [68]

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015

    2015

  61. [69]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021

    2021

  62. [70]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference opti- mization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  63. [71]

    Rahmanzadehgervi, L

    P. Rahmanzadehgervi, L. Bolton, M. R. Taesiri, and A. T. Nguyen. Vision language models are blind. arXiv preprint arXiv:2407.06581, 2024

    2024 arXiv

  64. [72]

    J. Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

    2016

  65. [73]

    N. L. Roux, M. G. Bellemare, J. Lebensold, A. Bergeron, J. Greaves, A. Fréchette, C. Pelletier, E. Thibodeau-Laufer, S. Toth, and S. Work. Tapered off-policy reinforce: Stable and efficient rein- forcement learning for llms. arXiv preprint arXiv:2503.14286, 2025

    2025 arXiv

  66. [74]

    Saikh, T

    T. Saikh, T. Ghosal, A. Mittal, A. Ekbal, and P. Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301, 2022

    2022

  67. [75]

    Sasse, S

    K. Sasse, S. Chen, J. Pond, D. Bitterman, and J. Osborne. Mapping bias in vision language models: Signposts, pitfalls, and the road ahead. arXiv preprint arXiv:2410.13146, 2024

    2024 arXiv

  68. [76]

    Schaul and J

    T. Schaul and J. Schmidhuber. Metalearning. Scholarpedia, 5(6):4650, 2010

    2010

  69. [77]

    Schmidhuber

    J. Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987

    1987

  70. [78]

    Schmidhuber

    J. Schmidhuber. A general method for incremental self-improvement and multi-agent learning. In Evolutionary Computation: Theory and Applications, pages 81–123. World Scientific, 1999

    1999

  71. [79]

    B. Settles. Active learning literature survey. 2009

    2009

  72. [80]

    Shinn, F

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 16

    2024

  73. [81]

    Silver, A

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016

    2016

  74. [82]

    Silver, J

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017

    2017

  75. [83]

    B. Song, R. Zhou, and F. Ahmed. Multi-modal machine learning in engineering design: A review and future directions. Journal of Computing and Information Science in Engineering, 24(1):010801, 2024

    2024

  76. [84]

    Sudalairaj, A

    S. Sudalairaj, A. Bhandwaldar, A. Pareja, K. Xu, D. D. Cox, and A. Srivastava. Lab: Large-scale alignment for chatbots. arXiv preprint arXiv:2403.01081, 2024

    2024 arXiv

  77. [85]

    Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y . Shen, C. Gan, L.-Y . Gui, Y .-X. Wang, Y . Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  78. [86]

    Z. Sun, Y . Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y . Yang, and C. Gan. Principle-driven self- alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024

    2024

  79. [87]

    Tenenbaum and E

    G. Tenenbaum and E. Goldring. A meta-analysis of the effect of enhanced instruction: cues, participation, reinforcement and feedback and correctives on motor skill learning. Journal of Research & Development in Education, 1989

    1989

  80. [88]

    J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011

    2011

  81. [89]

    S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024

    2024 arXiv

  82. [90]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  83. [91]

    G. Vallar. Spatial neglect, balint-homes’ and gerstmann’s syndrome, and other spatial disorders. Cns Spectrums, 12(7):527–536, 2007

    2007

  84. [92]

    Vedantam, C

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015

    2015

  85. [93]

    D. Wan, J. Cho, E. Stengel-Eskin, and M. Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02325, 2024

    2024 arXiv

  86. [94]

    X. Wang, J. Chen, Z. Wang, Y . Zhou, Y . Zhou, H. Yao, T. Zhou, T. Goldstein, P. Bhatia, F. Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint arXiv:2405.15973, 2024

    2024 arXiv

  87. [95]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  88. [96]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  89. [97]

    S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021

    2021 arXiv

  90. [98]

    J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  91. [99]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  92. [100]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 17

    2022 arXiv

  93. [101]

    Q. Ye, H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q. Qian, J. Zhang, and F. Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13040–13051, 2024

    2024

  94. [102]

    S. Yin, C. Fu, S. Zhao, T. Xu, H. Wang, D. Sui, Y . Shen, K. Li, X. Sun, and E. Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045, 2023

    2023 arXiv

  95. [103]

    H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y . Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023

    2023 arXiv

  96. [104]

    Young, B

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  97. [105]

    L. Yuan, D. Chen, Y .-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021

    2021 arXiv

  98. [106]

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  99. [107]

    Zelikman, G

    E. Zelikman, G. Harik, Y . Shao, V . Jayasiri, N. Haber, and N. D. Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024

    2024 arXiv

  100. [108]

    Zelikman, Y

    E. Zelikman, Y . Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  101. [109]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  102. [110]

    Zhang, Y

    D. Zhang, Y . Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024

    2024 arXiv

  103. [111]

    Zhang, F

    H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y . Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022

    2022 arXiv

  104. [112]

    Zhang, H

    H. Zhang, H. Li, F. Li, T. Ren, X. Zou, S. Liu, S. Huang, J. Gao, C. Li, J. Yang, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19–35. Springer, 2025

    2025

  105. [113]

    Zhang, B

    K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y . Zhang, J. Yang, C. Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024

    2024 arXiv

  106. [114]

    Z. Zhao, B. Wang, L. Ouyang, X. Dong, J. Wang, and C. He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839, 2023

    2023 arXiv

  107. [115]

    Y . Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024

    2024 arXiv

  108. [116]

    Y . Zhou, Z. Fan, D. Cheng, S. Yang, Z. Chen, C. Cui, X. Wang, Y . Li, L. Zhang, and H. Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622, 2024

    2024 arXiv

  109. [117]

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  110. [118]

    A tree with a donut as an ornament

    K. Zhu, L. Zhao, Z. Ge, and X. Zhang. Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 291–300, 2024. 18 A Sampling-based Visual Projection Workflow Instruction: Please describe the content of this image ...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.