Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generation

Fernando Gabriela Garcia; Hunter Young; Ryan Shaw; Spencer Burns

REVIEW 5 major objections 6 minor 32 references

Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage self-supervised training strategy, built around hierarchical self-captioning and internal sub-prompt planning, gives a large vision-language text-to-image model consistent wins over four established baselines on nine…

desk verdict Coherent two-stage self-supervised T2I method, but the evidence is thin and the fine-tuning/evaluation split may be contaminated. read the letter →

arxiv 2507.04151 v1 pith:OWTZRK4E submitted 2025-07-05 cs.CV

Fernando Gabriela Garcia , Spencer Burns , Ryan Shaw , Hunter Young This is my paper

classification cs.CV

keywords hierarchicalself-supervisiontext-to-imagegenerationlargevision-languagemodelscompositionalcontrolinternalplanningsemanticconsistencylossPlan2Genbenchmarkself-captioning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a text-to-image model can gain compositional control without expensively curated paired data by teaching its large vision-language backbone to caption images hierarchically and then use those captions as internal sub-prompts when generating. The proposed two-stage system, Hi-SSLVLM, first self-supervises multi-granularity grounding (global captions plus per-object captions, checked by a frozen CLIP encoder), then plans each image through an Internal Compositional Planning step and trains with a semantic consistency loss that ties every sub-prompt to the output image. On the Plan2Gen benchmark, evaluated by Gemini-2.0-Flash and InternVL3-78B across nine fine-grained dimensions, the paper reports that Hi-SSLVLM outscores Janus-Pro-1B, Stable Diffusion XL 1.0, DeepFloyd IF v1.0, and ControlNet-XL on every dimension, with average scores of 2.29 and 2.25. A four-variant ablation and a 20-rater human study are presented as supporting evidence. If the paper is right, fine-grained compositional adherence can be learned from uncurated image-caption pairs rather than requiring manual annotation of every object, attribute, and relationship.

What carries the argument

The load-bearing mechanism is the two-stage self-supervision loop. Stage 1, Multi-Granularity Visual-Language Grounding, lets the LVLM auto-regressively generate global and local captions for an image, with a frozen CLIP-based encoder providing pseudo-feedback through the alignment terms $L_{ ext{global}}$ and $L_{ ext{local}}$. Stage 2, Self-Refinement and Guided Image Generation, uses Internal Compositional Planning (ICP) to auto-regressively produce a sequence of sub-prompts $T_{ ext{sub}}$ that condition a latent diffusion decoder together with the original prompt, while the Semantic Consistency Loss $L_{ ext{consistency}}$ penalizes mismatch between each sub-prompt embedding and the generated image embedding. This loop lets the model generate its own structural annotations, plan compositions before rendering pixels, and correct itself during training.

What would settle it

Re-run the Plan2Gen prompts with each model generating several images per prompt and have Gemini-2.0-Flash and InternVL3-78B score each image multiple times; then compute per-dimension means with confidence intervals. If the 95% intervals for Hi-SSLVLM and the runner-up overlap on more than a few dimensions, or if repeated scoring of identical images shifts by more than about 0.05, the claim of consistent superiority is not supported.

Watch

Extended reading notes

Core claim

Hi-SSLVLM is a generative LVLM that produces images from complex prompts better than four established baselines because it is trained to know image composition before it generates it. The central claim is that a two-stage self-supervised objective suffices to internalize compositional semantics: Stage 1 forces the LVLM to write global and local captions for images and align them with a frozen visual encoder; Stage 2 makes the LVLM decompose each prompt into sub-prompts that condition a latent diffusion decoder, with a semantic consistency loss enforcing that each sub-prompt is reflected in the generated image. Measured on the Plan2Gen benchmark's nine dimensions, the authors claim consistent superiority under both automated LVLM judges and higher human Likert ratings.

Load-bearing premise

The load-bearing premise is that single-pass mean scores from two automated judges are meaningful at the observed resolution: the reported wins are 0.01 to 0.10 per dimension and 0.04 to 0.05 on averages, with no variance, confidence intervals, or inter-judge agreement reported, so if judge noise exceeds these deltas the superiority claim collapses to statistical ties.

Editorial extensions

If this is right

If the reported results hold, compositional text-to-image training no longer depends on meticulously annotated paired data: large uncurated web corpora appear sufficient to teach fine-grained visual semantics.
The Internal Compositional Planning step provides a concrete mechanism for improving the hardest dimensions in the paper, Text and Composition, because the model decomposes a prompt before rendering it.
The reported gains on unseen compositions imply the learned structure transfers to novel object-attribute combinations rather than memorizing training layouts.
Agreement between two automated judges and human raters suggests the advantage is perceptible, not an artifact of a single evaluator.
The ablation ordering, with ICP removal hurting most, identifies the sub-prompt planning step as the component most responsible for the claimed compositional control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: because the winning margins are 0.01 to 0.10 on individual dimensions, the paper's fine-grained ranking is only as strong as judge stability; a direct next experiment is repeated scoring of identical images to measure evaluator noise, which the paper does not report.
Extension: the same self-captioning and semantic-consistency losses are not specific to still images and could be applied to conditional video or layout-to-image generators, since both losses operate on text and image embeddings rather than pixels.
Extension: the semantic consistency term could be used as a training-free reranker at inference, selecting among several candidate images the one whose embedding best matches each planned sub-prompt.
Extension: a fairer cross-check of the headline claim would include the same ablations under both LVLM judges, since the paper reports ablation averages only under InternVL3-78B.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper builds a coherent two-stage self-supervised text-to-image system, but the evidence for its headline claim is thin, and there's a train/test contamination question sitting right in the introduction that the authors never address.

The good part: the architecture is clearly laid out. Stage 1 makes the LVLM generate hierarchical captions and aligns them with frozen CLIP; Stage 2 uses those abilities to plan sub-prompts and guides a latent diffusion decoder, with a semantic consistency loss that ties sub-prompts to the generated image. The method is a clean recombination of known pieces (BLIP-style bootstrapping, sub-prompt planning from refs [1] and [29], CLIP similarity losses), and the paper says so. The two-stage logic is sensible.

The problems are all in the evidence. First, the intro says fine-tuning and evaluation use the same 'specifically designed benchmarks for compositional generation [1]'. That is the Plan2Gen benchmark (though reference [1] is actually titled 'Draw all your imagine,' a separate sloppiness). No held-out split or disjointness check is reported, so Tables I and II may measure the model's memory of test prompts rather than compositional generalization. That is a load-bearing flaw. Second, the quantitative wins are tiny—0.01 to 0.10 on nine-point-ish scales, no error bars, no significance tests, and a best-of-four image selection for each model. The human study lists averages only, no variance or inter-annotator agreement. At that resolution the 'consistently surpasses all baselines' claim is mostly statistical tie. Third, the semantic consistency loss (Eq. 14) maximizes CLIP similarity between sub-prompts and the generated image; that is a reasonable training signal, but it is aligned with the same kind of semantic matching the LVLM judges are scoring, so part of the measured gap is likely the training objective showing up in the metric. Not fatal, but worth naming. The reader's claim that Eq. 14 doesn't match the described per-region mechanism is off: the text explicitly describes whole-image alignment in Stage 2; the per-region mechanism is only in Stage 1.

Citation wise, the paper is messy (wrong title for [1], GPT-3 cited through a video-language paper, a dense cluster of Y. Zhou refs), which makes me less confident about the authors' reading of the literature.

Who is it for? Someone working on T2I evaluation methodology could mine it as a cautionary example. The method itself is not a major advance. I would not cite it.

I'd send it to peer review anyway, because the contamination issue is checkable and the architecture is coherent enough that a serious referee could force the authors to clarify the split and release code/data. Without that, it's a reject.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces Hi-SSLVLM, a two-stage self-supervised framework for text-to-image generation built on an LVLM backbone. Stage 1 trains the model to produce global and local captions aligned with a frozen CLIP encoder, while Stage 2 uses an Internal Compositional Planning step to generate sub-prompts that condition a latent diffusion decoder, together with a semantic consistency loss. The paper reports evaluations on the Plan2Gen benchmark judged by Gemini-2.0-Flash and InternVL3-78B, claiming that Hi-SSLVLM outperforms Janus-Pro-1B, SDXL 1.0, DeepFloyd IF v1.0, and ControlNet-XL on all nine fine-grained dimensions, along with an ablation study and a human evaluation.

Significance. If the central claims were supported, the proposed method would offer a practically useful way to improve compositional text-to-image fidelity while reducing reliance on carefully curated paired data. The raw numbers in Tables I and II do show Hi-SSLVLM first on every dimension, so the paper is not contradicted by its own reported data. However, the empirical case as presented is not convincing. The most serious problem is that the same Plan2Gen benchmark used for evaluation is explicitly stated to be used for fine-tuning, making the reported superiority potentially an artifact of training on the test prompts. In addition, the best-of-four selection protocol uses the judge models themselves to select the evaluated image, and no variance, confidence intervals, or significance tests are reported for any table. The method is specified only schematically, with no model sizes, hyperparameters, or code release, so the experiments cannot be reproduced. These issues are load-bearing rather than cosmetic, and they prevent the paper from supporting its title claim of unlocking compositional control.

major comments (5)

[Section I (Introduction)] The Introduction states: 'For fine-tuning and evaluation, we utilize more meticulously curated datasets that contain complex compositional elements and diverse attributes, such as COCO [11] and specifically designed benchmarks for compositional generation [1].' Reference [1] is the Plan2Gen benchmark that is also used for all reported results (Tables I, II, V, VI, VII), and the paper gives no held-out split, prompt deduplication, or overlapping-prompt check. If the fine-tuning corpus contains the evaluation prompts or near-duplicates, Tables I/II and the abstract's 'consistently surpasses all established baselines' claim measure memorization rather than compositional generalization. The authors must either document a disjoint evaluation split (e.g., prompts excluded before fine-tuning) or re-run the comparisons on a benchmark not used in training; without that, the central claim is invalid.
[Section IV.A.2 (Experimental Setup, Datasets)] The evaluation protocol selects, for each prompt, the image judged best by the automated evaluators: 'For the automated evaluation metrics, the image that achieved the highest score among these four (as determined by the automated evaluators) was selected for final assessment.' This inflates the reported scores because the same LVLMs that produce the final judgment are used to choose the candidate, converting the evaluation into a max over draws rather than a typical performance estimate. Tables I and II report no standard deviations, confidence intervals, or significance tests; the claimed advantage over the runner-up is only 0.04-0.05 on the averages and 0.01-0.10 on individual dimensions. Without any measure of variance, the claim that Hi-SSLVLM 'consistently surpasses' all baselines on every dimension is not established.
[Section III (Method) and Section IV.A (Experimental Setup)] The architecture is described only schematically: the paper never identifies the specific LVLM backbone, the latent diffusion decoder, the exact Stage 1 and Stage 2 training corpora (beyond mentioning LAION-400M, Conceptual Captions, COCO, and [1]), the number of training steps, the hyperparameters lambda_global, lambda_local, and alpha, or the counts P and K for sub-prompts and detected objects. No code or model weights are released. This lack of specification makes the experiments irreproducible and prevents external verification of the claimed two-stage pipeline.
[Section IV.E.3 (Generalization to Unseen Compositions)] The paper asserts that a specialized test set of 100 prompts was 'deliberately excluded from the training data of any model,' but gives no operational procedure for guaranteeing this for closed baselines such as Janus-Pro-1B, SDXL, or DeepFloyd IF, whose training data are not public, nor for the authors' own model if it was trained on [1]. This unverifiable claim is load-bearing for the generalization result in Table VII; without a concrete protocol (e.g., prompt-level exclusion recorded before training and a duplicated-prompt check), Section IV.E.3 and Table VII should be removed or substantially reinterpreted.
[Section IV.D (Human Evaluation) and Table III (Ablation)] The human evaluation reports mean Likert scores from 20 annotators over 200 prompts, but gives no inter-annotator agreement (e.g., Krippendorff's alpha), no per-item standard deviation, and no significance test. The differences between Hi-SSLVLM and the next best baseline are 0.30 (fidelity), 0.40 (compositional accuracy), and 0.20 (aesthetics); without variance information these differences cannot be declared 'significantly higher.' The same lack of variance applies to the ablation study in Table III, where the full model exceeds the ablated variants by only 0.04-0.09, so the conclusion that each component is 'indispensable' is not supported.

minor comments (6)

[Section IV.A.2] The text states that the COCO-Stuff test set was used for evaluation, but no COCO-Stuff results appear anywhere in Section IV; either add the results or remove the claim.
[Section II.B] Reference [29], 'Improving compositional text-to-image generation with large vision-language models,' is cited as a directly relevant method, but this model is not included as a baseline in Tables I/II; its absence should be justified.
[Section IV.B] The semantic consistency loss in Eq. (14) uses frozen CLIP encoders, whereas the benchmark judges are Gemini-2.0-Flash and InternVL3-78B, so the loss and the evaluator are not literally identical; however, both reward text-image semantic alignment, and the paper should discuss this overlap and, ideally, include an evaluation metric not aligned with the training objective.
[Abstract and Section V] The abstract and conclusion use strong qualitative language such as 'impeccable text rendering' and 'unprecedented control,' but the paper contains no qualitative image examples (figures) to support these subjective characterizations.
[Throughout (notation and typos)] There are several presentation issues: 'T2I generation ,' has a stray space before the comma; 'a advanced Transformer-based network' should be 'an advanced Transformer-based network'; Eq. (11) introduces Tsub as a sequence while Eq. (14) treats it as a set; and Eq. (12) refers to projections of sub-prompt embeddings without showing the projection operator, which should be stated for clarity.
[Table III] The ablation study would be far more informative if it reported per-dimension scores rather than only averages, since the paper's central claim is about superiority on every fine-grained dimension.

Circularity Check

1 steps flagged · score 6.0 of 10

The paper fine-tunes on the same compositional benchmark [1] it later evaluates on, so the central Tables I/II superiority claim reduces to a fitted-input measurement rather than an independent prediction.

fitted input called prediction [Section I (Introduction) and Section IV.A.2 (Datasets)]
"“For fine-tuning and evaluation, we utilize more meticulously curated datasets that contain complex compositional elements and diverse attributes, such as COCO [11] and specifically designed benchmarks for compositional generation [1].” ... “Plan2Gen Benchmark Set: This meticulously curated dataset consists of a specialized collection of challenging textual prompts.”"

The paper’s central claim is that Hi-SSLVLM “consistently surpasses all established baselines across every single one of the nine fine-grained dimensions” on Plan2Gen-style benchmarks. Evaluation is run on the Plan2Gen Benchmark Set (Section IV.A.2), whose source is reference [1]. The same reference [1] is named as fine-tuning data in Section I, and the paper reports no held-out split, prompt exclusion, or disjointness check. Therefore the high scores in Tables I/II are not shown to be independent of the model’s training distribution; by the paper’s own stated setup, the reported “prediction” is a re-measurement of prompts the model was fitted on. This is a fitted-input-called-prediction circularity rather than an external generalization result.

full rationale

One load-bearing circular step is present. The paper explicitly says “For fine-tuning and evaluation, we utilize ... specifically designed benchmarks for compositional generation [1]” and then evaluates on the Plan2Gen Benchmark Set from that same reference, without stating that those prompts were withheld from fine-tuning. If the same prompt set is used, the central quantitative superiority claim in Section IV.B and Tables I/II reduces to memorization of the evaluation prompts; the paper provides no evidence to rule this out. The ablation and human evaluation inherit the same setup. The semantic consistency loss (Eq. 14) is not itself circular by construction: it optimizes frozen-CLIP image-text similarity, which is a different model from the Gemini-2.0-Flash and InternVL3-78B evaluators, so optimizing that loss and then measuring improvements on those evaluators is a legitimate, if expected, training effect. No self-citation chain is load-bearing; the cited benchmark and related works are by other authors. The score is 6 because the central claim is partially reduced by construction through the fine-tuning/evaluation overlap, while the method’s internal components retain independent descriptive content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The central empirical claims depend on a small set of unreported hyperparameters and on domain assumptions about the sensitivity and validity of judge-based scores and CLIP-based feedback. Stage 1 and Stage 2 losses (Eqs. 8-10, 14-15) all route through frozen CLIP encoders, so the entire mechanism inherits CLIP's failure modes for fine-grained attributes (texture, pose, lighting, text rendering) without any calibration presented.

free parameters (4)

lambda_global and lambda_local = not reported
Stage 1 alignment loss weights (Eq. 10) balancing global and local CLIP alignment; never given, so the balance of the claimed grounding signal is unspecified.
alpha = not reported
Stage 2 weight on the consistency loss (Eq. 15); never given, though the ablation claims the loss is critical.
Combine() fusion rule = not reported
Eq. 12 fuses the input prompt embedding and the sub-prompt embeddings, but the paper never states whether fusion is concatenation or weighted summation or what the weights are.
number of sub-prompts P and detected objects K = not reported
Both the planning step (Eq. 11) and the object set (Eq. 7) have unspecified sizes, which changes the strength of Eq. 9 and Eq. 14.

assumptions (5)

domain assumption Frozen CLIP encoders provide a valid, non-degenerate training signal for fine-grained local attribute alignment.
Equations 8 to 10 and 14 route all feedback through frozen CLIP encoders; the paper offers no evidence that CLIP cosine similarity is informative for the texture, pose, lighting, and text-rendering dimensions the benchmarks claim to measure.
domain assumption LVLM judge scores from Gemini-2.0-Flash and InternVL3-78B are valid and sensitive enough to rank models at 0.01 to 0.10 point deltas.
All quantitative claims rest on these single-point means without variance, confidence intervals, or inter-judge agreement (Section IV.A.3, Tables I-II).
domain assumption Captions and sub-prompts generated by the LVLM are of sufficient quality to serve as training targets and conditioning signals.
Stage 1 and Stage 2 both condition on the model's own text output (Eqs. 7 and 11); if self-generated captions drift or collapse to generic text, the losses in Eq. 9 and Eq. 14 can be minimized trivially.
domain assumption Web-scraped paired image-caption data (LAION-400M, Conceptual Captions) suffice for the stated self-supervision without further curation.
Section IV.A.2 names these datasets as pretraining data, but the paper gives no filtering or deduplication steps, and the 'self-supervised' framing still uses paired image-text data.
domain assumption The evaluation benchmark of reference [1] measures what the paper claims it measures.
The benchmark is the sole evaluation backbone, but the abstract misnames it ('Plan2Gen') and reference [1]'s actual title is different, so the benchmark's properties are inherited unverified.

invented entities (3)

Multi-Granularity Visual-Language Grounding (Stage 1)
purpose: Trains the LVLM to produce hierarchical global and local captions aligned to images via frozen CLIP pseudo-feedback (Eqs. 7-10).
No released artifacts, no qualitative demonstration of the generated hierarchical captions, and no calibration of the CLIP feedback; the only evidence is the aggregate ablation delta of 0.07.
Internal Compositional Planning (ICP)
purpose: Auto-regressively generates hierarchical sub-prompts that condition the diffusion decoder (Eqs. 11-12).
No sample plans are shown and the planning prompts are not quoted; the ablation delta attributed to ICP is 0.09 without variance.
Semantic Consistency Loss
purpose: Aligns each planned sub-prompt with the generated image (Eq. 14).
As written, the loss compares sub-prompts to the whole-image embedding, not to matching regions, so the claimed per-region mechanism is not independently evidenced; ablation delta is 0.04.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generation." pith.science (2026). https://pith.science/paper/OWTZRK4E

@misc{pith2026250704151,
  author       = {Pith},
  title        = {Pith review of: Unlocking Compositional Control: Self-Supervision for LVLM-Based Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWTZRK4E}},
  note         = {Machine review of arXiv:2507.04151}
}

read the original abstract

This paper introduces Hierarchical Self-Supervised LVLM (Hi-SSLVLM), a novel generative model designed to significantly advance text-to-image synthesis, particularly for complex and compositionally challenging prompts. Traditional methods often grapple with the high cost of meticulously curated paired image-text datasets and struggle with precise control over fine-grained visual attributes and intricate spatial relationships. Our Hi-SSLVLM addresses these limitations through a unique two-stage self-supervised learning strategy. The first stage, Multi-Granularity Visual-Language Grounding, enables the Large Vision-Language Model (LVLM) backbone to autonomously generate and align hierarchical captions (global and local) to images, cultivating a deep internal semantic understanding without reliance on extensive human annotation. The second stage, Self-Refinement and Guided Image Generation, leverages this acquired knowledge by an Internal Compositional Planning (ICP) mechanism, where the LVLM first formulates detailed textual sub-prompts to guide the image generation process, complemented by a novel Semantic Consistency Loss for precise output alignment. Comprehensive experiments against leading baselines, including Janus-Pro-1B, Stable Diffusion XL 1.0, DeepFloyd IF v1.0, and ControlNet-XL, on multi-dimensional benchmarks such as Gemini-2.0-Flash and InternVL3-78B, demonstrate Hi-SSLVLM's superior performance across all fine-grained metrics. An in-depth ablation study confirms the critical role of each proposed component. Furthermore, human evaluations corroborate our quantitative findings, highlighting Hi-SSLVLM's enhanced fidelity to prompt, compositional accuracy, and overall aesthetic quality, marking a significant step towards more controllable and semantically consistent open-ended text-to-image generation.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 13 canonical work pages

[1]

Draw all your imagine: A holistic benchmark and agent framework for complex instruction-based image generation,

Y . Zhou, J. Yuan, and Q. Wang, “Draw all your imagine: A holistic benchmark and agent framework for complex instruction-based image generation,” arXiv preprint arXiv:2505.24787 , 2025

arXiv 2025
[29]

Improving compositional text-to-image generation with large vision-language models,

S. Wen, G. Fang, R. Zhang, P. Gao, H. Dong, and D. N. Metaxas, “Improving compositional text-to-image generation with large vision-language models,” CoRR, vol. abs/2310.06311, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.06311 12

arXiv doi:10.48550/arxiv.2310.06311 2023
[11]

Microsoft COCO: common objects in context,

T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V , ser. Lecture Notes in Computer Science, D. J. Fleet, T. Pajdla, B. Schiele, and T. Tuytela...

doi:10.1007/978-3-319-10602-1 2014
[2]

Score: Story coherence and retrieval enhancement for ai narratives,

Q. Yi, Y . He, J. Wang, X. Song, S. Qian, X. Yuan, M. Zhang, L. Sun, K. Li, K. Lu et al., “Score: Story coherence and retrieval enhancement for ai narratives,” arXiv preprint arXiv:2503.23512 , 2025

arXiv 2025
[3]

Enhancing code llms with reinforcement learning in code generation,

J. Wang, Z. Zhang, Y . He, Y . Song, T. Shi, Y . Li, H. Xu, K. Wu, G. Qian, Q. Chen et al. , “Enhancing code llms with reinforcement learning in code generation,” arXiv preprint arXiv:2412.20367 , 2024

arXiv 2024
[4]

Diffusion models beat gans on image synthesis,

P. Dhariwal and A. Q. Nichol, “Diffusion models beat gans on image synthesis,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , M. Ranzato, A. Beygelzimer, Y . N. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 8780–8794. [Online]. A...

work page 2021
[5]

Triple sequence generative adversarial nets for unsupervised image captioning,

Y . Zhou, W. Tao, and W. Zhang, “Triple sequence generative adversarial nets for unsupervised image captioning,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7598–7602

work page 2021
[6]

Visual in-context learning for large vision-language models,

Y . Zhou, X. Li, Q. Wang, and J. Shen, “Visual in-context learning for large vision-language models,” in Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 . Association for Computational Linguis- tics, 2024, pp. 15 890–15 902

2024

Show all 32 references

[7]

Weak to strong generalization for large language models with multi-capabilities,

Y . Zhou, J. Shen, and Y . Cheng, “Weak to strong generalization for large language models with multi-capabilities,” in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https://openreview.net/forum?id=N1vYivuSKq

2025
[8]

Training medical large vision- language models with abnormal-aware feedback,

Y . Zhou, L. Song, and J. Shen, “Training medical large vision- language models with abnormal-aware feedback,” arXiv preprint arXiv:2501.01377, 2025

2025 arXiv
[9]

LAION-400M: open dataset of clip-filtered 400 million image-text pairs,

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “LAION-400M: open dataset of clip-filtered 400 million image-text pairs,” CoRR, vol. abs/2111.02114, 2021. [Online]. Available: https://arxiv.org/abs/2111. 02114

2021 arXiv
[10]

Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning,

P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. ...

2018
[12]

Beyond empathy: Integrating diagnostic and therapeutic reasoning with large language models for mental health counseling,

H. Hu, Y . Zhou, J. Si, Q. Wang, H. Zhang, F. Ren, F. Ma, and L. Cui, “Beyond empathy: Integrating diagnostic and therapeutic reasoning with large language models for mental health counseling,” arXiv preprint arXiv:2505.15715, 2025

2025
[13]

Multimodal event transformer for image-guided story ending generation,

Y . Zhou and G. Long, “Multimodal event transformer for image-guided story ending generation,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , 2023, pp. 3434–3444

2023
[14]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in 9th International Conference on Lear...

2021
[15]

Language models with image descriptors are strong few- shot video-language learners,

Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem, S. Chang, M. Bansal, and H. Ji, “Language models with image descriptors are strong few- shot video-language learners,” in Advances in Neural Information Processing Systems 35: Annual Conference...

2022
[16]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine L...

2021
[17]

BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation,

J. Li, D. Li, C. Xiong, and S. C. H. Hoi, “BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , ser. Proceedings of Mac...

2022
[18]

Flamingo: a visual language model for few-shot learning,

J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barre...

2022
[19]

Cross-lingual transfer of large language model by visually- derived supervision toward low-resource languages,

M. Muraoka, B. Bhattacharjee, M. Merler, G. Blackwood, Y . Li, and Y . Zhao, “Cross-lingual transfer of large language model by visually- derived supervision toward low-resource languages,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 3637– 3646

2023
[20]

Coca: Contrastive captioners are image-text foundation models,

J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation models,” Trans. Mach. Learn. Res. , vol. 2022, 2022. [Online]. Available: https://openreview.net/forum?id=Ee277P3AYC

2022
[21]

Tx-llava: Large language and vision assistant for temporal changes in chest x-rays,

H. Elgendy and H. Cholakkal, “Tx-llava: Large language and vision assistant for temporal changes in chest x-rays,” in 22nd IEEE International Symposium on Biomedical Imaging, ISBI 2025, Houston, TX, USA, April 14-17, 2025 . IEEE, 2025, pp. 1–4. [Online]. Available: https://doi...

2025
[22]

Mam: Modular multi-agent framework for multi-modal medical diagnosis via role-specialized collaboration,

Y . Zhou, L. Song, and J. Shen, “Mam: Modular multi-agent framework for multi-modal medical diagnosis via role-specialized collaboration,” arXiv preprint arXiv:2506.19835 , 2025

2025 arXiv
[23]

GLIDE: towards photorealistic image generation and editing with text-guided diffusion models,

A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “GLIDE: towards photorealistic image generation and editing with text-guided diffusion models,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltim...

2022
[24]

Taming transformers for high-resolution image synthesis,

P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25,

2021
[25]

Hierarchical text-conditional image generation with CLIP latents,

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with CLIP latents,” CoRR, vol. abs/2204.06125, 2022. [Online]. Available: https://doi.org/ 10.48550/arXiv.2204.06125

2022 doi arXiv
[26]

BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,

J. Li, D. Li, S. Savarese, and S. C. H. Hoi, “BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,” in International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , ser. Proceedings of Machi...

2023
[27]

Self-rewarding large vision- language models for optimizing prompts in text-to-image generation,

H. Yang, Y . Zhou, W. Han, and J. Shen, “Self-rewarding large vision- language models for optimizing prompts in text-to-image generation,” arXiv preprint arXiv:2505.16763 , 2025

2025
[28]

Evolvedirector: Approaching advanced text-to-image generation with large vision- language models,

R. Zhao, H. Yuan, Y . Wei, S. Zhang, Y . Gu, L. Ran, X. Wang, J. Z. Wu, D. J. Zhang, Y . Zhang, and M. Z. Shou, “Evolvedirector: Approaching advanced text-to-image generation with large vision- language models,” in Advances in Neural Information Processing Systems 38: Annual C...

2024
[162]

12 888–12 900

PMLR, 2022, pp. 12 888–12 900. [Online]. Available: https: //proceedings.mlr.press/v162/li22n.html

2022
[2021]

12 873–12 883

Computer Vision Foundation / IEEE, 2021, pp. 12 873–12 883. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021/ html/Esser\ Taming\ Transformers\ for\ High-Resolution\ Image\ Synthesis\ CVPR\ 2021\ paper.html

2021
[2022]

Available: http://papers.nips.cc/paper\ files/paper/2022/ hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html 11

[Online]. Available: http://papers.nips.cc/paper\ files/paper/2022/ hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html 11

2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Draw all your imagine: A holistic benchmark and agent framework for complex instruction-based image generation,

Y . Zhou, J. Yuan, and Q. Wang, “Draw all your imagine: A holistic benchmark and agent framework for complex instruction-based image generation,” arXiv preprint arXiv:2505.24787 , 2025

arXiv 2025

[2] [29]

Improving compositional text-to-image generation with large vision-language models,

S. Wen, G. Fang, R. Zhang, P. Gao, H. Dong, and D. N. Metaxas, “Improving compositional text-to-image generation with large vision-language models,” CoRR, vol. abs/2310.06311, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.06311 12

arXiv doi:10.48550/arxiv.2310.06311 2023

[3] [11]

Microsoft COCO: common objects in context,

T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V , ser. Lecture Notes in Computer Science, D. J. Fleet, T. Pajdla, B. Schiele, and T. Tuytela...

doi:10.1007/978-3-319-10602-1 2014

[4] [2]

Score: Story coherence and retrieval enhancement for ai narratives,

Q. Yi, Y . He, J. Wang, X. Song, S. Qian, X. Yuan, M. Zhang, L. Sun, K. Li, K. Lu et al., “Score: Story coherence and retrieval enhancement for ai narratives,” arXiv preprint arXiv:2503.23512 , 2025

arXiv 2025

[5] [3]

Enhancing code llms with reinforcement learning in code generation,

J. Wang, Z. Zhang, Y . He, Y . Song, T. Shi, Y . Li, H. Xu, K. Wu, G. Qian, Q. Chen et al. , “Enhancing code llms with reinforcement learning in code generation,” arXiv preprint arXiv:2412.20367 , 2024

arXiv 2024

[6] [4]

Diffusion models beat gans on image synthesis,

P. Dhariwal and A. Q. Nichol, “Diffusion models beat gans on image synthesis,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , M. Ranzato, A. Beygelzimer, Y . N. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 8780–8794. [Online]. A...

work page 2021

[7] [5]

Triple sequence generative adversarial nets for unsupervised image captioning,

Y . Zhou, W. Tao, and W. Zhang, “Triple sequence generative adversarial nets for unsupervised image captioning,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7598–7602

work page 2021

[8] [6]

Visual in-context learning for large vision-language models,

Y . Zhou, X. Li, Q. Wang, and J. Shen, “Visual in-context learning for large vision-language models,” in Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 . Association for Computational Linguis- tics, 2024, pp. 15 890–15 902

2024