PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

Hao Jiang; Jinlong Liu; Mushui Liu; Peng Zhang; Pipei Huang; Wanggui He

arxiv: 2604.12652 · v2 · submitted 2026-04-14 · 💻 cs.CV · cs.AI

PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

Jinlong Liu , Wanggui He , Peng Zhang , Mushui Liu , Hao Jiang , Pipei Huang This is my paper

Pith reviewed 2026-05-10 15:42 UTC · model grok-4.3

classification 💻 cs.CV cs.AI

keywords text-to-image generationreinforcement learningvision-language modelsprompt followingreward modelingannotation-free methodsDenseAlignBench

0 comments

The pith

PromptEcho constructs reward signals for text-to-image reinforcement learning by computing token-level cross-entropy loss in a frozen vision-language model using the original prompt as label.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents PromptEcho as a way to obtain high-quality rewards for reinforcement learning that improves how well text-to-image models follow complex prompts. It avoids human annotations and separate reward-model training by running a frozen VLM on the generated image and measuring how surprised the model is by the original prompt tokens. This produces a deterministic signal that scales with stronger VLMs and requires no task-specific fine-tuning. The authors also release DenseAlignBench, a new test set of dense captions, to measure prompt adherence more precisely than existing benchmarks. Experiments on two leading T2I models show large gains on the new benchmark plus consistent lifts on GenEval, DPG-Bench, and TIIFBench.

Core claim

PromptEcho extracts an image-text alignment reward directly from the token-level cross-entropy loss of any frozen VLM by feeding the generated image and using the original prompt as the target label. This re-uses the alignment knowledge already encoded during VLM pretraining, yielding a reward that is annotation-free, computationally light, and automatically improves as better open-source VLMs appear. On Z-Image and QwenImage-2512 the method delivers +26.8pp and +16.2pp net win rates on DenseAlignBench together with steady gains on three other prompt-following suites, while ablation studies confirm it surpasses inference-only scoring with the identical VLM and that larger VLMs produce higher

What carries the argument

PromptEcho reward: token-level cross-entropy loss of a frozen VLM whose input is the generated image and whose label is the original prompt.

If this is right

T2I models can be aligned to dense prompts through RL without collecting human preference data or training a dedicated reward model.
Reward quality increases automatically whenever a stronger open-source VLM is substituted, with no retraining required.
The same VLM used for reward computation outperforms simple inference-time scoring on prompt-following metrics.
Consistent gains appear across multiple independent benchmarks when the method is applied to current state-of-the-art T2I backbones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach lowers the cost of RL alignment for any generative model whose outputs can be scored by a VLM, potentially extending beyond images.
Developers may obtain better prompt adherence simply by swapping in larger VLMs rather than redesigning training pipelines.
Dense caption benchmarks like DenseAlignBench expose limitations that coarser metrics miss, suggesting future evaluation standards will shift toward richer textual descriptions.

Load-bearing premise

The token-level cross-entropy loss of a frozen VLM, with the original prompt as label, supplies a reliable reward that directly improves prompt-following quality when used in T2I reinforcement learning.

What would settle it

Training a T2I model with PromptEcho rewards and observing no improvement, or worse performance, on DenseAlignBench relative to a CLIP-Score baseline would falsify the central claim.

Figures

Figures reproduced from arXiv: 2604.12652 by Hao Jiang, Jinlong Liu, Mushui Liu, Peng Zhang, Pipei Huang, Wanggui He.

**Figure 2.** Figure 2: Qualitative comparison of 6 representative winning cases. Each row shows the full [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

**Figure 3.** Figure 3: Text rendering quality comparison before and after PromptEcho training (Section 4.4). [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

read the original abstract

Reinforcement learning (RL) can improve the prompt following capability of text-to-image (T2I) models, yet obtaining high-quality reward signals remains challenging: CLIP Score is too coarse-grained, while VLM-based reward models (e.g., RewardDance) require costly human-annotated preference data and additional fine-tuning. We propose PromptEcho, a reward construction method that requires \emph{no} annotation and \emph{no} reward model training. Given a generated image and a guiding query, PromptEcho computes the token-level cross-entropy loss of a frozen VLM with the original prompt as the label, directly extracting the image-text alignment knowledge encoded during VLM pretraining. The reward is deterministic, computationally efficient, and improves automatically as stronger open-source VLMs become available. For evaluation, we develop DenseAlignBench, a benchmark of concept-rich dense captions for rigorously testing prompt following capability. Experimental results on two state-of-the-art T2I models (Z-Image and QwenImage-2512) demonstrate that PromptEcho achieves substantial improvements on DenseAlignBench (+26.8pp / +16.2pp net win rate), along with consistent gains on GenEval, DPG-Bench, and TIIFBench without any task-specific training. Ablation studies confirm that PromptEcho comprehensively outperforms inference-based scoring with the same VLM, and that reward quality scales with VLM size. We will open-source the trained models and the DenseAlignBench.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PromptEcho gives a no-annotation reward for T2I RL by feeding generated images into a frozen VLM's token cross-entropy loss, but the evidence that this actually improves prompt following is still preliminary.

read the letter

The main point is that PromptEcho extracts a reward signal directly from a frozen VLM by computing token-level cross-entropy loss on the original prompt for each generated image. This sidesteps both human preference data and any reward-model training, which is the practical advance over CLIP Score or methods like RewardDance. They also release DenseAlignBench, a set of dense captions meant to test fine-grained prompt adherence more stringently than GenEval or DPG-Bench. The reported net win rates (+26.8 pp on their benchmark, plus gains elsewhere) and the scaling with VLM size are concrete enough to notice. The ablations showing it beats inference-time VLM scoring add a bit of internal consistency check. All of that is useful for anyone trying to run RL on T2I models without building new datasets. The soft spot is that the central claim still rests on an unproven correlation: that minimizing this particular loss under RL produces images that humans or independent metrics would judge as better aligned, rather than VLM-specific artifacts. The abstract does not include experimental details on baseline implementations, statistical tests, or checks for reward hacking and distribution shift between VLM training data and synthetic images, so the stress-test concern about reliability under optimization looks reasonable. The paper is aimed at groups already running RL for image generation and looking for lighter-weight rewards. A reader who wants to try the idea quickly would get value from the method and the benchmark. It is coherent enough and addresses a real bottleneck that it deserves a serious referee, even though the current evidence is thin on validation of the reward itself. I would send it to review with a request for more targeted experiments on whether the loss tracks human judgments and resists obvious failure modes.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces PromptEcho, an annotation-free reward construction method for reinforcement learning fine-tuning of text-to-image (T2I) models. It computes a reward as the token-level cross-entropy loss of a frozen vision-language model (VLM) with the original prompt as the label, directly using pre-trained alignment knowledge. The approach is applied to two T2I models and evaluated on a new DenseAlignBench benchmark (reporting +26.8pp / +16.2pp net win rate gains) plus GenEval, DPG-Bench, and TIIFBench, with ablations showing superiority to inference-based VLM scoring and scaling with VLM size. The paper also contributes DenseAlignBench for dense-caption prompt-following evaluation and plans to open-source models and the benchmark.

Significance. If the results hold under rigorous validation, PromptEcho offers a practical, scalable alternative to annotation-heavy or fine-tuned reward models for T2I alignment, with the key strengths of being deterministic, computationally efficient, and automatically improving as open-source VLMs advance. The introduction of DenseAlignBench addresses a gap in evaluating complex prompt adherence. These elements, combined with the parameter-free nature of the reward (no task-specific fitting), could meaningfully influence RL-based generative model training pipelines.

major comments (3)

[Experimental evaluation] The central empirical claims (Abstract) rest on reported gains such as +26.8pp net win rate on DenseAlignBench, yet the manuscript provides no details on the RL experimental setup, including algorithm choice, hyperparameters, number of optimization steps or samples, statistical significance testing, exact baseline reproductions, or controls for confounds. This is load-bearing for assessing whether the gains reflect genuine prompt-following improvements.
[Ablation studies] The claim that token-level CE loss from the frozen VLM constitutes a high-quality reward (Abstract and ablation studies) lacks explicit validation of its correlation with independent human or automated prompt-adherence metrics, analysis of distribution shift between VLM pretraining data and generated images, or examination of reward-hacking failure modes where low loss occurs without faithful adherence.
[DenseAlignBench] DenseAlignBench is positioned as a rigorous new benchmark for concept-rich dense captions, but its construction details, scale, diversity metrics, inter-annotator agreement, and direct comparison to existing suites (e.g., how it avoids the coarseness of GenEval) are insufficiently specified to support the claim of more rigorous testing.

minor comments (2)

[Abstract] Model names 'Z-Image' and 'QwenImage-2512' in the abstract should include citations or full references for reproducibility.
[Method] Notation for the reward computation (token-level cross-entropy) could be formalized with an equation in the method section to improve clarity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. The comments highlight important areas for improving clarity, reproducibility, and validation of our claims. We address each major comment below and will incorporate the requested details and analyses into the revised manuscript.

read point-by-point responses

Referee: [Experimental evaluation] The central empirical claims (Abstract) rest on reported gains such as +26.8pp net win rate on DenseAlignBench, yet the manuscript provides no details on the RL experimental setup, including algorithm choice, hyperparameters, number of optimization steps or samples, statistical significance testing, exact baseline reproductions, or controls for confounds. This is load-bearing for assessing whether the gains reflect genuine prompt-following improvements.

Authors: We agree that comprehensive details on the RL experimental setup are essential for reproducibility and for readers to evaluate the validity of the reported gains. In the revised manuscript, we will add a dedicated subsection describing the RL algorithm (PPO), all hyperparameters (learning rate, batch size, KL coefficient, etc.), the exact number of optimization steps and generated samples per step, statistical significance testing (e.g., paired t-tests or bootstrap confidence intervals on win rates), exact reproduction protocols for baselines, and controls for confounds such as fixed random seeds, consistent evaluation prompts, and image generation settings. These additions will directly address the load-bearing concerns. revision: yes
Referee: [Ablation studies] The claim that token-level CE loss from the frozen VLM constitutes a high-quality reward (Abstract and ablation studies) lacks explicit validation of its correlation with independent human or automated prompt-adherence metrics, analysis of distribution shift between VLM pretraining data and generated images, or examination of reward-hacking failure modes where low loss occurs without faithful adherence.

Authors: We acknowledge that stronger validation of the reward signal is needed. We will expand the ablation section to include: (1) quantitative correlation analysis between PromptEcho rewards and independent metrics (human preference ratings on a subset and automated scores such as VQA-based adherence and CLIPScore); (2) explicit discussion of distribution shift, noting that the frozen VLM was pretrained on large-scale image-text pairs that overlap with common T2I generation distributions, with qualitative examples; and (3) examination of potential reward-hacking cases, including failure examples where low cross-entropy loss does not imply faithful adherence, along with quantitative checks (e.g., reward vs. human alignment plots) and mitigation observations. These additions will provide the requested evidence. revision: yes
Referee: [DenseAlignBench] DenseAlignBench is positioned as a rigorous new benchmark for concept-rich dense captions, but its construction details, scale, diversity metrics, inter-annotator agreement, and direct comparison to existing suites (e.g., how it avoids the coarseness of GenEval) are insufficiently specified to support the claim of more rigorous testing.

Authors: We agree that more detailed documentation of DenseAlignBench is required to substantiate its advantages. In the revised manuscript, we will add an expanded section (or appendix) covering: construction methodology (prompt sourcing, dense caption generation process, and filtering criteria), scale (total number of prompts, images per prompt, and category distribution), diversity metrics (concept coverage statistics, average caption length, and lexical diversity), inter-annotator agreement (e.g., Cohen's kappa or percentage agreement on a validation subset), and a direct side-by-side comparison with GenEval and DPG-Bench that highlights how dense captions enable finer-grained evaluation of multi-concept adherence, thereby addressing the coarseness limitation. This will strengthen the benchmark's positioning. revision: yes

Circularity Check

0 steps flagged

No circularity: reward signal extracted from external frozen VLM without self-referential fitting or definition

full rationale

The paper defines PromptEcho explicitly as the token-level cross-entropy loss of a frozen pre-trained VLM (original prompt as label) applied to generated images. This construction uses no parameters fitted within the paper, no self-citation for the core reward mechanism, and no reduction of the claimed improvements to quantities defined by the method itself. Gains are measured on separate external benchmarks (DenseAlignBench, GenEval, DPG-Bench, TIIFBench) whose labels and scoring are independent of the VLM loss computation. No equations or steps in the provided text equate the reward or the RL outcome to the paper's own inputs by construction. The central assumption (that the loss proxies prompt adherence) is an empirical claim open to external verification rather than a definitional tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The method rests on the domain assumption that pre-trained VLMs already encode sufficient image-text alignment knowledge extractable via cross-entropy loss; no free parameters or new entities are introduced.

axioms (1)

domain assumption Pre-trained vision-language models encode useful image-text alignment knowledge that can be read out via token-level cross-entropy loss without further training.
Invoked to justify using the loss directly as reward.

pith-pipeline@v0.9.0 · 5583 in / 1255 out tokens · 26816 ms · 2026-05-10T15:42:37.865833+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

[1]

Carefully read and understand all requirements in the prompt: - Main subjects and objects - Actions and poses - Visual attributes (color, size, material, texture) - Composition and layout - Style and atmosphere - Any text or written elements - Spatial relationships (foreground, background, position) - Quantities and counting

work page
[2]

Examine Image A: - Which prompt requirements are accurately depicted? - Which prompt requirements are missing or incorrect? - Are there elements not mentioned in the prompt?

work page
[3]

Examine Image B: - Which prompt requirements are accurately depicted? - Which prompt requirements are missing or incorrect? - Are there elements not mentioned in the prompt?

work page
[4]

tie" **Preference Options:** -

Compare: - Which image more accurately captures the prompt requirements? - If both are equal in accuracy, select "tie" **Preference Options:** - "image_a": Image A better follows the prompt than Image B - "image_b": Image B better follows the prompt than Image A - "tie": Both images follow the prompt to a similar degree **Important Notes:** - Make your de...

work page
[5]

**main_title**: If the title has multiple lines (e.g., top line/ bottom line), each line is a separate element in the list

work page
[6]

**subtitle**: Text inferred to be subtitles, such as product titles or secondary headings

work page
[7]

**selling_points**: Product selling points, recommendations, or introductory phrases

work page
[8]

reasoning

**other_text**: Other text not belonging to the above categories. **Notes:** - Each piece of text should be extracted completely, preserving the original content - If a category has no corresponding content, return an empty list [] - Carefully read the context to determine the semantic role of each piece of text **Output Format (must strictly follow JSON ...

work page
[9]

If the title has multiple lines, each line is a separate element in the list

**main_title**: The largest and most prominent title text in the image. If the title has multiple lines, each line is a separate element in the list

work page
[10]

**subtitle**: Subtitles or supplementary text, such as product titles or secondary headings

work page
[11]

**selling_points**: Product selling points, recommendations, introductory phrases, efficacy features, etc

work page
[12]

main_title

**other_text**: Other text not belonging to the above categories. **Notes:** 13 - Only extract designed marketing text in the image; do not extract text printed on product packaging or physical objects - Each piece of text should be extracted completely, preserving the original content - Each field is a list of strings, arranged in visual reading order - ...

work page
[13]

Pay attention to distinguishing Chinese/English, uppercase/lowercase, numbers, etc

**Extract required text**: Carefully read the prompt and extract all text content required to appear in the image. Pay attention to distinguishing Chinese/English, uppercase/lowercase, numbers, etc. If the prompt does not explicitly require any text to appear in the image, expected_text is an empty string, and directly give 1 point

work page
[14]

Ignore text on product packaging; focus on the text required by the prompt

**Recognize image text**: Carefully examine the image and recognize all text appearing in it character by character. Ignore text on product packaging; focus on the text required by the prompt

work page
[15]

**Character-by-character comparison**: Perform strict character-by-character comparison between the prompt-required text and the actual text in the image: - Every required character must be completely correct; garbled text is unacceptable - No extra characters, missing characters, wrong characters, or garbled text allowed - No duplicate text (unless the p...

work page
[16]

expected_text

**Scoring**: - 1 point: All required text is rendered completely correctly in the image with no character-level errors - 0 points: Any text error exists **Output Format (strict JSON):** { "expected_text": "<all required text extracted from the prompt>", "found_text": "<all text actually recognized from the image>", "reasoning": "<detailed character-by-cha...

work page

[1] [1]

Carefully read and understand all requirements in the prompt: - Main subjects and objects - Actions and poses - Visual attributes (color, size, material, texture) - Composition and layout - Style and atmosphere - Any text or written elements - Spatial relationships (foreground, background, position) - Quantities and counting

work page

[2] [2]

Examine Image A: - Which prompt requirements are accurately depicted? - Which prompt requirements are missing or incorrect? - Are there elements not mentioned in the prompt?

work page

[3] [3]

Examine Image B: - Which prompt requirements are accurately depicted? - Which prompt requirements are missing or incorrect? - Are there elements not mentioned in the prompt?

work page

[4] [4]

tie" **Preference Options:** -

Compare: - Which image more accurately captures the prompt requirements? - If both are equal in accuracy, select "tie" **Preference Options:** - "image_a": Image A better follows the prompt than Image B - "image_b": Image B better follows the prompt than Image A - "tie": Both images follow the prompt to a similar degree **Important Notes:** - Make your de...

work page

[5] [5]

**main_title**: If the title has multiple lines (e.g., top line/ bottom line), each line is a separate element in the list

work page

[6] [6]

**subtitle**: Text inferred to be subtitles, such as product titles or secondary headings

work page

[7] [7]

**selling_points**: Product selling points, recommendations, or introductory phrases

work page

[8] [8]

reasoning

**other_text**: Other text not belonging to the above categories. **Notes:** - Each piece of text should be extracted completely, preserving the original content - If a category has no corresponding content, return an empty list [] - Carefully read the context to determine the semantic role of each piece of text **Output Format (must strictly follow JSON ...

work page

[9] [9]

If the title has multiple lines, each line is a separate element in the list

**main_title**: The largest and most prominent title text in the image. If the title has multiple lines, each line is a separate element in the list

work page

[10] [10]

**subtitle**: Subtitles or supplementary text, such as product titles or secondary headings

work page

[11] [11]

**selling_points**: Product selling points, recommendations, introductory phrases, efficacy features, etc

work page

[12] [12]

main_title

**other_text**: Other text not belonging to the above categories. **Notes:** 13 - Only extract designed marketing text in the image; do not extract text printed on product packaging or physical objects - Each piece of text should be extracted completely, preserving the original content - Each field is a list of strings, arranged in visual reading order - ...

work page

[13] [13]

Pay attention to distinguishing Chinese/English, uppercase/lowercase, numbers, etc

**Extract required text**: Carefully read the prompt and extract all text content required to appear in the image. Pay attention to distinguishing Chinese/English, uppercase/lowercase, numbers, etc. If the prompt does not explicitly require any text to appear in the image, expected_text is an empty string, and directly give 1 point

work page

[14] [14]

Ignore text on product packaging; focus on the text required by the prompt

**Recognize image text**: Carefully examine the image and recognize all text appearing in it character by character. Ignore text on product packaging; focus on the text required by the prompt

work page

[15] [15]

**Character-by-character comparison**: Perform strict character-by-character comparison between the prompt-required text and the actual text in the image: - Every required character must be completely correct; garbled text is unacceptable - No extra characters, missing characters, wrong characters, or garbled text allowed - No duplicate text (unless the p...

work page

[16] [16]

expected_text

**Scoring**: - 1 point: All required text is rendered completely correctly in the image with no character-level errors - 0 points: Any text error exists **Output Format (strict JSON):** { "expected_text": "<all required text extracted from the prompt>", "found_text": "<all text actually recognized from the image>", "reasoning": "<detailed character-by-cha...

work page