{"id":"18959356-ffe1-4a0e-a423-d413c91d23ec","arxiv_id":"2507.21391","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLaVA-Reward extracts reward scores from the hidden states of a multimodal LLM with a skip-connection cross-attention head, and reports state-of-the-art text-to-image evaluation across alignment, fidelity, and safety.","lead":"Researchers built LLaVA-Reward, a scoring model that judges images made from text prompts by reading the internal signals of a vision-language model instead of asking it to write a review. It is designed to rate images quickly across several criteria, and the team reports it outperforms earlier scoring models on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inference-time scaling comparison is uncontrolled: per-model steering temperature (λ=60/10/20) with no sensitivity analysis and no seed variance does not establish that LLaVA-Reward surpasses ImageReward/CLIPScore.","rationale":"The reader's weakest assumption already identified the same load-bearing concern: the inference-time scaling comparison uses different steering temperatures for LLaVA-Reward, ImageReward, and CLIPScore without sensitivity analysis, and the evaluations lack variance estimates. My stress-test confirms that this is the most fragile point of the central claim because the inference-time scaling result is explicitly advertised as a key contribution, and the reported margins on external metrics are small enough to be explained by hyperparameter choice or Monte Carlo seed noise. The benchmark evaluation portions of the paper are better supported by ablations and multiple datasets, though they also lack error bars; however, the stochastic and uncontrolled nature of the FK-steering experiment makes the downstream superiority claim the least secure. A sensitivity sweep with per-model λ calibration and repeated seeds would settle whether the advantage is real. Since the reader already assigned CONDITIONAL based on these issues, I recommend keeping that verdict unchanged rather than moving to accept or reject.","tokens_in":18838,"tokens_out":4980,"duration_ms":56814,"concrete_test":"Run FK steering on a held-out 100-prompt subset of GenEval and DrawBench with each reward model under a sweep of λ (e.g., 1, 5, 10, 20, 40, 60), choosing the best λ on a separate validation split, and repeat each configuration with 10 different SMC seeds, reporting mean and std of external metrics (GenEval score, HPSv2, VQAScore, LLM Grader). If ImageReward or CLIPScore at their best λ matches or beats LLaVA-Reward at its best λ, or if the LLaVA-Reward advantage is within seed noise, the inference-time scaling superiority claim should be removed or downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest downstream claim is that LLaVA-Reward improves diffusion inference-time scaling over ImageReward and CLIPScore (Sec. 4.5, Tab. 4, Tab. 9). This claim is not supported by the reported experiment. In Sec. 4.5, the authors state that the steering temperature λ is set to 60 for LLaVA-Reward, 10 for ImageReward, and 20 for CLIPScore, with no sensitivity analysis or calibration procedure. FK steering is a sequential Monte Carlo method; λ controls how aggressively the reward model's scores are turned into resampling weights. Since the three reward models have different output scales and different score distributions, fixing different λ values conflates model quality with hyperparameter choice. Moreover, the comparison appears to be based on a single stochastic run with particle number 4; no seed repetitions, confidence intervals, or significance tests are reported. The margins in Tab. 9 are small on several external metrics (e.g., SDXL GenEval HPSv2: 0.323 vs 0.316 vs 0.282; VQAScore: 0.721 vs 0.718 vs 0.694), so they are within plausible Monte Carlo noise. Because the inference-time scaling result is a central part of the abstract's claim that LLaVA-Reward outperforms existing methods in inference-time scaling, it needs controlled comparison before it can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLaVA-Reward, a multimodal reward model for text-to-image generation that extracts rewards from the hidden states of a pretrained MLLM (Phi-3.5-vision) instead of generating evaluation text or token probabilities. A Skip-connection Cross-Attention (SkipCA) reward head is proposed to combine early-layer visual features with later-layer text hidden states, and LoRA adapters are trained per evaluation perspective (alignment, fidelity/artifact, safety) using Bradley-Terry, General Preference Model (GPM), or cross-entropy objectives. The model is evaluated on MJ-Bench, TIFA 160, UnsafeDiff, and SMID, and is also used as a steering reward in Feynman-Kac (FK) inference-time scaling for Stable Diffusion v2.1 and SDXL. The authors report state-of-the-art results against CLIP-based, MLLM-based, and closed-source judges, and claim improved generation quality after FK steering.","tokens_in":19198,"tokens_out":6557,"duration_ms":76540,"significance":"If the empirical claims hold, the paper makes a useful contribution by showing that hidden-state reward extraction with a lightweight cross-attention head can be an efficient and flexible alternative to VQA-style and token-probability MLLM judges, with multi-perspective support through LoRA adapters. The paper also provides a fairly extensive ablation study (SkipCA vs MLP head, GPM vs BT, negative sample augmentation, hidden-state layer choice, base model size, LoRA vs full fine-tuning) and an inference-time comparison, which helps the community understand design trade-offs. However, the central downstream claim about inference-time scaling currently rests on an uncontrolled comparison, and the safety evaluation with ties is near chance, so the practical significance is not yet firmly established.","major_comments":[{"comment":"The inference-time scaling comparison is not controlled. The paper sets different steering temperatures for each reward model (lambda=60 for LLaVA-Reward, 10 for ImageReward, 20 for CLIPScore) without any sensitivity analysis or calibration procedure, even though these models have different output scales and score distributions. Since lambda controls how aggressively the reward is turned into resampling weights in FK steering, this choice alone could explain the observed differences. Moreover, the results are reported from a single run with no seed variance, confidence intervals, or significance tests; several margins in Tab. 9 are small (e.g., GenEval SDXL HPSv2 0.323 vs 0.316; VQAScore 0.721 vs 0.718) and could easily be within Monte Carlo noise. The abstract's claim that LLaVA-Reward 'outperforms ... in inference-time scaling' therefore needs a controlled comparison before it can be accepted.","section":"Sec. 4.5 / Tab. 4 / Tab. 9"},{"comment":"The proposed SkipCA module does not improve text-image alignment on MJ-Bench. LLaVA-Reward-Phi without SkipCA achieves 68.2% accuracy with ties and 68.6% without ties, whereas with SkipCA it achieves 66.1% and 66.2%, respectively. This is a direct negative result for the paper's central architectural motivation, which states that SkipCA 'enhances text-image correlation reasoning.' While SkipCA helps on the safety and artifact perspectives, the paper does not acknowledge or discuss that it hurts alignment; this omission is significant because alignment is one of the four claimed evaluation perspectives and the main cross-modal capability the module is designed to improve.","section":"Tab. 1 / Sec. 3"},{"comment":"The safety evaluation with ties is near chance. LLaVA-Reward-Phi reports 55.2% accuracy with ties on the MJ-Bench safety set. Combined with the without-ties accuracy of 92.1%, this implies that the model ties roughly 88% of the pairs, so it is barely better than random when ties are included. A safety judge that ties almost all pairs has limited practical value for ranking, and the paper does not report tie rates or discuss this behavior. The claim of 'state-of-the-art' safety evaluation is therefore overstated and should be qualified with tied-pair statistics.","section":"Tab. 1 (Safety rows)"},{"comment":"The cleanliness of the ImageReward-to-MJ-Bench training/test split rests entirely on the Levenshtein-distance filter, but no threshold or post-filter verification is reported. Since ImageReward is both the training source for LLaVA-Reward and a baseline in the same table, any residual near-duplicate prompts could inflate the reported advantage over ImageReward. The paper should specify the distance threshold, the number of removed prompts, and ideally a manual or embedding-based check for overlap, to rule out data leakage.","section":"Sec. 4.3 / Tab. 1"}],"minor_comments":[{"comment":"The table header 'LMM-based' should be 'MLLM-based' for consistency, and 'links' in Sec. 4.2 should be 'ties' (the MJ-Bench metric uses ties, not links).","section":"Tab. 1 / Sec. 4.2"},{"comment":"Table 3 is titled 'Evaluation of recent LMM-based safety judges on TIFA 160' but the columns report UnsafeDiff and SMID; the caption does not match the content.","section":"Tab. 3 caption"},{"comment":"The GPM reward difference in Eq. (2) is typeset as 'R≻rθp(ic,t), rθp(ir,t)' without explicit angle brackets around the inner product; this should be written as ⟨R≻ rθp(ic,t), rθp(ir,t)⟩ to be mathematically unambiguous.","section":"Eq. (2)"},{"comment":"The phrase 'Cross Attention Modal' should be 'Cross-Attention Module'.","section":"Sec. 3, 'Skip-connection Cross Attention Modal'"},{"comment":"The word 'setted' should be 'set', and the sentence beginning 'We can observe that each diffusion inference-time scaling variant tends to have a better result when evaluated on itself' should be expanded to clarify which of the reported metrics are self-referential and which are external.","section":"Sec. 4.5, last paragraph"}],"recommendation":"major_revision","confidential_remarks":"The skeptic's concern about the uncontrolled inference-time scaling comparison is well-founded and is the main reason for my major_revision recommendation. The negative alignment ablation for SkipCA and the high tie rate in safety evaluation also need to be addressed before the paper can support its headline claims. If the authors add a lambda sensitivity analysis, repeated seeds with error bars, and a proper discussion of the safety tie behavior, the contribution could become a solid paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this paper delivers a real, useful design. Taking the final hidden state of a decoder-only MLLM and injecting early visual features through a skip-connection cross-attention head is a clean fix for the diluted-visual-tokens problem, and the ablations justify it. The multi-perspective LoRA setup is practical, and the 0.35s inference cost is genuinely attractive against VQA-style judges. On MJ-Bench, TIFA-160, UnsafeDiff, and SMID, the margins over CLIP-scale and VQA baselines are large enough that I do not worry much about the missing error bars for the core evaluation claims.\n\nThe soft spot is the inference-time scaling story. The FK steering comparison uses lambda=60 for LLaVA-Reward, 10 for ImageReward, and 20 for CLIPScore, with no sensitivity analysis, one seed, and particle count 4. That is an uncontrolled comparison. The margins on external metrics are small in places (VQAScore 0.721 vs 0.718 on SDXL GenEval), and the qualitative examples are cherry-picked. The abstract's claim about 'inference-time scaling' should be toned down until this is redone with per-model lambda sweeps and multiple seeds. It is fixable, but as written it does not support the SOTA claim.\n\nAlso minor: safety accuracy with ties on MJ-Bench is 55.2%, which is near chance; the without-tie number is strong, but the tie handling needs discussion. Using LLaVA-Reward's own score as one of the metrics in Table 9 is self-referential, though the external metrics mostly support the finding anyway.\n\nThe limitations section is honest about training-data breadth and reward hacking. There is no fake precision here.\n\nBottom line: a solid systems paper with a genuinely new reward-head design and good ablations. The evaluation-head results are credible. The inference-time scaling section needs explicit sensitivity analysis and variance reporting. I would send it to a serious referee: it deserves referee time, and the flaws are correctable, not load-bearing. The reader's stress-test is on target.","headline":"A genuinely new reward-head design with credible evaluation results, but the inference-time scaling comparison is under-controlled and needs a lambda sweep before the SOTA claim stands.","tokens_in":19740,"tokens_out":2863,"would_cite":true,"duration_ms":31944,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hidden-state reward head with skip-connection cross-attention can score and steer text-to-image generation more like humans than CLIP- or response-based judges.","keywords":["text-to-image evaluation","reward models","multimodal large language models","hidden-state scoring","skip-connection cross-attention","preference learning","inference-time scaling","human-aligned evaluation"],"falsifier":"A temperature-sweep experiment holding the FK-steering strength equal across reward models (for instance, all at lambda=10, lambda=20, and lambda=60) on GenEval with SDXL would settle the inference-time-scaling claim; if ImageReward or CLIPScore matches or beats LLaVA-Reward at equal lambda, the claimed steering superiority collapses. On the evaluation side, an audit of the Levenshtein filtering step that checks whether any remaining MJ-Bench prompts or near-duplicates still appear in the ImageReward training set could reveal whether the alignment gains are inflated by leakage.","tokens_in":18688,"feed_emoji":"🖼️","tokens_out":15225,"duration_ms":154177,"temperature":0.7,"pith_summary":"LLaVA-Reward is a proposed judge for text-to-image generation that scores an image-prompt pair from the internal hidden state of a multimodal large language model, rather than from the text the model would write about it. The paper argues that this makes automatic evaluation faster and more flexible than current MLLM-based judges, which need instruction-following data and either generate verbose verdicts or read the probability of a rating token. It handles paired preference data through a Bradley-Terry or General Preference Model loss and unpaired binary safety data through cross-entropy. A Skip-connection Cross-Attention (SkipCA) reward head reconnects early visual features with the final text hidden state, and per-perspective LoRA adapters let one base MLLM cover alignment, fidelity, safety, and overall ranking. The paper reports that LLaVA-Reward beats CLIP-based, VQA-based, and token-likelihood judges on MJ-Bench, TIFA 160, UnsafeDiff, and SMID, and that it improves diffusion inference-time scaling on GenEval and DrawBench.","feed_headline":"Hidden-state reward model beats CLIP and VQA judges","feed_subtitle":"A cross-attention head over early visual features yields human-aligned scores and steers diffusion sampling to better images.","key_machinery":"The load-bearing component is the Skip-connection Cross-Attention (SkipCA) reward head. In a decoder-only MLLM the causal mask prevents visual tokens from being influenced by later text tokens, and visual-specific information is diluted in deep transformer layers; SkipCA counters this by taking projected visual tokens $e_v$ from just after the vision projector as keys and values, the final-layer hidden state $e_h$ as the query, running cross-attention, and mapping the result to a scalar or, under the General Preference Model, a vector reward. This gives the late text representation a second, direct look at early visual features; the paper's ablation shows the module matters most for safety, where removing it drops MJ-Bench safety accuracy without ties from 55.2 to 39.7. LoRA adapters, one per evaluation perspective, allow the same pretrained MLLM to switch tasks while only adding about 8 percent trainable parameters.","core_discovery":"The central claim is that a text-to-image reward model can be made accurate, cheap, and customizable by discarding the instruction-following and text-generation machinery of MLLMs and training a reward head directly on the model's hidden embeddings. Concretely, the paper shows that the EOS-token hidden state of Phi-3.5-vision, combined with visual tokens taken right after the vision projector via SkipCA, produces scores that agree with human preference better than the compared methods. On MJ-Bench, the Phi variant reaches 66.1% alignment accuracy with ties (66.2% without), 55.2% safety accuracy (92.1% without ties), and 91.1% fidelity accuracy (91.2% without ties), with the Qwen2.5-VL variant pushing fidelity to 94.3%. On TIFA 160 it reports a Pearson correlation of 71.1, on UnsafeDiff an F1 of 87.2, and on SMID an F1 of 75.7 for the 4.2B variant (78.1 for the 8.2B variant). In diffusion inference-time scaling with FK steering, it raises the mean GenEval score from 0.521 to 0.622 on Stable Diffusion v2.1 and from 0.563 to 0.645 on SDXL, above the ImageReward and CLIPScore steering baselines.","pith_inferences":["A testable extension the paper leaves implicit is a systematic sweep over base models and vision encoders; the two reported backbones already suggest the reward head transfers, but a controlled comparison would separate the contribution of SkipCA from the base model's visual capacity.","The paper connects the General Preference Model to direct preference optimization but only evaluates it for pairwise scoring, so using a GPM-trained LLaVA-Reward as the reward signal in a DPO-style training loop for a text-to-image model is a natural next experiment.","Because the safety variant is trained on binary UnsafeBench labels and still gains F1 on UnsafeDiff and SMID, the hidden-state approach looks promising as a cheap per-perspective safety screen for generated images, though that transfer would need validation on broader safety taxonomies."],"forward_implications":["The claimed result means automatic text-to-image judging can drop the instruction-following generation step entirely: a reward head on the EOS hidden state is more human-aligned and much faster than VQA-style judges, at 0.35 seconds per evaluation versus up to 7.01 seconds.","Since each perspective is a LoRA adapter on one base model, the same multimodal LLM can be re-purposed for alignment, fidelity, or safety by swapping adapters, making multi-perspective evaluation cheap to maintain.","The FK-steering gains on GenEval and DrawBench show the reward model can be used at sampling time to improve prompt adherence in Stable Diffusion v2.1 and SDXL without extra denoising steps.","Hard-negative preference pairs are a large part of the alignment gain: without the extra 70k pairs, MJ-Bench alignment accuracy drops by about 1.6 points, so data construction contributes as much as the architecture."],"supporting_citations":[{"why":"Supplies the ImageReward human-preference data converted into pairwise training pairs for alignment and fidelity, and is the closest CLIP-based baseline in the comparisons.","marker":"[62]"},{"why":"Defines MJ-Bench, the main multi-perspective text-to-image preference benchmark whose accuracy numbers anchor the evaluation claims.","marker":"[10]"},{"why":"Gives CLIPScore, a reference-free text-image scoring method used as a baseline in both evaluation and inference-time scaling.","marker":"[21]"},{"why":"VQAScore, the token-probability judge compared against on MJ-Bench, TIFA 160, and diffusion steering.","marker":"[32]"},{"why":"EvalAlign, the supervised-fine-tuned VQA judge that represents the instruction-following approach LLaVA-Reward is designed to replace.","marker":"[56]"},{"why":"Safety judge baseline (ImageGuard/T2ISafety) and the source of the cross-modality attention idea that SkipCA adapts.","marker":"[30]"},{"why":"LlavaGuard, the VQA-based safety judge compared on UnsafeDiff and SMID.","marker":"[20]"},{"why":"FK steering, the diffusion inference-time scaling method used to test LLaVA-Reward as a reward signal during sampling, including the different lambda settings.","marker":"[52]"},{"why":"The normalized Levenshtein distance metric used to filter MJ-Bench-like prompts out of the ImageReward training set to guard against train/test overlap.","marker":"[63]"},{"why":"The General Preference Model objective used for vector-valued reward outputs in the paired-preference training variant.","marker":"[65]"}],"fun_headline_variants":["Hidden-state reward model beats CLIP and VQA judges","SkipCA heads unlock efficient T2I reward models","Customizable reward model from MLLM hidden states","Faster, better reward scoring for text-to-image generation","LLaVA-Reward: hidden states replace text generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the assumption that the benchmark comparisons are configured fairly, especially that the much larger steering strength used for LLaVA-Reward in inference-time scaling (lambda=60 versus lambda=10 for ImageReward and lambda=20 for CLIPScore) does not by itself create the reported advantage.","fun_headline_variants_meta":{"raw":{"variants":["Hidden-state reward model beats CLIP and VQA judges","SkipCA heads unlock efficient T2I reward models","Customizable reward model from MLLM hidden states","Faster, better reward scoring for text-to-image generation","LLaVA-Reward: hidden states replace text generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1662,"prompt_tokens":1063,"completion_tokens":599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":679,"tokens_out":599,"duration_ms":7509,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:47:37.481414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A temperature-sweep experiment holding the FK-steering strength equal across reward models (for instance, all at lambda=10, lambda=20, and lambda=60) on GenEval with SDXL would settle the inference-time-scaling claim; if ImageReward or CLIPScore matches or beats LLaVA-Reward at equal lambda, the claimed steering superiority collapses. On the evaluation side, an audit of the Levenshtein filtering step that checks whether any remaining MJ-Bench prompts or near-duplicates still appear in the ImageReward training set could reveal whether the alignment gains are inflated by leakage.","supporting_citations":[{"cited_title":"Imagere- ward: Learning and evaluating human preferences for text- to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the ImageReward human-preference data converted into pairwise training pairs for alignment and fidelity, and is the closest CLIP-based baseline in the comparisons."},{"cited_title":"Evaluating text-to-visual generation with image-to-text gen- eration","cited_arxiv_id":null,"evidence_quote":"VQAScore, the token-probability judge compared against on MJ-Bench, TIFA 160, and diffusion steering."},{"cited_title":"A normalized levenshtein distance metric","cited_arxiv_id":null,"evidence_quote":"The normalized Levenshtein distance metric used to filter MJ-Bench-like prompts out of the ImageReward training set to guard against train/test overlap."}],"review_version":1}