{"id":"13c90785-b73e-4c63-84b7-7aeaf6fbef01","arxiv_id":"2511.19995","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CREward, trained only on Gemma-3-generated preference labels, predicts geometry/material/texture creativity rankings that correlate moderately with human designer judgments on a five-object benchmark.","lead":"The authors introduce CREward, a reward model that scores image creativity separately for geometry, material, and texture, trained on pairwise labels generated by a vision-language model instead of human raters. They report that it correlates with design-expert judgments on a small benchmark and use it to assess, filter, explain, and steer text-to-image generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human-benchmark images are not shown to be excluded from CREward's training pairs; Table 1 correlations may reflect image-specific memorization rather than generalizable creativity assessment.","rationale":"The reader's weakest assumption exactly identifies the most load-bearing issue: the human benchmark may not be disjoint from CREward's training data. I agree with this assessment and with the CONDITIONAL verdict. The paper's empirical case rests entirely on Table 1, and that table is only meaningful if CREward's predictions on CREBench-Human reflect generalization to unseen images rather than familiarity with images seen during training.\n\nI considered whether the more fundamental concern is that CREward, trained on Gemma-3 labels, cannot in principle exceed Gemma-3's own human correlation. That concern is not decisive: a student model trained on noisy teacher labels can outperform the teacher on a small evaluation set through regularization or by learning a smoother function, especially when the teacher is evaluated on only 100 pairs per object. So I do not treat that as the primary attack.\n\nI also considered the small sample size: only 5 objects, and rank correlations over 25 images per object have large standard errors. For example, texture rank correlation is 0.76 ± 0.12 for CREward vs 0.74 ± 0.06 for Gemini-2.5; the 'surpassing Gemini' claim is not statistically meaningful. However, this weakness would not, by itself, invalidate the central claim if the holdout were clean; it would only weaken the confidence. The holdout issue is more fundamental because it threatens the validity, not just the precision, of the central result.\n\nThe paper does include a limitations section acknowledging value/novelty trade-offs and slider entanglement, which is good-faith and counts in its favor. But the absence of any statement about excluding human-benchmark images from training is a concrete, fixable omission. The proposed test—releasing image IDs or hashes and recomputing Table 1 on a verified disjoint set—would settle the concern. Until then, the CONDITIONAL verdict is appropriate.","tokens_in":17744,"tokens_out":4120,"duration_ms":46421,"concrete_test":"Request the authors to release or report the exact image IDs / file hashes for (a) the 125 CREBench-Human images (25 per object × 5 objects) and (b) all images used to construct the 5,000 training pairs. Compute the intersection. If the intersection is non-empty, retrain CREward with those images excluded and recompute Table 1. If the intersection is empty, also verify that no near-duplicate images (e.g., same prompt/model/different seed) are present; if the rank correlations on a fresh held-out set from the same prompts are substantially lower than Table 1, the reported human alignment is not evidence of generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CREward, trained only on Gemma-3 labels, achieves human-level or better type-specific creativity rankings (Table 1: 0.59–0.76 rank correlation). This requires that the human benchmark (CREBench-Human) measures generalization, not memorization. The paper never establishes this.\n\nCREBench-Human (Sec. 3.1) is built from 5,000 generated images per object, from which 20 creative + 5 normal images are manually selected per object, for the same five objects used in training. CREward's training pairs (Sec. 3.3) are sampled from 60 prompts per object (8 object-agnostic + 12 object-specific per type) and 10 images per prompt, generated with the same or overlapping T2I models. If the 25 benchmark images per object are among the candidate images for those training pairs, then the 5,000 training pairs may contain the exact benchmark images. A frozen backbone with a trainable MLP head can memorize preferences for specific images over 20 epochs, especially when each image appears in multiple pairs, inflating rank correlations on the benchmark.\n\nThe paper's only stated split, 'fixed train–val–test split' (Sec. 3.4), refers to the LVLM-labeled pair set, not to the human benchmark. No sentence in Sec. 3.1, 3.3, or 3.4 says the human-benchmark images were held out from training. Because the benchmark was constructed first and the training pairs later, the default assumption in a careful review must be that overlap is possible, and the burden is on the authors to demonstrate non-overlap.\n\nThis is not an abstract concern: if even a subset of the 125 human-benchmark images appears in training, the headline comparison against Gemini-2.5 and the 'second-highest rank correlation' claim lose their evidentiary force. The concern is directly load-bearing and can be settled by a concrete data-release check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CREward, a type-specific creativity reward model that scores generated images along geometry, material, texture, and overall creativity axes. The authors collect a small human pairwise-preference benchmark on five object classes, find that Gemini-2.5 and Gemma-3 correlate with the aggregated human ranks, and then use Gemma-3 labels on 5,000 synthetic image pairs to train a frozen SigLIP backbone with an MLP reward head. They report rank correlations against the human benchmark (Table 1) and use the resulting model for model comparison, sample filtering, Grad-CAM explanations, and LoRA-based creativity sliders. The paper argues that LVLM-derived labels are sufficient to build an open-weight, human-aligned creativity metric.","tokens_in":18129,"tokens_out":6648,"duration_ms":71759,"significance":"If the reported correlations are not an artifact of benchmark overlap or evaluation confounds, CREward would be a practical and interpretable alternative to costly human creativity annotation, and the open-weight model would let others benchmark creative generation along disentangled axes. The paper's strengths are its explicit human benchmark, the systematic comparison of five vision backbones, the pairwise preference formulation with tie handling, and the candid discussion of limitations (novelty-vs-value bias, type entanglement). The main evaluation is, however, built on a very small human sample and on an unstated train/benchmark split, so the significance is conditional on fixing these issues.","major_comments":[{"comment":"The central evaluation in Table 1 requires that CREBench-Human measures generalization rather than memorization. The paper never states that the 25 benchmark images per object were excluded from the 5,000 training pairs. CREBench-Human is built by selecting 20 creative + 5 normal images from 5,000 generated images for the same five objects used in training, and the training pairs are sampled from generations obtained with the same prompt-generation recipe and, apparently, the same T2I models (Sec. 3.3). If any benchmark image is in the training-pair pool, a frozen backbone with a trainable MLP head can memorize preference labels over 20 epochs, inflating the Table 1 rank correlations. Please add an explicit holdout statement, or re-run the evaluation with a benchmark whose images are provably disjoint from training, and report which models/prompts generated the human benchmark images.","section":"Sec. 3.1, 3.3, 3.4"},{"comment":"The claim that Gemini-2.5 'exceeds inter-human correlation' compares the correlation of Gemini to the averaged human ranking against the mean annotator-to-annotator correlation. These quantities are not commensurable: averaging across annotators removes individual noise and mechanically increases correlation. The same issue affects the relative standing of CREward in Table 1, where CREward is also compared against the averaged human ranking. Please report LVLM/CREward correlations against each annotator separately (and their mean), or compute an inter-human baseline in the same aggregated metric (e.g., split-half correlation of averaged human ranks).","section":"Table 1 / Sec. 3.2"},{"comment":"The LoRA sliders are trained by maximizing the CREward score (Eq. 6), and the quantitative evaluation in Figure C uses that same CREward score as the primary success metric. This is partly self-confirming: the sliders are optimized for exactly the objective used to measure them. The LVLM improvement ratio in Figure D is a useful independent signal, but Gemma-3 is the same model family that generated the training labels, so it is not fully independent. Please report human ratings or a held-out LVLM (e.g., Gemini-2.5) for the slider comparison, and use CREward only as a monitoring signal rather than as the outcome measure.","section":"Sec. 4.2 / App. C.2.2, Eq. (6)"},{"comment":"The human benchmark is very small (5 objects, 5 annotators, 25 images per object, 100 pairs per object). With 25 items, the rank correlations have wide confidence intervals; for example, Texture CREward = 0.76 (0.12), Gemini = 0.74 (0.06), and Surprise = 0.49 (0.19) may not be statistically distinguishable. The statement that CREward 'surpasses even the closed-source Gemini-2.5 on texture creativity' is not supported without confidence intervals or a paired test across the five objects. Please provide such intervals or explicitly soften the claim.","section":"Table 1 / Sec. 3.5"}],"minor_comments":[{"comment":"The main text refers to 'Figure H' when presenting Grad-CAM results, but Figure H is a supplementary figure; the in-main reference should be Figure 9.","section":"Sec. 4.3"},{"comment":"The citation 'SDXL-DMD2 [36, 36]' duplicates reference 36; one entry should be [37]. Also, the TAESD decoder is referenced only by a footnote URL; a formal citation would be preferable.","section":"Sec. 3.4"},{"comment":"The clean-sample estimate is denoted \\hat{x}_{0,t} in the main text but \\bar{x}_{0,t} in the appendix. Please unify the notation.","section":"Eq. (3) and App. Eq. (5)-(6)"},{"comment":"The figure contains the typo 'CreReward Ranking'; it should be 'CREward Ranking'.","section":"Figure 2(d)"},{"comment":"Minor wording: 'Surprisescore' should be 'Surprise score', and the sentence beginning 'We use theSurprisescore' has a spacing issue.","section":"Sec. 3.5"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unverified train/benchmark split. If the authors can confirm that the CREBench-Human images are disjoint from the CREward training pairs, or re-run the evaluation with a strictly held-out human set, I would be willing to consider a significantly revised version. The comparison methodology for the 'exceeds inter-human' claim also needs correction. The contribution is useful for the creativity-evaluation niche, but the evidence is currently too weak to support the headline claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the idea is better than the headline experiments. Decomposing visual creativity into geometry/material/texture and training a per-axis reward from LVLM preferences is genuinely new and useful — it gives designers a cheap, interpretable score where they currently have monolithic Surprise/Rarity metrics or expensive human studies. CREBench is a real contribution, and the paper is candid about its own limits (novelty-over-value bias, entangled sliders).\n\nThe load-bearing claim is that CREward, trained only on Gemma-3 labels, aligns with human designer rankings (Table 1: 0.59–0.76 rank correlation). On reading it, the stress-test concern lands. The human benchmark (Sec. 3.1) takes 25 images per object from the same generative pipeline, same five objects, and plausibly the same prompt set used to build the 5,000 training pairs (Sec. 3.3); the paper never states the benchmark images were excluded from training. The 'fixed train–val–test split' in Sec. 3.4 refers to the LVLM-labeled pair set, not the human set. That is a load-bearing gap. It can be settled by releasing image IDs and confirming no overlap, but until then Table 1 could reflect the model having seen the benchmark images, inflating the correlations.\n\nTwo smaller issues. First, 'Gemini exceeds inter-human' (0.80 vs 0.71) is not apples-to-apples: the LVLM is scored against the averaged human ranking, while inter-human is pairwise annotator agreement, so the LVLM comparison has an advantage baked in. Second, with five annotators and 25 images per object, the standard deviations are wide enough that several headline differences (CREward vs Gemini on texture) are within noise.\n\nThe LoRA slider evaluation is partly self-confirming — improvement ratios come from Gemma-3, the same family that produced the training labels — but the negative-direction experiments are suggestive, and the paper flags the entanglement itself.\n\nWorth a serious referee — yes. The framing, benchmark, and applications warrant referee time. The revision needs: release of data/weights, an explicit holdout statement for CREBench-Human, and a statistically honest human baseline comparison. With those, the central claim becomes checkable.","headline":"Type-specific creativity reward is a genuinely useful idea; the headline human-alignment numbers need a holdout guarantee before they convince.","tokens_in":18680,"tokens_out":3925,"would_cite":true,"duration_ms":38967,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that visual creativity can be measured along three separate axes—geometry, material, and texture—and that a lightweight reward model trained only on vision-language-model labels can rank images much as human designers do.","keywords":["creativity reward model","type-specific creativity","geometry material texture","human perception alignment","vision-language model labels","pairwise preference learning","text-to-image diffusion","creative generation control"],"falsifier":"Build a fresh human benchmark from newly generated images and prompts that never appeared in CREward's training data, then recompute the same rank correlations. If the numbers fall to roughly the baseline metric's level, the claim of human-aligned generalization collapses; if they stay near 0.6–0.76, it survives.","tokens_in":17628,"feed_emoji":"🎨","tokens_out":5509,"duration_ms":55937,"temperature":0.7,"pith_summary":"The paper argues that visual creativity should not be a single undifferentiated score; it should be broken into geometry, material, and texture, mirroring the image-formation pipeline. The authors build a human benchmark and show that large vision-language models rank creative images in ways that correlate with human designer judgments. They then train CREward, a lightweight reward model, on pairwise labels produced by an open-weight vision-language model. On the human benchmark, CREward's type-specific rankings correlate with human rankings (0.59 geometry, 0.72 material, 0.76 texture, 0.61 overall), and it even outperforms a stronger closed-source model on texture creativity. If this holds, creativity can be evaluated, explained, and steered during image generation without costly human annotation.","feed_headline":"AI labels alone train a model to rank creative images like humans","feed_subtitle":"The CREward scorer breaks creativity into geometry, material, and texture and guides generation toward each type.","key_machinery":"The load-bearing object is CREward itself: a frozen vision backbone (SigLIP) feeding a five-layer MLP head that outputs four scalar scores—geometry, material, texture, and overall—trained with a pairwise logistic loss on 5,000 image-pair triplets labeled by an open-weight LVLM. The type decomposition, grounded in image-formation concepts of shape, surface-light interaction, and surface detail, is what carries the argument: it makes the reward signal interpretable and enables type-specific controls. For generation, one-step denoising estimates are scored by CREward and used to train lightweight LoRA adapters, while Grad-CAM is applied to the differentiable scores for attribution.","core_discovery":"The central claim is that a frozen vision backbone with a small reward head, trained only on pairwise preferences generated by an open-weight vision-language model, can reproduce human designers' relative judgments of creativity across geometry, material, and texture. The paper presents this as the first type-specific creativity reward model. Comparing CREward's rankings on a human benchmark with those of human annotators and large vision-language models, the authors report that CREward ranks second overall, ahead of its own teacher on material and overall creativity and ahead of the stronger closed-source model on texture. The same differentiable scores then serve as a reward signal to trai","pith_inferences":["A natural extension is to build a human benchmark on objects and prompts fully held out from CREward's training set; the sliders already hint at cross-object transfer, but the human-alignment numbers have not been verified there.","The high texture correlation (0.76) relative to low inter-human agreement (0.46) suggests CREward may be capturing a consensus that individual designers do not reliably express—testable by asking annotators to re-rank texture pairs after group discussion.","The LVLM-labeling recipe likely transfers beyond still images: the same geometry/material/texture decomposition could be applied to 3D assets or video frames.","Since the stronger closed-source LVLM still beat CREward overall, distilling labels from that model or ensembling multiple LVLM annotators would probably push an open-weight reward model past the reported ceiling."],"forward_implications":["Creativity evaluation can be scaled without human annotation: LVLM-generated labels can seed a fast, lightweight scorer that tracks human judgments.","Type-specific scores reveal which axis most strongly drives overall creativity—geometry correlates highest—so generative effort can be directed at shape rather than surface detail.","CREward can filter large batches of generated images, surfacing top creative examples for design inspiration in human–AI co-creation workflows.","Because the score is differentiable, it can be plugged into diffusion fine-tuning, yielding type-specific or mixed LoRA sliders that generalize to objects never seen during training.","The model provides a structured benchmark for comparing text-to-image systems on creative output, complementing fidelity-focused metrics."],"fun_headline_variants":["First creativity reward model splits into geometry, material, texture","AI labels train a creativity scorer that beats its teacher","Type-specific creativity reward ranks human-like, guides generation","CREward: reward model that specializes creativity scoring"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The human benchmark images are assumed to have been held out from the 5,000 pairs used to train CREward, but the paper does not state that exclusion, so the reported human correlations could partly reflect memorization of familiar images rather than learned creativity judgments.","fun_headline_variants_meta":{"raw":{"variants":["First creativity reward model splits into geometry, material, texture","AI labels train a creativity scorer that beats its teacher","Type-specific creativity reward ranks human-like, guides generation","CREward: reward model that specializes creativity scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":965,"prompt_tokens":681,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":425,"tokens_out":284,"duration_ms":4376,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:22:33.194119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a fresh human benchmark from newly generated images and prompts that never appeared in CREward's training data, then recompute the same rank correlations. If the numbers fall to roughly the baseline metric's level, the claim of human-aligned generalization collapses; if they stay near 0.6–0.76, it survives.","supporting_citations":[],"review_version":1}