{"id":"64a9b5ee-82a3-4e3e-a06b-d6885349b232","arxiv_id":"2508.12628","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multimodal language model, trained on a new annotated image-pair dataset and a recall-precision reinforcement reward, selects better advertising creatives and explains its choices, with measured offline and online gains.","lead":"This paper builds an AI system that looks at two advertising images for the same product and decides which one shoppers are more likely to click, explaining its choice in plain language. It introduces a new training dataset of 8,000+ annotated image pairs and a reinforcement-learning step designed to make that reasoning more accurate and useful.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Q10 click-likelihood labels are never validated against real user behavior; if annotator intuition diverges from CTR, the 0.791 offline accuracy and the reward in Eq. (4) optimize an unverified target.","rationale":"The reader's weakest assumption is the same one I find most load-bearing: Q10 label validity. The central claim has two pillars, offline superiority and online business gains. The online A/B test supports the deployment pillar directly, but the offline pillar—used to claim superiority over Co-Instruct, DepictQA-Wild, Q-Insight, and VAM—depends on labels that are never checked against actual clicks. The paper's own description of the online display strategy in Sec. 1 shows the platform already has a way to obtain real CTR feedback, so a validation study is feasible. I also noted a secondary inconsistency: the abstract says Creative4U is trained with CoT-SFT plus GRPO-RPR, while Table 2 reports the best result for GRPO-RPR alone and lower accuracy for the explicit CoT-SFT + GRPO-RPR combination. That is a real reporting issue, but it is less load-bearing because the empirical 0.791 result still stands and could be re-described accurately. The label-validity concern, by contrast, questions whether the benchmark itself measures what the paper claims. Since the concern is a missing validation rather than demonstrated failure, CONDITIONAL remains the right verdict and no adjustment is needed.","tokens_in":16540,"tokens_out":8800,"duration_ms":96342,"concrete_test":"Run an online 'small traffic display' experiment on a random sample of CreativePair test pairs: serve each image of a pair with equal traffic for the same product/query, record observed CTR per image, and compare the empirical CTR winner with the Q10 label across pairs. Report agreement rate and a confidence interval; if agreement is not significantly above 50%, the 0.791 offline accuracy measures annotator preference rather than true click likelihood, and the offline claims in Tables 1-3 must be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central offline claim—Creative4U selects the image users are more likely to click—rests entirely on the human Q10 answer in the Creative Evaluation Protocol (Sec. 3.2). The paper reports no inter-annotator agreement, no comparison of Q10 labels with logged click-through rates for the same creatives, and no measurement of annotation noise. Q10 is the ground truth for the accuracy reward R_acc in Eq. (4) and for every offline accuracy in Tables 1-3, so any systematic divergence between annotators' click intuition and actual user behavior propagates directly into the 0.791 headline number. The online A/B test (Table 4) is real click evidence, but it validates the full Creative4U pipeline against Random and Bandit, not the dataset labels, and it is not compared against the other offline baselines (Co-Instruct, DepictQA-Wild, Q-Insight); it also lacks confidence intervals and contains a prose/table mismatch in the reported CTR gain (7.07% vs 2.4%). Thus the 'outperforms existing ranking and comparison baselines' portion of the central claim is not independently grounded if the label target is invalid. This is a missing validation rather than an observed failure, but it is the load-bearing assumption of the offline evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an explainable advertising creative image selection paradigm. It introduces CreativePair, a dataset of 8,879 image pairs annotated with the Creative Evaluation Protocol (similarity, image-text consistency, five quality dimensions, and a final Q10 click-likelihood label) and with chain-of-thought reasoning generated by Qwen2.5-VL-72B-Instruct. The proposed Creative4U model is Qwen2.5-VL-7B trained with CoT-SFT followed by GRPO with rewards for format, answer accuracy, and CoT recall/precision. Experiments report offline accuracy 0.791, outperforming several IQA and ranking baselines, and an online A/B test on Taobao reporting CTR/RPM/CVR improvements over random and bandit baselines. The paper also describes a two-stage deployment pipeline and an explainability use case in AIGC feedback.","tokens_in":16881,"tokens_out":5967,"duration_ms":57128,"significance":"If valid, the work is significant: it is a first systematic, explainable approach to creative selection, with a structured protocol and dataset that the community can reuse; the GRPO-RPR reward design is a concrete, reproducible contribution; and the online deployment demonstrates practical value. The central offline claim, however, rests on human Q10 labels that are not validated against real click behavior, and the online evidence lacks statistical grounding. With those gaps addressed, the contribution would be solid.","major_comments":[{"comment":"The ground-truth labels for the selection task ('which image is the user more likely to click on') are never validated against actual user behavior. The paper reports no inter-annotator agreement, no comparison of Q10 labels with logged click-through rates for the same creative pairs, and no estimate of annotation noise. Since R_acc in Eq. (4) and every offline accuracy in Tables 1-3 depend on these labels, a systematic divergence between annotators' click intuition and real user behavior would directly invalidate the headline 0.791 and the offline comparisons. The online A/B test validates the full pipeline, not the dataset labels, so it cannot by itself rescue the offline claim. Please add annotation-quality metrics (e.g., Cohen's kappa or Krippendorff's alpha) and, if possible, a held-out subset where Q10 labels are compared with observed CTR for the same creatives.","section":"Section 3.2, Q10"},{"comment":"The CoT ground truth used for the recall and precision rewards, and for the reasoning-level metrics in Tables 1-3, is generated by Qwen2.5-VL-72B-Instruct and then audited by humans with a claimed 95.3% pass rate. This means the reasoning-quality metrics measure agreement with a model-generated protocol, not with independent human reasoning. The paper does not report how many CoT texts were rejected or what the refinements changed, nor whether the audit was performed by multiple annotators with agreement measurement. Please provide a human-written or fully human-verified reasoning subset and report agreement between the model-generated CoT and human reasoning on that subset, so the reasoning-level claims are not circular.","section":"Section 4.2, Eqs. (5)-(7) and Section 3.2"},{"comment":"The online A/B results are reported without confidence intervals or significance tests, so the reader cannot assess whether the CTR/RPM/CVR gaps are meaningful or within noise. Furthermore, the prose contradicts the table: without the online ranker the text reports '2.4% in CTR, 2.08% in RPM, and 0.96% in CVR', while Table 4 lists +7.07%, +6.92%, +2.48% for the same setting; with the online ranker the text says 1.32%/1.34%/0.35% while the table lists +3.75%/+3.86%/+1.69%. Please reconcile these numbers, report traffic allocation and the number of ad sets, and add statistical evaluation (e.g., bootstrap confidence intervals or p-values) for the reported improvements.","section":"Section 5.6, Table 4"},{"comment":"The VAM baseline is described as 'a visual-aware creative ranking method', but the original VAM (reference [22]) is a hybrid bandit model that relies on online feedback. The paper states that 'all baselines are trained on CreativePair with their default settings', yet gives no details on how a bandit algorithm is adapted to a static pairwise selection task. Without this adaptation protocol, the reported 0.608 accuracy for VAM is not interpretable, and the claim that Creative4U outperforms ranking-based approaches is weakened. Please specify how VAM was adapted, what its training data and loss were, and how the pairwise comparison was performed at inference.","section":"Section 5.1, Table 1, VAM row"}],"minor_comments":[{"comment":"The abstract says '8k annotated image pairs' while the body reports 8,879 pairs (7,088 training + 1,791 test). Please make the numbers consistent and clarify that filtering reduced the collected 10,000 pairs to 8,879.","section":"Abstract and Section 3.2"},{"comment":"References [32] and [33] appear to be the same work (DepictQA-Wild, arXiv:2405.18842). Please remove the duplicate or cite distinct versions if intended.","section":"References [32] and [33]"},{"comment":"The statement that 'a balanced weighting yields the best trade-off' and that the default setting attains 'the best F1 with near-best accuracy' is slightly misleading: ID 6 in Table 3 has accuracy 0.796, higher than the default 0.791, with comparable precision, at the cost of a small F1 decrease (0.510 vs 0.514). Please rephrase to avoid implying the default setting is strictly dominant.","section":"Section 5.4, Table 3"},{"comment":"The claim that the win rate of AIGC creatives against human-made ones improved from 12.2% to 44.5% lacks methodology: no description of how the comparison was set up, how many creatives were evaluated, or whether the judges were blind to condition. Please add this information or clearly label the result as an anecdotal report.","section":"Section 5.7"},{"comment":"The caption of Figure 6 says green highlights incorrect reasoning steps and red emphasizes correct ones, but the surrounding text discusses models being 'misled' without a consistent color-key. Please check the color assignments for consistency with the figure. Also, 'Image-Text Consistence' in Figure 1 and Algorithm 1 should read 'Consistency'.","section":"Figure 6 and general presentation"},{"comment":"The dataset and code are announced as 'will be made public', but no release URL or planned access mechanism is provided. Please include a link or a clear statement about how reviewers can access the data for verification.","section":"Section 1 and Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong industrial application with a useful dataset and a clean training recipe, but the three load-bearing validation gaps (human-label validation, CoT ground-truth provenance, and online statistical rigor) must be addressed before the claims can be fully trusted. The dataset-size and baseline-adaptation issues are secondary but should also be fixed. I recommend major revision rather than rejection because the weaknesses are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper ships a new dataset (CreativePair, ~8.9k pairs), a structured evaluation protocol for ad creatives, and a GRPO variant with recall/precision rewards on the reasoning trace. The offline accuracy is 0.791 on their test set, and the online A/B test on Taobao shows real CTR/RPM gains over random and a bandit baseline. That's genuine applied evidence, and the paper deserves a serious referee.\n\nWhat's new: the protocol is more structured than the usual \"pick the prettier one,\" and the dataset includes chain-of-thought reasoning aligned to that protocol. The GRPO-RPR reward design—explicitly rewarding coverage and precision of the comparison dimensions—is a reasonable extension of existing RL training for MLLMs. The paper also shows the model's rationales are useful in practice (AIGC win rate up from 12.2% to 44.5%), which is a nice qualitative result.\n\nThe soft spots are real but not fatal. First, the reasoning-level ground truth (CoT) is generated by Qwen2.5-VL-72B, not by humans. They do audit it, but the recall/precision rewards optimize toward the teacher's reasoning style, which may not be what human experts would write. Second, the final click-likelihood label (Q10) is never validated against actual click behavior for the same creatives. No inter-annotator agreement, no label-CTR comparison. That's a gap, but the online A/B test partially closes it: if the human labels were wildly off, a model trained on them would be unlikely to beat random in production. Still, they don't compare the other offline baselines online, so we don't know if the offline ranking against those baselines would transfer.\n\nThird, the reporting is sloppy. Table 4 shows Creative4U at +7.07% CTR without the ranker, but Sec 5.6 says 2.4%. That's a factor of three discrepancy. Also no confidence intervals or significance tests on any online metric. Fourth, code and data are promised but not yet public, so reproducibility is unverified.\n\nOverall: the core claim—that an MLLM can be trained to select better ad creatives with an explainable rationale—is supported by the online experiment. The paper is for people working on ad tech, image quality assessment, and applied MLLM reasoning. It's not a paradigm-shifting theoretical result, but it's a solid, deployable system. I'd send it to review with the expectation of heavy revision: add significance tests, fix the table/prose mismatch, report annotation agreement, and ideally run a small label-vs-CTR validation. The underlying building blocks are sound.\n\nRecommendation: accept conditional on those fixes. It's a serious paper worth refereeing.","headline":"Real online gains and a new dataset make this worth reading, but reporting sloppiness and the unvalidated Q10 label keep it from being a clean accept.","tokens_in":17387,"tokens_out":7290,"would_cite":false,"duration_ms":62767,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a vision-language model trained to compare two ad creatives can predict which one users are more likely to click and explain why, beating bandit and ranking baselines offline and lifting CTR by up to 7.07% in…","keywords":["advertising creative selection","image quality assessment","multimodal large language model","comparative reasoning","chain-of-thought","reinforcement learning","e-commerce advertising"],"falsifier":"Take a random sample of CreativePair pairs, have two independent annotation teams re-label them, and measure agreement on the final Q10 answer: near-chance agreement would mean the dataset's target is mostly annotation noise. Stronger: log the actual click outcome when a served product shows one creative and a replacement candidate, and test whether the human label predicts which image gets the higher CTR significantly above 0.5.","tokens_in":16358,"feed_emoji":"🛒","tokens_out":13437,"duration_ms":121195,"temperature":0.7,"pith_summary":"Creative4U claims that the problem of choosing which advertising image will perform best can be turned into a comparative reasoning task: given two creatives for the same product, the product title, and the user's search terms, a multimodal language model can decide which image users are more likely to click on and produce a step-by-step explanation of the decision. To support this, the paper introduces a dataset of 8,879 paired creatives annotated under a structured Creative Evaluation Protocol, each pair supplied with a chain-of-thought comparison generated by a large vision-language model and audited for consistency. A 7-billion-parameter multimodal model is then trained with supervised fine-tuning followed by group-relative policy optimization with recall and precision rewards, so the model is rewarded both for picking the right image and for covering the comparison dimensions that actually matter. The paper reports offline selection accuracy of 79.1% on held-out pairs, beating four baselines, and an online A/B test in which replacing the production offline selector lifts click-through rate by up to 7.07% relative to random selection. If the claim holds, advertisers facing a flood of AI-generated creatives could select offline, with explainable feedback, instead of spending traffic to learn which image wins.","feed_headline":"Creative selector explains its picks, lifts CTR 7.07%","feed_subtitle":"It also says why it chose that image, letting designers and AI tools fix the next creative instead of guessing.","key_machinery":"The load-bearing mechanism is the reward function in group-relative policy optimization: $R(y_i) = R_{\\mathrm{acc}}(y_i) + \\alpha_1 R_{\\mathrm{format}}(y_i) + \\alpha_2 R_{\\mathrm{rec}}(y_i) + \\alpha_3 R_{\\mathrm{prec}}(y_i)$. The two reasoning-level rewards are defined over the sets of protocol questions: $R_{\\mathrm{rec}} = |Q_{\\mathrm{correct}}|/|Q_{\\mathrm{gt}}|$ and $R_{\\mathrm{prec}} = |Q_{\\mathrm{correct}}|/|Q_{\\mathrm{pred}}|$, where $Q_{\\mathrm{correct}}$ counts dimensions on which the model's judgement matches the ground-truth annotation, $Q_{\\mathrm{gt}}$ is the set of dimensions where the two images actually differ, and $Q_{\\mathrm{pred}}$ is the set the model chose to discuss. Recall pushes the chain-of-thought to cover all decision-relevant differences; precision penalizes irrelevant or hallucinated comparisons. The Creative Evaluation Protocol supplies those dimensions in the first place: annotators compare image-text consistency, text in image, models and props, layout, product subject, and background before giving a final 'more likely to click' verdict. GRPO-RPR is this reward design plus a group-relative advantage update, and the paper's ablations show that the recall and precision terms are complementary: removing either one degrades accuracy or F1.","core_discovery":"The central claim is that explainable creative selection can be learned end-to-end from paired comparisons, and that the explanation quality is not a by-product but part of what makes the selection accurate. The model's output has two parts: a <think> chain in which the model compares the two creatives along seven protocol dimensions (query hit rate, title hit rate, text in image, models and props, layout, product subject, background), and an <answer> giving the final choice. Training uses a composite reward: an accuracy reward for the final answer, a format reward, and two reasoning-level rewards, where recall is the fraction of ground-truth differing dimensions that the model mentions correctly and precision is the fraction of mentioned dimensions that are correct. The policy is updated by normalizing these rewards within a sampled group. On the CreativePair test set, this reaches 0.791 accuracy, 0.799 recall, and 0.514 F1, ahead of the best baseline at 0.751 accuracy; in production, replacing the existing offline selector with the model's rankings lifts CTR by 7.07% without the downstream ranker and by 3.75% with it, and also lifts CVR and RPM.","pith_inferences":["The same recall–precision reward design should transfer to any MLLM task whose chain-of-thought can be checked against a fixed set of dimensions, such as document relevance or diagnostic image triage; the paper only demonstrates it on e-commerce creatives.","A natural validation the paper does not report is to compare CreativePair's human labels against the actual click-through rate of the same pairs served in production; if labels predict clicks, the offline metric is a faithful proxy, and if not, the model has been trained to match annotator taste rather than user behaviour.","The reported jump in AIGC win rate suggests the rationales could be fed back into the generation model itself as prompts or edit instructions, which the paper did not test; this would close the loop between selection and generation.","The two-stage coarse-to-fine selection recipe is a general pattern for converting a pairwise preference model into a ranked list under a candidate budget; it could be reused for any content domain where only pairwise judgments are available."],"forward_implications":["An e-commerce platform can rank AI-generated creatives offline from pairwise comparisons, avoiding small-traffic online exploration and still improving CTR, CVR, and RPM over both random selection and a bandit selector.","Because the selector emits dimension-level rationales, designers and AIGC pipelines receive actionable feedback; in the paper's deployment, iterating on that feedback raised the win rate of AI-generated creatives against human-made ones from 12.2% to 44.5%.","The coarse-to-fine deployment strategy keeps the cost of pairwise selection manageable: with up to 100 candidates, one pass against the current creative keeps roughly 12%, so the subsequent all-pairs ranking runs on a small survivor set.","The reward decomposition implies that reasoning quality is not decorative: the recall and precision rewards jointly drive decision accuracy, since the ablation without recall drops F1 from 0.514 to 0.376 and the ablation without precision drops accuracy to 0.729.","At 0.71 seconds per pair on the hardware used, the selector fits an offline batch pipeline; moving it to real-time serving would require lower latency."],"supporting_citations":[{"why":"Technical report for the open vision-language model used as the base of Creative4U.","marker":"[2]"},{"why":"Supplies the hybrid bandit baseline and the online bandit comparator that Creative4U is measured against in deployment.","marker":"[22]"},{"why":"A comparative image-quality baseline trained on CreativePair and evaluated against Creative4U.","marker":"[27]"},{"why":"A descriptive image-quality baseline that the paper compares with on accuracy and reasoning metrics.","marker":"[33]"},{"why":"The strongest accuracy baseline and the comparison point for GRPO with only accuracy and format rewards.","marker":"[11]"},{"why":"Introduces the group-relative policy optimization update that GRPO-RPR extends with reasoning-level rewards.","marker":"[19]"},{"why":"Chain-of-thought prompting, the basis for the CoT-SFT training stage that produces the supervised reasoning behaviour.","marker":"[24]"}],"fun_headline_variants":["AI explains ad picks, lifts CTR 7.07%","Creative4U: MLLM selector with reasons, +7.07% CTR","Explainable creative AI boosts CTR by 7.07%","Pick better ads with AI that tells you why","MLLM explains creative choices, CTR up 7.07%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human annotator's final answer — 'which image the user is more likely to click on' — matches what real users actually click; the paper does not report inter-annotator agreement or label-to-CTR validation, so if that intuition is wrong, the dataset target, the reward, and the offline accuracy all inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["AI explains ad picks, lifts CTR 7.07%","Creative4U: MLLM selector with reasons, +7.07% CTR","Explainable creative AI boosts CTR by 7.07%","Pick better ads with AI that tells you why","MLLM explains creative choices, CTR up 7.07%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000487,"raw_usage":{"total_tokens":2448,"prompt_tokens":1042,"completion_tokens":1406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":658,"tokens_out":1406,"duration_ms":11394,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:20:23.020239+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of CreativePair pairs, have two independent annotation teams re-label them, and measure agreement on the final Q10 answer: near-chance agreement would mean the dataset's target is mostly annotation noise. Stronger: log the actual click outcome when a served product shows one creative and a replacement candidate, and test whether the human label predicts which image gets the higher CTR significantly above 0.5.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid bandit baseline and the online bandit comparator that Creative4U is measured against in deployment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A comparative image-quality baseline trained on CreativePair and evaluated against Creative4U."}],"review_version":1}