{"id":"da69b3cb-9c91-4fa0-ab40-c751c8f1f22d","arxiv_id":"2508.18739","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DIVER trains an LLM with synthetic data plus multi-objective RL to emit several diverse, style-tagged ad headlines in one pass, reporting online ADVV and CTR gains.","lead":"The paper presents DIVER, a two-stage LLM pipeline (synthetic-data fine-tuning, then multi-objective reinforcement learning) that generates multiple diverse ad headlines in a single forward pass, deployed at Xiaohongshu. It reports a 4.0% advertiser-value gain and a 1.4% CTR gain, offering a concrete recipe for balancing diversity and quality in industrial text generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Online A/B test confound: serving-time user-profile matching may explain reported ADVV/CTR lifts","rationale":"The reader identified reward-proxy validity as the weakest assumption, and that is indeed a concern. However, the more directly load-bearing issue for the central business claim is the online A/B test confound: the serving-time user-profile selector is described only for DIVER, never for the baseline. The reader's rationale mentions an 'under-specified A/B design' but does not elevate it to the weakest assumption. My stress-test shifts the focus to this confound because the +4.0% ADVV and +1.4% CTR are the headline evidence, and they could be entirely due to the selector rather than to DIVER's generation improvements. This does not contradict the reader's conditional verdict; it adds a specific, testable condition that must be met before accepting the business impact claim. The offline evaluation is internally consistent, and the ablation studies show the proposed components affect the measured metrics, but the link from those offline gains to online ADVV/CTR is not demonstrated without controlling for the selector. Hence the paper remains conditional, not rejected, and I request the additional A/B detail or re-analysis.","tokens_in":12327,"tokens_out":3870,"duration_ms":37749,"concrete_test":"Request or design an online experiment with three arms: (1) DIVER-generated 30 headlines with the profile-matching selection; (2) Sampling+SFT-generated 30 headlines with the exact same profile-matching selection; (3) DIVER-generated 30 headlines served with random selection (no personalization). If the ADVV/CTR difference between (1) and (2) is not statistically significant while (1) vs (3) is significant, the reported lifts stem from the selector, not from DIVER's generation. Alternatively, re-analyze existing A/B logs: for each impression, compute the lift of the selected headline over a random headline from the same generated set; if that internal lift is ~4.0% ADVV, the baseline comparison is confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not the reward-proxy validity, but the under-specified online A/B contrast (Section 4.6, Table 4). The treatment arm generates 30 DIVER headlines and then selects the one most semantically similar to the user profile. The paper never states whether the 'Sampling + SFT' baseline uses the same 30-headline generation, the same similarity metric, and the same profile representation. If the baseline serves without user-profile matching (or with a different selector), the reported +4.0% ADVV and +1.4% CTR could be attributable to the selector itself, not to the SFT+RL generation improvements that are the paper's central contribution. The offline tables (Tables 1–2) show DIVER's diversity/quality gains, but those metrics are not validated as causal drivers of ADVV/CTR. Without a controlled comparison that isolates generation quality from selection effects, the headline business claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DIVER, a multi-stage framework for ad headline generation that combines a semantic-/stylistic-aware synthetic data pipeline (for cold-start SFT) with multi-objective reinforcement learning (GRPO) using five reward components: diversity, quality, CTR, quantity, and format. The authors report offline results on a 3,000-instance industrial test set showing lower PairBLEU/SelfBLEU and higher StyleCov than strong LLM baselines while maintaining NLI/ROUGE quality, plus ablations indicating each component matters. They also report an online A/B deployment on Xiaohongshu's Explore Feed, claiming +4.0% ADVV and +1.4% CTR relative to a 'Sampling + SFT' baseline, with the serving-time step of selecting the generated headline most semantically similar to the user profile.","tokens_in":12653,"tokens_out":4085,"duration_ms":46437,"significance":"If the claims hold, DIVER would be a useful industrial contribution: it shows that a diversity-aware reward can be combined with quality and CTR objectives without sacrificing faithfulness, and it is deployed at scale. The paper gives a concrete, reproducible-looking recipe (data augmentation, SFT, GRPO with five rewards) and includes thorough ablations that demonstrate each reward and each pipeline stage has a measurable effect. However, the two load-bearing empirical claims—offline diversity/quality balance and online business lift—are weakened by (i) direct overlap between the diversity reward and the offline diversity metrics, and (ii) an under-specified online A/B contrast that does not isolate generation quality from the serving-time selection mechanism. The lack of variance reporting and the single-split test further limit confidence in the precision of the reported numbers.","major_comments":[{"comment":"The online A/B test is under-specified. The text says DIVER generates 30 headlines and then selects 'the headline most semantically similar to the user profile.' For the 'Sampling + SFT' baseline, it is not stated whether the same 30-headline generation procedure, the same similarity metric, and the same user-profile representation are used. If the baseline serves without user-profile matching (or with a different selector), the reported +4.0% ADVV and +1.4% CTR may be driven by the selection step, not by the SFT+RL generation pipeline that is the paper's central contribution. Please specify the baseline serving protocol explicitly, and ideally include a third arm that uses the same selector on the baseline model's outputs.","section":"§4.6, Table 4"},{"comment":"There is a direct overlap between the diversity reward and the offline evaluation metrics. The reward is rdiversity = (1 − PairBLEU(Y) + Coverage(Y))/2, while Table 1's diversity columns include PairBLEU and StyleCov (and SelfBLEU/DisNGram/CosSim are correlated). Thus the offline diversity gains are partially by construction; the model is optimized against the same statistics that are later used to claim success. The paper should add an independent diversity evaluation (e.g., human judgments of distinctiveness, or a downstream task where diversity is not part of the training objective) or explicitly acknowledge that the offline diversity metrics are not independent of the reward.","section":"§3.2.1 vs §4.1, Tables 1–2"},{"comment":"No variance or significance testing is reported anywhere. All offline results are point estimates on a single 3,000-instance test split. For a paper making an industrial deployment claim, it is essential to report confidence intervals or at least significance tests (e.g., bootstrap or multiple evaluation seeds) for the main tables. This is not a perfunctory request: the differences in some quality columns (e.g., Rouge-1: 16.71 vs 17.93 for Qwen2.5-72B in Table 1) are small and may be within noise, which would affect the 'maintains high quality' claim.","section":"§4.2–§4.3, Tables 1–2"},{"comment":"The CTR reward model is trained on headlines generated by high-temperature SFT, labeled by the top and bottom third of actual CTR. During RL, the model being optimized shifts the distribution of generated headlines; there is no evidence that the CTR predictor remains calibrated or discriminative on DIVER-generated headlines. Since the online CTR lift is a headline result, the paper should report some validation of the CTR reward model on held-out head-to-head data, or at least discuss the distribution shift. Without this, the mechanism by which GRPO improves real CTR is unclear.","section":"Appendix A, 'Details about CTR Reward'"}],"minor_comments":[{"comment":"The normalization of PairBLEU and Coverage in rdiversity is ambiguous. Table 1 reports PairBLEU values like 2.08 and StyleCov as percentages (63.42%). If these are on different scales (0–100 vs 0–1), the arithmetic in Eq. (1) is undefined. Please state the assumed scales and, if necessary, normalize explicitly.","section":"§3.2.1, Eq. (1)"},{"comment":"Typo: 'All datasets used in this study used are properly licensed' should be 'All datasets used in this study are properly licensed.'","section":"Ethical Considerations"},{"comment":"The metric names 'Rouge-1', 'Rouge-2', 'Rouge-L' should be capitalized as 'ROUGE-1', 'ROUGE-2', 'ROUGE-L' to match standard usage (and the reference 'Chin-Yew, 2004' is also typically 'Lin, 2004').","section":"§4.1, Evaluation Metrics"},{"comment":"In the 'Home Improvement Fence Ad' column, 'User Type 1' is described as 'Male DIY & fitness lover'—'fitness lover' is not obviously relevant to a fence-installation ad; consider whether this is a typo (e.g., 'DIY & home-improvement lover').","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly a systems/industry paper, and the claimed deployment is impressive. However, the online A/B contrast is the kind of detail that an industrial paper must get right; as written, it does not rule out the selector as the cause of the lift. The diversity-reward/evaluation overlap is also a serious concern for the offline claims. I would be willing to look at a revised version that addresses the serving-protocol specification, adds independent or at least disambiguated diversity evaluation, and reports variance/significance. The manuscript's own Limitations section is honest about noise and fixed reward metrics, which is a point in its favor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competent industrial package: LLM-synthesized data with a 16-style taxonomy, cold-start SFT to emit N headlines in one pass, and a five-reward GRPO objective. The offline tables are internally consistent, and the ablations show each component matters, including diversity. The diversity reward directly targets pairwise BLEU and style coverage, which are also evaluation metrics, so there is some circularity—but the other diversity metrics (SelfBLEU, DisNGram, CosSim) also improve, so the effect is not purely self-fulfilling. Credit where due: this is a reasonable, deployable recipe, and the paper is honest about limitations (fixed reward metrics, long-tail scarcity, and so on).\n\nThe soft spot, as your stress-test note says, is the online A/B contrast. Section 4.6 says DIVER generates 30 headlines and then selects the one most semantically similar to the user profile. The baseline is described only as 'high-temperature sampling and SFT without synthetic data.' The paper never states whether that baseline also generates 30 headlines, applies the same similarity metric and profile representation, or does any user matching at all. If the baseline serves without the personalization step, the +4.0% ADVV and +1.4% CTR could come entirely from the selector, not from the SFT+RL generation improvements that are the paper's actual contribution. That is not a minor omission; it is the load-bearing claim. Also, no confidence intervals or significance tests appear anywhere, online or offline.\n\nThat said, the flaw is a reporting gap, not a contradiction in the data. There is no evidence the authors are hiding a negative result; the offline tables are plausible and the method is coherent. The paper would be much stronger with a proper A/B design that isolates generation from selection, plus variance estimates for the main metrics.\n\nWho this is for: people in applied NLG, ad-tech, and LLM fine-tuning who want a concrete system recipe. It will not reshape a scientific field, but it is a useful case study. I would send it to a serious referee, because the contribution is substantial and the missing details are fixable in revision. My recommendation: engage with it, ask the authors for the A/B arm definitions and per-run variance, and cite it if you work on text-generation diversity or advertising.\n\nFor the record, I disagree with any claim that this is a flawed or circular work outright—the reward–evaluation overlap is real but only partial, and the ablations compensate. The paper is worth reading and worth reviewing.","headline":"A solid industrial pipeline for generating diverse ad headlines, but the online A/B claim is under-specified: the serving-time selector may explain the lift, so take the +4.0% ADVV with caution.","tokens_in":13116,"tokens_out":1776,"would_cite":true,"duration_ms":21355,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims DIVER, a two-stage framework with synthetic data and multi-objective reinforcement learning, generates diverse ad headlines in one pass and, deployed at scale, raises advertiser value by 4.0% and click-through rate by 1.4%","keywords":["ad headline generation","diversity","large language models","reinforcement learning","multi-objective optimization","GRPO","advertising","personalization"],"falsifier":"An A/B test that keeps the serving-time user-profile selection step but replaces the reinforcement-learning-trained generator with the supervised-fine-tuning-only checkpoint from the paper's ablations: if the 4.0% advertiser-value and 1.4% click-through-rate gains disappear, the reinforcement-learning stage is the cause; if they persist, the gains come from the synthetic data or the selector.","tokens_in":12244,"feed_emoji":"📢","tokens_out":6446,"duration_ms":67857,"temperature":0.7,"pith_summary":"This paper tries to show that ad headline diversity can be treated as an explicit training objective rather than a by-product of sampling. It argues that two interventions—a synthetic data pipeline that annotates headlines with semantic keywords and stylistic types, and a reinforcement-learning stage that rewards low pairwise BLEU, style coverage, faithfulness, predicted click-through rate, correct count, and parseable format—let one LLM emit many diverse, high-quality headlines in one pass. Offline, the method beats closed- and open-weight baselines on diversity while staying near the best on quality. Online, on a large content-sharing platform, the deployed system improves advertiser value by 4.0% and click-through rate by 1.4% against a sampling-plus-supervised-fine-tuning baseline.","feed_headline":"Training for diversity lifts ad value 4% and CTR 1.4%","feed_subtitle":"A two-stage recipe—synthetic multi-headline data plus multi-objective reinforcement learning—makes output varied without losing faithfulness","key_machinery":"The mechanism is the set-level composite reward plus the two-stage training that makes it usable. Pairwise BLEU and style coverage are combined into a diversity score; a binary faithfulness classifier and a click-through prediction model supply quality and user-preference scores; quantity and JSON-format rules keep the output usable. Averaging these five rewards and optimizing with GRPO, a policy-gradient reinforcement-learning algorithm, pushes the model to generate N headlines that are simultaneously different, on-topic, engaging, and parseable, in one forward pass.","core_discovery":"The paper's central claim is that diversity in one-to-many generation is not a decoding-time accident; it can be optimized at the set level. DIVER first builds synthetic multi-headline training data by extracting a semantic keyword and a style (directness, emoji usage, rhetorical device) from real ad headlines, then fine-tunes a generator to follow those cues, then applies GRPO to maximize a composite reward over the whole generated set. The reward averages five signals—diversity (1 minus average pairwise BLEU combined with style coverage), faithfulness to the ad content, predicted click-through rate, output count against target, and JSON parseability. The paper reports that this produces th","pith_inferences":["The same two-stage recipe is likely portable to other one-to-many creative generation tasks, such as product descriptions, social post variants, or email subject lines, because nothing in the reward design is ad-specific except the click-through model.","The pairwise-BLEU part of the diversity reward may under-reward legitimate reuse of key product terms; embedding-based set diversity could capture semantic spread without penalizing those words.","The serving-time user-profile matcher is exactly where diversity and personalization meet: more diverse candidate sets give the matcher more room, so a sweep over candidate count could isolate how much of the online gain comes from training quality versus selection quality.","The online numbers conflate the trained generator with the serving selector; applying the same user-profile matcher to a stronger supervised-fine-tuning-only generator would pinpoint the marginal value of the reinforcement-learning stage."],"forward_implications":["A single forward pass can replace sampling-based diversity tricks: the model is trained to emit N diverse headlines at once, so serving needs no rejection sampling or prompt reshuffling.","Diversity and quality are not necessarily in tension: the composite reward finds a point where pairwise BLEU drops to 2.08 while NLI faithfulness stays near the best baselines.","Both stages carry weight: ablations show removing the synthetic data pipeline or the reinforcement-learning stage separately degrades diversity and faithfulness, and removing both is worst.","The gains materialize in business metrics: deployed with user-profile matching over 30 candidates, the system improves advertiser value by 4.0%, click-through rate by 1.4%, impressions by 2.4%, and cost per mile by 2.0%."],"supporting_citations":[{"why":"Supplies the possibility-exploration fine-tuning baseline and frames diversity as a training objective that the paper extends with multi-objective reinforcement learning.","marker":"(Mai and Carson-Berndsen, 2024)"},{"why":"Supplies GRPO, the policy-gradient algorithm used for the multi-objective reinforcement-learning stage.","marker":"(Shao et al., 2024)"},{"why":"Supplies BLEU, used both as the pairwise semantic-diversity reward and as an evaluation metric.","marker":"(Papineni et al., 2002)"},{"why":"Provides evidence that supervised fine-tuning and RLHF can shrink output diversity, motivating the explicit diversity reward.","marker":"(Kirk et al., 2024)"},{"why":"Defines the base model on which supervised fine-tuning and reinforcement learning are run; all comparisons and ablations use it.","marker":"(Qwen, 2025)"},{"why":"Supplies Self-BLEU, one of the diversity metrics used to compare against baselines.","marker":"(Zhu et al., 2018)"}],"fun_headline_variants":["Diverse ad headlines via LLMs lift value 4% and CTR 1.4%","Unlock diversity in ad headlines: 4% value, 1.4% CTR","LLM multi-objective training: diverse headlines boost CTR and value","Diversity-aware ad headline generation: 4% ADVV lift, 1.4% CTR","DIVER framework: set-level diversity optimization lifts ad metrics"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's results rest on the assumption that the five automated reward scores—especially the click-through model trained on the best and worst supervised-fine-tuning outputs—are faithful proxies for what makes an ad headline valuable; if that model is biased, the measured quality and the online gains could come from the serving-time user-profile matching rather than from the generation training.","fun_headline_variants_meta":{"raw":{"variants":["Diverse ad headlines via LLMs lift value 4% and CTR 1.4%","Unlock diversity in ad headlines: 4% value, 1.4% CTR","LLM multi-objective training: diverse headlines boost CTR and value","Diversity-aware ad headline generation: 4% ADVV lift, 1.4% CTR","DIVER framework: set-level diversity optimization lifts ad metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000599,"raw_usage":{"total_tokens":2624,"prompt_tokens":720,"completion_tokens":1904,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1797}},"tokens_in":464,"tokens_out":1904,"duration_ms":14607,"temperature":1.0,"reasoning_tokens":1797,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:15:02.890935+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An A/B test that keeps the serving-time user-profile selection step but replaces the reinforcement-learning-trained generator with the supervised-fine-tuning-only checkpoint from the paper's ablations: if the 4.0% advertiser-value and 1.4% click-through-rate gains disappear, the reinforcement-learning stage is the cause; if they persist, the gains come from the synthetic data or the selector.","supporting_citations":[{"cited_title":"Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning","cited_arxiv_id":"2412.03343","evidence_quote":"Supplies the possibility-exploration fine-tuning baseline and frames diversity as a training objective that the paper extends with multi-objective reinforcement learning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Self-BLEU, one of the diversity metrics used to compare against baselines."}],"review_version":1}