{"id":"caaa06e6-bb1f-4b79-8d59-b277025286fe","arxiv_id":"2501.11900","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SCAPE learns users' stylistic and content preferences from click history via LLM-inferred headline attributes, blends long- and short-term signals, and generates personalized headlines that beat prior models on PENS.","lead":"SCAPE is a headline-writing system that personalizes both the topic and the writing style of news headlines, using a large language model to infer style and content tastes from each reader's click history. It reports the best results so far on the PENS benchmark, though its style personalization is judged partly by the same model that builds the user profiles.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is unsupported as reported: the KDD 2024 baseline [16] is cited but absent from Table 1, so 'new benchmark' has not been established.","rationale":"The reader's weakest_assumption focuses on unvalidated LLM-inferred style/content labels, which is a real threat to the interpretation of the mechanism. However, the most load-bearing issue for the paper's headline claim is the missing KDD 2024 baseline: 'SOTA' is an empirical comparative claim, and the comparison table is incomplete. The reader did flag this in the rationale as a condition, but did not put it in weakest_assumption. I therefore say partial rather than agree. The verdict should remain CONDITIONAL because the missing comparison and judge-consistency issue do not by themselves disprove the method; they make the SOTA claim unverified. Adding the KDD 2024 baseline and an independent judge would settle it. I did not choose REJECT because the reported ROUGE gains are systematic across all metrics and the framework is coherent; the concern is about evidence completeness, not internal inconsistency.","tokens_in":7451,"tokens_out":5971,"duration_ms":61728,"concrete_test":"Obtain the Tan et al. (KDD 2024) system or its published PENS results and evaluate it under the exact protocol of Table 1 (same test split, ROUGE-1/2/L, Fact Score). If any ROUGE score meets or exceeds SCAPE's 34.26/14.79/28.36, the 'new SOTA benchmark' claim fails; if all are lower, the omitted-baseline concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 asserts that SCAPE 'sets a new benchmark for personalized headline generation in SOTA results.' The only way to support that claim is a complete comparison with the strongest existing methods. Reference [16] (Tan et al., KDD 2024, 'Enhancing Personalized Headline Generation via Offline Goal-Conditioned Reinforcement Learning with Large Language Models') is cited in the introduction as a personalized headline generation method but never appears in Table 1. The reported margins over the best included baseline (GTP) are small: ROUGE-1 34.26 vs 33.84, ROUGE-2 14.79 vs 14.23, ROUGE-L 28.36 vs 27.85. An omitted strong baseline could plausibly invert these differences. This is a completeness problem for the central claim, independent of the unvalidated LLM-inferred labels: even if the style-content mechanism is exactly as described, the paper has not shown that it is SOTA. The personalization judge (Qwen2.5-72B-Instruct) is also the same model family used to infer R_s and R_c, so the win-rate evidence in Figure 3 is at least partially a measure of self-consistency with the attribute extractor rather than alignment with human users.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCAPE, a framework for personalized news headline generation that jointly models content interests and stylistic preferences. It uses an instruction-tuned LLM to infer style and content attributes for each headline (Eq. 1), encodes them offline with an embedding-based LLM into a headline embedding table (Eqs. 2–3), and builds user representations by hierarchically fusing long- and short-term content and style embeddings with gating and attention. A contrastive disentanglement loss (Eq. 10) separates the four user representations, and the fused user vector is added to the decoder input embeddings of a FlanT5-base generator (Eq. 9). Experiments on PENS compare SCAPE with open-source LLMs, API-based LLMs, and prior personalized models. The paper reports higher ROUGE-1/2/L and Fact Scores than all listed baselines, with significance at p≤0.05, and presents LLM-judged personalization win rates in Figure 3. The central claim is that SCAPE sets a new state of the art for personalized headline generation.","tokens_in":7732,"tokens_out":2821,"duration_ms":31053,"significance":"If the empirical claims hold, the paper makes a useful contribution by moving personalized headline generation beyond content-only user modeling to include stylistic preferences, using LLM collaboration to obtain attribute labels without manual annotation. The hierarchical gated fusion and contrastive disentanglement of long/short-term content and style representations are reasonable architectural components, and the use of an offline embedding table is practical. The paper also has concrete strengths: it evaluates on the established PENS benchmark, compares against a broad set of LLM and personalized baselines, and reports Fact Scores in addition to ROUGE. However, the significance is currently undermined by evaluation gaps: the strongest directly relevant recent baseline, cited as [16], is absent from Table 1; the personalization judge is the same model family that produced the inferred attributes; and no variance or significance-test details are given for the small ROUGE margins. These issues are fixable, but they must be addressed before the SOTA claim is credible.","major_comments":[{"comment":"The claim that SCAPE 'sets a new benchmark for personalized headline generation in SOTA results' is not supported by the reported comparison because reference [16] (Tan et al., KDD 2024), which the introduction cites as a personalized headline generation method, is never included in Table 1. The margins over the strongest included baseline (GTP) are small: ROUGE-1 34.26 vs. 33.84, ROUGE-2 14.79 vs. 14.23, ROUGE-L 28.36 vs. 27.85. Without a comparison against [16], or a clear justification for its omission, the SOTA claim is incomplete. I ask the authors to add this baseline, or to qualify the claim to 'state of the art among the compared methods.'","section":"§3.2, Table 1"},{"comment":"The personalization evaluation uses Qwen2.5-72B-Instruct as the judge, and the same model family (Qwen2.5-72B-Instruct and GTE-Qwen2-7B-Instruct) is used in Section 2.2.1 to infer the style and content attributes R_s and R_c that shape the user representations. The reported win rates are therefore at least partly a measure of consistency between the judge and the attribute extractor, rather than alignment with real user preferences. I request an independent evaluation: human judgments on a sample, or a different judge model that was not involved in attribute inference, with agreement statistics between judges.","section":"§3.1.3, Figure 3"},{"comment":"Table 1 reports no variance, no number of runs or seeds, and no details on how the p≤0.05 significance level was computed. Given that the ROUGE improvements over GTP are under one point for all three ROUGE variants, these omissions are load-bearing for the superiority claim. Please provide means and standard deviations over multiple runs, state the test used (e.g., paired bootstrap or Wilcoxon), and report all compared systems under identical evaluation settings. Also, Fact Scores are listed as NA for EUI-PENS and GTP, so the claim about improved factual consistency over all baselines in Section 3.2 is not fully substantiated.","section":"§3.1.2, Table 1"},{"comment":"The prompts Pstyle and Pcontent that drive the LLM attribute inference are not reported, and the inferred attributes R_s and R_c are never validated against human judgments, clustering stability, or any external signal. Since the user fusion network and the final personalized injection both operate on these attributes, the framework's central mechanism rests on the unverified quality of these labels. Please release the prompts, show examples of inferred attributes, and provide a human agreement study or an ablation that replaces the LLM attribute extractor with an alternative (e.g., heuristic or supervised) extractor to demonstrate that the gains come from genuine style/content modeling rather than from properties of the particular LLM prompt.","section":"§2.2.1, Eq. (1)"}],"minor_comments":[{"comment":"There is a grammatical typo: 'Base on User A’s historical click patterns' should be 'Based on User A’s historical click patterns.'","section":"§1, Figure 1 caption and text"},{"comment":"The notation R_s and R_c is introduced as LLM responses, but the paper never states the output format (e.g., free text, JSON, or a fixed label set). Please specify the response schema, since it affects reproducibility of the embedding table.","section":"§2.2.1, Eq. (1)"},{"comment":"The implementation details list learning rates for 'subsequent steps' without specifying which module each learning rate applies to. Please clarify the correspondence between the three peak learning rates (1e-3, 1e-6, 1e-5) and the training stages or model components.","section":"§3.1.3"},{"comment":"The personalized injection is described as adding the user representation to every decoder token embedding, but the paper does not discuss whether this causes distribution shift at pre-training time or whether any normalization is applied. A sentence on how this interacts with the pre-trained FlanT5 decoder would improve clarity.","section":"§2.2.3, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short WWW Companion paper, and the core idea is timely and potentially useful. I do not believe rejection is warranted because the central mechanism is plausible and the reported improvements, while small, are consistent across metrics. However, the missing comparison with the cited KDD 2024 baseline and the use of the same LLM family for both attribute extraction and personalization judging are substantive evaluation gaps that currently prevent the SOTA claim. Both are addressable within the scope of a revision, so major revision is the appropriate recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SCAPE is a reasonable, well-scoped extension of personalized headline generation: it jointly models content interests and stylistic preferences from click histories, uses an LLM to infer unsupervised labels, and fuses long/short-term user representations with gating and contrastive loss. That combination is genuinely new in the PENS-based literature, and the architecture is clear. The reported ROUGE gains over FPG and GTP are consistent, and the Fact Score improvement is plausible.\n\nThe soft spots are in the evaluation. The SOTA claim in Section 3.2 is not supported as reported because Tan et al. (KDD 2024) [16] is cited in the introduction and then never appears in Table 1. If that method is a strong personalized baseline, omitting it inverts the meaning of \"new benchmark.\" The margins over GTP are small (ROUGE-1 34.26 vs 33.84), so this is not a minor omission. Second, the personalization win rates in Figure 3 are judged by Qwen2.5-72B-Instruct, the same model family used to infer the style/content attributes that shape user representations. That makes the win-rate evidence at least partly self-consistency, not user alignment. The ROUGE results are external and unaffected, but the personalization claim is the paper's reason to exist. Third, the prompts Pstyle and Pcontent are not reported, and the inferred labels are never validated against humans or clusters. That is an important assumption, but not a fatal one: the ROUGE gains could still hold even if the style labels are noisy, and the authors are honest about the lack of ground truth.\n\nMinor: no variance or significance test details beyond a p-value star, and the free parameters (margin, window K) are not analyzed. This is a 4-page companion paper, so some brevity is expected, but the missing baseline is not a brevity issue.\n\nWho gets value: researchers working on personalized text generation or news headline generation. It is a step forward, not a breakthrough. The paper deserves a serious referee; I would send it back with requests to add [16], release the prompts, and replace the same-model judge with an independent judge or human ratings. With those changes it could be a solid workshop-level contribution.","headline":"A plausible style-content personalization framework whose SOTA claim rests on an incomplete baseline table and a self-referential LLM judge.","tokens_in":8241,"tokens_out":2101,"would_cite":true,"duration_ms":21597,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCAPE shows that tailoring headlines to a reader's stylistic taste as well as content interest beats content-only personalization on the PENS benchmark.","keywords":["personalized headline generation","stylistic preferences","content interests","user preference modeling","large language models","contrastive learning","hierarchical fusion","PENS benchmark"],"falsifier":"If replacing the style embedding channel with random vectors leaves ROUGE, fact, and personalization win scores essentially unchanged, the reported advantage does not come from stylistic modeling.","tokens_in":1433,"feed_emoji":"📰","tokens_out":2102,"duration_ms":66698,"temperature":0.7,"pith_summary":"Personalized headline generation has mostly asked what a reader likes to read about, while ignoring how a reader likes a headline to sound. The paper proposes SCAPE, which has a large language model infer a style attribute and a content attribute for every headline, stores both as embeddings, and fuses a user's long- and short-term preferences on each dimension through gated attention and contrastive disentanglement. The fused user vector is added to every decoder token of a FlanT5 generator so the output can reflect both dimensions at once. On the PENS dataset, SCAPE reports higher ROUGE and fact scores than content-only personalized baselines and wins more pairwise comparisons when an LLM judge compares its headlines with the originals. The intended upshot is that panoramic user modeling, style plus content, is the right unit for personalized headline generation.","feed_headline":"Style-plus-content headlines beat content-only personalization","feed_subtitle":"Modeling stylistic taste plus content interest beats content-only personalization on PENS.","key_machinery":"The load-bearing components are (1) the LLM-based headline inference module that produces style and content responses $R_s$ and $R_c$ via prompts $P_{style}$ and $P_{content}$ and encodes them into style/content embeddings $E_s$ and $E_c$; (2) a hierarchical gated fusion network that computes long- and short-term content and style representations, gates them with $\\alpha$ and $\\beta$, and attends to the candidate article; and (3) a personalized injection module that adds the fused user representation to every token embedding of the FlanT5 decoder. A contrastive triplet loss, with long-/short-term proxies as positives and opposite-type proxies as negatives, keeps the four user representations disentangled so that style and content do not collapse into a single signal.","core_discovery":"SCAPE's central claim is that jointly modeling content interests and stylistic preferences in a user's historical clicks improves personalized headline generation beyond content-only user modeling. It gains this by having an LLM infer a style attribute and a content attribute for each headline, embedding both through instruction-following embeddings, and then combining the user's long- and short-term representations on each dimension with gated fusion and contrastive disentanglement. The style-content user vector is injected into the decoder's token embeddings. Experiments on PENS report state-of-the-art ROUGE-1/2/L and fact scores, and pairwise LLM-judged personalization wins over existing personalized baselines such as FPG and GTP.","pith_inferences":["If the inferred style labels are noisy or miss dimensions users actually care about, the reported gains could come from the content pipeline or the fusion architecture rather than from genuine style modeling; an ablation that replaces inferred styles with content features would test this.","The same two-prompt inference and embedding table recipe could be applied to other personalized text generation tasks, such as email subject lines, ad copy, or social posts, where style preferences are user-specific.","Because the paper does not validate the inferred attributes against human judgments, a testable extension is to collect human style labels for a sample of headlines and measure agreement with the LLM responses.","The pairwise LLM judge may encode its own stylistic biases; a human evaluation on a subset would check whether the judged personalization advantage holds outside the judge model."],"forward_implications":["A headline generator can personalize along stylistic dimensions such as interrogative phrasing, numbered lists, and exclamatory tone without resorting to a single global clickbait style.","Fact preservation and personalization can move together: SCAPE reports the highest fact score and the highest personalization win rate among the compared methods.","LLM-inferred headline attributes provide a workable weak-supervision signal for style and content, removing the need for manually annotated labels.","The token-level injection of the fused user vector makes user conditioning cheap: it only adds a vector to the existing decoder embeddings of a fixed backbone model."],"supporting_citations":[{"why":"Supplies the PENS dataset and the personalized headline generation task, including the gold-standard parallel headlines used for evaluation.","marker":"[2]"},{"why":"Provides the instruction-following text embedding method used to encode each headline together with inferred style and content attributes.","marker":"[13]"},{"why":"Supplies the long- and short-term interest disentanglement strategy that SCAPE adapts through contrastive triplet loss.","marker":"[22]"},{"why":"Defines the fact-preserved personalized headline generation baseline FPG and the fact-score metric used in the evaluation.","marker":"[19]"},{"why":"Provides the general-then-personal decoupling and pretraining approach for personalized headline generation that SCAPE compares against.","marker":"[15]"},{"why":"Establishes the personalized headline generation setup and the PENS-NRMS/NAML-style baselines that SCAPE must beat.","marker":"[1]"}],"fun_headline_variants":["Style and content beat content-only in personalized headlines","Personalized headlines get a style upgrade with SCAPE","Panoramic interests: style plus content wins for headline personalization","LLM-infused style and content fusion improves personalized headlines","Headlines tailored to your style and taste, not just content"],"cache_read_input_tokens":10368,"weakest_assumption_plain":"The framework's gains rest on the assumption that the LLM's inferred style and content labels, produced from hand-written prompts that are never shown, faithfully capture the stylistic and content dimensions that real readers care about.","fun_headline_variants_meta":{"raw":{"variants":["Style and content beat content-only in personalized headlines","Personalized headlines get a style upgrade with SCAPE","Panoramic interests: style plus content wins for headline personalization","LLM-infused style and content fusion improves personalized headlines","Headlines tailored to your style and taste, not just content"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2383,"prompt_tokens":815,"completion_tokens":1568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":1487}},"tokens_in":431,"tokens_out":1568,"duration_ms":10827,"temperature":1.0,"reasoning_tokens":1487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:44:35.479583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If replacing the style embedding channel with random vectors leaves ROUGE, fact, and personalization win scores essentially unchanged, the reported advantage does not come from stylistic modeling.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PENS dataset and the personalized headline generation task, including the gold-standard parallel headlines used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the instruction-following text embedding method used to encode each headline together with inferred style and content attributes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the long- and short-term interest disentanglement strategy that SCAPE adapts through contrastive triplet loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the fact-preserved personalized headline generation baseline FPG and the fact-score metric used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the general-then-personal decoupling and pretraining approach for personalized headline generation that SCAPE compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the personalized headline generation setup and the PENS-NRMS/NAML-style baselines that SCAPE must beat."}],"review_version":1}