{"id":"25b5b3c9-1fa5-41af-a443-b09de22f6b32","arxiv_id":"2502.04674","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AdParaphrase provides 725 human-preference-annotated paraphrase pairs of Japanese ad texts and shows fluency, length, noun count, and bracket use correlate with attractiveness.","lead":"This paper introduces AdParaphrase, a public dataset of paired Japanese ad texts that share the same meaning but differ in wording, rated by human judges for attractiveness. It finds that people prefer ads with higher fluency, more characters, more nouns, and bracket symbols, and that feeding these tips to GPT-4 makes generated ads more attractive in human evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The feature-attractiveness analysis is run on a post-hoc 316-pair high-agreement subset; if agreement correlates with the tested features, the reported associations do not generalize to the full 725-pair dataset.","rationale":"The reader's weakest assumption is exactly the load-bearing point. The paper's central empirical contribution is the set of feature-preference associations in Section 4.1.3 and the generation prompt built on those findings. Those associations are estimated only on the 316 high-agreement pairs. Because the filter is defined by the outcome variable (judge agreement), it can induce selection bias or Simpson-type reversal when features correlate with agreement. For example, bracket use might make preferences more decisive by making the difference salient, rather than making the text genuinely more attractive; the chi-square test cannot distinguish 'features cause preference' from 'features cause agreement,' and the excluded 409 pairs are the only way to check. The multiple-comparison issue is real but secondary: PPL, length, nouns, and brackets would likely survive a Bonferroni threshold at the reported sample sizes, while noun phrases would not. The ad-generation experiment also rests on a single stochastic run with no significance test for the headline contrast, which is an additional limitation; however, if the feature findings are subset artifacts, the prompt's 'tips' lose their empirical basis regardless of generation performance. The proposed re-analysis on all 725 pairs with agreement-level adjustment directly tests the selection-bias account, and because the dataset is released publicly, the check is feasible. If the associations survive, the concern is resolved; if not, the main analysis needs revision, but the dataset itself remains a valuable resource.","tokens_in":16337,"tokens_out":6176,"duration_ms":67234,"concrete_test":"Recompute the Section 4.1 analysis on all 725 paraphrase pairs, defining the outcome as the text preferred by the majority of the 10 judges (or by the proportion of judges preferring each text), and include agreement level as a covariate in a paired logistic regression to test whether the high-agreement subset is representative. If PPL, character length, noun count, noun phrases, and bracket use no longer remain significant with multiple-comparison control, the reported effects are an artifact of selecting high-agreement pairs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.2 restricts the chi-square analysis to the 316 of 725 paraphrase pairs with >=7/10 judges agreeing on the more attractive ad (Section 3.3), and Table 3 then reports significant associations for PPL, character length, nouns, noun phrases, and brackets. This is a post-hoc selection on the outcome. Agreement level is not a neutral filter: if features such as brackets or length polarize judges, the probability of reaching 7/10 agreement is correlated with those same features, so the conditional association between feature and majority preference can appear or change even when no association exists in the full population. No analysis of the 409 excluded pairs, and no comparison of feature distributions across agreement levels, is provided; the representativeness of the high-agreement subset is therefore load-bearing for every feature conclusion. The 24 unadjusted chi-square tests are a secondary concern, but the subset selection is the primary threat: the claimed p<0.01 findings in Section 4.1.3 are not demonstrated for the dataset as a whole.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AdParaphrase, a Japanese ad-text paraphrase dataset with human attractiveness judgments. It constructs 725 paraphrase pairs from existing datasets and human/LLM-generated paraphrases, collects ten preference judgments per pair, analyzes 24 linguistic features via chi-square tests on the 316 pairs with at least 7/10 judge agreement, and reports that preferred texts exhibit lower perplexity, longer character counts, more nouns and noun phrases, and greater bracket use. It then incorporates these findings into GPT-4 prompts with positive and negative few-shot examples, reporting human-evaluated improvements in paraphrase identification and attractiveness, and includes an analysis of alignment with predicted CTR and a small online A/B test.","tokens_in":16419,"tokens_out":6167,"duration_ms":56495,"significance":"If the feature findings are valid, AdParaphrase is a valuable public resource: the paraphrase-pair design controls for semantic content, the preference judgments use ten judges per pair, and the dataset, prompts, and analysis code are publicly released on GitHub. The generation experiment is concrete and falsifiable, with GPT-4-fewshot-findings-both reaching 36.6% attractiveness versus 23.2% for human expert paraphrases. However, the central statistical claim is currently undermined by a post-hoc subset selection and by the lack of multiple-comparison correction and significance testing for key generation comparisons, so the findings should be treated as promising but not yet established.","major_comments":[{"comment":"The chi-square analysis is restricted to the 316 of 725 pairs in which at least 7 of 10 judges agreed on the more attractive text. Because this filter is applied after observing the preference outcome, it is not a neutral design choice: if a linguistic feature (e.g., bracket use or length) polarizes judges, the probability of reaching 7/10 agreement is correlated with that feature, so associations in the selected subset need not reflect the full population. The paper offers no comparison of feature distributions across agreement levels and no analysis of the 409 excluded pairs. Since every reported feature-preference conclusion (lower PPL, longer character counts, more nouns/noun phrases, more brackets) derives from this subset, the central claim of the abstract and §4.1.3 is not demonstrated for the dataset as a whole. Please repeat the analysis on all 725 pairs (e.g., using a mixed-effects logistic regression or a continuous measure of preference strength) or provide a formal argument that agreement is independent of the tested features.","section":"§4.1.2–§4.1.3, Table 3"},{"comment":"Twenty-four chi-square tests are reported without any multiple-comparison correction. With a threshold of p<0.01, roughly 0.24 false positives are expected, and several p-values are close to 0.01 (word-length p=0.02, kanji p=0.02). Many features are also correlated (e.g., character length, noun count, noun phrases, and perplexity), which further complicates interpretation. Please apply a multiple-comparison correction (e.g., Benjamini-Hochberg or Bonferroni) and report adjusted p-values, or explicitly justify why unadjusted tests are appropriate.","section":"Table 3"},{"comment":"The claim that findings 'significantly improved performance' is not supported by a significance test: the only Fisher's exact test reported compares GPT-4-fewshot-both with GPT-4-fewshot-pos, not the findings-versus-no-findings comparisons, and the experiment is a single run (Appendix F states 'We report the results of a single experiment'). The success rates in Table 4 are therefore point estimates with unknown variance, and the superiority over the HUMAN baseline (36.6% vs. 23.2%) is not shown to be statistically reliable. Please provide confidence intervals, repeated runs, or significance tests for the key comparisons, ideally including GPT-4-zeroshot-findings versus GPT-4-zeroshot and GPT-4-fewshot-findings-both versus HUMAN.","section":"§4.2.3, Table 4, Appendix F"},{"comment":"The generation experiment has a circularity component: the findings embedded in the prompt are learned from the same type of human-preference judgments used to evaluate the generated texts, and the concluding sentence of §4.2.3 states that the results 'suggest that the linguistic features identified through the linguistic feature analysis (§4.1) contribute to enhancing the attractiveness of ad texts.' This is an application rather than an independent confirmation of a causal effect. The paper should explicitly acknowledge this and soften the causal wording, presenting the generation results as demonstrating that prompt-engineering with these features can steer an LLM on the same evaluation metric.","section":"§4.2, Fig. 9, §4.2.3"}],"minor_comments":[{"comment":"The paper calls one source dataset 'Ad Similarity' and cites Zhang et al. (2024), but the reference list entry for that citation is titled 'AdTEC: A unified benchmark for evaluating text quality in search engine advertising.' Please clarify the relationship between Ad Similarity and AdTEC and ensure the citation name matches the dataset name.","section":"§2.1"},{"comment":"The sentence 'we randomized the order of ad text pairs before presenting them to the judges' is ambiguous: it should state whether the order of the two texts within each pair was randomized, which is what mitigates position bias.","section":"§3.2"},{"comment":"The x-axis is labeled 'maximum number of votes between ad text pair'; the text explains that zero means all judges skipped, but it does not explain how ties other than 'all skip' are represented (e.g., 5–5 splits). Please add a clarifying sentence.","section":"Figure 3"},{"comment":"The text reports average Jaccard similarities of 0.560 and 0.602 for positive and negative examples, while Table 5 reports Jaccard between input and generated text; make clear that these are different quantities and that the table reports generation similarity, not example similarity.","section":"§4.2.3"},{"comment":"The pCTR alignment rates (54.7% in the high-agreement group and 44.5% in the low-agreement group) are compared informally; since 54.7% is not far from chance, please add a statistical test (e.g., a binomial test) and note that pCTR is a model prediction rather than observed click data.","section":"§5.1, Table 7"},{"comment":"The statement 'We report the results of a single experiment' should appear in the main text or at least in the main experimental section, not only in the appendix, so that readers are not misled about the precision of Table 4.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The dataset and the controlled paraphrase-pair design are commendable and could constitute a useful contribution to the ad-text and paraphrase communities. The primary blocker is the post-hoc high-agreement subset used for the feature analysis; the authors should be encouraged to provide a full-pair analysis or a convincing justification that the subset is representative. The multiple-comparison and single-run issues are also fixable. The paper's fit with the journal is good if these statistical concerns are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is a real contribution and the paper is worth engaging, but the headline feature findings are not as solid as the abstract suggests. The core problem is the 316-pair subset.\n\nWhat's new: a public paraphrase-controlled preference dataset for Japanese ad text, with 725 paraphrase pairs and 10 judges per pair. That fills a real gap. The bracket finding is concrete and novel for ad text. The generation experiment is a reasonable application, and the best model beating human expert paraphrase (36.6% vs 23.2% attractive success) is interesting, though it's a single run and no significance test is reported for that headline contrast.\n\nStrengths: annotation procedures are documented carefully; 5 annotators for paraphrase identification, 10 judges for preference, dummy questions for quality control, and the data is released. That is reproducible work, and they deserve credit for it.\n\nSoft spots: the feature analysis in §4.1.2 uses only 316 of 725 pairs where at least 7 of 10 judges agreed. That is a post-hoc selection on the outcome. If agreement correlates with features—for example, if brackets polarize judges or longer texts make preferences clearer—then the chi-square associations in Table 3 are conditional and need not generalize. The paper gives no analysis of the 409 excluded pairs and no comparison of feature distributions across agreement levels. The stress-test note is right: this is load-bearing.\n\nSecond, 24 chi-square tests are run with no multiple-comparison correction. With p<0.01, you'd expect about 0.24 false positives by chance, which is not huge, but combined with the subset issue the five 'significant' features could partly reflect selection. The noun and bracket findings might well hold, but the paper doesn't demonstrate that.\n\nThird, the generation experiment is a single run—Appendix F says so explicitly. Fisher's exact test is used only for some contrasts, not for the headline comparison of the best model against the human expert baseline. The numerical improvement is large, but without a significance test or multiple seeds, treat it as suggestive.\n\nThe online A/B test also improved only one of three ad groups, and pCTR alignment is only about 55% even in the high-agreement group, so the practical upside is limited. The authors acknowledge some of this.\n\nWho it's for: researchers working on ad text generation, evaluation, or Japanese ad linguistics. It's a useful dataset paper. The statistical analysis needs a rework, but the resource itself is solid.\n\nRecommendation: send to peer review. Require a re-analysis on the full dataset or a pre-registered subset, multiple-comparison control, and significance testing for the generation claims. Not a desk reject.","headline":"Useful new resource, but the feature-attractiveness findings rest on a post-hoc high-agreement subset and unadjusted chi-squares; the dataset deserves review, the statistical claims need rework.","tokens_in":17056,"tokens_out":1578,"would_cite":true,"duration_ms":15089,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Paraphrased ads are judged more attractive when they are more fluent, longer, more noun-heavy, and use bracket symbols.","keywords":["ad text generation","paraphrase dataset","human preferences","linguistic features","few-shot in-context learning","Japanese ad texts","attractiveness evaluation","chi-square analysis"],"falsifier":"Re-run the feature-versus-preference comparison on all 725 pairs using the continuous share of judges preferring each text instead of the majority label on only the 316 high-agreement pairs; if lower perplexity, longer length, noun density, and bracket use no longer predict preference, the central finding fails.","tokens_in":16032,"feed_emoji":"📣","tokens_out":8573,"duration_ms":71054,"temperature":0.7,"pith_summary":"This paper introduces AdParaphrase, a dataset of 725 pairs of Japanese ad texts that mean the same thing but differ in wording and style, each pair judged for attractiveness by ten people. It uses these matched pairs to isolate style from content, then tests 24 linguistic features against majority preference. The paper's central finding is that the ad text most people prefer tends to be more fluent (lower perplexity), longer in characters, richer in nouns and noun phrases, and likelier to contain Japanese bracket symbols, with all four associations significant at $p<0.01$. It then turns the finding into a prompt for GPT-4 that includes positive and negative before/after examples; that prompted model reaches a 36.6% attractive-success rate and 30.8% overall success in human evaluation, above the 23.2% attractiveness of paraphrases written by human advertising experts. The broader point is that content-independent stylistic preferences in ads can be measured with paired data and then used to make machine rewriting attractive.","feed_headline":"Fluent, longer, bracketed ads are judged more attractive","feed_subtitle":"A 725-pair preference dataset isolates why rewrites persuade; a GPT-4 prompt using those traits beats human rewrites.","key_machinery":"The central mechanism is the paired-paraphrase design: collecting ad-text pairs that are semantically equivalent, so that any systematic preference between the two members can only be attributed to wording and style. On those pairs, a chi-square test of independence tests whether the text preferred by the majority of judges is also the one scoring higher on each of 24 linguistic features. The generation side uses in-context learning: a GPT-4 prompt that states the four findings as 'tips' and supplies both positive demonstrations (where a rewrite was preferred) and negative demonstrations (where judges did not converge), which teaches the model both what attractiveness looks like and how aggressively to edit.","core_discovery":"On its own terms, the paper claims that human attractiveness judgments for paraphrased ad texts are not arbitrary: in the 316 of 725 pairs where at least seven of ten judges agreed, the majority-chosen text has significantly lower perplexity, significantly more characters, significantly more nouns and noun phrases, and significantly more bracket usage (all $p<0.01$ by chi-square test). The paper treats these as the linguistic signature of an attractive Japanese ad text. It further claims that encoding this signature into a GPT-4 instruction prompt, together with twenty positive examples (input less-preferred to output more-preferred) and twenty negative examples (pairs without consensus), yields ad rewrites that human judges find attractive 36.6% of the time and paraphrase-acceptable-plus-attractive 30.8% of the time, outperforming zero-shot GPT-4, few-shot variants without the findings, and human expert paraphrases (23.2% attractive).","pith_inferences":["The contrastive paraphrase design could be exported to other content types, such as product descriptions, headlines, or notifications, to separate style-driven preference from content-driven preference in the same way.","The bracket effect is plausibly tied to the visual landscape of Japanese search ads and the 15-character headline limit, so the same feature may not transfer to banner ads, long-form copy, or other languages; testing this would require a multilingual paired-preference dataset.","The 316-pair subset masks disagreement: if long or bracketed texts polarize judges, the feature signature could partly describe texts that produce consensus rather than texts that are broadly attractive.","A direct online A/B test with more ad groups would tell whether the best prompted model reliably lifts clicks and impressions; the paper's own three-group test improved only one of three ad groups."],"forward_implications":["Ad copywriters can act on four concrete style levers: write closer to natural fluency, use the full character budget, favor nouns and noun phrases, and mark key phrases with brackets.","LLM-based ad rewriting can exceed the attractiveness of human-expert paraphrases when prompted with preference-derived findings plus positive and negative examples, at least for Japanese search ads.","Ad texts preferred by seven or more judges align with predicted CTR in 54.7% of cases, so style improvements that win human votes can translate into modest click-performance gains.","The public dataset lets other researchers replay the preference analysis and test new features or rewriting strategies without access to proprietary ad log data.","Negative few-shot examples, drawn from pairs without clear consensus, appear to calibrate the amount of editing a model performs; models given only positive examples made smaller lexical changes and improved less."],"supporting_citations":[{"why":"Supplies Ad Similarity, the source of 706 candidate paraphrase pairs from ads rated as similar.","marker":"Zhang et al., 2024"},{"why":"Supplies CAMERA, the benchmark whose dev ads were paraphrased by humans and LLMs, and supports pCTR as a proxy for human evaluation.","marker":"Mita et al., 2024"},{"why":"Provides GPT-4, the model used both to create paraphrase candidates and to generate attractive rewrites in the in-context learning experiments.","marker":"OpenAI, 2024"},{"why":"Establishes few-shot in-context learning, the paradigm the generation prompts rely on.","marker":"Brown et al., 2020"},{"why":"Supplies GPT-2, the language model used to compute the perplexity feature that is one of the four significant preference predictors.","marker":"Radford et al., 2019"},{"why":"Supplies the Balanced Corpus of Contemporary Written Japanese used to compute word-frequency lexical-choice features.","marker":"Maekawa et al., 2010"},{"why":"Provides the attractiveness evaluation guidelines and criteria given to crowdsourced judges.","marker":"Wang et al., 2021"},{"why":"Provides the kappa agreement thresholds used to designate the 316 high-agreement pairs as fair agreement and thus analyzable.","marker":"Landis and Koch, 1977"},{"why":"Supplies the pCTR prediction approach used to test alignment between human preference and predicted clicks.","marker":"Hughes et al., 2019"},{"why":"Supplies Llama 2, the open-weight baseline used in both paraphrase-candidate generation and ad-text generation comparisons.","marker":"Touvron et al., 2023"}],"fun_headline_variants":["Fluent, long, bracketed ads win human preference","AdParaphrase dataset pinpoints attractive wording","Brackets and nouns boost ad appeal, study finds","GPT-4 rewrites ads using data-backed style traits","Linguistic recipe for attractive ads uncovered"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes the 316 pairs where at least seven judges agreed are representative, so the style preferences found on that subset would also hold for the 409 pairs where judges disagreed, and features like length or brackets do not themselves influence how much judges agree.","fun_headline_variants_meta":{"raw":{"variants":["Fluent, long, bracketed ads win human preference","AdParaphrase dataset pinpoints attractive wording","Brackets and nouns boost ad appeal, study finds","GPT-4 rewrites ads using data-backed style traits","Linguistic recipe for attractive ads uncovered"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1334,"prompt_tokens":965,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":581,"tokens_out":369,"duration_ms":23528,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:55:11.943221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the feature-versus-preference comparison on all 725 pairs using the continuous share of judges preferring each text instead of the majority label on only the 316 high-agreement pairs; if lower perplexity, longer length, noun density, and bracket use no longer predict preference, the central finding fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes few-shot in-context learning, the paradigm the generation prompts rely on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GPT-2, the language model used to compute the perplexity feature that is one of the four significant preference predictors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the attractiveness evaluation guidelines and criteria given to crowdsourced judges."}],"review_version":1}