{"id":"689f3044-7bee-47d6-8c88-36d1d906acca","arxiv_id":"1908.08936","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A fatigue-aware logistic-regression bandit that scores ad creatives by similarity to a user's past exposures improved CTR and post-impression CVR over baseline in a production A/B test.","lead":"This paper adds an advertising-fatigue score to a display-ad selection model and tests it in a real advertising system. In a one-week production A/B test, the fatigue-aware algorithm beat a baseline on click-through rate and post-impression conversions, though only some gaps were statistically significant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fatigue metric's hand-weighted text:image similarity is unvalidated, and the central claim lacks sensitivity analysis showing the result is robust to the arbitrary 3:1 weight.","rationale":"The central claim is that adding the fatigue predictor of Eq. (2) to a contextual Thompson-sampling bandit yields measurable gains in a production display-ad deployment. The only difference between FA and Baseline is the inclusion of kappa_i,t(at) and its square. Therefore, the validity of kappa is the load-bearing assumption: if the similarity-weighted history does not reflect fatigue, the algorithm's behavior reduces to an arbitrary creative-rotation heuristic, and the paper's 'fatigue-aware' interpretation collapses. Section 4.2 fixes the text:image weight at 3:1 with only an interpretability heuristic, and no sensitivity analysis is reported. Figures 9 and 10 cannot validate kappa because they measure 'fatigue' with the same kappa. The reader's weakest assumption points to this exact issue. A secondary concern is that Table 1 shows the CVR difference (FA 1.12 vs Baseline 1.10) without a significance star, so the abstract's 'superiority' overstates the evidence on that metric; the required confidence intervals are absent. However, the post-impression CVR difference is significant at p<0.01, so the primary finding is not purely a statistical artifact. Because the paper already earns conditional acceptance and the missing sensitivity analysis is a reasonable condition, the verdict remains conditional.","tokens_in":10362,"tokens_out":12492,"duration_ms":121310,"concrete_test":"Run a follow-up production A/B test in which the text:image similarity weight is set to 1:0 and 0:1 (and ideally 1:1) while all other settings remain unchanged, then compare the FA-versus-Baseline differences in CTR and post-impression CVR. If the FA advantage disappears or reverses under either alternative weight, the manually chosen 3:1 weight is a load-bearing assumption and the original result is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The fatigue metric kappa_i,t(at) = h_i,t · s(at) (Section 4.1, Eq. 3) is the only component that distinguishes the proposed algorithm from the baseline. The similarity vector s(at) is formed as a weighted average of BoW text similarity and MobileNetV2 image similarity, with the text weight fixed at 3 (Section 4.2). This weight is justified by interpretability, not by evidence, and no sensitivity analysis is provided. Because the post-hoc analyses in Figures 9 and 10 use the same kappa to define fatigue, they cannot independently validate the metric. If the 3:1 choice is arbitrary and the algorithm's advantage depends on it, the reported improvement may be an artifact of the similarity scoring rather than a fatigue effect. The central claim that Eq. (2) captures fatigue therefore rests on this unvalidated assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fatigue-aware ad creative selection algorithm for display advertising. The reward predictor in Eq. (2) augments a contextual logistic model with a quadratic term in a fatigue metric κ_i,t(a_t) = h_i,t · s(a_t), where h_i,t is the user's exposure history and s(a_t) is a similarity vector between the candidate creative and past creatives, computed as a weighted combination of text BoW and image MobileNetV2 cosine similarities (Section 4). The algorithm (Algorithm 2) embeds this predictor in a Thompson-sampling bandit. The paper reports a one-week production A/B test across three campaigns (Section 5) in which the fatigue-aware algorithm (FA) numerically outperforms a contextual-bandit baseline and random selection on CTR, overall CVR, and post-impression CVR, although only some differences are statistically significant. Post-hoc analyses in Section 6 examine the relationship between the fatigue measure and CTR/CVR.","tokens_in":10548,"tokens_out":4072,"duration_ms":35236,"significance":"If the result holds, the paper provides a simple, low-latency method for incorporating advertising fatigue into ad creative selection, with a real production deployment and realistic constraints. The derivation of the fatigue feature is non-circular because κ is constructed from exposure histories and precomputed similarities, not from the outcome. The production A/B test is a genuine strength, as are the practical details of implementation. However, the statistical support is partial, and the fatigue metric relies on an unvalidated similarity weighting, so the claim of demonstrated superiority needs further support before it can be accepted as stated.","major_comments":[{"comment":"The statistical evidence for the central claim is weaker than the text suggests. Only CTR (1.08, P<0.1) and post-impression CVR (1.09, P<0.01) are marked significant for FA versus Baseline; overall CVR (1.12) is not significant, and post-click CVR numerically favors Baseline (1.05 vs 1.04). The paper should report confidence intervals, describe the testing procedure, and address multiple comparisons across the four metrics and three campaigns.","section":"§5.2, Table 1"},{"comment":"The similarity score s(a,a') is a weighted average with text similarity given three times the weight of image similarity, justified solely by interpretability. Because κ is the only component that distinguishes FA from Baseline, the absence of any sensitivity analysis for this weight means the reported improvement could be an artifact of the chosen similarity scoring rather than a fatigue effect. The post-hoc validation in Section 6 uses the same κ to define fatigue and therefore cannot independently confirm the metric.","section":"§4.2, Eq. (3)"},{"comment":"The experiment duration is one week and the significance calculation is not described. The per-campaign CTRs in Table 2 vary widely (e.g., FA 1.21 and Baseline 0.95 for campaign A), and the paper notes campaign A has the least data; without confidence intervals or per-campaign significance tests, the stability claim for FA is not supported.","section":"§5.1 and Table 2"}],"minor_comments":[{"comment":"The caption reads 'Users jump to login screen of the game when click on these ads'; it should be 'when they click on these ads'.","section":"§1, Figure 1 caption"},{"comment":"The footnote says four campaigns were used but one had only one creative; the results in Table 2 cover three campaigns. Please clarify the discrepancy.","section":"§5.1, footnote 2"},{"comment":"The sentence 'The results clearly show that the proposed algorithm successfully increased both clicks and conversions' overstates the significance reported in Table 1.","section":"§5.2"},{"comment":"Figure 10 uses data from the random algorithm only; the main text should state this explicitly rather than leaving it to the caption.","section":"§6, Figure 10"},{"comment":"Several references contain garbled author names: [16] lists 'Clammer Koby' and 'Shie Monner' (should be Koby Crammer and Shie Mannor), and [20] appears malformed. Please correct the reference list.","section":"References"},{"comment":"There are typos: 'SGDCliassifier' and 'skleran' should be 'SGDClassifier' and 'scikit-learn', respectively; 'metrices' in Section 6 should be 'metrics'.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a useful industry case study with a real production experiment, but the statistical reporting and sensitivity analysis are not yet at the level needed to support the paper's strong claims. The authors should be asked to provide confidence intervals, multiple-testing considerations, and a sensitivity analysis of the similarity weighting. The reference list also needs correction. I recommend major revision rather than rejection, because the central idea is sound and the issues are addressable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a genuine production A/B test, not a simulation, and the central direction is plausible. The fatigue-aware algorithm (FA) adds a similarity-weighted exposure metric and its square to a logistic CTR predictor, then selects via Thompson sampling. It beat a well-tuned baseline on CTR (P<0.1) and post-impression CVR (P<0.01), and did not lose on overall CVR. That is a useful data point for the subfield.\n\nWhat is actually new: fatigue modeling already exists in news recommendation (Agarwal, Ma) and past-exposure features exist in display targeting (Gupta), but I don't know of a published combination of a similarity-weighted fatigue score over text and image, quadratic fatigue terms, and a live deployment on ad creatives. The paper also gives a clear system description, including how histories are capped (one impression per minute, 24-hour window) and how the similarity is computed offline. The post-hoc analyses—the algorithm lowers accumulated fatigue relative to baseline, and CTR declines with fatigue in random logs—support the mechanism, though they use the same kappa, so they are not an independent validation.\n\nThe soft spots are real but not fatal. First, the fatigue metric relies on a hand-picked 3:1 weight for text over image similarity, justified only by interpretability. Since kappa is the only feature distinguishing FA from baseline, the result could depend on that arbitrary choice. There is no sensitivity analysis. Second, the 'superiority' claim outruns the statistics: only one headline metric is significant at P<0.1, overall CVR is not significant, and post-click CVR favors baseline. No confidence intervals, no correction for multiple metrics, and per-campaign results are noisy. Third, no code or data are released, which limits reproducibility but is common in industry work.\n\nNone of this changes my read that the paper deserves a serious referee. The production setting is a genuine strength, and the weakness in the fatigue metric can be addressed with sensitivity analysis rather than rejection. I would send it to review with the request that the authors report confidence intervals, run the 3:1 weight robustness check, and reign in the language.\n\nFor you: worth reading if you work on ad creative selection or fatigue in recommenders; I'd bring it to a reading group only if the group cares about applied bandits. I'd cite it for the deployment result, with a caveat about the unvalidated similarity weight.","headline":"A genuine production A/B test of a fatigue-aware ad creative selector, with a plausible but modest effect and an under-validated similarity metric.","tokens_in":11062,"tokens_out":2513,"would_cite":true,"duration_ms":110387,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a similarity-weighted fatigue score to an ad-selection bandit raised click-through rate and conversions in a one-week production experiment.","keywords":["Display advertising","Advertising fatigue","Advertising repetition","Contextual bandit","Thompson sampling","Ad creative selection","Real-time bidding","Online experiment"],"falsifier":"Re-run the same one-week A/B test with the fatigue features included but with the history vector $h_{i,t}$ replaced by a scrambled or permuted version that destroys the true exposure signal while keeping the same distribution; if CTR and CVR gains persist, the improvement is not caused by the measured fatigue mechanism. A weaker and cheaper check is to print the fitted $b_1,b_2$ and the implied fatigue-response curve from the random-arm logs: the central claim requires the curve to be measurably negative in the high-fatigue region, not flat or positive.","tokens_in":10120,"feed_emoji":"📈","tokens_out":13593,"duration_ms":538185,"temperature":0.7,"pith_summary":"This paper sets out to show that online display ads can be chosen more effectively when the selection algorithm takes into account how tired a user has become of seeing an ad. It modifies an existing contextual-logistic selector—a bandit algorithm that balances trying different ads against showing the current best—by adding a fatigue score and its square as two extra features, where the fatigue score counts recent exposures to the same or similar creatives weighted by similarity. The modified algorithm ran for one week in a live production display-advertising system against the unmodified contextual bandit and against random selection. The paper reports that the fatigue-aware version outperformed both on click-through rate and on overall conversion rate, and that it served ads at lower measured fatigue levels. The practical point is that advertisers can counter ad wear-out with a small, low-latency change to the reward model rather than a new system.","feed_headline":"Fatigue-aware ad selection lifts clicks and conversions in live test","feed_subtitle":"A two-feature fatigue score added to a Thompson-sampling ad selector beat baseline and random in a one-week live test.","key_machinery":"The load-bearing object is the fatigue score $\\kappa_{i,t}(a_t)=h_{i,t}\\cdot s(a_t)$, where $h_{i,t}$ is the user's exposure-history vector and $s(a_t)$ is a vector of similarity scores between the candidate creative and each creative the user has recently seen. Each similarity score is a weighted average of bag-of-words text cosine similarity and image cosine similarity from a pre-trained image network, with text weighted three times higher. The score enters the logistic reward predictor through the quadratic terms $b_1\\kappa + b_2\\kappa^2$, so the fitted curve is allowed to be increasing, decreasing, or inverted-U rather than assuming the textbook shape. The model is retrained daily on the previous day's logs and used with Thompson sampling, and the similarity vectors are precomputed offline, leaving only a history lookup and an inner product per candidate at decision time.","core_discovery":"The central claim is that adding two fatigue features to the reward predictor—$b_1\\kappa_{i,t}(a_t)+b_2\\kappa_{i,t}(a_t)^2$ in the log-odds of a logistic model—produces better ad choices than either the same model without fatigue or random selection. In the one-week production A/B test, normalized click-through rate was 1.08 for the fatigue-aware algorithm versus 1.04 for the baseline (1.00 for random), and normalized post-impression conversion rate was 1.09 versus 1.04, with the CTR and post-impression CVR differences reaching statistical significance; on post-click CVR the baseline was numerically higher but the gap was not significant. The fatigue feature $\\kappa_{i,t}(a_t)$ is an inner product of the user's recent exposure-history vector with a vector of precomputed text-and-image similarity scores, which is why it can personalize without adding per-impression computation. The paper reads the result as operational evidence for the two-factor model of advertising repetition: initial repetition helps, fatigue eventually dominates, and a bandit that rotates creatives before that point can raise both clicks and conversions.","pith_inferences":["An untested extension: the 3:1 text-to-image weighting is a free parameter, and learning it from data could change the size of the gains.","The paper leaves these engineering choices untested: the 24-hour history window and one-impression-per-minute cap likely interact with the wear-out curve.","The same two-feature correction could apply to any recommender with pairwise-similar items and an exposure log, which would make the production result a proof of concept for content fatigue beyond ads.","The differing CTR and CVR fatigue curves visible in the paper's random-arm data suggest that optimizing clicks alone might over-rotate away from creatives that still convert."],"forward_implications":["Advertisers can add fatigue-awareness to an existing bandit by appending two features, with no change to the latency budget.","Creatives similar to what a user has already seen are served less often as exposure accumulates, reducing wear-out.","The fatigue feature can be tuned for different goals, since the observed relationship between fatigue, clicks, and conversions differs.","The daily retraining schedule means the method handles creatives that are added and removed without retraining the similarity network online.","The statistically significant production gains imply the fatigue score carries signal beyond the usual contextual features."],"supporting_citations":[{"why":"Supplies the Thompson-sampling update and posterior approximation used by both the baseline and the fatigue-aware selector.","marker":"[7]"},{"why":"Provides the contextual-bandit formulation and the replay-style tuning method used to set hyperparameters.","marker":"[18]"},{"why":"Prior work showing that repeated exposure to recommended items depresses CTR, motivating exposure-based features.","marker":"[2]"},{"why":"Prior user-fatigue model for news recommendation that uses past exposure and reactions as predictive features.","marker":"[19]"},{"why":"Meta-analysis of advertising repetition that is the empirical basis for the two-factor wear-in/wear-out curve.","marker":"[25]"},{"why":"Pre-trained image network whose pooled features define the image-similarity half of the fatigue score.","marker":"[23]"},{"why":"Mortal multi-armed bandit formulation that motivates how the algorithm handles creatives added and removed over time.","marker":"[6]"}],"fun_headline_variants":["Fatigue-aware bandit lifts ad CTR and conversions in production","Two fatigue features in ad selector beat baseline in live A/B","Psychology-informed ad rotation improves clicks and conversions","Fatigue-modeling ad bandit outperforms baseline in real test","Adding fatigue score to ad selection boosts performance live"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single number per user and candidate ad—a weighted sum of past exposures, with text similarity counted three times as heavily as image similarity—really captures how tired a user is of an ad, and that one curved relationship between that number and the chance of clicking or converting applies across all users and campaigns.","fun_headline_variants_meta":{"raw":{"variants":["Fatigue-aware bandit lifts ad CTR and conversions in production","Two fatigue features in ad selector beat baseline in live A/B","Psychology-informed ad rotation improves clicks and conversions","Fatigue-modeling ad bandit outperforms baseline in real test","Adding fatigue score to ad selection boosts performance live"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3131,"prompt_tokens":888,"completion_tokens":2243,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":2163}},"tokens_in":504,"tokens_out":2243,"duration_ms":154213,"temperature":1.0,"reasoning_tokens":2163,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:58:41.461241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same one-week A/B test with the fatigue features included but with the history vector $h_{i,t}$ replaced by a scrambled or permuted version that destroys the true exposure signal while keeping the same distribution; if CTR and CVR gains persist, the improvement is not caused by the measured fatigue mechanism. A weaker and cheaper check is to print the fitted $b_1,b_2$ and the implied fatigue-response curve from the random-arm logs: the central claim requires the curve to be measurably negative in the high-fatigue region, not flat or positive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Thompson-sampling update and posterior approximation used by both the baseline and the fatigue-aware selector."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior user-fatigue model for news recommendation that uses past exposure and reactions as predictive features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mortal multi-armed bandit formulation that motivates how the algorithm handles creatives added and removed over time."}],"review_version":1}