{"id":"ea074296-dfbe-4142-99a3-9054fa64bbf6","arxiv_id":"2506.16429","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An agentic marketing system combining Difference-in-Differences and Thompson sampling increased engagement and conversion in a 6.4-million-user field test, but the treatment was confounded with extra message volume.","lead":"This paper describes an AI system that decides when and how to send marketing messages to each user by combining causal effect estimates with a bandit algorithm, and reports a 6.4-million-user field test. A smart generalist should read it because it is a rare public account of a deployed personalization system at 150-million-user scale, even though the experiment's design has a confound.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing volume-matched control confounds personalization with extra message volume; Table 1's GMV CI for PF3 also contradicts the claim of significance across all use-cases.","rationale":"I read the paper as an applied systems report whose central empirical claim is that agentic personalization improves upper- and bottom-funnel metrics. For that claim to hold, the randomized contrast in Table 1 must isolate the personalization mechanism from the fact that treatment users received extra messages. The paper's own description shows it does not: the treatment group received agentically orchestrated messages layered on top of the common baseline, with no volume-matched or placebo control. This is exactly the reader's weakest assumption, and it is load-bearing because any positive lift could be explained by additional frequency or by any message at all, not by the agentic selection. The concrete test above is the minimum design needed to separate these explanations. I also note the internal inconsistency between the text's claim of significance across all use-cases and the PF3 GMV interval containing zero, but that is secondary. The reader's REJECT verdict follows from the same volume-confound gap, so I recommend no change to the verdict.","tokens_in":6438,"tokens_out":3858,"duration_ms":42462,"concrete_test":"Run a follow-up three-week experiment with four arms: (i) baseline rules-based messages only; (ii) baseline plus the same number and channel mix of non-personalized control messages selected randomly from the same modular action catalogue; (iii) baseline plus agentically personalized messages; and (iv) arm (iii) with the personalized timing/copy randomly scrambled. Compare intent, conversion, and GMV with 99% confidence intervals using the same DiD/PSM procedure. If arms (ii) and (iv) reproduce the lifts of arm (iii), the effect is message volume or mere message presence, not agentic personalization; if arm (iii) dominates both, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The experiment cannot separate personalization from additional message volume. Section 3 states treatment messages were \"layered on top of baseline rules-based messages delivered to both control and treatment groups,\" and the control group received no equivalent extra or placebo messages. The treated arm therefore received both (a) agentic selection of timing, copy, and content, and (b) additional messages that the control arm did not receive. Equation (2)'s DiD contrast ΔY_T − ΔY_C estimates the total effect of the package {agentic selection, extra sends}, not the incremental effect of personalization. Sustaining the central claim requires the counterfactual in which users receive the same number and type of extra messages without agentic selection. The absence of a volume-matched control breaks the unconfoundedness/parallel-trends assumption, so the confidence intervals in Table 1 cannot be attributed to personalization. A secondary inconsistency: the text claims \"significant improvements across both upper- and bottom-funnel events, across all considered use-cases,\" but Table 1 reports a 99% GMV CI for Transactional PF3 of [–0.31%, +22.55%], which includes zero and is therefore not significant at the stated level.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an agentic framework for cross-channel marketing personalisation, in which a Thompson-sampling bandit selects modular actions (timing, channel, message attributes) for individual users, and a difference-in-differences estimator with propensity-score matching estimates individual treatment effects on a weighted event-stream outcome. The authors report a three-week field experiment across 6.4 million users for four product features, with 99% confidence intervals for intent, conversion, and GMV improvements, and state that the system is deployed to 150 million users. The central empirical claim is that agentically orchestrated messaging produces significant incremental engagement and revenue compared with rules-based baseline messaging.","tokens_in":6708,"tokens_out":3829,"duration_ms":37640,"significance":"If the causal attribution were sound, this would be an important large-scale demonstration that automated, modular, bandit-based personalisation can improve marketing funnel outcomes in a deployed consumer application. The paper has real strengths: it describes a deployed system, reports a genuine field experiment at unusual scale, presents confidence intervals rather than bare point estimates, and adopts a modular action-space design that is operationally sensible. However, the experiment as described cannot support the paper's central claim, because the treatment group received additional agentic messages that the control group did not receive, and the text's claim of significance across all use-cases is contradicted by the reported GMV interval for one product feature. These are load-bearing issues in the empirical contribution, not presentation points.","major_comments":[{"comment":"The difference-in-differences contrast in Eq. (2) estimates the combined effect of agentic selection and additional message volume, not the incremental effect of personalisation. Section 3 states that agentically-orchestrated messages were 'layered on top of baseline rules-based messages delivered to both control and treatment groups,' and the control group received no equivalent extra or placebo messages. The treated arm therefore differs from control in both message selection and number of sends, so the unconfoundedness/parallel-trends assumption behind Eq. (2) cannot be maintained for the attribution to personalisation. A volume-matched control arm, or a factorial design with a non-agentic extra-message arm, is required; without such a control, the confidence intervals in Table 1 cannot be interpreted as effects of personalisation rather than of extra messaging volume.","section":"§3, Eq. (2)"},{"comment":"The text claims 'We observe significant improvements across both upper- and bottom-funnel events, across all considered use-cases,' but Table 1 reports a 99% confidence interval for Transactional PF3 GMV of [–0.31%, +22.55%], which includes zero and is therefore not significant at the stated level. At the reported confidence level, this is a null result for GMV for that feature. The claim should be qualified, or the significance statement corrected, so that the text and table are consistent.","section":"§3, Table 1"},{"comment":"The statistical details needed to validate the reported confidence intervals are absent. The paper calls the experiment 'randomised controlled,' but the Table 1 caption states that propensity score matching was applied to pair users 'who received at least one message' with those in the control group 'who did not,' which suggests non-compliance or a per-protocol analysis rather than an intention-to-treat analysis. The matching covariates, matching method, caliper, and variance-estimation procedure are not specified. Because the central empirical claim rests entirely on these confidence intervals, the reader cannot assess whether the intervals are valid or whether matching introduced selection bias.","section":"§3, Table 1 caption and §2"}],"minor_comments":[{"comment":"The outcome definition in Eq. (1) depends on several unspecified components: the functional form of the temporal weight w_t, the numerical value of the exponential decay half-life, the window length t_delta, and the empirical Bayes prior parameters. Reporting these values and a small sensitivity analysis would materially improve reproducibility and help readers gauge how much the conclusions depend on these choices.","section":"§2, Eq. (1)"},{"comment":"The term 'agentic' is not formally defined; the described system is a contextual Thompson-sampling bandit with modular actions and a human-in-the-loop action space. A precise definition of what makes the system agentic, or a reference to a specific agentic framework, would help position the contribution relative to existing bandit and recommender-systems work.","section":"§2, 'Decision-Making'"},{"comment":"The paper does not report the number or frequency of messages sent per user in each arm, despite delivery frequency being one of the personalised dimensions. Reporting message volumes would make the treatment/control contrast explicit and would also make the volume confound directly visible to the reader.","section":"§3, 'Results'"},{"comment":"The 'N/A' entry for GMV for the Account Creation product feature is unexplained; a sentence noting that the feature has no monetised transaction would improve table clarity.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is not supported by the experiment as designed: the treatment contrast is confounded with additional message volume, and the reported data themselves contain a significance inconsistency. The volume-matched control is a design-level omission that cannot be repaired post hoc from the currently described experiment. The paper is best framed as an industry case study; acceptance would require a redesign or a substantially more cautious causal claim, which is outside the scope of this manuscript as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper up front. First, the empirical core is a real three-week field experiment across 6.4 million users, reporting 99% confidence intervals for four product features. Second, the central claim that the gains come from personalization is not supported by the design, because the treatment group got extra messages on top of the baseline, and the control got none. That confound is the whole ballgame.\n\nWhat the paper does well: it is clearly written, honest about assembling known building blocks (DiD, Thompson sampling, Wolpertinger-style action decomposition, empirical Bayes priors), and the modular action space is a sensible way to tame the combinatorial explosion of marketing messages. The scale of deployment and the concrete numbers are useful as an existence proof that this kind of system can be built and run. The authors also deserve credit for reporting confidence intervals, not just point estimates.\n\nThe soft spots are significant. The text says treatment messages were layered on top of baseline rules-based messages delivered to both groups, with no placebo or volume-matched control for the treatment arm. Equation (2) therefore estimates the effect of the whole package—personalization plus extra sends—not the incremental effect of personalization. The claims about significance are also inconsistent with the reported numbers: the text says significant improvements across all use-cases, but Table 1 shows a 99% CI for GMV on Transactional PF3 that includes zero. That is not significant at the stated level. In addition, the propensity-score matching and the DiD estimator details are thin; I would want to see how control users were matched, how the variance was computed, and how the temporal decay and event weights were chosen.\n\nWho is this for? Practitioners building CRM or marketing orchestration systems will get a useful systems description and a cautionary tale about experimental design. Researchers in causal inference or bandits will not find new methodology. As a standalone paper, it does not establish the causal claim it leads with, but the underlying system and the field-experiment data are worth engaging with. I would not cite it in my own work, but I would bring it up in a reading group as a case study of confounded deployment experiments.\n\nMy recommendation to an editor: send it to peer review, because the scale and the question matter, but the current version needs a volume-matched control or placebo arm, full estimator details, and consistent reporting of non-significant effects. With those changes it could become a credible applied paper.","headline":"Big field experiment, but the central causal claim is confounded by extra message volume, and the text overstates what Table 1 actually shows.","tokens_in":7197,"tokens_out":1264,"would_cite":false,"duration_ms":14597,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic personalisation of cross-channel marketing lifts intent, conversion, and GMV in a 6.4-million-user field experiment.","keywords":["agentic personalisation","cross-channel marketing","Thompson sampling","Difference-in-Differences","individual treatment effect","contextual bandit","customer relationship management","field experiment"],"falsifier":"Run the same three-week experiment with two treatment arms and one extra control arm: the personalised agentic arm, a volume-matched arm that sends the same number of additional messages with population-level rules-based timing and copy, and a no-extra-message control. If the volume-matched arm reproduces the intent, conversion, and GMV gains, the causal effect is message volume, not personalisation; if only the personalised arm retains the gains, the central claim is supported. A secondary check is to test whether the pre-intervention event trends in the propensity-matched control are parallel to the treated group's trends over the same three weeks.","tokens_in":1823,"feed_emoji":"📈","tokens_out":2917,"duration_ms":109282,"temperature":0.7,"pith_summary":"The paper argues that marketing communication—what to send, by which channel, at what time, and how often—can be treated as a sequential decision-making problem and automated per user. Its proposed agent estimates the incremental effect of each message module on each user from event-stream data, using a Difference-in-Differences contrast with propensity-matched controls, and selects actions through Thompson sampling. The claim is validated by a three-week randomised field experiment across 6.4 million users of a multi-service app: agentically orchestrated messages layered on top of the standard rules-based baseline produced significant absolute increases in intent and conversion for four product features, plus relative GMV gains in the double digits. If the result holds, CRM teams could move from manual segmentation and one-off A/B tests to continuously adapting, user-level personalisation at scale, with marketers still controlling the catalogue of allowed message styles.","feed_headline":"Agentic marketing lifts intent and sales in 6.4M-user test","feed_subtitle":"An AI agent chose timing and copy for each message; four product features gained intent, conversion, and revenue.","key_machinery":"The load-bearing object is the agent: a loop that logs each user's event stream, converts events into weighted outcomes $Y^u_{[t_i,t_j]} = \\sum_{(t,e)\\in D^u} w_e \\, w_t$, estimates an Individual Treatment Effect for every action via the Difference-in-Differences contrast $\\Delta Y = (Y^T_{\\text{post}}-Y^T_{\\text{pre}})-(Y^C_{\\text{post}}-Y^C_{\\text{pre}})$ with propensity-matched controls, and selects the next message by Thompson sampling over a Beta-Bernoulli model, using an empirical Bayes prior for sparse context-action pairs. The action space is modularised in the style of the Wolpertinger architecture: message attributes are grouped into small action sets, and the chosen combination is mapped back to a catalogue of eligible messages, avoiding the combinatorial explosion over full message strings.","core_discovery":"The paper's central claim is that a modular, agentic decision-making policy—rather than pre-scripted rules—should decide the marketing messages each user receives, and that doing so yields measurable incremental engagement and revenue. The treatment group received agentically orchestrated push and in-app messages layered on top of the same rules-based baseline messages sent to both groups. For each candidate action (tone, emoji, value proposition, timing, channel, frequency), the system estimates an Individual Treatment Effect from weighted event streams, and Thompson sampling explores the action space while exploiting what is already known. The authors report that in a three-week randomised controlled experiment across 6.4 million users, intent and conversion rates increased significantly for all four product features, with absolute intent increases from about 0.70 to 2.45 percentage points, absolute conversion increases from 0.07 to 0.48 percentage points, and relative GMV gains reaching double digits for two of the three transactional features. On the strength of these results, the system is deployed to serve the full user base of 150 million users.","pith_inferences":["One natural next test is to separate content personalisation from message-volume effects by adding a non-personalised extra-message control; this would sharpen the causal claim beyond the current propensity-matched design.","The same modular ITE-plus-Thompson-sampling design could be applied to other user-facing decisions beyond marketing—onboarding flows, feature prompts, safety warnings, or recommendation slates—wherever incremental engagement rather than raw clicks is the objective.","Because the event weights are estimated as log-likelihood ratios for goal events, apps with sparse or incomplete event logging may need longer pre-periods or stronger priors before the ITE estimates stabilise.","Three weeks is a short window; a longer deployment would reveal whether the gains persist, decay as novelty wears off, or are offset by message fatigue as exploration continues."],"forward_implications":["Marketers can hand the per-user choices of timing, channel, and message components to the agent while retaining an explicit catalogue of allowed copy and offers.","New product features can be onboarded by defining their goal events and candidate message modules, reusing the same ITE and Thompson-sampling loop.","The modular action space keeps the action set manageable, so personalising over tone, emoji, value propositions, timing, and frequency does not require modelling every full message string.","Sustained double-digit GMV gains at this scale would make the incremental revenue from personalisation a primary business case for replacing rules-based CRM."],"supporting_citations":[{"why":"Supplies the nonlinear Difference-in-Differences identification framework that the ITE estimator relies on.","marker":"[2]"},{"why":"Provides interrupted time series analysis, which the sparse ITE modification extends.","marker":"[14]"},{"why":"Provides the empirical evaluation of Thompson sampling that motivates the arm-selection heuristic.","marker":"[4]"},{"why":"Supplies the tutorial-level treatment of Thompson sampling used to frame the contextual bandit.","marker":"[15]"},{"why":"Introduces the Wolpertinger architecture for large discrete action spaces, which motivates the modular action-space design.","marker":"[5]"},{"why":"Shows recent recommender-systems adoption of Wolpertinger-style action decomposition, supporting the modular mapping back to messages.","marker":"[17]"},{"why":"Supplies user-based collaborative filtering ideas behind the empirical Bayes prior for sparse context-action pairs.","marker":"[6]"},{"why":"Provides the Granger-causality perspective used to set event weights by their informativeness for goal events.","marker":"[16]"},{"why":"Establishes that Thompson sampling remains fast and optimal with less exploration, supporting the deployed exploration-exploitation trade-off.","marker":"[9]"}],"fun_headline_variants":["AI agent picks message timing and tone, lifts conversions in 6.4M test","Agent-driven marketing boosts intent and revenue across 6.4M users","Marketing agent learns each user's style, lifts sales in large-scale test","Personalized message choice by agent yields double-digit gains in test","AI orchestration of push and in-app messages lifts engagement in RCT"],"cache_read_input_tokens":9344,"weakest_assumption_plain":"The result stands on the assumption that the control group's behaviour is a valid counterfactual for what treatment users would have done without personalisation—in particular, that the measured gains come from personalising timing, content, and frequency, and not simply from the fact that treatment users received additional messages on top of the shared rules-based baseline.","fun_headline_variants_meta":{"raw":{"variants":["AI agent picks message timing and tone, lifts conversions in 6.4M test","Agent-driven marketing boosts intent and revenue across 6.4M users","Marketing agent learns each user's style, lifts sales in large-scale test","Personalized message choice by agent yields double-digit gains in test","AI orchestration of push and in-app messages lifts engagement in RCT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1659,"prompt_tokens":913,"completion_tokens":746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":529,"tokens_out":746,"duration_ms":6612,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:26:25.563946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-week experiment with two treatment arms and one extra control arm: the personalised agentic arm, a volume-matched arm that sends the same number of additional messages with population-level rules-based timing and copy, and a no-extra-message control. If the volume-matched arm reproduces the intent, conversion, and GMV gains, the causal effect is message volume, not personalisation; if only the personalised arm retains the gains, the central claim is supported. A secondary check is to test whether the pre-intervention event trends in the propensity-matched control are parallel to the treated group's trends over the same three weeks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the empirical evaluation of Thompson sampling that motivates the arm-selection heuristic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides interrupted time series analysis, which the sparse ITE modification extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that Thompson sampling remains fast and optimal with less exploration, supporting the deployed exploration-exploitation trade-off."}],"review_version":1}