Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Agentic Personalisation of Cross-Channel Marketing Experiences

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Agentic personalisation of cross-channel marketing lifts intent, conversion, and GMV in a 6.4-million-user field experiment.

desk verdict Big field experiment, but the central causal claim is confounded by extra message volume, and the text overstates what Table 1 actually shows. read the letter →

arxiv 2506.16429 v1 pith:3Q3NMRX2 submitted 2025-06-19 cs.AI cs.IRcs.LG

Sami Abboud , Eleanor Hanna , Olivier Jeunen , Vineesha Raheja , Schaun Wheeler This is my paper · ORCID
classification cs.AIcs.IRcs.LG
keywords agenticpersonalisationcross-channelmarketingThompsonsamplingDifference-in-Differencesindividualtreatmenteffectcontextualbanditcustomerrelationshipmanagementfieldexperiment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that marketing communication—what to send, by which channel, at what time, and how often—can be treated as a sequential decision-making problem and automated per user. Its proposed agent estimates the incremental effect of each message module on each user from event-stream data, using a Difference-in-Differences contrast with propensity-matched controls, and selects actions through Thompson sampling. The claim is validated by a three-week randomised field experiment across 6.4 million users of a multi-service app: agentically orchestrated messages layered on top of the standard rules-based baseline produced significant absolute increases in intent and conversion for four product features, plus relative GMV gains in the double digits. If the result holds, CRM teams could move from manual segmentation and one-off A/B tests to continuously adapting, user-level personalisation at scale, with marketers still controlling the catalogue of allowed message styles.

What carries the argument

The load-bearing object is the agent: a loop that logs each user's event stream, converts events into weighted outcomes $Y^u_{[t_i,t_j]} = \sum_{(t,e)\in D^u} w_e \, w_t$, estimates an Individual Treatment Effect for every action via the Difference-in-Differences contrast $\Delta Y = (Y^T_{\text{post}}-Y^T_{\text{pre}})-(Y^C_{\text{post}}-Y^C_{\text{pre}})$ with propensity-matched controls, and selects the next message by Thompson sampling over a Beta-Bernoulli model, using an empirical Bayes prior for sparse context-action pairs. The action space is modularised in the style of the Wolpertinger architecture: message attributes are grouped into small action sets, and the chosen combination is mapped back to a catalogue of eligible messages, avoiding the combinatorial explosion over full message strings.

What would settle it

Run the same three-week experiment with two treatment arms and one extra control arm: the personalised agentic arm, a volume-matched arm that sends the same number of additional messages with population-level rules-based timing and copy, and a no-extra-message control. If the volume-matched arm reproduces the intent, conversion, and GMV gains, the causal effect is message volume, not personalisation; if only the personalised arm retains the gains, the central claim is supported. A secondary check is to test whether the pre-intervention event trends in the propensity-matched control are parallel to the treated group's trends over the same three weeks.

Watch

Extended reading notes

Core claim

The paper's central claim is that a modular, agentic decision-making policy—rather than pre-scripted rules—should decide the marketing messages each user receives, and that doing so yields measurable incremental engagement and revenue. The treatment group received agentically orchestrated push and in-app messages layered on top of the same rules-based baseline messages sent to both groups. For each candidate action (tone, emoji, value proposition, timing, channel, frequency), the system estimates an Individual Treatment Effect from weighted event streams, and Thompson sampling explores the action space while exploiting what is already known. The authors report that in a three-week randomised controlled experiment across 6.4 million users, intent and conversion rates increased significantly for all four product features, with absolute intent increases from about 0.70 to 2.45 percentage points, absolute conversion increases from 0.07 to 0.48 percentage points, and relative GMV gains reaching double digits for two of the three transactional features. On the strength of these results, the system is deployed to serve the full user base of 150 million users.

Load-bearing premise

The result stands on the assumption that the control group's behaviour is a valid counterfactual for what treatment users would have done without personalisation—in particular, that the measured gains come from personalising timing, content, and frequency, and not simply from the fact that treatment users received additional messages on top of the shared rules-based baseline.

Editorial extensions

If this is right

  • Marketers can hand the per-user choices of timing, channel, and message components to the agent while retaining an explicit catalogue of allowed copy and offers.
  • New product features can be onboarded by defining their goal events and candidate message modules, reusing the same ITE and Thompson-sampling loop.
  • The modular action space keeps the action set manageable, so personalising over tone, emoji, value propositions, timing, and frequency does not require modelling every full message string.
  • Sustained double-digit GMV gains at this scale would make the incremental revenue from personalisation a primary business case for replacing rules-based CRM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One natural next test is to separate content personalisation from message-volume effects by adding a non-personalised extra-message control; this would sharpen the causal claim beyond the current propensity-matched design.
  • The same modular ITE-plus-Thompson-sampling design could be applied to other user-facing decisions beyond marketing—onboarding flows, feature prompts, safety warnings, or recommendation slates—wherever incremental engagement rather than raw clicks is the objective.
  • Because the event weights are estimated as log-likelihood ratios for goal events, apps with sparse or incomplete event logging may need longer pre-periods or stronger priors before the ITE estimates stabilise.
  • Three weeks is a short window; a longer deployment would reveal whether the gains persist, decay as novelty wears off, or are offset by message fatigue as exploration continues.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes an agentic framework for cross-channel marketing personalisation, in which a Thompson-sampling bandit selects modular actions (timing, channel, message attributes) for individual users, and a difference-in-differences estimator with propensity-score matching estimates individual treatment effects on a weighted event-stream outcome. The authors report a three-week field experiment across 6.4 million users for four product features, with 99% confidence intervals for intent, conversion, and GMV improvements, and state that the system is deployed to 150 million users. The central empirical claim is that agentically orchestrated messaging produces significant incremental engagement and revenue compared with rules-based baseline messaging.

Significance. If the causal attribution were sound, this would be an important large-scale demonstration that automated, modular, bandit-based personalisation can improve marketing funnel outcomes in a deployed consumer application. The paper has real strengths: it describes a deployed system, reports a genuine field experiment at unusual scale, presents confidence intervals rather than bare point estimates, and adopts a modular action-space design that is operationally sensible. However, the experiment as described cannot support the paper's central claim, because the treatment group received additional agentic messages that the control group did not receive, and the text's claim of significance across all use-cases is contradicted by the reported GMV interval for one product feature. These are load-bearing issues in the empirical contribution, not presentation points.

major comments (3)
  1. [§3, Eq. (2)] The difference-in-differences contrast in Eq. (2) estimates the combined effect of agentic selection and additional message volume, not the incremental effect of personalisation. Section 3 states that agentically-orchestrated messages were 'layered on top of baseline rules-based messages delivered to both control and treatment groups,' and the control group received no equivalent extra or placebo messages. The treated arm therefore differs from control in both message selection and number of sends, so the unconfoundedness/parallel-trends assumption behind Eq. (2) cannot be maintained for the attribution to personalisation. A volume-matched control arm, or a factorial design with a non-agentic extra-message arm, is required; without such a control, the confidence intervals in Table 1 cannot be interpreted as effects of personalisation rather than of extra messaging volume.
  2. [§3, Table 1] The text claims 'We observe significant improvements across both upper- and bottom-funnel events, across all considered use-cases,' but Table 1 reports a 99% confidence interval for Transactional PF3 GMV of [–0.31%, +22.55%], which includes zero and is therefore not significant at the stated level. At the reported confidence level, this is a null result for GMV for that feature. The claim should be qualified, or the significance statement corrected, so that the text and table are consistent.
  3. [§3, Table 1 caption and §2] The statistical details needed to validate the reported confidence intervals are absent. The paper calls the experiment 'randomised controlled,' but the Table 1 caption states that propensity score matching was applied to pair users 'who received at least one message' with those in the control group 'who did not,' which suggests non-compliance or a per-protocol analysis rather than an intention-to-treat analysis. The matching covariates, matching method, caliper, and variance-estimation procedure are not specified. Because the central empirical claim rests entirely on these confidence intervals, the reader cannot assess whether the intervals are valid or whether matching introduced selection bias.
minor comments (4)
  1. [§2, Eq. (1)] The outcome definition in Eq. (1) depends on several unspecified components: the functional form of the temporal weight w_t, the numerical value of the exponential decay half-life, the window length t_delta, and the empirical Bayes prior parameters. Reporting these values and a small sensitivity analysis would materially improve reproducibility and help readers gauge how much the conclusions depend on these choices.
  2. [§2, 'Decision-Making'] The term 'agentic' is not formally defined; the described system is a contextual Thompson-sampling bandit with modular actions and a human-in-the-loop action space. A precise definition of what makes the system agentic, or a reference to a specific agentic framework, would help position the contribution relative to existing bandit and recommender-systems work.
  3. [§3, 'Results'] The paper does not report the number or frequency of messages sent per user in each arm, despite delivery frequency being one of the personalised dimensions. Reporting message volumes would make the treatment/control contrast explicit and would also make the volume confound directly visible to the reader.
  4. [Table 1] The 'N/A' entry for GMV for the Account Creation product feature is unexplained; a sentence noting that the feature has no monetised transaction would improve table clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the treatment-effect estimate comes from a randomized comparison, not from the fitted outcome weights or self-citations.

full rationale

The paper's derivation chain is not circular. Equation (1) defines a weighted outcome using event weights w_e = ln P(e|Goal)/P(e|¬Goal), which are fitted from observed data, but these weights are not used to predict the treatment effect; the causal estimate in Eq. (2) contrasts pre/post outcome changes between treated and control users in a randomized experiment, and Table 1 reports the resulting treatment-effect estimates and confidence intervals. The self-citations ([7], [8]) are used only to motivate reward modelling and sequential decision-making and are not load-bearing for the experimental result. No equation is assumed into existence; the fitted w_e do not determine the DiD contrast. The absence of a volume-matched control is an inferential identification concern, not circularity, and the PF3 GMV confidence interval including zero is a consistency or correctness issue, not a circularity issue. Consequently, no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework rests on standard causal and bandit assumptions, plus a fitted outcome-reward definition. The free parameters (event weights, half-life, window, prior) are not reported, so the exact causal estimates cannot be reconstructed. No new physical or scientific entities are introduced.

free parameters (4)
  • event weight w_e = estimated from historical data; no values provided
    Defines the outcome in Eq. (1) as a log-likelihood ratio of observing a goal event after event e. This fitted weight determines the reward that Thompson sampling optimizes, and its uncertainty is not propagated.
  • temporal decay half-life = tuneable; value not reported
    Sets the exponential recency weighting w_t in Eq. (1); a hand-chosen hyperparameter that changes the outcome and hence the ITE estimates.
  • pre/post window length t_delta = value not reported
    Eq. (3) defines pre and post windows around the intervention; this hyperparameter determines which events enter the Difference-in-Differences estimate.
  • empirical Bayes prior parameters for Beta-Bernoulli Thompson sampling = imputed from similar users; no details given
    Priors for P(Delta Y | A, X) are estimated from user similarity; no formula or implementation is provided.
assumptions (4)
  • domain assumption Parallel trends hold between users who receive agentic messages and the propensity-score-matched control group.
    Required for the Difference-in-Differences estimate in Eq. (2) to identify the incremental effect of treatment. The paper invokes DiD in Section 2 but provides no supporting trend or balance diagnostics.
  • domain assumption Unconfoundedness holds after propensity score matching: all confounders of messaging and engagement are captured in the matching model.
    The Table 1 note says propensity score matching was applied, but no covariates, balance statistics, or matching algorithm are described. If unmeasured confounders remain, the ITE estimates are biased.
  • domain assumption No interference between users: one user's treatment does not affect another user's outcomes (SUTVA).
    The DiD and bandit framework treats each user as an independent unit; cross-user effects such as social sharing or market-level saturation are not modeled.
  • ad hoc to paper The outcome Y_u as a weighted sum of logged events with weights w_e = log P(e|Goal) / P(e|not Goal) is a valid scalar reward for the business funnel.
    Eq. (1) defines the reward; the weights are fitted from historical event-goal co-occurrences. There is no argument that optimizing this weighted proxy is equivalent to optimizing true incremental GMV or conversions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agentic Personalisation of Cross-Channel Marketing Experiences." pith.science (2026). https://pith.science/paper/3Q3NMRX2

@misc{pith2026250616429,
  author       = {Pith},
  title        = {Pith review of: Agentic Personalisation of Cross-Channel Marketing Experiences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3Q3NMRX2}},
  note         = {Machine review of arXiv:2506.16429}
}
read the original abstract

Consumer applications provide ample opportunities to surface and communicate various forms of content to users. From promotional campaigns for new features or subscriptions, to evergreen nudges for engagement, or personalised recommendations; across e-mails, push notifications, and in-app surfaces. The conventional approach to orchestration for communication relies heavily on labour-intensive manual marketer work, and inhibits effective personalisation of content, timing, frequency, and copy-writing. We formulate this task under a sequential decision-making framework, where we aim to optimise a modular decision-making policy that maximises incremental engagement for any funnel event. Our approach leverages a Difference-in-Differences design for Individual Treatment Effect estimation, and Thompson sampling to balance the explore-exploit trade-off. We present results from a multi-service application, where our methodology has resulted in significant increases to a variety of goal events across several product features, and is currently deployed across 150 million users.

Figures

Figures reproduced from arXiv: 2506.16429 by the authors.

Figure 1
Figure 1. An illustration of the feedback loop in our setup. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 canonical work pages

  1. [1]

    Gilligan-Lee

    Oriol Corcoll Andreu, Athanasios Vlontzos, Michael O’Riordan, and Ciaran M. Gilligan-Lee. 2024. Contrastive representations of high-dimensional, structured treatments. arXiv:2411.19245 [stat.ML]

    arXiv 2024

  2. [2]

    Susan Athey and Guido W. Imbens. 2006. Identification and Inference in Non- linear Difference-in-Differences Models. Econometrica 74, 2 (2006), 431–497. https://doi.org/10.1111/j.1468-0262.2006.00668.x

    arXiv 2006

  3. [3]

    Rajeev Batra and Michael L. Ray. 1986. Situational Effects of Advertising Repetition: The Moderating Influence of Motivation, Ability, and Opportu- nity to Respond. Journal of Consumer Research 12, 4 (1986), 432–445. http: //www.jstor.org/stable/254303

    work page 1986

  4. [4]

    Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thomp- son Sampling. In Advances in Neural Information Processing Systems , J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.), Vol. 24. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2011/file/ e53a0a2978c28872a4505bdb51db06dc-Paper.pdf

    work page 2011

  5. [5]

    Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2016. Deep Reinforcement Learning in Large Discrete Action Spaces. arXiv:1512.07679 [cs.AI]

    arXiv 2016

  6. [6]

    Ekstrand, John T

    Michael D. Ekstrand, John T. Riedl, and Joseph A. Konstan. 2011. Collaborative Filtering Recommender Systems. Foundations and Trends® in Human–Computer Interaction 4, 2 (2011), 81–173. https://doi.org/10.1561/1100000009

    doi:10.1561/1100000009 2011

  7. [7]

    Olivier Jeunen. 2021. Offline Approaches to Recommendation with Online Success . Ph. D. Dissertation. University of Antwerp

    work page 2021

  8. [8]

    Olivier Jeunen, Thorsten Joachims, Harrie Oosterhuis, Yuta Saito, and Flavian Vasile. 2022. CONSEQUENCES — Causality, Counterfactuals and Sequential Decision-Making for Recommender Systems. In Proceedings of the 16th ACM Conference on Recommender Systems (Seattle, WA, USA) (RecSys ’22). ACM, New York, NY, USA, 654–657. https://doi.org/10.1145/3523227.3547409

    arXiv 2022

Show all 21 references
  1. [9]

    Tianyuan Jin, Xianglin Yang, Xiaokui Xiao, and Pan Xu. 2023. Thompson Sam- pling with Less Exploration is Fast and Optimal. In Proc. of the 40th International Conference on Machine Learning (Proc. of Machine Learning Research, Vol. 202) . PMLR, 15239–15261. https://proceedings...

    2023

  2. [10]

    Jean Kaddour, Yuchen Zhu, Qi Liu, Matt J Kusner, and Ricardo Silva. 2021. Causal Effect Inference for Structured Treatments. In Advances in Neural Information Processing Systems , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curra...

    2021

  3. [11]

    Ann Kronrod and Joel Huber. 2019. Ad wearout wearout: How time can re- verse the negative effect of frequent advertising repetition on brand prefer- ence. International Journal of Research in Marketing 36, 2 (2019), 306–324. https://doi.org/10.1016/j.ijresmar.2018.11.008

    2019 doi

  4. [12]

    Vineet Kumar and Werner Reinartz. 2018. Customer relationship management. Springer

    2018

  5. [13]

    Lakshmanan, Mitul Tiwari, and Sam Shah

    Pei Lee, Laks V.S. Lakshmanan, Mitul Tiwari, and Sam Shah. 2014. Modeling impression discounting in large-scale recommender systems. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, New York, USA)(KDD ’14). ACM, N...

    2014

  6. [14]

    David McDowall, Richard McCleary, and Bradley J Bartos. 2019. Interrupted time series analysis. Oxford University Press

    2019

  7. [15]

    Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen

    Daniel J. Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. 2018. A Tutorial on Thompson Sampling. Foundations and Trends ® in Machine Learning 11, 1 (2018), 1–96. https://doi.org/10.1561/2200000070

    2018 doi

  8. [16]

    Ali Shojaie and Emily B. Fox. 2022. Granger Causality: A Review and Recent Advances. Annual Review of Statistics and Its Application 9, Volume 9, 2022 (2022), 289–319. https://doi.org/10.1146/annurev-statistics-040120-010930

    2022 doi

  9. [17]

    Aayush Singha Roy, Edoardo D’Amico, Elias Tragos, Aonghus Lawlor, and Neil Hurley. 2023. Scalable Deep Q-Learning for Session-Based Slate Recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23) . ACM, 877–882. https://doi.org/10.1145/36049...

    2023

  10. [18]

    Smith, Felix Greaves, and Trishan Panch

    Andrew L. Smith, Felix Greaves, and Trishan Panch. 2023. Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language Models. PLOS Digital Health 2, 11 (11 2023), 1–3. https://doi.org/10.1371/journal.pdig.0000388

    2023 doi

  11. [19]

    Thompson

    William R. Thompson. 1933. On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika 25, 3/4 (1933), 285–294. http://www.jstor.org/stable/2332286

    1933

  12. [20]

    Francesco Trovo, Stefano Paladino, Marcello Restelli, and Nicola Gatti. 2020. Sliding-window thompson sampling for non-stationary settings. Journal of Artificial Intelligence Research 68 (2020), 311–364

    2020

  13. [21]

    Caroline Tynan and Jennifer Drayton

    A. Caroline Tynan and Jennifer Drayton. 1987. Market segmentation. Journal of Marketing Management 2, 3 (1987), 301–335. https://doi.org/10.1080/0267257X. 1987.9964020

    1987

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.