Development of an End-to-end Machine Learning System with Application to In-app Purchases

Adrian Chmielewski-Anders; Anders Englesson; Christoffer {\AA}hrling; Dionysios Varelas; Elena Bonan; Lewis Anderson

REVIEW 3 major objections 5 minor 17 references

Development of an End-to-end Machine Learning System with Application to In-app Purchases

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ContentRNN, a GRU-based model with a calibration layer, beats a popularity baseline at predicting the day of a player's next in-app purchase, both offline and in an online A/B test.

desk verdict A useful, honest industry case study of an end-to-end ML platform for purchase timing; the model itself is a known-technique combination and the empirical lift claims are under-evidenced, but the system description earns a careful read. read the letter →

arxiv 2412.12390 v1 pith:P62QSG4M submitted 2024-12-16 cs.LG

Dionysios Varelas , Elena Bonan , Lewis Anderson , Anders Englesson , Christoffer {AA}hrling , Adrian Chmielewski-Anders This is my paper · ORCID

classification cs.LG

keywords MachineLearningPlatformMobileGamingPersonalizationIn-appPurchasesPurchaseTimingPredictionContentRNNA/BTesting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a neural network called ContentRNN—a GRU layer combined with content features and a post-hoc calibration layer—predicts the day of a player's next in-app purchase better than a popularity baseline built from three purchase-frequency segments. Offline, ContentRNN improves mean absolute error from 3.84 to 3.36 days, accuracy from 0.39 to 0.41, precision from 0.18 to 0.27, and F1 from 0.25 to 0.29. In an online A/B test against the baseline, the model produces a 20% increase on one of the key business metrics. The paper then details the end-to-end production system—feature engineering platform, training pipelines, batch inference, and monitoring—that makes these daily predictions available to the personalization engine. A sympathetic reader should care because the work shows a concrete path from a predicted next-purchase distribution to personalized offer timing in a live mobile game.

What carries the argument

The central objects are the popularity baseline, the ContentRNN architecture, and the CDF-based problem formulation. The baseline partitions players into three segments by purchase frequency and, for each segment, assigns the most frequent next-purchase day as the prediction. ContentRNN takes numerical features, a time-series feature vector, and categorical embeddings; the time series feeds a GRU layer, all features are concatenated, then two ReLU hidden layers (sizes 64 and 32) lead to a 16-output softmax. A calibration layer, applied element-wise after training, adjusts the probability estimates, and the loss is a weighted sum of cross-entropy and mean-squared error with tuned weights a and b. The problem formulation sums per-day purchase probabilities to form a CDF, with the offer shown at the CDF's median day; this CDF construction is what connects the model's daily probability estimates to the offer-timing business rule.

What would settle it

On the training data, count how many players make two or more purchases within a single N-day window; if that fraction is non-negligible, the summed per-day probabilities are not a valid CDF and the median-based offer rule is undefined.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that adding a sequence model to a tabular neural network, together with a calibration layer, improves the accuracy of next-purchase-day prediction for mobile game players. The model estimates the probability of a purchase on each of the next N days, sums those probabilities into a CDF, and triggers an offer on the day corresponding to the CDF's median for eligible players. ContentRNN's GRU layer captures short-term behavioral signals that the tabular network misses, while the calibration layer yields better probability estimates; the two-stage training—end-to-end first, then a second pass with all layers frozen except the calibration layer—is part of the recipe. The reported offline gains (MAE 3.36 vs 3.84, accuracy 0.41 vs 0.39, F1 0.29 vs 0.25) and the 20% lift on a key business metric in an online A/B test are the evidence the paper offers for this claim.

Load-bearing premise

The problem framing assumes that a player makes at most one purchase within the N-day window, so that summing per-day purchase probabilities yields a true CDF; if multiple purchases per window are possible, the summed quantity is not a probability distribution and the median-based offer rule is not well-defined.

Editorial extensions

If this is right

If the online improvement generalizes, then timing offers by predicted next-purchase day can raise key business metrics relative to segment-based popularity timing.
The combined GRU-plus-tabular architecture with a calibration layer is a reusable template for time-until-event prediction in other mobile game actions.
The production platform described—feature engineering with daily validation tests, cloud training pipelines, PySpark batch inference, and third-party monitoring with retraining triggers—can support daily predictions at hundred-million scale.
The two-pass training and the a·CE + b·MSE loss are effective for ordinal next-event-day prediction and can be carried over to similar problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same time-until-next-action framing could be applied to other game events such as churn or level completion, with the model controlling when to send interventions; the paper does not propose these extensions.
The CDF construction is only a true probability distribution under a single-purchase assumption per window; a survival-analysis formulation that handles repeated purchases and non-purchasing players would be a more defensible target and the paper does not address it.
The reported 20% business-metric lift could be decomposed by an ablation study that removes the calibration layer or the GRU layer in the online test; the paper does not report such ablations.
The paper's batch-inference cadence could be shifted to real-time scoring, which the authors list as future work, potentially enabling offers timed to within minutes rather than days.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, it is a genuine industry case study from King, and its main value is the detailed description of a production ML platform: feature engineering, training pipelines, deployment, monitoring, retraining, and the operational lessons. Second, the headline empirical claim—ContentRNN beating a popularity baseline—rests on point estimates and a one-sentence A/B result, so the actual size of the effect is not established. That is a reporting gap, not evidence that the work is wrong.

What is actually new: the specific application of a GRU plus calibration layer to predict the day of a player's next in-app purchase, and the combination of that model with a production system that makes daily batch predictions for personalization. The paper does not claim to introduce a new learning method; it cites the components. That is honest. The offline metrics (MAE 3.36 vs 3.84, F1 0.29 vs 0.25) are plausible and directionally consistent, and the popularity baseline is a sensible, strong reference point.

The weak spots are real but proportionate. The offline results are single numbers on a proprietary test set—no confidence intervals, no test-set size, no repeated-seed variability. The MAE gap could easily be sampling noise in a large, noisy purchase-timing task. The online A/B test is reported as “20% increase on one of the key business metrics” with no primary metric, duration, sample size, or confidence interval. If that figure was selected from several tracked metrics, multiple comparisons are a genuine concern. The CDF construction in Section 3 is also under-specified: summing per-day purchase probabilities only gives a valid distribution if the events are mutually exclusive (i.e., you are modeling the day of the first purchase after the snapshot) and if players who never purchase in the window are censored. The paper does not state either condition. These are clarity issues that a referee should ask to be fixed.

Who gets value: practitioners building ML platforms for personalization, especially in mobile gaming or similar industries. It is a good reference for how to structure feature engineering, training, deployment, and monitoring in a real system. The citation pattern is appropriate and the reflection on challenges is candid.

For peer review: yes, send it out. It is an applied paper whose contribution is the system, not the novelty of the model. A serious referee can ask for the missing uncertainty quantification and experimental details without dismissing the paper. I would not cite it in my own work, but I would bring it to a reading group focused on applied ML systems.

Referee Report

3 major / 5 minor

Summary. The paper describes the development of an end-to-end machine learning system at King for predicting when a player will make their next in-app purchase, with the goal of personalizing offer timing. The proposed model, ContentRNN, combines a GRU layer with content-based features and a calibration layer, and is compared offline against a popularity baseline using MAE, accuracy, precision, and F1. The paper reports offline improvements (MAE 3.36 vs. 3.84) and a 20% increase on an unspecified key business metric in an online A/B test. The bulk of the paper then details the production ML platform: feature engineering, training pipelines, batch deployment, monitoring, and infrastructure.

Significance. If the reported improvements are statistically reliable, this is a valuable industrial case study: it demonstrates a concrete application of a neural model to purchase-timing prediction and provides a unusually complete description of the surrounding ML platform, including data validation, retraining policies, monitoring, and infrastructure-as-code. The paper's clear separation of offline and online evaluation and its candid discussion of challenges (training/serving skew, retraining cadence, communication overhead) are strengths. The main limitation is that the empirical superiority claims rest on point estimates with no uncertainty quantification, and the problem formulation in Section 3 is underspecified; these issues prevent the reader from fully assessing the validity of the headline results.

major comments (3)

[Section 3] The CDF construction is not well-defined as written. The equation P(purchase in d1,...,dN | X_i) = sum_{n=1}^N P(purchase on d_n | X_i) is only a valid probability if the events "purchase on d_n" are mutually exclusive across days, which holds only if each player makes at most one purchase in the window and the probabilities refer to the day of the next purchase. The paper does not state this assumption, does not explain how players who never purchase within the N-day horizon are handled (censoring), and does not discuss how the model labels such cases. This matters because the median of the CDF is used to decide offer eligibility; if the summed quantity is not a distribution, the rule is not well-founded. Please clarify the target variable and state the exclusivity/censoring assumptions explicitly.
[Section 5.4, Figure 3] The offline results report only point estimates on a proprietary test set, with no test-set size, confidence intervals, error bars, or significance tests. The reported MAE gap of 0.48 (3.84 vs. 3.36) and F1 gap of 0.04 (0.25 vs. 0.29) could plausibly be within sampling error for a large, noisy purchase-timing task. To support the claim that ContentRNN outperforms the popularity baseline, the authors should report the test-set size, per-metric confidence intervals (e.g., bootstrap or repeated-seed variation), and ideally per-segment results. Without this, the offline superiority claim is not established.
[Section 5.5] The online A/B test result is reported in a single sentence: "Against the popularity baseline we observed 20% increase on one of the key business metrics." The paper does not specify which metric was primary, how many metrics were tracked, the test duration, the sample size, the confidence interval, or whether the metric was pre-specified. If the 20% figure was selected post hoc from several sensitive business metrics, it could reflect multiple comparisons rather than a genuine effect. Please provide the A/B test design and uncertainty quantification, or explicitly label the result as an anecdotal internal observation rather than a statistically validated claim.

minor comments (5)

[Section 5.1] The popularity baseline equation uses count({p purchases on d_n | p in S_m}) / count(S_m). This is only a probability distribution if every player in the segment has exactly one next purchase day in the window; please clarify the treatment of players without a purchase, which also relates to the censoring issue in Section 3.
[Section 5.3] The calibration layer is described as trained in a second pass with all other layers frozen, but no details are given about the calibration training data, loss, or validation procedure; a sentence or two would help reproducibility.
[Section 7.1] There is a typo: "can proved to be difficult" should be "can prove to be difficult." Also, the description of the retraining trigger uses "3 standard deviations" without stating over what time window or metric; please specify.
[Section 8.3] The sentence "Being a able to serve more use cases in real time" contains a typo ("a able").
[References] The GRU layer is attributed to reference [1], which is the Bahdanau et al. neural machine translation paper; the original GRU paper (Cho et al., 2014) is a more appropriate citation for the architecture.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported improvements are empirical comparisons of independently trained models, not derivations that reduce to fitted inputs.

full rationale

The paper's load-bearing claims are that ContentRNN outperforms a popularity baseline on a proprietary test set (Figure 3: MAE 3.84 vs 3.36, accuracy 0.39 vs 0.41, F1 0.25 vs 0.29) and that an online A/B test observed a 20% increase on one key business metric (Section 5.5). These are empirical measurements of trained systems, not tautological consequences of the model definitions. The popularity baseline is constructed from purchase-frequency segments independently of ContentRNN's learned predictions; the neural model is trained on features and labels and evaluated on a held-out test set, so the offline metrics are not fitted values renamed as predictions. The Section 3 formulation, which sums per-day purchase probabilities into a CDF, is a modeling assumption about the target label rather than an equation that makes the results equivalent to their inputs. The paper cites external work for RNNs, calibration, and ML platforms, and none of those citations define the model's predictive target or are used to establish superiority in a self-referential way. The absence of confidence intervals, A/B test significance details, and a clearer statement about mutually exclusive purchase events is a reporting and robustness limitation, not circularity. The derivation chain is therefore self-contained with respect to the paper's stated problem and evaluation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on tuned hyperparameters, an unstated exclusivity assumption in the CDF construction, and undisclosed details about data splits, feature correctness, and the A/B test. No new physical or conceptual entities are introduced.

free parameters (4)

Loss weights a and b = not disclosed
Tuned via cross-validation in Section 5.2 to weight cross-entropy and MSE losses; no values or search grid provided.
Network architecture sizes = 64, 32, 16
Dense layer sizes and softmax output size in Section 5.2 are chosen by the authors with no ablation study.
Popularity baseline segment thresholds = not disclosed
The three player segments in Sections 2.1 and 5.1 are based on purchase frequency percentiles chosen from data, and they define the baseline being compared.
Retraining threshold = 3 standard deviations
The monitoring policy in Section 7.1 triggers retraining when prediction distribution shifts more than 3 standard deviations; an ad hoc choice.

assumptions (4)

domain assumption The quantity in Eq. (1) is a valid cumulative distribution function for the next purchase day.
Requires mutual exclusivity of daily purchase events and at most one purchase in the window; not stated in Section 3.
domain assumption The training, validation, and test splits are temporally ordered and free of leakage.
Section 6.2.1 states a 12/2/2 month split but does not describe deduplication, leakage checks, or how players are assigned to splits.
domain assumption The A/B test is statistically sound and the reported 20% lift is attributable to ContentRNN.
Section 5.5 gives no sample size, duration, significance level, or pre-registered metric.
domain assumption The engineered features and labels in the feature engineering platform are computed correctly.
Section 6.1.2 mentions automated validation tests, but their coverage and the label definition for non-purchasers are not described.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Development of an End-to-end Machine Learning System with Application to In-app Purchases." pith.science (2026). https://pith.science/paper/P62QSG4M

@misc{pith2026241212390,
  author       = {Pith},
  title        = {Pith review of: Development of an End-to-end Machine Learning System with Application to In-app Purchases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P62QSG4M}},
  note         = {Machine review of arXiv:2412.12390}
}

read the original abstract

Machine learning (ML) systems have become vital in the mobile gaming industry. Companies like King have been using them in production to optimize various parts of the gaming experience. One important area is in-app purchases: purchases made in the game by players in order to enhance and customize their gameplay experience. In this work we describe how we developed an ML system in order to predict when a player is expected to make their next in-app purchase. These predictions are used to present offers to players. We briefly describe the problem definition, modeling approach and results and then, in considerable detail, outline the end-to-end ML system. We conclude with a reflection on challenges encountered and plans for future work.

Figures

Figures reproduced from arXiv: 2412.12390 by the authors.

**Figure 1.** Discounted offer 3 PROBLEM DEFINITION The target of this model is to identify for each player the probability of making a purchase any day within the next 𝑁 days. Essentially, this requires the estimation of the cumulative distribution function (CDF) for each player over the following 𝑁 days of interest. According to this, a player is eligible to receive an offer if they have not made a purchase until the day that … view at source ↗

**Figure 2.** ContentRNN Architecture Model MAE Accuracy Precision F1 Popularity baseline 3.84 0.39 0.18 0.25 ContentRNN 3.36 0.41 0.27 0.29 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Structure of a (re) training pipeline. Purple colored [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Inference pipeline • Difference between the training and inference processes (training/serving skew) • Change in the feature distribution (covariate shift) • Change in the relation between features and label (concept shift) The system continuously monitors input featur…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

[1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs.CL]

arXiv 2016
[2]

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Ben- netot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Ex- plainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Informatio...

doi:10.1016/j.inffus.2019.12.012 2020
[3]

Vaishak Belle and Ioannis Papantonis. 2021. Principles and practice of explainable machine learning. Frontiers in big Data (2021), 39

work page 2021
[4]

Ekaba Bisong. 2019. Kubeflow and Kubeflow Pipelines . Apress, Berkeley, CA, 671–685. https://doi.org/10.1007/978-1-4842-4470-8_46

work page doi:10.1007/978-1-4842-4470-8_46 2019
[5]

Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2017. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. In Proceedings of IEEE Big Data

2017
[6]

Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. 2019. Continual Learning in Practice. arXiv:1903.05202 [stat.ML]

work page Pith review arXiv 2019
[7]

Weinberger

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. https://doi.org/10.48550/ARXIV.1706.04599

arXiv doi:10.48550/arxiv.1706.04599 2017
[8]

Turnbull, Bren- dan M

Malay Haldar, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Williams, Bradley C. Turnbull, Bren- dan M. Collins, and Thomas Legrand. 2019. Applying Deep Learning to Airbnb Search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining . ACM. https://doi.org...

arXiv 2019

Show all 17 references

[9]

Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber’s Machine Learning Platform. https://eng.uber.com/michelangelo-machine- learningplatform/

2017
[10]

Theofilos Kakantousis, Antonios Kouzoupis, Fabio Buso, Gautier Berthou, Jim Dowling, and Seif Haridi. 2019. Horizontally scalable ml pipelines with a feature store. In Proc. 2nd SysML Conf., Palo Alto, USA

2019
[11]

Igor L Markov, Pavlos A Apostolopoulos, Mia Garrard, Yin Huang, Tanvi Gupta, Anika Li, Cesar Cardoso, George Han, Ryan Maghsoudian, Norm Zhou, et al
[12]

Markov, Hanson Wang, Nitya S

Igor L. Markov, Hanson Wang, Nitya S. Kasturi, Shaun Singh, Mia R. Garrard, Yin Huang, Sze Wai Celeste Yuen, Sarah Tran, Zehui Wang, Igor Glotov, Tanvi Gupta, Peng Chen, Boshuang Huang, Xiaowen Xie, Michael Belkin, Sal Uryasev, Sam Howie, Eytan Bakshy, and Norm Zhou. 2022. Loo...

2022
[13]

Baylor, Eric Breck, Heng-Tze Cheng, Jarek Wilkiewicz, Levent Koc, Lukasz Lew, Martin A

Akshay Naresh Modi, Chiu Yuen Koo, Chuan Yu Foo, Clemens Mewald, Denis M. Baylor, Eric Breck, Heng-Tze Cheng, Jarek Wilkiewicz, Levent Koc, Lukasz Lew, Martin A. Zinkevich, Martin Wicke, Mustafa Ispir, Neoklis Polyzotis, Noah Fiedel, Salem Elie Haykal, Steven Whang, Sudip Roy,...

2017
[14]

Dominick Rocco. 2021. What is a Feature Store? https://www.phdata.io/blog/ what-is-a-feature-store/

2021
[15]

Manasi Vartak and Samuel Madden. 2018. MODELDB: Opportunities and Chal- lenges in Managing Machine Learning Models. IEEE Data Eng. Bull. 41, 4 (2018), 16–25. http://sites.computer.org/debull/A18dec/p16.pdf

2018
[16]

Franklin, Scott Shenker, and Ion Stoica

Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Re- silient Distributed Datasets: A Fault-Tolerant Abstraction for in-Memory Cluster Computing. In Proceedings of the 9th USENIX ...

2012
[2023]

arXiv preprint arXiv:2302.14139 (2023)

Scalable End-to-End ML Platforms: from AutoML to Self-serve. arXiv preprint arXiv:2302.14139 (2023)

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs.CL]

arXiv 2016

[2] [2]

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Ben- netot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Ex- plainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Informatio...

doi:10.1016/j.inffus.2019.12.012 2020

[3] [3]

Vaishak Belle and Ioannis Papantonis. 2021. Principles and practice of explainable machine learning. Frontiers in big Data (2021), 39

work page 2021

[4] [4]

Ekaba Bisong. 2019. Kubeflow and Kubeflow Pipelines . Apress, Berkeley, CA, 671–685. https://doi.org/10.1007/978-1-4842-4470-8_46

work page doi:10.1007/978-1-4842-4470-8_46 2019

[5] [5]

Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2017. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. In Proceedings of IEEE Big Data

2017

[6] [6]

Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. 2019. Continual Learning in Practice. arXiv:1903.05202 [stat.ML]

work page Pith review arXiv 2019

[7] [7]

Weinberger

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. https://doi.org/10.48550/ARXIV.1706.04599

arXiv doi:10.48550/arxiv.1706.04599 2017

[8] [8]

Turnbull, Bren- dan M

Malay Haldar, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Williams, Bradley C. Turnbull, Bren- dan M. Collins, and Thomas Legrand. 2019. Applying Deep Learning to Airbnb Search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining . ACM. https://doi.org...

arXiv 2019