Pith. sign in

REVIEW 4 major objections 4 minor 16 references

Addressing bias in Recommender Systems: A Case Study on Data Debiasing Techniques in Mobile Games

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Applying three debiasing methods to mobile-game recommender data, this case study finds that the largest accuracy gains require a randomized holdout dataset, while the no-random-data method IPS yields small and inconsistent improvements.

desk verdict Useful industrial case study with new numbers on mobile-game debiasing, but the headline claim about randomized data is confounded and the evaluation has gaps. read the letter →

arxiv 2411.18716 v1 pith:XS3VL5PF submitted 2024-11-27 cs.LG

Yixiong Wang , Maria Paskevich , Hui Wang This is my paper · ORCID
classification cs.LG
keywords recommendersystemsdebiasingimplicitfeedbackmobilegamesinversepropensityscoringdoublyrobustlearningAutoDebiasmatrixfactorization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a case study of three data-debiasing techniques for recommender systems, tested on mobile-game purchasing data where feedback is implicit (purchase or no purchase). It first validates the techniques on two public explicit-feedback datasets, then applies them to three proprietary implicit-feedback datasets from a mobile-game company, using a matrix-factorization model trained on biased data as the baseline. The central finding is that methods requiring a randomized holdout dataset—Doubly Robust learning and AutoDebias—produce the largest accuracy gains, with Doubly Robust learning cutting RMSE by 45.40% on one proprietary dataset. Inverse Propensity Scoring, which does not require randomized data, improves diversity metrics but gives only limited and inconsistent accuracy gains. The paper concludes that significant performance gains still depend on collecting randomized datasets, despite the operational cost and user-experience risk of doing so.

What carries the argument

The load-bearing mechanism is the randomized (uniform) holdout dataset $D_U$: a slice of data in which users received random recommendations, treated as unbiased ground truth for preference. The paper splits any available randomized data into 5% for training ($D_U$), 5% for validation, and 90% for testing, and this split feeds Doubly Robust learning's combination of propensity weighting with imputation and AutoDebias's meta-learning weight assignment. The baseline is matrix factorization trained only on biased data, and improvements are expressed as percentage changes in RMSE, AUC, NDCG@5, Gini index, and Entropy. The presence or absence of $D_U$ is what separates the large gains from the small ones.

What would settle it

Collect an independent, freshly randomized holdout for the dataset on which DR cut RMSE by 45.40%, train DR on the original biased set, and test on that new holdout; if the RMSE gain largely disappears, the reported improvement was tied to the particular split rather than to genuine debiasing. A second check is to specify a proper unbiased test set for the dataset lacking randomized data and see whether IPS's reported diversity gains and accuracy losses still hold there.

Watch

Extended reading notes

Core claim

The central claim is empirical rather than methodological: on implicit-feedback mobile-game shop data, debiasing techniques that use a randomized dataset can yield large improvements over a biased baseline, while the technique that avoids randomized data cannot. The strongest reported result is a 45.40% reduction in RMSE from Doubly Robust learning on one proprietary dataset, together with a 7.07% AUC gain and gains in NDCG@5 and Gini. AutoDebias also improves accuracy metrics, especially on the two public datasets, but its training time grows dramatically. On the proprietary dataset without any randomized data, IPS improves diversity metrics but worsens RMSE and leaves NDCG@5 essentially flat. The authors' stated conclusion is that 'significant performance gains still depend on the challenging task of collecting randomized datasets.'

Load-bearing premise

The load-bearing premise is that the randomized holdout data is an unbiased ground truth for what players actually prefer, and that Gini and Entropy movements really measure reduced bias; for the one dataset that has no randomized data, there is also no stated unbiased test benchmark, so its results rest on an unexamined evaluation choice.

Editorial extensions

If this is right

  • Game developers who cannot run randomized holdouts should expect only modest accuracy gains from IPS, mainly in diversity metrics like Gini and Entropy.
  • Building a pipeline that collects randomized recommendation data is likely to matter more than the choice of debiasing model, since the two methods with access to such data dominate the accuracy improvements.
  • Doubly Robust learning on mid-sized implicit-feedback data can simultaneously improve accuracy and diversity, but the roughly 386% training-time increase on Set B means compute cost must be budgeted.
  • On the largest proprietary dataset, AutoDebias improved AUC and NDCG@5 but degraded RMSE, Gini, and Entropy relative to the biased baseline, so scale changes which debiasing method wins.
  • Implicit-feedback datasets produce less consistent debiasing results than explicit-feedback public datasets, suggesting that purchase/no-purchase signals weaken the transfer of these methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper identifies randomized data as the bottleneck, a natural next step it does not test is whether online exploration (e.g., contextual bandits) can supply the same unbiased signal more cheaply than a static randomized holdout.
  • A sharper test of whether DR genuinely debiases would measure exposure equity directly against the randomized benchmark, rather than inferring it from Gini and Entropy movement.
  • The 45.40% RMSE result on one dataset could be checked for robustness by collecting a second, independently randomized sample and testing on it; if the gain does not transfer, it may be an artifact of one split.
  • Combining IPS-style variance control with a small randomized sample might recover some of DR's gains when full randomized collection is unaffordable, a middle ground the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a case study of three debiasing techniques—Inverse Propensity Scoring (IPS), Doubly Robust learning (DR), and AutoDebias—applied to recommender systems for mobile games. The authors evaluate these methods against a biased matrix-factorization baseline (MF(biased)) on two public explicit-feedback datasets (COAT, YahooR3!) and three proprietary implicit-feedback datasets from King (Sets A, B, C), using RMSE, AUC, NDCG@5, Gini, Entropy, and training time. The central conclusion, stated in Section 6, is that significant performance gains still depend on collecting randomized datasets, since DR and AutoDebias, which require such data, show large improvements on some internal datasets while IPS, which does not, offers more limited gains.

Significance. If the main conclusion were established, the paper would provide practically useful guidance for mobile-game recommender systems: randomized data collection is a key enabler for debiasing gains, and methods that avoid such data (IPS) yield only modest improvements. The paper's strengths are its use of real-world implicit-feedback data from a commercial mobile gaming setting, the inclusion of confidence intervals from repeated runs in Table 4, and a reproducible artifact-based training pipeline. However, the headline causal claim is not isolated by the experimental design, and several evaluation choices need to be tightened before the conclusion can be accepted. The paper's internal admission in Section 5.2 that results are 'less consistent' across internal datasets is itself a signal that the cross-dataset generalization claim should be treated cautiously.

major comments (4)
  1. [§4 and §6] The central conclusion that 'significant performance gains still depend on the challenging task of collecting randomized datasets' is not established because the comparison confounds the randomized-data effect with other factors. In §4, DR and AutoDebias are trained with an additional 5% randomized subset D_U, while IPS and MF(biased) are not; moreover, DR and AutoDebias are more complex estimators, and IPS is the only no-randomized-data method tested. The large gains reported in Table 3 (e.g., Set B DR RMSE -45.40%) could therefore reflect extra training data, model capacity, or method choice rather than the randomized origin of D_U. A control arm that replaces D_U with an equal-sized biased sample, or a comparison against MF(uniform) from Table 4 as a randomized-data baseline, is needed before the §6 claim can be accepted.
  2. [§4 and Table 3] The test set for Set A is unspecified. Set A has no randomized data, so the 'data split strategy' described in §4—5% for D_U, 5% for D_V, 90% for D_Te—cannot be applied to it. Yet the paper reports IPS results on Set A in Table 3. It is unclear what data was used to compute RMSE, AUC, NDCG, Gini, and Entropy for Set A, and whether the evaluation set is biased or unbiased. This undermines the only IPS-only internal result and should be clarified.
  3. [Tables 2 and 3] The percentage improvements in Tables 2 and 3 are reported without confidence intervals or significance tests, even though Table 4 provides absolute values with 95% confidence intervals. Several claimed improvements are small relative to the intervals; for example, Set B DR's NDCG@5 gain of 0.68% appears to be within the overlap of the confidence intervals for MF(biased) and DR in Table 4. The authors should report uncertainty for the relative improvements or clearly state which differences are statistically distinguishable.
  4. [§3.3, §5.2, and Table 4] Gini and Entropy are treated as measures of debiasing quality, but no evidence is given that improving these diversity metrics corresponds to reducing the bias identified in §3.1 (selection, exposure, and position bias). In Table 4, MF(uniform) and MF(biased) have nearly identical Gini and Entropy on COAT and YahooR3!, which suggests these metrics may be insensitive to the biases at play. The paper should either validate Gini/Entropy as debiasing measures using a known unbiased benchmark or temper the claims about 'better balance' and 'debiasing quality' in §5.2.
minor comments (4)
  1. [Throughout] There are several typos and inconsistent spellings: 'propreitery' in §3.2, 'NGCG@5' in §5.1, 'fine-tunning' in §3.3, and inconsistent 'AutoDebias'/'Autodebias' capitalization.
  2. [Table 3 and §5.2] The 'Training time (sec)' column in Table 3 actually shows percentage changes, not seconds; the heading should be 'Training time change' to match Table 2 and avoid confusion.
  3. [§3.3] The notation in the NDCG and Entropy formulas is unclear: 'rel_i' is not fully defined, and 'n is a total number of items u in a dataset' reads awkwardly; also the AUC formula's 'rank_{u,i}' should be defined as the rank of the positive item among all items for user u.
  4. [Table 4] The MF(uniform) results are reported in Table 4 but are never discussed in the text or used as a comparison baseline in Tables 2 and 3; including this model in the relative comparisons would strengthen the analysis of the value of randomized data.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper applies published debiasing methods to external and proprietary data and draws an empirical, confounded-but-not-circular conclusion.

full rationale

The paper's derivation chain is not circular. The debiasing methods (IPS, DR, AutoDebias) are taken from the external literature, and the percentage improvements in Tables 2 and 3 are out-of-sample comparisons against an MF(biased) baseline on held-out randomized test data as described in Section 4. No parameter is fitted to a subset and then reported as a prediction of a closely related quantity: the models are trained on D_T or D_U and evaluated on D_Te. The only self-citation, DFSNet, is mentioned in Related Work as prior King work but is not used in any experiment and does not support the conclusion. The central claim in Section 6—'significant performance gains still depend on the challenging task of collecting randomized datasets'—is an empirical interpretation of the observed results, not an equation that reduces to its inputs. The skeptic's concern that the design confounds randomized-data availability with model complexity and extra training data is a correctness threat, not a circularity: the paper never claims to isolate the randomized-data mechanism. Section 5.2's admission that internal results are 'less consistent' and Section 4's omission of the Set A test set are transparency limitations, but they do not make the derivation circular. Overall, no circular step can be quoted and exhibited.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical comparison rests on the availability of unbiased randomized data and on the validity of the chosen metrics. The paper does not introduce new theoretical constructs; its main assumptions are domain-level commitments about what unbiased data looks like and what debiasing quality means.

free parameters (3)
  • MF latent factors and regularization = not reported (tuned on validation set)
    Matrix Factorization baseline depends on these hyperparameters; they affect all percentage improvement comparisons.
  • IPS propensity model parameters = not reported
    IPS requires estimating observation propensities, but the paper does not describe the propensity model or its estimation.
  • AutoDebias meta-learning parameters = not reported
    AutoDebias trains an additional meta-model; its hyperparameters are tuned on D_V but values are not given.
assumptions (3)
  • domain assumption Randomized holdout data is unbiased and reflects true user preference
    Section 4 divides randomized data into training support, validation, and test; the evaluation treats D_U as ground truth for debiasing quality.
  • domain assumption Binary purchase/no-purchase feedback is a valid proxy for user preference
    Section 3.1 describes implicit feedback as binary response; the debiasing evaluation uses purchases as positive signals without accounting for missing negatives.
  • domain assumption Gini and Entropy improvements indicate reduced bias
    Section 3.3 defines Gini and Entropy as evaluation metrics, but the paper does not demonstrate that more diverse recommendations correspond to less biased models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Addressing bias in Recommender Systems: A Case Study on Data Debiasing Techniques in Mobile Games." pith.science (2026). https://pith.science/paper/XS3VL5PF

@misc{pith2026241118716,
  author       = {Pith},
  title        = {Pith review of: Addressing bias in Recommender Systems: A Case Study on Data Debiasing Techniques in Mobile Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XS3VL5PF}},
  note         = {Machine review of arXiv:2411.18716}
}
read the original abstract

The mobile gaming industry, particularly the free-to-play sector, has been around for more than a decade, yet it still experiences rapid growth. The concept of games-as-service requires game developers to pay much more attention to recommendations of content in their games. With recommender systems (RS), the inevitable problem of bias in the data comes hand in hand. A lot of research has been done on the case of bias in RS for online retail or services, but much less is available for the specific case of the game industry. Also, in previous works, various debiasing techniques were tested on explicit feedback datasets, while it is much more common in mobile gaming data to only have implicit feedback. This case study aims to identify and categorize potential bias within datasets specific to model-based recommendations in mobile games, review debiasing techniques in the existing literature, and assess their effectiveness on real-world data gathered through implicit feedback. The effectiveness of these methods is then evaluated based on their debiasing quality, data requirements, and computational demands.

Figures

Figures reproduced from arXiv: 2411.18716 by the authors.

Figure 1
Figure 1. Examples of content placements in Candy Crush Soda Saga (left) and Candy Crush Saga (right), highlighting biases: selection bias with a prominently placed product (left) and exposure bias with limited visibility, where products are hidden behind the "More Offers" button (right) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Debiasing results on open datasets (COAT and yahooR3!). The graphs show the percentage change in metrics (AUC, RMSE, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Debiasing results on internal datasets (Set A, Set B and Set C). The graphs show the percentage change in metrics (AUC, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Implicit feedback for recommender systems

    Oard, Douglas W.& Jinmook Kim. Implicit feedback for recommender systems . Proceedings of the AAAI workshop on recommender systems. Vol. 83. 1998

    work page 1998

  2. [2]

    ACM Trans

    Jiawei Chen and Hande Dong and Xiang Wang and Fuli Feng and Meng Wang & Xiangnan He,Bias and Debias in Recommender System: A Survey and Future Directions. ACM Trans. Inf. Syst. 41, 3, Article 67 (2023) Manuscript submitted to ACM 8 Wang et al. Table 4. Performance metrics across different models and datasets, with 95% confidence intervals. Dataset Model R...

    work page 2023

  3. [3]

    In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2010)

    Harald Steck, Training and testing of recommender systems on data missing not at random . In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2010). Association for Computing Machinery, New York, NY, USA, 713–722

    work page 2010

  4. [4]

    In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML 2016)

    Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims, Recommendations as Treatments: Debiasing Learning and Evaluation. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML 2016). JMLR.org, 1670–1679

    work page 2016

  5. [5]

    In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2022)

    Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Rui Zhang, and Jie Sun, A Generalized Doubly Robust Learning Framework for Debiasing Post-Click Conversion Rate Prediction . In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2022). Association for Computing Machinery, New York, NY, USA, 252–262

    work page 2022

  6. [6]

    In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)

    Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang,AutoDebias: Learning to Debias for Recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021). Association for Computing Machinery, New York, NY, USA, 21–30

    work page 2021

  7. [7]

    In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys 2020)

    Andrés Villa, Vladimir Araujo, Francisca Cattan, and Denis Parra,Interpretable Contextual Team-aware Item Recommendation: Application in Multiplayer Online Battle Arena Games. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys 2020)

    work page 2020

  8. [8]

    In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM 2020)

    Qilin Deng, Kai Wang, Minghao Zhao, Zhene Zou, Runze Wu, Jianrong Tao, Changjie Fan, and Liang ChenPersonalized Bundle Recommendation in Online Games. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM 2020)

    work page 2020

Show all 16 references
  1. [9]

    Zaman.Recommendation Applications and Systems at Electronic Arts

    Meng Wu, John Kolen, Navid Aghdaie, and Kazi A. Zaman.Recommendation Applications and Systems at Electronic Arts. In Proceedings of the Eleventh ACM Conference on Recommender Systems (RecSys 2017)

    2017

  2. [10]

    Workshop of ACM Conference on Recommender Systems (RecSys 2020)

    Cao, Lele & Asadi, Sahar & Biasielli, Matteo & Sjöberg, Michael.Debiasing Few-Shot Recommendation in Mobile Games. Workshop of ACM Conference on Recommender Systems (RecSys 2020)

    2020

  3. [11]

    Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020)

    Liu, Dugang and Cheng, Pengxiang and Dong, Zhenhua and He, Xiuqiang and Pan, Weike and Ming, Zhong A general knowledge distillation framework for counterfactual recommendation via uniform data. Proceedings of the 43rd International ACM SIGIR Conference on Research and Developm...

    2020

  4. [12]

    2024, Vertex AI

    Google. 2024, Vertex AI. Retrieved December 1, 2023 from https://cloud.google.com/vertex-ai, Manuscript submitted to ACM Addressing bias in Recommender Systems: A Case Study on Data Debiasing Techniques in Mobile Games 9

    2024

  5. [13]

    Marlin, Benjamin M., and Richard S. Zemel. Collaborative prediction and ranking with non-random missing data. Proceedings of the third ACM conference on Recommender systems (RecSys 2009)

    2009

  6. [14]

    In Proceedings of the 25th conference on user modeling, adaptation and personalization, pp

    Felício, Crícia Z., Klérisson VR Paixão, Celia AZ Barcelos, and Philippe Preux.A multi-armed bandit model selection for cold-start user recommendation . In Proceedings of the 25th conference on user modeling, adaptation and personalization, pp. 32-40. 2017

    2017

  7. [15]

    In 2017 IEEE International Conference on Big Knowledge (ICBK), pp

    Wang, Lu, Chengyu Wang, Keqiang Wang, and Xiaofeng He.Biucb: A contextual bandit algorithm for cold-start and diversified recommendation. In 2017 IEEE International Conference on Big Knowledge (ICBK), pp. 248-253. IEEE, 2017

    2017

  8. [16]

    Sitharama Iyengar, Larisa Shwartz, and Genady Ya Grabarnik.Online interactive collaborative filtering using multi-armed bandit with dependent arms

    Wang, Qing, Chunqiu Zeng, Wubai Zhou, Tao Li, S. Sitharama Iyengar, Larisa Shwartz, and Genady Ya Grabarnik.Online interactive collaborative filtering using multi-armed bandit with dependent arms. IEEE Transactions on Knowledge and Data Engineering 31, no. 8 (2018): 1569-1580....

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.