{"id":"2f13a7de-ddc5-4906-9b70-7e0531795067","arxiv_id":"2411.18716","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An industry case study showing that IPS, Doubly Robust learning, and AutoDebias produce inconsistent debiasing improvements on implicit-feedback mobile game recommender data.","lead":"This case study applies three known debiasing techniques to recommender data from mobile games, using both public explicit-feedback datasets and King's proprietary implicit-feedback data. It finds that debiasing gains are inconsistent on implicit feedback, and that methods requiring randomized data perform best but at high computational cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §6 claim that significant gains require randomized data is not established: no ablation isolates the randomized-data effect from extra training data, model complexity, and method choice. The large DR/AutoDebias gains could persist with a biased D_U.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing issue for the central claim is not the Set A test-set ambiguity alone; it is that the experiments never isolate the causal effect of randomized data. The conclusion in §6 requires showing that the presence of randomized data, not the presence of extra training data, model complexity, or estimator choice, drives the observed gains. Since no ablation replaces the randomized D_U with a biased sample of the same size, the large DR and AutoDebias improvements on Sets B and C are confounded. The reader identified related weaknesses—missing MF(uniform) baseline and unspecified Set A test set—but did not frame them as a causal-attribution problem. My proposed test is a single, cheap computational check that would settle the concern: if the gains persist when D_U is drawn from biased data, the headline claim is false; if they vanish, the claim is supported. This does not change the reader's conditional verdict, but it makes the required condition more precise: the authors should either run this ablation or soften the Section 6 conclusion to 'methods using additional data, including randomized collections, showed gains' rather than attributing the gains specifically to randomization.","tokens_in":8495,"tokens_out":8736,"duration_ms":80361,"concrete_test":"For Set B and Set C, rerun the exact DR and AutoDebias pipelines under identical hyperparameters, validation, and test splits, but construct D_U by sampling uniformly at random from the biased training set D_T instead of from the randomized collection. Compare RMSE, AUC, NDCG@5, Gini, and Entropy to the published Table 3/4 values. If the large gains persist with a biased D_U, the §6 conclusion is unsupported; if gains disappear, the randomized origin is causal. Repeat on COAT and yahooR3! as a sanity check, and also report the existing MF(uniform) row as a reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion in §6—'significant performance gains still depend on the challenging task of collecting randomized datasets'—is a causal statement about the value of randomized data. The experimental design in §4 and §5 does not isolate that factor. DR and AutoDebias differ from the IPS arm in at least three confounded ways: (i) they receive an additional training signal from the 5% randomized subset D_U (the §4 split reserves 5% of randomized data for D_U and 90% for D_Te); (ii) they are more complex estimators (imputation-based and meta-learning); and (iii) IPS is the only representative of the no-randomized-data family. The strongest supporting numbers—Set B DR RMSE -45.40% and Set C AutoDebias AUC +2.61% (Table 3)—therefore cannot be attributed specifically to the randomized origin of D_U. They might reflect extra data, model capacity, or hyperparameter tuning. This reading is consistent with the paper's own admission that internal results are 'less consistent' (§5.2) and with the fact that on Set C AutoDebias degrades RMSE by +64.50% while improving AUC by only 2.61%. A control arm replacing D_U with an equal-sized biased sample is needed before the headline claim can be accepted. The unspecified Set A test set (§4) further weakens the only IPS-only internal result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a case study of three debiasing techniques—Inverse Propensity Scoring (IPS), Doubly Robust learning (DR), and AutoDebias—applied to recommender systems for mobile games. The authors evaluate these methods against a biased matrix-factorization baseline (MF(biased)) on two public explicit-feedback datasets (COAT, YahooR3!) and three proprietary implicit-feedback datasets from King (Sets A, B, C), using RMSE, AUC, NDCG@5, Gini, Entropy, and training time. The central conclusion, stated in Section 6, is that significant performance gains still depend on collecting randomized datasets, since DR and AutoDebias, which require such data, show large improvements on some internal datasets while IPS, which does not, offers more limited gains.","tokens_in":8760,"tokens_out":3838,"duration_ms":36732,"significance":"If the main conclusion were established, the paper would provide practically useful guidance for mobile-game recommender systems: randomized data collection is a key enabler for debiasing gains, and methods that avoid such data (IPS) yield only modest improvements. The paper's strengths are its use of real-world implicit-feedback data from a commercial mobile gaming setting, the inclusion of confidence intervals from repeated runs in Table 4, and a reproducible artifact-based training pipeline. However, the headline causal claim is not isolated by the experimental design, and several evaluation choices need to be tightened before the conclusion can be accepted. The paper's internal admission in Section 5.2 that results are 'less consistent' across internal datasets is itself a signal that the cross-dataset generalization claim should be treated cautiously.","major_comments":[{"comment":"The central conclusion that 'significant performance gains still depend on the challenging task of collecting randomized datasets' is not established because the comparison confounds the randomized-data effect with other factors. In §4, DR and AutoDebias are trained with an additional 5% randomized subset D_U, while IPS and MF(biased) are not; moreover, DR and AutoDebias are more complex estimators, and IPS is the only no-randomized-data method tested. The large gains reported in Table 3 (e.g., Set B DR RMSE -45.40%) could therefore reflect extra training data, model capacity, or method choice rather than the randomized origin of D_U. A control arm that replaces D_U with an equal-sized biased sample, or a comparison against MF(uniform) from Table 4 as a randomized-data baseline, is needed before the §6 claim can be accepted.","section":"§4 and §6"},{"comment":"The test set for Set A is unspecified. Set A has no randomized data, so the 'data split strategy' described in §4—5% for D_U, 5% for D_V, 90% for D_Te—cannot be applied to it. Yet the paper reports IPS results on Set A in Table 3. It is unclear what data was used to compute RMSE, AUC, NDCG, Gini, and Entropy for Set A, and whether the evaluation set is biased or unbiased. This undermines the only IPS-only internal result and should be clarified.","section":"§4 and Table 3"},{"comment":"The percentage improvements in Tables 2 and 3 are reported without confidence intervals or significance tests, even though Table 4 provides absolute values with 95% confidence intervals. Several claimed improvements are small relative to the intervals; for example, Set B DR's NDCG@5 gain of 0.68% appears to be within the overlap of the confidence intervals for MF(biased) and DR in Table 4. The authors should report uncertainty for the relative improvements or clearly state which differences are statistically distinguishable.","section":"Tables 2 and 3"},{"comment":"Gini and Entropy are treated as measures of debiasing quality, but no evidence is given that improving these diversity metrics corresponds to reducing the bias identified in §3.1 (selection, exposure, and position bias). In Table 4, MF(uniform) and MF(biased) have nearly identical Gini and Entropy on COAT and YahooR3!, which suggests these metrics may be insensitive to the biases at play. The paper should either validate Gini/Entropy as debiasing measures using a known unbiased benchmark or temper the claims about 'better balance' and 'debiasing quality' in §5.2.","section":"§3.3, §5.2, and Table 4"}],"minor_comments":[{"comment":"There are several typos and inconsistent spellings: 'propreitery' in §3.2, 'NGCG@5' in §5.1, 'fine-tunning' in §3.3, and inconsistent 'AutoDebias'/'Autodebias' capitalization.","section":"Throughout"},{"comment":"The 'Training time (sec)' column in Table 3 actually shows percentage changes, not seconds; the heading should be 'Training time change' to match Table 2 and avoid confusion.","section":"Table 3 and §5.2"},{"comment":"The notation in the NDCG and Entropy formulas is unclear: 'rel_i' is not fully defined, and 'n is a total number of items u in a dataset' reads awkwardly; also the AUC formula's 'rank_{u,i}' should be defined as the rank of the positive item among all items for user u.","section":"§3.3"},{"comment":"The MF(uniform) results are reported in Table 4 but are never discussed in the text or used as a comparison baseline in Tables 2 and 3; including this model in the relative comparisons would strengthen the analysis of the value of randomized data.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry case study with real proprietary data, which is valuable. However, the headline claim about randomized data is not yet supported because of the confounded comparison; the authors should either add a control experiment (e.g., equal-sized biased D_U) or substantially soften the Section 6 conclusion. The unspecified Set A test set is a solvable but necessary clarification. If the authors can address the major comments, the paper could be a useful practical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain take: this is a legitimate industrial case study, and the empirical numbers are new. The authors apply three standard debiasing methods (IPS, DR, AutoDebias) to implicit-feedback data from King mobile games, report training time alongside accuracy, and use COAT and YahooR3 to sanity-check the pipeline. That fills a real gap: most debiasing work is on explicit feedback and public datasets. The paper is honest about results being inconsistent across internal datasets, and Table 4 does report confidence intervals on the absolute values. For a practitioner at a gaming company, this is a useful starting point.\n\nThe soft spots are real, and the stress-test note is right about the main one. The conclusion in Section 6 says significant gains depend on collecting randomized data, but the design never isolates that factor. DR and AutoDebias get the extra 5% randomized training subset D_U, they are more complex estimators, and IPS is the only no-randomized-data representative. A control arm that replaces D_U with an equal-sized biased sample is missing. So the 45% RMSE improvement from DR on Set B cannot be cleanly attributed to the randomized origin of the data. The paper even notes the internal results are \"less consistent,\" which is a clue that the headline overreaches.\n\nOther issues are worth naming but are smaller. The test set for Set A is never specified — Set A has no randomized data, so what was the evaluation benchmark? MF(uniform) appears in Table 4 for the other sets but is not used as a baseline in the percentage-change tables, which weakens the comparison. Gini and Entropy are treated as debiasing quality, but they are diversity/inequality metrics, not direct measures of bias. And the percentage improvements in Tables 2 and 3 lack confidence intervals, so we cannot tell whether the differences are noise.\n\nWho is this for? Applied researchers and engineers in gaming or similar implicit-feedback settings who want a quick map of the trade-offs. It is not a theoretical contribution, and the central causal claim should be read as a hypothesis, not a result.\n\nRecommendation: send it to peer review. The data is proprietary but the experiments are reproducible in structure, and the topic matters. The referee should push for the missing control arm, a defined test protocol for Set A, and a less sweeping conclusion.","headline":"Useful industrial case study with new numbers on mobile-game debiasing, but the headline claim about randomized data is confounded and the evaluation has gaps.","tokens_in":9285,"tokens_out":1721,"would_cite":true,"duration_ms":18110,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying three debiasing methods to mobile-game recommender data, this case study finds that the largest accuracy gains require a randomized holdout dataset, while the no-random-data method IPS yields small and inconsistent improvements.","keywords":["recommender systems","debiasing","implicit feedback","mobile games","inverse propensity scoring","doubly robust learning","AutoDebias","matrix factorization"],"falsifier":"Collect an independent, freshly randomized holdout for the dataset on which DR cut RMSE by 45.40%, train DR on the original biased set, and test on that new holdout; if the RMSE gain largely disappears, the reported improvement was tied to the particular split rather than to genuine debiasing. A second check is to specify a proper unbiased test set for the dataset lacking randomized data and see whether IPS's reported diversity gains and accuracy losses still hold there.","tokens_in":8281,"feed_emoji":"🎮","tokens_out":9190,"duration_ms":75472,"temperature":0.7,"pith_summary":"This paper is a case study of three data-debiasing techniques for recommender systems, tested on mobile-game purchasing data where feedback is implicit (purchase or no purchase). It first validates the techniques on two public explicit-feedback datasets, then applies them to three proprietary implicit-feedback datasets from a mobile-game company, using a matrix-factorization model trained on biased data as the baseline. The central finding is that methods requiring a randomized holdout dataset—Doubly Robust learning and AutoDebias—produce the largest accuracy gains, with Doubly Robust learning cutting RMSE by 45.40% on one proprietary dataset. Inverse Propensity Scoring, which does not require randomized data, improves diversity metrics but gives only limited and inconsistent accuracy gains. The paper concludes that significant performance gains still depend on collecting randomized datasets, despite the operational cost and user-experience risk of doing so.","feed_headline":"Randomized data unlocks 45% RMSE cut in mobile-game recommendations","feed_subtitle":"Doubly robust learning beat no-random-data IPS on mobile-game shop data, at heavy compute cost.","key_machinery":"The load-bearing mechanism is the randomized (uniform) holdout dataset $D_U$: a slice of data in which users received random recommendations, treated as unbiased ground truth for preference. The paper splits any available randomized data into 5% for training ($D_U$), 5% for validation, and 90% for testing, and this split feeds Doubly Robust learning's combination of propensity weighting with imputation and AutoDebias's meta-learning weight assignment. The baseline is matrix factorization trained only on biased data, and improvements are expressed as percentage changes in RMSE, AUC, NDCG@5, Gini index, and Entropy. The presence or absence of $D_U$ is what separates the large gains from the small ones.","core_discovery":"The central claim is empirical rather than methodological: on implicit-feedback mobile-game shop data, debiasing techniques that use a randomized dataset can yield large improvements over a biased baseline, while the technique that avoids randomized data cannot. The strongest reported result is a 45.40% reduction in RMSE from Doubly Robust learning on one proprietary dataset, together with a 7.07% AUC gain and gains in NDCG@5 and Gini. AutoDebias also improves accuracy metrics, especially on the two public datasets, but its training time grows dramatically. On the proprietary dataset without any randomized data, IPS improves diversity metrics but worsens RMSE and leaves NDCG@5 essentially flat. The authors' stated conclusion is that 'significant performance gains still depend on the challenging task of collecting randomized datasets.'","pith_inferences":["Because the paper identifies randomized data as the bottleneck, a natural next step it does not test is whether online exploration (e.g., contextual bandits) can supply the same unbiased signal more cheaply than a static randomized holdout.","A sharper test of whether DR genuinely debiases would measure exposure equity directly against the randomized benchmark, rather than inferring it from Gini and Entropy movement.","The 45.40% RMSE result on one dataset could be checked for robustness by collecting a second, independently randomized sample and testing on it; if the gain does not transfer, it may be an artifact of one split.","Combining IPS-style variance control with a small randomized sample might recover some of DR's gains when full randomized collection is unaffordable, a middle ground the paper does not explore."],"forward_implications":["Game developers who cannot run randomized holdouts should expect only modest accuracy gains from IPS, mainly in diversity metrics like Gini and Entropy.","Building a pipeline that collects randomized recommendation data is likely to matter more than the choice of debiasing model, since the two methods with access to such data dominate the accuracy improvements.","Doubly Robust learning on mid-sized implicit-feedback data can simultaneously improve accuracy and diversity, but the roughly 386% training-time increase on Set B means compute cost must be budgeted.","On the largest proprietary dataset, AutoDebias improved AUC and NDCG@5 but degraded RMSE, Gini, and Entropy relative to the biased baseline, so scale changes which debiasing method wins.","Implicit-feedback datasets produce less consistent debiasing results than explicit-feedback public datasets, suggesting that purchase/no-purchase signals weaken the transfer of these methods."],"supporting_citations":[{"why":"Supplies the inverse propensity scoring method and the COAT dataset, providing the no-randomized-data baseline and the first public validation set.","marker":"[4]"},{"why":"Supplies the Doubly Robust learning framework whose application yields the 45.40% RMSE reduction on one proprietary dataset.","marker":"[5]"},{"why":"Supplies AutoDebias, the bias-agnostic meta-learning method that achieves the largest accuracy gains on the public datasets.","marker":"[6]"},{"why":"Supplies the split of randomized data into 5% training, 5% validation, and 90% test, used for DR and AutoDebias in all experiments.","marker":"[11]"},{"why":"Supplies the YahooR3! dataset, the second explicit-feedback benchmark used to validate the methods before applying them to implicit feedback.","marker":"[13]"},{"why":"Supplies the bias taxonomy that justifies selecting IPS, DR, and AutoDebias for selection, exposure, and position bias.","marker":"[2]"},{"why":"Establishes the missing-not-at-random data problem that motivates why biased training data needs debiasing in the first place.","marker":"[3]"}],"fun_headline_variants":["45% RMSE cut in mobile-game recs needs randomized data","Debiasing without randomized data fails in game recs","Randomized data beats classic debiasing in mobile game recs","Doubly robust learning slashes RMSE 45% in game recommendations","AutoDebias boosts accuracy but at huge compute cost in games"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the randomized holdout data is an unbiased ground truth for what players actually prefer, and that Gini and Entropy movements really measure reduced bias; for the one dataset that has no randomized data, there is also no stated unbiased test benchmark, so its results rest on an unexamined evaluation choice.","fun_headline_variants_meta":{"raw":{"variants":["45% RMSE cut in mobile-game recs needs randomized data","Debiasing without randomized data fails in game recs","Randomized data beats classic debiasing in mobile game recs","Doubly robust learning slashes RMSE 45% in game recommendations","AutoDebias boosts accuracy but at huge compute cost in games"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1238,"prompt_tokens":887,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":503,"tokens_out":351,"duration_ms":102060,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:56:14.855849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect an independent, freshly randomized holdout for the dataset on which DR cut RMSE by 45.40%, train DR on the original biased set, and test on that new holdout; if the RMSE gain largely disappears, the reported improvement was tied to the particular split rather than to genuine debiasing. A second check is to specify a proper unbiased test set for the dataset lacking randomized data and see whether IPS's reported diversity gains and accuracy losses still hold there.","supporting_citations":[{"cited_title":"In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML 2016)","cited_arxiv_id":null,"evidence_quote":"Supplies the inverse propensity scoring method and the COAT dataset, providing the no-randomized-data baseline and the first public validation set."},{"cited_title":"In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the Doubly Robust learning framework whose application yields the 45.40% RMSE reduction on one proprietary dataset."},{"cited_title":"In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)","cited_arxiv_id":null,"evidence_quote":"Supplies AutoDebias, the bias-agnostic meta-learning method that achieves the largest accuracy gains on the public datasets."},{"cited_title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the split of randomized data into 5% training, 5% validation, and 90% test, used for DR and AutoDebias in all experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the YahooR3! dataset, the second explicit-feedback benchmark used to validate the methods before applying them to implicit feedback."},{"cited_title":"ACM Trans","cited_arxiv_id":null,"evidence_quote":"Supplies the bias taxonomy that justifies selecting IPS, DR, and AutoDebias for selection, exposure, and position bias."},{"cited_title":"In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2010)","cited_arxiv_id":null,"evidence_quote":"Establishes the missing-not-at-random data problem that motivates why biased training data needs debiasing in the first place."}],"review_version":1}