{"id":"987a9a9d-941a-452b-9025-a89f4962ed0c","arxiv_id":"2501.15816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AdaF2M2 combines feature masking with a state-aware adapter to reduce over-reliance on ID features, yielding small offline and online gains across five base recommendation models.","lead":"This paper proposes AdaF2M2, a recommendation framework that randomly masks features during training and adds a state-aware adapter that reweights features for different users and items. The authors report small but consistent offline gains and modest online A/B improvements in Douyin Music, with the largest relative gains on cold-start items.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline online gains (+1.37%/+1.89%) cannot be traced to the reported A/B tables, and the 95% confidence intervals in Table 1 are formatted in a way that contradicts the 'statistically significant' claim; this makes the central quantitative claim unverifiable as presented.","rationale":"The most load-bearing concern is not the counterfactual assumption discussed in Section 3.2, though that is a legitimate mechanism-level worry. The central claim is quantitative: AdaF2M2 yields cumulative +1.37% active days and +1.89% app duration. The evidence for this claim rests on online A/B tests, but the paper's reported confidence intervals are internally inconsistent, and the headline cumulative figures do not appear in any table or derivation. If the CIs are taken at face value as intervals for the relative improvement, the 'statistically significant' assertions in the text are contradicted; if they are intervals for the baseline metric, then the precision of the lift is unknown. Either way, the reader cannot verify the paper's most important empirical assertion. The missing DropoutNet baseline and counterfactual assumption are important secondary issues, but they do not directly threaten the existence of the reported lift. This concern is actionable: the authors can supply raw A/B data, correctly computed CIs, and the aggregation rule. Because the paper still shows consistent positive point estimates across many experiments, conditional acceptance remains the appropriate disposition; the flaw is in reporting and verifiability, not a demonstrated failure of the method. Hence the reader's CONDITIONAL verdict is unchanged, but the stated weakest assumption should be replaced with the statistical traceability concern.","tokens_in":16539,"tokens_out":5445,"duration_ms":50953,"concrete_test":"Request the user-level A/B logs for the ranking, retrieval, and cold-start experiments and the definition of 'cumulative improvement.' Recompute a paired bootstrap 95% CI for the relative lift of each row in Tables 1-4 and for the pooled cumulative lift. If the cumulative CIs include zero, or if the reported +1.37%/+1.89% cannot be reproduced from the per-experiment lifts (e.g., via a precision-weighted meta-analysis), the central quantitative claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim quantifies cumulative online improvements of +1.37% and +1.89% (abstract/conclusion), yet no table or text explains how these numbers are aggregated from the per-experiment lifts in Tables 1-4, which report, for example, +0.212% active days / +0.442% duration for ranking and +0.066%/+0.073% active days for retrieval. The bracketed values in Table 1 are written as [-0.063%, +0.063%] for a reported +0.212% lift: if these are 95% CIs for the relative improvement, they contain 0, contradicting the claim of statistical significance; if they are intervals for the underlying metric, the relative-lift CI is not reported. The same pattern appears in Tables 2, 3, and 4. Without a correct CI for the lift or a transparent aggregation of the cumulative effect, the evidence does not establish the headline improvement. This is a reporting/correctness issue, not a disagreement with the method's plausibility.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AdaF2M2, a model-agnostic framework for recommender-system feature modeling. It combines a feature-mask mechanism, which generates k randomly masked augmented copies of each sample and trains them with a task-oriented cross-entropy auxiliary loss, with a state-aware adapter that multiplies each feature embedding by a sigmoid-gated weight computed from active-day statistics, ID embeddings, embedding norms, and interaction counts. The framework is applied to five base models (FM, YouTube DNN, AFN, DCN-V2, EulerNet) and evaluated offline on MovieLens-1M and two industrial datasets (DouyinMusic-4B and -20B), as well as online through A/B tests in ranking, retrieval, and item cold-start tasks. The paper reports cumulative online improvements of +1.37% on user active days and +1.89% on app duration, and claims deployment in multiple applications of Douyin Group.","tokens_in":16805,"tokens_out":6866,"duration_ms":62534,"significance":"The contribution is empirical and engineering-oriented. If the reported gains hold up, AdaF2M2 is practically valuable because it is model-agnostic, imposes no additional serving latency (the mask forward passes are training-only), and shows consistent offline gains across five base models and two data scales. The inclusion of a public dataset alongside billion-scale industrial logs and online A/B tests gives the evaluation unusual breadth for a conference paper. However, the paper's central quantitative claims currently rest on incomplete statistical reporting: the confidence intervals for the online lifts are not interpretable as presented, and the cumulative headline numbers are not derivable from the tables. These issues are fixable in revision, but they are not merely cosmetic.","major_comments":[{"comment":"Table 1's caption says the square brackets are 95% confidence intervals for online metrics, but the Overall row reports Active Days +0.212% with interval [-0.063%, +0.063%]. If these brackets are CIs for the relative lift, the interval includes 0, which contradicts the claim in Section 4.2 that the improvement is statistically significant; if they are CIs for the underlying absolute metric, then the CI for the relative lift is not reported. The same pattern appears in Tables 2, 3, and 7, and Table 4 gives no intervals at all. Please state precisely what the intervals denote and report the CI (or a test) for the relative improvement itself, along with sample sizes and experiment duration.","section":"Table 1 (and Tables 2, 3, 7)"},{"comment":"The headline numbers +1.37% and +1.89% cumulative improvements on Active Days and Duration are repeated in the abstract, contributions, and conclusion, but no table or equation explains how they are aggregated from the per-experiment lifts in Tables 1-4. For example, Table 1 reports +0.212%/+0.442% for ranking, Table 3 reports +0.066%/+0.195% and +0.073%/+0.184% for two retrieval models, and Table 4 reports additional cold-start lifts. Without the aggregation formula, weighting, or experimental periods, the central quantitative claim cannot be verified or reproduced from the paper. Please document the cumulative calculation and report the component lifts with their uncertainties.","section":"Abstract / Section 5"},{"comment":"The text states 'We report the offline results via three random runs' and 'Statistically significant improvement is marked with bold font in the tables,' yet Tables 5 and 6 show only point estimates (RelaImpr values) with no standard deviations, confidence intervals, or p-values, and no bold markers are visible in the plain-text rendering. Relative AUC improvements of 0.13%-0.71% may be smaller than run-to-run variance. Please add dispersion measures or significance tests, and ensure the bold marking is present and defined.","section":"Section 4.3 / Tables 5-6"},{"comment":"The feature-mask mechanism explicitly relies on the counterfactual assumption that a high-active user with only the basic features available at the new-user stage would receive a prediction consistent with the same user's later-stage behavior. This assumption is named but not tested. A concrete way to test it is to compare masked-sample predictions against actual new-user labels for users who later become active, or to evaluate a variant trained only on unmasked samples on a new-user holdout. Without such evidence, the auxiliary masked loss remains a plausible but unvalidated mechanism for improving cold-start generalization.","section":"Section 3.2"},{"comment":"The state-aware adapter is built on four empirical signals (active days, ID embeddings, embedding norms, interaction counts), but the paper gives no evidence that these signals capture the state information that matters for feature weighting. The ablation in Tables 7-8 shows the adapter helps offline but not online in isolation, so the mechanism's contribution is not fully characterized. Please add ablations that remove or replace each state signal, or otherwise demonstrate that the chosen signals are sufficient for the claimed responsiveness.","section":"Section 3.3"}],"minor_comments":[{"comment":"The phrase 'The main contributions of our work are summarized into four folds' should be 'four aspects' or 'fourfold'; the current wording is nonstandard.","section":"Section 1"},{"comment":"The negative value for Dislike (-1.594% overall) is not discussed; please clarify the direction of improvement for constraint metrics and whether a decrease in Dislike is the desired outcome.","section":"Table 1"},{"comment":"The industrial dataset paragraphs report sampling time spans but not how the two DouyinMusic datasets differ in feature sets; a sentence on feature compatibility would help readers assess the effect of dataset size.","section":"Section 4.1"},{"comment":"The heatmaps have no color scale or numeric range for the reported average feature weights, which makes the comparison across features hard to interpret; the phrase 'the limitation of pages' also appears to be a leftover editing note.","section":"Figure 2"},{"comment":"The serving stage returns only the adapter prediction; it would be helpful to state explicitly that the [MASK] embeddings are not used at serving time, though this is implied in Section 3.4.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is essentially an industrial case study with a modest algorithmic novelty (masked feature augmentation plus state-conditioned feature weighting). The breadth of experiments is a genuine strength, and I do not think the novelty level alone blocks publication. The blocking issue is the unreportable statistics: the online CIs and the cumulative headline numbers need to be fixed or the claims need to be toned down. I would be comfortable with major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part. The framework is a clean combination of two known ideas: per-feature masking with a task-oriented auxiliary loss (rather than contrastive/SSL), and a state-aware adapter that takes empirical signals (active days, ID embedding norms, interaction counts) rather than just the feature embeddings themselves. That is new enough to be worth a look. The evaluation is broad: five base models, two industrial datasets at billion-sample scale, one public set, and online A/B tests in ranking, retrieval, and cold-start. If the numbers are right, this is a genuinely useful model-agnostic add-on with no serving latency cost.\n\nNow the problem. The headline quantitative claims are not verifiable from what is reported. The abstract and conclusion state +1.37% and +1.89% cumulative online improvements, but no table or text explains how these are aggregated from the per-experiment lifts like +0.212% active days for ranking and +0.066% for retrieval. And the 95% confidence intervals in Tables 1-4 and 7 are written as symmetric intervals centered at zero (e.g., [-0.063%, +0.063%] for a reported +0.212% lift). As formatted, they cannot be confidence intervals for the relative improvement; if they are intervals for the underlying metric, then the CI for the lift is simply absent. Either way, the 'statistically significant' claims are unsupported by the displayed numbers. The offline tables similarly show only relative AUC/UAUC changes with no error bars, even though the text says results come from three random runs and marks significance in bold; no bold appears in the tables.\n\nThe mechanism itself is plausible. The counterfactual assumption in Section 3.2 — that a model trained on masked features transfers to the new-user state — is the kind of assumption that could break, and the paper does not test sensitivity to mask probability or the auxiliary loss weight. The missing DropoutNet comparison is a real gap, since that is the closest prior method. But these are fixable issues, not a fundamental flaw in the core idea. The offline consistency across five models is a good sign.\n\nWho this paper is for: practitioners in industry who want a drop-in feature-mask and adapter module, and researchers working on cold-start or feature weighting. It is not a methods breakthrough but it is a solid engineering contribution that could become a useful baseline.\n\nI would send it to peer review — it deserves a serious referee — but the reviewer should insist on corrected interval reporting, an explicit aggregation formula for the cumulative gains, and a DropoutNet comparison. As it stands, the paper's own presented evidence does not establish the headline numbers.","headline":"Plausible industrial feature-mask/adapter framework with consistent small offline gains, but the online evidence as reported does not support the headline cumulative improvements.","tokens_in":17330,"tokens_out":3619,"would_cite":false,"duration_ms":32737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaF2M2 claims that randomly masking feature embeddings during training and reweighting features with a state-aware adapter lets recommendation models learn non-ID features and improve online engagement metrics such as active days and app…","keywords":["recommendation systems","feature representation learning","feature masking","state-aware adapter","long-tail distribution","cold-start","CTR prediction","model-agnostic framework"],"falsifier":"Re-run the online A/B test on the same DCN-V2 ranking task with a two-week horizon; if Active Days and Duration show no statistically significant gain, the central claim fails. Alternatively, construct a toy dataset where IDs perfectly determine the label and non-ID features carry no signal, then check whether the feature-mask auxiliary loss still improves test AUC; if it does, the claimed mechanism of learning non-ID features is not what drives the gain.","tokens_in":16355,"feed_emoji":"📈","tokens_out":5726,"duration_ms":50559,"temperature":0.7,"pith_summary":"Recommendation models trained on skewed real-world data tend to lean on user and item IDs, leaving age, gender, genre, and other non-ID features under-trained. AdaF2M2 is a model-agnostic wrapper that attacks this with two coordinated pieces: random feature masking during training, which forces the model to predict from non-ID features, and a state-aware adapter that re-weights features for new versus old users and hot versus cold items. The paper reports that adding the wrapper to five base models improves offline AUC and UAUC, and that online A/B tests in a music recommendation service show cumulative gains of +1.37% in user active days and +1.89% in app duration. If these results hold, the framework offers a deployment path for better cold-start and long-tail performance without changing the serving-time architecture.","feed_headline":"Masking IDs helps recommenders learn non-ID features","feed_subtitle":"A model-agnostic feature-mask plus state-aware adapter lifted active days and app duration in online A/B tests.","key_machinery":"The framework has two coordinated mechanisms. The feature-mask mechanism generates k augmented copies of each training sample by randomly replacing feature embeddings with distinct learned [MASK] embeddings, then runs each copy through the base model and supervises it with the original label, so the model learns to make predictions from non-ID features when IDs are masked. The state-aware adapter takes four empirical state signals (active days, concatenated ID embeddings, norms of ID embeddings, and interaction counts) and produces Sigmoid weights that rescale feature embeddings before prediction, allowing different features to matter for new versus old users and cold versus hot items. Keeping the two paths separate prevents the adapter's low weights from shrinking gradients for non-ID features.","core_discovery":"The paper's central claim is that the harmful effects of popularity bias on feature learning can be countered by training a base recommendation model on masked-feature augmented samples while separately learning per-feature weights from empirical user and item state signals. During training, each sample is copied k times, and in each copy every feature embedding is replaced by its own learned [MASK] embedding with probability drawn from the range [0.1, 0.5]; all copies are scored by the same base model with the same click or finish label, so the model must learn to use whatever non-masked features remain. The state-aware adapter takes active days, concatenated ID embeddings, ID-embedding norms, and interaction counts as input and produces Sigmoid weights applied to all feature embeddings before the final prediction. The mask path and the adapter path are trained with separate losses, so low adapter weights cannot starve non-ID embeddings of gradient. The paper claims that this combination, applied on top of FM, YouTube DNN, AFN, DCN-V2, and EulerNet, improves AUC and UAUC on public and on 4-billion- and 20-billion-sample industrial datasets and produces statistically significant online gains.","pith_inferences":["A plausible reading not tested in the paper is that the feature-mask auxiliary loss acts partly as a regularizer or noise-injection mechanism; comparing it with standard embedding dropout at matched compute would show how much of the gain is specific to masking IDs versus general noise robustness.","The four state signals are treated as sufficient, but the paper gives no evidence that they exhaust relevant user and item state; adding session context, content embeddings, or recency signals to the adapter could produce even sharper weights.","The counterfactual premise could be tested directly with logged data: for users with complete histories, compare a masked-feature prediction to the same user's actual early-stage logged behavior; divergence would suggest tuning the mask probability range or using state-conditioned masks.","The framework's success on engagement metrics suggests it could complement popularity-bias correction and debiasing methods, since it addresses the representation side of long-tail data rather than only re-ranking or sampling."],"forward_implications":["Feature-mask training should reduce reliance on user and item IDs, so cold-start users and items should get better predictions from age, gender, genre, and similar meta features.","Because adapter weights come from state signals rather than from the features themselves, the weight distribution should shift appropriately as a user moves from new to active or an item from cold to hot.","Serving latency is unchanged by the mask mechanism, since masked forwards run only during training; the deployed model is still a single forward pass with adapter-weighted embeddings.","The framework is model-agnostic, so any ranking or two-tower retrieval model that takes embeddings as input can be wrapped without redesigning the feature interaction layer.","If the reported online numbers generalize, a production recommender can expect engagement gains of a few tenths of a percent per model upgrade, which is substantial relative to typical production improvements."],"supporting_citations":[{"why":"Supplies the FM base model used in both retrieval experiments and offline comparisons.","marker":"[30]"},{"why":"Supplies the YouTube DNN two-tower architecture used as a base model for retrieval and ranking.","marker":"[9]"},{"why":"Supplies the AFN ranking baseline, representing adaptive feature-interaction models that learn weights from features themselves.","marker":"[8]"},{"why":"Supplies the DCN-V2 industrial ranking model on which the main online A/B test is run.","marker":"[44]"},{"why":"Supplies the EulerNet ranking baseline used to test universality across recent feature-interaction models.","marker":"[40]"},{"why":"Supplies the DropoutNet idea of masking ID embeddings, which the feature-mask mechanism directly extends.","marker":"[41]"},{"why":"Supplies MaskNet, the self-supervised mask-based comparison method against which the task-oriented loss is contrasted.","marker":"[49]"}],"fun_headline_variants":["Masking ID features helps recommenders learn non-ID signals","Feature mask and adapter lift Douyin recommendations","AdaF2M2 trains recommenders to ignore popularity bias","Masking user IDs boosts active days and app time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a prediction made from masked, non-ID features for a high-active user should match that same user's behavior in an earlier, less-active stage; if this counterfactual is wrong, the extra training signal teaches a distribution that does not match how the model is actually served.","fun_headline_variants_meta":{"raw":{"variants":["Masking ID features helps recommenders learn non-ID signals","Feature mask and adapter lift Douyin recommendations","AdaF2M2 trains recommenders to ignore popularity bias","Masking user IDs boosts active days and app time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1485,"prompt_tokens":1071,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":687,"tokens_out":414,"duration_ms":4465,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:55:43.824371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the online A/B test on the same DCN-V2 ranking task with a two-week horizon; if Active Days and Duration show no statistically significant gain, the central claim fails. Alternatively, construct a toy dataset where IDs perfectly determine the label and non-ID features carry no signal, then check whether the feature-mask auxiliary loss still improves test AUC; if it does, the claimed mechanism of learning non-ID features is not what drives the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the YouTube DNN two-tower architecture used as a base model for retrieval and ranking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DCN-V2 industrial ranking model on which the main online A/B test is run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FM base model used in both retrieval experiments and offline comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EulerNet ranking baseline used to test universality across recent feature-interaction models."},{"cited_title":"Nature communications 12, 1 (2021), 1992","cited_arxiv_id":null,"evidence_quote":"Supplies the DropoutNet idea of masking ID embeddings, which the feature-mask mechanism directly extends."}],"review_version":1}