{"id":"01ca0087-b733-4cd8-9cea-c22ef4fe6402","arxiv_id":"2502.18478","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding low-weight noisy copies of training examples improves Meta's ads ranking by about 0.1% to 0.3% relative Normalized Entropy, and slightly outperforms self-consistency regularization.","lead":"A Meta team proposes a simple regularization method that adds slightly perturbed copies of training examples with a lower weight in the loss, and reports small gains in a billion-scale ads ranking system. The evidence is a handful of internal experiments with tiny relative improvements and no released code or confidence intervals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 compares LSPR and SCR under different augmentation budgets: LSPR perturbs the entire batch and doubles batch size, while SCR perturbs only a small fraction, so the claimed 'consistently better' result may be an artifact of the unmatched setup rather than of the loss-balancing mechanism.","rationale":"I read the paper as making an empirical industrial claim: LSPR, a simple lambda-weighted noisy-copy loss, improves NE over production baseline and over SCR. The strongest support is Table 2 and the online paragraph. The most vulnerable part is not the method's plausibility—noise injection and consistency are established regularization ideas—but the attribution of the measured 0.03 pp advantage to the LSPR mechanism. The authors themselves state an asymmetric experimental protocol in Section 5.2.2, so the comparison is confounded even before considering missing variance estimates. This concern does not require assuming run-to-run noise is large; it holds even if every number in Table 2 is exactly reproducible, because the treatment includes a different augmentation budget. My proposed matched-pair test would settle whether the claimed LSPR-vs-SCR advantage survives. The reader's weakest assumption (noise/CI) overlaps with this, but I would name the unmatched setup as the prior concern. Verdict remains CONDITIONAL: method may be useful, but Table 2 as reported does not establish the central comparative claim.","tokens_in":11802,"tokens_out":4001,"duration_ms":36918,"concrete_test":"Re-run the final-stage ranker comparison in Table 2 as a matched-pair experiment: fix the optimizer, learning-rate schedule, number of steps, and the perturbation fraction to be identical for SCR and LSPR (e.g., both methods perturb 100% of each batch, or both perturb the same small fraction), changing only the auxiliary loss form or weighting. If LSPR's NE advantage over SCR does not persist at a level above run-to-run variance (report at least 5 seeds per cell), the Section 5.2.2 result is confounded by augmentation budget.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is not just missing error bars; Table 2 is an uncontrolled comparison. Section 5.2.2 says that for LSPR 'we perturbed the entire batch each time, leading to doubled batch size,' whereas for SCR 'we only perturbed a small fraction of points in each batch.' The headline comparison therefore varies two things at once: the loss-balancing rule and the augmentation budget / effective batch size. LSPR's advantage over SCR in Table 2 (0.13 vs 0.10, 0.11 vs 0.08, 0.10 vs 0.07) is only 0.03 percentage points of relative NE. If even part of that gap comes from seeing twice as many perturbed examples per step, the claim that LSPR 'performs consistently better compared to SCR' is not supported. Section 4.4 also says the loss weight and perturb details 'will be added to the final version,' so the setup cannot be reconstructed; no confidence intervals or repeat runs are given for Table 2, and the online 0.1%-0.2% claim is reported without launch statistics. The numerical analysis in Section 5.1 does not fix this because it is a small synthetic linear-model simulation with a different setup (MSE, not NE; no production architecture).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Loss-Balanced Small Perturbation Regularization (LSPR), a perturbation-based regularization method that adds noisy copies of training examples to the training batch and down-weights them in the supervised loss, in contrast to Self-Consistency Regularization (SCR), which adds an auxiliary MSE consistency loss. The authors report offline relative Normalized Entropy (NE) gains of 0.1%–0.3% for SCR and 0.07%–0.13% for LSPR over a production baseline in a multi-stage industrial ads ranking system, and claim an online top-line improvement of 0.1%–0.2% from LSPR. They also present a numerical analysis on a small linear model (Section 5.1) suggesting that LSPR aligns better with the optimal weights than SCR. The paper's central claim is that LSPR consistently outperforms SCR across data-availability settings and is the first perturbation-based regularization deployed in a billion-scale ads ranking system.","tokens_in":11994,"tokens_out":3020,"duration_ms":28693,"significance":"If the empirical claims are correct, the contribution is practically significant: a simple, architecture-agnostic regularizer that yields 0.1%–0.2% relative NE gains at billion-user scale is valuable for industrial recommender systems, and the paper provides a clear algorithmic description (Algorithm 1, Eq. 2) that others could implement. The industrial deployment description and the multi-stage integration details are useful to practitioners. However, the evidence presented is currently too weak to establish the central superiority claim: the headline comparison in Table 2 is an uncontrolled experiment, no uncertainty quantification is provided for the small offline differences, and the numerical analysis in Section 5.1 is a synthetic simulation rather than a derivation or a production-scale evaluation. The claimed novelty of being the 'first' industrial-scale deployment is also sensitive to the exact scope of related work (e.g., Yao et al., 2021, report large-scale item recommendations with perturbation-based SSL).","major_comments":[{"comment":"The headline comparison between LSPR and SCR is uncontrolled. The text states that for LSPR 'we perturbed the entire batch each time, leading to doubled batch size,' whereas for SCR 'we only perturbed a small fraction of points in each batch.' Thus Table 2 simultaneously varies the loss-balancing mechanism and the augmentation budget/effective batch size. The 0.02–0.03 percentage-point gaps between LSPR and SCR (0.13 vs 0.10, 0.11 vs 0.08, 0.10 vs 0.07) could plausibly be explained by the larger number of perturbed examples seen by LSPR, rather than by the loss-balancing rule itself. To support the claim that LSPR 'performs consistently better compared to SCR,' the authors should report a controlled comparison with matched perturbation budgets (e.g., equal numbers of perturbed examples per step or equal total compute), or add ablations that isolate the loss-balancing component.","section":"Section 5.2.2, Table 2"},{"comment":"The offline gains and the online claim lack uncertainty quantification. The relative NE differences in Table 2 are 0.07%–0.13% relative to baseline, which are small enough to be within run-to-run noise for industrial training pipelines, yet the paper reports no confidence intervals, no repeated runs, and no significance tests. Likewise, the online claim of '0.1% to 0.2% relative improvement' is described only qualitatively ('consistently across multiple launches') with no launch statistics, segment-level variance, or guardrail metrics. Without these, the central empirical claim that LSPR is consistently better than SCR is not statistically supported.","section":"Table 2 and Section 5.2.3"},{"comment":"The numerical analysis does not provide a derivation of LSPR's advantage; it writes down the LSPR and SCR gradient updates as definitions and then simulates a two-layer linear model with MSE loss and selected hyperparameters (omega in {0.1, 0.9}, lambda in {0.001, 1}, eta = 1.4). Because the simulation uses a different loss (MSE) and architecture than the production setting (NE on a large ranking model), and because the hyperparameters are chosen by the authors, Figure 4 does not independently establish that LSPR outperforms SCR in the real system. The authors should either provide an analytic characterization of when LSPR dominates SCR (e.g., in terms of the bias-variance trade-off of the gradient updates) or base the superiority claim on controlled real-data experiments with proper ablations.","section":"Section 5.1, Eqs. (12)–(15)"},{"comment":"Reproducibility is severely limited by missing hyperparameter details. Section 4.4 states that the dense-feature perturbation distribution, sparse-feature dropout rate, and loss weight values 'will be added to the final version of our paper.' Without these values, and without any description of the search ranges or the final chosen settings, the experiments in Table 2 cannot be reconstructed. Since the central claim is empirical and the gains are small, the exact hyperparameter choices are load-bearing and must be reported in the main text or an appendix.","section":"Section 4.4"}],"minor_comments":[{"comment":"There are grammatical errors that should be corrected, e.g., 'We have successfully demonstrate' in the abstract and 'weight them down in the the loss term calculation' in Section 1.","section":"Abstract, Section 1"},{"comment":"The text refers to 'SCP weight updates' when the intended acronym is SCR; this typo appears twice and should be fixed.","section":"Section 5.1, Eqs. (14)–(15)"},{"comment":"The sentence defining the LSPR loss, 'where ŷ′_i is the model's prediction on the perturbed input x′_i traditional supervised loss function,' is incomplete and should be rewritten.","section":"Section 4.2, Eq. (2)"},{"comment":"The caption says 'ω denotes noise sample weight,' but in Eqs. (12)–(15) ω appears as the perturbation magnitude scaling the noise z; the notation and terminology should be aligned.","section":"Figure 4 caption"},{"comment":"Several references are malformed or incomplete, e.g., 'Wang F, Wang Y, Li D, Gu H, Lu T, Zhang P, and Gu N' and 'LeCun Y, Bengio Y, and Hinton G'; these should be formatted consistently with the rest of the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong industrial-practice candidate but the central scientific claim—that LSPR consistently beats SCR—is currently supported by an uncontrolled comparison and no statistical evidence. I would be willing to accept after the authors provide controlled experiments with matched augmentation budgets, confidence intervals or repeated runs, and the missing hyperparameter details. The 'first of its kind' novelty claim should also be carefully scoped against prior large-scale SSL work in recommender systems, since the current wording may overstate the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new piece here is LSPR, which is simply training with noisy copies of examples, downweighted by lambda in the loss, applied at billion-scale in Meta's ads ranking. That is genuinely new as a deployment claim; the algorithm itself is a weighted version of Bishop's training-with-noise, which they cite. The paper is worth a look because it shows perturbation-based regularization can survive industrial-scale sparse features and production constraints.\n\nWhat it does well: the method is simple to state, the system description gives some idea of where such regularization helps across retrieval, early, and final ranking, and the offline gains of 0.1-0.3% relative NE are consistent in direction with the online claim of 0.1-0.2%. The authors also cite Bishop and acknowledge the lineage, which is more than some industrial papers do.\n\nThe soft spots are real. The main comparison in Table 2 is not controlled: LSPR perturbs the entire batch and doubles batch size, SCR perturbs only a small fraction, so the 0.03 percentage point gap between LSPR and SCR could just be more augmentation per step. The paper reports no confidence intervals, no repeated runs, and no significance tests for these differences. Hyperparameters are explicitly deferred to \"the final version,\" so the setup cannot be reproduced from the text. The online results are given without launch statistics. The numerical analysis in Section 5.1 is a simulation with chosen lambda and omega, not a derivation, and it uses MSE on a linear model while the real experiments use NE on a deep network, so it is illustrative at best. The \"first of its kind\" claim is also overstated given that noise injection has a long history; what is first is the specific deployment, not the idea.\n\nNone of this proves the method does not work. The central claim, that a downweighted noise term helps at scale, is plausible and the offline direction matches the online direction. But the evidence as written does not separate LSPR's loss-balancing mechanism from the extra perturbed data it consumes.\n\nGive this paper a serious referee. It reports a real deployed change to a billion-scale system, and the confound in Table 2 is exactly what a good reviewer should force the authors to untangle. I would not cite it in its current form because the method is not novel enough and the results are not well enough supported. I would bring it to a reading group, though, as a case study in how industrial deployment papers can under-report comparison setup.","headline":"LSPR is a minor twist on training-with-noise with a plausible industrial story, but the headline comparison against SCR is confounded and the evidence is too thin to accept as-is.","tokens_in":12635,"tokens_out":1851,"would_cite":false,"duration_ms":16851,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Perturbation-based regularization, applied for the first time at billion-user scale, improves industrial ads ranking.","keywords":["ads ranking","perturbation-based regularization","self-consistency regularization","loss-balanced small perturbation regularization","normalized entropy","industrial recommender systems","data augmentation","click-through rate prediction"],"falsifier":"Re-run the final-stage ranker experiment from Table 2 multiple times with different random seeds and data subsamples, computing the mean and confidence interval of the relative NE gain for LSPR versus SCR at each data-availability level; if LSPR's gain is not consistently positive across repetitions, the paper's central claim that LSPR performs consistently better than SCR collapses.","tokens_in":11535,"feed_emoji":"📈","tokens_out":6584,"duration_ms":52044,"temperature":0.7,"pith_summary":"The paper is trying to establish that perturbation-based regularization—adding small noisy copies of training examples—can work at the scale of a billion-user ads ranking system, and that a specific new variant, Loss-Balanced Small Perturbation Regularization (LSPR), beats the established Self-Consistency Regularization (SCR) baseline. LSPR adds perturbed examples to the training batch but down-weights their supervised loss by a factor $\\lambda < 1$, instead of adding an auxiliary loss that pushes perturbed and original predictions closer together. The paper reports relative Normalized Entropy gains of about 0.1% to 0.3% offline with SCR, and consistent 0.10% to 0.13% relative NE gains for LSPR over the production baseline in the final-stage ranker, with LSPR outperforming SCR at every data-availability level. It also reports 0.1% to 0.2% top-line improvements in online launches. A sympathetic reader would care because the method is simple enough to deploy across many production deep learning models and is claimed to generalize across ranking stages, surfaces, and traffic.","feed_headline":"A noise trick lifts billion-scale ad ranking by 0.1-0.2%","feed_subtitle":"Down-weighting perturbed training copies beats self-consistency in offline and online ad-ranking tests.","key_machinery":"The load-bearing object is the LSPR loss, a weighted sum of the ordinary supervised loss on clean examples and the same supervised loss on perturbed copies: $L_{\\mathrm{LSPR}}(y, \\hat{y}, \\hat{y}') = L_{\\mathrm{supervised}}(y, \\hat{y}) + \\lambda L_{\\mathrm{supervised}}(y, \\hat{y}')$, with $\\lambda < 1$. Perturbations are drawn from the same distribution class as the features—Gaussian noise for dense features and dropout for sparse features—so the noisy copy keeps the original label. The mechanism differs from SCR in that LSPR never tries to make clean and perturbed predictions equal; it treats the perturbed point as a down-weighted supervised example, which the paper argues disturbs learning dynamics less and doubles the effective batch size. This loss is what carries both the offline NE gains and the claimed online improvements, and the paper's linear-model analysis uses it to show better alignment with the optimal weight $W^*$ than SCR.","core_discovery":"The central claim is that a regularizer which simply copies each training example, adds feature-consistent noise (Gaussian perturbation to dense features and dropout to sparse features), and includes the noisy copy in the supervised loss with a scalar weight $\\lambda < 1$ improves a billion-scale ads ranking system. The paper's stated discovery is that this LSPR loss, $L_{\\mathrm{LSPR}} = L_{\\mathrm{supervised}}(y, \\hat{y}) + \\lambda L_{\\mathrm{supervised}}(y, \\hat{y}')$, performs consistently better than Self-Consistency Regularization, which instead enforces consistency between the predictions on original and perturbed inputs through an MSE auxiliary term. In the final-stage ranker experiments, LSPR yields 0.13%, 0.11%, and 0.10% relative NE gains over the production baseline at 33%, 66%, and 100% data availability, versus SCR's 0.10%, 0.08%, and 0.07%; the paper also reports that the offline gains carry over to online top-line metrics at 0.1% to 0.2%. The numerical analysis with linear models is offered as evidence that LSPR's gradient updates align better with the optimal model weights and reach lower weight-space error than SCR.","pith_inferences":["Editorial inference: the paper's numerical analysis only covers linear models, so an immediate extension is to test whether LSPR's better alignment with optimal weights on linear systems also holds for the deep final-stage ranker.","Editorial inference: because LSPR only changes how perturbed copies are weighted, it could be combined with other augmentation families such as feature masking or mixup-style interpolation; the paper only tests Gaussian noise and dropout.","Editorial inference: the reported gains are small (0.07% to 0.13% relative NE), and without confidence intervals or repeated-run statistics, a reader should treat the offline advantage as a point estimate rather than a proven difference; this is a gap the paper does not address.","Editorial inference: the paper mentions that perturbation-dependent weights are a future direction; a natural testable extension is to weight perturbed copies by their noise magnitude or by feature importance rather than a uniform $\\lambda$."],"forward_implications":["LSPR can be dropped into any supervised deep learning model by cloning each batch, perturbing features, and down-weighting the clone's loss; no auxiliary head or consistency loss is needed.","At 33%, 66%, and 100% of training data, LSPR's relative NE gain over the production baseline is consistently larger than SCR's, so the advantage holds as data becomes more available.","Because retrieval-stage and early-stage ranking models also improved under perturbation-based regularization, the technique applies across the full multi-stage ranking pipeline, not just the final ranker.","If the online 0.1% to 0.2% top-line improvements are stable across multiple launches, the method is usable in continuously trained production environments, not only offline retraining.","The improvement is measured relative to production baselines that were already state-of-the-art, so the resulting ranking quality gain is on top of mature industrial ads systems."],"supporting_citations":[{"why":"Supplies the classical result that training with noise is equivalent to Tikhonov regularization, the theoretical anchor for LSPR's noise-based perturbation.","marker":"Bishop (1995)"},{"why":"Defines the self-consistency regularization baseline that LSPR is compared against.","marker":"Sinha and Dieng (2021)"},{"why":"Provides the linear feedforward network learning-curve framework used for the numerical analysis comparing LSPR and SCR.","marker":"Werfel et al. (2003)"},{"why":"Establishes embedding-level perturbation and feature masking for large-scale recommender systems, the template for the data augmentation strategy.","marker":"Yao et al. (2021)"},{"why":"Defines Normalized Entropy, the offline metric used to measure all reported gains.","marker":"He et al. (2014)"},{"why":"Supplies dropout as the sparse-feature perturbation technique used in both SCR and LSPR.","marker":"Srivastava et al. (2014b)"}],"fun_headline_variants":["Loss-balanced noise beats self-consistency in ads ranking","New regularizer lifts billion-scale ad ranking up to 0.2%","Perturbation trick outperforms self-consistency for ads models","LSPR: simple noise addition improves industrial ads ranking","Weighted noise copies boost ad ranking in billion-parameter models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported relative Normalized Entropy differences of 0.07% to 0.13% over the production baseline are real and not run-to-run noise; the paper gives no confidence intervals, repeated runs, or significance tests for these differences.","fun_headline_variants_meta":{"raw":{"variants":["Loss-balanced noise beats self-consistency in ads ranking","New regularizer lifts billion-scale ad ranking up to 0.2%","Perturbation trick outperforms self-consistency for ads models","LSPR: simple noise addition improves industrial ads ranking","Weighted noise copies boost ad ranking in billion-parameter models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1447,"prompt_tokens":1057,"completion_tokens":390,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":673,"tokens_out":390,"duration_ms":4139,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T04:09:14.905068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the final-stage ranker experiment from Table 2 multiple times with different random seeds and data subsamples, computing the mean and confidence interval of the relative NE gain for LSPR versus SCR at each data-availability level; if LSPR's gain is not consistently positive across repetitions, the paper's central claim that LSPR performs consistently better than SCR collapses.","supporting_citations":[],"review_version":1}