{"id":"485c892a-ea91-4ced-8367-91b6d8eacf17","arxiv_id":"2412.00430","paper_version":6,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors propose a 'Performance Law' for sequential recommendation models that predicts HR and NDCG from model layers, embedding dimension, and number of tokens divided by Approximate Entropy, then uses the fitted formula to search for optimal configurations.","lead":"This paper fits an equation that predicts hit-rate and NDCG for sequential recommendation models from model depth, embedding size, and a new data-quality score based on Approximate Entropy. The authors use this fitted curve to pick supposedly optimal model configurations without training every option.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq 7 is never tested out-of-sample; with 7+ free parameters fitted per dataset, in-sample R^2 does not establish a predictive law, and Table 2's fitted w5 varies by orders of magnitude and sign across datasets.","rationale":"The reader identified the unproven linear performance-loss mapping as the weakest assumption. That is a real gap, but it is not the most load-bearing failure of the paper. Eq 7 does not actually use the linear map; it is a heuristic parametric form taken from the authors' prior work. The decisive issue is empirical: the Performance Law is never subjected to out-of-sample prediction. All reported correlations are in-sample fits, and the fitted parameters in Table 2 are unstable across datasets, with w5 ranging over many orders of magnitude and flipping sign. Such instability directly contradicts the claimed universality of Eq 7. A concrete held-out test, as described above, would settle whether the law predicts at all. If it does not, the verdict should remain REJECT; if it does, the paper would still need to address the missing derivation, but the central empirical claim would be substantially supported. I therefore agree with the reader's rejection while giving a different and, in my view, more fundamental reason: the central claim is not actually tested in the paper.","tokens_in":15430,"tokens_out":3500,"duration_ms":37604,"concrete_test":"Fit Eq 7 on one dataset using only configurations with N in {2,4,8} and demb in {64,128,256}, then train the model at held-out configurations such as N=6, demb=192 and N=12, demb=384 and compare predicted vs actual HR@10 and NDCG@10, reporting absolute errors and whether the predicted optimum matches the empirical optimum. Additionally, fit all parameters on three public datasets and predict the fourth; if w5 must be re-fit per dataset to obtain any reasonable prediction, the law is not predictive across dataset scales.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Eq 7 predicts HR@10 and NDCG@10 from N, demb, #Tokens, and ApEn. The reported validation is entirely in-sample: Eq 7 is fitted on the same runs whose metrics are plotted, and the R^2/figures measure goodness of fit, not predictive accuracy. No configuration is held out, and no dataset is predicted from fits on other datasets. This is not a minor omission: Table 2 shows the fitted exponent w5 varying from 0.0595 to 70.35 for HR and from -29.95 to 18.46 for NDCG across datasets, sometimes negative. A law with dataset-dependent parameters that change sign and magnitude is not a law that 'enables accurate predictions across various dataset scales'. The theoretical route is also indirect: Section 3.3 states 'Performance = 1 - kL', but k is never used, and Eq 7 is imported by analogy from Wu and Tang 2024 rather than derived from the theorems. Theorem 3.5 gives only a bound, and its proof requires Zt <= 1, which is asserted rather than established. The ApEn data-quality measure is a plausible contribution, but it does not rescue an unvalidated performance law. If Eq 7 cannot predict held-out configurations, the central claim fails regardless of in-sample fit quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a 'Performance Law' for sequential recommendation models, proposing that ranking metrics HR@10 and NDCG@10 can be predicted from model depth N, embedding dimension d_emb, token count, and Approximate Entropy (ApEn) through Eq. (7), with effective data parameter D' = #Tokens · ApEn' = #Tokens/ApEn. The authors give theorems linking ApEn to data scale and loss to model dimensions, fit Eq. (7) on four datasets, and use the fitted law to search for optimal configurations and compare architectures.","tokens_in":15907,"tokens_out":7803,"duration_ms":73273,"significance":"If the Performance Law were valid, it would be practically valuable: a small number of training runs plus token count and ApEn could predict ranking metrics and identify compute-optimal configurations without exhaustive training. The paper also makes a plausible contribution in proposing ApEn as a data-quality measure for recommender data, and it reports a broad experimental matrix across four datasets and several architectures. However, the theoretical derivation has load-bearing gaps and the empirical validation is entirely in-sample, so the central predictive claim is not established by the evidence in the manuscript.","major_comments":[{"comment":"Lemma 3.2 is false for r<1, the regime used by the paper (ApEn is computed with r=0). With two-point distributions x=(0.5,0.5), y=(0.9,0.1) and r=0.5, the left side is -1 and the right side is approximately -1.737, violating the claimed inequality. The proof reverses the inequality after dividing by the negative number ln r, and it invokes ln(y/x)≤y/x−1 in the wrong direction. Since Theorem 3.3 relies on this lemma to conclude that data scale is bounded by #Tokens·ApEn', the data-parameter part of Eq. (7) is not supported.","section":"Section 3.3, Lemma 3.2, Eqs. (10)-(11)"},{"comment":"Even apart from Lemma 3.2, the proof of Theorem 3.3 contains unjustified steps. The requirement that 'p(si) and pxy are minimal' is undefined and is an assumption rather than a property of the data. In Eq. (19), the inequality log(1/p_xy)≥1 is not valid for arbitrary transition probabilities (it requires p_xy≤1/e), and the claim that 'each term has a minimum value of 1/(|U|·Smax)' is asserted without proof; states with zero observed transitions have p_xy=0. The bound in Eq. (21) that the entropy of the sequence distribution is at least S_max is also not established. The theorem therefore does not prove D=#Tokens·ApEn'.","section":"Section 3.3, Theorem 3.3, Eqs. (15)-(21)"},{"comment":"The paper states 'we assume a linear relationship between model performance and loss, expressed as Performance = 1−kL'. No derivation, measurement, or citation supports this fixed linear map between sampled-softmax training loss and thresholded ranking metrics HR@10/NDCG@10. The constant k is never fitted or used anywhere in the subsequent equations, and the arguments in Section 3.3 only concern loss, so the transfer from loss scaling to ranking-metric scaling has no demonstrated basis.","section":"Section 3.3, linear performance-loss assumption"},{"comment":"Theorem 3.5 does not justify the functional form of Eq. (7). The proof assumes Zt≤1, which is asserted to be 'common in the overfitting phenomenon' but is not proven for the models studied. Even if Eq. (24) held, it is a bound of the form log(Zt)+1/Zt between two expressions, not an exact additive identity; the final sentence 'Analogously to (Wu & Tang, 2024), we factorize the product into the structure of Equation 7' supplies the missing step by reference to a paper by two of the present authors, with no derivation. The same symbol p1 appears in both the N and d_emb terms of Eq. (7), while the fitted examples in Eqs. (29)-(30) contain additional constants, so even the notational form is inconsistent.","section":"Theorem 3.5, Eq. (24), Eq. (7)"},{"comment":"All reported validation of Eq. (7) is in-sample: the same runs used to fit the seven or more free parameters are those whose R² and fitted curves are displayed, and no configuration is held out. Table 2 shows the fitted exponent w5 ranging from 0.0595 to 70.35 for HR@10 and from -29.95 to 18.46 for NDCG@10 across datasets, with sign changes. A law whose fitted parameters vary this widely across datasets does not support the abstract's claim that the method 'enables accurate predictions across various dataset scales and model sizes'; out-of-sample prediction of held-out configurations or datasets is required and is not reported. Section 5.4.1 similarly reports chosen optima but does not compare predicted metric values with later measurements.","section":"Section 5.3, Table 2"}],"minor_comments":[{"comment":"The parameter p1 appears in both the N and d_emb terms but likely is intended to be two different parameters; the fitted equations (29)-(30) use different constants, so the notation in Eq. (7) should be corrected.","section":"Eq. (7)"},{"comment":"The caption of Fig. 2 refers to 'KuaiRand-1K' while Table 1 and the text refer to 'KuaiRand-pure'; please make the dataset names consistent across the manuscript.","section":"Figures 2-5 and Table 1"},{"comment":"Several references are incomplete; for example, 'Wang, H., et al.' lacks a year and venue, and the ICML submission template header remains in the arXiv version.","section":"References"},{"comment":"The text states 'both w1 and w2 are positive values,' but Table 4 reports w3 and w4, not w1 and w2; the relationship between the table and the statement needs clarification.","section":"Section 5.4.2, Table 4"},{"comment":"The column heading '1/DL' is not defined in the text or caption, and it is unclear how it relates to the fitted data parameter D' discussed later.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"Given that the central predictive claim is supported only by in-sample fits and the theoretical route contains a false lemma, I cannot recommend publication. The ApEn data-quality idea may be worth pursuing, but the current manuscript would need a substantive rewrite and genuine out-of-sample evaluation to establish a predictive law."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's headline claim is a \"Performance Law\" that predicts HR@10 and NDCG@10 from model layers, embedding dim, token count, and an Approximate Entropy data-quality term. That claim isn't supported by the experiments, because every fit is in-sample. The useful piece is the ApEn adjustment, which is plausible and worth a careful look.\n\nWhat's genuinely new: applying the Wu & Tang 2024 performance-law functional form to sequential recommendation, replacing parameters with layers/embedding dim, and using ApEn' = 1/ApEn to fold data quality into the effective token count. The empirical observation that the fitted data parameter D' tracks Tokens/ApEn roughly linearly across four datasets is the most interesting thing in the paper. If that relationship is stable, it's a useful engineering heuristic.\n\nWhere it falls short: Eq 7 is not derived. The theorems are meant to justify it, but Lemma 3.2's inequality reverses for r < 1 (the regime they use), Theorem 3.3 relies on a \"minimum probability\" step that's just asserted, and Theorem 3.5 assumes Zt ≤ 1 without argument. More importantly, there's no out-of-sample validation anywhere. The law has seven-plus free parameters; fitting it to the same runs whose metrics you plot and reporting R² doesn't show prediction. Table 2 makes this vivid: fitted w5 for HR ranges from 0.0595 to 70.35 and even goes negative for NDCG. A law whose exponent flips sign across datasets is not a law that \"enables accurate predictions across various dataset scales.\" The Performance = 1 - kL link to loss is also asserted, not measured, and k never appears again.\n\nThat said, this isn't a throwaway. The ApEn idea is sensible, and a proper held-out test could turn it into something citable. The paper would be much stronger if it held out a few configurations per dataset, or fit on three datasets and predicted the fourth, and reported errors, not just R². A serious referee could push the authors to do that, so I'd send it to review rather than desk reject, but with the expectation of major revision. The reader's rejection is fair on the current evidence; it's just not the end of the story for the core idea.","headline":"A promising data-quality idea wrapped in a predictive law that isn't actually tested; the in-sample fits don't carry the weight of the claims.","tokens_in":16370,"tokens_out":2968,"would_cite":false,"duration_ms":28740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single formula predicts recommender ranking performance.","keywords":["sequential recommendation","scaling laws","performance law","approximate entropy","data quality","hit rate","NDCG","transformer recommendation"],"falsifier":"On a held-out dataset, compute ApEn and token counts, fit Equation (7) using only a few small runs, then predict HR@10 and NDCG@10 at larger depths and embedding sizes; if the predicted values fall outside the fitted residual band, or if the measured loss and HR pairs are visibly non-monotone, the law fails.","tokens_in":15234,"feed_emoji":"📈","tokens_out":8958,"duration_ms":77636,"temperature":0.7,"pith_summary":"This paper tries to establish a Performance Law for sequential recommendation models: a closed-form equation that predicts ranking metrics (HR@10 and NDCG@10) from the number of transformer layers, the item embedding dimension, the number of training tokens, and a data-complexity measure called Approximate Entropy. The practical payoff would be large: a few small training runs plus cheap data statistics would let developers forecast accuracy, locate the best model configuration, and avoid exhaustive sweeps. The authors argue that ordinary scaling laws fail for recommenders because loss and ranking performance diverge and because data quality matters as much as data quantity. They fit the law on four datasets and report that it tracks HR@10 and NDCG@10 across dataset scales and model sizes, including the U-shaped decay that appears when models grow too large.","feed_headline":"One Performance Law predicts recommendation accuracy across sizes","feed_subtitle":"A few small training runs plus data complexity can forecast HR@10 and NDCG@10.","key_machinery":"The load-bearing object is Equation (7), a parametric curve with a depth term in $N$, an embedding term in $d_{\\mathrm{emb}}$, and a data term in $D' = \\#\\mathrm{Tokens}\\cdot\\mathrm{ApEn}'$. Approximate Entropy (with tolerance $r=0$ here) is a time-series regularity statistic: it counts how often nearby patterns recur, and the paper uses $\\mathrm{ApEn}' = 1/\\mathrm{ApEn}$ so that larger values mean cleaner, less redundant data. Theorem 3.3 argues that the effective data scale is bounded by tokens times $\\mathrm{ApEn}'$, using Kraft's inequality and Markov-chain entropy; Theorem 3.5 bounds loss by $\\log(n) + 1/n$ with $n = S_{\\max}d_{\\mathrm{emb}}$, which produces the log-plus-decay shape of the model-size terms. The paper assumes Performance $= 1 - kL$ to convert loss into ranking metrics, and Equation (7) is the assembled fit.","core_discovery":"The central claim is the Performance Law, Equation (7):\n\n$$\\text{Performance} = w_1\\left(\\log N + \\frac{p_1}{$N^{{w_3}}$}\\right) + w_2\\left(\\log d_{\\mathrm{emb}} + \\frac{p_1}{d_{\\mathrm{emb}}^{w_4}}\\right) + \\log D' + \\frac{p_2}{D'^{w_5}},$$\n\nwith $D' = \\#\\mathrm{Tokens}\\cdot \\mathrm{ApEn}'$ (where $\\mathrm{ApEn}' = 1/\\mathrm{ApEn}$, the reciprocal of Approximate Entropy). The paper argues that this single equation describes HR@10 and NDCG@10 of decoder-only transformer sequential recommenders across datasets, model depths, and embedding sizes, and that it can locate both a global optimum and the best configuration under a fixed model-size budget. The law is assembled by combining a log-and-decay loss bound (Theorem 3.5) with a data-scale bound (Theorem 3.3) that replaces raw token count by tokens times reciprocal Approximate Entropy, joined through the assumed linear relationship between ranking performance and loss.","pith_inferences":["An unstated extension of the linear-loss assumption: a calibrated link between softmax loss and thresholded ranking metrics, such as a logistic or sampling-corrected transform, might make the Performance Law hold outside the fitted regimes.","Setting Approximate Entropy's tolerance to zero counts only exact repeats; for noisy interaction logs, a small positive tolerance would make the quality measure less brittle and is a testable variant of the paper's recipe.","If the effective-data bound is universal, the same $D' = \\mathrm{Tokens}\\cdot\\mathrm{ApEn}'$ substitution could be applied to other recommendation families and other metrics, turning the law into a general data-curation tool (for example, predicting when de-duplicating interactions helps more than adding new ones)."],"forward_implications":["If Equation (7) holds, HR@10 and NDCG@10 for unseen configurations can be predicted from a few small training runs, token counts, and Approximate Entropy, without full grid searches.","The same fitted law can identify both the global optimum and the constrained optimum (for example, a fixed depth and embedding size), which the paper demonstrates by selecting a practical online-style configuration.","By comparing fitted exponents across model architectures and numerical precisions, the law ranks how much each model family stands to gain from scaling up.","Because the data term is $\\log D' + p_2/D'^{w_5}$, the law predicts diminishing returns and eventual decay from both model-size and data-size directions, explaining when bigger models or more data stop helping."],"supporting_citations":[{"why":"Defines the neural scaling-law paradigm that the paper adapts to sequential recommendation.","marker":"Kaplan et al., 2020"},{"why":"Supplies the Chinchilla loss form L(N,D) whose decomposition Equation (7) refines.","marker":"Hoffmann et al., 2022a"},{"why":"Introduces Approximate Entropy, the data-quality measure the paper inverts to ApEn'.","marker":"Pincus, 1991"},{"why":"Provides the Kraft inequality used in the proof that effective data scale is bounded by tokens times ApEn'.","marker":"Kraft, 1949"},{"why":"Gives the loss lower bound and associative-memory analysis behind Theorem 3.5's log-plus-decay structure.","marker":"Niu et al., 2024"},{"why":"Analogous performance-law factorization that the paper explicitly follows in assembling Equation (7).","marker":"Wu & Tang, 2024"},{"why":"Cited for the loss-performance link that motivates the assumed linear relation Performance = 1 - kL.","marker":"Du et al., 2024"},{"why":"Documents the inverted-U performance behavior that the decay terms in Equation (7) are designed to capture.","marker":"Belkin et al., 2019"}],"fun_headline_variants":["One law predicts recommender accuracy from size and data quality","Data complexity, not just size, sets sequential recommender performance","Performance Law: forecast HR@10 from model scale and data entropy","Approximate Entropy unlocks scaling law for sequential recommenders","Predict recommender accuracy via one law: scale plus data entropy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole law rests on the assumption that ranking metrics rise and fall linearly with the training loss; if that relationship bends, the predictions have no foundation.","fun_headline_variants_meta":{"raw":{"variants":["One law predicts recommender accuracy from size and data quality","Data complexity, not just size, sets sequential recommender performance","Performance Law: forecast HR@10 from model scale and data entropy","Approximate Entropy unlocks scaling law for sequential recommenders","Predict recommender accuracy via one law: scale plus data entropy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1423,"prompt_tokens":958,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":574,"tokens_out":465,"duration_ms":4319,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:24:29.821151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out dataset, compute ApEn and token counts, fit Equation (7) using only a few small runs, then predict HR@10 and NDCG@10 at larger depths and embedding sizes; if the predicted values fall outside the fitted residual band, or if the measured loss and HR pairs are visibly non-monotone, the law fails.","supporting_citations":[],"review_version":1}