{"id":"0f6d23cc-2e4b-46a8-ada5-4a0e86ec82b1","arxiv_id":"2411.15223","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"An xDeepFM variant with multi-head attention and an FM replacing the linear part reports AUC 0.7850 and Logloss 0.4628 on a 500k sample of Criteo.","lead":"This paper proposes a click prediction model that adds multi-head attention to xDeepFM and replaces its linear layer with a factorization machine. It reports small gains over two older models on a public ad dataset, but provides no code, no significance tests, and its own table shows other models beating it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are not trustworthy because hyperparameters were tuned on the same test set used for the final comparison; the 0.0072 AUC improvement over xDeepFM may be selection bias.","rationale":"The reader's verdict of REJECT is well supported. The central empirical claim, that the proposed model significantly improves CTR prediction accuracy, rests entirely on test-set metrics reported in Sections 3.3.1-3.3.4. The paper explicitly uses the test set for hyperparameter selection: the learning rate, embedding dimension, and number of attention heads are each chosen by comparing AUC and Logloss on the same 20% test split that later appears in Table 3. This invalidates the final comparison as an unbiased estimate of model quality. The absence of validation data, seed variation, or confidence intervals means the reported 0.0072 AUC gain could be selection noise rather than a real architectural improvement. The internal contradiction in Table 3, where DMCNN and DTM beat the proposed model on one metric each, further weakens the abstract's claim of outperforming other state-of-the-art methods. I found no machine-checked proofs or released code that could independently support the result. The proposed modification itself is a reasonable incremental idea, but the evidence as presented does not establish the claimed accuracy improvement. Therefore, the reader's REJECT verdict should stand; my stress-test does not change it.","tokens_in":9619,"tokens_out":2720,"duration_ms":24640,"concrete_test":"Re-run the experiment with a strict train/validation/test split (e.g., 70/10/20 or nested cross-validation): tune learning rate, embedding size, and attention heads on the validation split only, freeze the chosen hyperparameters, and then evaluate once on the untouched test split. Repeat with at least 5 random seeds and report mean and standard deviation of AUC and Logloss for both the proposed model and xDeepFM under identical conditions. If the 0.0072 AUC gain over xDeepFM does not persist on the untouched test set, or is within one standard deviation, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that the dataset was divided 8:2 into training and test sets, and Sections 3.3.1-3.3.3 use AUC/Logloss on that test set to choose the learning rate (0.05), embedding size (8), and number of attention heads (2). Section 3.3.4 then reports the model with these chosen hyperparameters on the same test set and attributes the 0.0072 AUC gain over xDeepFM to the architecture. Because the test set was already used for model selection, the final numbers are optimistically biased; with multiple configurations evaluated on the same test split, best-of-test selection alone can create a spurious advantage over baselines that were not tuned in the same way. The paper reports no separate validation split, no multiple seeds, and no error bars, so the claimed significance cannot be assessed. In addition, the broad claim that the model outperforms other state-of-the-art methods is contradicted by the paper's own Table 3, where DMCNN has higher AUC (0.7994) and DTM has lower Logloss (0.4488) than the proposed model. The load-bearing weakness is the test-set tuning: without a genuinely held-out evaluation, the architecture's actual contribution is unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an enhanced xDeepFM model for advertising click-through rate prediction, replacing the linear component with a Factorization Machine and adding a multi-head attention layer. The authors evaluate the model on a 500,000-sample subset of the Criteo dataset, report AUC and Logloss, and claim that their model outperforms DeepFM, xDeepFM, and other state-of-the-art methods. The core claim is unsupported because the final test metrics are obtained from the same test set used to select hyperparameters, and Table 3 internally contradicts the claim of universal superiority.","tokens_in":9873,"tokens_out":2529,"duration_ms":23415,"significance":"If the claimed improvements were established with a sound evaluation protocol, the architectural modifications (multi-head attention plus FM-based low-order interaction modeling) would be a modest but potentially useful incremental contribution to CTR prediction research. However, the paper provides no credible evidence for this claim: the evaluation is methodologically invalid, the results are not reproducible from the information given, and the reported numbers are not internally consistent. The paper ships no code, no detailed preprocessing steps, and no statistical significance analysis, so even the descriptive contribution is weakened. The central idea is not original enough to carry the paper by itself without a trustworthy empirical demonstration.","major_comments":[{"comment":"The experimental protocol invalidates the headline claim. Sections 3.3.1 through 3.3.3 select the learning rate (0.05), embedding size (8), and number of attention heads (2) by comparing AUC and Logloss on the test set, and Section 3.3.4 then reports the final comparison on that same test set. Because the test set has already been used for model selection, the final metrics are not independent; best-of-test selection can create a spurious advantage over baselines that were not tuned on the same split. The paper must either use a separate validation set for hyperparameter selection and reserve the test set for a single final evaluation, or use nested cross-validation. Without this, the reported 0.0072 AUC improvement over xDeepFM cannot be attributed to the architecture.","section":"§3.3.1–§3.3.4"},{"comment":"The broad claim that the model outperforms other state-of-the-art methods is contradicted by the paper's own Table 3. The proposed model achieves AUC 0.7850 and Logloss 0.4628, but DMCNN achieves a higher AUC (0.7994) and DTM achieves a lower Logloss (0.4488) on the same reported benchmark. The abstract and Section 3.3.4 state that the model outperforms other state-of-the-art methods without qualification, which is not supported by these data. The authors should restrict their claims to the two baselines actually compared in their own experiments, or explain why the other rows are not considered in the comparison.","section":"Table 3 and Abstract"},{"comment":"No measures of uncertainty are reported. All comparisons are single point estimates with no error bars, no multiple seeds, and no statistical significance tests. Given that the reported difference in AUC between the proposed model and xDeepFM is only 0.0072, and the difference between runs with different hyperparameter settings in Tables 1 and 2 is of a similar magnitude (e.g., 0.7849 vs. 0.7850), the claimed improvement may be within run-to-run noise. The authors should report mean and standard deviation over multiple random seeds and perform a paired significance test (e.g., a paired bootstrap or Wilcoxon test) on the test predictions.","section":"§3.3.4"},{"comment":"The experimental description is missing crucial reproducibility details. The paper states that 500,000 samples were used for training, but it is unclear whether the full sampled dataset is 500,000 samples split 80/20 or whether 500,000 is the training size only. There is no description of how missing values were handled, how the categorical features were one-hot encoded or embedded, what the feature cardinalities are, or how the baseline models (DeepFM, xDeepFM, SEDAFM, MAME-DFM, DTM, DMCNN) were configured and tuned. Without these details, no one can reproduce or verify the results, and the comparison may not be fair if the baselines were not tuned with the same effort.","section":"§3.1 and §3.3.4"}],"minor_comments":[{"comment":"The column header 'Number of Long Attention Head' appears to be a typo; it should read 'Number of Attention Heads'.","section":"Table 2"},{"comment":"The sentence 'The test results of different models on the same dataset are shown in Table 3 and Figures 2 and 3' is confusing because Figures 2 and 3 are the architecture diagrams of CIN and the multi-head attention mechanism, not experimental result plots. The reference should be corrected to point to the relevant result figures or be removed.","section":"§3.3.4"},{"comment":"The AUC formula in Eq. (8) is written as a single line without the rank-sum variable on the left-hand side; the notation should be cleaned up and the 'rank' variable explained consistently.","section":"§3.2"},{"comment":"The hyperparameter experiments are presented as if the best configuration is unambiguous, but the differences are very small (e.g., AUC 0.7849 at embedding size 8 versus 0.7844 at size 6). The authors should acknowledge that these differences may not be practically significant and would benefit from statistical testing even for the hyperparameter choice.","section":"§3.3.1–§3.3.3"},{"comment":"The abstract claims that the improvements are achieved 'without significantly increasing computational complexity,' but no runtime, parameter count, or computational cost experiments are reported. This claim should either be substantiated or removed.","section":"Abstract and §3.3.4"},{"comment":"Several references are incomplete or inconsistently formatted; for example, Vaswani (2017), Hochreiter (1997), and O'Shea (2015) lack full author lists and titles, while other entries include them. The reference list should be brought to a consistent, complete format.","section":"References"},{"comment":"The manuscript contains numerous typographical and formatting issues ('School of Dig Data', 'X DEEP FM', garbled equations, and inconsistent use of styles in the formulas). A thorough proofreading and LaTeX cleanup is needed before publication.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is a preprint-style submission with a weak evaluation methodology. The core issue is not merely a missing error bar but the use of the test set for hyperparameter selection, which calls into question the validity of every reported result. The internal inconsistency in Table 3 also suggests that the authors did not carefully check their own numbers. Even though the methodology could in principle be fixed by re-running experiments with a proper validation split, the current manuscript would need substantial new experiments, and given the lack of code and preprocessing details, I do not see a straightforward path to a sound paper within a normal revision cycle. I therefore recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a modest, clearly described modification of xDeepFM—multi-head attention on the embeddings plus an FM replacing the linear term. That is a legitimate idea, and the paper does a fair job explaining the pieces. The problem is the evaluation: the authors tune learning rate, embedding dimension, and attention heads on the test set (Sections 3.3.1–3.3.3) and then report the chosen configuration's score on the same test set in 3.3.4. That invalidates the headline gain of +0.0072 AUC over xDeepFM; it could easily be selection bias. There are no multiple seeds or error bars to suggest otherwise.\n\nI also checked Table 3. Against the baselines the authors cite, DMCNN gets AUC 0.7994 and DTM gets Logloss 0.4488, both better than the paper's 0.7850/0.4628 on one metric. So the abstract's 'outperforms other state-of-the-art methods' is not what their own table shows. The correct statement would be 'competitive with or better than DeepFM and xDeepFM on this subset.'\n\nWhat the paper does well: the model description is readable, the equations are standard, and the CIN/attention/FM fusion is coherent. If the authors rerun the experiments with a proper validation split and multiple seeds, the result would be a small but honest engineering contribution. As it stands, the central quantitative claim is unsupported.\n\nI would not bring this to a reading group or cite it in its current form. It is not a waste of time—the idea is sane and the writing is fine—but the evaluation needs to be redone from scratch. I'd lean toward desk reject for a serious venue: the flaw is load-bearing and the reported numbers are not trustworthy. If the authors come back with a clean protocol, it could be a workshop-level paper.","headline":"The architecture is a reasonable incremental variant, but the test-set tuning makes the reported gains untrustworthy and the abstract overstates the results.","tokens_in":10396,"tokens_out":2542,"would_cite":false,"duration_ms":22926,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding a multi-head attention layer and replacing xDeepFM's linear component with a Factorization Machine improves CTR prediction accuracy on the Criteo dataset.","keywords":["CTR prediction","xDeepFM","multi-head attention","factorization machine","Criteo dataset","AUC","Logloss","feature interaction"],"falsifier":"Retrain the proposed model and xDeepFM on Criteo with a train/validation/test split, tune only on validation, and report the best-validation-epoch test AUC and Logloss across several random seeds. If the AUC advantage over xDeepFM does not reproduce on a test set that was never used for model selection, the central claim of significant improvement is not supported.","tokens_in":9397,"feed_emoji":"📈","tokens_out":7323,"duration_ms":58783,"temperature":0.7,"pith_summary":"The paper proposes an enhanced xDeepFM model for advertising click-through rate prediction and reports that it outperforms the original xDeepFM, DeepFM, and several other baselines on a 500,000-sample Criteo subset. The two modifications are a multi-head attention layer placed before the deep network, with residual connections and layer normalization, and a Factorization Machine replacing the linear component. On the reported test set the model reaches AUC 0.7850 and Logloss 0.4628, a gain of 0.0072 AUC and 0.0061 Logloss over xDeepFM. If the result holds, it offers a relatively contained way to improve CTR prediction accuracy on sparse, high-dimensional advertising data.","feed_headline":"Improved xDeepFM hits 0.7850 AUC on Criteo CTR benchmark","feed_subtitle":"Multi-head attention plus an FM branch beats xDeepFM by 0.0072 AUC and 0.0061 Logloss on 500k Criteo samples.","key_machinery":"The two load-bearing additions are the multi-head attention layer, computed as $\\text{MultiHead}(Q,K,V)=\\text{Concat}(\\text{head}_1,\\ldots,\\text{head}_h)W^O$ with each head an attention operation, and the Factorization Machine replacing the linear term, with prediction $\\hat{y}(x)=w_0+\\sum_i w_i x_i+\\sum_i\\sum_{j>i}\\langle v_i,v_j\\rangle x_i x_j$. The attention layer is inserted ahead of the DNN with residual connections and layer normalization so original embedding information is retained while feature correlations are captured from multiple subspaces. The FM branch contributes low-order interaction modeling that the linear branch of xDeepFM cannot express. These outputs are combined with the CIN and DNN outputs and mapped by a sigmoid to produce the final click probability.","core_discovery":"The paper's central claim is that its enhanced xDeepFM architecture predicts ad clicks more accurately than the model it builds on. The final prediction is the sigmoid-weighted fusion of three branches: an FM branch for first- and second-order feature interactions, a Compressed Interaction Network for explicit high-order interactions, and a DNN branch whose input embeddings are first passed through a multi-head attention layer with residuals and layer normalization. On the Criteo benchmark the paper reports test AUC 0.7850 and Logloss 0.4628, compared with 0.7778 and 0.4689 for xDeepFM, and interprets these differences as evidence that both modifications contribute to better modeling of feature relationships in sparse ad data.","pith_inferences":["A direct testable extension is to repeat the hyperparameter search using a separate validation split and touch the test set only once; the reported gain would be materially stronger evidence if it survives that protocol.","The baselines SEDAFM, MAME-DFM, DTM, and DMCNN are compared using numbers from other sources rather than retrained under identical conditions, so an apples-to-apples retraining comparison would clarify whether the improvement is specific to this implementation.","The claim that computational complexity does not significantly increase is stated, not measured; a parameter-count and runtime comparison against xDeepFM would make it testable.","Re-running the same two modifications on other sparse CTR datasets or larger Criteo samples, with multiple seeds, would show whether the 0.0072 AUC gain is stable or dataset-specific."],"forward_implications":["If the reported results are accepted, porting these two modifications into an existing xDeepFM-based CTR system yields a gain of roughly 0.0072 AUC on a large, sparse advertising dataset.","The lower Logloss alongside the higher AUC indicates that predicted click probabilities become both better ranked and better calibrated.","Because the attention layer is preposed and the DNN and CIN components are unchanged, the modification can be layered onto similar feature-interaction architectures without redesigning their training pipelines.","The combination of FM, CIN, and attention-enhanced DNN lets one network cover first-order, second-order, explicit high-order, and implicit high-order feature interactions simultaneously."],"supporting_citations":[{"why":"Supplies the xDeepFM architecture that the paper modifies and the primary baseline whose AUC and Logloss the reported gains are measured against.","marker":"Lian et al. [2018]"},{"why":"Provides the DeepFM model used as a comparison baseline and the embedding-output-sharing design that the proposed model inherits.","marker":"Guo et al. [2017]"},{"why":"Defines the Factorization Machine whose prediction formula replaces the linear component of xDeepFM.","marker":"Rendle [2010]"},{"why":"Supplies the multi-head attention mechanism and scaled dot-product attention formulas adopted in the proposed model.","marker":"Vaswani [2017]"},{"why":"Motivates combining attention with xDeepFM-style automatic feature interaction learning.","marker":"Song et al. [2019]"},{"why":"Introduces the Deep and Cross Network that contributes the cross-network lineage behind xDeepFM's CIN component.","marker":"Wang et al. [2017]"},{"why":"Provides the AUC evaluation metric used to compare all models in the experiments.","marker":"Lobo et al. [2008]"},{"why":"Provides the Logloss evaluation metric used to compare predicted probability quality across models.","marker":"Bishop [1995]"}],"fun_headline_variants":["Multi-head attention boosts xDeepFM CTR accuracy on Criteo","Enhanced xDeepFM with attention and FM wins Criteo AUC","FM plus attention lifts xDeepFM to 0.7850 AUC on Criteo","Better CTR prediction: xDeepFM upgraded with attention and FM","Attention-FM hybrid improves ad click prediction on Criteo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hyperparameters chosen by inspecting AUC and Logloss on the test set in Sections 3.3.1 through 3.3.3 generalize to the final reported test numbers; if they do not, the 0.0072 AUC gain could reflect selection bias rather than a genuine improvement in the model.","fun_headline_variants_meta":{"raw":{"variants":["Multi-head attention boosts xDeepFM CTR accuracy on Criteo","Enhanced xDeepFM with attention and FM wins Criteo AUC","FM plus attention lifts xDeepFM to 0.7850 AUC on Criteo","Better CTR prediction: xDeepFM upgraded with attention and FM","Attention-FM hybrid improves ad click prediction on Criteo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2438,"prompt_tokens":917,"completion_tokens":1521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1427}},"tokens_in":533,"tokens_out":1521,"duration_ms":73939,"temperature":1.0,"reasoning_tokens":1427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:50:42.573966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the proposed model and xDeepFM on Criteo with a train/validation/test split, tune only on validation, and report the best-validation-epoch test AUC and Logloss across several random seeds. If the AUC advantage over xDeepFM does not reproduce on a test set that was never used for model selection, the central claim of significant improvement is not supported.","supporting_citations":[{"cited_title":"xdeepfm: Combining explicit and implicit feature interactions for recommender systems","cited_arxiv_id":null,"evidence_quote":"Supplies the xDeepFM architecture that the paper modifies and the primary baseline whose AUC and Logloss the reported gains are measured against."},{"cited_title":"Autoint: Automatic feature interaction learning via self-attentive neural networks","cited_arxiv_id":null,"evidence_quote":"Motivates combining attention with xDeepFM-style automatic feature interaction learning."},{"cited_title":"Auc: a misleading measure of the performance of predictive distribution models","cited_arxiv_id":null,"evidence_quote":"Provides the AUC evaluation metric used to compare all models in the experiments."}],"review_version":1}