{"id":"77654d93-ab28-4332-9d98-6856c1f0f026","arxiv_id":"2411.13700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CETNet combines two CTR models with separate embeddings, KL collaboration, and entropy-based confidence fusion, yielding small AUC gains on public benchmarks but not in the internal deployment.","lead":"This paper proposes CETNet, an ensemble method for click-through rate prediction that trains two models with separate embedding tables, aligns them with a KL-divergence loss, and fuses their outputs using an entropy-based confidence weight. It reports small AUC gains over individual models on three public datasets, though the confidence-based fusion did not help in the internal Meta deployment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central novelty—entropy-confidence fusion—is undercut by its own Section 4.5, where it failed in the large-scale setting; the claim that fusion drives CETNet's gains is unsubstantiated.","rationale":"The reader's weakest assumption is correct; I agree. The concern is load-bearing because the paper's contribution list includes confidence-based fusion as a key mechanism, and the deployment section contains a negative result for it. The public ablation is not sufficient because it is a single run, and the 'w/o Confidence-based Fusion' and 'Multi-Embedding + concat' rows are not clearly defined enough to reconcile with Section 4.5. A controlled comparison with uniform weights would isolate the value of entropy weighting. The verdict should remain CONDITIONAL: the paper needs this test and/or release of code before the central claim can be accepted. No change from the reader's conditional verdict.","tokens_in":16114,"tokens_out":3851,"duration_ms":37800,"concrete_test":"On the three public datasets, run CETNet with three fusion variants under identical settings (same multi-embedding, same KL weight, same hyperparameters): (a) entropy-weighted softmax, (b) uniform weights, (c) plain concatenation. Use at least 5 random seeds and report mean +/- std AUC/gAUC plus paired bootstrap confidence intervals for differences. If variant (a) is not significantly better than (b) or (c) on at least two datasets, the confidence-fusion claim should be dropped and the framework reframed as a collaborative multi-embedding ensemble with concatenation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.4, CETNet's fusion weight is exp(-H(P)), treating low entropy as confidence. This assumption is unvalidated: for a Bernoulli click probability, entropy is minimal for predictions near 0 or 1 regardless of correctness, so a confidently wrong model can dominate. More importantly, the paper's own large-scale deployment (Section 4.5) states that this confidence-based fusion 'does not result in significant improvements or stable learning in our internal setting,' while simple concatenation achieved a better NE gain (-0.062% vs -0.051%). This directly contradicts Table 3, where removing confidence-based fusion degrades results on all public datasets. The discrepancy is unresolved: either the fusion mechanism is dataset-dependent and not a reliable component, or the public benchmark numbers are not statistically robust (single seed, no error bars). Since the abstract and introduction present confidence-based fusion as a key contribution, the central claim that CETNet 'dynamically balances contributions' is the least secure link in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CETNet, an ensemble framework for CTR prediction that trains two models (InterFormer and DHEN) with separate embedding tables, aligns their predictions with symmetric KL divergence, and fuses their output embeddings using softmax weights derived from negation entropy. The authors evaluate CETNet on AmazonElectronics, TaobaoAds, KuaiVideo, Criteo, and Avazu, report ablations, and describe a large-scale internal deployment at Meta. The paper's central claim is that CETNet outperforms individual models and state-of-the-art baselines, and that confidence-based fusion is a key component that lets the framework achieve this with smaller embedding sizes than multi-embedding baselines.","tokens_in":16327,"tokens_out":9484,"duration_ms":65740,"significance":"The question addressed, how to combine diverse CTR models without duplicating full embedding costs, is practically important, and the paper's comparisons against the multi-embedding paradigm on Criteo/Avazu are a useful addition to the literature. The broad public-benchmark evaluation, multiple ensemble combinations, and one-epoch experiments are strengths. However, the signature confidence-based fusion component is explicitly reported as unsuccessful in the internal deployment, and the public results are all single-seed runs with no variance estimates. These issues need to be resolved before the stated contributions can be considered established.","major_comments":[{"comment":"Section 4.5 reports that confidence-based fusion 'does not result in significant improvements or stable learning' in the large-scale internal setting, whereas simple concatenation achieves a larger NE gain (-0.062% vs. -0.051%). Since confidence-based fusion is a central claimed contribution of the paper (abstract and Section 1) and Table 3's ablation says removing it degrades public-benchmark results, the paper needs to address this contradiction. At minimum, provide a multi-seed analysis on the public datasets to show whether the fusion component has a stable positive effect, and explicitly discuss in which settings it does and does not help.","section":"4.5 / Figure 3"},{"comment":"The text states that the InterFormer+DHEN combination 'achieves the highest gAUC, AUC, and lowest LogLoss across all datasets.' Table 1 contradicts this for LogLoss on TaobaoAds: CETNet (InterFormer+DHEN) has LogLoss 0.1930, while CETNet (InterFormer+DCNv2) has 0.1921, CETNet (DIN+DIEN) has 0.1922, and InterFormer alone has 0.1924. The claim should be corrected to reflect the actual metric(s) for which the combination is best, and the authors should explain why LogLoss does not follow the same pattern.","section":"4.2 / Table 1"},{"comment":"The confidence measure used in the fusion module is the negation entropy of each model's scalar click probability. For a Bernoulli distribution, entropy is a deterministic function of p that is minimized as p approaches 0 or 1, irrespective of whether the prediction is correct; a confidently wrong model can therefore receive the largest fusion weight. The paper does not validate that exp(-H(P)) is a useful confidence signal for CTR predictions (e.g., by correlating it with per-instance error or showing calibration), so the theoretical justification for the fusion mechanism is currently weak.","section":"3.4"},{"comment":"All experimental tables appear to be based on single runs with one random seed, and no standard deviations, confidence intervals, or significance tests are reported. Many of the claimed differences are small (e.g., Table 1, KuaiVideo AUC 0.7456 vs. 0.7453; Table 3, Amazon gAUC 0.8860 vs. 0.8851), so it is difficult to establish that the improvements are not run-to-run noise. Please provide multiple-seed results with summary statistics for the main claims in Tables 1 and 3.","section":"4.1 / Tables 1 and 3"},{"comment":"The Criteo/Avazu experiments compare 'Ours-sum' and 'Ours-concat' for DNN, IPNN, DCNv2, and FinalMLP, but the manuscript never specifies how these variants relate to the CETNet architecture described in Section 3. For example, it is unclear whether symmetric KL alignment and confidence-based fusion are used in these configurations, or whether the reader should view them as simple concatenation/summation ensembles over the existing models. Without this specification, the parameter-efficiency claim relative to the multi-embedding baseline cannot be fully assessed.","section":"4.4.2 / Table 4"}],"minor_comments":[{"comment":"The table header row contains the typo 'InteFormer'; it should be 'InterFormer'.","section":"Table 1"},{"comment":"The text says 'such as an increase in gAUC from 0.6637 to 0.6620,' but 0.6620 is a decrease; the sentence should be corrected.","section":"4.3"},{"comment":"The section says 'six state-of-the-art models' but lists seven model names, with four non-sequential models and three sequential models; the count and list should be aligned.","section":"4.1.2"},{"comment":"Figure 3 lacks axis labels and a clear definition of the NE gain metric; please label the axes and state the exact formula used for NE gain.","section":"Figure 3"},{"comment":"The KL divergence formula takes logarithms of predicted probabilities without an epsilon; a numerical stabilization note or clipping strategy should be added for cases where predictions are 0 or 1.","section":"3.3.1"},{"comment":"The abstract uses the term 'general softmax' while Section 3.4.1 describes standard softmax; the terminology should be made consistent.","section":"Abstract / 3.4.1"},{"comment":"The manuscript retains template placeholders such as 'Do Not Use This Code', 'Conference acronym XX', and the copyright block; these should be removed before submission.","section":"Title page"},{"comment":"The labels 'Standard' and 'Ensemble' in Figure 2 are not defined in the caption; please clarify whether they refer to single-model predictions before and after collaborative training.","section":"4.4.1 / Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely fixable with a revision that reframes the contribution around the ensemble and concatenation results, demotes confidence-based fusion to an exploratory variant, adds error bars, and corrects the overclaims. The internal-deployment result currently works against the paper's main narrative, so the authors need to be transparent about the discrepancy and, if possible, provide evidence on which fusion strategy is appropriate in which settings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a plausible engineering paper with a real credibility problem at its center. The proposed framework, CETNet, ensembles two CTR models (InterFormer and DHEN) with separate embedding tables, symmetric KL divergence to align their predictions, and a confidence-based fusion that weights each model's embedding by exp(-entropy). The pieces are all known—multi-embedding from Guo et al., deep mutual learning from Zhang et al., entropy fusion from the multimodal literature—but the combination is new for CTR. On three public datasets, CETNet beats its individual components by small but consistent margins (roughly 0.1–0.3% AUC), and the Criteo/Avazu experiments suggest parameter efficiency relative to single- and multi-embedding baselines. The paper is clearly written, the ablation is standard, and the authors deserve credit for reporting their Meta deployment honestly.\n\nThe soft spot is the confidence-fusion mechanism itself. Section 3.4 assumes that lower Bernoulli entropy means higher confidence, which is shaky—for a click probability, entropy is lowest near 0 and 1 regardless of correctness, so a confidently wrong model can dominate. More damaging, Section 4.5 states that in the large-scale internal setting the confidence-based fusion “does not result in significant improvements or stable learning,” and simple concatenation gave a better NE gain. That directly contradicts the public-benchmark ablation in Table 3, where removing fusion degrades results across all datasets. The authors don't resolve this discrepancy; they just note it. Combined with single-seed runs and no error bars, the evidence that the signature fusion component helps is thin.\n\nThere are also reproducibility gaps: the paper reports tuning learning rate, weight decay, and alpha, but gives values only for alpha. The code is not released. The gains are small enough that they could be noise, and the lack of variance estimates makes that impossible to rule out.\n\nWho is this for? Practitioners who want a concrete recipe for ensembling two existing CTR models—that part is usable. Researchers will find a modest extension of known ideas plus a useful reminder that a component that works in one setting can fail in another. The paper deserves a serious referee because the core question—does adaptive confidence weighting add value over concatenation in CTR ensembles?—is meaningful and the authors have gathered substantial data around it. But as written, the central claim is not supported. I'd recommend sending it to review with a request for major revision: multi-seed results, full hyperparameter disclosure, and a direct attempt to reconcile Table 3 with Section 4.5.\n\nBest,","headline":"A useful ensemble recipe with a load-bearing fusion component that its own deployment experiment undermines.","tokens_in":16883,"tokens_out":2921,"would_cite":false,"duration_ms":27095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Entropy-weighted ensemble of two CTR models beats either model alone with smaller embeddings.","keywords":["Recommender systems","CTR prediction","collaborative learning","knowledge distillation","ensemble","multi-embedding","confidence-based fusion"],"falsifier":"Measure, across the three public datasets, the per-instance correlation between each component model's negation entropy and whether that model's prediction is closer to the ground-truth label than the other model's; if the correlation is zero or negative, the confidence-based fusion cannot be carrying the reported gains and concatenation should perform equally, which the ablation contradicts.","tokens_in":15946,"feed_emoji":"🎯","tokens_out":7979,"duration_ms":71885,"temperature":0.7,"pith_summary":"CETNet is a training framework that ensembles two click-through-rate models—InterFormer, which learns sequential and heterogeneous feature interactions, and DHEN, which learns hierarchical high-order interactions—each with its own independent embedding table. The two models are trained jointly with a symmetric KL-divergence term that aligns their predictions, and their embeddings are combined by a confidence-based fusion that weights each model by the negation entropy of its predicted click probability. The paper claims this collaborative ensemble outperforms either model alone and state-of-the-art baselines on three public datasets, and that on two additional datasets it matches or beats the multi-embedding paradigm while using substantially smaller embedding tables. The wider claim is that model diversity plus aligned joint training is a more efficient path to accuracy than simply scaling one model's embedding size.","feed_headline":"Two-model ensemble beats bigger embeddings for click prediction","feed_subtitle":"Separate embedding tables plus confidence weighting lift accuracy on live ad data.","key_machinery":"The mechanism is a confidence-based fusion built on the negation entropy of each model's scalar click probability: $C = -H(\\hat y)$ with $H(\\hat y)=-\\hat y\\log\\hat y-(1-\\hat y)\\log(1-\\hat y)$, detached from the computation graph. Softmax over the two confidence scores gives weights $w_{IF}$ and $w_{DHEN}$, and the final embedding is the weighted concatenation $[w_{IF}\\cdot e_{IF}\\parallel w_{DHEN}\\cdot e_{DHEN}]$, fed to a readout layer with a sigmoid for the fused prediction. Around this sit three binary cross-entropy losses (one per model plus one for the fused output) and a symmetric KL-divergence term $\\mathcal{L}_{kl}=\\frac12 KL(\\hat y_{IF}\\|\\hat y_{DHEN})+\\frac12 KL(\\hat y_{DHEN}\\|\\hat y_{IF})$ that aligns the two models' predictions during joint training. The separate embedding tables provide the diversity; the KL term provides the collaboration; the entropy weighting provides per-instance dynamic balancing.","core_discovery":"The paper shows that complementary feature-interaction models can be made to cooperate better than either can perform alone, and that this cooperation is what unlocks gains without expanding embedding capacity. Specifically, it shows that giving InterFormer and DHEN separate embedding tables and training them with symmetric KL divergence prevents one model from dominating, while softmax weighting over the negation entropy of each model's click probability lets the more confident model's embedding carry more weight in the fused prediction. In experiments across AmazonElectronics, TaobaoAds, and KuaiVideo, the InterFormer+DHEN ensemble achieves the highest gAUC, AUC, and lowest LogLoss among all listed methods, and ablation removes each component (multi-embedding, KL alignment, confidence fusion, gradient stopping) degrades performance. On Criteo and Avazu, the framework with simple sum or concatenation fusion beats single-embedding and multi-embedding baselines at 2x or 3x embedding size where the baselines need 10x, showing the efficiency claim. In the industrial deployment, concatenation proved more stable than the confidence weighting, yet still gave a normalized-entropy gain of -0.062% over the baseline.","pith_inferences":["Since the large-scale industrial deployment found confidence-based fusion unstable and simple concatenation superior, the entropy weighting may be dataset- or calibration-dependent; a direct test is to compare full fusion against concatenation on the three public datasets while measuring per-instance entropy-correctness correlation.","The paper's own results suggest the benefit depends on architectural distance between the paired models: InterFormer+DHEN outperforms InterFormer+InterFormer, so a systematic study of how much diversity (for example, interaction-order coverage or sequence-vs-hierarchy) drives the gain would sharpen the design rule.","A stronger confidence signal might come from the models' disagreement or from calibrated probabilities rather than the entropy of the raw sigmoid output, since uncalibrated probabilities conflate certainty with confidence."],"forward_implications":["Ensembling architecturally different models with separate embedding tables can improve CTR accuracy even when total embedding parameters are fewer than a single-model baseline.","The symmetric KL term prevents one model from dominating, and the in-ensemble versions of both InterFormer and DHEN outperform their independently trained versions.","Removing the confidence-based fusion, the KL alignment, the multi-embedding, or the gradient stop each degrades performance in the ablations.","With only one epoch of training, the framework still beats single- and multi-embedding baselines on Criteo at 2x or 3x embedding size, which matches industry training budgets.","On the large-scale industrial dataset, the concatenation variant of the fusion gives a normalized-entropy gain, showing the multi-embedding ensemble transfers to production-scale data."],"supporting_citations":[{"why":"supplies the multi-embedding paradigm and the embedding-collapse analysis that motivate and anchor the efficiency comparison.","marker":"[6]"},{"why":"defines InterFormer, the sequential and heterogeneous feature-interaction model used as one ensemble component.","marker":"[29]"},{"why":"defines DHEN, the hierarchical high-order interaction model used as the other ensemble component.","marker":"[31]"},{"why":"inspires the confidence-based fusion mechanism via negation-entropy weighting.","marker":"[7]"},{"why":"also cited as a source for the confidence-based fusion weights.","marker":"[32]"},{"why":"provides the deep mutual learning template of models learning from each other that the KL-alignment term implements.","marker":"[33]"},{"why":"defines normalized entropy (NE), the evaluation metric used in the industrial deployment comparison.","marker":"[9]"},{"why":"provides the open-source implementation and tuned hyperparameter settings used for the baseline comparisons.","marker":"[38]"}],"fun_headline_variants":["Ensemble of complementary models beats larger embeddings for CTR","CETNet: Smaller embeddings, smarter fusion for click prediction","Collaborative ensemble lifts CTR without scaling embedding size","Confidence-weighted fusion boosts ensemble CTR prediction","Two models, separate embeddings, better click prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model's certainty, measured as the negation entropy of its predicted click probability, reliably indicates when that model's embedding deserves more weight in the final prediction.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble of complementary models beats larger embeddings for CTR","CETNet: Smaller embeddings, smarter fusion for click prediction","Collaborative ensemble lifts CTR without scaling embedding size","Confidence-weighted fusion boosts ensemble CTR prediction","Two models, separate embeddings, better click prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2340,"prompt_tokens":1013,"completion_tokens":1327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1253}},"tokens_in":629,"tokens_out":1327,"duration_ms":10765,"temperature":1.0,"reasoning_tokens":1253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:58:50.600144+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, across the three public datasets, the per-instance correlation between each component model's negation entropy and whether that model's prediction is closer to the ground-truth label than the other model's; if the correlation is zero or negative, the confidence-based fusion cannot be carrying the reported gains and concatenation should perform equally, which the ablation contradicts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the multi-embedding paradigm and the embedding-collapse analysis that motivate and anchor the efficiency comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"also cited as a source for the confidence-based fusion weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines normalized entropy (NE), the evaluation metric used in the industrial deployment comparison."}],"review_version":1}