{"id":"57893ffa-37e7-451c-831b-486fe5463f51","arxiv_id":"2411.15731","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OptFusion automatically learns fusion connections and operations in deep CTR models and reports consistent AUC improvements on Criteo, Avazu, and KDD12.","lead":"This paper introduces OptFusion, a method that automatically learns how to fuse the shallow and deep components of click-through-rate prediction models, choosing both connections and fusion operations during training. It reports small but consistent AUC gains over strong baselines on three large ad-click datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Architecture selection on the training set (Eq. 16, Algorithm 1) without a held-out validation split risks overfitting and could inflate the small reported AUC gains; the claim that both variants are significant on all datasets also conflicts with Table 3, where OptFusion-Hard on KDD12 lacks a…","rationale":"The reader identified the same weakest assumption, and I agree. The no-validation selection protocol is the most load-bearing concern because it directly threatens the internal validity of the reported improvements. The improvements are small (0.001–0.003 AUC), and without a validation split, the architecture can adapt to training-data idiosyncrasies. The paper does not report variance or the number of runs behind the t-test, making the asterisks unverifiable. Two smaller inconsistencies reinforce the concern: (1) Table 3 shows OptFusion-Hard without an asterisk on KDD12, contradicting the text's claim that both hard and soft are significant on all datasets; (2) the reported Criteo Logloss improvement (0.0023) does not match the table values (best baseline EDCN Logloss 0.4419 vs OptFusion-Soft 0.4408 gives 0.0011). These are not fatal on their own, but they indicate the significance evidence needs closer scrutiny. I did not find an additional independent weakness beyond the reader's; the code release and the ablation studies are points in favor. A validation-split rerun would settle whether the central claim holds.","tokens_in":17090,"tokens_out":7517,"duration_ms":62856,"concrete_test":"Repeat the full OptFusion pipeline on Criteo, Avazu, and KDD12 with a random 80/20 temporal-preserving split of each training set. In the selection stage (Eq. 16), optimize Θ and α, β on the 80% training subset, and after convergence measure the architecture's loss on the 20% held-out subset; select the architecture with the best validation loss (or early-stop on validation). Then retrain Θ on the full original training set (or the 80% subset) with the selected architecture fixed, and evaluate on the original test set. Recompute the AUC/Logloss gap to the best baseline and the paired t-test across at least 5 seeds, reporting mean±std. If the improvement on any dataset becomes <0.001 AUC or fails to reach p<0.05, the reported gains are at least partly a selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4.2) is that OptFusion outperforms all SOTA baselines 'by a significant margin.' The support rests on Eq. 16, which jointly optimizes model parameters Θ and architecture parameters α, β on the same training set D, and Algorithm 1, which uses only D for selection. The selected architecture is then fixed and Θ retrained on the same D (Eq. 17). No validation set is used for architecture selection. Standard NAS practice (e.g., DARTS) holds out a validation split for architecture parameters because minimizing training loss during selection can overfit the architecture to the training data. With reported AUC improvements of only 0.0011–0.0036, such overfitting could plausibly account for the apparent gains. Moreover, the significance claim is internally inconsistent: in Table 3, OptFusion-Hard is not marked significant on KDD12 (no asterisk), yet the text says 'both soft and hard, outperforms all the SOTA baselines over three datasets ... by a significant margin.' The t-test details (number of runs, variance) are not reported, so the asterisks cannot be independently verified. The released code is a positive, but the experimental protocol does not currently rule out selection overfitting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OptFusion, a framework for automatically learning fusion design in deep CTR prediction models. It defines a search space over fusion connections and fusion operations, relaxes the discrete choices with continuous architecture parameters α and β, uses a straight-through estimator for connection learning, and jointly optimizes model and architecture parameters in a one-shot selection stage, followed by a retraining stage with fixed architecture. Experiments on Criteo, Avazu, and KDD12 compare OptFusion with a range of CTR baselines and NAS methods; the paper reports AUC and LogLoss gains, an efficiency analysis, and ablation studies on fusion operations, selection algorithms, shallow components, and the number of components. Code is released.","tokens_in":17388,"tokens_out":4215,"duration_ms":38654,"significance":"If the reported gains hold, the paper addresses a genuinely underexplored aspect of CTR model design: fusion connections and operations, rather than component architectures. The focused search space is a sensible alternative to broader NAS methods, and the one-shot selection idea is reasonable. Strengths include public code, evaluation on three large-scale datasets, ablation studies that probe the influence of fusion operations and the selection algorithm, and an efficiency analysis. The main concerns are experimental protocol: architecture selection is performed on the training set without a held-out validation split, the statistical significance claims are not fully supported by the reported details, and several directly related fusion-design baselines are omitted.","major_comments":[{"comment":"Architecture parameters α and β are optimized on the same training set D that is later used for retraining (Eq. 17) and for the reported test performance, with no held-out validation split used for architecture selection. Since the reported AUC improvements are only 0.0011–0.0036, selection overfitting on the training loss could plausibly account for part or all of the apparent gains. Please use a held-out validation split for architecture selection, or provide repeated data-split or repeated-run evidence that the selected architectures generalize stably; also report the number of runs and variance for the main results.","section":"§3.4.1 (Eq. 16), Algorithm 1"},{"comment":"The text states that “OptFusion, both soft and hard, outperforms all the SOTA baselines over three datasets” with a “significant margin,” but in Table 3 OptFusion-Hard on KDD12 is not marked with an asterisk, so it is not claimed to be statistically significantly better than the best baseline (EDCN). This internal inconsistency affects the central claim. Please either weaken the claim to match the table or provide the missing significance evidence, and specify the t-test protocol: number of runs, whether tests are paired, and the variance of the reported metrics.","section":"§4.2, Table 3"},{"comment":"The related work discusses FinalMLP, EulerNet, and MaskNet as fusion-focused CTR models, yet none of these appears in the baseline comparison in Section 4.1.3 or in Table 3. Since these methods are directly relevant to the paper's claim of outperforming “all SOTA baselines,” the claim is stronger than the evidence. Please add these baselines or explicitly restrict the comparison to the listed methods.","section":"§4.1.3 (baselines) and §5.1 (related work)"},{"comment":"The text says that “both Soft and Hard methods exhibit significantly superior performance compared to models with fixed fusion operations,” but Table 4 reports no significance tests or variance. Moreover, on Criteo the ADD-only configuration achieves AUC 0.8111 and LogLoss 0.4422, quite close to Hard (0.8108/0.4413) and Soft (0.8113/0.4408), so the fixed-operation ablation does not uniformly show a large gap. Because the experiments use the searched connections with fixed operations, they also do not isolate the benefit of operation selection from connection learning. Please report repeated runs and tests, and temper or support the claim.","section":"§4.4.1, Table 4"},{"comment":"The evidence for the one-shot joint selection algorithm, which is one of the paper's stated contributions, is thin: the differences over the sequential selection are 0.0004 in AUC on both Criteo and Avazu (0.8113 vs 0.8109 and 0.7938 vs 0.7934), with no significance tests or repeated-run variance. This small unquantified gap does not convincingly establish that the entanglement between connection and operation selection is beneficial. Please provide variance estimates or a significance test, or frame the one-shot advantage as suggestive rather than established.","section":"§4.4.2, Table 5"}],"minor_comments":[{"comment":"The identity “2×(1+3+···+2n−1)+2(n+1) = 2n^2+2n+1” appears arithmetically inconsistent: the sum of the first n odd numbers is n^2, so the expression evaluates to 2n^2+2n+2. Please check the derivation and the resulting search-space size.","section":"§3.1 (search space analysis)"},{"comment":"The stopping criterion “while not converged” is not defined; please specify the convergence condition or report the number of epochs used in the selection stage.","section":"Algorithm 1"},{"comment":"The footnote reports a two-sided t-test with p<0.05 but does not state the number of runs, whether the test is paired, or which variance is used; adding these details would make the significance claims verifiable.","section":"Table 3 footnote"},{"comment":"The statement that the optimal learning rate and L2 regularization from the initial training are reused in retraining is useful, but it is unclear whether all baselines receive the same hyperparameter tuning budget; please clarify the tuning protocol for baselines and OptFusion variants.","section":"§4.1.4 (implementation details)"},{"comment":"The case-study figure may be difficult to read in print; please consider larger labels or a table summarizing the selected connections and operations for each dataset.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nYou should know two things about this paper. First, it is a genuinely useful application of NAS to fusion design in CTR models: it narrows the search space to just the fusion connections and operations, and jointly learns both with a one-shot algorithm. The experiments are extensive, the ablations are well designed, and the code is public. That part holds up. Second, the experimental protocol has a real flaw that the stress-test note correctly identifies: architecture parameters α and β are optimized on the same training set used for retraining (Eq. 16 and Algorithm 1), with no held-out validation split. That is unusual — DARTS and most NAS methods hold out a validation set for architecture selection precisely because minimizing training loss can overfit the architecture. With AUC gains of only 0.0011–0.0036, selection overfitting could account for a meaningful chunk of the reported improvement.\n\nThere is also an internal inconsistency: the text says both soft and hard variants 'outperform all SOTA baselines over three datasets ... by a significant margin,' but Table 3 does not mark OptFusion-Hard as significant on KDD12, and the t-test details (number of runs, variance) are not given anywhere. That is a straightforward fix: either report the actual p-values and run counts, or soften the claim. Two expert-fusion baselines named in Related Work (FinalMLP, EulerNet) are also absent from the experiments; their inclusion would strengthen the SOTA claim.\n\nNone of this is fatal. The central claim — that learned fusion helps and that the search can be made efficient — is plausible and consistent with the ablations, including the compatibility study across different shallow components. The paper deserves a serious referee, but it needs revision before acceptance. I would ask the authors to rerun selection with a validation split, report variance over multiple runs, add the missing baselines, and align the significance claims with the table. If those changes are made, the practical contribution is solid. If not, I'd be wary of citing the precise numbers.\n\nFor a reader working on CTR models or practical NAS, this is worth a look. For a reviewer, treat it as promising but in need of the above fixes.","headline":"Useful NAS-for-CTR application with a real protocol flaw — architecture selection on the train set and missing variance — that should be fixed before the precise numbers are trusted.","tokens_in":17874,"tokens_out":3751,"would_cite":false,"duration_ms":29974,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CTR model's fusion design can be learned automatically, beating fixed stacked and parallel fusion on three large ad-click datasets.","keywords":["click-through rate prediction","fusion learning","connection learning","operation selection","one-shot learning","differentiable architecture search","neural architecture search","feature interaction"],"falsifier":"Run OptFusion's selection stage on one split of a dataset, then evaluate the chosen fusion architecture on a disjoint held-out split against a fixed naive fusion design; if the gap disappears or reverses, the reported gains come from selection overfitting rather than from learning better fusions.","tokens_in":16910,"feed_emoji":"🧩","tokens_out":9068,"duration_ms":80325,"temperature":0.7,"pith_summary":"The paper argues that the way a deep click-through rate (CTR) model stitches together its shallow and deep feature-interaction components—the fusion connections and the fusion operations—is a neglected and dataset-dependent design choice, and that this choice can be learned automatically rather than fixed by hand. It introduces OptFusion, which searches over a small directed-acyclic-graph space of component connections and a candidate set of fusion operations (addition, Hadamard product, concatenation, attention) using a one-shot differentiable algorithm. On three large-scale ad-click datasets, the learned fusions outperform existing CTR models, including expert-designed fusion and neural-architecture-search baselines, by margins of 0.0011, 0.0021, and 0.0036 AUC over the best baseline. A case study shows the learned fusions differ by dataset, which the paper reads as evidence that no single hand-crafted fusion design is best.","feed_headline":"Learned fusion design beats hand-set fusion in CTR models","feed_subtitle":"OptFusion automatically picks how model parts connect and combine, lifting AUC by up to 0.0036 on three ad-click sets.","key_machinery":"The central machinery is a one-shot differentiable search over a fusion-only component DAG. The graph contains an embedding block, $n$ shallow and $n$ deep interaction components, and an output block; each component may take inputs from any lower-level component, with connection presence decided by a straight-through unit-step function of $\\alpha$ and fusion operation expressed as a softmax-weighted mixture over ADD, PROD, CONCAT, and ATT. During selection, Eq. 16 optimizes model parameters $\\Theta$ together with $\\alpha$ and $\\beta$ on the training loss; after selection, $\\alpha^*$ is thresholded to binary connections, $\\beta^*$ is kept as a soft mixture (OptFusion-Soft) or hard-coded to the argmax operation (OptFusion-Hard), and $\\Theta$ is retrained by Eq. 17. This converts an exponentially large discrete fusion search, of size $O(2^{2n}\\cdot k^n)$, into a single continuous optimization.","core_discovery":"On its own terms, the paper's claim is that fusion—the choice of which components in a deep CTR model feed into which others, and how their outputs are combined—is a learnable architectural dimension, and that learning it end-to-end improves prediction. OptFusion parameterizes every candidate edge of a component DAG with a connection weight $\\alpha$ and every candidate fusion operation with a weight $\\beta$, relaxes the discrete choices via a straight-through estimator and a softmax, optimizes both together with model weights on the CTR loss during a selection stage, then fixes the selected architecture and retrains the model. In experiments, both the hard and the soft variant outperform all compared baselines on Criteo, Avazu, and KDD12; the reported AUC improvements over the best baseline are 0.0011, 0.0021, and 0.0036, marked statistically significant at $p < 0.05$. The paper further claims that the learned architectures differ across datasets, with parallel-favoring fusion on Criteo and stacked-favoring fusion on Avazu and KDD12, evidence that a single hand-designed fusion choice leaves performance on the table.","pith_inferences":["A natural transfer test the paper does not run: freeze the fusion architecture found on one dataset and measure how much of the AUC gain survives on another; if most of the gain evaporates, the value of fusion search is dataset-specific tuning rather than a universal architectural improvement.","Because architecture selection is performed on the same training set used for retraining, a held-out evaluation of the selection procedure itself would clarify whether the reported margins reflect genuine fusion quality or selection overfitting; the paper's significance tests compare final models, not selection robustness.","The same joint connection-and-operation learning idea could apply wherever hand-set fusion is standard, such as multi-modal or multi-task networks; success there would show the principle generalizes beyond CTR feature interactions.","The case study's finding that ADD and PROD dominate on two of three datasets suggests the practical gain of operation search comes mostly from choosing among parameter-free combiners, which may be a cheaper hypothesis to test than full operation search."],"forward_implications":["Existing fixed fusion designs leave measurable accuracy on the table; selecting fusion automatically yields consistent AUC and LogLoss improvements over stacked, parallel, and expert-designed fusion on all three datasets.","The best fusion design is data-dependent, since OptFusion's search on Criteo converges to a parallel-leaning architecture while Avazu and KDD12 favor stacked structures; models deployed on new data should therefore search rather than reuse a preset fusion.","A fusion-only search space is more effective and faster than full neural architecture search for this problem: OptFusion beats the broader NAS baselines while keeping total training time lower.","Jointly learning connections and operations in one pass beats sequential selection, because the choice of fusion operation changes which connections are useful and vice versa.","The learned fusion transfers across explicit interaction components, so OptFusion can upgrade models built on different shallow blocks without redesigning them."],"supporting_citations":[{"why":"Supplies the one-shot learning template that OptFusion adapts for jointly learning fusion connections and operations.","marker":"[30]"},{"why":"Provides the straight-through estimator that makes discrete connection states differentiable during selection.","marker":"[1]"},{"why":"Supplies the selection-then-retraining paradigm and the thresholding rule used to fix connections after search.","marker":"[18]"},{"why":"A neural-architecture-search baseline with a much larger search space; the comparison shows a fusion-only search is more effective.","marker":"[31]"},{"why":"A weight-sharing neural-architecture-search baseline whose official implementation is reused; the central efficiency and effectiveness comparison.","marker":"[42]"},{"why":"The expert-designed dense-fusion model that is the strongest baseline; OptFusion must beat it on all datasets.","marker":"[3]"},{"why":"A baseline whose stacked and parallel variants perform differently across datasets, motivating the need for learned fusion.","marker":"[37]"},{"why":"Records that fusion design performance varies across datasets, evidence that fusion design matters.","marker":"[38]"},{"why":"A parallel-fusion baseline and the cited convention that a 0.1 percent AUC improvement counts as significant.","marker":"[5]"}],"fun_headline_variants":["OptFusion learns fusion design, boosting CTR AUC on 3 datasets","Auto fusion: OptFusion picks connections and ops for better CTR","Learning fusion connections and operations lifts CTR AUC","OptFusion automates fusion design, beating fixed connections in ad CTR","Fusion design is learnable: OptFusion boosts CTR on real datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Architecture parameters are selected on the same training set later used to retrain the model, so the method assumes that minimizing the training loss during selection finds fusion designs that generalize to new clicks rather than designs that only fit the training data.","fun_headline_variants_meta":{"raw":{"variants":["OptFusion learns fusion design, boosting CTR AUC on 3 datasets","Auto fusion: OptFusion picks connections and ops for better CTR","Learning fusion connections and operations lifts CTR AUC","OptFusion automates fusion design, beating fixed connections in ad CTR","Fusion design is learnable: OptFusion boosts CTR on real datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2703,"prompt_tokens":1022,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":1605}},"tokens_in":638,"tokens_out":1681,"duration_ms":12081,"temperature":1.0,"reasoning_tokens":1605,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:57:29.786087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OptFusion's selection stage on one split of a dataset, then evaluate the chosen fusion architecture on a disjoint held-out split against a fixed naive fusion design; if the gap disappears or reverses, the reported gains come from selection overfitting rather than from learning better fusions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the one-shot learning template that OptFusion adapts for jointly learning fusion connections and operations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the selection-then-retraining paradigm and the thresholding rule used to fix connections after search."}],"review_version":1}