{"id":"ed0753fe-d1e1-4181-b67e-a80faa3b0339","arxiv_id":"2411.09852","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A bidirectional, interleaved interaction module with a separate Cross Arch for selective summarization improves CTR prediction over unidirectional fusion baselines by small margins on public and industrial data.","lead":"InterFormer is a CTR prediction module that lets static user features and behavior sequences inform each other in both directions, layer by layer, while summarizing information through a separate gating path. The authors report small but consistent gains on three public benchmarks and a 0.15% normalized-entropy gain plus 24% QPS gain in Meta's internal Ads deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on sub-0.001 AUC gaps from single runs; the paper's own Table 3 already shows a gAUC deficit on KuaiVideo, so statistical significance and consistency are the load-bearing unknowns.","rationale":"The reader's conditional verdict is well calibrated, and the weakest assumption is correctly identified as run-to-run variance: the public benchmark improvements are of size 0.0002-0.0014 in absolute AUC, a range that is commonly within training noise unless paired significance testing is reported. I read the architecture as internally coherent: the bidirectional flows in Eqs. (7)-(9) are clearly specified, the Cross Arch is a reasonable design, and the Figure 2 ablation orders align with the paper's story. However, Table 3 adds a sharper, non-statistical problem that reinforces the reader's concern: InterFormer's gAUC on KuaiVideo is below DIEN's, so the phrase 'consistently outperforms' and 'state-of-the-art performance on three public datasets' is not literally supported by the paper's own numbers, regardless of significance. The industrial claims are difficult to audit because the baseline is unnamed and the 24% QPS gain is entangled with system co-design. None of this changes the appropriate verdict: the paper is conditionally acceptable, with the condition being multi-seed reproducible experiments, significance tests, correction or qualification of the KuaiVideo gAUC result, and disclosure of the industrial baseline and code. If the requested checks fail, the central SOTA claim should be downgraded; if they pass, the conditional is satisfied.","tokens_in":19404,"tokens_out":7326,"duration_ms":68886,"concrete_test":"Run InterFormer and the strongest baselines (DIEN, TransAct) on TaobaoAds and KuaiVideo with at least 10 random seeds under the BARS configuration, and report mean ± std plus paired bootstrap 95% confidence intervals for AUC and gAUC. Compute the fraction of runs in which InterFormer beats the best baseline on every dataset and metric; if the Taobao AUC gap (0.0009) or the KuaiVideo gAUC gap (-0.0014) is not significant at p<0.05, the 'consistent SOTA' claim fails. Also re-run the Figure 2 variants (sole, sep, n2s, s2n, int) with matched parameter counts to separate the interleaving effect from capacity, and release the code so the Table 3 numbers can be reproduced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"InterFormer's central claim is that it consistently beats SOTA on three public benchmarks plus an internal dataset. The public support is Table 3, and the decisive margins are tiny: TaobaoAds AUC 0.6528 vs 0.6519 (an absolute gap of 0.0009), Amazon AUC 0.8865 vs 0.8851 (0.0014), and KuaiVideo AUC 0.7453 vs 0.7451 (0.0002). Section 5.2.1 reports these as single runs with no seeds, error bars, confidence intervals, or significance tests, so none of the SOTA margins can be distinguished from training noise. The problem is amplified by an internal inconsistency in the same table: on KuaiVideo, InterFormer's gAUC is 0.6637, lower than DIEN's 0.6651, while Section 5.2.1 asserts consistent SOTA and 'outperforms the best competitor by up to 0.9% in gAUC.' If gAUC is a primary metric, the claim is already false on one of three datasets; if it is not, the paper should say so. The industrial 0.15% NE gain and 24% QPS gain are self-reported against an unnamed internal baseline, without confidence intervals or an ablation separating architectural gains from the model-system co-design optimizations described in Section 5.3.2, so they cannot independently rescue the benchmark claim. The architecture is coherent and the Figure 2 ablations directionally support bidirectional interleaving, but those ablations do not control parameter counts, leaving open the possibility that part of the gain is added capacity rather than the interleaving mechanism itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InterFormer, a module for click-through rate prediction that learns heterogeneous information interactions between non-sequence features and user behavior sequences in an interleaving style. The architecture combines an Interaction Arch for behavior-aware non-sequence feature interactions, a Sequence Arch with personalized feed-forward networks and multi-head attention for context-aware sequence modeling, and a Cross Arch that selectively summarizes gated information for exchange between the two modes. The authors report state-of-the-art AUC, gAUC, LogLoss, and NE results on three public benchmarks and an internal Meta dataset, an industrial deployment with 0.15% NE gain and 24% QPS gain, and ablations supporting the bidirectional interleaving design.","tokens_in":19737,"tokens_out":3059,"duration_ms":33097,"significance":"If the empirical claims hold, InterFormer offers a useful and broadly applicable architectural pattern for CTR prediction, extending earlier unidirectional sequence-to-nonsequence designs with a bidirectional, interleaved information flow and a separate summarization arch. The architecture equations are internally coherent, the method is compatible with several interaction backbones, and the public-benchmark experiments are conducted within the BARS framework, which is a reproducible setup. The ablations in Figure 2 show a consistent ordering best-to-worst of int, n2s approximately s2n, sep, sole across three backbones, which directionally supports the central mechanism. However, the headline state-of-the-art claims rest on very small differences from single runs, and one stated gAUC claim is contradicted by the paper's own table, so the empirical support currently lags behind the strength of the claims.","major_comments":[{"comment":"The text states that InterFormer outperforms the best competitor by up to 0.9% in gAUC, but Table 3 shows that InterFormer's gAUC on KuaiVideo is 0.6637, which is lower than DIEN's 0.6651. The largest gAUC gain in Table 3 is 0.0008 on AmazonElectronics (0.8843 vs. 0.8835), which is about 0.09% relative, not 0.9%. This internally inconsistent claim must be corrected, and the paper should either identify the dataset and metric on which each percentage gain is achieved or qualify that gAUC is not a primary metric.","section":"§5.2.1, Table 3"},{"comment":"The state-of-the-art claim is based on single-run benchmark scores with no error bars, confidence intervals, or significance tests. The decisive margins are very small: 0.0009 AUC on TaobaoAds, 0.0014 AUC on AmazonElectronics, and 0.0002 AUC on KuaiVideo, all of which could plausibly be within run-to-run variance for these datasets and model families. Without multiple seeds or a significance test, the central claim that InterFormer consistently achieves state-of-the-art AUC on public benchmarks is not yet established.","section":"§5.2.1, Table 3"},{"comment":"The ablations that attribute performance gains to the interleaving learning style and to selective information aggregation are not parameter-matched. The int, n2s, s2n, sep, and sole scenarios differ in the number and size of modules, and the selective aggregation comparison in Figure 3 varies the aggregation mechanism without controlling for capacity. As a result, part of the observed improvement could be due to added model capacity rather than to the bidirectional interleaving or selective aggregation mechanism itself. Parameter-matched ablations, or equivalent-capacity baselines, are needed to support the mechanism-level interpretation.","section":"§5.2.2, Figures 2 and 3"},{"comment":"The industrial results are reported as a 0.15% NE gain and a 24% QPS gain against an unnamed internal SOTA model, with no confidence intervals or run-to-run variability information. Additionally, the QPS gain appears to include the model-system co-design optimizations described in Section 5.3.2 (communication overlap, FLOP reallocation, and kernel fusion), so the architecture's own efficiency contribution is not separated from systems engineering. The paper should identify the baseline model, state how many runs or evaluation periods were used, and disentangle architectural from systems-level gains.","section":"§5.3.1 and §5.3.2"}],"minor_comments":[{"comment":"The sentence beginning 'public benchmark datasets are carried out in Section 5.2' is ungrammatical; it should be revised to 'public benchmark experiments are carried out' or similar.","section":"§5.1"},{"comment":"The AmazonElectronics dataset description says there are 1,689,188 samples, but the subsequent split mentions 2.60M training and 0.38M test samples, which total about 2.98M and are inconsistent with the stated sample count. This should be corrected or clarified.","section":"Appendix B.1"},{"comment":"The figure caption says the CLS token appending happens only at the first layer, and step 3 of Algorithm 1 similarly performs the prepending before the main loop, but the main loop in Algorithm 1 does not explicitly mark this one-time step. Making the one-time nature explicit in the algorithm pseudocode would improve clarity.","section":"Figure 1 and Algorithm 1"},{"comment":"The related work section cites many papers on graph learning and time-series forecasting that are not directly relevant to CTR prediction or heterogeneous interaction learning; tightening the citation list to the most relevant prior work would improve readability.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The central architecture is coherent and the direction of the contribution is appropriate for the journal, but the empirical support needs substantial strengthening before the state-of-the-art claim can be accepted. The internal inconsistency between the text's gAUC claim and Table 3 is the kind of issue that should be caught by the authors and the reviewer process. I would also note that the reference list contains a large number of citations to the authors' own prior work and to papers co-authored by one of the senior authors, several of which are only tangentially related; this may warrant a nudge toward a more selective citation list."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"InterFormer is a plausible, cleanly described compositional architecture for CTR. It interleaves non-sequence and sequence streams bidirectionally with a separate Cross Arch for summarization, and the ablations in Figure 2 support the central mechanism claim: bidirectional information flow (int) beats unidirectional (n2s, s2n) and early aggregation (sole, sep) across three interaction backbones. That is real evidence and worth crediting. The industrial story – 0.15% NE gain and 24% QPS gain at Meta scale – is relevant even if it is self-reported.\n\nThe soft spot is the empirical support for the headline SOTA claim. The decisive margins are sub-0.001 AUC (TaobaoAds 0.0009, KuaiVideo 0.0002, Amazon 0.0014), all from single runs with no seeds, error bars, or significance tests. Those numbers could easily flip under run-to-run variance. Worse, Table 3 shows InterFormer's gAUC on KuaiVideo (0.6637) below DIEN's (0.6651), while Section 5.2.1 says it outperforms the best competitor by up to 0.9% in gAUC. That is an internal inconsistency, and the 0.9% figure does not match any arithmetic reading of the table. It needs correction and multi-seed reporting.\n\nTwo further concerns, both moderate. The Figure 2 ablations do not control parameter count, so part of the gain may be added capacity rather than the interleaving mechanism itself. And the internal 0.15% NE gain is against an unnamed baseline, bundled with the model-system co-design optimizations of Section 5.3.2; without an ablation separating architecture from system tricks, the internal claim cannot independently verify the benchmark story. These issues are addressable. They do not sink the architecture – the mechanism evidence is directionally sound – but they do sink the current SOTA claim.\n\nThis paper is for people working on CTR architectures and industrial recommendation systems. It is a useful case study in combining known parts and in how to evaluate such combinations. It deserves a serious referee, though the referee should demand multi-seed variance, a corrected gAUC statement, and code or detailed experimental records before accepting the SOTA claim.","headline":"A sensible compositional CTR architecture with a clean ablation story, but the headline SOTA margins come from single runs and one table contradicts the text's gAUC claim.","tokens_in":20400,"tokens_out":2836,"would_cite":true,"duration_ms":25289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InterFormer claims that bidirectional interleaving of static and sequence features improves click-through rate prediction, with up to 0.14% AUC gain on benchmarks and 0.15% NE gain in an industrial deployment.","keywords":["CTR prediction","heterogeneous information","bidirectional interaction","sequence modeling","feature interaction","interleaving architecture","selective aggregation","industrial recommendation"],"falsifier":"Re-run the benchmark comparisons with multiple random seeds (for example, ten) on one public dataset such as Amazon-Electronics and compute confidence intervals for the AUC difference between InterFormer and the strongest baseline; if the interval straddles zero, the SOTA claim collapses. For the industrial claim, an A/B test that fails to reproduce the 0.15% normalized-entropy gain or the 0.6% topline improvement at the same scale would falsify it.","tokens_in":19179,"feed_emoji":"📈","tokens_out":6959,"duration_ms":64000,"temperature":0.7,"pith_summary":"This paper argues that click-through rate prediction is held back by two design choices in existing models: information flows only one way, from static features to behavior sequences, and sequence information is aggressively summarized early on. InterFormer is proposed as a module that interleaves non-sequence and sequence processing across stacked layers, letting each mode inform the other, while a separate Cross Arch selects and compresses information before exchange. The authors report state-of-the-art results on three public benchmarks, with up to 0.14% AUC improvement over the strongest baseline, and an industrial-scale evaluation on a 70-billion-sample dataset showing a 0.15% normalized-entropy gain and 24% higher query throughput. If these margins hold, the design offers a reusable building block for large-scale advertising and recommendation systems.","feed_headline":"Bidirectional signal exchange lifts click prediction by 0.14% AUC","feed_subtitle":"InterFormer interleaves static and sequence features, gaining 0.15% NE and 24% QPS at industrial scale.","key_machinery":"The load-bearing object is the InterFormer block, defined by three cooperating arches. The Interaction Arch takes non-sequence features plus sequence summarization as input and returns behavior-aware non-sequence embeddings; the Sequence Arch combines a Personalized FeedForward Network, which projects sequence embeddings using non-sequence summarization as a query, with multi-head attention and a prepended CLS token; the Cross Arch gates and summarizes both modes before exchange. Because each arch preserves the shape of its input, the paper can stack multiple InterFormer layers without aggressive pooling, and the same block is compatible with different interaction backbones such as dot product, DCNv2, and DHEN.","core_discovery":"The central claim is that heterogeneous information in CTR prediction is best integrated by bidirectional, interleaved interaction rather than by unidirectional conditioning or early concatenation. InterFormer alternates an Interaction Arch, which models feature interactions among non-sequence embeddings enriched with summarized behavior, and a Sequence Arch, which models behavior sequences with multi-head attention conditioned on summarized non-sequence context. The Cross Arch keeps each mode's full representation intact while extracting gated low-dimensional summaries—CLS tokens, pooling-by-multihead-attention tokens, and recent items for sequences; gated MLP compression for non-sequences—so that neither mode is pooled prematurely. The authors assert that this design yields mutually beneficial learning, and they support it with ablations in which bidirectional flow consistently beats both unidirectional directions and selective aggregation beats average pooling, MLP, and multi-head-attention early summarization.","pith_inferences":["A direct test of the core claim is to run the public-benchmark comparisons across multiple seeds and report confidence intervals; this would separate the small AUC and NE margins from training noise.","The separate-summarization principle could transfer to other multi-modal ranking tasks where token counts differ, since the Cross Arch decouples selection from interaction.","If the 24% query-throughput gain is robust, it suggests that co-scheduling communication-bound interaction modules and computation-bound sequence modules could speed up other large-scale ranking workloads.","The paper's observation that adding long sequences improves NE by 0.14% hints that interleaving may increase the benefit of sequence feature scaling, a hypothesis testable on public long-sequence datasets."],"forward_implications":["InterFormer is claimed to beat 11 state-of-the-art baselines on Amazon-Electronics, TaobaoAds, and KuaiVideo, with gains up to 0.9% in gAUC, 0.14% in AUC, and 0.54% in LogLoss.","A three-layer InterFormer is claimed to improve normalized entropy by 0.15% over the internal state-of-the-art model at similar FLOPs, while delivering a 24% queries-per-second gain from overlapping communication and computation.","The bidirectional interleaving style is claimed to be universally beneficial across dot-product, DCNv2, and DHEN backbones, with performance ordered from weakest to strongest across scenarios: no exchange, separate arches, single-direction flows, and full bidirectional flow.","The results claim that selective aggregation improves CTR quality compared to aggressive early summarization by average pooling, MLP, or multi-head attention.","Pilot launches of the model in the paper's industrial advertising system are claimed to have produced a 0.6% improvement in topline metrics."],"supporting_citations":[{"why":"provides the Deep & Cross network used as a backbone Interaction Arch and as a baseline.","marker":"[47]"},{"why":"provides the DHEN ensemble used as the Interaction Arch backbone in experiments.","marker":"[79]"},{"why":"supplies the multi-head attention mechanism central to the Sequence Arch.","marker":"[43]"},{"why":"is a sequential baseline representing the unidirectional flow the paper argues against.","marker":"[85]"},{"why":"is a sequential baseline whose attention module is compared against InterFormer.","marker":"[84]"},{"why":"supplies pooling-by-multihead-attention tokens used as sequence summarization.","marker":"[17]"},{"why":"supplies the self-gating mechanism used in Cross Arch summarization.","marker":"[4]"},{"why":"supplies the MaskNet used to unify and denoise multiple behavior sequences.","marker":"[50]"},{"why":"supplies rotary position embeddings applied to sequence tokens.","marker":"[38]"},{"why":"provides the evaluation pipeline used for public benchmark experiments.","marker":"[86]"}],"fun_headline_variants":["Interleaved heterogeneous interaction lifts CTR accuracy","Avoid early pooling, interleave signals for better CTR","Bidirectional interleaving beats one-way CTR models","InterFormer interleaves modes for sharper click prediction","Learn heterogeneous signals in both directions for CTR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported differences—roughly 0.0005 to 0.001 in AUC and 0.15% in normalized entropy—reflect genuine improvement rather than run-to-run variance, because the key benchmark and industrial tables report single runs without error bars or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["Interleaved heterogeneous interaction lifts CTR accuracy","Avoid early pooling, interleave signals for better CTR","Bidirectional interleaving beats one-way CTR models","InterFormer interleaves modes for sharper click prediction","Learn heterogeneous signals in both directions for CTR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2681,"prompt_tokens":905,"completion_tokens":1776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1704}},"tokens_in":521,"tokens_out":1776,"duration_ms":12839,"temperature":1.0,"reasoning_tokens":1704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:14:18.589211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark comparisons with multiple random seeds (for example, ten) on one public dataset such as Amazon-Electronics and compute confidence intervals for the AUC difference between InterFormer and the strongest baseline; if the interval straddles zero, the SOTA claim collapses. For the industrial claim, an A/B test that fails to reproduce the 0.15% normalized-entropy gain or the 0.6% topline improvement at the same scale would falsify it.","supporting_citations":[{"cited_title":"Highway Transformer: Self-Gating Enhanced Self-Attentive Networks","cited_arxiv_id":"2004.08178","evidence_quote":"supplies the self-gating mechanism used in Cross Arch summarization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the evaluation pipeline used for public benchmark experiments."}],"review_version":1}