REVIEW 4 major objections 6 minor 37 references
HIT Model: A Hierarchical Interaction-Enhanced Two-Tower Model for Pre-Ranking Systems
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Two-tower pre-ranking model with generators and multi-head matching lifts online GMV by 1.66% and ROI by 1.55% while keeping serving latency near vanilla two-tower.
desk verdict HIT reports a deployed pre-ranking model with real gains, but the printed generation loss contradicts the paper's own cross-tower interaction story, so the code is the make-or-break. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pair of generators $g_k(\cdot)$ (two-layer MLPs taking only static features) that produce mimic vectors $\mathbf{r}^m_k$, trained by the generation loss $\mathcal{L}_{g_u}=-\frac{1}{N}\sum_i (y_i\,\mathrm{Dist}(\mathbf{r}_u,\mathbf{r}^m_1)+(1-y_i)\,\mathrm{Dist}(\mathbf{r}_u,\mathbf{r}^m_2))$ (and its ad-tower analogue), together with the multi-head representers of Eqs. (5)-(6) and the max-then-sum scoring rule of Eq. (7). The generators' outputs are L2-normalized, concatenated with the user dense vector $\mathbf{e}_u$, and fed into the tower DNN, so the generated information influences the embeddings that the representers see. 'Stop gradient' operations on both the generator input/output and the representer output used in the generation loss are intended to let the generators fit high-level representations without disturbing the backbone or letting the auxiliary loss interfere with the main training. The multi-head representer then partitions each tower's embedding into $J$ subspaces; the matching score is the sum over user heads of the maximum similarity to any ad head, which the paper interprets as finding the most relevant ad attribute for each user interest while keeping ad embeddings pre-cacheable.
What would settle it
Inspect the released source code for the generation loss: if the user-tower generator's target in Eq. (8) is the user tower's own multi-head representation $\mathbf{r}_u$ with the printed negative sign, then the generator never receives ad-side information and the paper's central mechanism is contradicted; alternatively, run an ablation that trains each generator against the opposite tower's representation instead of the same tower's—if HIT's AUC is unchanged, the claimed interaction is not the source of the gains.
Extended reading notes
Core claim
The paper's central claim is that a two-tower pre-ranking model can be made 'interaction-enhanced' by (1) adding a dual-generator module, in which two small MLPs (one per label type, here positive/negative) produce 'mimic' vectors from static features that are concatenated into the tower inputs, with a cosine-similarity generation loss aligning those mimic vectors with the tower's high-level representer output; and (2) replacing the single dot product with a multi-head representer that projects user and ad embeddings into $J$ latent subspaces and computes $\hat{y} = \sum_{j_u=1}^{J}\max_{j_a}(\mathbf{r}_{u,(j_u)})^\top \mathbf{r}_{a,(j_a)}$. The paper argues that the generators simulate the opposite tower's representations, 'exposing' each tower to cross-domain information before encoding, and that the multi-head representer captures multi-faceted user interests and multi-dimensional ad attributes. It further claims that this design preserves the two-tower serving advantage because ad representations are pre-computed and cached; only the user side is computed at request time. The paper reports consistent offline superiority across MovieLens, Amazon (Electro), and Alibaba datasets, and online gains in GMV and ROI on Tencent's platform.
Load-bearing premise
The model only deserves the name 'interaction-enhanced' if each tower's generator is genuinely trained on information from the other tower; if the training loss only mirrors a tower's own representation, the cross-domain interaction story collapses, even if the accuracy numbers survive.
Editorial extensions
If this is right
- Pre-ranking systems can adopt HIT as a drop-in upgrade to an existing two-tower pipeline, since it adds only static-feature MLPs and linear head projections to the served model.
- The reported online A/B gains (GMV +1.66%, ROI +1.55%) suggest that the interaction improvements measured offline by AUC translate into revenue-relevant ranking changes at scale.
- The max-then-sum matching generalizes late-interaction scoring (e.g., MaxSim-style matching) to a pre-cached two-tower setting, providing a template for other retrieval/pre-ranking models.
- Because HIT separates generators by label type, the architecture can be extended to multi-task or multi-class labels by increasing the number of generators $K$, as the paper itself notes.
- The 'stop gradient' recipe shows how to add auxiliary representation-mimicking losses to a two-tower model without destabilizing the main CTR training.
Reading between the lines
- A close reading of Eq. (8) suggests the printed generation loss may train each generator against its own tower's multi-head representation rather than the opposite tower's; if the released code matches the equations, the model's gains would come from a form of self-distillation or regularization rather than the claimed cross-tower interaction, and the mechanism would need to be re-described even i
- The static-only generator input trades away dynamic features at training time by design; a natural extension is train-time distillation of dynamic signals into the static generator, which could recover some of the lost information without online cost.
- The paper's hierarchy (coarse-grained generation then fine-grained matching) parallels distillation-plus-MaxSim structures in retrieval; one could test whether a single well-chosen generator shared across label types performs comparably, isolating the value of the dual-generator split.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HIT, a two-tower pre-ranking model for online display advertising, augmented with two components: generators that are claimed to pre-generate holistic vectors incorporating coarse-grained user-ad interactions, and multi-head representers that project embeddings into multiple latent subspaces for fine-grained matching. The authors report offline improvements over ten baselines on MovieLens, Amazon (Electro), and Alibaba datasets, and online A/B test gains of +1.66% GMV and +1.55% ROI on Tencent's platform, with code released. The central novelty is the generation loss, which is intended to transfer coarse-grained cross-tower information between the user and ad towers.
Significance. If the mechanism worked as described, HIT would be a valuable industrial contribution: it preserves two-tower serving efficiency while incorporating interaction signals, and the paper provides public-dataset experiments, ablation studies, an online A/B test, and open-source code. However, the printed equations do not implement the claimed cross-tower generation, and the sign of the generation loss contradicts its stated objective. The empirical results cannot be interpreted as evidence for the proposed interaction mechanism until this is resolved.
major comments (4)
- [Section 4.3.1, Eq. (8)] The generation loss L_gu is written as -1/N Σ [y_i Dist(r_u, r^m_1) + (1-y_i) Dist(r_u, r^m_2)], where r_u is the user tower's own multi-head representation. This compares the user generator outputs with the same tower's representation, not with the ad tower's representation as stated in Section 4.1 ('the generators in the user tower simulate ad representations'). The cross-domain interaction mechanism, which is the paper's central contribution, is therefore absent from the printed mathematics. Furthermore, since Dist is defined as cosine distance, the leading negative sign makes the objective maximize rather than minimize the distance, contradicting the sentence in Section 4.3.1 that the loss 'is designed to minimize the distance between r_u and r^m_k'. The authors should either correct Eq. (8) (and the corresponding ad-tower loss) or provide evidence from the released code that the implemented loss differs from the printed one; as written, the claim of interaction enhancement is unsupported.
- [Section 4.1 and Eq. (3)] The description of the generator in Eq. (3) is ambiguous with respect to the generator's target. The text says r^m_{k,δ} denotes 'mimic users/ads' and that the user-tower generator 'simulates ad representations', but the generator takes only user static features x_ub as input and Eq. (8) ties its output to r_u. If the generator is meant to output an ad-like vector from user features, the loss should compare it with r_a (or an ad-side quantity); if it is meant to output a user-like vector, the cross-tower story in the introduction is incorrect. Either way, the manuscript must clarify what the generator produces and against which representation it is trained.
- [Section 5.2, Table 2] The claim that 'the improvement is significant at α=0.01' appears without any reported standard deviations, error bars, or description of the statistical test (e.g., paired t-test across repetitions). The table reports only average values over 10 repetitions, which is insufficient to substantiate the significance statement for the paper's main offline result.
- [Section 5.5, Table 4] The online A/B test reports +1.66% GMV and +1.55% ROI relative to the MVKE control, but no confidence intervals, significance levels, or information about the number of users/impressions assigned to each arm are provided. Given that these numbers are used to support the deployment claim, a statistical characterization is necessary to assess whether the gains are meaningful and not within normal fluctuation.
minor comments (6)
- [Section 4.3.1, Eq. (8)] The sample index i does not appear on r_u, r^m_1, or r^m_2; clarify that these quantities are per-sample.
- [Section 4.2, Eq. (7)] The notation j_u and j_a for the head indices is inconsistent with the earlier use of j; use a consistent subscript convention.
- [Figure 3] The labels 'Mimic User' and 'Mimic Ad' next to the generators are inconsistent with Eq. (8), which trains the generators against the same tower's representation.
- [Section 5.1] Hyperparameters α, J, and z are fixed without any sensitivity analysis or tuning procedure; report how these were chosen.
- [Section 5.5, Figure 6] The 'Success Rate' metric is not defined; state what it measures.
- [General] The paper uses the ACM template with 'Conference acronym 'XX' and copyright 2018 placeholders; these need to be updated before publication.
Circularity Check
No significant circularity: HIT's empirical claims are benchmarked externally; the generation-loss inconsistency is an internal correctness issue, not a circular derivation.
full rationale
The derivation chain of HIT is not circular in the sense targeted by this analysis. The main objective (Eq. 9) is cross-entropy on observed click labels; the generation loss (Eq. 8) is an auxiliary self-distillation term with stop-gradient operations (Sec. 4.3.3), not a fitted quantity that is later reported as a prediction. Offline AUC/CEloss comparisons in Table 2 are against external baselines (DSSM, DAT, MVKE, IntTower, Wide&Deep, DCN, AutoInt, COLD), and the online A/B test in Sec. 5.5 compares against a deployed MVKE control on GMV/ROI. No parameter is fitted to a subset and then 'predicted' on a closely related quantity. The self-citations (Refs. [4], [26]) appear as background examples of Tencent advertising research and are not load-bearing for the HIT architecture. The most salient concern in the paper is not circularity but internal consistency: Eq. (8) contains only r_u, the user tower's own multi-head representation, and with Dist defined as cosine distance the leading negative sign would maximize rather than minimize distance. This would undermine the claimed cross-tower mimicry mechanism if the implementation matches the printed equations. However, an equation that fails to implement the stated mechanism is a correctness/consistency problem, not a case of the paper's conclusions being equivalent to its inputs by construction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- alpha (generation loss weight) =
1e-3
- J (number of representer heads) =
2
- K (number of generators) =
2
- z (representer output dimension) =
16
assumptions (4)
- standard math Cosine distance and L2 normalization behave as stated in Eqs. (2), (7) and (8).
- domain assumption Pre-cached ad embeddings are the binding serving constraint.
- domain assumption Static features alone suffice for generator input.
- domain assumption Random 80/20 split of interaction samples is a valid evaluation protocol.
Cite this review
Pith. "Pith review of HIT Model: A Hierarchical Interaction-Enhanced Two-Tower Model for Pre-Ranking Systems." pith.science (2026). https://pith.science/paper/JKG7QNJS
@misc{pith2026250519849,
author = {Pith},
title = {Pith review of: HIT Model: A Hierarchical Interaction-Enhanced Two-Tower Model for Pre-Ranking Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/JKG7QNJS}},
note = {Machine review of arXiv:2505.19849}
}
abstract
Online display advertising platforms rely on pre-ranking systems to efficiently filter and prioritize candidate ads from large corpora, balancing relevance to users with strict computational constraints. The prevailing two-tower architecture, though highly efficient due to its decoupled design and pre-caching, suffers from cross-domain interaction and coarse similarity metrics, undermining its capacity to model complex user-ad relationships. In this study, we propose the Hierarchical Interaction-Enhanced Two-Tower (HIT) model, a new architecture that augments the two-tower paradigm with two key components: $\textit{generators}$ that pre-generate holistic vectors incorporating coarse-grained user-ad interactions through a dual-generator framework with a cosine-similarity-based generation loss as the training objective, and $\textit{multi-head representers}$ that project embeddings into multiple latent subspaces to capture fine-grained, multi-faceted user interests and multi-dimensional ad attributes. This design enhances modeling effectiveness without compromising inference efficiency. Extensive experiments on public datasets and large-scale online A/B testing on Tencent's advertising platform demonstrate that HIT significantly outperforms several baselines in relevance metrics, yielding a $1.66\%$ increase in Gross Merchandise Volume and a $1.55\%$ improvement in Return on Investment, alongside similar serving latency to the vanilla two-tower models. The HIT model has been successfully deployed in Tencent's online display advertising system, serving billions of impressions daily. The code is available at https://github.com/HarveyYang123/HIT_model.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al
-
[2]
Jiaping Gui, Stuart Mcilroy, Meiyappan Nagappan, and William GJ Halfond. 2015. Truth in advertising: The hidden cost of mobile ads for software developers. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , Vol. 1. IEEE, 100–110
work page 2015
-
[3]
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction.arXiv preprint arXiv:1703.04247 (2017)
arXiv 2017
-
[4]
Mengzhuo Guo, Wuqi Zhang, Congde Yuan, Binfeng Jia, Guoqing Song, Hua Hua, Shuangyang Wang, and Qingpeng Zhang. 2024. A Bayesian Multi-Armed Bandit Algorithm for Bid Shading in Online Display Advertising. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management . 4506–4513
work page 2024
-
[5]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management . 2333–2338
2013
-
[6]
Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. 2019. FiBiNET: combining fea- ture importance and bilinear feature interaction for click-through rate prediction. In Proceedings of the 13th ACM conference on recommender systems . 169–177
2019
-
[7]
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969 (2019)
arXiv 2019
-
[8]
Cong Jiang, Zhongde Chen, Bo Zhang, Yankun Ren, Xin Dong, Lei Cheng, Xinxing Yang, Longfei Li, Jun Zhou, and Linjian Mo. 2024. GATS: Generative Audience Targeting System for Online Advertising. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2920–2924
work page 2024
Show all 37 references
-
[9]
Dan Li, Yang Yang, Hongyin Tang, Jiahao Liu, Qifan Wang, Jingang Wang, Tong Xu, Wei Wu, and Enhong Chen. 2022. VIRT: Improving representation-based text matching via virtual interaction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing ...
2022
-
[10]
Xiangyang Li, Bo Chen, HuiFeng Guo, Jingjie Li, Chenxu Zhu, Xiang Long, Sujian Li, Yichao Wang, Wei Guo, Longxia Mao, et al. 2022. Inttower: the next generation of two-tower model for pre-ranking system. In Proceedings of the 31st ACM International Conference on Information & ...
2022
-
[11]
Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, et al. 2022. Ernie- search: Bridging cross-encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval. arXiv prepri...
2022 arXiv
-
[12]
H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al
-
[13]
Wentao Ouyang, Xiuwu Zhang, Shukui Ren, Li Li, Kun Zhang, Jinmei Luo, Zhaojie Liu, and Yanlong Du. 2021. Learning graph meta embeddings for cold-start ads in click-through rate prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Developmen...
2021
-
[14]
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang
-
[15]
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. Autoint: Automatic feature interaction learning via self- attentive neural networks. InProceedings of the 28th ACM international conference on information and knowledge management ...
2019
-
[16]
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008)
2008
-
[17]
In 2016 IEEE 16th international conference on data mining (ICDM)
Product-based neural networks for user response prediction. In 2016 IEEE 16th international conference on data mining (ICDM) . IEEE, 1149–1154
2016
-
[18]
Yaozheng Wang, Dawei Feng, Dongsheng Li, Xinyuan Chen, Yunxiang Zhao, and Xin Niu. 2016. A mobile recommendation system based on logistic regression and gradient boosting decision trees. In 2016 international joint conference on neural networks (IJCNN). IEEE, 1896–1902
2016
-
[19]
Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, and Furu Wei
-
[20]
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. In Proceedings of the ADKDD’17 . 1–7
2017
-
[21]
Yanheng Wei, Lianghua Huang, Yanhao Zhang, Yun Zheng, and Pan Pan. 2022. An intelligent advertisement short video production system via multi-modal retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . 3368–3372
2022
-
[22]
Melanie Wiese, Carla Martínez-Climent, and Dolores Botella-Carrubi. 2020. A framework for Facebook advertising effectiveness: A behavioral perspective. Journal of Business Research 109 (2020), 76–87
2020
-
[23]
Zhenhui Xu, Meng Zhao, Liqun Liu, Lei Xiao, Xiaopeng Zhang, and Bifeng Zhang
-
[24]
Zhe Wang, Liqin Zhao, Biye Jiang, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai
-
[25]
Yantao Yu, Weipeng Wang, Zhoutian Feng, and Daiyue Xue. 2021. A dual aug- mented two-tower model for online large-scale recommendation. DLP-KDD (2021)
2021
-
[26]
Congde Yuan, Mengzhuo Guo, Chaoneng Xiang, Shuangyang Wang, Guoqing Song, and Qingpeng Zhang. 2022. An actor-critic reinforcement learning model for optimal bidding in online display advertising. In Proceedings of the 31st ACM International Conference on Information & Knowledg...
2022
-
[27]
Hengyu Zhang, Junwei Pan, Dapeng Liu, Jie Jiang, and Xiu Li. 2024. Deep Pattern Network for Click-Through Rate Prediction. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1189–1199
2024
-
[28]
Wei Zhang, Dai Li, Chen Liang, Fang Zhou, Zhongke Zhang, Xuewei Wang, Ru Li, Yi Zhou, Yaning Huang, Dong Liang, et al. 2024. Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta. In Companion Proceedings of the ACM on Web Conference 20...
2024
-
[29]
Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiwang Yang, Xiaobing Liu, Jiliang Tang, and Hui Liu. 2021. Dear: Deep reinforcement learning for online advertising impression in recommender systems. In Proceedings of the AAAI conference on artificial intelligence , Vol. 35. 750–758
2021
-
[30]
Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. 2019. Sampling-bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM conference on recommender systems . 269–277
2019
-
[31]
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...
2018
-
[36]
Zuowu Zheng, Changwang Zhang, Xiaofeng Gao, and Guihai Chen. 2022. HIEN: hierarchical intention embedding network for click-through rate prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . 322–331
2022
-
[2013]
In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining . 1222–1230
-
[2016]
In Proceedings of the 1st workshop on deep learning for recommender systems
Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems . 7–10
-
[2020]
CoRR abs/2007.16122 (2020)
COLD: Towards the Next Generation of Pre-Ranking System. CoRR abs/2007.16122 (2020). arXiv preprint arXiv:2007.16122 (2020)
2020 arXiv
-
[2021]
arXiv preprint arXiv:2112.08723 (2021)
Distilled dual-encoder model for vision-language understanding. arXiv preprint arXiv:2112.08723 (2021)
2021 arXiv
-
[2022]
In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Mixture of virtual-kernel experts for multi-objective user profile modeling. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4257–4267
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.