REVIEW 3 major objections 3 minor 26 references
NAM: A Normalization Attention Model for Personalized Product Search In Fliggy
T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A normalization attention model that scales personalization by item popularity improves conversion prediction in Fliggy product search.
desk verdict A workmanlike industrial paper with a real 0.8% online lift, but the gate's motivation is partly confounded and the statistics are thinner than claimed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is Inverse Item Frequency, $\mathrm{IIF}_i = 1/\#U_i$, the reciprocal of the number of users who interacted with item $i$, used twice. In the Global Normalized Multi-Head Self-Attention (GN_MHSA) and Global Normalized Target Attention (GN_TA), an attention mask $M_{Q,K} = \mathrm{SiLU}(\sqrt{\mathrm{IIF}_Q (\mathrm{IIF}_K)^T} \odot W_{M_{Q,K}})$ multiplies the scaled dot-product, so attention between two items is suppressed or boosted according to the geometric mean of their IIFs, mimicking cosine similarity's frequency normalization. In the IIF-based personalization gate, the target item's own IIF is passed through SiLU with a learnable scalar and multiplied elementwise into the personalized representation, making personalization a monotone function of target-item rarity. The same scalar counting signal thus determines both which historical items are attended to and how strongly the personalized summary is used.
What would settle it
Take NAM and rerun the offline evaluation with the target-item IIF values randomly permuted across items while keeping all other features fixed: if CTCVR AUC and GAUC do not drop materially, the personalization gate is not carrying the claimed effect. Alternatively, compute PCOC for NAM within single query-category pairs; if long-tail items in low-personalization categories such as commodity staples show no calibration gain, then the single scalar $\#U_i$ is not a sufficient popularity signal.
Extended reading notes
Core claim
The paper's central claim is that conversion estimation in personalized search is systematically miscalibrated by popularity, and that the calibration can be fixed inside the attention mechanism. Popular items are overestimated and long-tail items underestimated when attention relies on co-occurrence frequency; the paper's data tables show predicted-over-actual conversion ratios (PCOC) far from 1 at both extremes. NAM encodes inverse item frequency $\mathrm{IIF}_i = 1/\#U_i$ directly into attention: a SiLU-activated mask scales the query-key product in both the self-attention over user behavior and the target attention, and a separate gate $\mathrm{SiLU}(\mathrm{IIF}_{i_t} W_{\mathrm{IIF}_{i_t}}) \odot O'$ controls how much of the personalized representation reaches the towers. The outcome, according to the paper, is that NAM improves CTCVR AUC and GAUC over all baselines offline, narrows the PCOC gap for both popular and long-tail items, and lifts online conversion by 0.8%.
Load-bearing premise
The load-bearing premise is that the count of users who have interacted with an item is enough to know how much personalization that item needs, and that this relationship holds consistently across queries, categories, and time.
Editorial extensions
If this is right
- Attention weights in product search should carry an item-frequency normalization term, not just query-user relevance, to avoid systematic over- and underestimation.
- A single target-item popularity scalar can serve as the personalization gate, giving a cheap and interpretable answer to "when to personalize".
- Conversion models built on entire-space multi-task learning can gain up to 0.002 GAUC offline from this normalization without changing the tower structure.
- Deploying NAM in place of the production attention baseline should raise post-click conversion rate in e-commerce search by roughly 0.8% at similar traffic.
- The PCOC corrections imply popular-item exposure may become relatively less valuable and long-tail inventory more valuable, shifting ranking toward niche items.
Reading between the lines
- The paper leaves implicit that IIF is a stand-in for item maturity and bandwagon pressure; one could test whether sales volume, price-band, or category-specific popularity changes the learned gate.
- The gate depends only on the target item's IIF, but the paper's own Table 3 suggests query intent and user state also shape the retargeting ratio; conditioning the gate on those features is a natural extension.
- Because the offline and online comparisons both target Fliggy travel products, an untested extension is whether the same IIF normalization transfers to general merchandise, where repeat purchase and price sensitivity differ.
- A direct test of the "when to personalize" claim would be an A/B experiment holding the attention normalization fixed and toggling only the IIF gate; the paper's ablations do this offline but not online.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NAM, a normalization attention model for personalized product search at Fliggy. The model introduces two components: (1) a global normalized multi-head self-attention (GN_MHSA) and target attention (GN_TA) that incorporate inverse item frequency (IIF) into attention weights, and (2) an IIF-based personalization gate that scales the personalized representation by a function of the target item's IIF. The authors motivate the design with three data-driven intuitions, including a claim that popular items are overestimated and long-tail items underestimated, that globally normalized similarity improves hit rate, and that conversion of long-tail items is more correlated with user behavior. They report offline AUC/GAUC improvements over several baselines on a large industrial dataset, ablation results, a PCOC analysis, and an online A/B test with a 0.8% CTCVR lift.
Significance. If the result holds, the paper makes a practical contribution by showing that simple, parameter-efficient popularity normalization and gating can improve conversion estimation in industrial product search. The strengths are that the model is evaluated both offline and online, the two proposed components are isolated in ablations, and the intuitions are grounded in platform data. However, the significance is tempered by the lack of statistical significance tests, the tiny offline gains, and a conceptual gap between the motivating analysis and the actual gating mechanism. The SiLU range claim is also mathematically incorrect. These issues limit the strength of the central claim as currently written, but the core idea is plausible and worth further validation.
major comments (3)
- [Section 3.1 (Eq. 4)] The text states: 'we use SiLU instead of Sigmoid or ReLU here to ensure that the values of M_Q,K are within the range of 0 to 1 since the values of IIF are always positive.' This is mathematically incorrect: SiLU(x) = x * sigmoid(x), which grows unboundedly as x increases, so the values of M_Q,K are not guaranteed to lie in [0,1]. If the [0,1] range is actually needed for the mask, the authors should use a bounded activation such as sigmoid; otherwise, the justification for choosing SiLU should be revised.
- [Section 3.3 (Eq. 14) and Table 3] Intuition 3 is supported by Table 3, which compares conversion rates between IsRetarget=1 and IsRetarget=0, i.e., whether the user directly interacted with the target item before. However, the gating mechanism in Eq. (14) depends only on the target item's IIF and does not condition on whether the target appears in, or is similar to, the user's historical behavior. Therefore, the gate is a popularity-indexed proxy for item familiarity, not an implementation of the stated similarity-based intuition. The offline ablations do not separate these two cases, so the empirical support for the gate's intended interpretation is incomplete. Please clarify the role of the gate or provide additional evidence that IIF alone captures the heterogeneity described in Intuition 3.
- [Section 4.2 (Table 5) and Section 4.4] The paper repeatedly uses the word 'significantly' (e.g., 'significantly outperforms' in the abstract and in Section 4.2), but no significance test, confidence interval, or variance estimate is reported. In Table 5, the improvement over the second-best baseline is only 0.001 in CTCVR AUC and 0.002 in CTCVR GAUC, which may be within run-to-run variance for industrial-scale data. Similarly, the online A/B test in Section 4.4 reports a 0.8% CTCVR lift with no statistical significance testing, p-values, confidence intervals, or description of the experiment design (e.g., number of users, duration, randomization unit). Please include such statistical evidence, or temper the claims accordingly.
minor comments (3)
- [Table 3] The table is garbled in the submitted text; for example, the row for level (0,1%] reads '10.501 22.45% 5.20%' and the column alignment is inconsistent. Please reformat the table so that IsRetarget, ExposureRate, CTCVR, and Retarget/NonRetarget are clearly separated.
- [Section 3.1, Eq. (4)] The notation uses the elementwise product symbol ⊙ both for the scalar-matrix product and for the matrix multiplication in Eq. (5); consider making the broadcasting explicit or using a different symbol for scalar scaling.
- [Section 2, Table 2] The definitions of Co-occurrence, Jaccard, and Cosine similarity are missing the union/intersection operators in the rendered text (they appear as blank spaces). These equations should be typeset correctly.
Circularity Check
No significant circularity: the IIF-based components are motivated by empirical observations and validated on held-out data, not derived from the target metric.
full rationale
The paper's derivation chain is empirical rather than definitional. The three intuitions in Section 2 are supported by platform-log analyses (Tables 1-3), and these observations motivate the IIF-based normalization and gating components (Sections 3.1-3.3). The components contain learnable parameters (W_MQ,K, W_Mit,K', W_IIFit) that are trained against held-out CTCVR labels, and the reported offline AUC/GAUC improvements and the online A/B test are computed on unseen data. No fitted quantity is renamed as a prediction: IIF is a precomputed item statistic (1/#U_i), not an optimized parameter, and the gate in Eq. 14 is learned from labels rather than set to reproduce the Table 3 ratios. The baselines are external published models (AEM, ZAM, TEM, DAM, QEM, HEM) and the ablations isolate component contributions; there is no load-bearing self-citation chain or imported uniqueness theorem. The Table 3 evidence contrasts retargeted versus non-retargeted items while the gate conditions on target-item popularity alone, which is a construct-validity concern about the intuition-to-implementation link, but it does not make any equation or fitted parameter identical to the target output. The PCOC analysis reports the model's calibrated error relative to actual CTCVR, not a prediction derived from the metric itself. Overall, the central claim (IIF-aware attention and gating improve conversion estimation) is independently evaluated and does not reduce to its inputs by construction.
Assumptions & free parameters
free parameters (4)
- W_M_QK =
learned (not reported)
- W_M_it_Kp =
learned (not reported)
- W_IIF_it =
learned (not reported)
- Architecture hyperparameters =
H=4, d_ei=98, towers 512/256/128, lr=0.001, batch 1024
assumptions (4)
- standard math Scaled dot-product attention with softmax yields useful query/item representations.
- domain assumption Item popularity can be measured by the number of distinct users who interacted with the item (#U_i), and IIF = 1/#U_i is a stable signal.
- domain assumption The conversion-rate heterogeneity described by Intuition 3 is a stable property of user behavior across queries and time.
- ad hoc to paper SiLU applied to positive IIF inputs keeps the mask M_Q,K within [0,1].
Cite this review
Pith. "Pith review of NAM: A Normalization Attention Model for Personalized Product Search In Fliggy." pith.science (2026). https://pith.science/paper/UTJNJJB7
@misc{pith2026250608382,
author = {Pith},
title = {Pith review of: NAM: A Normalization Attention Model for Personalized Product Search In Fliggy},
year = {2026},
howpublished = {\url{https://pith.science/paper/UTJNJJB7}},
note = {Machine review of arXiv:2506.08382}
}
read the original abstract
Personalized product search provides significant benefits to e-commerce platforms by extracting more accurate user preferences from historical behaviors. Previous studies largely focused on the user factors when personalizing the search query, while ignoring the item perspective, which leads to the following two challenges that we summarize in this paper: First, previous approaches relying only on co-occurrence frequency tend to overestimate the conversion rates for popular items and underestimate those for long-tail items, resulting in inaccurate item similarities; Second, user purchasing propensity is highly heterogeneous according to the popularity of the target item: it is less correlated with the user's historical behavior for a popular item and more correlated for a long-tail item. To address these challenges, in this paper we propose NAM, a Normalization Attention Model, which optimizes ''when to personalize'' by utilizing Inverse Item Frequency (IIF) and employing a gating mechanism, as well as optimizes ''how to personalize'' by normalizing the attention mechanism from a global perspective. Through comprehensive experiments, we demonstrate that our proposed NAM model significantly outperforms state-of-the-art baseline models. Furthermore, we conducted an online A/B test at Fliggy, and obtained a significant improvement of 0.8% over the latest production system in conversion rate.
Figures
Reference graph
Works this paper leans on
-
[1]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher
-
[2]
Qingyao Ai, Daniel N Hill, SVN Vishwanathan, and W Bruce Croft. 2019. A zero attention model for personalized product search. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management . 379–388
work page 2019
-
[3]
Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W Bruce Croft. 2017. Learning a hierarchical embedding model for personalized product search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval . 645–654
work page 2017
-
[4]
Elias Bassani, Pranav Kasela, and Gabriella Pasi. 2024. Denoising Attention for Query-aware User Modeling. In Findings of the Association for Computational Linguistics: NAACL 2024. 2368–2380
work page 2024
-
[5]
Paul N Bennett, Ryen W White, Wei Chu, Susan T Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012. Modeling the impact of short-and long-term behavior on search personalization. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval . 185–194
work page 2012
-
[6]
Keping Bi, Qingyao Ai, and W Bruce Croft. 2020. A transformer-based embedding model for personalized product search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval . 1521–1524
work page 2020
-
[7]
JohnS Breese DavidHeckerman CarlKadie. 1998. Empirical analysis of predictive algorithms for collaborative filtering. Microsoft Research Microsoft Corporation One Microsoft Way Redmond, W A98052 (1998)
work page 1998
-
[8]
Òscar Celma and Pedro Cano. 2008. From hits to niches? or how popular artists can bias music recommendation and discovery. In Proceedings of the 2nd KDD workshop on large-scale recommender systems and the netflix prize competition . 1–8
work page 2008
Show all 26 references
-
[9]
Susan T Dumais. 2016. Personalized Search: Potential and Pitfalls.. In CIKM. 689
2016
-
[10]
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107 (2018), 3–11
2018
-
[11]
Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yinglong Wang, Jun Ma, and Mohan Kankanhalli. 2019. Attentive long short-term preference modeling for person- alized product search. ACM Transactions on Information Systems (TOIS) 37, 2 (2019), 1–27
2019
-
[12]
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of- experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining . 1930–1939
2018
-
[13]
Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrie...
2018
-
[14]
Joel Myerson, Leonard Green, and Missaka Warusawitharana. 2001. Area under the curve as a measure of discounting. Journal of the experimental analysis of behavior 76, 2 (2001), 235–243
2001
-
[15]
Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve re- stricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) . 807–814
2010
-
[16]
Matjaž Perc. 2014. The Matthew effect in empirical data. Journal of The Royal Society Interface 11, 98 (2014), 20140378
2014
-
[17]
Parikshit Sondhi, Mohit Sharma, Pranam Kolari, and ChengXiang Zhai. 2018. A taxonomy of queries for e-commerce search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval . 1245–1248
2018
-
[18]
Ning Su, Jiyin He, Yiqun Liu, Min Zhang, and Shaoping Ma. 2018. User intent, behaviour, and perceived satisfaction in product search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining . 547–555
2018
-
[19]
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In Proceedings of the 14th ACM Conference on Recommender Systems. 269–278
2020
-
[20]
Jaime Teevan, Susan T Dumais, and Daniel J Liebling. 2008. To personalize or not to personalize: modeling queries with variation in user intent. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 163–170
2008
-
[21]
Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. 2020. Entire space multi-task modeling via post-click behavior decomposi- tion for conversion rate prediction. In Proceedings of the 43rd International ACM SIGIR conference on research and develo...
2020
-
[22]
Xiangyu Zhao, Zhendong Niu, and Wei Chen. 2013. Opinion-based collaborative filtering to solve popularity bias in recommender systems. InDatabase and Expert Systems Applications: 24th International Conference, DEXA 2013, Prague, Czech Republic, August 26-29, 2013. Proceedings,...
2013
-
[23]
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 5941–5948
2019
-
[24]
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...
2018
-
[25]
Han Zhu, Junqi Jin, Chang Tan, Fei Pan, Yifan Zeng, Han Li, and Kun Gai. 2017. Optimized cost per click in taobao display advertising. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining . 2191–2200
2017
-
[2019]
arXiv preprint arXiv:1907.13286 (2019)
The unfairness of popularity bias in recommendation. arXiv preprint arXiv:1907.13286 (2019)
2019 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.