Efficient user history modeling with amortized inference for deep learning recommendation models

Aman Gupta; Fedor Borisyuk; Lars Hertel; Neil Daftary; Rahul Mazumder

REVIEW 1 major objections 4 minor 2 cited by

Efficient user history modeling with amortized inference for deep learning recommendation models

T0 review · 1 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Appending candidates with cross-attention makes amortized history inference exact, cutting serving latency by 30% in production.

desk verdict A clean, honest systems paper: the append-with-cross-attention recipe plus amortized inference is genuinely useful, and the load-bearing equivalence claim is plausible but under-specified on positional encodings. read the letter →

arxiv 2412.06924 v1 pith:KWCOJYRR submitted 2024-12-09 cs.LG cs.IR

Lars Hertel , Neil Daftary , Fedor Borisyuk , Aman Gupta , Rahul Mazumder This is my paper · ORCID

classification cs.LGcs.IR

keywords recommendersystemsuserhistorymodelingearlyfusioncross-attentionamortizedinferenceTransformerencoderDLRMlatencyreduction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that for DLRM-style recommender models, early fusion of the candidate item can be done by appending it to the end of the user-history sequence with cross-attention, and that this performs on par with the more expensive concatenation fusion. This choice is what unlocks amortized inference: instead of running the Transformer once per candidate, all m candidates are appended and processed in one forward pass, producing candidate outputs equivalent to regular inference. The paper proves the equivalence, derives the complexity reduction, and reports that on the deployed Feed and Ads ranking surfaces the amortized version cuts latency by about 30% while keeping or improving engagement. The significance is that Transformer-based user-history encoders, which are notoriously costly to serve, can be made much cheaper without changing the model's learned weights or quality.

What carries the argument

The load-bearing mechanism is the cross-attention formulation of appending: queries are computed from the history-plus-candidate sequence, while keys and values are computed from the history items only, so the candidate attends to the history but history items cannot attend to the candidate. This makes each candidate's output a function of the history and its own embedding alone, which is precisely what allows the m candidates to be packed into one sequence of length n+m and processed in a single Transformer forward pass; the reshaping of the tensor between m x (n+1) x d and 1 x (n+m) x d makes the encoder output compatible with the rest of the DLRM-style MLP. The paper also matches parameter counts between fusion methods by tuning key and feedforward dimensions, and uses attention-matrix inspection on a Feed example to contrast the near-constant pairwise pattern learned by concatenation with the diagonal, sequence-spreading pattern learned by appending.

What would settle it

Run the same candidate through the same trained Transformer both as the sole appended item at position n+1 and as the second of two appended items at position n+2; if the two output vectors differ when any absolute-position encoding is present, the claimed equivalence is false.

Watch

Extended reading notes

Core claim

The central claim is that under the cross-attention definition of Equation (3), the m candidate outputs obtained from a single amortized forward pass are equivalent to the outputs of m regular inferences, because each candidate attends only to the fixed user history and to itself, never to other candidates. Given that equivalence, the paper's other main result is empirical: appending with cross-attention matches concatenation in accuracy, winning on two of four public datasets, tying within significance thresholds on the Feed and Ads offline metrics, and exhibiting qualitatively different attention patterns. The paper then verifies the complexity analysis with CPU/GPU benchmarks and reports a production deployment in which amortized inference reduces p90/p99 latency and CPU usage by 30% or more compared with non-amortized appending, with Feed engagement rising to +0.18%. The production result is described as unexpected, since amortization is only an inference optimization, and the paper attributes the extra engagement to the latency reduction itself.

Load-bearing premise

The equivalence between amortized and regular inference rests on the Transformer output for a candidate being independent of the candidate's absolute position in the input sequence, since regular inference places the candidate at position n+1 while amortized inference places it at positions n+1 through n+m.

Editorial extensions

If this is right

If the equivalence holds, the Transformer encoder for user history is computed once per request rather than once per candidate, so the history-encoding portion of inference cost falls by roughly a factor of m.
Because the empirical comparison found appending on par with concatenation, practitioners can adopt the cheaper fusion without sacrificing the reported engagement or AUC gains.
The complexity analysis implies the relative savings grow with history length n, making amortized inference increasingly attractive for lifelong or long-sequence user behavior models.
Amortized inference is orthogonal to other attention-efficiency tricks such as sparse attention, progressive sampling, and multi-query attention, so those can be stacked on top for further speed-ups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The equivalence claim silently assumes the encoder is invariant to a candidate's absolute position in the input sequence; since the paper does not say whether positional encodings are used, a reader should verify that the deployed Transformer uses relative or no positional information before relying on the equivalence.
The complexity comparison suggests that amortization is most beneficial when the candidate count m is large relative to the history length n; for very short histories and small batches, overhead may dominate.
The attention-pattern difference indicates that concatenation can degrade into near-DIN pairwise scoring, so in domains where sequential dependency truly matters, appending may change model behavior even if aggregate metrics match.
A natural extension is to combine amortized inference with two-stage lifelong behavior models that first reduce the history with a general search unit, since the same position-invariance condition would apply to the reduced sequence.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a read. The paper does one concrete thing well: it shows that you can append the candidate to the user history with cross-attention, get parity with the more expensive concatenation approach, and then exploit that format to serve m candidates in one Transformer pass. That combination — append + cross-attention + amortized inference for pointwise DLRM rankers — is not in the cited prior work, and the production numbers (30% latency reduction, small CPU savings) back up the complexity argument. Credit where due: the FLOP analysis is simple and correct, the benchmarking in Figure 3 matches it, and the deployment results are real evidence, not a toy simulation.

The soft spots are real but not fatal. The exact-equivalence claim in Section 2.2 holds only if the encoder is invariant to the candidate's absolute position. Regular inference puts the candidate at position n+1; amortized inference puts it at n+j. If positional embeddings or position-dependent attention biases are used, the outputs differ. The paper never says whether positional encodings are used or disabled, so the central efficiency claim rests on an unstated architectural condition. That is a documentation gap, not a demonstrated flaw — a position-free cross-attention encoder would satisfy it, and the production results suggest it works in their setting. But it needs to be stated.

The parity claim between append and concat is weaker evidence: no error bars, and the models are matched on parameter count by using different embedding dimensions per method, which confounds the comparison. The conclusion rests on two of four public datasets going each way plus internal results within significance thresholds. That is roughly fine for a systems paper but not a strong statistical statement.

One minor oddity: the engagement lift for amortized inference over regular append in production is attributed to latency, which is plausible and consistent with the cited online-latency literature, but it is a post-hoc explanation.

Who gets value: practitioners serving Transformer-based history encoders in ranking systems, and researchers working on early fusion or amortized inference. The paper deserves a serious referee — the contribution is clear, the deployment evidence is real, and the missing position-encoding disclosure is fixable at review time.

Referee Report

1 major / 4 minor

Summary. The paper studies early fusion of the candidate item in Transformer-based user history encoders for deep learning recommendation models (DLRMs). It compares concatenating the candidate to each history item against appending it as a separate token with cross-attention, and reformulates the M-FALCON amortized inference scheme for DLRMs by processing all candidate items in a single forward pass. The authors report that appending with cross-attention performs on par with concatenating across four public datasets and two internal LinkedIn surfaces, and that amortized inference reduces production latency by about 30% on LinkedIn Feed and Ads. The paper also includes attention-matrix visualizations, FLOP-based complexity arguments, and CPU/GPU benchmarks.

Significance. If the core equivalence and the empirical parity results hold, this is a practical contribution: it offers a low-overhead way to reduce the serving cost of Transformer-based user history encoders without sacrificing ranking quality. The paper provides public-dataset experiments, production A/B tests, and a deployment case study, which are valuable for both academic and industrial audiences. The central theoretical claim, however, rests on an unstated assumption about positional encodings, and the complexity analysis contains a technical error; these need to be addressed before the efficiency claims can be fully accepted.

major comments (1)

[Section 2.2, Eq. (3)] The equivalence claim between amortized and regular inference is asserted unconditionally, but it depends on the candidate output being invariant to the candidate's absolute position in the input sequence. In regular inference the candidate is at position n+1, while in amortized inference the m candidates occupy positions n+1 through n+m; if the Transformer uses learned or sinusoidal positional encodings, or any position-dependent attention bias, the query vectors for the same candidate in the two regimes differ and the outputs are no longer equivalent. The manuscript nowhere states whether positional encodings are used. Because the reported 30% latency reduction in Table 3 and the Feed engagement results in Section 3.4 are presented as consequences of this equivalence, the authors must either document that the model uses no positional encodings and justify that choice, or modify the amortized inference scheme to replicate the regular-inference positions. This is a load-bearing assumption that cannot remain implicit.

minor comments (4)

[Section 3.3] The theoretical complexity of amortized inference is stated as O(l(n+m)^2 d) for the attention term, which overestimates the cost because cross-attention restricts keys and values to the n history tokens; the correct attention complexity is O(l(n+m) n d). The qualitative conclusion that amortization helps as n grows still holds after this correction, but the formula and the text should be updated. Additionally, the ratio of regular to amortized cost increases with n but saturates rather than growing linearly, so the phrase 'increases linearly as n grows' is inaccurate.
[Section 3.1] The claim that appending 'performs similar to' concatenating is based on point estimates without reported standard errors or significance tests; reporting confidence intervals or paired tests across random seeds would strengthen this central empirical claim.
[Section 2.1, Table 1] The parameter-matching procedure between Append and Concat is not fully specified; while the table lists different embedding and feedforward/key dimensions, reporting actual parameter counts or the search procedure would help readers assess whether the comparison is fair.
[Section 5] There is a typo in the final section: 'consistenly' should be 'consistently.' Also, the abstract contains a grammatical comma splice ('Using the latter method, allows us to...').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the amortized-inference equivalence is a direct mathematical consequence of the paper's cross-attention equations, and all quality/latency claims are measured outcomes.

full rationale

The central derivation in Section 2.2 is not circular. The claim that under cross-attention (Equations 3-5) the candidate outputs in amortized inference equal those from regular inference is a formal identity: K and V are computed only from the history tokens, so each candidate's attention output depends only on its own query and the shared history representation, not on other candidates or on the candidate's absolute position, provided the encoder is position-invariant. This is a construction-level equality, not a fitted parameter renamed as a prediction. The append-vs-concat performance comparison (Table 2) is an empirical measurement on public and internal datasets, and the latency reductions (Table 3) are deployment measurements. M-FALCON [13] is explicitly cited as external prior work, and no load-bearing argument rests on the authors' own unpublished claims. The only notable caveat is that the equivalence silently assumes the absence of absolute or relative positional embeddings, since these would make the candidate query position-dependent; this is a correctness/robustness risk, not circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The amortization equivalence is structural, not fitted; no parameter is tuned to force the result. The key axioms are the history-only key/value construction and the absence of absolute position dependence. The paper's offline parity claim does depend on hand-chosen hyperparameters in Table 1, but those are experimental design choices, not fit parameters in the derivation.

assumptions (2)

ad hoc to paper The sequence encoder uses no absolute position encodings (or position-equivalent features) in the attention computation.
Section 2.2's equivalence of amortized and regular inference requires the candidate at position n+1 in regular inference to have the same representation as candidates at positions n+1...n+m in amortized inference; the paper does not state whether positional encodings are used.
standard math Multi-head attention with K and V restricted to history tokens is applied exactly as written in Equations (3)-(5), with no candidate-to-candidate interaction via masked attention or other mechanisms.
This follows directly from the definitions, but it is a stated construction rather than a derived fact.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient user history modeling with amortized inference for deep learning recommendation models." pith.science (2026). https://pith.science/paper/KWCOJYRR

@misc{pith2026241206924,
  author       = {Pith},
  title        = {Pith review of: Efficient user history modeling with amortized inference for deep learning recommendation models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWCOJYRR}},
  note         = {Machine review of arXiv:2412.06924}
}

read the original abstract

We study user history modeling via Transformer encoders in deep learning recommendation models (DLRM). Such architectures can significantly improve recommendation quality, but usually incur high latency cost necessitating infrastructure upgrades or very small Transformer models. An important part of user history modeling is early fusion of the candidate item and various methods have been studied. We revisit early fusion and compare concatenation of the candidate to each history item against appending it to the end of the list as a separate item. Using the latter method, allows us to reformulate the recently proposed amortized history inference algorithm M-FALCON \cite{zhai2024actions} for the case of DLRM models. We show via experimental results that appending with cross-attention performs on par with concatenation and that amortization significantly reduces inference costs. We conclude with results from deploying this model on the LinkedIn Feed and Ads surfaces, where amortization reduces latency by 30\% compared to non-amortized inference.

Figures

Figures reproduced from arXiv: 2412.06924 by the authors.

**Figure 2.** Attention activations from the first Transformer [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Inference time for 100 forward passes using regular [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation
cs.IR 2025-06 conditional novelty 5.0 of 10

TransAct V2 adds lifelong user sequences and a next-action loss to Pinterest's CTR model, reporting online gains of +6.35% repin volume and -12.80% hide volume.
An Industrial-Scale Sequential Recommender for LinkedIn Feed Ranking
cs.IR 2026-02 conditional novelty 4.0 of 10

A transformer-based sequential recommender, Feed SR, improved LinkedIn Feed time spent by 2.10% in an online A/B test and now serves the majority of Feed traffic.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages · cited by 2 Pith papers

[1]

Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. 2022. Sampling is all you need on modeling long-term user behaviors for CTR prediction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management . 2974–2983

2022
[2]

Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. 2023. TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3785–3794

work page 2023
[3]

Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. InProceedings of the 1st international workshop on deep learning practice for high-dimensional sparse data. 1–4

2019
[4]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al
[5]

Pinterest Engineering. [n. d.]. GPU-accelerated MLInference at Pinterest — Pinter- est Engineering. https://medium.com/@Pinterest_Engineering/gpu-accelerated- ml-inference-at-pinterest-ad1b6a03a16d. [Accessed 09-05-2024]

work page 2024
[6]

Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin Liu, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng. 2023. PS-SA: An Efficient Self-Attention via Progressive Sampling for User Behavior Sequence Modeling. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 4639–4645

work page 2023
[7]

Barrie Kersbergen and Sebastian Schelter. 2021. Learnings from a Retail Rec- ommendation System on Billions of Interactions at bol. com. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) . IEEE, 2447–2452

work page 2021
[8]

Hitesh Khandelwal, Viet Ha-Thuc, Avishek Dutta, Yining Lu, Nan Du, Zhihao Li, and Qi Hu. 2021. Jointly Optimize Capacity, Latency and Engagement in Large-scale Recommendation Systems. In Proceedings of the 15th ACM Conference on Recommender Systems. 559–561

work page 2021

Show all 16 references

[9]

Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In Proceedings of the 29th ACM International Conference on Informat...

2020
[10]

Aixin Sun. 2023. Take a Fresh Look at Recommender Systems from an Evaluation Standpoint. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2629–2638

2023
[11]

Hao Wang, Jianxun Lian, Mingqi Wu, Haoxuan Li, Jiajun Fan, Wanyue Xu, Chaozhuo Li, and Xing Xie. 2023. Convformer: Revisiting transformer for se- quential user modeling. arXiv preprint arXiv:2308.02925 (2023)

2023 arXiv
[12]

Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and An- drew Zhai. 2023. Transact: Transformer-based realtime user action model for recommendation at pinterest. In Proceedings of the...

2023
[13]

Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhao- jie Gong, Fangda Gu, Michael He, et al. 2024. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. arXiv preprint arXiv:2402.17152 (2024)

2024 arXiv
[14]

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...

2018
[15]

Tianyu Zhu, Yansong Shi, Yuan Zhang, Yihong Wu, Fengran Mo, and Jian-Yun Nie. 2024. Collaboration and Transition: Distilling Item Transitions into Multi- Query Self-Attention for Sequential Recommendation. In Proceedings of the 17th ACM International Conference on Web Search a...

2024
[2016]

In Proceedings of the 1st workshop on deep learning for recommender systems

Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems . 7–10

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. 2022. Sampling is all you need on modeling long-term user behaviors for CTR prediction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management . 2974–2983

2022

[2] [2]

Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. 2023. TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3785–3794

work page 2023

[3] [3]

Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. InProceedings of the 1st international workshop on deep learning practice for high-dimensional sparse data. 1–4

2019

[4] [4]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al

[5] [5]

Pinterest Engineering. [n. d.]. GPU-accelerated MLInference at Pinterest — Pinter- est Engineering. https://medium.com/@Pinterest_Engineering/gpu-accelerated- ml-inference-at-pinterest-ad1b6a03a16d. [Accessed 09-05-2024]

work page 2024

[6] [6]

Jiacen Hu, Zhangming Chan, Yu Zhang, Shuguang Han, Siyuan Lou, Baolin Liu, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng. 2023. PS-SA: An Efficient Self-Attention via Progressive Sampling for User Behavior Sequence Modeling. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 4639–4645

work page 2023

[7] [7]

Barrie Kersbergen and Sebastian Schelter. 2021. Learnings from a Retail Rec- ommendation System on Billions of Interactions at bol. com. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) . IEEE, 2447–2452

work page 2021

[8] [8]

Hitesh Khandelwal, Viet Ha-Thuc, Avishek Dutta, Yining Lu, Nan Du, Zhihao Li, and Qi Hu. 2021. Jointly Optimize Capacity, Latency and Engagement in Large-scale Recommendation Systems. In Proceedings of the 15th ACM Conference on Recommender Systems. 559–561

work page 2021