Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Hi-SAM claims that disentangling shared cross-modal semantics from modality-specific detail in semantic IDs, plus a hierarchy-aware transformer with anchor memory, consistently beats prior recommenders and lifts online response rate by 6.55

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:59 UTC pith:RMVLFQI4

load-bearing objection Credible architecture, plausible mechanism, but the empirical case is under-specified: baseline feature pipelines and the H parameter conflict need pinning down before the headline gains can be evaluated. the 4 major comments →

arxiv 2602.11799 v2 pith:RMVLFQI4 submitted 2026-02-12 cs.AI cs.IR

Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation

Pingjun Pan , Tingting Zhou , Peiyao Lu , Tingting Fei , Hongxiang Chen , Chuanjiang Luo This is my paper
classification cs.AI cs.IR
keywords multi-modal recommendationsemantic IDsdisentangled tokenizationhierarchical transformercold-start recommendationcross-modal alignmentmemory-anchor attentionlarge-scale recommendation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Prior semantic-ID recommenders fail, the paper argues, in two places: tokenizers tangle shared cross-modal meaning with modality-specific detail, and transformers flatten the natural hierarchy of users, items, and tokens. Hi-SAM targets both: a Disentangled Semantic Tokenizer aligns image, text, and behavior embeddings on a hypersphere, quantizes shared consensus and per-modality nuance in separate codebooks, and enforces their separation with a mutual-information penalty; a Hierarchical Memory-Anchor Transformer splits positional encoding into inter-item and intra-item subspaces and lets history be accessed only through compressed anchor tokens. On two public datasets and one industrial dataset, Hi-SAM reports consistent gains over all baselines, with the largest improvements in cold-start scenarios. The authors also report a 6.55% gain in online Response Rate and a 35% latency reduction in production, which is why the design matters: it turns multimodal item content into discrete IDs that serve sparse interaction data better and remain cheap enough to deploy.

Core claim

Central claim: disentangled semantic IDs plus a hierarchy-aware decoder let multimodal content beat both sparse-ID and prior semantic-ID recommenders overall, not just in cold start. The DST aligns image, text, and behavior embeddings on a hypersphere, quantizes shared consensus with one codebook stack, recovers per-modality detail from residuals, and minimizes mutual information between the two. The HMAT encodes item order and within-item position in orthogonal RoPE subspaces and forces historical attention through compressed anchor tokens. Reported evidence: higher AUC and GAUC on all datasets, several-point cold-start gains, a 6.55% online Response Rate lift, and 35% lower latency.

What carries the argument

Two coupled mechanisms carry the argument. (1) Disentangled Semantic Tokenizer (DST): cross-modal geometric alignment projects each modality onto a unit hypersphere and shrinks the volume of the parallelotope they span; Disentangled Modal-Residual Quantization then aggregates aligned features, quantizes residuals with shared codebooks for consensus, unfolds the leftover residual into subspaces, and uses each modality's aligned features as queries to recover modality-specific codes, with an upper-bound mutual-information term enforcing that specific codes carry no shared information. (2) Hierarchical Memory-Anchor Transformer (HMAT): Hierarchical RoPE gives each token a coordinate (item index

Load-bearing premise

The load-bearing premise is Section 4.1's assertion that baselines were 'strictly aligned' in feature configuration and model complexity, plus the claim that the behavior encoder, trained on time-isolated samples, is leakage-free; if either fails, the reported GAUC gaps may reflect encoder capacity or leakage rather than the architecture.

What would settle it

Re-run the main comparison with every baseline given the same input features and encoder stack as Hi-SAM (same vision-language and behavior embeddings, same compute budget) and with the behavior encoder trained without the time-isolation restriction; if the GAUC gaps shrink to noise, the reported gains are mostly encoder capacity or leakage. A second check: remove the mutual-information term from DST and see whether the layer-wise code-correlation diagonal pattern in Figure 3 disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, semantic-ID recommenders can surpass sparse-ID models on overall ranking, closing the gap that earlier semantic-ID methods left open.
  • Cold-start users are the main beneficiaries: the online A/B test reports a +16.62% Response Rate lift for cold-start users, alongside offline GAUC and AUC gains of several points.
  • Hierarchical position encoding plus anchor memory makes long multimodal sequences computationally tractable, with about 35% lower response time at matched compute.
  • Pre-training the decoder on next-token prediction before supervised fine-tuning adds consistent gains, suggesting that learning semantics and modeling preferences should be decoupled.
  • The framework scales smoothly with depth and sequence length, with GAUC rising as compute increases.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference, not the paper's claim: the mutual-information penalty is directly testable—if DST truly disentangles, ablating one modality's specific code should degrade prediction for that modality's attributes while leaving others intact.
  • My inference: part of the gain may come from noise filtering rather than semantic richness; an ablation replacing semantic IDs with random but hierarchy-preserving tokens would isolate the two effects.
  • My inference: because the online baseline is a DLRM, the 6.55% lift likely bundles encoder-capacity gains with architecture gains; an online A/B against the strongest semantic-ID baseline would separate them.
  • My inference: a natural untested extension is reusing Hi-SAM's item codes across domains or platforms, since content-level IDs should transfer better than sparse IDs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript proposes Hi-SAM, a two-stage framework for multimodal semantic-ID-based recommendation. Stage 1 (DST) aligns image/text/behavior embeddings on a hypersphere (CGA, via GRAM-style volume minimization) and quantizes them into shared and modality-specific discrete tokens (DMRQ) under a mutual-information penalty. Stage 2 (HMAT) introduces Hierarchical RoPE with split inter-item/intra-item subspaces and Memory-Anchor Attention, along with anchor-based KV-cache eviction. Offline experiments on Amazon Books, Movies & TV, and an industrial dating dataset report consistent state-of-the-art gains, with especially large cold-start improvements; online A/B reports a +6.55% Response Rate gain and a 35% latency reduction. The appendix contains derivations of the vCLUB MI bound, the quantization losses, and the H-RoPE attention expansion.

Significance. The design is well-motivated and timely: semantic-ID recommenders do suffer from flat tokenization and cross-modal redundancy, and the proposed disentangled tokenizer plus hierarchy-aware transformer is a plausible and practically relevant direction. The anchor-based attention with lossless KV-cache eviction is also a useful industrial contribution. The paper provides clear derivations and fairly detailed hyperparameters, which is a strength. However, the central empirical attribution is currently under-supported: the claimed gains may reflect the capacity of the large pretrained encoders or the behavioral encoder rather than the proposed architecture, the leakage barrier for the behavioral encoder is not specified, and the offline results lack any uncertainty quantification. If these issues are resolved, the work would make a solid contribution to large-scale multimodal recommendation.

major comments (4)
  1. [§4.1, §4.2, Appendix B.2] The claim that all baselines are 'strictly aligned' in feature configuration is not substantiated. Only Hi-SAM's input pipeline is specified: BLIP-2 (2.7B) for image/text plus a SASRec-based behavioral encoder (Appendix B.2); the baselines use official implementations or 'strictly follow original papers.' The reported gains — e.g., Industrial GAUC from 0.6131 (PSRQ+MCCA) to 0.6410 (Hi-SAM-Small) in Table 2 — may therefore reflect dense encoder capacity rather than DST/HMAT. The same confound affects Table 3, where DST is compared against QARM/PSRQ under a common HSTU backbone: if the tokenizers receive different content features, the ablation does not isolate the tokenizer. Additionally, MTGR receives extra cross features, contradicting the 'same feature set' statement. Please provide per-baseline encoder/preprocessing specifications and add an experiment where every tokenizer is applied
  2. [§4.1 Implementation Details vs. Appendix B.2] The number of PSGR subspaces is inconsistent: Section 4.1 states H=8, while Appendix B.2 states H=4. PSGR is a load-bearing component of DMRQ (Section 3.2.2, Eq. (2)), so the discrepancy is not cosmetic. Please state the actual value used for Tables 2–4 and correct the affected text, or re-run the results with the correct configuration.
  3. [§4.2, Table 2] No measure of uncertainty is reported for any offline metric. The data are from a single 90/10 chronological split; there are no seeds, standard deviations, or significance tests. Several headline differences on the public datasets are small (e.g., Book GAUC 0.6501 vs. 0.6588; Book Cold AUC 0.6877 vs. 0.6924), so the claim of 'consistently outperforms all baselines across all metrics' is not yet established. Please report repeated-seed means with error bars and paired significance tests, or at least bootstrap confidence intervals. Online A/B results in Table 5 should likewise report confidence intervals and should account for testing multiple variants.
  4. [Appendix B.2] The behavioral encoder's leakage barrier is asserted but not specified. The text says the SASRec encoder is 'trained on samples strictly isolated by time from the downstream ranking data,' but it does not state the isolation interval, how user/item identity is handled, or how behavioral embeddings are computed at test time. Since Figure 2 shows the behavioral modality alone is the strongest input and cold-start gains are a core claim, any temporal overlap would inflate Cold GAUC (e.g., 0.5835 in Table 2). Please give the exact protocol, including the exclusion-window length, and provide a leakage check (e.g., behavior-embedding accuracy under randomly permuted time).
minor comments (6)
  1. [§3.2.1, Eq. (1)] K is undefined. Specify how many negatives are used and whether they are in-batch negatives or sampled from a separate memory bank.
  2. [§3.3.2, Eq. (10)] The condition 'k=[ANC]' compares a key index with a token. Rephrase as 'the key token is an Anchor Token' to avoid confusion.
  3. [§4.2, Table 2] Clarify whether the 'w/PT+SFT' row is Hi-SAM-Large with semantic pretraining; the row label is ambiguous as printed.
  4. [§3.4.2] In 'One-Pass Parallel Ranking,' specify how the candidate coordinates are reset — for all candidate tokens or only item tokens — and how this interacts with the H-RoPE positions of the action tokens.
  5. [§4.5, Figure 4(c)] Provide the exact (depth, sequence length, GFLOPs) values for the plotted points. As printed, the scaling-law claim cannot be reproduced from the figure.
  6. [§4.6, Table 5] Report the A/B test duration, number of users per bucket, and statistical significance. Also state whether the DLRM baseline was concurrently updated during the two-month experiment.

Circularity Check

0 steps flagged

No significant circularity; Hi-SAM's derivation is self-contained against external methods and external benchmarks.

full rationale

The paper's derivation chain does not reduce to its own inputs. The DST tokenizer is built from external, independently published components (GRAM [6], RQ-VAE [28], vCLUB [5]); its objectives (Eq. 1, Eq. 2) are geometric alignment, residual reconstruction, and mutual-information minimization, none of which contains the recommendation metric being predicted. The HMAT stage is trained with standard next-token/NLL objectives (Eq. 11 and the pre-training loss) over the discrete token stream; no equation defines an output in terms of the target GAUC, Cold GAUC, or Response Rate. The only author self-citations are PSRQ+MCCA [37] and Climber [40]. PSRQ+MCCA is used as a baseline competitor whose reported results are compared against, not as evidence for Hi-SAM's effectiveness; Climber is cited for an established one-pass ranking/inference trick and does not carry the central claim. Evaluation is against external Amazon 2023 benchmarks and a production DLRM baseline, so the main results are externally falsifiable. Concerns raised in the reader's take about baseline feature-alignment and possible leakage are experimental-validity risks, not circularity: even if the 'strictly aligned' claim (Sec. 4.1) were false, it would weaken attribution but would not make any derivation equivalent to its inputs. The H=8 vs H=4 discrepancy and under-specified baseline encoders are transparency/correctness issues, not circular steps. Therefore, no circularity is found.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The central claim rests on standard quantization and alignment machinery imported from external works (GRAM, RQ-VAE, vCLUB) plus several hand-chosen hyperparameters not all of which are reported (beta, lambda) or consistently reported (H). The only truly new invented objects are architectural: anchor tokens and the hierarchical RoPE split. No external falsifiable handle is provided for these components beyond the paper's own benchmarks.

free parameters (6)
  • Codebook size (512 x 256) = 512 codes x 256 dims
    Capacity of shared and modality-specific codebooks in DST; chosen by hand. Reported in Section 4.1 and Appendix B.2.
  • Quantization depth N_sh=3, N_sp=1 = 3 shared layers + 1 per modality
    Token length affects model complexity and information retention; chosen by hand. Section 4.1.
  • PSGR subspace count H = 8 (Section 4.1) vs 4 (Appendix B.2)
    Internal inconsistency; H controls how residuals are disentangled before semantic probing.
  • H-RoPE base frequencies = B_inter=10^4, B_intra=10^2
    Set for long inter-item vs short intra-item sequences; chosen by design. Section 3.3.1.
  • Loss weights beta and lambda = not reported
    Weights of quantization vs MI loss in Eq. (2); values not given, so exact reproduction is impossible.
  • Maximum sequence length = 300
    All models trained with max length 300; affects complexity and results. Section 4.1.
axioms (5)
  • domain assumption GRAM volume minimization on the unit hypersphere is a valid way to align multiple modalities
    Adopted from [6]; used in Section 3.2.1. If the Gram determinant is not a meaningful alignment measure for text/image/behavior embeddings, DST starts from misaligned features.
  • standard math vCLUB with a Gaussian variational family yields a reliable upper bound on mutual information
    Used in Section 3.2.2 and Appendix A.1. The bound is valid for the chosen family but can be loose, so minimizing it does not strictly guarantee disentanglement.
  • domain assumption RQ-VAE residual quantization converges to useful codebooks
    Central to DST shared branch; adopted from [28] without new convergence guarantees. Section 3.2.2.
  • domain assumption The SASRec behavioral encoder trained on time-separated samples contains no label leakage
    Appendix B.2 states this but does not detail the isolation procedure; leakage would inflate the reported gains.
  • standard math Block-diagonal rotation applies separately to the first d/2 and last d/2 dimensions without cross-interference
    Appendix A.3 derives the decoupled attention score; this is standard linear algebra if the split is exact.
invented entities (3)
  • Anchor Token ([ANC]) no independent evidence
    purpose: Compressed per-item summary; historical items are accessed only through anchors
    Internal architectural device; its benefits are measured only through downstream metrics in this paper.
  • H-RoPE coordinate (m,n) with split inter/intra subspaces no independent evidence
    purpose: Restore item-level and token-level hierarchy in position encoding
    Novel positional scheme; no falsifiable prediction outside the model's own evaluations.
  • Shared and modality-specific codebooks no independent evidence
    purpose: Disentangle cross-modal consensus from modality-specific residuals
    Internal representational constructs, evaluated only in aggregate.

pith-pipeline@v1.3.0-alltime-deepseek · 19401 in / 17015 out tokens · 139755 ms · 2026-08-02T23:59:01.728146+00:00 · methodology

0 comments
read the original abstract

Multi-modal recommendation has gained traction as items possess rich attributes like text and images. Semantic ID-based approaches effectively discretize this information into compact tokens. However, two challenges persist: (1) Suboptimal Tokenization: existing methods (e.g., RQ-VAE) lack disentanglement between shared cross-modal semantics and modality-specific details, causing redundancy or collapse; (2) Architecture-Data Mismatch: vanilla Transformers treat semantic IDs as flat streams, ignoring the hierarchy of user interactions, items, and tokens. Expanding items into multiple tokens amplifies length and noise, biasing attention toward local details over holistic semantics. We propose Hi-SAM, a Hierarchical Structure-Aware Multi-modal framework with two designs: (1) Disentangled Semantic Tokenizer (DST): unifies modalities via geometry-aware alignment and quantizes them via a coarse-to-fine strategy. Shared codebooks distill consensus while modality-specific ones recover nuances from residuals, enforced by mutual information minimization; (2) Hierarchical Memory-Anchor Transformer (HMAT): splits positional encoding into inter- and intra-item subspaces via Hierarchical RoPE to restore hierarchy. It inserts Anchor Tokens to condense items into compact memory, retaining details for the current item while accessing history only through compressed summaries. Experiments on real-world datasets show consistent improvements over SOTA baselines, especially in cold-start scenarios. Deployed on a large-scale social platform serving millions of users, Hi-SAM achieved a 6.55% gain in the core online metric.

Figures

Figures reproduced from arXiv: 2602.11799 by Chuanjiang Luo, Hongxiang Chen, Peiyao Lu, Pingjun Pan, Tingting Fei, Tingting Zhou.

Figure 1
Figure 1. Figure 1: The architecture of Hi-SAM, which consists of the Disentangled Semantic Tokenizer (DST) stage and the Hierarchical [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ablation study on different modality combinations. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Layer-wise cosine similarity heatmaps of the base [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Scalability analysis of Hi-SAM regarding (a) model depth, (b) sequence length, and (c) computational cost (GFLOPs). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: 3D t-SNE visualizations of the latent space before [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 9 linked inside Pith

  1. [1]

    Understanding scaling laws for recommendation models.arXiv preprint arXiv:2208.08489, 2022

    Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Understanding scaling laws for recommendation models.arXiv preprint arXiv:2208.08489, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Tallrec: An effective and efficient tuning framework to align large language model with recommendation

    Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. InProceedings of the 17th ACM conference on recommender systems, pages 1007–1014, 2023

    2023

  4. [4]

    Wide & deep learning for recommender systems

    Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. InProceedings of the 1st workshop on deep learning for recommender systems, pages 7–10, 2016

    2016

  5. [5]

    Club: A contrastive log-ratio upper bound of mutual information

    Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779–1788. PMLR, 2020

    2020

  6. [6]

    Gramian multimodal representation learning and alignment

    Giordano Cicchetti, Eleonora Grassucci, Luigi Sigillo, Danilo Comminiello, et al. Gramian multimodal representation learning and alignment. InProceedings of International Conference on Learning Representations (ICLR 2025). ICLR, 2025

    2025

  7. [7]

    Early vs late fusion in multimodal convolutional neural networks

    Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. Early vs late fusion in multimodal convolutional neural networks. In2020 IEEE 23rd international conference on information fusion (FUSION), pages 1–6. IEEE, 2020

    2020

  8. [8]

    Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5)

    Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). InProceedings of the 16th ACM conference on recommender systems, pages 299–315, 2022

    2022

  9. [9]

    Deepfm: a factorization-machine based neural network for ctr prediction.arXiv preprint arXiv:1703.04247, 2017

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr prediction.arXiv preprint arXiv:1703.04247, 2017

    Pith/arXiv arXiv 2017

  10. [10]

    Mtgr: Industrial-scale generative recommendation framework in meituan

    Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. Mtgr: Industrial-scale generative recommendation framework in meituan. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 5731–5738, 2025

    2025

  11. [11]

    Vbpr: visual bayesian personalized ranking from implicit feedback

    Ruining He and Julian McAuley. Vbpr: visual bayesian personalized ranking from implicit feedback. InProceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  12. [12]

    Towards universal sequence representation learning for recommender sys- tems

    Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender sys- tems. InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 585–593, 2022

    2022

  13. [13]

    Learning vector-quantized item representation for transferable sequential recommenders

    Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. InProceedings of the ACM Web Conference 2023, pages 1162–1171, 2023

    2023

  14. [14]

    Bridging language and items for retrieval and recommendation.arXiv preprint arXiv:2403.03952, 2024

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation.arXiv preprint arXiv:2403.03952, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Multimodal representation learning for recommendation in internet of things.IEEE Internet of Things Journal, 6(6):10675–10685, 2019

    Zhenhua Huang, Xin Xu, Juan Ni, Honghao Zhu, and Cheng Wang. Multimodal representation learning for recommendation in internet of things.IEEE Internet of Things Journal, 6(6):10675–10685, 2019

    2019

  16. [16]

    Self-attentive sequential recommenda- tion

    Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommenda- tion. In2018 IEEE international conference on data mining (ICDM), pages 197–206. IEEE, 2018

    2018

  17. [17]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  18. [18]

    Au- toregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Au- toregressive image generation using residual quantization. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11523– 11532, 2022

    2022

  19. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  20. [20]

    Vt- fsl: Bridging vision and text with llms for few-shot learning.arXiv preprint arXiv:2509.25033, 2025

    Wenhao Li, Qiangchang Wang, Xianjing Meng, Zhibin Wu, and Yilong Yin. Vt- fsl: Bridging vision and text with llms for few-shot learning.arXiv preprint arXiv:2509.25033, 2025

    arXiv 2025

  21. [21]

    Qarm: Quantitative alignment multi-modal recommendation at kuaishou

    Xinchen Luo, Jiangxia Cao, Tianyu Sun, Jinkai Yu, Rui Huang, Wei Yuan, Hezheng Lin, Yichen Zheng, Shiyao Wang, Qigen Hu, et al. Qarm: Quantitative alignment multi-modal recommendation at kuaishou. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 5915– 5922, 2025

    2025

  22. [22]

    Entire space multi-task model: An effective approach for estimating post-click conversion rate

    Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. InThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1137–1140, 2018

    2018

  23. [23]

    Multimodal movie recommendation system using deep learning.Mathematics, 11(4):895, 2023

    Yongheng Mu and Yun Wu. Multimodal movie recommendation system using deep learning.Mathematics, 11(4):895, 2023

    2023

  24. [24]

    Multimodal deep learning

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al. Multimodal deep learning. InICML, volume 11, pages 689–696, 2011

    2011

  25. [25]

    Balanced multimodal learning via on-the-fly gradient modulation

    Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8238–8247, 2022

    2022

  26. [26]

    When text-as-vision meets semantic ids in generative recom- mendation: An empirical study.arXiv preprint arXiv:2601.14697, 2026

    Shutong Qiao, Wei Yuan, Tong Chen, Xiangyu Zhao, Quoc Viet Hung Nguyen, and Hongzhi Yin. When text-as-vision meets semantic ids in generative recom- mendation: An empirical study.arXiv preprint arXiv:2601.14697, 2026

    arXiv 2026

  27. [27]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  28. [28]

    Recommender systems with generative retrieval.Advances in Neural Information Processing Systems, 36:10299–10315, 2023

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. Recommender systems with generative retrieval.Advances in Neural Information Processing Systems, 36:10299–10315, 2023

    2023

  29. [29]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Pith/arXiv arXiv 1908

  30. [30]

    The persistence and transience of memory.Neuron, 94(6):1071–1084, 2017

    Blake A Richards and Paul W Frankland. The persistence and transience of memory.Neuron, 94(6):1071–1084, 2017

    2017

  31. [31]

    Scaling law for recommendation models: Towards general-purpose user representations

    Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlén Ramström, Jisu Jeong, Jung-Woo Ha, and Kyung-Min Kim. Scaling law for recommendation models: Towards general-purpose user representations. InProceedings of the AAAI con- ference on artificial intelligence, volume 37, pages 4596–4604, 2023

    2023

  32. [32]

    Better generalization with semantic ids: A case study in ranking for recommendations

    Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, et al. Better generalization with semantic ids: A case study in ranking for recommendations. InProceedings of the 18th ACM Conference on Recommender Systems, pages 1039–1044, 2024

    2024

  33. [33]

    Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. InProceedings of the 28th ACM international conference on information and knowledge management, pages 1441–1450, 2019

    2019

  34. [34]

    Setrank: A setwise bayesian approach for collaborative ranking from implicit feedback

    Chao Wang, Hengshu Zhu, Chen Zhu, Chuan Qin, and Hui Xiong. Setrank: A setwise bayesian approach for collaborative ranking from implicit feedback. InProceedings of the aaai conference on artificial intelligence, volume 34, pages 6127–6136, 2020

    2020

  35. [35]

    Deep & cross network for ad click predictions

    Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. InProceedings of the ADKDD’17, pages 1–7. 2017

    2017

  36. [36]

    Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems

    Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. InProceedings of the web conference 2021, pages 1785–1797, 2021

    2021

  37. [37]

    Progressive semantic residual quantization for multimodal-joint interest modeling in music recommendation

    Shijia Wang, Tianpei Ouyang, Qiang Xiao, Dongjing Wang, Yintao Ren, Songpei Xu, Da Guo, and Chuanjiang Luo. Progressive semantic residual quantization for multimodal-joint interest modeling in music recommendation. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 6119–6127, 2025

    2025

  38. [38]

    Multi-modal self- supervised learning for recommendation

    Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. Multi-modal self- supervised learning for recommendation. InProceedings of the ACM web confer- ence 2023, pages 790–800, 2023

    2023

  39. [39]

    A survey on large language models for recommendation.World Wide Web, 27(5):60, 2024

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommendation.World Wide Web, 27(5):60, 2024

    2024

  40. [40]

    Climber: Toward efficient scaling laws for large recommendation models

    Songpei Xu, Shijia Wang, Da Guo, Xianwen Guo, Qiang Xiao, Bin Huang, Guanlin Wu, and Chuanjiang Luo. Climber: Toward efficient scaling laws for large recommendation models. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 6193–6200, 2025

    2025

  41. [41]

    Where to go next for recommender systems? id-vs

    Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? id-vs. modality- based recommender models revisited. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2639–2649, 2023

    2023

  42. [42]

    Soundstream: An end-to-end neural audio codec.IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 30:495–507, 2021

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec.IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 30:495–507, 2021

    2021

  43. [43]

    Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152, 2024

    Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Wukong: Towards a scaling law for large-scale recommendation.arXiv preprint arXiv:2403.02545, 2024

    Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. Wukong: Towards a scaling law for large-scale recommendation.arXiv preprint arXiv:2403.02545, 2024. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Pan et al

    Pith/arXiv arXiv 2024

  45. [45]

    Scaling law of large sequential recommendation models

    Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. Scaling law of large sequential recommendation models. InProceedings of the 18th ACM Conference on Recommender Systems, pages 444–453, 2024

    2024

  46. [46]

    Mining latent structures for multimedia recommendation

    Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. Mining latent structures for multimedia recommendation. InProceedings of the 29th ACM international conference on multimedia, pages 3872–3880, 2021

    2021

  47. [47]

    Latent structure mining with contrastive modality fusion for multimedia recommendation.IEEE Transactions on Knowledge and Data Engineering, 35(9): 9154–9167, 2022

    Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Mengqi Zhang, Shu Wu, and Liang Wang. Latent structure mining with contrastive modality fusion for multimedia recommendation.IEEE Transactions on Knowledge and Data Engineering, 35(9): 9154–9167, 2022

    2022

  48. [48]

    Collm: Integrating collaborative embeddings into large language models for recommendation.IEEE Transactions on Knowledge and Data Engineering, 2025

    Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: Integrating collaborative embeddings into large language models for recommendation.IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  49. [49]

    H2o: Heavy- hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy- hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    2023

  50. [50]

    Personalized multi modal alignment encoding for ctr-recommendation in wechat

    Jiawei Zheng, Hao Gu, Lingling Yi, Jie Wen, and Chuan Chen. Personalized multi modal alignment encoding for ctr-recommendation in wechat. InProceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 6301–6308, 2025

    2025

  51. [51]

    Deep interest network for click-through rate prediction

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1059–1068, 2018

    2018

  52. [52]

    Deep interest evolution network for click-through rate prediction

    Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 5941–5948, 2019

    2019

  53. [53]

    Semantic Shifts

    Hongyu Zhou, Xin Zhou, Zhiwei Zeng, Lingzi Zhang, and Zhiqi Shen. A com- prehensive survey on multimodal recommender systems: Taxonomy, evaluation, and future directions.arXiv preprint arXiv:2302.04473, 2023. A Supplement to Method A.1 Derivation of Mutual Information Minimization In this section, we provide the detailed derivation of the Mutual Informati...

    Pith/arXiv arXiv 2023