REVIEW 3 major objections 6 minor 44 references
This paper introduces M3L-10M and M3L-20M, two large-scale, fully documented, and reproducible multimodal versions of MovieLens-10M and 20M, with text, image, audio, and video features extracted from plots, posters, and trailers.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 22:47 UTC pith:FS5GCQO4
load-bearing objection A useful, mostly transparent large-scale multimodal MovieLens release, but the undocumented 534-item drop in ML-20M needs to be explained before trusting the coverage numbers. the 3 major comments →
Binge Watch: Reproducible Multimodal Benchmarks Datasets for Large-Scale Movie Recommendation on MovieLens-10M and 20M
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that M3L-10M and M3L-20M provide the first fully documented, reproducible, large-scale multimodal versions of MovieLens. They preserve 100% of users and about 94% of ratings, while adding textual features from plot summaries, image features from posters, and acoustic and video features from trailers, encoded with a suite of public pretrained models. The paper further claims that a modality-fingerprinting analysis, based on MovieLens genome tags, shows text is the most narratively precise while video and audio capture stylistic, non-redundant signals; in the quantitative pilot, combining text and audio lets a standard model (VBPR) slightly beat a pure collaborative-filter
What carries the argument
The load-bearing artifact is the dataset construction pipeline: query a public movie metadata API for each MovieLens item to obtain plot, poster URL, and trailer URL; download posters and trailers; encode text with sentence transformers (MiniLM, MPNet, CLIP-Text), images with VGG16, ViT, and CLIP-Image, video with SlowFast, R(2+1)D, and MViT, and audio with VGGish, Whisper, and AST; then release the URL mappings and the resulting fixed-size embeddings while deleting raw media to avoid copyright issues. The second piece is the modality-fingerprinting analysis that quantifies each modality's narrative versus stylistic contribution using manually classified genome tags.
Load-bearing premise
The paper assumes that a movie without a downloadable trailer is safe to drop from the dataset, so that the remaining 71% of ML-20M movies still represents the original MovieLens distribution well enough for benchmarking.
What would settle it
A simple check: correlate trailer availability with release year, average rating, or rating count. If older or less popular movies are disproportionately excluded, the dataset no longer represents the full MovieLens population, and recommendation results on it would not generalize to the full catalog.
If this is right
- Researchers can benchmark multimodal recommenders on a dataset with four modalities and about 9.4M or 18.8M ratings without redoing data collection.
- Audio and video features, rarely used in previous MovieLens-based studies, become available for testing whether they improve performance in sparse and cold-start settings.
- The released embeddings support apples-to-apples comparison of fusion strategies since all models share the same input features.
- The documented pipeline can be re-run or extended to add new encoders or refresh outdated URLs, keeping the resource alive.
- If the preliminary result holds, text+audio combinations may become a new standard configuration, not just text+image.
Where Pith is reading between the lines
- The heavy pruning of items without trailers (about 29% of ML-20M movies are dropped) likely skews the dataset toward movies that are popular or new enough to have trailers; a testable extension is to compare recommendation accuracy on the dropped items using only plot/poster features.
- The frozen embeddings from fixed encoder versions may age as better encoders appear; an extension would be to release raw text and image URLs so future work can re-encode with newer models.
- The same pipeline could transfer to other domains (books, music, podcasts) where a public metadata API supplies covers, descriptions, and audio/video previews, giving multimodal recommender research more than one benchmark.
- Because the paper only ran its pilot on M3L-10M, the claim that audio is the key modality remains untested at the 20M scale; an obvious next step is to run the same models on M3L-20M.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents M3L-10M and M3L-20M, two multimodal augmentations of MovieLens-10M and MovieLens-20M. The construction pipeline queries TMDB for plot, poster, and trailer URLs, downloads available trailers, and encodes text/image/video/audio with multiple pretrained encoders. The authors release URL mappings, extracted embeddings, and code. They provide qualitative analyses (t-SNE, modality fingerprinting) and a preliminary quantitative comparison of VBPR, LATTICE, FREEDOM, and BPR on M3L-10M. The central claim is that these datasets are fully documented, reproducible, and large-scale resources for multimodal recommendation.
Significance. If the resource works as described, it addresses a real gap: public large-scale multimodal datasets with audio/video features. The release of mappings, extracted features, and pipeline code is a genuine contribution and goes beyond many prior works that do not release data. The paper's quantitative claims about the audio modality are preliminary and would need stronger experimental support. Overall, the dataset resource has the potential to be a useful benchmark, provided the construction is fully documented and the item filtering is transparent.
major comments (3)
- [§3.1, Tables 2 and 3] The paper never accounts for the difference between the official MovieLens-20M item count (27,278) and the N=26,744 used as the 'original' population in Tables 2 and 3. None of the three pipeline steps described in §3.1 removes 534 movies before trailer filtering. Since all coverage percentages are computed relative to this reduced denominator, the reported coverage overstates the true overlap with MovieLens-20M (19,009/27,278 ≈ 69.7% vs 71.1%). This undocumented filtering step directly conflicts with the 'fully documented and reproducible' claim and must be corrected by either documenting the step (e.g., removing movies with no TMDB ID) or reporting official counts.
- [§3.1(3), Tables 2 and 3] The trailer-availability filter removes 28.9% of ML-20M items (26,744 → 19,009). The paper notes geographic restrictions and deleted videos but does not analyze whether missingness is random. If trailer availability correlates with popularity, age, genre, or country, M3L-20M is a biased subset of MovieLens-20M. Given the paper's emphasis on 'preserving original user-item interactions' and 'highest possible item coverage,' a bias analysis (e.g., missingness vs release year, popularity, genre) should be added, and the limitations explicitly discussed.
- [§5, Table 5] The quantitative validation uses a single 8:1:1 split and no repeated runs, so the differences in Table 5 (e.g., VBPR Text+Audio 0.1938 vs Text+Video 0.1919) are within plausible noise. The claim that 'Audio modality is a key component of the best-performing configuration for every model' is stronger than the evidence allows. Please report multiple runs with standard deviations or significance tests, or reframe the results as exploratory. This is not a blocker for the dataset release, but it is load-bearing for the paper's empirical claim.
minor comments (6)
- [§3] The statement 'ML-10M is a subset of ML-20M... can even be directly extracted from the larger one' is inaccurate or at least imprecise: MovieLens-10M and MovieLens-20M are separate releases with different user sets (69,878 vs 138,493 in Table 3). If ML-10M were extracted from ML-20M, the user/item counts would differ from the original ML-10M. Please clarify the relationship and specify which official version of each dataset was used.
- [Figure 2 caption and §4.1] The caption attributes specific statements about Psycho and Awakenings to Text and Image that are not present in §4.1. For example, the caption says Text places Psycho and Awakenings in 'overlapping central regions,' but §4.1 discusses Star Wars/Star Trek and Platoon for Text. Align the caption and body text to avoid confusion.
- [§3.1(3)] The text says 'all retrieved images are available for download,' but Table 2 shows poster coverage below 100% (91.91% for ML-20M). Clarify why some movies lack posters despite this statement.
- [Table 1] Typo in header: 'Reproducibile' should be 'Reproducible'. Also, 'ML-20MCineFinder' is missing a space.
- [References] Reference [19] is incomplete ('[n. d.]' with no year or venue); provide a full citation.
- [§4.2] The manual classification of genome tags into 'narrative' and 'stylistic' categories is a subjective step. Ensure the exact tag lists and normalization code are in the repository, as claimed, so the fingerprinting analysis is reproducible.
Circularity Check
No circularity: the dataset is assembled from external sources and pretrained encoders, with no prediction reducing to its inputs.
full rationale
The paper's central contribution is a resource, not a derived prediction. M3L-10M and M3L-20M are constructed by querying TMDB for plots/posters/trailers and encoding them with external pretrained models (MiniLM, MPNet, CLIP, VGG16, ViT, SlowFast, R(2+1)D, MViT, VGGish, Whisper, AST). The claimed coverage statistics are measured after that external data-collection process, and the recommendation experiments in Section 5 validate the released features against a separate interaction signal (BPR, VBPR, LATTICE, FREEDOM) rather than defining or fitting the resource. The 'protocol follows [23]' statement is a normal methodological self-citation and is not load-bearing: the prior work does not supply the MovieLens-10M/20M data, the TMDB/YouTube raw sources, or the encoder outputs that constitute the present release. No equation or construction step defines a target quantity in terms of itself, no fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. The observed discrepancy between the original ML-20M item count (27,278) and the paper's N=26,744 in Tables 2 and 3 is a documentation/reproducibility concern about an undocumented pre-filter, but it is not circularity: it does not make the released dataset equivalent to its inputs by construction. Accordingly, no circular step is identified and the score is 0.
Axiom & Free-Parameter Ledger
free parameters (1)
- Genome-tag narrative/stylistic classification
axioms (4)
- domain assumption TMDB API and YouTube are reliable and legally usable sources for plots, posters, and trailers.
- domain assumption Pretrained encoders produce embeddings that faithfully capture semantic, visual, acoustic, and motion content.
- domain assumption MovieLens genome relevance scores and genre labels are valid ground truth for judging latent spaces.
- domain assumption Trailer availability is not systematically correlated with movie characteristics.
read the original abstract
As Multimodal Recommender Systems gain interest, high-quality datasets with multimedia side information have become essential. However, much of the current literature reports experiments that rely on small-scale, undocumented, or non-public datasets. In this paper, we introduce M3L-10M and M3L-20M, two large-scale, fully documented and reproducible datasets that enrich MovieLens-10M and MovieLens-20M with multimodal features. Following a documented pipeline, we collect movie plots, posters, and trailers and extract features using state-of-the-art encoders. We publicly release raw data mappings, extracted features, and complete datasets to foster reproducibility and advance the field. Qualitative and quantitative analyses demonstrate the quality of our datasets across multiple perspectives. This work establishes a foundational resource for large-scale, multimodal movie recommendation. Our resource is available at: https://zenodo.org/records/18499145, with source code at https://github.com/giuspillo/M3L_10M_20M.
Figures
Reference graph
Works this paper leans on
-
[1]
Kingma DP Ba J Adam et al. 2014. A method for stochastic optimization.arXiv preprint arXiv:1412.69801412, 6 (2014). Binge Watch: Reproducible Multimodal Benchmarks Datasets for Large-Scale Movie Recommendation on MovieLens-10M and 20M Conference’17, July 2017, Washington, DC, USA
arXiv 2014
-
[2]
Matteo Attimonelli, Danilo Danese, Angela Di Fazio, Daniele Malitesta, Claudio Pomo, and Tommaso Di Noia. 2025. Large-scale Benchmarks for Multimodal Recommendation with Ducho.Expert Systems with Applications(2025), 130813
2025
-
[3]
Desheng Cai, Shengsheng Qian, Quan Fang, Jun Hu, and Changsheng Xu. 2022. Adaptive anti-bottleneck multi-modal graph learning network for personalized micro-video recommendation. InProceedings of the 30th ACM International Con- ference on Multimedia. 581–590
2022
-
[4]
Feiyu Chen, Junjie Wang, Yinwei Wei, Hai-Tao Zheng, and Jie Shao. 2022. Break- ing isolation: Multimodal graph fusion for multimedia recommendation by edge- wise modulation. InProceedings of the 30th ACM International Conference on Multimedia. 385–394
2022
-
[5]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. InInterna- tional Conference on Learning Representations. https:...
2021
-
[6]
Xiaoyu Du, Zike Wu, Fuli Feng, Xiangnan He, and Jinhui Tang. 2022. Invariant representation learning for multimedia recommendation. InProceedings of the 30th ACM international conference on multimedia. 619–628
2022
-
[7]
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. 2021. Multiscale Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 6824–6835. https://arxiv.org/abs/2104.11227
Pith/arXiv arXiv 2021
-
[8]
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slow- fast networks for video recognition. InProceedings of the IEEE/CVF international conference on computer vision. 6202–6211
2019
-
[9]
Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. InInterspeech 2021. 571–575. doi:10.21437/Interspeech.2021-698
-
[10]
Ruining He and Julian McAuley. 2016. VBPR: visual bayesian personalized ranking from implicit feedback. InProceedings of the AAAI conference on artificial intelligence, Vol. 30
2016
-
[11]
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 131–135
2017
-
[12]
Fei Lei, Zhongqi Cao, Yuning Yang, Yibo Ding, and Cong Zhang. 2023. Learning the user’s deeper preferences for multi-modal recommendation systems.ACM Transactions on Multimedia Computing, Communications and Applications19, 3s (2023), 1–18
2023
-
[13]
Qidong Liu, Jiaxi Hu, Yutian Xiao, Xiangyu Zhao, Jingtong Gao, Wanyu Wang, Qing Li, and Jiliang Tang. 2024. Multimodal recommender systems: A survey. Comput. Surveys57, 2 (2024), 1–17
2024
-
[14]
Qijiong Liu, Jieming Zhu, Yanting Yang, Quanyu Dai, Zhaocheng Du, Xiao-Ming Wu, Zhou Zhao, Rui Zhang, and Zhenhua Dong. 2024. Multimodal pretraining, adaptation, and generation for recommendation: A survey. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6566–6576
2024
-
[15]
Xiaohao Liu, Zhulin Tao, Jiahong Shao, Lifang Yang, and Xianglin Huang. 2022. Elimrec: Eliminating single-modal bias in multimedia recommendation. InPro- ceedings of the 30th ACM international conference on multimedia. 687–695
2022
-
[16]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PmLR, 8748–8763
2021
-
[17]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning. PMLR, 28492–28518
2023
-
[18]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme
-
[19]
Mehmet Sarıçiçek, Rukiye Orman, MURAT Dener, and Harun Kınacı. [n. d.]. CineFinder: A Movie Recommendation System Using Visual and Textual Deep Features. ([n. d.])
-
[20]
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007. Collaborative filtering recommender systems. InThe adaptive web: methods and strategies of web personalization. Springer, 291–324
2007
-
[21]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Net- works for Large-Scale Image Recognition. In3rd International Conference on Learn- ing Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.1556
Pith/arXiv arXiv 2015
-
[22]
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding.Advances in neural information processing systems33 (2020), 16857–16867
2020
-
[23]
Giuseppe Spillo, Elio Musacchio, Cataldo Musto, Marco de Gemmis, Pasquale Lops, and Giovanni Semeraro. 2025. See the movie, hear the song, read the book: Extending movielens-1m, last. fm-2k, and DBbook with multimodal data. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 847–856
2025
-
[24]
Rui Sun, Xuezhi Cao, Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang, and Kai Zheng. 2020. Multi-modal knowledge graphs for recommender systems. InProceedings of the 29th ACM international conference on information & knowledge management. 1405–1414
2020
-
[25]
Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-Seng Chua. 2022. Self-supervised learning for multimedia recommenda- tion.IEEE Transactions on Multimedia25 (2022), 5107–5116
2022
-
[26]
Zhulin Tao, Yinwei Wei, Xiang Wang, Xiangnan He, Xianglin Huang, and Tat- Seng Chua. 2020. Mgat: Multimodal graph attention network for recommendation. Information Processing & Management57, 5 (2020), 102277
2020
-
[27]
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018. A closer look at spatiotemporal convolutions for action recognition. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition. 6450–6459
2018
-
[28]
Qifan Wang, Yinwei Wei, Jianhua Yin, Jianlong Wu, Xuemeng Song, and Liqiang Nie. 2021. Dualgnn: Dual graph neural network for multimedia recommendation. IEEE Transactions on Multimedia25 (2021), 1074–1084
2021
-
[29]
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou
-
[30]
Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020. Graph-refined convolutional network for multimedia recommendation with implicit feedback. InProceedings of the 28th ACM international conference on multimedia. 3541–3549
2020
-
[31]
Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. InProceedings of the 27th ACM international conference on multimedia. 1437–1445
2019
-
[32]
Zixuan Yi, Zijun Long, Iadh Ounis, Craig Macdonald, and Richard Mccreadie
-
[33]
Zixuan Yi, Xi Wang, Iadh Ounis, and Craig Macdonald. 2022. Multi-modal graph contrastive learning for micro-video recommendation. InProceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 1807–1811
2022
-
[34]
Penghang Yu, Zhiyi Tan, Guanming Lu, and Bing-Kun Bao. 2023. Multi-view graph convolutional network for multimedia recommendation. InProceedings of the 31st ACM international conference on multimedia. 6576–6585
2023
-
[35]
Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang
-
[36]
Hongyu Zhou, Yinan Zhang, Aixin Sun, and Zhiqi Shen. 2025. Does Multimodality Improve Recommender Systems as Expected? A Critical Analysis and Future Directions. arXiv:2508.05377 [cs.IR] https://arxiv.org/abs/2508.05377
Pith/arXiv arXiv 2025
-
[37]
Hongyu Zhou, Xin Zhou, Zhiwei Zeng, Lingzi Zhang, and Zhiqi Shen. 2023. A comprehensive survey on multimodal recommender systems: Taxonomy, evalua- tion, and future directions.arXiv preprint arXiv:2302.04473(2023)
Pith/arXiv arXiv 2023
-
[38]
Xin Zhou. 2023. Mmrec: Simplifying multimodal recommendation. InProceedings of the 5th ACM International Conference on Multimedia in Asia Workshops. 1–2
2023
-
[39]
Xin Zhou and Zhiqi Shen. 2023. A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation. InProceedings of the 31st ACM international conference on multimedia. 935–943
2023
-
[40]
Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023. Bootstrap latent representations for multi- modal recommendation. InProceedings of the ACM web conference 2023. 845–854
2023
-
[2012]
BPR: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618(2012)
Pith/arXiv arXiv 2012
-
[2020]
InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’20)
MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 485, 13 pages
-
[2021]
InProceedings of the 29th ACM international conference on multimedia
Mining latent structures for multimedia recommendation. InProceedings of the 29th ACM international conference on multimedia. 3872–3880
-
[2025]
Enhancing recommender systems: Deep modality alignment with large multi-modal encoders.ACM Transactions on Recommender Systems3, 4 (2025), 1–25
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.