REVIEW 5 major objections 5 minor 1 cited by
RecBench-MD, a benchmark of 19 foundation models across 15 datasets and 10 domains, finds that in-domain fine-tuning beats cross-dataset transfer and that multi-domain training improves adaptability.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A large-scale benchmark finds that in-domain fine-tuning works best for foundation model recommenders, while cross-dataset and multi-domain training help in new scenarios.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A genuinely large and reusable benchmark for foundation-model recommendation; the resource is real, but the split description and variance reporting need to be fixed before the comparative claims can be trusted. the 5 major comments →
Evaluating Recabilities of Foundation Models: A Multi-Domain, Multi-Dataset Benchmark
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the paper's terms, the discovery is a set of measured regularities. Prompt-based ranking beats embedding-based matching at zero shot, aligning with pretraining. In-domain fine-tuning performs best and sets an upper bound cross-dataset transfer rarely exceeds. Cross-dataset fine-tuning generally helps but can hurt: Yelp and Hotel degrade most often, and models with strong zero-shot ability can lose it after tuning on unrelated data. Multi-domain training beats zero-shot and narrows the gap to the in-domain bound as models grow. The paper reads these as evidence that text-based transfer works without shared users or items, and that larger models exploit diverse training domains better.
What carries the argument
The load-bearing structure is RecBench-MD's eight-setting taxonomy (Settings A through H), which separates zero-shot from zero-resource evaluation and crosses single-dataset, single-domain, cross-dataset, cross-domain, and multi-domain conditions. Each setting isolates one question: intrinsic ability, in-domain learning, same-domain transfer, cross-domain transfer, or multi-domain fusion. Two evaluation heads keep the ranking from depending on one recommendation paradigm: prompt-based click prediction, which turns Yes/No output logits into a probability, and embedding-based matching, which measures cosine similarity between user and item encodings. The paper also introduces the Reciprocal Ra
Load-bearing premise
The load-bearing premise is that reducing every dataset to about 20,000 test and 100,000 fine-tuning interactions preserves each domain's real interaction structure; if those subsamples are unrepresentative, the transfer and multi-domain rankings could reflect sampling artifacts rather than genuine model abilities.
What would settle it
Re-run the cross-dataset and multi-domain tables on full datasets or on several fresh random subsamples with documented seeds. If the top-ranked fine-tuning sets, the degraded cells (Yelp and Hotel), and the multi-domain-versus-in-domain gaps change materially across draws, the paper's transfer conclusions do not survive.
If this is right
- For a new recommendation scenario with no in-domain labels, fine-tuning a large foundation model on existing datasets is a viable warm start, with in-domain fine-tuning remaining the ceiling.
- Multi-domain training is a dependable way to raise zero-shot ability, and its payoff grows with model scale.
- Single-dataset or single-domain evaluations of recommendation foundation models are unreliable; rankings should be averaged across multiple datasets and domains.
- Fine-tuning order matters in sequential multi-domain training: the dataset with the strongest single-dataset performance is best used second, and a weaker dataset can usefully go first.
- Prompt-based and embedding-based paradigms reward different models, so both need to be reported separately.
Where Pith is reading between the lines
- The paper groups domains by label, not by measured distributional similarity; a follow-up could quantify transferability as a function of item-text embedding distance or user-behavior overlap, which might explain why same-ecosystem pairs transfer especially well.
- The rankings are rankings of adaptation protocols as much as of base models, since everything is fine-tuned with LoRA and binary cross-entropy; changing rank, sequence length, or loss could reorder the leaderboard.
- The red-degraded cells imply practitioners should treat cross-dataset fine-tuning as conditional: when zero-shot performance is already strong or the domain gap is wide, freezing the model may beat transferring.
- Because the subsampling procedure is not documented, a re-sampling audit is the natural next check; if rankings flip across subsample draws, the transfer conclusions are not yet stable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RecBench-MD, a benchmark for evaluating foundation-model 'recabilities' in recommendation settings. It defines eight settings (A–H) covering zero-shot, single-domain, cross-dataset, cross-domain, and multi-domain fine-tuning; evaluates 19 foundation models on 15 datasets spanning 10 domains; and reports AUC for two recommendation approaches (prompt-based and embedding-based). The headline findings are that in-domain fine-tuning (Settings C/E) outperforms zero-shot, cross-dataset transfer is practically useful, and multi-domain training improves adaptability, with Setting G as a robust zero-resource strategy and Setting H as an upper bound.
Significance. If the methodology is sound, this would be the most comprehensive public benchmark of foundation-model recommendation ability to date. The released code and data, the breadth of models/datasets, and the explicit taxonomy of eight settings are genuine contributions. However, the empirical findings currently rest on an unspecified and potentially overlapping train/test split, and several stated conclusions are contradicted by the paper's own tables. The benchmark's value as a trustworthy reference therefore depends on resolving these issues.
major comments (5)
- [Section 4, Implementation Details; Section 3.1, Setting E] The paper does not specify how the ~20,000 test samples and ~100,000 fine-tuning samples are drawn. In the pair-wise click-prediction setup, the sampling unit is critical: if user–item interactions are sampled independently, the same user and possibly the same interaction can appear in both sets, enabling direct memorization and inflating AUC for every fine-tuned setting. Setting E explicitly states 'with potential overlap,' which turns this from an omission into a design choice. Please state whether splits are user-disjoint, item-disjoint, temporal, or random interaction-level, and report overlap statistics. If overlap exists, the fine-tuning comparisons in Tables 3–6 must be rerun under a non-overlapping split.
- [Section 4, final paragraph] The statement that 'statistically significant differences are observed (p <0.05)' is unsupported. No test statistic, variance estimate, or confidence interval is reported anywhere. Tables 3–5 give only point estimates of AUC, so the reader cannot judge whether headline gaps (e.g., Setting H vs. Setting G in Table 5) are noise. Specify the significance test, its pairing structure (across runs, datasets, or both), and report standard deviations or confidence intervals for the main comparisons.
- [Section 5.2, Table 3] The text claims that Setting C and Setting E 'consistently achieve superior performance' over the zero-shot baseline, but Table 3 contains clear counterexamples: OPT-1B Setting E on H&M is 0.5208 vs. zero-shot 0.5650 and on CDs is 0.4926 vs. 0.4994; Llama-3-8B Setting E on H&M is 0.5378 vs. zero-shot 0.7690. The claim should be weakened to 'on average' or should identify the conditions under which fine-tuning does not help.
- [Section 5.4, Table 5] The text says Setting H 'underperforms Setting G on five other datasets (Movie., Yelp, Steam, Elec., Hotel.)', but Table 5 shows the opposite for several entries: BERTbase Steam has H=0.6817 > G=0.6189, BERTbase Hotel has H=0.4869 > G=0.4795, and OPT-1B Yelp has H=0.5014 > G=0.4853. The robustness claim about Setting G needs to be restated to match the actual cross-model pattern.
- [Section 4, Datasets] The description says the fine-tuning set is randomly split 9:1 into training and validation, but it does not explain how the separate ~20k test set is constructed or whether it is disjoint from the fine-tuning set at the interaction/user level. For datasets in Table 2 that have both a test set and a fine-tuning set from the same original source, clarify whether users or items are shared between the two sets; this is essential for interpreting the transfer results.
minor comments (5)
- [Section 4 and Section 5.1] Model names are inconsistent: 'Misrtal-27B' vs. 'Mistral-2 7B', 'Qwen-27B' vs. 'Qwen-2 7B', and 'GLM-49B' vs. 'GLM-4 9B'. Please unify the names with the model list.
- [Table 1] The header 'RECBENCHRECBENCH-MD' appears corrupted; it should read 'RecBench-MD' or the consistent benchmark name used in the text.
- [Figure 2] The heatmap has no color scale or legend, so the claim that 'light-colored (yellow) areas indicate better recabilities' cannot be quantified. Add a colorbar and numeric normalization details.
- [Section 3.1 footnote] The definition of 'zero-resource' is confusing: Setting A is called 'zero-resource zero-shot' but involves no fine-tuning at all, while the footnote says zero-resource means fine-tuning on some datasets and testing on unseen ones. Clarify the terminology.
- [Throughout] There are several grammatical and typographical errors, e.g., 'The full evaluation results will available', 'small than 0.5', and 'Misrtal'. These should be cleaned up in a revision.
Circularity Check
No significant circularity: the paper reports measured benchmark results rather than deriving predictions from fitted inputs.
full rationale
RecBench-MD is an empirical benchmark paper. Its central claims—that in-domain fine-tuning achieves optimal performance, that cross-dataset transfer is practically useful, and that multi-domain training improves adaptability—are summaries of measured AUC values across Settings A–H. There is no step where a 'prediction' reduces to a fitted parameter or to a self-citation. The taxonomy of settings is a descriptive organization, not a derivation. Setting H is explicitly described as an upper bound because it is fine-tuned on datasets that share domains with the test sets; this is a benchmark design choice, not a hidden prediction. The RRA metric (Eq. 4) is a descriptive ranking summary, not a fitted quantity. Self-citations (e.g., Liu et al. 2025b for metric choice) concern standard evaluation practices and are not load-bearing. The only notable concern is the unspecified split unit and the phrase 'with potential overlap' for Setting E, which could threaten validity through user leakage, but the paper does not reduce any stated conclusion to that overlap, and circularity analysis cannot confirm leakage from the text alone. Thus the benchmark conclusions are self-contained empirical observations rather than circular derivations.
Axiom & Free-Parameter Ledger
free parameters (4)
- LoRA rank and alpha =
rank=32, alpha=128
- Learning rate =
1e-4
- User sequence truncation length =
20
- Test and fine-tune set sizes =
~20,000 test, ~100,000 fine-tune
axioms (6)
- standard math Binary cross-entropy loss is an appropriate objective for click prediction.
- standard math The softmax over Yes/No logits gives a valid click probability.
- domain assumption Cosine similarity of the last token embeddings is a valid user-item matching score.
- ad hoc to paper LoRA fine-tuning with the same hyperparameters is an adequate and fair proxy for full fine-tuning across all 19 models.
- domain assumption The selected textual features (title, description, etc.) are sufficient item representations for recommendation.
- domain assumption The standardized subsamples (20,000 test and 100,000 training examples) are representative of the original datasets.
Cite this review
Pith. "Pith review of Evaluating Recabilities of Foundation Models: A Multi-Domain, Multi-Dataset Benchmark." pith.science (2026). https://pith.science/paper/3AGJCW5P
@misc{pith2026250821354,
author = {Pith},
title = {Pith review of: Evaluating Recabilities of Foundation Models: A Multi-Domain, Multi-Dataset Benchmark},
year = {2026},
howpublished = {\url{https://pith.science/paper/3AGJCW5P}},
note = {Machine review of arXiv:2508.21354}
}
read the original abstract
Comprehensive evaluation of the recommendation capabilities of existing foundation models across diverse datasets and domains is essential for advancing the development of recommendation foundation models. In this study, we introduce RecBench-MD, a novel and comprehensive benchmark designed to assess the recommendation abilities of foundation models from a zero-resource, multi-dataset, and multi-domain perspective. Through extensive evaluations of 19 foundation models across 15 datasets spanning 10 diverse domains -- including e-commerce, entertainment, and social media -- we identify key characteristics of these models in recommendation tasks. Our findings suggest that in-domain fine-tuning achieves optimal performance, while cross-dataset transfer learning provides effective practical support for new recommendation scenarios. Additionally, we observe that multi-domain training significantly enhances the adaptability of foundation models. All code and data have been publicly released to facilitate future research.
Figures
Forward citations
Cited by 1 Pith paper
-
RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation
A from-scratch model that tokenizes items into hierarchical codes and predicts next-item codes reaches higher average zero-shot AUC on 8 datasets than LLM recommenders up to 7B parameters.
Reference graph
Works this paper leans on
-
[1]
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1007–1014, 2023a. Keqin Bao, Jizhi Zhang, Yang Zhang, Wang Wenjie, Fuli Feng, and Xiangnan He. Large lan...
-
[4]
Revisiting Word Embeddings in the LLM Era
Matthew Freestone and Shubhra Kanti Karmaker Santu. Word embeddings revisited: Do llms offer something new. arXiv preprint arXiv:2402.11094,
work page internal anchor Pith review Pith/arXiv arXiv
-
[5]
Cross-domain recom- mendation via cluster-level latent factor model
Sheng Gao, Hao Luo, Da Chen, Shantao Li, Patrick Gallinari, and Jun Guo. Cross-domain recom- mendation via cluster-level latent factor model. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part II 13, pages 161–176. Springer,
work page 2013
-
[7]
Lei Guo, Li Tang, Tong Chen, Lei Zhu, Quoc Viet Hung Nguyen, and Hongzhi Yin. Da-gcn: A domain-aware attentive graph convolution network for shared-account cross-domain sequential recommendation. arXiv preprint arXiv:2105.03300,
-
[9]
Beyond Utility: Evaluating LLM as Recommender
Chumeng Jiang, Jiayin Wang, Weizhi Ma, Charles LA Clarke, Shuai Wang, Chuhan Wu, and Min Zhang. Beyond utility: Evaluating llm as recommender. arXiv preprint arXiv:2411.00331,
work page internal anchor Pith review Pith/arXiv arXiv
-
[10]
Llava-onevision: Easy visual task transfer
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326,
-
[11]
Language model evolutionary algorithms for recommender systems: Benchmarks and algorithm comparisons
Jiao Liu, Zhu Sun, Shanshan Feng, and Yew-Soon Ong. Language model evolutionary algorithms for recommender systems: Benchmarks and algorithm comparisons. arXiv preprint arXiv:2411.10697, 2024a. Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, et al. Llmrec: Benchmarking large language ...
-
[12]
Learning Multi-Aspect Item Palette: A Semantic Tokenization Framework for Generative Recommendation
Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. Once: Boosting content-based recom- mendation with both open-and closed-source large language models. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 452–461, 2024b. Qijiong Liu, Jieming Zhu, Zhaocheng Du, Lu Fan, Zhou Zhao, and Xiao-Ming Wu. Learning mult...
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[13]
Accessed: 2025-05-08. Hoang Ngo and Dat Quoc Nguyen. Recgpt: Generative pre-training for text-based recommendation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Short Papers, Bangkok, Thailand, August 11-16, 2024, pages 302–313. Association fo...
work page 2025
-
[14]
URL https://openai.com/gpt. Large language model. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530,
-
[15]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay ...
-
[16]
Rethinking the evaluation for conversational recommendation in the era of large language models
Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. Rethinking the evaluation for conversational recommendation in the era of large language models. arXiv preprint arXiv:2305.13112,
-
[17]
Huggingface’s transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771,
Pith/arXiv arXiv 1910
-
[19]
A survey on large language models for recommendation
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024a. Xuansheng Wu, Huachi Zhou, Yucheng Shi, Wenlin Yao, Xiao Huang, and Ninghao Liu. Could small language models serve as recommenders? towards data-cen...
work page 2024
-
[20]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671,
-
[21]
Opt: Open pre-trained transformer language models
12 Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,
-
[22]
Language models as recommender systems: Evalua- tions and limitations
Yuhui Zhang, Hao Ding, Zeren Shui, Yifei Ma, James Zou, Anoop Deo- ras, and Hao Wang. Language models as recommender systems: Evalua- tions and limitations. In NeurIPS 2021 Workshop on I (Still) Can’t Believe It’s Not Better ,
work page 2021
-
[23]
doi: 10.1109/TKDE. 2024.3392335. URL https://doi.org/10.1109/TKDE.2024.3392335. Sashuai Zhou, Weinan Gan, Qijiong Liu, Ke Lei, Jieming Zhu, Hai Huang, Yan Xia, Ruiming Tang, Zhenhua Dong, and Zhou Zhao. Recbase: Generative foundation model pretraining for zero-shot recommendation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Langu...
arXiv 2024
-
[25]
The results suggest that using a lower-ranked dataset in the first step, followed by the top-performing dataset in the second step, tends to produce the best outcomes for the target test set. C.2 Additional Evaluation Metrics In the main text, we report only the AUC metric due to the space constraints. Here, we provide additional evaluation metrics, inclu...
work page 1903
-
[2013]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[2019]
Jiahao Wu, Qijiong Liu, Hengchang Hu, Wenqi Fan, Shengcai Liu, Qing Li, Xiao-Ming Wu, and Ke Tang. Leveraging large language models (llms) to empower training-free dataset condensation for content-based recommendation. arXiv preprint arXiv:2310.09874,
work page internal anchor Pith review Pith/arXiv arXiv
-
[2022]
Chatglm: A family of large language models from glm-130b to glm-4 all tools
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793,
-
[2023]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
-
[2024]
Differential private knowledge transfer for privacy-preserving cross-domain recommendation
Chaochao Chen, Huiwen Wu, Jiajie Su, Lingjuan Lyu, Xiaolin Zheng, and Li Wang. Differential private knowledge transfer for privacy-preserving cross-domain recommendation. In Proceedings of the ACM web conference 2022, pages 1455–1465,
work page 2022
-
[2025]
Cross-domain recommendation: challenges, progress, and prospects
Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. Cross-domain recommendation: challenges, progress, and prospects. arXiv preprint arXiv:2103.01696,
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.