REVIEW 3 major objections 5 minor 32 references
HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read HotelMatch-LLM claims a new best result in natural-language hotel retrieval by pairing a small query encoder with a large document encoder and pooling every property image into a fixed set of visual tokens.
desk verdict A credible applied IR paper with a practical asymmetric SLM/LLM architecture, but the shared GPT-4o label source for training and evaluation means the claimed gains are internally consistent yet externally unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the asymmetric dual encoder: a small language model (110M parameters) encodes queries, while a large language model (335M to 7B parameters) encodes hotels, and a linear projection aligns the query embedding dimension with the document embedding dimension. On the image side, all property images pass through a frozen CLIP visual encoder, their patch embeddings are mean-pooled across images into a fixed $49 \times d$ tensor, linearly projected into the LLM embedding space, and inserted between `<image_start>` and `<image_end>` tokens before the hotel text, so the number of images does not change the model input length. The final document embedding comes from the LLM's CLS token or mean-pooled token representation, and training combines three losses as $L = 0.7 L_{\text{Ret}} + 0.2 L_{\text{MLM}} + 0.1 L_{\text{VisF}}$, where the MLM and visual-facility objectives are calibrated to geography and amenities.
What would settle it
Re-annotate a random sample of HotelMatch test queries with human relevance judgments and rerun the Table 3 comparison; if the HotelMatch-LLM versus MARVEL MRR gap shrinks or reverses on the human-labeled subset, the synthetic labels, not the architecture, are the likely source of the reported gain.
Extended reading notes
Core claim
The central discovery is that an asymmetric retrieval architecture, small model for queries and large model for documents, together with full-gallery image pooling, sets a new best result for multimodal hotel retrieval. On the main multimodal test set, HotelMatch-LLM reaches MRR@10 0.681 and nDCG@10 0.600, beating MARVEL (0.603 and 0.503) and VISTA (0.582 and 0.465), and it also leads on the vision-driven, text-driven, and out-of-distribution sets. The authors attribute the improvement to three jointly trained objectives (retrieval, masked language modeling, and visual facility learning), to processing all property images via mean pooling of CLIP patch embeddings into 49 visual tokens, and to separate learning rates for the small and large encoders, which they show is critical: shared learning rates give MRR 0.315 versus 0.681 with separate rates. The method generalizes across LLM backbones, with MRR rising to 0.719 when a 7B model embeds hotels.
Load-bearing premise
The load-bearing premise is that GPT-4o's binary relevance labels, used both for training and for measuring all models, match what human travelers would judge relevant; the paper's only direct check is a 0.95 Pearson correlation between MUMIC-derived facility lists and actual images, not a human agreement study on the retrieval labels.
Editorial extensions
If this is right
- Hotel platforms can index every photo of a property at fixed cost: 49 pooled visual tokens are independent of gallery size, so a 300-image hotel costs the same to embed as a 10-image hotel.
- Query-time cost stays close to a 110M-parameter model (18.69 ms per query versus 31.07 ms for MARVEL), so the accuracy gains need not raise serving latency.
- The multi-task objective matters: removing the masked-language-modeling loss drops MRR from 0.681 to 0.650, and removing both auxiliary losses drops it to 0.632, indicating that city and country prediction is load-bearing.
- Effectiveness scales with the document-side backbone: swapping the large encoder to a 1.5B or 7B model raises MRR to 0.694 and 0.719, so the approach benefits from bigger hotel encoders without slowing queries.
- Natural-language property search becomes feasible as a primary interface, since the model handles free-form multimodal phrases and out-of-distribution constraints such as "3-star hotel near a subway station, 30 minutes from Tokyo station."
Reading between the lines
- The same asymmetric recipe should transfer to other verticals with rich multimodal documents, such as real-estate listings, vehicle inventory, or product catalogs, where indexing is offline and query latency matters; the paper does not test this.
- Mean pooling all images into 49 patches is cheap and fixed-size, but it likely dilutes distinctive single-image features such as a standout view from one photo; a testable variant would pool per-patch with attention weights or keep top-activation patches.
- The vision- and text-driven test queries are generated from the hotel content itself, so those two sets may overstate performance on real user phrasing; a harder evaluation would use logged user queries with human relevance judgments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents HotelMatch-LLM, a multimodal dense retriever for hotel search that combines a small language model (SLM) for query encoding with a large language model (LLM) for document encoding, and augments text with visual tokens obtained by mean-pooling CLIP patch embeddings over all property images. The model is trained with a weighted combination of contrastive retrieval loss, masked language modeling over city/country tokens, and visual facility classification. Experiments on the HotelMatch dataset (3.1M hotels, four test sets) report MRR@10 of 0.681 versus 0.603 for MARVEL, with gains on all test sets, alongside ablations, a full-ranking comparison, and latency measurements.
Significance. If the reported results are valid, the paper makes a useful practical contribution: the asymmetric architecture keeps query-time cost near that of a small model while leveraging a large model offline, and the patch-pooling mechanism provides a simple way to handle arbitrarily large image galleries. The promise to release the test set is a positive step for reproducibility. However, the central evaluation rests on GPT-4o-generated relevance labels, which the paper does not validate against human judgments; this substantially tempers the strength of the 'state-of-the-art' claim, although the comparative framework with fine-tuned baselines is internally coherent.
major comments (3)
- [Section 4 (Evaluation) and Figure 4 / A.2] The evaluation labels are generated by GPT-4o with the same binary-relevance instruction used to produce training labels, so HotelMatch-LLM is optimized (Eq. 15) to reproduce GPT-4o judgments and then measured against GPT-4o judgments. The only validity evidence in Section A.2 is a Pearson correlation of 0.95 between annotations made from real images and from MUMIC facility tags; this validates the image-to-text proxy, not the correctness of the relevance judgments themselves. Without a human-labeled test set or an explicit demonstration that GPT-4o's binary labels agree with human raters (e.g., Cohen's kappa on a sample), the reported superiority over MARVEL and VISTA may reflect alignment with the labeler's biases rather than true hotel-search relevance. The limitation statement in Section 8 acknowledges dependence on synthetic labels but does not condition the main conclusions on this caveat.
- [Section 4 (Evaluation)] The sentence 'Irrelevant queries are excluded' is a problematic reporting choice. If queries without any relevant document in the top-100 candidate set are dropped, the metrics in Table 3 are computed on an unknown favorable subset. The paper should report the number of excluded queries, include them in the evaluation with zero gains, and state the exclusion criterion. This is essential because Table 3 is the central evidence for the SOTA claim.
- [Section 6 (Full-ranking, Table 7)] The full-ranking evaluation annotates only the top-10 documents per model, again with GPT-4o. This truncation penalizes models that place a relevant document outside the top-10 without a principled basis, and the use of the same labeler does not resolve the validity concern. The paper should either use a fixed candidate pool with complete labels or report the evaluation on a human-labeled subset.
minor comments (5)
- [Section 1] The phrase 'with generating 667 billion in revenue' is ungrammatical; the sentence needs revision. Additionally, footnote 2 contains the visible artifact 'wreview=falseork' that should be removed.
- [Section 5] The research-question numbering is inconsistent: the list assigns RQ3 to multi-task optimization, but the results subsection uses RQ3 for generalization across LLMs and leaves the multi-task ablation unnumbered.
- [Table 3] The header contains 'MRR@' without a cutoff in the second column block; use 'MRR@10' consistently in the table and captions.
- [Table 5] The first row reports only 'GTR-Large-335M' for the query embedder without stating the hotel embedder; clarify that both sides use GTR-Large (or specify the configuration explicitly).
- [Section A.2] The heading 'A.2.1 Domain-specific Multi-task Optimization' appears in the middle of the relevance-label appendix and seems to be a leftover; the content immediately following is a figure caption, so the heading should be removed or renamed.
Circularity Check
Evaluation reuses the GPT-4o labels used for training, making the SOTA claim self-referential rather than human-validated.
-
fitted input called prediction
[Sections 3.3 (Eq. 10/15), 4 (Evaluation), A.2 (Figure 4)]
"We train HotelMatch-LLM using synthetic relevance labels generated by GPT-4o given a pair of query and property ... We use the prompt shown in Figure 4 detailed in Section A.2. ... We utilize GPT-4o for synthetic annotations of top-100 candidates, using the prompt shown in Figure 4 detailed in Section A.2, reducing reliance on costly human labeling while maintaining high quality, and performance is evaluated using MRR and nDCG at top-10 results. ..."
Training L_Ret (Eq. 10) optimizes P(d+|q) over positives chosen by GPT-4o's binary judgments; the test MRR/nDCG is computed from GPT-4o's binary judgments using the same Figure 4 prompt. So the fitted objective and the reported metric are the same labeling function. The only validation (A.2) is a 0.95 Pearson correlation between image-input and MUMIC-tag-input GPT-4o annotations, not GPT-4o-vs-human agreement, and Section 8 concedes the model 'relies heavily on the quality of the synthetic data generated by GPT-4o.' The headline SOTA gap (0.681 vs 0.603) therefore measures alignment with the training labeler; the comparison with baselines is informative only if GPT-4o relevance is assumed to equal human relevance, which is cited, not demonstrated.
full rationale
The main derivation chain — architecture, loss composition (Eq. 15), image pooling, learning-rate asymmetry, ablations (Tables 4–6), and generalization across backbones (Table 5) — is self-contained and does not reduce to a fit or to a self-citation. The MUMIC-based facility labels and the MUMIC tags in the relevance prompt come from prior work that is externally published, and the 0.95 correlation is at least an internal consistency check for the image-to-text proxy. The load-bearing weakness is the identity of the label source: training positives/negatives and test judgments are both produced by GPT-4o with the same Figure 4 prompt. The model is therefore optimized to reproduce GPT-4o's notion of relevance and then evaluated against that same notion. This does not make the baseline comparison meaningless (all systems are scored on the same labels), but it does mean the reported SOTA margin is not evidence about human-judged relevance; no human agreement is reported, and Section 8 explicitly conditions the results on synthetic label quality. Hence a moderate circularity score of 4: the central claim has independent comparative content, but the target metric is self-referential and unvalidated against humans.
Assumptions & free parameters
free parameters (5)
- lambda_1 (retrieval loss weight) =
0.7
- lambda_2 (MLM loss weight) =
0.2
- lambda_3 (VisF loss weight) =
0.1
- SLM learning rate =
5e-4
- LLM learning rate =
5e-6
assumptions (3)
- domain assumption GPT-4o generated relevance labels are a valid proxy for human relevance judgments
- domain assumption MUMIC facility labels from hotel images are accurate enough to train visual facility learning
- domain assumption The top-100 candidates from the fine-tuned CLIP retriever contain the relevant hotels for re-ranking evaluation
Cite this review
Pith. "Pith review of HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval." pith.science (2026). https://pith.science/paper/SZLKGFJT
@misc{pith2026250607296,
author = {Pith},
title = {Pith review of: HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZLKGFJT}},
note = {Machine review of arXiv:2506.07296}
}
read the original abstract
We present HotelMatch-LLM, a multimodal dense retrieval model for the travel domain that enables natural language property search, addressing the limitations of traditional travel search engines which require users to start with a destination and editing search parameters. HotelMatch-LLM features three key innovations: (1) Domain-specific multi-task optimization with three novel retrieval, visual, and language modeling objectives; (2) Asymmetrical dense retrieval architecture combining a small language model (SLM) for efficient online query processing and a large language model (LLM) for embedding hotel data; and (3) Extensive image processing to handle all property image galleries. Experiments on four diverse test sets show HotelMatch-LLM significantly outperforms state-of-the-art models, including VISTA and MARVEL. Specifically, on the test set -- main query type -- we achieve 0.681 for HotelMatch-LLM compared to 0.603 for the most effective baseline, MARVEL. Our analysis highlights the impact of our multi-task optimization, the generalizability of HotelMatch-LLM across LLM architectures, and its scalability for processing large image galleries.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12b. arXiv preprint arXiv:2410.07073
arXiv 2024
-
[4]
Zeta Alpha AI. 2024. Zeta-alpha-e5-mistral. https://huggingface.co/zeta-alpha-ai/Zeta-Alpha-E5-Mistral. Accessed: 2024-12-11
work page 2024
-
[5]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkow...
work page 2022
-
[6]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3
arXiv 2023
-
[7]
Aditi Chaudhary, Karthik Raman, and Michael Bendersky. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.107 It ' s all relative! -- a synthetic query generation approach for improving zero-shot relevance prediction . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1645--1664, Mexico City, Mexico. Association for Comput...
-
[8]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library
arXiv 2024
Show all 32 references
-
[9]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361
2020 arXiv
-
[10]
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...
2020 doi
-
[11]
Omar Khattab and Matei Zaharia. 2020. https://doi.org/10.1145/3397271.3401075 Colbert: Efficient and effective passage search via contextualized late interaction over bert . In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Informatio...
2020
-
[12]
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428
2024 arXiv
-
[13]
Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. https://arxiv.org/abs/2409.15700 Making text embedders few-shot learners . Preprint, arXiv:2409.15700
2024 arXiv
-
[14]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org
2023
-
[15]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://openreview.net/forum?id=w0H2xGHlkw Visual instruction tuning . In Thirty-seventh Conference on Neural Information Processing Systems
2023
-
[16]
Jingjing Liu, Chang Liu, and Nicholas J Belkin. 2020. Personalization in text information retrieval: A survey. Journal of the Association for Information Science and Technology, 71(3):349--369
2020
-
[17]
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. https://doi.org/10.18653/v1/2023.eacl-main.148 MTEB : Massive text embedding benchmark . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages...
2023 doi
-
[18]
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.669 Large dual encoders are generalizable retrievers . In Proceedings of the 2022 Con...
2022 doi
-
[19]
Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W. Lee. 2024. Any-precision llm: Low-cost deployment of multiple, different-sized llms. In Proceedings of the 41st International Conference on Machine Learning
2024
-
[20]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch
2017
-
[21]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...
2021
-
[22]
Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, and Daniel Campos
Hossein A. Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, and Daniel Campos. 2024. https://doi.org/10.1145/3626772.3657942 Synthetic test collections for retrieval evaluation . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in I...
2024
-
[23]
Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR’94, pages 232--241. Springer
1994
-
[24]
Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. https://doi.org/10.1145/3626772.3657707 Large language models can accurately predict searcher preferences . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Informat...
2024
-
[25]
Online travel booking statistics
TravelPerk . Online travel booking statistics. https://www.travelperk.com/blog/online-travel-booking-statistics/. Accessed: 2024-12-10
2024
-
[26]
Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor. arXiv preprint arXiv:2406.06519
2024 arXiv
-
[27]
Fengjun Wang, Sarai Mizrachi, Moran Beladev, Guy Nadav, Gil Amsalem, Karen Lastmann Assaraf, and Hadas Harush Boker. 2023. https://doi.org/10.1609/aaai.v37i13.26850 Mumic - multimodal embedding for multi-label image classification with tempered sigmoid . In Proceedings of the ...
2023 doi
-
[28]
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. https://doi.org/10.18653/v1/2024.acl-long.642 Improving text embeddings with large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...
2024 doi
-
[29]
Bennett, Junaid Ahmed, and Arnold Overwijk
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. https://openreview.net/forum?id=zeFrfgyZln Approximate nearest neighbor negative contrastive learning for dense text retrieval . In International Conference o...
2021
-
[30]
Dun Zhang. 2024. Stella\_en\_1.5b\_v5. https://huggingface.co/dunzhang/stella_en_1.5B_v5. Accessed: 2024-12-11
2024
-
[31]
Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.175 VISTA : Visualized text embedding for universal multi-modal retrieval . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu...
2024 doi
-
[32]
Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.783 MARVEL : Unlocking the multi-modal capability of dense retrieval via visual module plugin . In Proceedings of the 62nd Annual Me...
2024 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.