REVIEW 3 major objections 5 minor 5 references
Hotel Recommendation System
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper's best hotel-recommendation model exploits a data leak to reach 0.496 MAP@5.
desk verdict Honest Kaggle write-up whose headline number is a known data-leak lookup, not a reusable method; clear about its own limitation but the central claim doesn't generalize. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the data-leak lookup: a matching procedure that, for each test row, searches the training set for a row with the same values on user location country, orig destination distance, user location region, user location city, and hotel market. When an exact match is found, the test row is assigned the matched row's hotel cluster; partial matches are also searched, and rows with no match fall back to the most frequent clusters. This lookup is what carries the argument because the pure leak solution alone scores 0.491, a larger gain over the ensemble's 0.450 than any other single improvement. The second machinery is the probability-averaged ensemble of random forest, gradient boosting, SGD, and naive Bayes, which supplies predictions for the rows the lookup cannot resolve.
What would settle it
Take the test set and mark each row as matched or unmatched based on whether an exact match on the five leak features exists in the training set. Compute MAP@5 separately for the matched and unmatched subsets under the EL+DL rule: if the matched subset's score is substantially higher than the unmatched subset's, the reported 0.496 is driven by the leak. Alternatively, retrain the ensemble after removing or permuting the five leak features; if the score falls to approximately 0.450, the leak is the source of the gain.
Extended reading notes
Core claim
The central discovery is that the Ensemble Learning plus Data Leak (EL+DL) method achieves a MAP@5 of 0.496 on the test set, the highest score among all the methods the paper tries. The data-leak component assigns a test row the hotel cluster of a training row with identical values on the five leak features, falling back to the ensemble average when no match exists. The paper is explicit that this solution does not generalize and exists only because Expedia's data was flawed. It still concludes that the combined model performs best, and attributes the gain to a synergistic effect of combining weak learners with the leak lookup.
Load-bearing premise
The best model's performance rests on the premise that a test row's true hotel cluster can be recovered by finding a training row with identical values for user location country, orig destination distance, user location region, user location city, and hotel market. The paper itself admits this match is imperfect because hotel clusters can change seasonally, yet the EL+DL method still relies on this leak for its largest gains.
Editorial extensions
If this is right
- On the original Expedia test set, any future method that ignores the leak features would need to beat 0.496 to claim a genuine improvement over the paper's combined model.
- The leak-based lookup is a leaderboard-oriented tactic: it provides an upper bound for this specific test set, but will not transfer to new data where the leak is fixed.
- The paper's results imply that when benchmark data contains a confirmed leak, a hybrid of exact-match lookup and a classifier is a reliable way to boost scores, at the cost of honest generalization.
Reading between the lines
- Editorial inference: the reported 0.496 is largely a measure of how thoroughly the test set's rows can be found in the training data, not of recommendation quality; on leak-free data, the method should land near the ensemble's 0.450.
- A direct test of this would be to re-split the training data chronologically and retrain the ensemble on the earlier portion, evaluating on the later portion; the EL+DL score should collapse to roughly the ensemble-only score.
- The same leak-matching probe could be applied to other public datasets with announced leaks to quantify how much of a public leaderboard score is attributable to leakage rather than modeling.
- The paper implicitly shows that seasonal cluster reassignment sets a ceiling on the leak lookup's accuracy; a temporal model of cluster changes, if added, could push the leak match rate even higher.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a solution to the Expedia hotel recommendation Kaggle challenge. The authors describe preprocessing of a 37.6-million-row training set, feature engineering (including distance-matrix completion and PCA on latent features), and multiple classifiers (random forest, SGD, naive Bayes, XGBoost), which are combined into ensembles. Their headline result is a MAP@5 score of 0.496 on the Kaggle test set, achieved by an ensemble learning plus data leak (EL+DL) method that assigns test rows the hotel cluster of matching training rows using a set of leak features. The paper also reports a non-leak ensemble score of 0.450, a baseline of 0.069, and an upper-bound comparison to the competition's 0.60219 state-of-the-art score.
Significance. If taken at face value, the paper's central claim is that a hotel recommendation system can achieve MAP@5 = 0.496 on Expedia data. That claim, however, rests on a data leak that the authors themselves acknowledge: EL+DL directly copies training labels for test rows matching on leak features (Sections 3.3, 4.7, 4.8, 7). This is not a generalizable predictive model, and the paper's own text concedes that the method 'does not generalize well and exists only because the data provided by Expedia was flawed.' The non-leak results are competent but unremarkable: an ensemble of standard classifiers at 0.450, with no validation set, no error bars, and no temporal holdout. The paper is transparent about the leak, which is a point of honesty, but the headline result is a leaderboard artifact rather than a scientific contribution to recommender systems.
major comments (3)
- [Section 4.7/4.8 and Section 7] The reported best result, EL+DL with MAP@5 = 0.496, is produced by a lookup procedure that assigns test rows the hotel cluster of a training row with identical values for user location country, orig destination distance, user location region, user location city, and hotel market. Because the test ground-truth labels are the competition's held-out labels, this procedure effectively retrieves answers rather than learning a mapping. The paper's own conclusion in Section 7 states that 'Ensemble Learning with Data Leak model performs best giving a score of 0.496 on test data,' and Table 3 shows the entire gain from the non-leak ensemble (0.450) to EL+DL (0.496) is attributable to the leak. This makes the leak load-bearing for the paper's main claim, and a leaderboard trick cannot support a generalizable recommendation-system contribution.
- [Section 3.3 and Section 4.7] The leak-based method's validity depends on the unstated assumption that matching on the five leak features uniquely identifies the correct hotel cluster in the test ground truth, but the paper itself says this is not guaranteed because 'hotels can change cluster assignments, as hotel popularity and prices have seasonal characteristics.' No experiment quantifies the precision of exact matches, how partial matches are resolved, or how performance degrades as the time gap between matched train and test rows increases. Without such quantification, the 0.496 score cannot be separated from a stale-label artifact: the lookup may simply be retrieving clusters that were correct at training time but no longer at booking time.
- [Section 2.2 and Section 6] The evaluation reports only MAP@5 values on the Kaggle test set and offers no validation set, no cross-validation on a held-out temporal split, and no error bars. The preprocessing step that prunes non-bookers from the training set changes the training distribution relative to the test set, which contains only bookers, yet the effect of this mismatch is not analyzed. As a result, the differences among the models in Table 3 (e.g., RF at 0.421 vs. XGB at 0.432, or DL at 0.491 vs. EL+DL at 0.496) are not shown to be statistically significant, and the generalizable performance of the non-leak methods on this task is not established.
minor comments (5)
- [Abstract] The phrase 'given hundred distinct clusters' should be 'given one hundred distinct clusters.'
- [Section 3.2] The MAP@5 formula is imprecise: the sum over k of P(k) is not the standard average precision at rank 5, and the denominator |U| and the variable n are not clearly defined in relation to the test set size and the number of recommended clusters.
- [Section 4.9] The distance-matrix completion description leaves important details unspecified, such as the exact convergence criterion, the rank of the iterative SVD, and the number of iterations, which prevents reproduction of the reported 0.463 MAP@5 score.
- [Section 6] The sentence 'this method does not generalizes well' contains a grammatical error and should read 'does not generalize well.'
- [Section 6] The discussion of the 'state of the art result' compares against Kaggle leaderboard scores but does not report the authors' own leaderboard rank or whether the EL+DL submission was actually accepted by Kaggle's evaluation, which would clarify the practical status of the reported score.
Circularity Check
No circularity: Data Leak is a transparent lookup, not a circular derivation.
full rationale
The paper makes no pretense that the Data Leak solution is a learned, generalizable model. Section 4.7 describes it as a simple classification technique leveraging a leakage in the data, and Section 7 explicitly concedes that 'this method does not generalizes well and exists only because the data provided by Expedia was flawed.' The reported MAP@5 for DL and EL+DL is a Kaggle leaderboard score obtained by matching test rows to training rows on leak features and copying the training label. This is a lookup or transductive shortcut, not a circular derivation: the test labels are not used to construct the predictions, and the method's predictions are exactly what the leak description says (training labels of matched rows). No fitted parameter is renamed as a prediction, no theorem is imported from prior work by these authors, and there are no self-citations at all. The paper also acknowledges the seasonal instability of hotel clusters as a reason the leak-match is imperfect, further showing that the authors are not claiming the leak is a generalizable result. The conclusion that ensemble synergy is reaffirmed by the EL+DL score is a misattribution of the score gain to ensembling rather than to the leak, but that is a logical or validity concern, not circularity. The derivation chain is self-contained relative to the stated method: the individual models are standard classifiers, the leak is explicitly named, and the best score is transparently attributed to the leak. No circular step is exhibited.
Assumptions & free parameters
free parameters (3)
- PCA components =
20
- Distance matrix completion threshold =
100 km
- Hyperparameters for classifiers =
Not specified
assumptions (3)
- domain assumption Expedia's hotel cluster labels are the ground truth for user preference
- domain assumption MAP@5 is the appropriate evaluation metric for the recommendation task
- ad hoc to paper The data leak features (user location country, orig destination distance, user location region, user location city, hotel market) uniquely identify the same hotel cluster across train and test
Cite this review
Pith. "Pith review of Hotel Recommendation System." pith.science (2026). https://pith.science/paper/CZO47PPA
@misc{pith2026190807498,
author = {Pith},
title = {Pith review of: Hotel Recommendation System},
year = {2026},
howpublished = {\url{https://pith.science/paper/CZO47PPA}},
note = {Machine review of arXiv:1908.07498}
}
read the original abstract
One of the first things to do while planning a trip is to book a good place to stay. Booking a hotel online can be an overwhelming task with thousands of hotels to choose from, for every destination. Motivated by the importance of these situations, we decided to work on the task of recommending hotels to users. We used Expedia's hotel recommendation dataset, which has a variety of features that helped us achieve a deep understanding of the process that makes a user choose certain hotels over others. The aim of this hotel recommendation task is to predict and recommend five hotel clusters to a user that he/she is more likely to book given hundred distinct clusters.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
G. Huming and L. Weili. A hotel recommendation system based on collaborative filtering and rankboost algorithm. In 2010 Second International Conference on Multimedia and Information Technology , April 2010
work page 2010
-
[3]
A. Levi, O. Mokryn, C. Diot, and N. Taft. Finding a needle in a haystack of reviews: Cold start context-based hotel recommender system. In Proceedings of the Sixth ACM Conference on Recommender Systems , RecSys '12, New York, NY, USA, 2012. ACM
work page 2012
-
[4]
X. Liu, B. Xu, Y. Zhang, Q. Yan, L. Pang, Q. Li, H. Sun, and B. Wang. Combination of diverse ranking models for personalized expedia hotel searches. ArXiv , abs/1311.7679, 2013
work page Pith review arXiv 2013
-
[5]
Saga , Yoshihiro Hayashi , and H
R. Saga , Yoshihiro Hayashi , and H. Tsuji . Hotel recommender system based on user's preference transition. In 2008 IEEE International Conference on Systems, Man and Cybernetics , pages 2437--2442, Oct 2008
work page 2008
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.