{"id":"d6abe61f-d686-45ae-b50f-62066365bdc4","arxiv_id":"1908.07498","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":1.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of standard classifiers plus a data-leakage lookup achieves MAP@5 of 0.496 on the Expedia hotel recommendation test set.","lead":"This paper reports a machine learning pipeline for recommending hotel clusters using Expedia's Kaggle dataset. Its best score comes from exploiting a known data leak, not from a generalizable model.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best result depends on unvalidated leak-match precision under seasonal cluster changes; a temporal holdout would settle whether the 0.496 score is a generalizable result or a stale-label artifact.","rationale":"The paper is transparent about the leak and even flags that the leak method does not generalize, which is creditworthy. But transparency does not repair the central claim: the reported best score is achieved by a lookup keyed to features whose label stability is explicitly time-dependent. The training-set MAP in Table 3 is not evidence because lookup on training rows is circular. The Kaggle test score is a single point estimate with no error bars, no code, and no analysis of match accuracy. The most decisive missing piece is a temporal holdout: the Kaggle split is already time-ordered, so a within-training time split would let us measure whether exact matches retain their labels under the seasonal drift the paper acknowledges. If they do not, the 0.496 result is a non-generalizable artifact and the paper's central claim fails. This is a correctness risk, not just a novelty objection, and it is the same assumption the reader's weakest_assumption identifies, so agreement is complete and the reject verdict stands.","tokens_in":8293,"tokens_out":4593,"duration_ms":50233,"concrete_test":"Construct a temporal holdout from the training set: train on rows with check-in dates before a cutoff (e.g., the first nine months) and test on rows with check-in dates after the cutoff, using only the pre-cutoff rows in the DL lookup. Compute MAP@5 for XGB, EL, DL, and EL+DL on this holdout, and also compute the exact-match precision of DL as the fraction of matched test rows whose matched training label equals the holdout label. If exact-match precision is substantially below the implied precision on the Kaggle test set, or if EL+DL's margin over EL collapses, the central result is not a robust recommendation method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (MAP@5 = 0.496, Section 7) is produced by EL+DL, where DL assigns a test row the hotel cluster of a training row with the same values for user location country, orig destination distance, user location region, user location city, and hotel market (Sections 3.3 and 4.7). The load-bearing assumption is that identical leak-feature values imply the same hotel cluster in the test ground truth. The paper itself says this is not guaranteed, because 'hotels can change cluster assignments, as hotel popularity and prices have seasonal characteristics' (Section 3.3), and later concedes the method 'does not generalizes well and exists only because the data provided by Expedia was flawed' (Section 7). Yet no experiment quantifies the precision of exact matches, how partial matches are resolved, or how performance degrades as the time gap between matched train and test rows grows. Without that quantification, the 0.496 score cannot be separated from a stale-label artifact: the lookup may simply be retrieving clusters that were correct at training time but no longer at booking time. The paper therefore does not establish a generalizable recommendation system; it establishes a leaderboard trick.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a solution to the Expedia hotel recommendation Kaggle challenge. The authors describe preprocessing of a 37.6-million-row training set, feature engineering (including distance-matrix completion and PCA on latent features), and multiple classifiers (random forest, SGD, naive Bayes, XGBoost), which are combined into ensembles. Their headline result is a MAP@5 score of 0.496 on the Kaggle test set, achieved by an ensemble learning plus data leak (EL+DL) method that assigns test rows the hotel cluster of matching training rows using a set of leak features. The paper also reports a non-leak ensemble score of 0.450, a baseline of 0.069, and an upper-bound comparison to the competition's 0.60219 state-of-the-art score.","tokens_in":8690,"tokens_out":2655,"duration_ms":28921,"significance":"If taken at face value, the paper's central claim is that a hotel recommendation system can achieve MAP@5 = 0.496 on Expedia data. That claim, however, rests on a data leak that the authors themselves acknowledge: EL+DL directly copies training labels for test rows matching on leak features (Sections 3.3, 4.7, 4.8, 7). This is not a generalizable predictive model, and the paper's own text concedes that the method 'does not generalize well and exists only because the data provided by Expedia was flawed.' The non-leak results are competent but unremarkable: an ensemble of standard classifiers at 0.450, with no validation set, no error bars, and no temporal holdout. The paper is transparent about the leak, which is a point of honesty, but the headline result is a leaderboard artifact rather than a scientific contribution to recommender systems.","major_comments":[{"comment":"The reported best result, EL+DL with MAP@5 = 0.496, is produced by a lookup procedure that assigns test rows the hotel cluster of a training row with identical values for user location country, orig destination distance, user location region, user location city, and hotel market. Because the test ground-truth labels are the competition's held-out labels, this procedure effectively retrieves answers rather than learning a mapping. The paper's own conclusion in Section 7 states that 'Ensemble Learning with Data Leak model performs best giving a score of 0.496 on test data,' and Table 3 shows the entire gain from the non-leak ensemble (0.450) to EL+DL (0.496) is attributable to the leak. This makes the leak load-bearing for the paper's main claim, and a leaderboard trick cannot support a generalizable recommendation-system contribution.","section":"Section 4.7/4.8 and Section 7"},{"comment":"The leak-based method's validity depends on the unstated assumption that matching on the five leak features uniquely identifies the correct hotel cluster in the test ground truth, but the paper itself says this is not guaranteed because 'hotels can change cluster assignments, as hotel popularity and prices have seasonal characteristics.' No experiment quantifies the precision of exact matches, how partial matches are resolved, or how performance degrades as the time gap between matched train and test rows increases. Without such quantification, the 0.496 score cannot be separated from a stale-label artifact: the lookup may simply be retrieving clusters that were correct at training time but no longer at booking time.","section":"Section 3.3 and Section 4.7"},{"comment":"The evaluation reports only MAP@5 values on the Kaggle test set and offers no validation set, no cross-validation on a held-out temporal split, and no error bars. The preprocessing step that prunes non-bookers from the training set changes the training distribution relative to the test set, which contains only bookers, yet the effect of this mismatch is not analyzed. As a result, the differences among the models in Table 3 (e.g., RF at 0.421 vs. XGB at 0.432, or DL at 0.491 vs. EL+DL at 0.496) are not shown to be statistically significant, and the generalizable performance of the non-leak methods on this task is not established.","section":"Section 2.2 and Section 6"}],"minor_comments":[{"comment":"The phrase 'given hundred distinct clusters' should be 'given one hundred distinct clusters.'","section":"Abstract"},{"comment":"The MAP@5 formula is imprecise: the sum over k of P(k) is not the standard average precision at rank 5, and the denominator |U| and the variable n are not clearly defined in relation to the test set size and the number of recommended clusters.","section":"Section 3.2"},{"comment":"The distance-matrix completion description leaves important details unspecified, such as the exact convergence criterion, the rank of the iterative SVD, and the number of iterations, which prevents reproduction of the reported 0.463 MAP@5 score.","section":"Section 4.9"},{"comment":"The sentence 'this method does not generalizes well' contains a grammatical error and should read 'does not generalize well.'","section":"Section 6"},{"comment":"The discussion of the 'state of the art result' compares against Kaggle leaderboard scores but does not report the authors' own leaderboard rank or whether the EL+DL submission was actually accepted by Kaggle's evaluation, which would clarify the practical status of the reported score.","section":"Section 6"}],"recommendation":"reject","confidential_remarks":"The paper's core contribution is undermined by the authors' own admission that the best score comes from a confirmed data leak. This is not a fixable presentation issue: the central claim would need to be redefined entirely to remove the leak-based result, and the remaining contribution is a fairly standard ensemble application without rigorous evaluation. The paper also appears to be a course-project write-up (author IDs and the report style) rather than a research paper, and it provides no code or reproducibility artifacts. I recommend rejection, though the authors' transparency about the leak is noteworthy and should be acknowledged in the decision letter."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this one is a course-style Kaggle write-up, not a research paper. The headline MAP@5 of 0.496 comes from the 'Ensemble Learning plus Data Leak' model, and the data leak is exactly what it sounds like: for test rows that match a training row on five leak features, the method copies the training label. The paper is unusually honest about this: Section 3.3 explains the leak, Section 7 admits it does not generalize and exists only because Expedia's data was flawed. Credit is due for that transparency, and for a clean comparison of six standard classifiers on a large dataset, with sensible exploratory analysis.\n\nThe soft spots are structural. There is no code, no error bars, no temporal holdout. The leak's precision is never quantified; the paper notes hotel clusters can change seasonally, which means the lookup could be retrieving stale labels, but no experiment measures how often that happens. So the central number overstates generalizable performance. The non-leak ensemble scores 0.450, well below the competition winner's 0.602, and the leak's contribution is a lookup, not a learned mapping. Section 7 itself concedes this. On the plus side, the distance-matrix completion is credited to the first-place blog, and the leak trick is credited to another participant. That is good scholarly hygiene.\n\nIs there anything new? No. The models are off-the-shelf, the feature engineering is standard, and the results are a Kaggle leaderboard entry. The paper would not change how recommendation systems are built. Its value is as a cautionary example of how a test-set leak can inflate scores and an honest account of why leaderboard tricks are not methods.\n\nWho is it for? Students learning how to run a Kaggle pipeline, and people teaching data-leakage detection. It deserves a blog post or an appendix, not a serious referee slot in a research venue. If the authors wanted to turn it into something reviewable, they would need to add a temporal validation study that estimates the leak's hit rate and shows whether the non-leak ensemble generalizes beyond the competition window. As is, I'd desk-reject it.","headline":"Honest Kaggle write-up whose headline number is a known data-leak lookup, not a reusable method; clear about its own limitation but the central claim doesn't generalize.","tokens_in":9072,"tokens_out":2295,"would_cite":false,"duration_ms":21824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's best hotel-recommendation model exploits a data leak to reach 0.496 MAP@5.","keywords":["hotel recommendation","Expedia dataset","data leakage","MAP@5","ensemble learning","random forest","gradient boosting","distance matrix completion"],"falsifier":"Take the test set and mark each row as matched or unmatched based on whether an exact match on the five leak features exists in the training set. Compute MAP@5 separately for the matched and unmatched subsets under the EL+DL rule: if the matched subset's score is substantially higher than the unmatched subset's, the reported 0.496 is driven by the leak. Alternatively, retrain the ensemble after removing or permuting the five leak features; if the score falls to approximately 0.450, the leak is the source of the gain.","tokens_in":8105,"feed_emoji":"🏨","tokens_out":8593,"duration_ms":69920,"temperature":0.7,"pith_summary":"The paper claims that the best way to maximize MAP@5 on Expedia's hotel recommendation test set is to combine an ensemble of four standard classifiers with a direct lookup that exploits a known data leak. The leak lets a test row inherit the hotel cluster of a training row that matches on five features: user location country, orig destination distance, user location region, user location city, and hotel market. The paper reports that this combined approach scores 0.496 on the competition test set, beating every individual model. A careful reader should care because the result indicates that this particular leaderboard rewarded detecting leakage rather than building a generalizable recommender.","feed_headline":"Best hotel picks came from a data leak, not a better model","feed_subtitle":"A 0.496 MAP@5 was earned by matching test rows to training data, not by generalizing.","key_machinery":"The key machinery is the data-leak lookup: a matching procedure that, for each test row, searches the training set for a row with the same values on user location country, orig destination distance, user location region, user location city, and hotel market. When an exact match is found, the test row is assigned the matched row's hotel cluster; partial matches are also searched, and rows with no match fall back to the most frequent clusters. This lookup is what carries the argument because the pure leak solution alone scores 0.491, a larger gain over the ensemble's 0.450 than any other single improvement. The second machinery is the probability-averaged ensemble of random forest, gradient boosting, SGD, and naive Bayes, which supplies predictions for the rows the lookup cannot resolve.","core_discovery":"The central discovery is that the Ensemble Learning plus Data Leak (EL+DL) method achieves a MAP@5 of 0.496 on the test set, the highest score among all the methods the paper tries. The data-leak component assigns a test row the hotel cluster of a training row with identical values on the five leak features, falling back to the ensemble average when no match exists. The paper is explicit that this solution does not generalize and exists only because Expedia's data was flawed. It still concludes that the combined model performs best, and attributes the gain to a synergistic effect of combining weak learners with the leak lookup.","pith_inferences":["Editorial inference: the reported 0.496 is largely a measure of how thoroughly the test set's rows can be found in the training data, not of recommendation quality; on leak-free data, the method should land near the ensemble's 0.450.","A direct test of this would be to re-split the training data chronologically and retrain the ensemble on the earlier portion, evaluating on the later portion; the EL+DL score should collapse to roughly the ensemble-only score.","The same leak-matching probe could be applied to other public datasets with announced leaks to quantify how much of a public leaderboard score is attributable to leakage rather than modeling.","The paper implicitly shows that seasonal cluster reassignment sets a ceiling on the leak lookup's accuracy; a temporal model of cluster changes, if added, could push the leak match rate even higher."],"forward_implications":["On the original Expedia test set, any future method that ignores the leak features would need to beat 0.496 to claim a genuine improvement over the paper's combined model.","The leak-based lookup is a leaderboard-oriented tactic: it provides an upper bound for this specific test set, but will not transfer to new data where the leak is fixed.","The paper's results imply that when benchmark data contains a confirmed leak, a hybrid of exact-match lookup and a classifier is a reliable way to boost scores, at the cost of honest generalization."],"supporting_citations":[],"fun_headline_variants":["Top hotel scores came from data leak, not model smarts","Hotel recommendation 'win' was just a data copy","Expedia data flaw boosts hotel model score to 0.496","No generalization: hotel AI match training rows","Hotel AI wins by memorizing, not learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The best model's performance rests on the premise that a test row's true hotel cluster can be recovered by finding a training row with identical values for user location country, orig destination distance, user location region, user location city, and hotel market. The paper itself admits this match is imperfect because hotel clusters can change seasonally, yet the EL+DL method still relies on this leak for its largest gains.","fun_headline_variants_meta":{"raw":{"variants":["Top hotel scores came from data leak, not model smarts","Hotel recommendation 'win' was just a data copy","Expedia data flaw boosts hotel model score to 0.496","No generalization: hotel AI match training rows","Hotel AI wins by memorizing, not learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3435,"prompt_tokens":760,"completion_tokens":2675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":376,"completion_tokens_details":{"reasoning_tokens":2597}},"tokens_in":376,"tokens_out":2675,"duration_ms":19019,"temperature":1.0,"reasoning_tokens":2597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:17:14.272024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the test set and mark each row as matched or unmatched based on whether an exact match on the five leak features exists in the training set. Compute MAP@5 separately for the matched and unmatched subsets under the EL+DL rule: if the matched subset's score is substantially higher than the unmatched subset's, the reported 0.496 is driven by the leak. Alternatively, retrain the ensemble after removing or permuting the five leak features; if the score falls to approximately 0.450, the leak is the source of the gain.","supporting_citations":[],"review_version":1}