{"id":"9f7a7857-a73a-4898-b78f-48f6c354e2ce","arxiv_id":"2501.03072","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new OpenTable dataset with 19,536 ratings, four criteria dimensions, and known user-ID cleaning issues is released for multi-criteria recommender research.","lead":"This paper releases a new benchmark dataset of restaurant ratings crawled from OpenTable.com, containing 19,536 ratings with one overall score and four criteria scores. It gives recommender systems researchers a new multi-criteria dataset, but the dataset's user identification has known reliability issues.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 2.4's user-ID rules are the central risk: merging all 'Unknown user' reviews into one ID and then deleting duplicate pairs can create pseudo-users and discard real ratings, so the benchmark's user dimension needs quantitative validation.","rationale":"The central claim is a data-release benchmark claim, so the relevant question is whether the released user-item rating matrix can support meaningful multi-criteria recommendation experiments. The reader identified the user-ID construction as the weakest assumption. My reading of Section 2.4 confirms this, and adds the point that the duplicate-removal rule is not an independent cleaning step: it is a direct artifact of merging all 'Unknown user' reviews. Therefore the cleaned file may be internally inconsistent even if the raw crawl was accurate, because spurious duplicates caused by the merge are resolved by deleting one rating rather than by separating identities. This is a concrete, falsifiable concern: its impact depends entirely on how many anonymous rows exist and how many duplicate conflicting pairs exist. The author's transparency is a point in favor, and the dataset appears real and mostly plausible. Given no quantitative validation is supplied, a conditional verdict is appropriate. I therefore keep the reader's CONDITIONAL rather than upgrading to ACCEPT or downgrading to REJECT, pending the audit described in concrete_test. I agree with the reader's weakest assumption.","tokens_in":2340,"tokens_out":4494,"duration_ms":45332,"concrete_test":"Download opentable_cleaned.csv from the Kaggle or IEEE DataPort repositories. Compute (a) the number of rows, distinct items, and distinct pairs whose username is 'Unknown user' (or whose assigned user ID is the merged anonymous ID); (b) the number of duplicate pairs in the uncleaned file with conflicting ratings; (c) the per-user mean and median after excluding the merged anonymous ID. If the anonymous ID accounts for more than a few percent of ratings or if its removal changes the mean from 14.9 by more than, say, 20%, the benchmark's user dimension is materially distorted and the benchmark claim should be weakened or the data re-released with anonymous reviews separated; if the anonymous fraction is negligible, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4 contains the load-bearing step for the benchmark claim. The released data identifies users by treating username+city as unique and assigning every 'Unknown user' review to a single ID, then the cleaned file removes duplicate pairs keeping the last. If any material fraction of OpenTable reviews are anonymous, the 1,309-user count and all per-user statistics (mean 14.9, median 3) are built on a pseudo-user whose history mixes many unrelated diners. Worse, the dedup rule follows from the merge: two different anonymous diners reviewing the same restaurant become the same pair with different ratings, and 'keeping the last' deletes one real rating instead of fixing the identity error. Since multi-criteria recommendation algorithms rely on user histories to learn preferences, this directly affects whether the dataset supports the stated benchmark use. The paper is transparent about the scheme, but it provides no counts for the 'Unknown user' rows or duplicate conflicting pairs, so the magnitude of the distortion is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper releases a crawled OpenTable.com dataset containing 19,536 ratings by 1,309 users on 91 restaurants. Each rating includes an overall score plus criteria scores for food, service, ambience, and value, all on a 1–5 scale. The authors report basic statistics (83.6% sparsity; mean 14.9 and median 3 ratings per user; mean 214 and median 250 ratings per restaurant) and present the dataset as a benchmark for multi-criteria recommender systems. Section 2.4 describes the user-identification scheme: username plus city is treated as a unique user, and all reviews with the default anonymous username are assigned to a single UserID. The authors also provide a cleaned file in which duplicate pairs are removed by keeping the last entry. The paper is a data descriptor with no algorithmic derivation or fitting.","tokens_in":2543,"tokens_out":2769,"duration_ms":27952,"significance":"If the data are valid, the release is useful for multi-criteria recommender-system research because few public datasets provide criteria-level ratings for restaurants. The strengths are that the data are publicly hosted on Kaggle and IEEE DataPort, that both raw and cleaned files are offered, and that the authors transparently disclose the user-identification heuristic. However, the benchmark claim rests on an unvalidated assumption about user identity: merging all anonymous reviews into one ID and then deduplicating pairs can distort user histories, which are central to recommender evaluation. No quantitative evidence is given for the magnitude of this distortion, so the dataset's suitability as a benchmark is not yet established.","major_comments":[{"comment":"The user identification rule is the load-bearing step for the benchmark claim. Treating every 'Unknown user' review as the same UserID creates a pseudo-user whose history mixes unrelated diners, and the paper gives no count of how many of the 19,536 ratings come from this pseudo-user or how many duplicate pairs with conflicting ratings exist. Without these counts, the reported 1,309-user total and the per-user statistics (mean 14.9, median 3) are not validated as descriptions of real user histories. Please add these statistics and, if the fraction of anonymous ratings is material, re-release the data with a separate handling of anonymous reviews or clearly mark the pseudo-user as not usable for user-modeling experiments.","section":"§2.4"},{"comment":"The deduplication rule for the cleaned file (keep the last entry per pair) is not a neutral cleaning step under the merged-identity rule: two different anonymous diners who visited the same restaurant become a single pair with different multi-criteria ratings, and deleting all but the last discards one real rating. The paper should report the number of removed duplicate rows, the number of conflicting rating sets among them, and show that the rating distributions in Figure 2 are stable under this deletion; if conflicts are frequent, the cleaned file should not be presented as the default benchmark file.","section":"§2.4"},{"comment":"The benchmark claim for multi-criteria recommender systems depends on being able to learn per-user preferences from histories, yet with median 3 ratings per user and a possibly large merged anonymous user, the effective number of users with usable histories may be much smaller than 1,309. Please provide the per-user rating-count distribution after removing the 'Unknown user' ID, and specify whether the intended benchmark evaluation uses all users or a filtered set with a minimum history length.","section":"§2.3"}],"minor_comments":[{"comment":"Typo: 'preferencs' should be 'preferences' in the first paragraph.","section":"§1"},{"comment":"Table 1 is referenced in the text, but the table content does not actually appear in the manuscript; please include the multi-criteria rating example or remove the reference.","section":"§1"},{"comment":"The criteria names are inconsistent: the text says 'satisfaction of service and ambience' while Figure 2 labels them 'Service' and 'Ambience'; align the names across the text and figures.","section":"§2.2"},{"comment":"The raw and cleaned files are mentioned but not described in enough detail to reproduce the cleaning; a short data dictionary and a count of rows removed during deduplication would improve the data descriptor.","section":"§2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a data descriptor, and the main risk is not circularity but inadequate validation of the user dimension. The self-citations to prior work using the same dataset are transparent and do not constitute a problem. If the authors provide the requested quantitative diagnostics about anonymous ratings, duplicate conflicts, and post-cleanup user histories, the paper could be acceptable for publication; without those diagnostics, the benchmark claim is not sufficiently supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does what it says: it releases a new OpenTable dataset with overall and four-criteria ratings, publicly hosted on Kaggle and IEEE DataPort. That artifact is genuinely new, and the paper is transparent about how it was collected and cleaned. Credit where due: the author discloses the limitations of the user-ID scheme in Section 2.4, which is more upfront than most data papers. The stats (19,536 ratings, 1,309 users, 91 restaurants, 83.6% sparsity) are plausible and straightforward.\n\nThe soft spot is the user-identification procedure, and it is load-bearing. The paper assigns one ID to every “Unknown user” review, then removes duplicate pairs keeping only the last. That means the 1,309 users include a single pseudo-user who may be an aggregate of many distinct anonymous diners. Worse, the dedup rule follows from the merge: two different anonymous users rating the same restaurant become the same pair, and “keeping the last” deletes one real rating rather than fixing the identity error. The paper provides no counts for the “Unknown user” rows or the conflicting duplicate pairs, so the magnitude of the distortion is unquantified. Since multi-criteria recommendation algorithms learn from per-user histories, the benchmark claim in the abstract is stronger than the evidence currently supports.\n\nThat said, the paper is not hopeless. If the author were to report the share of anonymous reviews and duplicate conflicts, or release a version that discards anonymous entries entirely, the dataset could be usable. The cleanest fix is to reposition it as a multi-criteria rating prediction dataset rather than a user-centric benchmark, unless the validation shows the identity problem is small.\n\nI would send this to peer review with a request for revision, not desk reject it. A serious referee can ask for the necessary numbers or a revised dataset. The paper deserves engagement because the artifact is new and the author is honest about the methodology. My own verdict is conditional: the central claim holds only if the user-ID distortion is shown to be small.","headline":"An honest but flawed OpenTable multi-criteria rating dataset; the user-identification scheme in Section 2.4 needs quantitative validation before the benchmark claim can be taken at face value.","tokens_in":3028,"tokens_out":1637,"would_cite":false,"duration_ms":38919,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper releases an OpenTable dataset with 19,536 multi-criteria restaurant ratings for recommender-system benchmarks.","keywords":["recommender systems","multi-criteria recommendations","OpenTable dataset","benchmark dataset","restaurant ratings","multi-criteria ratings","data release","collaborative filtering"],"falsifier":"A direct check would be to take a random sample of the 'Unknown user' entries in the cleaned file and inspect whether their ratings on the same restaurant vary widely or appear in bursts that suggest multiple different people; if they do, the identity mapping has merged distinct users and the per-user statistics are unreliable. Alternatively, re-crawl a subset of the same restaurants and see whether the public review pages display a more fine-grained user identifier that contradicts the username-plus-city mapping.","tokens_in":2109,"feed_emoji":"🍽️","tokens_out":5290,"duration_ms":47283,"temperature":0.7,"pith_summary":"The paper's central claim is that the crawled OpenTable data set can serve as a benchmark for multi-criteria recommender systems. It contains 19,536 real restaurant ratings by 1,309 users on 91 restaurants, each with an overall rating and ratings on four criteria: food, service, ambience, and value. The data set is meant to fill a gap: although OpenTable.com is a real-world example of multi-criteria ratings, no open data from it had been available for research. A careful reader would care because multi-criteria recommenders need real data with correlated aspect ratings, and this release provides that in a small, manageable package with known sparsity.","feed_headline":"OpenTable dataset gives 19,536 multi-criteria ratings","feed_subtitle":"Overall plus food, service, ambience, and value scores let researchers test multi-criteria recommenders.","key_machinery":"The central object is the dataset itself, structured as a table of pairs with five rating columns: overall, food, service, ambience, and value. It is carried by the user-identity heuristic that turns nicknames into stable user IDs, and by the cleaned file that resolves duplicate entries. The dataset's value is that it attaches four aspect ratings to an overall rating on real restaurant reviews, which is exactly the input shape multi-criteria recommendation algorithms need.","core_discovery":"The author's own claim is that the released data set is a benchmark for multi-criteria recommendations. The data set includes 19,536 ratings from 1,309 users on 91 restaurants, with each rating consisting of an overall 1-5 score plus four criterion scores for food quality, service satisfaction, ambience, and overall value. The author reports a rating sparsity of 83.6%, an average of 14.9 ratings per user, and an average of 214 ratings per restaurant. Because the crawl records nicknames rather than user IDs, the author maps users by the combination of username and city, assigns a single ID to all 'Unknown user' entries, and also releases a cleaned file that keeps only the last rating for each unique pair.","pith_inferences":["A user of the data should note that the username-plus-city identity heuristic could conflate two people who share a nickname in the same city, or split one person who reviews from different cities, so per-user histories should be treated as approximate.","Giving every 'Unknown user' entry the same ID could seriously distort the user distribution, since anonymous reviews may come from many distinct diners; analyses that rely on per-user history should run robustness checks excluding that ID.","The cleaned file's keep-last-entry rule biases toward the most recent review per user and restaurant, which may matter for studying rating dynamics or preference drift.","A natural extension is to treat the four criterion ratings as predictor variables and examine how well they explain the overall rating, connecting this benchmark to earlier work on multi-criteria ranking."],"forward_implications":["Researchers can use the overall rating column alone for traditional single-rating recommender benchmarks, or use all five columns for multi-criteria recommenders.","The four criterion ratings allow direct comparison of algorithms that predict the overall rating from aspect ratings against algorithms that use only the overall rating.","The reported sparsity, user, and restaurant statistics give a clear baseline expectation for how dense or sparse the benchmark is, so new methods can be evaluated against the same data geometry.","The cleaned file provides a standard rating matrix without duplicate pairs, making it directly usable in standard recommender-system libraries.","Releasing the dataset on public platforms lets other groups replicate the author's prior experiments on this data."],"supporting_citations":[{"why":"Defines multi-criteria recommender systems and supplies the research context that motivates a benchmark dataset.","marker":"[1]"},{"why":"Prior work by the author that already used this OpenTable data to demonstrate a multi-criteria ranking framework.","marker":"[2]"},{"why":"Prior work connecting multi-criteria decision making to recommender systems, cited as evidence of the data's utility.","marker":"[3]"},{"why":"Prior work on hybrid multi-criteria preference ranking, showing the dataset supports ranking experiments.","marker":"[4]"},{"why":"The author's open-source library used to evaluate multi-criteria recommendation algorithms on this data set.","marker":"[5]"}],"fun_headline_variants":["19,536 multi-criteria ratings to test your recommenders","Multi-criteria recommender benchmark: OpenTable dataset","OpenTable data: 19,536 ratings with 5 criteria each","New benchmark for multi-criteria RS: 19,536 ratings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole dataset's usefulness rests on the assumption that treating username plus city as a unique user, and merging every 'Unknown user' into one identity, preserves real user histories closely enough that the rating data still reflects genuine preferences.","fun_headline_variants_meta":{"raw":{"variants":["19,536 multi-criteria ratings to test your recommenders","Multi-criteria recommender benchmark: OpenTable dataset","OpenTable data: 19,536 ratings with 5 criteria each","New benchmark for multi-criteria RS: 19,536 ratings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3435,"prompt_tokens":793,"completion_tokens":2642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":2570}},"tokens_in":409,"tokens_out":2642,"duration_ms":19797,"temperature":1.0,"reasoning_tokens":2570,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:59:48.835394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check would be to take a random sample of the 'Unknown user' entries in the cleaned file and inspect whether their ratings on the same restaurant vary widely or appear in bursts that suggest multiple different people; if they do, the identity mapping has merged distinct users and the per-user statistics are unreliable. Alternatively, re-crawl a subset of the same restaurants and see whether the public review pages display a more fine-grained user identifier that contradicts the username-plus-city mapping.","supporting_citations":[{"cited_title":"In: Recommender systems handbook, pp","cited_arxiv_id":null,"evidence_quote":"Defines multi-criteria recommender systems and supplies the research context that motivates a benchmark dataset."},{"cited_title":"IEEE Access 10, 90715--90725 (2022)","cited_arxiv_id":null,"evidence_quote":"Prior work by the author that already used this OpenTable data to demonstrate a multi-criteria ranking framework."},{"cited_title":"In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces","cited_arxiv_id":null,"evidence_quote":"Prior work connecting multi-criteria decision making to recommender systems, cited as evidence of the data's utility."},{"cited_title":"Hybrid Multi-Criteria Preference Ranking by Subsorting","cited_arxiv_id":"2306.11233","evidence_quote":"Prior work on hybrid multi-criteria preference ranking, showing the dataset supports ranking experiments."},{"cited_title":"In: 2024 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) (2024)","cited_arxiv_id":null,"evidence_quote":"The author's open-source library used to evaluate multi-criteria recommendation algorithms on this data set."}],"review_version":1}