REVIEW 3 major objections 4 minor 7 references
OpenTable data with multi-criteria ratings
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper releases an OpenTable dataset with 19,536 multi-criteria restaurant ratings for recommender-system benchmarks.
desk verdict An honest but flawed OpenTable multi-criteria rating dataset; the user-identification scheme in Section 2.4 needs quantitative validation before the benchmark claim can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the dataset itself, structured as a table of <user, item> pairs with five rating columns: overall, food, service, ambience, and value. It is carried by the user-identity heuristic that turns nicknames into stable user IDs, and by the cleaned file that resolves duplicate <user, item> entries. The dataset's value is that it attaches four aspect ratings to an overall rating on real restaurant reviews, which is exactly the input shape multi-criteria recommendation algorithms need.
What would settle it
A direct check would be to take a random sample of the 'Unknown user' entries in the cleaned file and inspect whether their ratings on the same restaurant vary widely or appear in bursts that suggest multiple different people; if they do, the identity mapping has merged distinct users and the per-user statistics are unreliable. Alternatively, re-crawl a subset of the same restaurants and see whether the public review pages display a more fine-grained user identifier that contradicts the username-plus-city mapping.
Extended reading notes
Core claim
The author's own claim is that the released data set is a benchmark for multi-criteria recommendations. The data set includes 19,536 ratings from 1,309 users on 91 restaurants, with each rating consisting of an overall 1-5 score plus four criterion scores for food quality, service satisfaction, ambience, and overall value. The author reports a rating sparsity of 83.6%, an average of 14.9 ratings per user, and an average of 214 ratings per restaurant. Because the crawl records nicknames rather than user IDs, the author maps users by the combination of username and city, assigns a single ID to all 'Unknown user' entries, and also releases a cleaned file that keeps only the last rating for each unique <user, item> pair.
Load-bearing premise
The whole dataset's usefulness rests on the assumption that treating username plus city as a unique user, and merging every 'Unknown user' into one identity, preserves real user histories closely enough that the rating data still reflects genuine preferences.
Editorial extensions
If this is right
- Researchers can use the overall rating column alone for traditional single-rating recommender benchmarks, or use all five columns for multi-criteria recommenders.
- The four criterion ratings allow direct comparison of algorithms that predict the overall rating from aspect ratings against algorithms that use only the overall rating.
- The reported sparsity, user, and restaurant statistics give a clear baseline expectation for how dense or sparse the benchmark is, so new methods can be evaluated against the same data geometry.
- The cleaned file provides a standard <user, item> rating matrix without duplicate pairs, making it directly usable in standard recommender-system libraries.
- Releasing the dataset on public platforms lets other groups replicate the author's prior experiments on this data.
Reading between the lines
- A user of the data should note that the username-plus-city identity heuristic could conflate two people who share a nickname in the same city, or split one person who reviews from different cities, so per-user histories should be treated as approximate.
- Giving every 'Unknown user' entry the same ID could seriously distort the user distribution, since anonymous reviews may come from many distinct diners; analyses that rely on per-user history should run robustness checks excluding that ID.
- The cleaned file's keep-last-entry rule biases toward the most recent review per user and restaurant, which may matter for studying rating dynamics or preference drift.
- A natural extension is to treat the four criterion ratings as predictor variables and examine how well they explain the overall rating, connecting this benchmark to earlier work on multi-criteria ranking.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper releases a crawled OpenTable.com dataset containing 19,536 ratings by 1,309 users on 91 restaurants. Each rating includes an overall score plus criteria scores for food, service, ambience, and value, all on a 1–5 scale. The authors report basic statistics (83.6% sparsity; mean 14.9 and median 3 ratings per user; mean 214 and median 250 ratings per restaurant) and present the dataset as a benchmark for multi-criteria recommender systems. Section 2.4 describes the user-identification scheme: username plus city is treated as a unique user, and all reviews with the default anonymous username are assigned to a single UserID. The authors also provide a cleaned file in which duplicate <user, item> pairs are removed by keeping the last entry. The paper is a data descriptor with no algorithmic derivation or fitting.
Significance. If the data are valid, the release is useful for multi-criteria recommender-system research because few public datasets provide criteria-level ratings for restaurants. The strengths are that the data are publicly hosted on Kaggle and IEEE DataPort, that both raw and cleaned files are offered, and that the authors transparently disclose the user-identification heuristic. However, the benchmark claim rests on an unvalidated assumption about user identity: merging all anonymous reviews into one ID and then deduplicating <user, item> pairs can distort user histories, which are central to recommender evaluation. No quantitative evidence is given for the magnitude of this distortion, so the dataset's suitability as a benchmark is not yet established.
major comments (3)
- [§2.4] The user identification rule is the load-bearing step for the benchmark claim. Treating every 'Unknown user' review as the same UserID creates a pseudo-user whose history mixes unrelated diners, and the paper gives no count of how many of the 19,536 ratings come from this pseudo-user or how many duplicate <user,item> pairs with conflicting ratings exist. Without these counts, the reported 1,309-user total and the per-user statistics (mean 14.9, median 3) are not validated as descriptions of real user histories. Please add these statistics and, if the fraction of anonymous ratings is material, re-release the data with a separate handling of anonymous reviews or clearly mark the pseudo-user as not usable for user-modeling experiments.
- [§2.4] The deduplication rule for the cleaned file (keep the last entry per <user,item> pair) is not a neutral cleaning step under the merged-identity rule: two different anonymous diners who visited the same restaurant become a single <user,item> pair with different multi-criteria ratings, and deleting all but the last discards one real rating. The paper should report the number of removed duplicate rows, the number of conflicting rating sets among them, and show that the rating distributions in Figure 2 are stable under this deletion; if conflicts are frequent, the cleaned file should not be presented as the default benchmark file.
- [§2.3] The benchmark claim for multi-criteria recommender systems depends on being able to learn per-user preferences from histories, yet with median 3 ratings per user and a possibly large merged anonymous user, the effective number of users with usable histories may be much smaller than 1,309. Please provide the per-user rating-count distribution after removing the 'Unknown user' ID, and specify whether the intended benchmark evaluation uses all users or a filtered set with a minimum history length.
minor comments (4)
- [§1] Typo: 'preferencs' should be 'preferences' in the first paragraph.
- [§1] Table 1 is referenced in the text, but the table content does not actually appear in the manuscript; please include the multi-criteria rating example or remove the reference.
- [§2.2] The criteria names are inconsistent: the text says 'satisfaction of service and ambience' while Figure 2 labels them 'Service' and 'Ambience'; align the names across the text and figures.
- [§2.4] The raw and cleaned files are mentioned but not described in enough detail to reproduce the cleaning; a short data dictionary and a count of rows removed during deduplication would improve the data descriptor.
Circularity Check
No circularity: the paper is a dataset release whose statistics are direct crawl outputs; self-citations are contextual, not load-bearing.
full rationale
The paper makes no derivation and offers no fitted prediction. Its central claim is that the released OpenTable data contain 19,536 ratings from 1,309 users on 91 restaurants, with overall ratings plus food, service, ambience, and value criteria. These numbers are reported as direct outputs of the crawl and the released files, not as consequences of any model, equation, or fitted parameter. Section 2.3 mentions that the authors examined MCRS algorithms over this dataset in prior work [2,3,4] using MCRecKit [5]; those self-citations are contextual reports of prior experimentation and do not provide the argument that the dataset is a benchmark, nor do they define the dataset's content. Section 2.4 discloses the user-identity rule (username+city as a user, with all 'Unknown user' entries merged into one ID) and the deduplication rule (keep the last entry per <user,item>). These are data-cleaning assumptions that affect validity, but they are not circular: no claimed result is defined in terms of itself, and no quantity is fitted and then reported as a prediction. The only potential concern is correctness/representativeness of the anonymized user IDs, which is outside the circularity definition. Therefore the paper is self-contained with respect to its release claim, and the score is 0.
Assumptions & free parameters
assumptions (3)
- ad hoc to paper The combination of username and city uniquely identifies a real user on OpenTable.
- ad hoc to paper Keeping only the last entry for each <user, item> pair in the cleaned file preserves the data's utility without introducing bias.
- domain assumption The web crawl captured the displayed ratings faithfully and completely.
Cite this review
Pith. "Pith review of OpenTable data with multi-criteria ratings." pith.science (2026). https://pith.science/paper/PMESR672
@misc{pith2026250103072,
author = {Pith},
title = {Pith review of: OpenTable data with multi-criteria ratings},
year = {2026},
howpublished = {\url{https://pith.science/paper/PMESR672}},
note = {Machine review of arXiv:2501.03072}
}
read the original abstract
With the development of recommender systems (RSs), several promising systems have emerged, such as context-aware RS, multi-criteria RS, and group RS. Multi-criteria recommender systems (MCRSs) are designed to provide personalized recommendations by considering user preferences in multiple attributes or criteria simultaneously. Unlike traditional RSs that typically focus on a single rating, these systems help users make more informed decisions by considering their diverse preferences and needs across various dimensions. In this article, we release the OpenTable data set which was crawled from OpenTable.com. The data set can be considered as a benchmark data set for multi-criteria recommendations.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Recommender systems handbook, pp
Adomavicius, G., Manouselis, N., Kwon, Y.: Multi-criteria recommender systems. In: Recommender systems handbook, pp. 769--803. Springer (2010)
work page 2010
-
[2]
IEEE Access 10, 90715--90725 (2022)
Zheng, Y., Wang, D.: Multi-criteria ranking: Next generation of multi-criteria recommendation framework. IEEE Access 10, 90715--90725 (2022)
work page 2022
-
[3]
In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces
Zheng, Y., Wang, D.: Multi-criteria decision making and recommender systems. In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. pp. 181--184 (2023)
work page 2023
-
[4]
Hybrid Multi-Criteria Preference Ranking by Subsorting
Zheng, Y., Wang, D.X.: Hybrid multi-criteria preference ranking by subsorting. arXiv preprint arXiv:2306.11233 (2023)
work page Pith review arXiv 2023
-
[5]
Zheng, Y., Wang, D.X., Ruan, Q.: MCR ec K it: An open-source library for multi-criteria recommendations. In: 2024 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) (2024)
work page 2024
-
[6]
, " * write output.state after.block = add.period write
ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 'after.sentence := #3 '...
-
[7]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize ":" * " " *...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.