REVIEW 3 major objections 4 minor 19 references
Using the Open Meta Kaggle Dataset to Evaluate Tripartite Recommendations in Data Markets
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Recommendation accuracy hinges on data market use case
desk verdict A clean taxonomy and honest benchmark for tripartite data-market recommendations, but the UC3/UC4 conclusions rest on an unvalidated co-interaction proxy that should be tested before the preference for CF is accepted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the tripartite graph connecting users, datasets, and services, which yields four distinct recommendation tasks (UC1 through UC4). Because datasets and services have no direct user interactions in the dataset–service tasks, the paper constructs a collaboration network: a dataset and a service are linked whenever a user has interacted with both. This derived relation supplies both the training signal for collaborative filtering and the ground truth for evaluating UC3 and UC4.
What would settle it
Run the same four use cases on a real data market where dataset–service compatibility is labeled by domain experts or by actual service usage, and check whether collaborative filtering still beats most-popular for UC3 and UC4; if it does not, the conclusion depends on the co-interaction proxy rather than on the market structure.
Extended reading notes
Core claim
The central claim is that recommendation accuracy strongly depends on the given use case. In the simplest case, recommending datasets to users where only 45 candidates exist, the non-personalized most-popular baseline beats collaborative filtering (P@1 0.823 vs 0.705). In the three harder cases—services to users, datasets to services, and services to datasets—collaborative filtering wins across nearly all metrics; the hardest case, recommending services for datasets, remains essentially unsolved, with CF reaching only P@1 0.022 and MP achieving 0.000 on every metric. The authors conclude there is no one-size-fits-all solution, and that linking services and datasets will require more sophisticated methods than popularity or standard collaborative filtering.
Load-bearing premise
For the two tasks that link datasets to services, the paper treats 'a user interacted with both' as evidence of a genuine dataset–service match, and uses that same derived link both to train and to score the algorithms; if that proxy is wrong, the comparison is unsupported.
Editorial extensions
If this is right
- In small-candidate, user-to-item tasks such as recommending datasets to users, simple popularity ranking can outperform personalization, so data market operators need not deploy complex models for every surface.
- For larger candidate sets or entity-to-entity recommendation, collaborative filtering is the better default among the two tested approaches.
- The dataset-to-service task (UC3) shows the best collaborative filtering results across all four use cases, suggesting that the derived dataset–service interaction graph carries usable signal.
- Recommending services to datasets is not solved by either baseline, indicating that a separate algorithmic effort is needed for this direction.
- The proposed four-use-case framing and architecture give data market practitioners a concrete evaluation template.
Reading between the lines
- If the co-interaction proxy overstates genuine dataset–service compatibility, the relative ordering of MP and CF in UC3 and UC4 may not transfer to real marketplaces; a direct or content-based ground truth could change the ranking.
- Applying content-based features of datasets and services could be a natural next step: the paper itself points to content-based filtering and word embeddings, but does not test them.
- The same evaluation template could be applied to other three-entity platforms, such as code repositories linking developers, packages, and issues, to test whether the use-case-dependent accuracy pattern generalizes.
- The near-perfect P@1 of both algorithms in UC3 suggests a single highly-connected dataset dominates relevance; a degree-aware baseline or per-service analysis would clarify whether the metric reflects genuine service diversity.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses recommendation in data markets by extending the usual bipartite user-item view to a tripartite relationship among users, datasets, and services. It defines four use cases—datasets for users (UC1), services for users (UC2), datasets for services (UC3), and services for datasets (UC4)—and proposes a system architecture built on the ScaR framework. Using the Meta Kaggle dataset, the authors construct a collaboration network between datasets and services by linking any two entities that a user has interacted with, and then evaluate a most-popular (MP) and a collaborative-filtering (CF) algorithm on all four use cases. The reported results show MP outperforming CF in UC1, CF outperforming MP in UC2, UC3, and UC4, and very low accuracy overall in UC4. The paper concludes that recommendation accuracy strongly depends on the use case, with simple popularity-based approaches sufficient when the candidate set is limited and personalized approaches preferable for more complex entity-to-entity recommendation tasks.
Significance. The paper makes a useful contribution by explicitly defining a taxonomy of tripartite recommendation use cases for data markets and by providing a transparent evaluation on a public dataset with standard baselines. Its strengths include the clear separation of the four use cases, the use of the open Meta Kaggle dataset, and the honest acknowledgment of limitations such as the simulated nature of the data market and the restriction to two algorithms. If the results are accepted, the work would be a valuable starting point for a largely under-researched area and would support the practical message that algorithm choice should be tailored to the recommendation scenario. However, the significance is currently limited because the UC3 and UC4 results rest on an unvalidated co-interaction proxy for dataset-service compatibility, and because the evaluation populations are very small with no uncertainty quantification. The headline claim that CF should be favored for complex entity-to-entity settings is therefore not yet fully supported.
major comments (3)
- [3.1, 3.2, Table 2] The evaluation of UC3 and UC4 is built on an unvalidated proxy: a link between a dataset and a service is created whenever a user has interacted with both, and this same derived relation is used both as the collaborative-filtering training signal and as the ground-truth relevance in the evaluation. Consequently, the UC3 and UC4 results measure how well algorithms predict this co-interaction heuristic, not genuine dataset-service compatibility in a data market. The P@1=1.000 value for both MP and CF in UC3, explained by one highly connected dataset being relevant for all 2,338 evaluated services, is exactly the pattern one would expect if the construction rule, rather than any independent compatibility signal, defines relevance. The density of the derived network (95,249 interactions among 45 × 3,334 possible pairs) makes this concern particularly acute. The paper should either validate the proxy against an independent signal, analyze sensitivity to the construction rule, or substantially soften the conclusions drawn from UC3 and UC4.
- [3.2, Table 2] The evaluation provides no uncertainty quantification on very small test populations: UC1 has only 17 test users and UC4 only 44 test datasets. Reported differences such as UC2 P@1 (0.103 for MP vs. 0.137 for CF) and UC1 P@1 (0.823 vs. 0.705) are not accompanied by confidence intervals, per-entity variance, or significance tests. The qualitative claim that recommendation accuracy strongly depends on the use case, and the specific MP-vs-CF ordering, would be more secure if the authors reported paired tests or confidence intervals, or explicitly framed the results as preliminary given the small samples.
- [3.2] The CF algorithm is underspecified. The text states that for UC3 and UC4 'we calculate similarities between datasets and services, respectively,' but it does not specify the similarity measure, neighborhood size, rating normalization, or handling of the dense derived network. Because the entire comparison rests on this implementation, the missing details prevent replication and make it impossible to separate the effects of the algorithm from the effects of specific implementation choices. The authors should provide the exact CF formulation or release the code used for the experiments.
minor comments (4)
- [3.1, Table 1] The number of user/dataset interactions is given as 2,926 in the text of Section 3.1 but as 2,962 in Table 1; these values should be reconciled.
- [2, Figure 2] There is a typo in Section 2 ('gives an a detailed' should be 'gives a detailed'), and the caption of Figure 2 says 'is build upon' instead of 'is built upon'.
- [2.1] In the description of UC4, 'which makes this use cases more complex' should be 'which makes this use case more complex'.
- [3.2] Footnote 8, which says 'For 10 recommended entities, Precision typically reaches its highest value for k = 1,' is confusing because the paper reports P@1, not precision at 10; the footnote should be clarified or removed.
Circularity Check
No circular reasoning in the evaluation; the UC3/UC4 co-interaction proxy is a validity concern, not a circularity.
full rationale
The paper's empirical comparisons are a standard offline evaluation: observed interactions are split into training and test sets, MP and CF are computed from training interactions only, and metrics are computed on held-out interactions. No parameter is fitted to the test labels, and neither algorithm is defined in terms of the outcome it is asked to predict. The central conclusion that recommendation accuracy depends on the use case is an empirical finding, not a definitional consequence. The main limitation is that UC3/UC4 relevance labels are constructed by co-interaction, i.e., a dataset-service link is created when a user has interacted with both entities, and this same proxy is used for both training and testing. This is an unvalidated modeling assumption and a threat to external validity, but it is not circular: the test links are not derived from the recommendation scores, and the paper explicitly discloses the construction. Self-citations to ScaR [9,10] and to a collaboration-network paper [5] describe infrastructure or motivate the link construction but do not carry the empirical argument. Hence no specific circular step can be exhibited.
Assumptions & free parameters
free parameters (1)
- Minimum interactions threshold =
11
assumptions (3)
- domain assumption The Meta Kaggle dataset approximates a real-world data market.
- ad hoc to paper An indirect link between a dataset and a service, created when a user has interacted with both, is a valid representation of dataset-service relationships.
- domain assumption Standard offline train/test evaluation with a fixed split is appropriate for comparing recommenders in this setting.
Cite this review
Pith. "Pith review of Using the Open Meta Kaggle Dataset to Evaluate Tripartite Recommendations in Data Markets." pith.science (2026). https://pith.science/paper/YC45QWLE
@misc{pith2026190804017,
author = {Pith},
title = {Pith review of: Using the Open Meta Kaggle Dataset to Evaluate Tripartite Recommendations in Data Markets},
year = {2026},
howpublished = {\url{https://pith.science/paper/YC45QWLE}},
note = {Machine review of arXiv:1908.04017}
}
read the original abstract
This work addresses the problem of providing and evaluating recommendations in data markets. Since most of the research in recommender systems is focused on the bipartite relationship between users and items (e.g., movies), we extend this view to the tripartite relationship between users, datasets and services, which is present in data markets. Between these entities, we identify four use cases for recommendations: (i) recommendation of datasets for users, (ii) recommendation of services for users, (iii) recommendation of services for datasets, and (iv) recommendation of datasets for services. Using the open Meta Kaggle dataset, we evaluate the recommendation accuracy of a popularity-based as well as a collaborative filtering-based algorithm for these four use cases and find that the recommendation accuracy strongly depends on the given use case. The presented work contributes to the tripartite recommendation problem in general and to the under-researched portfolio of evaluating recommender systems for data markets in particular.
Figures
Reference graph
Works this paper leans on
-
[1]
José María Cavanillas, Edward Curry, and Wolfgang Wahlster (Eds.). 2016. New Horizons for a Data-Driven Economy . Springer, Cham, Switzerland. DOI:http: //dx.doi.org/10.1007/978-3-319-21569-3
-
[2]
Edward Curry. 2016. The big data value chain: definitions, concepts, and theo- retical approaches. In New horizons for a data-driven economy . Springer, Cham, 29–37
work page 2016
-
[3]
Ernesto Damiani, Paolo Ceravolo, Fulvio Frati, Valerio Bellandi, Ronald Maier, Isabella Seeber, and Gabriela Waldhart. 2015. Applying recommender systems in collaboration environments. Computers in Human Behavior 51 (2015), 1124–1133
work page 2015
-
[4]
Daniela Godoy and Alejandro Corbellini. 2016. Folksonomy-Based Recom- mender Systems: A State-of-the-Art Review. International Journal of Intelli- gent Systems 31, 4 (2016), 314–346. DOI:http://dx.doi.org/10.1002/int.21753 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/int.21753
-
[5]
Ilire Hasani-Mavriqi, Dominik Kowald, Denis Helic, and Elisabeth Lex. 2018. Consensus dynamics in online collaboration systems. Computational Social Networks 5, 1 (2018), 2
work page 2018
-
[6]
Jonathan L Herlocker, Joseph A Konstan, Loren G Terveen, and John T Riedl
-
[7]
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422–446
2002
-
[8]
Tom Kenter and Maarten De Rijke. 2015. Short text similarity with word embed- dings. In Proc. of CIKM’15. ACM, 1411–1420
work page 2015
Show all 19 references
-
[9]
Emanuel Lacic, Dominik Kowald, Denis Parra, Martin Kahr, and Christoph Trat- tner. 2014. Towards a scalable social recommender engine for online marketplaces: The case of apache solr. In Proceedings of the 23rd International Conference on World Wide Web. ACM, 817–822
2014
-
[10]
Emanuel Lacic, Matthias Traub, Dominik Kowald, and Elisabeth Lex. 2015. ScaR: Towards a Real-Time Recommender Framework Following the Microservices Architecture. In Proceedings of LSRS2015 Workshop at RecSys 2015
2015
-
[11]
Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2011. Content-based recommender systems: State of the art and trends. In Recommender systems handbook. Springer, 73–105
2011
-
[12]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv e-prints (Jan 2013), arXiv:1301.3781. arXiv:cs.CL/1301.3781
2013 arXiv
-
[13]
Denis Parra and Peter Brusilovsky. 2009. Collaborative filtering for social tagging systems: an experiment with CiteULike. InProceedings of the third ACM conference on Recommender systems. ACM, 237–240
2009
-
[14]
Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011. Introduction to recom- mender systems handbook. Springer
2011
-
[15]
Alan Said and Alejandro Bellogín. 2014. Comparative recommender system evaluation: benchmarking recommendation frameworks. In Proceedings of the 8th ACM Conference on Recommender systems . ACM, 129–136
2014
-
[16]
Andrew I Schein, Alexandrin Popescul, Lyle H Ungar, and David M Pennock
-
[17]
Matthias Traub, Heimo Gursch, Elisabeth Lex, and Roman Kern. 2017. Data Mar- ket Austria (Institute of Systems Sciences, Innovation and Sustainability Reports) , Romana Rauter, Martina Zimek, Aisma Linda Kiesnere, and Rupert J. Baumgart- ner (Eds.). Institute of Systems Scienc...
2017
-
[2002]
In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Methods and metrics for cold-start recommendations. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 253–260
-
[2004]
ACM Transactions on Information Systems (TOIS) 22, 1 (2004), 5–53
Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS) 22, 1 (2004), 5–53
2004
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.