REVIEW 4 major objections 5 minor 8 references
Generating Diverse Synthetic Datasets for Evaluation of Real-life Recommender Systems
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A Python framework generates controllable synthetic categorical datasets for recommender-system experiments, letting users tune distributions, feature interactions, and noise without real user data.
desk verdict Useful open-source tool paper, but the experiments don't yet verify the generator delivers what it claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the CategoricalClassification class, which represents datasets as integer arrays of categorical encodings and generates target vectors from built-in or user-defined decision functions, including nonlinear combinations and clustering. Feature correlations are produced by vector rotations, and feature interactions are created as explicit pairwise combinations (AND, OR, XOR, sums and squares of sums) that can then be removed from the visible feature set, letting the experimenter control exactly which predictive signal is available to a model. Reproducibility comes from random seeds, so the same configuration yields the same data each time.
What would settle it
Take a real-world recommender dataset with known feature interactions and run the Section 3.2 protocol: if logistic regression matches or beats DeepFM on sum-of-squares interaction tasks but the synthetic generator's same configuration predicts otherwise, the representativeness assumption fails. More directly, rerun the framework's AutoML simulation on a real production dataset with 900 features; if the tuned-model trend reverses relative to the non-tuned search, the claimed isolation of model behavior breaks down.
Extended reading notes
Core claim
On its own terms, the central claim is that a fully synthetic, controllably generated categorical dataset is a sufficient test bed for studying recommender-system algorithms, provided the generator can express the properties the experiment cares about. The paper argues that CategoricalClassification achieves this by letting a user specify value distributions, target-vector functions, correlated or combined features, and noise levels, all with deterministic seeds for reproducibility. The three use cases are presented as evidence: a caching extension to HyperLogLog that stays exact for low-cardinality counts; an observed DeepFM advantage over logistic regression as feature interactions grow, with a sum-of-squares exception; and a demonstration that AutoML feature selection without hyperparameter tuning can be misleading. The intended consequence is that researchers can run controlled experiments on synthetic data and expect the qualitative findings to transfer to real deployments.
Load-bearing premise
The load-bearing premise is that synthetic categorical data built from chosen distributions, explicit feature interactions, and 20 percent noise behaves like real-life recommender data, so conclusions drawn about model performance on these datasets will carry over to production systems; the paper offers no direct validation of this transfer.
Editorial extensions
If this is right
- A researcher can generate a large, sparse, high-cardinality categorical dataset in one function call and then iteratively modify it, adding noise, correlations, or missing values, without touching real user data.
- The caching-augmented HyperLogLog variant stays deterministic for low-cardinality features and only switches to probabilistic counting when memory limits require it, reducing high-impact estimation errors.
- Under the generated configurations, DeepFM generally outperforms logistic regression when pairwise feature interactions are present, but sum-of-squares interactions form an exception worth investigating.
- AutoML feature-selection runs that skip hyperparameter tuning can produce rankings whose apparent positive trend is not reproduced by tuned models, implying that such searches should be interpreted with caution.
Reading between the lines
- If the representativeness assumption holds, the same generator could serve as a shared benchmark for recommender-system evaluation, letting different papers compare results on identically configured datasets.
- The generator's ability to remove interaction features from the visible set while keeping their signal in the target creates a direct test for feature-interaction detection methods, a use case the paper does not develop.
- A natural extension would be to invert the pipeline: fit the generator's parameters to a real dataset's marginal distributions and interaction strengths, then generate synthetic variants to test how robust a model's behavior is to perturbations of those parameters.
- The paper does not compare against existing synthetic-data generators, so an obvious next experiment is a head-to-head where the same model-evaluation protocol is run on data from CategoricalClassification and from GAN- or CART-based generators.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents CategoricalClassification, a Python framework for generating synthetic high-dimensional categorical datasets with user-controlled attributes such as feature distributions, cardinalities, feature interactions, correlations, and noise. The authors argue that unlike existing synthetic-data approaches that focus on privacy or fidelity to real data, their deterministic and modular framework enables controlled, reproducible generation of production-scale data tailored to recommender-system evaluation. They demonstrate the framework in three use cases: benchmarking HyperLogLog-style probabilistic counting algorithms with a proposed caching mechanism; comparing DeepFM and logistic regression on datasets with increasingly complex feature interactions; and simulating AutoML feature-selection searches with the Outrank tool. The paper concludes that the framework effectively isolates model behavior in controlled scenarios and provides a freely available open-source package.
Significance. If the central claims hold, the framework is a practically useful contribution: it is open source, installable via pip, deterministic via random seeds, and modular through a NumPy/SciPy-like API. The three use cases span different stages of the recommender-system pipeline (stream counting, model bias, feature selection) and illustrate how controlled synthetic datasets can expose failure modes that are difficult to isolate in real data. The proposed caching mechanism for HyperLogLog is a concrete algorithmic idea that could be of independent interest. However, the paper's core assertion that datasets are 'statistically coherent' and generated with 'controlled attributes' is not quantitatively verified; the experiments also lack error bars, multiple runs, and precise configuration descriptions. These gaps are central because the use-case conclusions are drawn from the generated data and would be compromised if the generator does not actually implement the advertised statistical structure.
major comments (4)
- [Section 2, Table 1] The paper claims that the framework generates datasets with 'controlled attributes' and 'statistically coherent' structure, but it provides no quantitative verification that the requested properties actually appear in the output. Specifically, there are no checks that user-specified distributions, feature cardinalities, correlation levels, noise rates (e.g., the stated 20% categorical noise), or AND/OR/XOR/square feature interactions are realized in generated datasets. The only evidence is the visual PCA/density plot in Figure 1. Because Use Cases 2 and 3 depend on the generator faithfully implementing these controlled structures, I ask for a validation section or appendix that measures, for a set of configurations, the empirical distribution fit, correlation recovery, interaction presence, and noise rate against the requested values.
- [Section 3.2, paragraph 2] The dataset construction procedure is ambiguous and appears self-contradictory: 'create various feature interactions based on pairwise combinations of relevant features, which are subsequently removed from our generated dataset. We then iteratively remove the resulting combination features.' It is unclear whether the original relevant features are retained along with the combination features, and what 'iteratively remove' means for the configurations in Figure 3. This ambiguity prevents a reader from re-deriving the experimental setup. Please clarify the exact generation and removal procedure, and also specify the model architectures, the meaning of 'one epoch,' the 'minimal hyperparameter tuning' used, and the number of repeated runs/seeds, since Figure 3 reports only point estimates.
- [Section 3.3, last paragraph] The AutoML use case concludes that 'AutoML for feature selection without hyperparameter optimisation can be misleading,' but Outrank was evaluated without hyperparameter optimization while DeepFM and logistic regression were evaluated with tuning. This comparison confounds feature-selection quality with the absence of hyperparameter tuning, making the conclusion one-sided. The paper acknowledges the lack of HPO in Outrank, but the stated conclusion goes beyond what the experiment can support. Either compare Outrank with an HPO-enabled variant or restrict the claim to the specific setting, and note that the observed gap may reflect the tuning asymmetry rather than a property of the generated datasets.
- [Section 3.1, Figure 2] The results in Figure 2 are described as distributions of computation times over 'more than 2k synthetic datasets,' but no details are given about the range of generated configurations (feature cardinalities, distributions, noise levels) or about repeated runs and variance. The novelty claim for the caching mechanism depends on the comparison being robust across configurations. Please report the generation configuration space and provide variance estimates or at least the seeds and full configuration list.
minor comments (5)
- [Section 4] There is a typo in the Discussion: 'alogrithm' should be 'algorithm'.
- [Section 4] In the final paragraph, 'researches' should be 'researchers'.
- [Section 1, reference [6]] The text says 'SynEva' but the reference title says 'Synevarec'; please make the names consistent.
- [Section 3.3, Figure 4 caption] The notation in the evolution sets such as 'OR3' and 'IRR50' is not defined; specify the naming convention (e.g., interaction type plus feature index) so the reader can interpret which features were selected.
- [Figure 3 caption] The x-axis label 'Synthetic dataset configs' could be made more explicit by defining the numbered configurations in the caption text (e.g., which combinations of AND, OR, XOR, and square features each configuration contains).
Circularity Check
No circularity: dataset parameters are user-specified inputs and all evaluated models are external.
full rationale
The paper presents a synthetic data generator and uses it in three application studies; it does not derive a prediction from its own fitted values. The generator's parameters (distributions, cardinalities, feature interactions, noise level) are inputs supplied by the user, and the subsequent evaluations are performed by external or standard algorithms such as HyperLogLog, DeepFM, logistic regression, SGDClassifier, and Outrank. No quantity estimated from the generated data is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The only affiliation-related element is that Use Case 3 employs Outrank, an open-source package associated with the authors, but this is a tool application rather than a load-bearing citation that closes an argument. The paper's unverified claim that the generated data actually possesses its advertised statistical structure is a validation or correctness gap, not a circularity, because the framework's outputs are not used to justify the framework's own premises. Accordingly, no circular step can be identified and quoted.
Assumptions & free parameters
assumptions (2)
- domain assumption Synthetic categorical data generated from user-specified distributions, feature interactions, and noise is statistically coherent and representative of real-life recommender system data.
- domain assumption The predefined feature combination operators (AND, OR, XOR, sum of squares, square of sums) capture the kinds of feature interactions that matter in real recommender data.
Cite this review
Pith. "Pith review of Generating Diverse Synthetic Datasets for Evaluation of Real-life Recommender Systems." pith.science (2026). https://pith.science/paper/XYUDRH5N
@misc{pith2026241206809,
author = {Pith},
title = {Pith review of: Generating Diverse Synthetic Datasets for Evaluation of Real-life Recommender Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/XYUDRH5N}},
note = {Machine review of arXiv:2412.06809}
}
read the original abstract
Synthetic datasets are important for evaluating and testing machine learning models. When evaluating real-life recommender systems, high-dimensional categorical (and sparse) datasets are often considered. Unfortunately, there are not many solutions that would allow generation of artificial datasets with such characteristics. For that purpose, we developed a novel framework for generating synthetic datasets that are diverse and statistically coherent. Our framework allows for creation of datasets with controlled attributes, enabling iterative modifications to fit specific experimental needs, such as introducing complex feature interactions, feature cardinality, or specific distributions. We demonstrate the framework's utility through use cases such as benchmarking probabilistic counting algorithms, detecting algorithmic bias, and simulating AutoML searches. Unlike existing methods that either focus narrowly on specific dataset structures, or prioritize (private) data synthesis through real data, our approach provides a modular means to quickly generating completely synthetic datasets we can tailor to diverse experimental requirements. Our results show that the framework effectively isolates model behavior in unique situations and highlights its potential for significant advancements in the evaluation and development of recommender systems. The readily-available framework is available as a free open Python package to facilitate research with minimal friction.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Synthetic Data and Simulators for Recommendation Systems: Current State and Future Directions
A. Lesnikowski, G. d. S. P. Moreira, S. Rabhi, K. Byleen-Higley, Synthetic data and simulators for recommendation systems: current state and future directions, arXiv preprint arXiv:2112.11022 (2021)
work page Pith review arXiv 2021
-
[2]
M. Slokom, Comparing recommender systems using synthetic data, in: Proceedings of the 12th ACM Conference on Recommender Systems, 2018, pp. 548–552
work page 2018
-
[3]
A. Berlioz, A. Friedman, M. A. Kaafar, R. Boreli, S. Berkovsky, Applying differential privacy to matrix factorization, in: Proceedings of the 9th ACM Conference on Recommender Systems, 2015, pp. 107–114
work page 2015
-
[4]
N. Antulov-Fantulin, M. Bošnjak, V. Zlatić, M. Grčar, T. Šmuc, Synthetic sequence generator for recommender systems–memory biased random walk on a sequence multilayer network, in: Discovery Science: 17th International Conference, DS 2014, Bled, Slovenia, October 8-10, 2014. Proceedings 17, Springer, 2014, pp. 25–36
work page 2014
-
[5]
Scalable Realistic Recommendation Datasets through Fractal Expansions
F. Belletti, K. Lakshmanan, W. Krichene, Y.-F. Chen, J. Anderson, Scalable realistic recommendation datasets through fractal expansions, arXiv preprint arXiv:1901.08910 (2019)
work page Pith review arXiv 2019
-
[6]
V. Provalov, E. Stavinova, P. Chunaev, Synevarec: A framework for evaluating recommender systems on synthetic data classes, in: 2021 International Conference on Data Mining Workshops (ICDMW), IEEE, 2021, pp. 55–64
work page 2021
-
[7]
W. Shen, Deepctr: Easy-to-use,modular and extendible package of deep-learning based ctr models, https://github.com/shenweichen/deepctr, 2017
work page 2017
-
[8]
H. Guo, R. Tang, Y. Ye, Z. Li, X. He, Deepfm: a factorization-machine based neural network for ctr prediction, arXiv preprint arXiv:1703.04247 (2017)
arXiv 2017
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.