{"id":"4ed3a277-3faa-4ad1-82b0-d65b834a493e","arxiv_id":"2412.06809","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors present CategoricalClassification, a modular framework for generating diverse synthetic categorical datasets to evaluate recommender systems in controlled experiments.","lead":"This paper introduces an open-source Python framework that generates synthetic categorical datasets with controllable distributions, feature interactions, and noise for evaluating recommender systems. It demonstrates the tool on three use cases: probabilistic counting, bias detection, and AutoML feature selection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never verifies that the generated data actually has the controlled statistical structure it claims, so the use-case conclusions may be artifacts of the generator rather than properties of the framework.","rationale":"The reader's weakest assumption (representativeness of synthetic categorical data) is real, but it is not the most load-bearing part of the central claim, because the framework is explicitly positioned as a controllable, completely synthetic tool rather than a real-data simulator. The more direct threat is internal: Section 2 promises precise control over distributions, interactions, noise, and correlations, but no evidence is given that the generated arrays satisfy these promises. Use Cases 2 and 3 would be meaningless if the advertised feature interactions or noise level are not actually present. Moreover, the textual description of how combinations are created and removed is ambiguous, making the experiments not independently reproducible from the paper. The suggested statistical checks would settle this concern without requiring any real dataset. Because the open-source package exists, these checks are feasible and cheap. The paper remains a plausible workshop contribution; nothing here demands rejection. The conditionality already assigned by the reader is appropriate, so no change to the verdict is needed.","tokens_in":5309,"tokens_out":6388,"duration_ms":59317,"concrete_test":"Pick the Section 3.2 configuration with AND, OR, and XOR interactions, 4 relevant plus 750 irrelevant features, 10k rows, and 20% noise. Independently re-implement the described target rule: generate the four relevant categorical features, compute the three intended interaction features, combine them into a nonlinear target, and add exactly 20% label noise. Generate one dataset with a fixed seed from CategoricalClassification under the same configuration. Then: (a) compare the generated target vector to the independently re-created ground-truth target with noise removed; the Hamming distance should be approximately 20%; (b) fit a probe model for each intended interaction feature and check that it is recoverable from the generated columns; (c) run chi-square or Kolmogorov-Smirnov tests on the requested long-tail and normal cardinalities.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two load-bearing parts: controlled attributes and statistical coherence. Section 2 describes a large API surface (Table 1), but the paper reports no quantitative check that requested feature distributions, cardinalities, correlations, 20% noise, or AND/OR/XOR/square feature interactions actually appear in the output. Use Cases 2 and 3 depend on exactly this. Section 3.2 also contains an ambiguous, unreproducible setup: 'create various feature interactions based on pairwise combinations of relevant features, which are subsequently removed' followed by 'We then iteratively remove the resulting combination features'—it is unclear whether original relevant features or their combinations are in the training set, so the Figure 3 and AutoML results cannot be re-derived from the text. If the generator's implementation silently mis-specifies interactions or noise, then the observed DeepFM-vs-logistic-regression advantage and the 'AutoML without hyperparameter tuning can be misleading' conclusion are artifacts of a bug, not evidence for the framework. This is a correctness risk distinct from representativeness: even a purely artificial, non-realistic generator must actually implement the controlled structure it advertises. The Discussion's own note that GANs/VAEs could 'further enrich the diversity and realism' is a self-acknowledged limitation, but the more basic unverified step is whether the current deterministic process delivers the requested statistics at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CategoricalClassification, a Python framework for generating synthetic high-dimensional categorical datasets with user-controlled attributes such as feature distributions, cardinalities, feature interactions, correlations, and noise. The authors argue that unlike existing synthetic-data approaches that focus on privacy or fidelity to real data, their deterministic and modular framework enables controlled, reproducible generation of production-scale data tailored to recommender-system evaluation. They demonstrate the framework in three use cases: benchmarking HyperLogLog-style probabilistic counting algorithms with a proposed caching mechanism; comparing DeepFM and logistic regression on datasets with increasingly complex feature interactions; and simulating AutoML feature-selection searches with the Outrank tool. The paper concludes that the framework effectively isolates model behavior in controlled scenarios and provides a freely available open-source package.","tokens_in":5542,"tokens_out":2959,"duration_ms":28518,"significance":"If the central claims hold, the framework is a practically useful contribution: it is open source, installable via pip, deterministic via random seeds, and modular through a NumPy/SciPy-like API. The three use cases span different stages of the recommender-system pipeline (stream counting, model bias, feature selection) and illustrate how controlled synthetic datasets can expose failure modes that are difficult to isolate in real data. The proposed caching mechanism for HyperLogLog is a concrete algorithmic idea that could be of independent interest. However, the paper's core assertion that datasets are 'statistically coherent' and generated with 'controlled attributes' is not quantitatively verified; the experiments also lack error bars, multiple runs, and precise configuration descriptions. These gaps are central because the use-case conclusions are drawn from the generated data and would be compromised if the generator does not actually implement the advertised statistical structure.","major_comments":[{"comment":"The paper claims that the framework generates datasets with 'controlled attributes' and 'statistically coherent' structure, but it provides no quantitative verification that the requested properties actually appear in the output. Specifically, there are no checks that user-specified distributions, feature cardinalities, correlation levels, noise rates (e.g., the stated 20% categorical noise), or AND/OR/XOR/square feature interactions are realized in generated datasets. The only evidence is the visual PCA/density plot in Figure 1. Because Use Cases 2 and 3 depend on the generator faithfully implementing these controlled structures, I ask for a validation section or appendix that measures, for a set of configurations, the empirical distribution fit, correlation recovery, interaction presence, and noise rate against the requested values.","section":"Section 2, Table 1"},{"comment":"The dataset construction procedure is ambiguous and appears self-contradictory: 'create various feature interactions based on pairwise combinations of relevant features, which are subsequently removed from our generated dataset. We then iteratively remove the resulting combination features.' It is unclear whether the original relevant features are retained along with the combination features, and what 'iteratively remove' means for the configurations in Figure 3. This ambiguity prevents a reader from re-deriving the experimental setup. Please clarify the exact generation and removal procedure, and also specify the model architectures, the meaning of 'one epoch,' the 'minimal hyperparameter tuning' used, and the number of repeated runs/seeds, since Figure 3 reports only point estimates.","section":"Section 3.2, paragraph 2"},{"comment":"The AutoML use case concludes that 'AutoML for feature selection without hyperparameter optimisation can be misleading,' but Outrank was evaluated without hyperparameter optimization while DeepFM and logistic regression were evaluated with tuning. This comparison confounds feature-selection quality with the absence of hyperparameter tuning, making the conclusion one-sided. The paper acknowledges the lack of HPO in Outrank, but the stated conclusion goes beyond what the experiment can support. Either compare Outrank with an HPO-enabled variant or restrict the claim to the specific setting, and note that the observed gap may reflect the tuning asymmetry rather than a property of the generated datasets.","section":"Section 3.3, last paragraph"},{"comment":"The results in Figure 2 are described as distributions of computation times over 'more than 2k synthetic datasets,' but no details are given about the range of generated configurations (feature cardinalities, distributions, noise levels) or about repeated runs and variance. The novelty claim for the caching mechanism depends on the comparison being robust across configurations. Please report the generation configuration space and provide variance estimates or at least the seeds and full configuration list.","section":"Section 3.1, Figure 2"}],"minor_comments":[{"comment":"There is a typo in the Discussion: 'alogrithm' should be 'algorithm'.","section":"Section 4"},{"comment":"In the final paragraph, 'researches' should be 'researchers'.","section":"Section 4"},{"comment":"The text says 'SynEva' but the reference title says 'Synevarec'; please make the names consistent.","section":"Section 1, reference [6]"},{"comment":"The notation in the evolution sets such as 'OR3' and 'IRR50' is not defined; specify the naming convention (e.g., interaction type plus feature index) so the reader can interpret which features were selected.","section":"Section 3.3, Figure 4 caption"},{"comment":"The x-axis label 'Synthetic dataset configs' could be made more explicit by defining the numbered configurations in the caption text (e.g., which combinations of AND, OR, XOR, and square features each configuration contains).","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-length submission with a practical tool and several concrete use cases. The main gap is not novelty or circularity but the absence of verification that the generator produces the claimed statistical structures; this is fixable with additional experiments and clarification. The ambiguous construction in Section 3.2 is the most serious reproducibility issue and should be resolved before acceptance. The paper's fit for RecSys workshop venues is reasonable, but for a journal the evaluation rigor would need to be raised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a useful tool paper with a real gap in the toolkit, but the evidence doesn't yet show the tool does what it says. The authors built CategoricalClassification, an open-source Python package for generating categorical datasets with controlled distributions, cardinalities, feature interactions, noise, and custom target functions. That fills a need in recommender-systems evaluation, where synthetic data is often too simple or locked to real-data privacy concerns. The API looks clean and modular, and the three use cases — probabilistic counting, bias detection, AutoML feature selection — are sensible demonstrations. Credit where due: the code is public, the package is on PyPI, and the first use case runs over 2k generated datasets. That's reproducible infrastructure, not just a claim.\n\nThe soft spots are real and concentrated in the experiments. Most important: the paper never checks that the generated data actually has the statistical structure it claims. No verification that requested distributions, cardinalities, interaction types (AND/OR/XOR/squares), or the 20% noise actually appear in the output. Figure 1 is a visual illustration, not a test. This matters because use cases 2 and 3 draw conclusions about DeepFM and AutoML behavior from those assumed structures. If the generator silently mis-specifies an interaction, the conclusions are artifacts. This is a correctness risk independent of the representativeness question.\n\nSecond, the setup in Section 3.2 is ambiguous: 'create various feature interactions based on pairwise combinations of relevant features, which are subsequently removed' followed by 'We then iteratively remove the resulting combination features' — it's unclear which features stay in the training set. That makes the DeepFM-vs-logistic-regression results impossible to re-derive from the text.\n\nThird, the experiments lack error bars and multiple runs, hyperparameters are loosely described, and the AutoML use case itself notes Outrank has no hyperparameter optimization, which makes the comparison one-sided. These don't sink the framework, but they make the empirical demonstrations exploratory. The representativeness of synthetic data for real workloads is also unvalidated, though that is a lower bar for a tool paper.\n\nWho is this for? Researchers who need a quick, flexible generator for controlled categorical experiments and want to build on public code. The paper deserves a serious referee because the tool is real and the use cases are well motivated, but the referee should push for a validation section checking generator outputs against parameters, plus reproducible scripts. I'd want that before citing the empirical results.\n\nRecommendation: engage with this as a workshop-level tool paper; require the validation check and the ambiguity fix before publication.","headline":"Useful open-source tool paper, but the experiments don't yet verify the generator delivers what it claims.","tokens_in":6066,"tokens_out":5116,"would_cite":false,"duration_ms":33221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Python framework generates controllable synthetic categorical datasets for recommender-system experiments, letting users tune distributions, feature interactions, and noise without real user data.","keywords":["synthetic data generation","categorical datasets","recommender system evaluation","HyperLogLog","DeepFM","AutoML","feature interactions","dataset generation"],"falsifier":"Take a real-world recommender dataset with known feature interactions and run the Section 3.2 protocol: if logistic regression matches or beats DeepFM on sum-of-squares interaction tasks but the synthetic generator's same configuration predicts otherwise, the representativeness assumption fails. More directly, rerun the framework's AutoML simulation on a real production dataset with 900 features; if the tuned-model trend reverses relative to the non-tuned search, the claimed isolation of model behavior breaks down.","tokens_in":5129,"feed_emoji":"📊","tokens_out":3483,"duration_ms":30204,"temperature":0.7,"pith_summary":"The paper introduces CategoricalClassification, a Python framework for generating fully synthetic, high-dimensional categorical datasets for evaluating recommender-system components. The authors aim to fill a gap left by synthetic-data methods that either mimic real data or focus on narrow structures: here the user controls feature distributions, cardinalities, feature interactions (AND, OR, XOR, squares), and noise, so experiments can isolate specific algorithmic behaviors. They demonstrate the framework on three tasks: benchmarking HyperLogLog-style probabilistic counters, comparing DeepFM and logistic regression under increasing feature-interaction complexity, and simulating AutoML feature-selection searches. If the framework works as claimed, researchers get a reproducible, modular way to create production-scale test data without access to sensitive real-world logs.","feed_headline":"Controlled synthetic data stresses recommender models","feed_subtitle":"New Python tool tunes distributions, feature interactions, and noise to benchmark counters, bias, and AutoML.","key_machinery":"The carrying mechanism is the CategoricalClassification class, which represents datasets as integer arrays of categorical encodings and generates target vectors from built-in or user-defined decision functions, including nonlinear combinations and clustering. Feature correlations are produced by vector rotations, and feature interactions are created as explicit pairwise combinations (AND, OR, XOR, sums and squares of sums) that can then be removed from the visible feature set, letting the experimenter control exactly which predictive signal is available to a model. Reproducibility comes from random seeds, so the same configuration yields the same data each time.","core_discovery":"On its own terms, the central claim is that a fully synthetic, controllably generated categorical dataset is a sufficient test bed for studying recommender-system algorithms, provided the generator can express the properties the experiment cares about. The paper argues that CategoricalClassification achieves this by letting a user specify value distributions, target-vector functions, correlated or combined features, and noise levels, all with deterministic seeds for reproducibility. The three use cases are presented as evidence: a caching extension to HyperLogLog that stays exact for low-cardinality counts; an observed DeepFM advantage over logistic regression as feature interactions grow, with a sum-of-squares exception; and a demonstration that AutoML feature selection without hyperparameter tuning can be misleading. The intended consequence is that researchers can run controlled experiments on synthetic data and expect the qualitative findings to transfer to real deployments.","pith_inferences":["If the representativeness assumption holds, the same generator could serve as a shared benchmark for recommender-system evaluation, letting different papers compare results on identically configured datasets.","The generator's ability to remove interaction features from the visible set while keeping their signal in the target creates a direct test for feature-interaction detection methods, a use case the paper does not develop.","A natural extension would be to invert the pipeline: fit the generator's parameters to a real dataset's marginal distributions and interaction strengths, then generate synthetic variants to test how robust a model's behavior is to perturbations of those parameters.","The paper does not compare against existing synthetic-data generators, so an obvious next experiment is a head-to-head where the same model-evaluation protocol is run on data from CategoricalClassification and from GAN- or CART-based generators."],"forward_implications":["A researcher can generate a large, sparse, high-cardinality categorical dataset in one function call and then iteratively modify it, adding noise, correlations, or missing values, without touching real user data.","The caching-augmented HyperLogLog variant stays deterministic for low-cardinality features and only switches to probabilistic counting when memory limits require it, reducing high-impact estimation errors.","Under the generated configurations, DeepFM generally outperforms logistic regression when pairwise feature interactions are present, but sum-of-squares interactions form an exception worth investigating.","AutoML feature-selection runs that skip hyperparameter tuning can produce rankings whose apparent positive trend is not reproduced by tuned models, implying that such searches should be interpreted with caution."],"supporting_citations":[{"why":"Supplies the context that synthetic data and simulators are still nascent in recommender systems, motivating the need for a new generator.","marker":"[1]"},{"why":"Describes SynRec, a CART-based prior framework for partially synthetic data that the paper positions itself against.","marker":"[2]"},{"why":"Describes SynEvaRec, a GAN-based synthetic-data framework based on real data, serving as a baseline approach.","marker":"[6]"},{"why":"Provides the DeepFM model used in the algorithmic-bias use case to compare against logistic regression.","marker":"[8]"}],"fun_headline_variants":["Synthetic data generator tailors recommender tests","Python tool spins custom datasets for recommender benchmarking","Diverse synthetic datasets for recommender evaluation","Controllable synthetic data probes recommender algorithms","Modular synthetic data for recommender system experiments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that synthetic categorical data built from chosen distributions, explicit feature interactions, and 20 percent noise behaves like real-life recommender data, so conclusions drawn about model performance on these datasets will carry over to production systems; the paper offers no direct validation of this transfer.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data generator tailors recommender tests","Python tool spins custom datasets for recommender benchmarking","Diverse synthetic datasets for recommender evaluation","Controllable synthetic data probes recommender algorithms","Modular synthetic data for recommender system experiments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1284,"prompt_tokens":900,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":314}},"tokens_in":516,"tokens_out":384,"duration_ms":3250,"temperature":1.0,"reasoning_tokens":314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:25:57.285968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real-world recommender dataset with known feature interactions and run the Section 3.2 protocol: if logistic regression matches or beats DeepFM on sum-of-squares interaction tasks but the synthetic generator's same configuration predicts otherwise, the representativeness assumption fails. More directly, rerun the framework's AutoML simulation on a real production dataset with 900 features; if the tuned-model trend reverses relative to the non-tuned search, the claimed isolation of model behavior breaks down.","supporting_citations":[{"cited_title":"Synthetic Data and Simulators for Recommendation Systems: Current State and Future Directions","cited_arxiv_id":"2112.11022","evidence_quote":"Supplies the context that synthetic data and simulators are still nascent in recommender systems, motivating the need for a new generator."},{"cited_title":"Slokom, Comparing recommender systems using synthetic data, in: Proceedings of the 12th ACM Conference on Recommender Systems, 2018, pp","cited_arxiv_id":null,"evidence_quote":"Describes SynRec, a CART-based prior framework for partially synthetic data that the paper positions itself against."},{"cited_title":"Provalov, E","cited_arxiv_id":null,"evidence_quote":"Describes SynEvaRec, a GAN-based synthetic-data framework based on real data, serving as a baseline approach."}],"review_version":1}