{"id":"ef111a53-c5d1-41c4-bbf4-b54c6de152cb","arxiv_id":"2507.11821","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MNIST-Gen automatically builds MNIST-style datasets for user-defined categories using CLIP semantic scoring, reinforcement learning, and hierarchical category definitions, demonstrated on Tree-MNIST and Food-MNIST.","lead":"MNIST-Gen is a pipeline that automatically creates small grayscale image datasets in the MNIST format from user-defined category descriptions. It lets researchers build custom benchmarks for niche domains without spending weeks on manual image collection and labeling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Categorization accuracy is unvalidated: the paper claims no ground truth yet cites Food-11 labels; the 89.7% figure should be checked against them.","rationale":"Most of the pipeline components (CLIP embeddings, three processing modes, DQN) are described concretely, and the experimental tables are internally consistent, but the evaluation does not independently verify the labels that determine both the headline accuracy and the downstream model performance. The strongest claim is an accuracy number, and accuracy is only meaningful relative to a reference. The paper's own limitation section concedes the absence of ground truth, while Section IV.A points to a labeled source (Food-11) that could have provided one. This is not a matter of disagreement with scientific consensus; it is an internal inconsistency that can be resolved by a single comparison. The reader's weakest assumption is partially on point—the human-review labels are not an independent referee—but the more precise issue is that a labeled dataset appears to have been available and was not used as such, or if it was used, the protocol and results should be reported. I am not arguing that the framework is fraudulent or useless; the modular design and qualitative examples are plausible. But the central quantitative claim cannot be assessed from the preprint until this is settled.","tokens_in":13644,"tokens_out":5864,"duration_ms":60978,"concrete_test":"Use the original Food-11 category labels for the 5,000 images used in Section IV.A as an external ground truth, and compute macro-F1 and Cohen's kappa between the Smart Batch CLIP auto-categorization and the Food-11 labels after defining a mapping between the 10 MNIST-Gen categories and Food-11's classes. If the resulting agreement is substantially below the reported 89.7%, the headline accuracy claim is not established; if the Food-11 labels were used in the original evaluation, Section V's \"no ground truth\" statement is contradicted and the paper must clarify the protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—85% automatic categorization and 89.7% overall label accuracy on Food-MNIST (Table I)—is not verifiable as reported because the evaluation lacks an independent reference. Section V states \"there is no definitive ground truth against which to benchmark performance,\" but Section IV.A says the 5,000 Food-MNIST images were \"retrieved ... from the Kaggle Food-11 dataset provided by [30] to compare and validate subcategory labeling.\" Food-11 is a curated, labeled dataset; if its category labels were available to the authors, CLIP-based categorization accuracy should be measured against them rather than against human-review labels that were themselves generated with CLIP suggestions (Section III.G). The reported 89.7% \"against human review\" can therefore be circular: the same model that proposes categorizations also informs the reviewer, so agreement does not establish correctness. A second, related weakness is that the RL reward in Eq. (16) includes ModelAcc from a model trained on the current batch; the 5–8% accuracy improvement claimed for RL filtering (Section IV.E) is then partially a selection artifact of optimizing the same metric used for evaluation. Both issues bear directly on the paper's headline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MNIST-Gen, a modular pipeline for generating MNIST-style grayscale image datasets from user-specified hierarchical category definitions. The system combines keyword-driven image retrieval, CLIP-based semantic analysis, three processing modes (individual review, smart batch, fast batch), a Deep Q-Learning agent for adaptive sample filtering, and a category-theoretic formalization of the transformation pipeline. As proof of concept, the authors construct Tree-MNIST and Food-MNIST and evaluate them by training classical and deep classifiers. The headline claims are 85% automatic categorization accuracy, an 89.7% label accuracy in smart batch mode, a 5–8% classification accuracy improvement from RL+semantic filtering over random selection, and an 80% reduction in manual annotation time compared with manual curation.","tokens_in":13915,"tokens_out":4558,"duration_ms":54475,"significance":"If the empirical claims were properly supported, MNIST-Gen would be a practically useful contribution for researchers who need small, task-specific benchmark datasets without publishing full datasets. The modular architecture, the hierarchical semantic representation with explicit semantic characteristics, and the integration of CLIP with a DQN-based selection policy are interesting and clearly described. The paper also includes honest limitations (Section V) and provides two new datasets, though the datasets themselves are not yet released. However, the evaluation as it stands does not justify the quantitative claims: the RL reward includes the same downstream accuracy metric used for evaluation, the categorization accuracy is not benchmarked against an independent ground truth even though one exists (Food-11), ambiguous images are removed before evaluation without accounting for the resulting selection bias, and no error bars or significance tests are reported. These issues are load-bearing because they concern exactly the numbers announced in the abstract and conclusion.","major_comments":[{"comment":"The RL reward in Eq. (16) includes λ3·ModelAcc, the accuracy of a lightweight model trained on the current batch. The paper then reports in Section IV.E that the RL+Semantic filtering variant achieves a 5–8% improvement in classification accuracy over random and CLIP-only filtering. This is circular: the filter is optimized against the same type of accuracy signal that is later used as the outcome measure. To support the claim, the authors should train the final downstream classifiers on a fixed, pre-registered test split that is never exposed to the RL loop, and report accuracy on that independent test set for all three filtering variants (random, CLIP-only, RL+CLIP). Without this separation, the reported improvement is partly a selection artifact.","section":"Section IV.E and Eq. (16)"},{"comment":"The paper states in Section V that “there is no definitive ground truth against which to benchmark performance,” yet Section IV.A says the 5,000 Food-MNIST images were “retrieved from the Kaggle Food-11 dataset provided by [30] to compare and validate subcategory labeling.” Food-11 is a labeled dataset; if the original labels were available, categorization accuracy should be measured against them rather than against human-review labels that were themselves generated with CLIP suggestions (Section III.G). At minimum, the authors should explain why Food-11 labels cannot serve as ground truth, and they should define exactly what the “Accuracy %” column in Table I measures (agreement with a human reviewer who saw CLIP suggestions, agreement with Food-11 labels, or something else).","section":"Section IV.A and Section V"},{"comment":"The evaluation excludes “visually ambiguous images that contained multiple food types from different categories (e.g., eggs and fries on the same plate)” before computing classification accuracy. This removes exactly the hard cases that a real-world dataset-generation pipeline would encounter, and the degree of filtering may affect the three processing modes differently. The authors should report the number and proportion of removed images, show the classification accuracy on the removed set, and run the processing-mode comparison on the full retrieved set (or at least demonstrate that the exclusion rule does not change the ranking of modes). Without this, the reported accuracies in Tables I–III are conditional on an unquantified favorable filter.","section":"Section IV.A"},{"comment":"All accuracy numbers are reported as point estimates without error bars, confidence intervals, or significance tests, and it is unclear how many independent runs were performed. With dataset sizes of 1,500 and 5,000 images and an 80/20 split, a single run is not sufficient to establish a 5–8% difference between filtering variants; the observed gap could be within run-to-run noise. Please provide results over multiple seeds with standard deviations and, ideally, a paired statistical test for the RL+Semantic versus Semantic-only comparison.","section":"Tables II–III and Section IV.E"}],"minor_comments":[{"comment":"The code and data links are placeholders (\"(mnist-gen-code)\" and \"(data-batch1)\"); actual URLs or a DOI should be provided before publication so the claimed reproducibility can be checked.","section":"Footnote 1"},{"comment":"The \"Accuracy %\" column needs a precise definition: accuracy against which reference (human review, original Food-11 labels, or something else), how the human review was conducted, and whether the human reviewers saw CLIP's category suggestions.","section":"Table I"},{"comment":"Please clarify which binarization method was actually used in the experiments: Eq. (15) defines a fixed threshold θ, while Appendix A.2 mentions Otsu's method and adaptive RL-learned thresholds.","section":"Eq. (15) and Appendix A.2"},{"comment":"The natural transformation diagram is not rendered correctly in the text; please typeset the commutative diagram properly and define the objects and arrows in the diagram explicitly.","section":"Section III.E"},{"comment":"Table V shows only three of the ten Food-MNIST main categories; label it as a representative excerpt and include the full table in a supplement if space is limited.","section":"Table V"},{"comment":"The phrase \"near-optimal class distribution entropy\" should be quantified, for example by reporting the actual entropy value relative to the uniform distribution.","section":"Section IV.E"},{"comment":"The category-theoretic formalism is not connected to any measurable prediction or experimental outcome in the paper; consider stating explicitly what explanatory work the formalism does, or presenting it more briefly as a conceptual framing.","section":"Section II.C and Section III.C–E"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the writing is generally clear, but the current evaluation does not support the abstract's quantitative claims. I would be willing to reconsider after a revision that adds an independent evaluation protocol (e.g., test split never used in RL, comparison against Food-11 labels, and error bars). The category-theory sections currently read as decorative rather than load-bearing; either connect them to the experimental design or shorten them substantially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clearly written, genuinely useful pipeline for generating MNIST-style domain-specific datasets, and the time-savings idea is sound. But the headline numbers (85% auto-categorization, 89.7% label accuracy) are not verified: there is no independent ground truth, and the RL reward includes the same downstream accuracy later cited as evidence of quality. The stress-test note is right.\n\nThe genuinely new part is the integration: CLIP-based hierarchical categorization with configurable JSON category definitions, three processing modes (individual, smart batch, fast batch), and a DQN filter that discards redundant or noisy samples. The two generated datasets, Tree-MNIST and Food-MNIST, are plausible outputs, and the reported reduction in annotation time from 25 to 5.2 hours is believable if the pipeline works as described. The modular design is a real strength; replacing any stage, say swapping CLIP for another model, is straightforward.\n\nThe soft spots are real and mostly concern evaluation. First, the paper claims there is 'no definitive ground truth' in Section V, but Section IV.A says images were retrieved from the Food-11 dataset, which is curated and labeled. If those labels are available, the right test is to compare MNIST-Gen's categorization against them. Getting 89.7% agreement with human reviewers who were shown CLIP suggestions is not the same as correctness. Second, Eq. (16) puts ModelAcc in the RL reward, and then the paper presents downstream model accuracy as evidence that RL filtering improves the dataset. That is circular to a meaningful degree: the filter is rewarded for selecting batches on which a model trains well, so a 5–8% gain is at least partly a selection artifact. Third, the category-theory section III.C-E is ornamentation. The functoriality 'proof' in the appendix just observes that pure functions compose. It does no work, and the paper would lose nothing if it were cut. Finally, code and data are promised but not provided; as it stands, the experiments are unverifiable.\n\nNone of this debunks the core usefulness of the tool. A reader who wants to build a small, specialized MNIST-style benchmark could still learn from the design. But as a research paper, the empirical claims need rework: release the code, actually evaluate against Food-11 labels or another external set, and re-frame the RL section so the reward does not contain the evaluation metric. With those changes, this would be a solid systems paper. Right now it is a conditional accept.\n\nWho is this for: people working on dataset generation, low-resource benchmarks, or weakly-supervised labeling. It deserves a serious referee; the framework is real and the idea of composable, user-defined hierarchical categories is worth airing. I would send it to review but with a strong request for code/data and independent validation.","headline":"A useful dataset-generation pipeline whose headline accuracy claims are unvalidated and partially circular; deserves peer review but needs reworked evaluation.","tokens_in":14412,"tokens_out":2872,"would_cite":false,"duration_ms":36162,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MNIST-Gen builds custom MNIST-style image datasets on demand, auto-labeling 85% of images with modest human review.","keywords":["hierarchical semantic categorization","CLIP","MNIST-style dataset generation","reinforcement learning","human-in-the-loop","category theory","Food-MNIST","Tree-MNIST"],"falsifier":"Run the smart-batch pipeline on an image set with fixed ground-truth labels, such as a held-out subset of Food-101, and measure the auto-assigned categories against those labels; if the auto-categorization agreement is far below 85% while human review still corrects many errors, the headline accuracy and the claimed time savings would not survive a labeled benchmark.","tokens_in":13441,"feed_emoji":"🖼️","tokens_out":4152,"duration_ms":43867,"temperature":0.7,"pith_summary":"The paper presents MNIST-Gen, an automated framework that generates MNIST-style grayscale image datasets for user-defined categories. It claims that combining CLIP-based semantic embeddings, hierarchical category definitions, and a reinforcement-learning agent with human feedback lets the system auto-categorize 85% of images in smart-batch mode while cutting annotation time by about 80%, from 25.0 to 5.2 hours on a 5,000-image food dataset. Two proof-of-concept datasets, Tree-MNIST and Food-MNIST, are produced and benchmarked, and the RL-filtered versions improve downstream classification accuracy by 5–8% over unfiltered or CLIP-only versions. The authors argue this lowers the barrier to task-specific benchmark creation in domains where standard datasets are missing or costly to publish.","feed_headline":"Custom MNIST datasets on demand: 85% auto-labeled","feed_subtitle":"A CLIP-plus-RL pipeline cuts annotation time by 80%, creating task-specific benchmarks without hand labeling.","key_machinery":"The argument is carried by the hierarchical semantic score, which combines CLIP text–image similarity, characteristic matching, and visual attributes; by the category-theoretic functor $F: \\mathcal{I} \\to \\mathcal{M}$ that guarantees compositionality of the pipeline; and by a DQN agent whose reward balances semantic confidence, class-balance entropy, model accuracy, and redundancy. The functor framing licenses replacing or reordering any transformation step without changing the overall dataset semantics, and the RL reward is what drives the reported 5–8% downstream accuracy improvement.","core_discovery":"The central claim is that a dataset-generation pipeline can be made compositional and adaptive enough that researchers can construct small, publication-ready MNIST-style benchmarks for arbitrary hierarchical categories with minimal manual labeling. The authors establish this through a six-stage pipeline—keyword retrieval, hierarchical category definition, CLIP semantic analysis, three processing modes, RL-based sample filtering, and a transformation chain modeled as composable morphisms—and support it with two generated datasets. On Food-MNIST's smart-batch mode, the system auto-categorizes 85% of images with an overall accuracy of 89.7% against human review, and the RL-filtered version yields 73.84% CNN accuracy on the 10-class task; on Tree-MNIST the CNN reaches 86.51%.","pith_inferences":["The no-ground-truth evaluation means the headline 85% accuracy measures agreement with CLIP-guided human review, not agreement with a fixed external label set; a fair reader should treat it as a process claim rather than a label-quality guarantee.","A direct testable extension would be to run the smart-batch pipeline on a labeled benchmark such as a Food-101 subset and report auto-categorization agreement with the known labels, converting the framework's claims into falsifiable numbers.","The RL reward's model-accuracy term creates a feedback loop that may inflate the measured 5–8% gain; an ablation with that term zeroed would isolate the semantic-filtering contribution.","The functoriality guarantee holds only for deterministic transformations, so the modularity claim would need re-testing if stochastic or learnable stages are inserted into the pipeline."],"forward_implications":["Researchers can create task-specific MNIST-style benchmarks without collecting and hand-labeling data from scratch.","In smart-batch mode, a 5,000-image dataset can be curated in about 5 hours rather than 25, with 15% of images needing human review.","Reinforcement-learning filtering, trained on the same pipeline, adds 5–8% downstream classification accuracy over CLIP-only selection.","Because the pipeline is modeled as composable morphisms, swapping in a new semantic model or preprocessing step should preserve consistency across stages."],"supporting_citations":[{"why":"Defines the MNIST format and benchmark the framework emulates.","marker":"[1]"},{"why":"FashionMNIST is the standard extension whose scope MNIST-Gen generalizes.","marker":"[2]"},{"why":"Snorkel is the programmatic weak-supervision baseline that MNIST-Gen contrasts with its hierarchical semantic pipeline.","marker":"[17]"},{"why":"CLIP supplies the semantic text–image embeddings that carry the categorization scoring.","marker":"[23]"},{"why":"Provides the applied category-theory formalism for the composable pipeline.","marker":"[24]"},{"why":"U2-Net performs the background-removal transformation in the preprocessing chain.","marker":"[29]"},{"why":"The Food-11 dataset is the source of the 5,000 images used to build and evaluate Food-MNIST.","marker":"[30]"}],"fun_headline_variants":["Generate custom MNIST-style datasets with 85% auto-labeling","MNIST-Gen: build your own benchmark in 80% less time","CLIP+RL pipeline cuts dataset creation time by 80%","From trees to food: on-demand MNIST datasets via RL","Category theory meets RL for custom MNIST benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that CLIP-based semantic similarity, validated by human review, is a good enough proxy for true category membership, because the images come from unlabeled public repositories and the paper states there is no definitive ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Generate custom MNIST-style datasets with 85% auto-labeling","MNIST-Gen: build your own benchmark in 80% less time","CLIP+RL pipeline cuts dataset creation time by 80%","From trees to food: on-demand MNIST datasets via RL","Category theory meets RL for custom MNIST benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1334,"prompt_tokens":971,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":275}},"tokens_in":587,"tokens_out":363,"duration_ms":4323,"temperature":1.0,"reasoning_tokens":275,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:00:54.833419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the smart-batch pipeline on an image set with fixed ground-truth labels, such as a held-out subset of Food-101, and measure the auto-assigned categories against those labels; if the auto-categorization agreement is far below 85% while human review still corrects many errors, the headline accuracy and the claimed time savings would not survive a labeled benchmark.","supporting_citations":[{"cited_title":"Snorkel: Rapid training data creation with weak supervision,","cited_arxiv_id":null,"evidence_quote":"Snorkel is the programmatic weak-supervision baseline that MNIST-Gen contrasts with its hierarchical semantic pipeline."},{"cited_title":"Fong and D","cited_arxiv_id":null,"evidence_quote":"Provides the applied category-theory formalism for the composable pipeline."},{"cited_title":"U2-net: Going deeper with nested u-structure for salient object detection,","cited_arxiv_id":null,"evidence_quote":"U2-Net performs the background-removal transformation in the preprocessing chain."},{"cited_title":"Food/non-food image classifi- cation and food categorization using pre-trained googlenet model,","cited_arxiv_id":null,"evidence_quote":"The Food-11 dataset is the source of the 5,000 images used to build and evaluate Food-MNIST."}],"review_version":1}