{"id":"4e7570f9-1a74-466e-9b17-27448b092905","arxiv_id":"2510.06596","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A regression-based composite of existing image-quality and trainability metrics, called SDQM, correlates with YOLOv11 mAP (r=0.87 on one split; 0.78±0.09 k-fold) on three synthetic-to-real detection benchmarks.","lead":"The paper introduces SDQM, a composite score meant to rate how useful a synthetic image dataset will be for training an object detector, claiming it predicts YOLOv11 accuracy better than existing quality metrics. If it holds up, teams building synthetic datasets could screen many candidates without running a full train-evaluate cycle.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Model/feature selection on the full dataset before the train/eval split makes the reported r=0.87 an optimistic estimate; redo selection inside CV folds.","rationale":"The reader's weakest_assumption focused on the 10-epoch mAP ground truth and the unsound feature-extractor criterion, but their rationale also identified selection-before-split as a concern. I agree that the 10-epoch target is fragile, but I judge the more load-bearing issue to be the leaky selection pipeline: the paper's most quantitative evidence (r=0.87, k-fold 0.78) is computed after the regression family and feature subsets were chosen on the full dataset. This directly inflates the central correlation and undermines the claim that SDQM is a predictive surrogate. The proposed concrete test—nested CV with all selection inside folds—would settle whether the correlation survives honest held-out evaluation. Because the reader already assigned CONDITIONAL and this concern is within their rationale, the verdict does not need to change; it remains CONDITIONAL pending the test.","tokens_in":15294,"tokens_out":2863,"duration_ms":24608,"concrete_test":"Run nested cross-validation: for each outer fold, perform all model/feature/sub-component selection (choice of random forest vs linear/XGBoost, removal of MAUVE/Authenticity/Clusterability, and the Appendix E p-value-based term selection) using only the training portion, then evaluate on the held-out fold. Report the mean Pearson and Spearman across outer folds. If the mean Pearson drops below roughly 0.6–0.7 or is no better than a simple linear baseline fit inside folds, the r=0.87 claim is not supported as an unbiased held-out estimate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SDQM predicts YOLOv11n mAP50 with r=0.87—rests on an evaluation split that is not truly held out. Section 3.3 performs the exploratory analysis that selects random forest as the regression method, removes MAUVE/Authenticity/Clusterability via backward feature reduction 'from the data,' and uses correlations in Fig. 1 computed over all collected datapoints. Appendix E further selects per-component statistical tests (A-D, ED, K-S) and quadratic/interaction terms via regressions on those same datapoints. Only in Section 4 are the collected values shuffled and split into train/eval. Thus the regression family, the feature subset, and the sub-component transforms are all chosen using evaluation datapoints. The k-fold mean (Pearson 0.78±0.09) does not fix this, because every fold reuses the same pre-selected feature set and regression family. The headline Table 2 r=0.8719 is therefore an optimistic estimate of how SDQM would perform on genuinely new dataset pairs. A separate but related fragility is that the ground truth is 10-epoch mAP50, never validated against converged training; if 10-epoch ranking differs from converged ranking, SDQM may predict an artifact. The selection leakage alone is sufficient to cast doubt on the headline correlation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SDQM, an integrated metric for evaluating synthetic object-detection datasets without requiring converged model training. SDQM combines established metrics (MAUVE, Frontier Integral, α-Precision/β-Recall/Authenticity, V-Usable Information) with new components (Dataset Separability, Clusterability, spatial/box/pixel/label distribution matches) and fuses them via random forest regression to predict YOLOv11n mAP50. Experiments use an evolutionary algorithm to create varied subset pairs from RarePlanes, DIMO, and WASABI, yielding 364 training and 64 evaluation subset pairs. The paper reports a held-out Pearson correlation of r=0.8719 for SDQM versus mAP50, with an average k-fold Pearson correlation of 0.78±0.091, and compares SDQM against several existing scalar metrics.","tokens_in":15489,"tokens_out":3126,"duration_ms":27942,"significance":"If the r=0.87 result were obtained under a genuinely clean held-out protocol, SDQM would be a practically valuable cheap surrogate for training-based dataset evaluation in object detection, with clear uses in synthetic-data generation, selection, and debugging. The paper has real strengths: it releases code, uses three diverse dataset pairs, evaluates multiple feature extractors, and reports k-fold variability. However, the headline correlation is currently not trustworthy because model and feature selection are performed on the full dataset before the train/eval split, and because the ground truth is only 10-epoch mAP50, never validated against converged training. These issues are fixable within the manuscript's scope, so the central idea is defensible but needs a substantially reworked evaluation.","major_comments":[{"comment":"Selection leakage invalidates the headline held-out r. Section 3.3 performs backward feature reduction on all collected data points to remove MAUVE, Authenticity, and Clusterability before the train/eval split described in Section 4. Appendix D selects the random forest regression family by comparing methods on the same full data, and Appendix E selects per-sub-metric statistical tests and quadratic/interaction terms via regressions on those same data points. The k-fold cross-validation in Section 4 reuses the pre-selected feature set and model family in every fold, so the reported r=0.8719 (Table 2) and k-fold mean 0.78±0.091 are optimistic. Please redo all feature/model/transform selection inside the training folds (nested CV) or otherwise demonstrate that the reported correlation holds on a truly untouched evaluation set.","section":"§3.3, §4, Appendices D/E"},{"comment":"The ground truth used throughout is mAP50 of YOLOv11n trained for only ten epochs. The paper never checks whether this 10-epoch score ranks datasets the same way as converged training. If short-training mAP is an artifact of optimization dynamics rather than dataset quality, SDQM may be predicting that artifact. Please validate the proxy by training at least a subset of the curated pairs to convergence (e.g., 50–100 epochs) and reporting rank correlation between 10-epoch and converged mAP50, or otherwise justify epoch choice.","section":"§4, Model Training"},{"comment":"The comparison against existing metrics is unfair and therefore not informative as reported. SDQM is a 9-feature random forest fitted to mAP50, while the baselines are raw scalar metrics evaluated without any calibration or fitting. A more appropriate comparison would apply the same fitting protocol to each baseline — e.g., fit each scalar metric through a monotonic regression on training folds and report cross-validated Pearson/Spearman correlations with confidence intervals. As presented, Table 2 conflates the benefit of the feature set with the benefit of regression fitting.","section":"Table 2"},{"comment":"The feature-extractor selection criterion is not sufficient to establish information quality. The paper selects GroundingDino-tiny because it yields high cosine similarity and low Euclidean distance between paired real and synthetic DIMO images. But a degenerate constant encoder would trivially maximize cosine similarity (all pairs identical) and minimize Euclidean distance. The criterion therefore does not measure whether the encoder preserves useful semantic information. Please add a control or a task-based probe, such as linear-probe classification accuracy or a comparison against random/constant embeddings.","section":"Appendix A"}],"minor_comments":[{"comment":"The abstract and contributions state that SDQM 'sets a new standard' and 'demonstrated a strong correlation (r=0.87)' without acknowledging the k-fold variability (0.78±0.091) and the selection-leakage caveat. Please moderate the claim or qualify it with the CV estimate.","section":"Abstract/§1"},{"comment":"The log cluster metric uses 'log' without a base; specify the base. Also clarify how the expression behaves when the squared term is zero or very small.","section":"§3.1, Eq. (1)"},{"comment":"The text says 'this process results in 364 training subsets and 64 evaluation subsets,' but the preceding description of the evolutionary algorithm and random additions is not fully formal. Please clarify whether the 64 evaluation subsets are disjoint from the training subsets and how the 'randomly selected' extra subsets were incorporated.","section":"§4"},{"comment":"Appendix E states that Dataset Separability regression assigns a negative weight to validation accuracy a and a positive weight to parameter count p, then says 'this establishes a positive correlation' with mAP50 where 'higher values ... indicate greater difficulty in distinguishing.' The sign logic is unclear: higher a should mean easier separation, so the intended combination needs a more precise explanation.","section":"Appendix E"},{"comment":"There are several typos and formatting issues: 'W ASABI' spacing, 'futhermore', 'displayes' (Appendix D), and Table 5's mixed text prompts. The code link is given, but no license or version/commit hash is provided for reproducibility.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the experimental setup is ambitious, but the evaluation protocol as written is not sufficient to support the headline r=0.87. The selection-leakage issue is the most serious; it should be addressed by moving all feature/model selection inside CV folds. The 10-epoch mAP proxy also needs validation against converged training. These are substantial but tractable changes, so I recommend major revision rather than rejection. I also suggest the authors compare SDQM against fitted, cross-validated versions of the baseline metrics, not just raw scalars."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious paper that deserves a referee, but the central number is not trustworthy as stated. The interesting contribution is not any single component—MAUVE, alpha-Precision, beta-Recall, Authenticity, V-usable information are all established—but the integration. The object-detection adaptation of V-Information (Eqs. 2–4 with YOLOv11n) is genuinely new, and Dataset Separability via AutoKeras plus the Clusterability log term are reasonable additions. The authors also deserve credit for being honest about the evolutionary algorithm failing to converge for most sub-metrics, and for reporting k-fold variance and Spearman, not just the best Pearson split. Code is public.\n\nThe soft spots are real and they hit the headline. Section 3.3 does backward feature reduction and regression-method selection on the full dataset before the train/eval split described in Section 4. That means the held-out split is not truly held out: the feature set, the regression family, and per-component transforms (Appendix E) were all chosen using evaluation datapoints. The k-fold mean of 0.78 Pearson doesn't fix it, because every fold reuses the same pre-selected pipeline. The honest estimate is the k-fold Spearman of 0.58, which is moderate, not strong. The baseline comparison in Table 2 is also structurally unfair—a fitted nine-feature random forest versus raw scalar metrics, with no FID or IS included. And the ground truth itself is fragile: 10-epoch YOLOv11n mAP is never validated against converged training, so SDQM might be predicting a training artifact rather than true dataset quality. The feature-extractor selection criterion in Appendix A—cosine similarity between real/synthetic DIMO pairs—would be trivially satisfied by a constant encoder, so that argument needs rethinking.\n\nNone of this is fatal; all of it is fixable. The paper identifies a real problem, proposes a sensible architecture, and reports enough detail for the flaws to be caught. The fix list: move all selection inside CV folds, fit the baselines as regressors too, add FID/IS, validate against longer training on at least one dataset, and replace the feature-extractor criterion. If those are done, the approach could be useful for sim-to-real dataset filtering.\n\nRecommendation: send to peer review with major revision. The problem is important, the integration is novel, and the evidence, honestly reported, is enough to warrant referee time.","headline":"The integrated-metric recipe is plausible and some pieces are new, but the headline r=0.87 is inflated by selection leakage and a 10-epoch mAP ground truth; the honest number is closer to the k-fold Spearman of 0.58.","tokens_in":16202,"tokens_out":1549,"would_cite":false,"duration_ms":14377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SDQM predicts object-detection mAP from dataset quality alone, at r=0.87.","keywords":["synthetic data","dataset quality","object detection","evaluation metric","mAP","domain gap","feature embeddings","random forest regression"],"falsifier":"Train object detectors to convergence (e.g., 100+ epochs) on a diverse set of real/synthetic subset pairs, compute SDQM for each pair, and measure the Pearson correlation between SDQM and converged mAP50. If the correlation drops well below 0.87—or below the correlation of a cheap baseline like β-Recall—the metric is an artifact of early-training dynamics rather than a true dataset-quality measure.","tokens_in":14960,"feed_emoji":"🎯","tokens_out":4581,"duration_ms":37898,"temperature":0.7,"pith_summary":"SDQM is an integrated metric that aims to score how useful a synthetic image dataset will be for training an object detection model, without requiring the model to be trained to convergence. The paper combines several existing and newly designed sub-metrics—covering pixel intensities, bounding-box geometry, label statistics, spatial layout, and embedding-space overlap—and feeds them into a random forest regressor that outputs a single quality score. The central claim is that this score correlates strongly (Pearson r=0.87 on one evaluation split; 0.78±0.091 in k-fold) with the mAP50 achieved by a YOLOv11n model trained on the dataset, far above existing metrics like α-Precision, β-Recall, and MAUVE. If true, SDQM gives a cheap screening tool for synthetic data generation and selection.","feed_headline":"SDQM predicts object-detection mAP without training: r=0.87","feed_subtitle":"New composite metric scores synthetic datasets from pixels, boxes, and embeddings, beating existing generative-model metrics.","key_machinery":"The central object is the SDQM score itself: a random forest regression over selected sub-metrics that maps a (real, synthetic) dataset pair to a single number. The regression is the mechanism that lifts correlation beyond any individual sub-metric, and it is trained on mAP50 values from a YOLOv11n model trained for ten epochs. The V-Information sub-metric—an extension of V-usable information to object detection—carries the most weight; it measures the change in predictive entropy of a COCO-pretrained YOLOv11n after brief fine-tuning on synthetic data. Feature embeddings for the distribution-comparison sub-metrics come from a GroundingDINO-tiny encoder, selected by cosine similarity between","core_discovery":"The paper claims that dataset quality for object detection can be captured by a composite of interpretable sub-metrics, independently of a converged training run. SDQM is built from components measuring pixel-intensity match, bounding-box match, label overlap, spatial distribution difference, dataset separability, clusterability, α-Precision, β-Recall, Authenticity, MAUVE, Frontier Integral, and an extension of V-usable information called V-Information. Sub-metrics are pruned via correlation and backward feature reduction, and the survivors are combined with random forest regression trained against actual YOLOv11n mAP50 scores on synthetic/real subset pairs selected by an evolutionary algori","pith_inferences":["Because the regression is trained on 10-epoch YOLOv11n mAP, the metric may be tuned to early-training behavior; its correlation with fully converged mAP is untested and could be weaker.","The feature-extractor selection criterion—cosine similarity between real and synthetic DIMO pairs—could be satisfied trivially by a constant encoder, so SDQM's validity depends on the encoder actually capturing semantics.","The reported r=0.87 comes from one train/eval split; the k-fold average is lower (0.78), so the headline correlation may overstate typical performance.","SDQM could be repurposed as an acquisition function for active learning or data pruning on real datasets, not just synthetic-data screening."],"forward_implications":["A team generating synthetic data can rank candidate datasets by SDQM before spending GPU-hours on training.","SDQM can be used as an objective in generation or selection loops, replacing costly train-and-evaluate iteration.","The metric's components give diagnostic feedback: a low Pixel Intensity Match or low β-Recall pinpoints why a synthetic dataset may underperform.","The same structure is portable to instance segmentation, localization, and classification by swapping the detector used for V-Information.","SDQM provides a benchmark for future research on mixed real-synthetic dataset curation."],"fun_headline_variants":["No training: SDQM predicts mAP, r=0.87","SDQM: score synthetic data, predict mAP without training","SDQM: composite metric outperforms prior synthetic-data scores","SDQM: evaluate synthetic data quality without costly training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The metric is validated against the mAP50 of a YOLOv11n model trained for only ten epochs on small, evolutionarily curated subset pairs, and the paper assumes that this short-training mAP orders datasets the same way as fully converged training.","fun_headline_variants_meta":{"raw":{"variants":["No training: SDQM predicts mAP, r=0.87","SDQM: score synthetic data, predict mAP without training","SDQM: composite metric outperforms prior synthetic-data scores","SDQM: evaluate synthetic data quality without costly training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3640,"prompt_tokens":722,"completion_tokens":2918,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2860}},"tokens_in":466,"tokens_out":2918,"duration_ms":16582,"temperature":1.0,"reasoning_tokens":2860,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T11:08:54.796844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train object detectors to convergence (e.g., 100+ epochs) on a diverse set of real/synthetic subset pairs, compute SDQM for each pair, and measure the Pearson correlation between SDQM and converged mAP50. If the correlation drops well below 0.87—or below the correlation of a cheap baseline like β-Recall—the metric is an artifact of early-training dynamics rather than a true dataset-quality measure.","supporting_citations":[],"review_version":1}