{"id":"0e1b76b9-e74f-47dd-91cb-bc0ffa20a603","arxiv_id":"2506.16116","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Combining natural and dermatology image quality datasets to train a CNN improves dermatology image quality prediction compared with dermatology-only training, but the claimed 'optimal across domains' result is only partially supported.","lead":"The authors built a new artificially distorted dermatology image dataset with human quality ratings and trained neural networks on it together with six general-purpose image quality datasets. They report that this cross-domain training improves quality scoring for dermatology photos, which could help teledermatology services screen out unusable images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own B5 results contradict the central claim: DIQA-only beats All on the dermatology test set (PLCC 0.867 vs 0.836; SROCC 0.847 vs 0.813), so 'cross-domain training provides better results' holds only for B0/B3 and lacks significance testing.","rationale":"The reader selected MOS rescaling as the weakest assumption, but the paper's own tables contain a more direct and more damaging problem: for the largest model tested, dermatology-only training beats cross-domain training on the target domain. This is not a subtle scaling artifact; it is a visible inversion of the claimed effect. The paper's conclusion ('cross-domain training ... provides better results than training exclusively on dermatology data') is asserted as a general statement, yet it fails for EfficientNet-B5 in both PLCC and SROCC. The authors' suggestion that B5 All is overfitting is speculative and does not explain why B5 DIQA-only generalizes so well on the same test distribution. Additionally, the claim that cross-domain training yields 'optimal performance across domains' is contradicted on nearly every natural dataset by single-domain baselines in the same tables. Without significance testing, even the B0/B3 improvements could be noise; the reported standard deviations are large enough that the differences may not be reliable. Together, these points indicate that the central conclusion is not established by the presented evidence. The verdict should move from CONDITIONAL to REJECT because the internal results actively refute the paper's headline claim, and the missing data and code prevent verification. That said, the dataset construction and cross-subject split design are reasonable contributions, and a revised claim limited to smaller models with significance testing might be salvageable, but as written the argument fails.","tokens_in":15081,"tokens_out":3593,"duration_ms":39321,"concrete_test":"Recompute the DIQA test-set comparison between All and DIQA-only for B0, B3, and B5, reporting per-split paired differences and a paired significance test (e.g., Wilcoxon signed-rank or paired t-test over the 5 splits). If B5 is not significantly better for All—or is significantly worse—and if B0/B3 differences are not significant, then the central claim of superior cross-domain training for dermatology IQA is not supported by the evidence.","verdict_should_be":"REJECT","load_bearing_attack":"The central assertion (Abstract; Section 5) is that cross-domain training ('All') yields better dermatology IQA than dermatology-only training. Tables 3 and 4 directly undermine this for the largest model. On the DIQA test set, B5 trained only on DIQA reaches PLCC 0.8666±0.0400 and SROCC 0.8466±0.0420, while B5 trained on All reaches PLCC 0.8363±0.0349 and SROCC 0.8125±0.0364. The direction flips, and the paper's dismissal of this as 'overfitting' (Section 5) is unexplained—if anything, the DIQA-only B5's high in-domain correlation indicates that additional natural data can hurt at larger capacity. The broader claim of 'optimal performance across domains' is also contradicted by the same tables: single-domain models outperform All on their own datasets (e.g., B0 KonIQ-10k 0.9134 vs All 0.8759; B0 SPAQ 0.9133 vs All 0.9052; B0 GFIQA-20k 0.9648 vs All 0.9610). Because no significance tests are reported, the apparent B0/B3 benefits (0.055 and 0.029 PLCC) could be within split-to-split noise. The MOS rescaling limitation (Discussion) is a plausible mechanism for label mismatch, but the B5 reversal is a direct, internal inconsistency that does not depend on rescaling assumptions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes to improve no-reference image quality assessment (IQA) for dermatology by jointly training an EfficientNet on natural-domain IQA databases (KonIQ-10k, SPAQ, GFIQA-20k, Kadid-10k, BIQ2021, LIVE-ItW) and on a newly introduced artificially distorted dermatology image database, Legit.Health-DIQA-Artificial, annotated by 40 non-expert observers. The authors compare three model sizes (B0, B3, B5) trained on each single dataset and on all datasets combined, reporting PLCC and SROCC on every test set across five random splits. They conclude that cross-domain training yields optimal performance across domains and improves dermatology IQA relative to dermatology-only training.","tokens_in":15418,"tokens_out":5630,"duration_ms":57954,"significance":"The new dermatology IQA dataset, despite being limited to artificial distortions, fills a gap in the field and could serve as a benchmark for future work. The use of subject-wise splits to prevent leakage between a pristine image and its distorted views is a methodological strength that improves the reliability of the internal comparisons. However, the central claim of the paper is not supported by its own results: B5 trained only on the dermatology dataset outperforms B5 trained on all datasets, and single-domain models outperform the All model on some natural datasets. The lack of significance testing and the acknowledged MOS rescaling problem further weaken the evidence. If the claims were properly scoped and supported by statistical analysis, this could be a useful contribution to teledermatology and cross-domain IQA.","major_comments":[{"comment":"The central claim that cross-domain training 'provides better results than training exclusively on dermatology data' is contradicted by the EfficientNet-B5 results on the DIQA test set: B5 trained only on DIQA reaches PLCC 0.8666±0.0400 and SROCC 0.8466±0.0420, whereas B5 trained on All reaches PLCC 0.8363±0.0349 and SROCC 0.8125±0.0364. The paper's dismissal of this as 'overfitting' (Section 5) is unexplained; the DIQA-only B5 actually shows the highest in-domain correlation, so additional natural data appears to hurt rather than help at this capacity. The conclusion must either be restricted to B0/B3 or supported by an analysis of why the B5 All model underperforms.","section":"Tables 3 and 4; Section 5"},{"comment":"The abstract's claim that cross-domain training 'yields optimal performance across domains' is not supported by the paper's own tables: single-domain models outperform All on their own test sets in several cases, e.g., B0 KonIQ-10k PLCC 0.9134 vs. 0.8759, B0 SPAQ 0.9133 vs. 0.9052, B0 GFIQA-20k 0.9648 vs. 0.9610, and B0 KonIQ-10k SROCC 0.8925 vs. 0.8427. If the claim means something weaker, such as 'competitive on natural datasets and improved on the dermatology dataset for smaller models,' it should be stated precisely.","section":"Table 3; Abstract"},{"comment":"The validity of the cross-domain training rests on the commensurability of the MOS scores from different datasets after linear rescaling to [1,10]. The paper itself concedes in the Discussion that without dataset-specific MOS scaling, images from different quality-feature distributions may be assigned similar scores, reducing the effectiveness of combining datasets. Since this admission directly applies to the central experiment, the paper should either demonstrate that the reported improvements are robust to alternative rescaling schemes, or temper the conclusion accordingly.","section":"Section 3.3; Section 5"},{"comment":"No statistical significance testing is reported: all conclusions are drawn from means and standard deviations over five random splits, and some differences central to the claims fall within one standard deviation or have overlapping intervals (e.g., B0 All vs. B0 DIQA on DIQA PLCC 0.8021±0.0550 vs. 0.7470±0.0525). The paper should report paired significance tests (e.g., Wilcoxon signed-rank or paired t-test across splits) for the comparisons that underlie the main claim.","section":"Section 4"}],"minor_comments":[{"comment":"Typographical errors: 'entirely different form a natural image' should be 'entirely different from a natural image', and 'independent form each other' should be 'independent from each other'.","section":"Discussion and Section 3.3"},{"comment":"In the concluding section, 'DIQA modles' should be corrected to 'DIQA models'.","section":"Conclusion"},{"comment":"The statement that the dataset and code are not publicly available limits reproducibility; the authors should at least release the quality ratings in anonymized form or explain the intellectual property restrictions in more detail.","section":"Data Availability Statement"},{"comment":"The paper does not compare against any existing NR-IQA baselines (e.g., BRISQUE, NIQE, or other deep learning models), so the absolute correlation values are difficult to interpret; adding such baselines would strengthen the evaluation.","section":"Section 3.1 and Table 1"},{"comment":"The description of the splitting strategy is confusing: Kadid-10k is said to have image clusters, yet the next sentence states that 'all images from Kadid-10k, KonIQ-10k, GFIQA-20k, and SPAQ are independent samples'; please clarify whether the split unit is a cluster or an individual image for each dataset.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper repeatedly builds on the authors' preliminary work [32] without a clear novelty statement; the relationship to that earlier paper should be made explicit. In addition, the non-availability of the dataset and code is a concern for a paper whose main contribution is a new benchmark. The manuscript fits the scope of the conference but needs substantial revision to align the claims with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2506.16116. The paper has a real asset: a new dermatology IQA dataset (Legit.Health-DIQA-Artificial) with 40 human ratings per image, and a careful subject-wise split to avoid leakage. That part is solid. The systematic comparison across six natural IQA databases and three EfficientNet sizes is also a reasonable effort, and the idea of pooling natural and dermatology data to compensate for small medical datasets is a legitimate question in the teledermatology niche.\n\nThe problem is the headline claim. The abstract and Section 5 say cross-domain training yields 'optimal performance across domains,' but the tables don't support that. On the dermatology test set, the B5 model trained only on DIQA beats the B5 trained on All (PLCC 0.867 vs 0.836; SROCC 0.847 vs 0.813). On the natural datasets, single-domain models outperform All on their own test sets (e.g., B0 KonIQ 0.913 vs All 0.876; B0 SPAQ 0.913 vs All 0.905). The benefit of All is seen only for B0/B3 on DIQA, and without significance tests those gaps could be split noise. The paper dismisses the B5 reversal as 'overfitting,' but nothing in the results supports that explanation.\n\nThere's also a deeper issue the authors concede: they naively rescale each dataset's scores to [1,10] without proper dataset-specific MOS scaling. That undermines the premise that pooling the labels produces commensurable training targets. And since the data and code aren't released, the results aren't independently checkable. No external NR-IQA baselines are compared either, so the only benchmark is the paper's own internal comparison.\n\nWho would get value from this? Someone building a practical teledermatology quality filter might use the dataset as a starting point, and the experimental setup is instructive. But the central claim needs to be reworked, with significance tests and a direct comparison to existing IQA models. That's a major revision, not a trivial fix.\n\nMy call: send it to peer review. The dataset and careful subject-wise split deserve referee time, and the flaws are addressable. But if I were the editor, I'd make clear that the abstract and conclusions need to be aligned with the actual evidence before acceptance.","headline":"The paper's own tables undercut its central claim about cross-domain training, but the new dermatology IQA dataset and systematic evaluation still deserve peer review with major revisions.","tokens_in":15995,"tokens_out":2765,"would_cite":false,"duration_ms":29869,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training an image-quality model jointly on natural and dermatology image datasets outperforms dermatology-only training for predicting the quality of skin images, and supports this with a new 1,800-image dermatology…","keywords":["image quality assessment","teledermatology","cross-domain training","no-reference image quality","deep learning","mean opinion score","dermatology imaging","EfficientNet"],"falsifier":"Re-run the cross-domain versus dermatology-only comparison after replacing naive linear rescaling of mean opinion scores with dataset-specific calibration, for instance a per-dataset monotonic mapping to a common perceptual scale; if the 'All' models then no longer outperform dermatology-only models on Legit.Health-DIQA-Artificial, the reported benefit is an artifact of score rescaling rather than genuine cross-domain transfer.","tokens_in":14875,"feed_emoji":"🩺","tokens_out":6595,"duration_ms":64835,"temperature":0.7,"pith_summary":"Teledermatology works only when patient photos are good enough for a clinician to read, but automated quality checking for skin images has little data and has not used the large natural-image quality databases that exist for everyday photos. This paper tries to establish that a model trained jointly on natural image-quality datasets and a new dermatology image-quality dataset outperforms a model trained on dermatology data alone, while staying accurate on natural images. To make the test possible, the authors built and rated a new dermatology benchmark from 100 pristine skin images distorted in seven ways, producing 1,800 annotated images with 40 human ratings each. If the claim holds, teledermatology platforms could automatically flag poor-quality patient photos before review, and other small medical-imaging domains could borrow the same cross-domain recipe.","feed_headline":"Cross-domain training sharpens dermatology image quality scores","feed_subtitle":"Mixing natural and dermatology photo ratings beats dermatology-only models, the paper reports.","key_machinery":"The mechanism is cross-domain training of a no-reference IQA regression model: a convolutional EfficientNet (B0, B3, or B5) with two fully-connected layers, trained with mean squared error loss to predict a scalar quality score, and evaluated with Pearson's linear correlation coefficient and Spearman's rank correlation coefficient. The enabling data object is the new Legit.Health-DIQA-Artificial dataset, built by taking 100 pristine dermatology images and applying seven distortion families (JPEG compression, Gaussian blur, pixelation, sharpening, brightness, color, contrast) at multiple strengths, yielding 1,800 images annotated by 40 non-expert observers on a 1-to-10 scale; subject-based splits prevent leakage between training and test views. To combine databases, the paper linearly rescales each natural dataset's scores to the 1-to-10 range and reweights classes to correct score imbalance. The argument turns on comparing single-domain models, natural-only models, and 'All' models across the same test sets.","core_discovery":"The central claim is that cross-domain training, training one no-reference image quality model simultaneously on several natural image quality databases and a new dermatology image quality database, yields better dermatology image quality prediction than training on dermatology data alone, while preserving or improving performance on natural images. The authors support this with a new benchmark, Legit.Health-DIQA-Artificial: 100 pristine dermatology images from multiple sources, each distorted in seven ways at multiple strengths to produce 1,800 rated images, with 40 non-expert ratings per image averaged into a mean opinion score. Across five random splits, EfficientNet models trained on all datasets ('All') achieve the highest average correlation across test sets and outperform single-domain models on the dermatology test set in most configurations, although the largest EfficientNet-B5 variants are noted as the stated exception.","pith_inferences":["Beyond the paper, the scaling question is left open; a direct next experiment would replace naive linear rescaling with dataset-specific MOS calibration and check whether the cross-domain advantage on dermatology images grows, shrinks, or disappears.","Because the new dermatology dataset contains only artificial distortions, the practical benefit for teledermatology still needs confirmation on authentic patient-taken photos with real blur, lighting, and framing problems.","The poor mutual transfer between the two artificial-distortion datasets, Kadid-10k and the new DIQA set, hints that single-domain models may be memorizing content rather than distortions; this could be tested by measuring performance on distorted versions of unseen content.","If the cross-domain recipe generalizes, other small medical imaging fields with scarce quality ratings, such as retinal or dermoscopic imaging, could follow the same pattern: pool large natural IQA databases with a small curated medical set rather than collecting thousands of medical ratings."],"forward_implications":["In most model sizes tested, the 'All' cross-domain models beat both natural-only and dermatology-only models on the new dermatology test set; the largest EfficientNet-B5 variant is the stated exception, where the paper suspects overfitting.","No single natural dataset is enough: the best natural-only model (SPAQ) reaches roughly 0.64 correlation on dermatology images, well below the cross-domain models, so relying on a natural IQA model alone is not reliable for teledermatology.","Cross-domain models also keep strong accuracy on natural test sets, roughly 0.86 to 0.96 PLCC on KonIQ-10k, SPAQ, and GFIQA-20k, so one model can serve multiple domains instead of requiring a separate dermatology model.","Increasing input resolution and model size helps cross-domain training on most datasets, whereas for single-domain training the effect is dataset-dependent."],"supporting_citations":[{"why":"Supplies KonIQ-10k, a large natural authentic-distortion database used as one of the training sources.","marker":"[16]"},{"why":"Supplies Kadid-10k, a large artificially distorted database whose distortion types inspired the new dermatology dataset's distortions.","marker":"[21]"},{"why":"Supplies SPAQ, the natural dataset whose single-domain models transfer best to the dermatology test set.","marker":"[10]"},{"why":"Supplies GFIQA-20k, a large natural database used in training and testing.","marker":"[45]"},{"why":"Supplies BIQ2021, a large natural database used for training and validation only because of its fixed splits.","marker":"[1]"},{"why":"Supplies LIVE In the Wild, the common benchmark used as an external test set across all models.","marker":"[11]"},{"why":"The authors' prior DIQA work, which supplies the initial method and evidence that this paper extends.","marker":"[32]"},{"why":"Supplies the restricted augmentation strategy (resize, random horizontal flip, limited random crop) appropriate for IQA training.","marker":"[46]"},{"why":"Supplies EfficientNet, the convolutional architecture used for all models in the study.","marker":"[47]"},{"why":"Supplies the class-weighting scheme used to balance the quality-score distribution during training.","marker":"[6]"}],"fun_headline_variants":["Cross-domain training improves skin image quality scoring","Mixing dermatology and natural photo ratings boosts IQA","New benchmark: cross-domain training outperforms dermatology-only","Combining datasets produces better dermatology image quality models","Training on multiple domains sharpens teledermatology image checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that linearly rescaling each natural image-quality database's mean opinion scores to the same 1-to-10 range makes scores from different databases comparable enough that pooling them into one training set helps rather than hurts; the paper itself concedes that without dataset-specific scaling, images with different quality features can end up with similar scores.","fun_headline_variants_meta":{"raw":{"variants":["Cross-domain training improves skin image quality scoring","Mixing dermatology and natural photo ratings boosts IQA","New benchmark: cross-domain training outperforms dermatology-only","Combining datasets produces better dermatology image quality models","Training on multiple domains sharpens teledermatology image checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1836,"prompt_tokens":913,"completion_tokens":923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":845}},"tokens_in":529,"tokens_out":923,"duration_ms":9818,"temperature":1.0,"reasoning_tokens":845,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:46.827608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the cross-domain versus dermatology-only comparison after replacing naive linear rescaling of mean opinion scores with dataset-specific calibration, for instance a per-dataset monotonic mapping to a common perceptual scale; if the 'All' models then no longer outperform dermatology-only models on Legit.Health-DIQA-Artificial, the reported benefit is an artifact of score rescaling rather than genuine cross-domain transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies KonIQ-10k, a large natural authentic-distortion database used as one of the training sources."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Kadid-10k, a large artificially distorted database whose distortion types inspired the new dermatology dataset's distortions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SPAQ, the natural dataset whose single-domain models transfer best to the dermatology test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GFIQA-20k, a large natural database used in training and testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies BIQ2021, a large natural database used for training and validation only because of its fixed splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies LIVE In the Wild, the common benchmark used as an external test set across all models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The authors' prior DIQA work, which supplies the initial method and evidence that this paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the restricted augmentation strategy (resize, random horizontal flip, limited random crop) appropriate for IQA training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the class-weighting scheme used to balance the quality-score distribution during training."}],"review_version":1}