{"id":"9d7f2d54-1dad-4bab-afe3-1f8543fba73c","arxiv_id":"1908.08505","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ColorNet, a CNN fine-tuned on a newly aligned 180-image subjective dataset, reports Pearson correlation 0.937 with human colorfulness ratings, beating classical colorfulness metrics.","lead":"The authors train a convolutional neural network, ColorNet, to rate the colorfulness of natural images, and combine two existing subjective colorfulness databases into one 180-image training set. A VGG-based version reports higher correlation with human colorfulness ratings than four classical image-processing metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The combined ground-truth scale is fit from 12 anchor images and 3 viewers per source; a leave-one-out refit of Eq. 1 would show whether the Table 1 margin is label artifact or real.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the linear alignment in Eq. 1, fitted on 12 images per database with three expert viewers, is assumed to hold for all 180 images. This is the most fundamental issue because the alignment error contaminates both the training labels and the evaluation target; any subsequent significance test, error bar, or architecture comparison is conditional on those labels. I considered alternative concerns, including the absence of uncertainty quantification, the fact that ColorNet-ResNet has SROCC 0.889 slightly below CQE1's 0.896, and the lack of released code or data. These are real but secondary: they can be addressed after the ground-truth scale is shown to be stable. The leave-one-out anchor refit is a cheap, decisive check. If it passes, the conditional acceptance can later be upgraded; if it fails, the headline comparison in Table 1 cannot be trusted. Therefore I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":8440,"tokens_out":6385,"duration_ms":71943,"concrete_test":"For each source database, perform leave-one-out on the 12 anchor images: fit Eq. 1 on the other 11 images, predict the held-out Anchor score, and compute RMSE and worst-case error on the 1-9 Anchor scale. Then refit the complete Table 1 evaluation 12 times, each time omitting one anchor image from the alignment and regenerating the Combined labels; track the VGG-versus-CQE1 PCC/SROCC gap. If the leave-one-out Anchor RMSE exceeds roughly 1 unit on the 1-9 scale, or if the VGG advantage becomes nonsignificant under a test for correlated correlations (e.g., Steiger's test) in any refit, the central claim is not established beyond the chosen alignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 constructs the Combined 180-image ground truth by fitting y = ax + b (Eq. 1) between Anchor scores and each source database score. The fit uses only 12 selected images per database and Anchor scores from only three expert viewers, yet the fitted mapping is applied to all images in both databases. The paper states that the relationship is linear with 'very high correlation scores' but reports neither the correlations, residuals, nor confidence intervals for a and b. The validation in Fig. 1(e), plotting Combined vs Anchor, is in-sample: the same 24 images were used to estimate the mapping, so it cannot certify extrapolation to the remaining 156 images. If the true source-to-anchor relationship is nonlinear away from the selected points, or if the three-viewer Anchor scale is unstable, every training label in the Combined dataset is biased. Because ColorNet-VGG is trained end-to-end on those labels, it can absorb and reproduce that bias, while the classical metrics in Table 1 cannot adapt to it. Thus the reported PCC advantage of ColorNet-VGG (0.937) over CQE1 (0.895) and the SROCC advantage (0.921 vs 0.896) may be inflated by label miscalibration rather than by genuinely better perceptual estimation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ColorNet, the first deep learning-based colorfulness estimation metric for natural images. Because no sufficiently large colorfulness dataset exists, the authors combine two existing subjective databases (EPFL and UCL) using a new 'Anchor' subjective test on 24 selected images rated by three expert viewers. They fit linear mappings y = ax + b between the Anchor scores and each source database's scores, yielding a 180-image combined dataset with aligned subjective scores. They then fine-tune three CNN feature extractors (VGG, ResNet, MobileNet) with a shared rating network and evaluate them using 10-fold cross-validation. Table 1 reports that ColorNet-VGG achieves Pearson correlation 0.937 and Spearman correlation 0.921 with the aligned subjective scores, outperforming Hasler, Panetta CQE1/CQE2, and Yendrikhovskij. The authors also provide qualitative examples of monotonic responses to color saturation and hue changes.","tokens_in":8764,"tokens_out":3313,"duration_ms":35745,"significance":"If the reported correlations are robust, this is a useful contribution: it introduces a learning-based alternative to classical colorfulness formulas, and the dataset-alignment procedure addresses a real practical bottleneck (the absence of a shared subjective scale across databases). The held-out 10-fold evaluation is an appropriate and more trustworthy design than fitting and testing on the same images, and the authors compare against four standard classical metrics. However, the central quantitative claim rests on an alignment fitted from only 24 images and three expert viewers, and the reported PCC/SROCC differences come without uncertainty quantification or significance testing. These issues must be addressed before the superiority claim over classical metrics can be considered established.","major_comments":[{"comment":"The linear alignment y = ax + b is fitted on only 12 selected images per database, with Anchor scores obtained from just three expert viewers. The paper states that the relationship is linear with 'very high correlation scores' but does not report the correlations, residuals, or confidence intervals for a and b. Because the aligned scores are used both as training labels and as the ground truth for the quantitative comparison in Table 1, any miscalibration of Eq. (1) propagates directly into the claimed advantage of ColorNet-VGG. Please report alignment fit diagnostics and perform a leave-one-anchor-out or bootstrap refit of Eq. (1) to show that the PCC/SROCC rankings in Table 1 are stable under realistic variation in the anchor mapping.","section":"Section 3, 'Combining the Datasets', Eq. (1)"},{"comment":"The best ColorNet variant is selected by comparing VGG-, ResNet-, and MobileNet-based features on the same 10-fold splits, yet no error bars, confidence intervals, or significance tests accompany the reported PCC and SROCC values. With only 18 test images per fold, the difference between ColorNet-VGG (0.937) and CQE1 (0.895) could lie within sampling noise, especially since model selection is performed on the same evaluation data. Please provide per-fold results, bootstrap confidence intervals, or a paired significance test (e.g., Wilcoxon on per-fold correlations or Fisher z-transformation) to support the claim that ColorNet-VGG outperforms the classical methods.","section":"Section 5, 'Quantitative Evaluation' and Table 1"},{"comment":"The selected 12 images per database are described as representative because they 'cover the whole quality scale' of the source scores, but no evidence is given that they are representative under the Anchor experiment's subjective judgments. The validation in Fig. 1(e) plots Combined versus Anchor scores using the same 24 images that were used to estimate the alignment, so it is an in-sample check and cannot certify extrapolation to the remaining 156 images. Please provide an external validation (e.g., a small hold-out set of newly rated images) or explicitly discuss this limitation and its possible effect on the combined labels.","section":"Section 3, 'Selection of Images from Each Dataset'"}],"minor_comments":[{"comment":"The running header reads 'ESTIMA TING' with an internal space; this appears to be a typesetting artifact and should be corrected.","section":"Title and header"},{"comment":"The loss function sums over k but uses the index i for the image and rating; please make the index notation consistent, e.g., sum over i of |y_i - phi^j_r(phi^j_f(X_i))|.","section":"Eq. (2)"},{"comment":"The description of the 10-fold protocol says one piece is used for validation in each iteration, which implies training uses only 8 of 10 pieces; please clarify how the validation piece is chosen and how the final averaged results relate to the validation split.","section":"Section 5, 'Training and Implementation Details'"},{"comment":"The phrase 'the first deep learning-based colorfulness estimation metric' should be qualified as 'to the best of our knowledge' to be precise about novelty.","section":"Abstract and Introduction"},{"comment":"Figure 3 shows only a few hand-picked examples without quantitative agreement among observers; please either state that this is illustrative only or add a small multi-rater study to support the qualitative claim.","section":"Section 5, 'Qualitative Evaluation'"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the combined ground-truth scale is built from a very small anchor experiment (24 images, 3 expert viewers), and the evaluation does not quantify uncertainty. If the authors can demonstrate that the Table 1 ordering survives a refitting of Eq. (1) and add significance tests, the paper would be much stronger. I would also encourage the authors to release the combined dataset and code, which would increase the reproducibility and impact of this work. The paper is within the scope of the journal and the central idea is sound; the issues are fixable with additional analysis rather than being fundamental."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the first deep-learning colorfulness estimator, and the way they merge two subjective datasets via a common-anchor experiment is a real contribution. The soft spot is also the core: the anchor scores come from only three expert viewers, and the linear alignment is fit on just 12 images per source database, with no reported residuals or confidence intervals.\n\nWhat the paper does well. The idea is straightforward and honest: take two existing colorfulness databases with different scaling methods (rating vs. pairwise), run a small anchor study on a common subset, fit a linear mapping, concatenate into 180 labeled images, then fine-tune a CNN on those labels. They compare three CNN backbones against four classical formulas using PCC and SROCC. ColorNet-VGG comes out ahead (0.937/0.921 vs. CQE1 at 0.895/0.896). The qualitative check on crafted images is a nice sanity test. The writing is clear and the related work covers the classical baselines well.\n\nWhere it gets shaky. The anchor experiment uses three expert viewers. That is very few for psychometric scaling, even with an adaptive design. The paper says the relationship between anchor scores and source scores is linear with \"very high correlation\" but gives no numbers, and the validation plot in Fig. 1(e) uses the same 24 images that defined the mapping. So there is no evidence the linear fit extrapolates to the other 156 images. If the true relationship is even mildly nonlinear, every training label in the Combined set inherits that bias, and a flexible CNN can absorb it while the classical metrics cannot. That would inflate the reported margin. Also: no error bars or significance tests on the PCC/SROCC difference, the best architecture (VGG) is chosen on the same 180-image dataset, and no code or aligned dataset is released. The 25-second inference time is impractical but not central to the claim.\n\nIs the central argument salvageable? Yes. The comparison is internally consistent—classical metrics are evaluated on the same aligned labels—so the paper isn't circular. But the headline advantage could shrink under a more robust ground truth. A leave-one-out refit of the alignment parameters, a few more anchor viewers, and error bars would settle it. Those are reasonable revisions, not a rejection.\n\nWho this is for: anyone working on image quality assessment, tone mapping, or aesthetics who wants a colorfulness predictor and needs to know a learning-based option exists. It deserves a serious referee, but I'd ask the authors to release the dataset and report the anchor diagnostics before treating Table 1 as authoritative.","headline":"First learning-based colorfulness metric, and the dataset-alignment trick is genuinely useful; the weak point is the ground truth, built from 3 expert viewers and a linear fit over 12 images per source.","tokens_in":9220,"tokens_out":2265,"would_cite":true,"duration_ms":27669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned convolutional network predicts perceived colorfulness more accurately than classical hand-crafted metrics, with Pearson correlation 0.937 on a 180-image combined subjective dataset.","keywords":["colourfulness","CNN","color metric","deep learning","image quality assessment","subjective test alignment","psychometric scaling","color perception"],"falsifier":"Conduct a fresh subjective experiment on a held-out set of natural images, or re-run the Anchor experiment with many more non-expert viewers and a rank-based alignment, then compare ColorNet-VGG's predictions against the newly collected scores and against the classical opponent-color formula; if ColorNet-VGG no longer shows a higher Pearson or Spearman correlation with the fresh ratings, the claimed advantage is an artefact of the alignment. Alternatively, train and test ColorNet-VGG separately on each original database without the linear alignment; if the advantage over classical metrics disappears, the claim depends on the combined labels.","tokens_in":8281,"feed_emoji":"🎨","tokens_out":7353,"duration_ms":70034,"temperature":0.7,"pith_summary":"This paper sets out to show that a convolutional neural network can estimate the perceived colorfulness of a natural image more accurately than hand-crafted color-statistics formulas. Because no large annotated colorfulness dataset existed, the authors combine two public subjective databases by running a small anchoring experiment on a 24-image common subset and linearly remapping both databases onto one rating scale, yielding 180 images with aligned human scores. They then fine-tune a pretrained CNN as a feature extractor and attach a small rating network to predict colorfulness. In 10-fold cross-validation, the VGG-based ColorNet variant reaches Pearson correlation 0.937 and Spearman correlation 0.921 with human ratings, outperforming the four classical metrics tested. If correct, this establishes deep learning as a viable route to colorfulness estimation, which matters for tone-mapping, aesthetic analysis, and image-quality assessment.","feed_headline":"CNN rates image colorfulness closer to humans than classical formulas","feed_subtitle":"ColorNet-VGG scores 0.937 Pearson correlation with human ratings, beating four hand-crafted metrics on 180 images.","key_machinery":"The load-bearing component is the dataset-alignment procedure plus the two-stage CNN. For alignment, 12 images are selected from each of two public databases to cover the full colorfulness range, and three expert viewers compare them pairwise using an adaptive square design, scaled by Thurstone Case V psychometric scaling to a rating scale of 1 to 9. A linear mapping $y = ax + b$ is fitted between the anchor scores and each database's original scores, with $a_{\\mathrm{EPFL}}=0.8748$, $b_{\\mathrm{EPFL}}=1.4350$ and $a_{\\mathrm{UCL}}=1.1388$, $b_{\\mathrm{UCL}}=6.8759$, producing the 180-image Combined dataset. The model itself is the feature network followed by the rating network, trained end-to-end with the L1 loss, and the best configuration is ColorNet-VGG.","core_discovery":"ColorNet is a two-stage model: a feature network taken from an ImageNet-pretrained CNN (VGG16, ResNet18, or MobileNet) with its classification head removed, followed by a rating network consisting of a dropout layer, a 10-unit fully connected layer, a ReLU, and a single output unit. It is trained end-to-end with an L1 loss on 180 images whose subjective colorfulness scores were aligned by a linear mapping through an anchoring experiment. The paper's central result is that ColorNet-VGG predicts these aligned human ratings with Pearson correlation 0.937 and Spearman correlation 0.921, compared with 0.841/0.884 for the classical opponent-color formula, 0.895/0.896 and 0.312/0.415 for the two logarithmic CQE versions, and 0.843/0.834 for the saturation-based model. The authors interpret this as evidence that learned high-level features capture colorfulness information that low-level color statistics miss, and as a first deep-learning step in color-perception modeling.","pith_inferences":["The reported advantage may be partly tied to the specific linear alignment and the very small anchor set; a rank-based or probabilistic alignment would likely change absolute scores but might preserve the relative ordering of models.","Because the 10-fold protocol leaves only 18 test images per fold, the performance difference between ColorNet-VGG and the classical formula should be re-examined with confidence intervals or significance tests, which the paper does not report.","The same two-stage transfer-learning recipe could be applied to other subjective perceptual attributes, such as brightness, naturalness, or contrast, using anchor-aligned databases, and might yield similar gains over hand-crafted metrics.","ColorNet-Mobile's lower performance suggests that for this perceptual regression task, depth and residual connections matter more than parameter efficiency; depth-wise separable convolutions may lose color-sensitive gradient information."],"forward_implications":["Colorfulness can be treated as a perceptual attribute learnable from a small, aligned dataset, rather than as a fixed statistical formula.","The anchor-alignment method offers a template for merging subjective datasets collected under different psychophysical protocols (rating versus pairwise comparison) into one training set for a deep metric.","Fine-tuned ImageNet features already encode enough color information to outperform specially designed opponent-color statistics, suggesting that other low-level perceptual attributes, such as contrast or naturalness, may also be learnable this way.","A dataset of only 180 images is sufficient for a first learning-based colorfulness model, providing a starting point for scaling to larger and more diverse color-perception data."],"supporting_citations":[{"why":"Supplies the EPFL subjective colorfulness dataset with rating-scale scores and the opponent-color-statistics baseline that ColorNet must beat.","marker":"[5]"},{"why":"Supplies the UCL subjective dataset with pairwise-comparison scores, the other component of the 180-image combined dataset.","marker":"[19]"},{"why":"Defines the saturation-based colorfulness index used as one of the four classical baselines.","marker":"[16]"},{"why":"Defines the two CQE colorfulness metrics in logarithmic opponent-color space that serve as baselines.","marker":"[17]"},{"why":"Contributes the low-cost common-set method for aligning subjective tests, which the paper adapts with an Anchor experiment.","marker":"[31]"},{"why":"Provides the psychometric scaling method and software used to convert pairwise comparisons into colorfulness scores.","marker":"[29]"},{"why":"Supplies the VGG16 architecture whose features are fine-tuned in the best-performing ColorNet-VGG model.","marker":"[22]"},{"why":"Supplies the ResNet architecture used for the ColorNet-ResNet variant.","marker":"[23]"},{"why":"Supplies the MobileNet architecture used for the ColorNet-Mobile variant.","marker":"[24]"}],"fun_headline_variants":["AI colorfulness score beats classical formulas in human match","ColorNet: deep learning rates image color like humans","First deep metric for image colorfulness outperforms classical","CNN colorfulness rating aligns better with human perception"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The linear transformation $y = ax + b$ that aligns each public database to the Anchor scores is fitted on only 12 selected images per database judged by 3 expert viewers, and the paper assumes this fit extends to all 180 images; if the anchor ratings are unreliable or the relationship is not linear, the combined ground-truth scores become miscalibrated and the reported correlations cannot be trusted.","fun_headline_variants_meta":{"raw":{"variants":["AI colorfulness score beats classical formulas in human match","ColorNet: deep learning rates image color like humans","First deep metric for image colorfulness outperforms classical","CNN colorfulness rating aligns better with human perception"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1362,"prompt_tokens":887,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":503,"tokens_out":475,"duration_ms":4966,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:37:56.586173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct a fresh subjective experiment on a held-out set of natural images, or re-run the Anchor experiment with many more non-expert viewers and a rank-based alignment, then compare ColorNet-VGG's predictions against the newly collected scores and against the classical opponent-color formula; if ColorNet-VGG no longer shows a higher Pearson or Spearman correlation with the fresh ratings, the claimed advantage is an artefact of the alignment. Alternatively, train and test ColorNet-VGG separately on each original database without the linear alignment; if the advantage over classical metrics disappears, the claim depends on the combined labels.","supporting_citations":[{"cited_title":"In our paper, we removed the three fully connected layers of the VGG16 architecture and simply fed the resulting feature vector into the proposed rating network","cited_arxiv_id":null,"evidence_quote":"Supplies the EPFL subjective colorfulness dataset with rating-scale scores and the opponent-color-statistics baseline that ColorNet must beat."},{"cited_title":"Learning-Based Tone Mapping Operator for Image Matching,","cited_arxiv_id":null,"evidence_quote":"Supplies the UCL subjective dataset with pairwise-comparison scores, the other component of the 180-image combined dataset."},{"cited_title":"Calibrated image appearance reproduction,","cited_arxiv_id":null,"evidence_quote":"Defines the saturation-based colorfulness index used as one of the four classical baselines."},{"cited_title":"Learning-based tone mapping operator for efficient image matching,","cited_arxiv_id":null,"evidence_quote":"Defines the two CQE colorfulness metrics in logarithmic opponent-color space that serve as baselines."},{"cited_title":"Very deep convolutional networks for large-scale image recognition,","cited_arxiv_id":null,"evidence_quote":"Contributes the low-cost common-set method for aligning subjective tests, which the paper adapts with an Anchor experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the psychometric scaling method and software used to convert pairwise comparisons into colorfulness scores."},{"cited_title":"Improving color reproduction ac- curacy on cameras,","cited_arxiv_id":null,"evidence_quote":"Supplies the VGG16 architecture whose features are fine-tuned in the best-performing ColorNet-VGG model."},{"cited_title":"Analysis of public image and video databases for qual- ity assessment,","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet architecture used for the ColorNet-ResNet variant."},{"cited_title":"Selecting scenes for 2D and 3D subjective video quality tests,","cited_arxiv_id":null,"evidence_quote":"Supplies the MobileNet architecture used for the ColorNet-Mobile variant."}],"review_version":1}