Human Perceptual Evaluations for Image Compression

R. Manmatha; Srikar Appalaraju; Yash Patel

REVIEW 3 major objections 5 minor 20 references

Human Perceptual Evaluations for Image Compression

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep-learning codecs with higher MS-SSIM can look worse to people than engineered codecs, a four-dataset user study finds.

desk verdict Useful cautionary result about MS-SSIM, but the statistical foundation is too weak to carry the conclusion as reported. read the letter →

arxiv 1908.04187 v1 pith:NUE4IVFO submitted 2019-08-09 eess.IV cs.CV

Yash Patel , Srikar Appalaraju , R. Manmatha This is my paper · ORCID

classification eess.IVcs.CV

keywords imagecompressionperceptualqualityMS-SSIMPSNRhumanevaluationtwo-alternativeforcedchoicedeeplearningBPG

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the standard perceptual similarity metric MS-SSIM, and to a lesser degree PSNR, can rank image compression methods in a way that contradicts what human viewers prefer. The authors reproduce two learned codecs (Mentzer et al., Ballé et al.) and compare them against two engineered codecs (BPG, JPEG-2000) at four bit rates on four standard datasets. Crowd-sourced pairwise judgments show BPG is preferred over the learned methods even though the learned methods have higher MS-SSIM. If this holds, metric-only comparisons in the learned-compression literature can be misleading about which method actually looks better.

What carries the argument

The central object is MS-SSIM (multi-scale structural similarity), a metric comparing luminance, contrast, and structure across image scales; it serves both as the training loss for the learned codecs and as the headline evaluation metric whose reliability is tested. The testing instrument is a two-alternative forced-choice (2AFC) protocol: a viewer sees the original with two compressed versions and a synchronized magnifier, and must choose the more similar version. Six pairwise comparisons among two learned and two engineered codecs are run at four bit rates (0.23, 0.37, 0.67, and 1.0 bpp) over 143 images, with five votes per pair aggregated by majority to decide which method wins each image. This aggregation is what exposes the mismatch between metric ordering and human preference.

What would settle it

Re-run the same six pairwise comparisons with screened, expert raters and enough votes per pair to compute inter-rater reliability and confidence intervals. If expert preference no longer favors BPG over the learned codecs at matched bit rates, or if the preference flips once attention checks are enforced, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms: optimizing MS-SSIM as a training loss produces codecs whose outputs score higher on MS-SSIM but are judged visually worse by human observers than BPG, an engineered codec with lower MS-SSIM. Across Kodak, Urban100, Set14, and Set5, human majority votes favor BPG at essentially all bit rates; when BPG is removed, Mentzer et al. leads at low bit rates and JPEG-2000 at high bit rates, again not matching the MS-SSIM or PSNR orderings. The author concludes that MS-SSIM and PSNR are not reliable proxies for perceptual quality in this setting.

Load-bearing premise

The conclusion rests on crowd workers' five unvalidated votes per image pair being a correct measure of perceptual quality; if those votes are noisy or biased, the observed reversal of MS-SSIM rankings could be an artifact of the voting procedure.

Editorial extensions

If this is right

If the claim is correct, published comparisons that rank learned codecs above engineered ones purely by MS-SSIM can invert the true perceptual ordering.
MS-SSIM-optimized models can overfit the metric: their higher scores do not translate into images people prefer, so reporting only such scores is insufficient evidence of progress.
BPG, a non-learned codec, remains a strong perceptual baseline that learned methods should be required to beat in human evaluations, not just in metric values.
Releasing implementations or reconstructed images on standard datasets becomes a necessary part of compression papers, since metric numbers alone cannot support the quality claim.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A likely mechanism the paper leaves implicit is Goodhart-type metric overfitting: because learned codecs train directly against MS-SSIM, they exploit what the metric ignores, such as local blur, and a direct test would be whether models trained with a perceptual or adversarial loss rank closer to human judgments.
The finding suggests practical deployment decisions for image delivery should incorporate small human preference tests on representative content, since a codec that wins on MS-SSIM may deliver visibly worse text and edges to end users.
The paper does not report inter-rater agreement or significance tests; an extension would be to measure vote confidence, and one can predict that BPG's advantage concentrates in images with text, faces, or sharp edges, where blurring is most visible.
If the metric community responds by building a learned perceptual metric that correlates with 2AFC votes, the same pairwise-judgment data could be reused as a training set, making human evaluation pipelines a component of future codec development rather than a final check.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Here's my take on arXiv:1908.04187. The paper's core observation is that MS-SSIM can rank learned codecs above engineered ones while human observers prefer the engineered result. That's a useful caution for the compression community, and the qualitative pattern is visible in their plots. But as reported, the evidence doesn't carry the weight of the claim.

What's genuinely new: a pairwise human preference dataset for four codecs (JPEG-2000, BPG, Mentzer et al., Ball et al.) across four datasets and four bitrates, with a clean 2AFC interface and a synchronized magnifier. That's a legitimate empirical contribution, and the authors are honest enough to cite their own prior work [12] for the same general observation.

The soft spots are real and load-bearing. The statistical analysis is almost absent. Five binary votes per pair cannot yield per-pair significance even at unanimity (binomial p≈0.06). Aggregating to counts of 'best method' per image without a sign test, permutation test, or confidence interval makes the headline result an anecdote. No worker screening, attention checks, or inter-rater reliability are reported, so a few inattentive workers could sway the tallies. On top of that, the authors retrained Ball et al. with an MS-SSIM loss (footnote 1) rather than using the published model, so the comparison is against their re-instantiation, not the method as released. They also don't release the data or code, which hurts reproducibility.

I agree with the stress-test note: the central claim is not statistically grounded as it stands. That said, the qualitative finding is plausible and consistent with prior work, so I don't think this is a throwaway. It needs a serious revision—add proper significance testing, report worker demographics and screening, justify the retraining, and release the data. If those are addressed, this could be a useful reference.

My recommendation: send it to peer review, but with the expectation of major revision. The question is important enough to deserve referee time, and the study design is a reasonable starting point.

Best, [Your name]

Referee Report

3 major / 5 minor

Summary. The paper reports a human perceptual study comparing two engineered codecs (JPEG-2000 and BPG) with two learned codecs (Mentzer et al. and Ball et al.) at four bit rates (0.23, 0.37, 0.67, 1.0 bpp) on four standard datasets (Kodak, Urban100, Set14, Set5). Using a forced-choice 2AFC MTurk task with five judgments per pair, the authors tally, per image, which of the four methods is most preferred. They find that BPG is preferred over the learned methods even though the learned methods have higher MS-SSIM, and they conclude that MS-SSIM-based rankings of learned vs. engineered compression can be misleading. The paper also plots MS-SSIM and PSNR rate-distortion curves and compares them with the human-preference tallies.

Significance. If the central finding is reliable, it challenges the common practice of ranking learned image codecs by MS-SSIM and strengthens the case for evaluating compression with user studies or better perceptual metrics. The evaluation has a clear strength: the human judgments were collected independently of the metrics being criticized, so the core comparison is not circular, and the 2AFC protocol with a full-image view plus a synchronized magnifier is a reasonable design. The paper also covers multiple datasets and bit rates and reports the raw number of HITs. However, the claim as stated is not yet load-bearing because the paper lacks the statistical analysis needed to show that the vote tallies are distinguishable from chance or from worker noise, and because one of the learned methods is a retrained variant rather than the published model.

major comments (3)

[Section 3, Figures 3–7] The statement that 'BPG performs significantly better than all the other methods' is not supported by any significance test or confidence interval. With five binary votes per pair, a 4-1 tally has a two-sided binomial p of 0.375 under chance, so individual pair tallies cannot carry significance; the paper aggregates to the number of images for which a method is best, but no sign test, permutation test, or paired analysis across images is provided. A paired sign test across the 143 images, or a bootstrap confidence interval on the proportion of images for which BPG is preferred, would be needed to make the central claim statistically grounded.
[Section 2] The paper reports 17,160 HITs but gives no information about the number of unique workers, the distribution of votes across workers, or any quality-control measures such as attention checks or screening. Without this, the aggregate tallies could be driven by a small number of inattentive or systematically biased workers; the authors should report inter-rater reliability (for example, Krippendorff's alpha or a simple agreement statistic on the five votes per pair) and the number of unique workers per condition.
[Footnote 1] The Ball et al. model was retrained by the authors with an MS-SSIM loss rather than the MSE loss used in the original paper, and the claim that this 'does not change the general conclusions' is an assertion rather than a demonstrated fact. Since the paper's title and abstract are about deep learning techniques in general and specifically name Ball et al., the authors should either use the published pretrained models or provide evidence (e.g., comparable rate-distortion behavior) that their retrained variant represents the method as released. Without this, the comparison may describe the authors' reimplementation rather than the named method.

minor comments (5)

[Section 1, Figure 1 caption] The caption reports an MS-SSIM ordering ('Mentzer et al. > Ball et al. > BPG > JPEG-2000') without specifying the bpp at which the example is taken; the bit rate and the metric values should be stated.
[Section 3, paragraph 3] The text reads 'MS-SIM is still a better loss function than PSNR'; this should be 'MS-SSIM' for consistency with the rest of the paper.
[Section 2] The exact wording of the MTurk task instructions is not reported; providing the worker-facing instructions and the layout of the HIT would improve reproducibility of the study.
[Section 3, Figures 3 and 5] The metric curves are described as interpolated between trained checkpoints, but the figures do not mark the measured data points; adding markers at the actual bpp values would clarify the interpolation and allow readers to verify the comparisons.
[Section 4] The sentence 'Note that similar observations have been drawn in [12]' cites the authors' own preprint without stating what those observations are; either summarize the relevant finding or cite independent work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's conclusion rests on external human-preference judgments, not on a fitted parameter or self-citation chain.

full rationale

The paper's central claim is that learned compression methods can obtain higher MS-SSIM scores yet be judged worse by human observers. This is an empirical, externally grounded comparison: the human judgments are collected independently of the metric values being criticized, via pairwise 2AFC evaluations on Amazon Mechanical Turk (Section 2). No parameter is fitted from the human data to force the conclusion, and no equation is derived from the target claim. The MS-SSIM and PSNR curves (Section 3) are standard metric computations, and the human-vote tallies are independent measurements. The only self-citation, reference [12], appears in the Discussion as 'Note that similar observations have been drawn in [12]' — it is a supporting remark, not the load-bearing evidence for the paper's own user study. Concerns about the statistical reliability of the MTurk votes (small numbers of judgments per pair, absence of attention checks or significance tests) are correctness or robustness concerns, not circularity: the votes are not constructed from the MS-SSIM rankings. Because there is no derived quantity that reduces to an input by definition, no fitted input relabeled as prediction, and no load-bearing self-citation chain, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the reliability of the MTurk judgments and the representativeness of the datasets, not on any derived model parameters. The only hand-set number that directly affects the outcome is the number of votes per pair.

free parameters (1)

Number of human votes per pair = 5
The authors chose five judgments per comparison pair without a power analysis; this sample size controls the confidence of each per-image winner and is a hand-set experimental parameter.

assumptions (3)

domain assumption Two-alternative forced choice (2AFC) with whole images plus a linked magnifier is a valid method for measuring perceptual similarity to the original.
The study's ground truth is built on this protocol; no validation or comparison against other subjective methods is provided in this paper.
domain assumption Amazon Mechanical Turk workers, without screening or attention checks, provide reliable perceptual judgments representative of human viewers.
Section 2 describes the MTurk setup but reports no qualification requirements, attention checks, or inter-rater agreement.
domain assumption Kodak, Urban100, Set14, and Set5 are representative test sets for image compression evaluation.
These are standard datasets in the literature; the paper assumes they generalize to general photographic content.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Human Perceptual Evaluations for Image Compression." pith.science (2026). https://pith.science/paper/NUE4IVFO

@misc{pith2026190804187,
  author       = {Pith},
  title        = {Pith review of: Human Perceptual Evaluations for Image Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUE4IVFO}},
  note         = {Machine review of arXiv:1908.04187}
}

read the original abstract

Recently, there has been much interest in deep learning techniques to do image compression and there have been claims that several of these produce better results than engineered compression schemes (such as JPEG, JPEG2000 or BPG). A standard way of comparing image compression schemes today is to use perceptual similarity metrics such as PSNR or MS-SSIM (multi-scale structural similarity). This has led to some deep learning techniques which directly optimize for MS-SSIM by choosing it as a loss function. While this leads to a higher MS-SSIM for such techniques, we demonstrate using user studies that the resulting improvement may be misleading. Deep learning techniques for image compression with a higher MS-SSIM may actually be perceptually worse than engineered compression schemes with a lower MS-SSIM.

Figures

Figures reproduced from arXiv: 1908.04187 by the authors.

**Figure 2.** Sample instance from MTurk HIT. Entire images are [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Evaluation on the Kodak dataset [7] using MS-SSIM (left), PSNR (middle) and human study (right). In this case, Ball [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Human study on the Kodak (left most) [7], Urban100 (second from left) [8], Set14 (second from right) [20] and Set5 (right most) [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: Evaluation on the Urban100 dataset [8] using MS-SSIM (left), PSNR (middle) and human study (right). [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]

**Figure 6.** Figure 6: Evaluation on Set14 dataset [20] using MS-SSIM (left), PSNR (middle) and human study (right). In this case, Ball [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 7.** Figure 7: Evaluation on Set5 dataset [4] using MS-SSIM (left), PSNR (middle) and human study (right). [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

[12]

Deep Perceptual Compression

Y . Patel, S. Appalaraju, and R. Manmatha. Deep Perceptual Compression. arXiv preprint arXiv:1907.08310

work page Pith review arXiv 1907
[1]

Ball ´e, V

J. Ball ´e, V . Laparra, and E. P. Simoncelli. End- to-end optimized image compression. arXiv preprint arXiv:1611.01704, 2016

arXiv 2016
[2]

Ball ´e, D

J. Ball ´e, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior.arXiv preprint arXiv:1802.01436, 2018

arXiv 2018
[3]

F. Bellard. Bpg image format, 2014

work page 2014
[4]

Bevilacqua, A

M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi- Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012

work page 2012
[5]

T. Boutell. Png (portable network graphics) speciﬁcation version 1.0. Technical report, 1997

work page 1997
[6]

How much data do we create every day? the mind- blowing stats everyone should read, 2018

Forbes. How much data do we create every day? the mind- blowing stats everyone should read, 2018

work page 2018
[7]

R. Franzen. Kodak lossless true color image suite. source: http://r0k. us/graphics/kodak, 4, 1999

work page 1999

Show all 20 references

[8]

Huang, A

J.-B. Huang, A. Singh, and N. Ahuja. Single image super- resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015

2015
[9]

Johnston, D

N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. J. Hwang, J. Shor, and G. Toderici. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. structure, 10:23, 2017

2017
[10]

J. Lee, S. Cho, and S.-K. Beack. Context-adaptive entropy model for end-to-end optimized image compression. ICLR, 2019

2019
[11]

Mentzer, E

F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool. Conditional probability models for deep im- age compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4394– 4402, 2018

2018
[13]

Rippel and L

O. Rippel and L. Bourdev. Real-time adaptive image com- pression. arXiv preprint arXiv:1705.05823, 2017

2017 arXiv
[14]

Russakovsky, J

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. In- ternational journal of computer vision , 2015

2015
[15]

M. S. Sajjadi, B. Sch ¨olkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In Computer Vision (ICCV), 2017 IEEE Interna- tional Conference on, pages 4501–4510. IEEE, 2017

2017
[16]

C. E. Shannon. A mathematical theory of communication. Bell system technical journal , 27(3):379–423, 1948

1948
[17]

Skodras, C

A. Skodras, C. Christopoulos, and T. Ebrahimi. The jpeg 2000 still image compression standard. IEEE Signal pro- cessing magazine, 2001

2000
[18]

G. K. Wallace. The jpeg still picture compression standard. IEEE transactions on consumer electronics , 1992

1992
[19]

Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multiscale struc- tural similarity for image quality assessment. In The Thrity- Seventh Asilomar Conference on Signals, Systems & Com- puters, 2003, volume 2, pages 1398–1402. Ieee, 2003

2003
[20]

Zeyde, M

R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces. Springer, 2010. 5

2010

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [12]

Deep Perceptual Compression

Y . Patel, S. Appalaraju, and R. Manmatha. Deep Perceptual Compression. arXiv preprint arXiv:1907.08310

work page Pith review arXiv 1907

[2] [1]

Ball ´e, V

J. Ball ´e, V . Laparra, and E. P. Simoncelli. End- to-end optimized image compression. arXiv preprint arXiv:1611.01704, 2016

arXiv 2016

[3] [2]

Ball ´e, D

J. Ball ´e, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior.arXiv preprint arXiv:1802.01436, 2018

arXiv 2018

[4] [3]

F. Bellard. Bpg image format, 2014

work page 2014

[5] [4]

Bevilacqua, A

M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi- Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012

work page 2012

[6] [5]

T. Boutell. Png (portable network graphics) speciﬁcation version 1.0. Technical report, 1997

work page 1997

[7] [6]

How much data do we create every day? the mind- blowing stats everyone should read, 2018

Forbes. How much data do we create every day? the mind- blowing stats everyone should read, 2018

work page 2018

[8] [7]

R. Franzen. Kodak lossless true color image suite. source: http://r0k. us/graphics/kodak, 4, 1999

work page 1999