{"id":"f08b9b84-d189-4e45-a18d-4adb3a2963f1","arxiv_id":"1908.04187","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Human preference judgments contradict MS-SSIM rankings: learned compression methods with higher MS-SSIM are often judged visually worse than BPG and JPEG-2000.","lead":"This paper runs Amazon Mechanical Turk user studies comparing two deep learned compression methods against JPEG-2000 and BPG. It finds that human viewers prefer the engineered codecs even when the learned methods score higher on the MS-SSIM metric.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated MTurk votes are the load-bearing element: with five judgments per pair, no screening or attention checks, and no significance or reliability analysis, the claimed human-preference ranking is not statistically grounded.","rationale":"The reader identified the same weakest assumption: the MTurk judgments are treated as ground truth without validation. My read confirms that this is the most load-bearing part of the argument. The setup is otherwise reasonable: 2AFC is standard, the magnifying-glass interface is a good way to inspect local detail, and the datasets and bpp levels are sensible. But none of that matters if the five votes per pair come from unsupervised, possibly inattentive workers. The paper gives no evidence that the vote tallies would replicate, and the lack of significance tests means even the qualitative 'BPG is best' conclusion is not established. I also note a secondary concern about the retrained learned codecs, but the human-judgment issue is sufficient to keep the verdict conditional. If the proposed reanalysis is performed and the preference survives, the central claim would be substantially strengthened.","tokens_in":4430,"tokens_out":5255,"duration_ms":55347,"concrete_test":"Release or reconstruct the per-image, per-pair vote counts and fit a mixed-effects logistic regression with worker and image random effects for each bpp and dataset, using BPG vs Mentzer as the focal contrast, and report the 95% confidence interval and inter-rater reliability. If the BPG preference remains significant after accounting for worker and image variance, and the effect is not carried by a handful of workers, then the reliability objection is resolved; otherwise the paper's central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the MS-SSIM ordering of the four codecs is contradicted by human perception. The only evidence for that contradiction is the 2AFC MTurk voting summarized in Section 3, but the reliability of those votes is never demonstrated. Five binary judgments per image-pair is very low: a unanimous 5-0 pair has a two-sided binomial p of 0.0625 under chance, and a 4-1 pair has p = 0.375, so individual pairs cannot carry significance. The paper aggregates to the number of images for which a method is best without a sign test, permutation test, confidence interval, or inter-rater agreement measure. No screening or attention checks are reported, and no information is given about how many unique workers produced the 17,160 HITs; a few inattentive or systematic workers could dominate the tallies. The authors also state in Section 3 that BPG performs significantly better without any statistical test. Because the entire conclusion that MS-SSIM rankings are misleading depends on these vote counts, the claim is not yet supported beyond anecdote. A secondary issue is that both learned codecs were retrained by the authors (with Ball et al. switched to an MS-SSIM loss) rather than using the published pretrained models, so the result may describe their retrained instantiations rather than the methods as released, but the human-judgment reliability problem is the more immediate blocker.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a human perceptual study comparing two engineered codecs (JPEG-2000 and BPG) with two learned codecs (Mentzer et al. and Ball et al.) at four bit rates (0.23, 0.37, 0.67, 1.0 bpp) on four standard datasets (Kodak, Urban100, Set14, Set5). Using a forced-choice 2AFC MTurk task with five judgments per pair, the authors tally, per image, which of the four methods is most preferred. They find that BPG is preferred over the learned methods even though the learned methods have higher MS-SSIM, and they conclude that MS-SSIM-based rankings of learned vs. engineered compression can be misleading. The paper also plots MS-SSIM and PSNR rate-distortion curves and compares them with the human-preference tallies.","tokens_in":4638,"tokens_out":4001,"duration_ms":42084,"significance":"If the central finding is reliable, it challenges the common practice of ranking learned image codecs by MS-SSIM and strengthens the case for evaluating compression with user studies or better perceptual metrics. The evaluation has a clear strength: the human judgments were collected independently of the metrics being criticized, so the core comparison is not circular, and the 2AFC protocol with a full-image view plus a synchronized magnifier is a reasonable design. The paper also covers multiple datasets and bit rates and reports the raw number of HITs. However, the claim as stated is not yet load-bearing because the paper lacks the statistical analysis needed to show that the vote tallies are distinguishable from chance or from worker noise, and because one of the learned methods is a retrained variant rather than the published model.","major_comments":[{"comment":"The statement that 'BPG performs significantly better than all the other methods' is not supported by any significance test or confidence interval. With five binary votes per pair, a 4-1 tally has a two-sided binomial p of 0.375 under chance, so individual pair tallies cannot carry significance; the paper aggregates to the number of images for which a method is best, but no sign test, permutation test, or paired analysis across images is provided. A paired sign test across the 143 images, or a bootstrap confidence interval on the proportion of images for which BPG is preferred, would be needed to make the central claim statistically grounded.","section":"Section 3, Figures 3–7"},{"comment":"The paper reports 17,160 HITs but gives no information about the number of unique workers, the distribution of votes across workers, or any quality-control measures such as attention checks or screening. Without this, the aggregate tallies could be driven by a small number of inattentive or systematically biased workers; the authors should report inter-rater reliability (for example, Krippendorff's alpha or a simple agreement statistic on the five votes per pair) and the number of unique workers per condition.","section":"Section 2"},{"comment":"The Ball et al. model was retrained by the authors with an MS-SSIM loss rather than the MSE loss used in the original paper, and the claim that this 'does not change the general conclusions' is an assertion rather than a demonstrated fact. Since the paper's title and abstract are about deep learning techniques in general and specifically name Ball et al., the authors should either use the published pretrained models or provide evidence (e.g., comparable rate-distortion behavior) that their retrained variant represents the method as released. Without this, the comparison may describe the authors' reimplementation rather than the named method.","section":"Footnote 1"}],"minor_comments":[{"comment":"The caption reports an MS-SSIM ordering ('Mentzer et al. > Ball et al. > BPG > JPEG-2000') without specifying the bpp at which the example is taken; the bit rate and the metric values should be stated.","section":"Section 1, Figure 1 caption"},{"comment":"The text reads 'MS-SIM is still a better loss function than PSNR'; this should be 'MS-SSIM' for consistency with the rest of the paper.","section":"Section 3, paragraph 3"},{"comment":"The exact wording of the MTurk task instructions is not reported; providing the worker-facing instructions and the layout of the HIT would improve reproducibility of the study.","section":"Section 2"},{"comment":"The metric curves are described as interpolated between trained checkpoints, but the figures do not mark the measured data points; adding markers at the actual bpp values would clarify the interpolation and allow readers to verify the comparisons.","section":"Section 3, Figures 3 and 5"},{"comment":"The sentence 'Note that similar observations have been drawn in [12]' cites the authors' own preprint without stating what those observations are; either summarize the relevant finding or cite independent work.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is statistical: the human-preference ranking that drives the paper's conclusion is presented as a set of aggregate tallies with no significance tests, confidence intervals, or worker-reliability measures. This is fixable within the manuscript's scope by adding a sign test or permutation analysis, reporting worker counts and inter-rater reliability, and addressing the retrained-model issue for Ball et al. The self-citation [12] does not appear to be load-bearing and I do not see circularity, but the link to that preprint should be clarified in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my take on arXiv:1908.04187. The paper's core observation is that MS-SSIM can rank learned codecs above engineered ones while human observers prefer the engineered result. That's a useful caution for the compression community, and the qualitative pattern is visible in their plots. But as reported, the evidence doesn't carry the weight of the claim.\n\nWhat's genuinely new: a pairwise human preference dataset for four codecs (JPEG-2000, BPG, Mentzer et al., Ball et al.) across four datasets and four bitrates, with a clean 2AFC interface and a synchronized magnifier. That's a legitimate empirical contribution, and the authors are honest enough to cite their own prior work [12] for the same general observation.\n\nThe soft spots are real and load-bearing. The statistical analysis is almost absent. Five binary votes per pair cannot yield per-pair significance even at unanimity (binomial p≈0.06). Aggregating to counts of 'best method' per image without a sign test, permutation test, or confidence interval makes the headline result an anecdote. No worker screening, attention checks, or inter-rater reliability are reported, so a few inattentive workers could sway the tallies. On top of that, the authors retrained Ball et al. with an MS-SSIM loss (footnote 1) rather than using the published model, so the comparison is against their re-instantiation, not the method as released. They also don't release the data or code, which hurts reproducibility.\n\nI agree with the stress-test note: the central claim is not statistically grounded as it stands. That said, the qualitative finding is plausible and consistent with prior work, so I don't think this is a throwaway. It needs a serious revision—add proper significance testing, report worker demographics and screening, justify the retraining, and release the data. If those are addressed, this could be a useful reference.\n\nMy recommendation: send it to peer review, but with the expectation of major revision. The question is important enough to deserve referee time, and the study design is a reasonable starting point.\n\nBest,\n[Your name]","headline":"Useful cautionary result about MS-SSIM, but the statistical foundation is too weak to carry the conclusion as reported.","tokens_in":5194,"tokens_out":2589,"would_cite":false,"duration_ms":27289,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep-learning codecs with higher MS-SSIM can look worse to people than engineered codecs, a four-dataset user study finds.","keywords":["image compression","perceptual quality","MS-SSIM","PSNR","human evaluation","two-alternative forced choice","deep learning","BPG"],"falsifier":"Re-run the same six pairwise comparisons with screened, expert raters and enough votes per pair to compute inter-rater reliability and confidence intervals. If expert preference no longer favors BPG over the learned codecs at matched bit rates, or if the preference flips once attention checks are enforced, the paper's central claim is falsified.","tokens_in":4195,"feed_emoji":"🖼️","tokens_out":4205,"duration_ms":41653,"temperature":0.7,"pith_summary":"This paper tries to establish that the standard perceptual similarity metric MS-SSIM, and to a lesser degree PSNR, can rank image compression methods in a way that contradicts what human viewers prefer. The authors reproduce two learned codecs (Mentzer et al., Ballé et al.) and compare them against two engineered codecs (BPG, JPEG-2000) at four bit rates on four standard datasets. Crowd-sourced pairwise judgments show BPG is preferred over the learned methods even though the learned methods have higher MS-SSIM. If this holds, metric-only comparisons in the learned-compression literature can be misleading about which method actually looks better.","feed_headline":"Higher MS-SSIM can mean worse-looking images, user study finds","feed_subtitle":"Deep-learning codecs outscore BPG and JPEG-2000 on the metric, but people prefer the engineered codecs.","key_machinery":"The central object is MS-SSIM (multi-scale structural similarity), a metric comparing luminance, contrast, and structure across image scales; it serves both as the training loss for the learned codecs and as the headline evaluation metric whose reliability is tested. The testing instrument is a two-alternative forced-choice (2AFC) protocol: a viewer sees the original with two compressed versions and a synchronized magnifier, and must choose the more similar version. Six pairwise comparisons among two learned and two engineered codecs are run at four bit rates (0.23, 0.37, 0.67, and 1.0 bpp) over 143 images, with five votes per pair aggregated by majority to decide which method wins each image. This aggregation is what exposes the mismatch between metric ordering and human preference.","core_discovery":"On the paper's own terms: optimizing MS-SSIM as a training loss produces codecs whose outputs score higher on MS-SSIM but are judged visually worse by human observers than BPG, an engineered codec with lower MS-SSIM. Across Kodak, Urban100, Set14, and Set5, human majority votes favor BPG at essentially all bit rates; when BPG is removed, Mentzer et al. leads at low bit rates and JPEG-2000 at high bit rates, again not matching the MS-SSIM or PSNR orderings. The author concludes that MS-SSIM and PSNR are not reliable proxies for perceptual quality in this setting.","pith_inferences":["A likely mechanism the paper leaves implicit is Goodhart-type metric overfitting: because learned codecs train directly against MS-SSIM, they exploit what the metric ignores, such as local blur, and a direct test would be whether models trained with a perceptual or adversarial loss rank closer to human judgments.","The finding suggests practical deployment decisions for image delivery should incorporate small human preference tests on representative content, since a codec that wins on MS-SSIM may deliver visibly worse text and edges to end users.","The paper does not report inter-rater agreement or significance tests; an extension would be to measure vote confidence, and one can predict that BPG's advantage concentrates in images with text, faces, or sharp edges, where blurring is most visible.","If the metric community responds by building a learned perceptual metric that correlates with 2AFC votes, the same pairwise-judgment data could be reused as a training set, making human evaluation pipelines a component of future codec development rather than a final check."],"forward_implications":["If the claim is correct, published comparisons that rank learned codecs above engineered ones purely by MS-SSIM can invert the true perceptual ordering.","MS-SSIM-optimized models can overfit the metric: their higher scores do not translate into images people prefer, so reporting only such scores is insufficient evidence of progress.","BPG, a non-learned codec, remains a strong perceptual baseline that learned methods should be required to beat in human evaluations, not just in metric values.","Releasing implementations or reconstructed images on standard datasets becomes a necessary part of compression papers, since metric numbers alone cannot support the quality claim."],"supporting_citations":[{"why":"Defines MS-SSIM, the metric whose reliability is tested and which the learned codecs optimize as a loss.","marker":"[19]"},{"why":"One of the two learned codecs reproduced and evaluated; its high MS-SSIM is shown not to match human preference.","marker":"[11]"},{"why":"The other learned codec reproduced and evaluated, trained here with an MS-SSIM loss instead of the original MSE loss.","marker":"[1]"},{"why":"BPG, the engineered codec that wins most human pairwise comparisons despite lower MS-SSIM.","marker":"[3]"},{"why":"JPEG-2000, the engineered codec used as a baseline in all pairwise comparisons.","marker":"[17]"},{"why":"Supplies the two-alternative forced-choice protocol for perceptual similarity evaluation.","marker":"[15]"},{"why":"The authors' related work making similar observations about perceptual metrics, cited as independent support.","marker":"[12]"},{"why":"ImageNet, the training data used to reproduce the learned codecs at the four bit rates.","marker":"[14]"}],"fun_headline_variants":["MS-SSIM not a reliable proxy for perceived image quality","Higher MS-SSIM, worse visual quality: user study","Deep learning codecs beat metric, lose to BPG in human test","Metric says better, eyes say worse: image compression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on crowd workers' five unvalidated votes per image pair being a correct measure of perceptual quality; if those votes are noisy or biased, the observed reversal of MS-SSIM rankings could be an artifact of the voting procedure.","fun_headline_variants_meta":{"raw":{"variants":["MS-SSIM not a reliable proxy for perceived image quality","Higher MS-SSIM, worse visual quality: user study","Deep learning codecs beat metric, lose to BPG in human test","Metric says better, eyes say worse: image compression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2470,"prompt_tokens":797,"completion_tokens":1673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":1604}},"tokens_in":413,"tokens_out":1673,"duration_ms":12559,"temperature":1.0,"reasoning_tokens":1604,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:09:27.393461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same six pairwise comparisons with screened, expert raters and enough votes per pair to compute inter-rater reliability and confidence intervals. If expert preference no longer favors BPG over the learned codecs at matched bit rates, or if the preference flips once attention checks are enforced, the paper's central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MS-SSIM, the metric whose reliability is tested and which the learned codecs optimize as a loss."},{"cited_title":"Mentzer, E","cited_arxiv_id":null,"evidence_quote":"One of the two learned codecs reproduced and evaluated; its high MS-SSIM is shown not to match human preference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BPG, the engineered codec that wins most human pairwise comparisons despite lower MS-SSIM."},{"cited_title":"Skodras, C","cited_arxiv_id":null,"evidence_quote":"JPEG-2000, the engineered codec used as a baseline in all pairwise comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the two-alternative forced-choice protocol for perceptual similarity evaluation."},{"cited_title":"Deep Perceptual Compression","cited_arxiv_id":"1907.08310","evidence_quote":"The authors' related work making similar observations about perceptual metrics, cited as independent support."},{"cited_title":"Russakovsky, J","cited_arxiv_id":null,"evidence_quote":"ImageNet, the training data used to reproduce the learned codecs at the four bit rates."}],"review_version":1}