{"id":"b08b6147-47f5-4154-98e0-3c7746d1489e","arxiv_id":"2412.13137","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark of six lossy compression schemes for whole slide images shows deep-learning codecs outperform JPEG-like methods on quality metrics but fail to generalize, and proposes a deep feature similarity metric to replace task-specific downstream evaluation.","lead":"Researchers compared six lossy image compression methods on digital pathology slides, testing both human-perceived image quality and automated analysis accuracy. They propose a feature-similarity score as a shortcut for evaluating compression quality, and find that deep-learning codecs need fine-tuning per task and still fall short of clinical readiness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that feature cosine similarity 'aligns very well' with downstream performance rests only on a qualitative ordering at one bitrate; no quantitative correlation is reported, and the appendix shows a compression level that improves downstream accuracy over the original…","rationale":"The reader's weakest-assumption analysis identified the same load-bearing concern: the feature-similarity metric is not quantitatively validated and may not transfer across tasks, architectures, or compression artifacts. My stress-test sharpens this: the paper provides only a qualitative ordering at one bitrate, and the appendix contains a concrete empirical case where compression improves downstream performance over the original, violating the metric's foundational assumption that the original image is the upper baseline. This is not an external consensus disagreement but an internal consistency issue: the reported results in the appendix contradict the general claim made in the main text. The verdict should remain CONDITIONAL because the paper has other credible contributions (a benchmark of six compression schemes, the RMS dataset, and a negative finding on neural codec generalization), but the central metric claim requires quantitative validation before acceptance. The proposed test directly supplies the missing evidence by correlating the metric with observed downstream metrics across schemes and bitrates, and by checking the appendix's contradictory case.","tokens_in":15161,"tokens_out":2833,"duration_ms":29601,"concrete_test":"Compute Spearman rank correlation between the proposed mean cosine similarity (Ciga ResNet18 features) and actual downstream performance across the six compression schemes at multiple matched bitrates (e.g., 0.1, 0.3, 0.5, 0.7, 1.0) for both PanNuke Dice and Camelyon16 classification AUC. Report the correlation coefficient, its confidence interval, and a significance test. If the rank correlation is not significant or changes sign across bitrates, the claim that the metric 'aligns very well' with downstream performance is refuted. Additionally, using the appendix data, compute the Ciga feature cosine similarity for UNI's best-performing JPEG70 condition versus original; the metric must rank JPEG70 as superior to the original to be consistent with the observed AUC, which would directly test the core premise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central novel contribution is the task-agnostic metric: cosine similarity between deep features of original and compressed images. The only direct evidence in §4.2 is a qualitative statement that SQLC has the highest mean similarity and also performed best in the two downstream tasks. No correlation coefficient, scatter plot, statistical test, or error analysis is provided, and the metric is evaluated only at a single bitrate (0.5 bpp) on a combined test set. This does not establish that the metric 'aligns very well' with downstream performance across schemes, bitrates, or tasks. More seriously, the appendix's RMS classification experiments provide a direct counterexample to the metric's premise. Table 1 shows that the UNI foundation model achieves its best AUC (0.9575) with JPEG70 compression, outperforming the original uncompressed data (0.9383). The metric is based on the assumption stated in §3.2 that the original image is the upper baseline and that images more similar to the original yield comparable downstream performance. If compression can improve downstream accuracy relative to the original, then distance-to-original in feature space cannot be a universal proxy for downstream utility. The appendix result does not use the same Ciga features as the proposed metric, so it does not falsify the specific Ciga-based implementation, but it does undermine the general claim that the metric is task-agnostic and standardized. Absent a quantitative validation, the headline claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks six lossy compression schemes (JPEG, JPEG-XL, WebP, CAI, SPL2, SQLC) for whole-slide images in two tracks: perceptual quality (PSNR, MS-SSIM, LPIPS) and downstream tasks (nucleus segmentation with nnU-Net on PanNuke, and classification on Camelyon16). It introduces a task-agnostic evaluation metric computed as the cosine similarity between deep features of original and compressed images, using Ciga's ResNet18 pathology foundation model as the feature extractor. The authors claim this metric 'aligns very well' with actual downstream performance, allowing standardized evaluation without per-task testing. They also collect an 'initially uncompressed' RMS dataset for evaluating initial compression and report encode/decode times. The main contributions are the new metric, the broad comparative evaluation across perceptual and downstream criteria, and the uncompressed RMS dataset.","tokens_in":15359,"tokens_out":4606,"duration_ms":40984,"significance":"If the proposed task-agnostic metric were validated, it would provide a valuable, standardized tool for evaluating lossy compression schemes in digital pathology, reducing the need for expensive per-task downstream evaluation. The paper also provides a useful broad comparison of conventional and neural codecs, including a new uncompressed dataset and runtime analysis, which are strengths for reproducibility and practical uptake. However, the central validation of the metric is currently qualitative, and an appendix result directly contradicts the assumption that the original image is the upper baseline. The paper's strengths include the use of public datasets and checkpoints, transparent experimental design, and the inclusion of compute-time measurements.","major_comments":[{"comment":"The central claim that feature cosine similarity 'aligns very well' with downstream performance is not quantitatively supported. The only evidence is a qualitative observation that SQLC has the highest mean similarity and also performed best in the classification and segmentation tasks, at a single bitrate (0.5 bpp) on a combined test set. No correlation coefficient, scatter plot, or statistical test links the cosine similarity values to Dice scores or classification accuracy, and no evaluation across multiple bitrates or tasks is presented. Please provide a quantitative comparison across schemes and bitrates, with uncertainty estimates, to substantiate the claim.","section":"Section 4.2, Figure 6"},{"comment":"The appendix's RMS classification experiments provide a direct counterexample to the metric's premise. The UNI foundation model achieves its best AUC (0.9575) with JPEG70 compression, outperforming the original uncompressed data (0.9383). This contradicts the assumption stated in Section 3.2 that the original image is the upper baseline and that images more similar to the original yield comparable downstream performance. Although the appendix uses UNI features rather than the Ciga features used in the proposed metric, it undermines the general claim that the metric is task-agnostic. The paper should either restrict the claim to the specific feature extractor used or explain why this result does not violate the premise.","section":"Appendix, Table 1"},{"comment":"The main text states that 'we collected 100 randomly selected subjects' and extracted 1000 tiles per subject, resulting in 100,000 uncompressed tiles, while the Appendix states that the RMS dataset 'consists of 96 subjects' (52 Embryonal and 44 Alveolar). This inconsistency affects the reproducibility of the uncompressed dataset and the downstream experiments. Please clarify the correct dataset size and reconcile the numbers.","section":"Section 2.2, RMS dataset"},{"comment":"The paper asserts that the RMS data 'has not been previously lossy compressed' and is 'ideally suited to evaluate compression schemes for the initial compression,' but provides no verification that the WSI container or scanner pipeline introduced no lossy steps. Since both the perceptual evaluation and the proposed metric treat the original as the gold standard, this assumption is load-bearing. Please provide evidence from container metadata or scanner settings, or at least discuss the risk that prior lossy processing could bias the results.","section":"Section 2.2, RMS uncompressed claim"}],"minor_comments":[{"comment":"In the appendix, reference [6] is cited for the UNI foundation model, but the appendix reference list identifies [6] as OpenSlide; UNI is reference [8]. The citation should be corrected.","section":"Appendix, References"},{"comment":"The subscript in the cosine similarity notation appears garbled as 's!'#'; please use a readable symbol such as s_cos to denote the metric.","section":"Section 3.2, Eq. (1)"},{"comment":"The caption reads 'we the logarithmic time' and appears to be missing a verb; it should read 'we show the logarithmic time.'","section":"Figure 7 caption"},{"comment":"The appendix is not referenced anywhere in the main text. Add a cross-reference in Section 4 or 5 so readers are aware of the additional RMS classification experiments.","section":"Appendix"},{"comment":"There is a typo in the phrase 'WSI comression' in the discussion; it should be 'WSI compression.'","section":"Section 5, Discussion"},{"comment":"The statement 'we do not consider approaches with auto encoders, that solely perform dimensionality reduction' is confusing because the neural codecs in the study are autoencoders; clarify that the distinction is about dimensionality-reduction-only methods without quantization and bitstream coding.","section":"Section 5, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the task-agnostic metric, and its validation is currently too weak: the 'aligns very well' claim rests on a single qualitative observation at one bitrate. The appendix result with UNI is a serious challenge to the underlying assumption and should be directly addressed. I recommend major revision rather than rejection because the idea is promising and the criticism is addressable with additional quantitative analysis and a more careful statement of scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a useful benchmark paper: six lossy compression schemes, four datasets, perceptual and downstream tracks, plus a newly assembled uncompressed RMS dataset. Second, the headline novelty—a task-agnostic feature-similarity metric—is plausible but under-validated. The paper says the metric 'aligns very well' with downstream performance, but the evidence is a qualitative ordering at one bitrate, not a correlation, scatter, or test.\n\nThe strongest empirical contributions are the uncompressed RMS dataset and the finding that perceptual fine-tuning doesn't transfer: SPL2, trained on JPEG80 data, does poorly on uncompressed RMS, and neural codecs generalize poorly across compression schemes. The PanNuke segmentation and Camelyon16 classification experiments are reasonably structured, and the encoding/decoding time evaluation is a practical addition. The authors are also honest in the appendix, where they call the RMS classification experiments preliminary and list their own limitations.\n\nThe soft spots are real. The metric claim is the central contribution, and it rests on a single paragraph in §4.2. We don't get a single number linking cosine similarity to Dice or AUC. The metric is only evaluated at 0.5 bpp on a combined test set, so we can't tell whether the alignment holds across bitrates or tasks. The appendix gives a direct reason to be cautious about the 'task-agnostic' framing: the UNI foundation model gets its best AUC on JPEG70-compressed RMS data, beating the original uncompressed data. That doesn't falsify the specific Ciga-feature metric, since UNI uses different features, but it does undermine the assumption that distance-to-original is a universal proxy for downstream utility. There's also a reproducibility issue: the main text says 100 RMS subjects, the appendix says 96, and no code or evaluation scripts are provided. That's fixable but needs to be addressed.\n\nOverall this is a solid empirical contribution with an overclaimed metric. The benchmark itself is worth having, and the negative result about neural codec generalization is credible. The metric needs quantitative validation before the central claim is accepted.\n\nMy recommendation: send it to peer review. A good referee can push for the correlation analysis and the dataset consistency check. If those get fixed, the paper becomes a useful reference for the pathology compression community.","headline":"Useful compression benchmark with a valuable uncompressed RMS dataset, but the headline task-agnostic metric is only qualitatively validated and the appendix undercuts its universal framing.","tokens_in":16003,"tokens_out":2430,"would_cite":false,"duration_ms":22910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a task-agnostic metric, the cosine similarity between deep features of an original and a compressed whole-slide image, predicts downstream task performance well enough to standardize evaluation of lossy compression…","keywords":["whole slide images","lossy compression","feature similarity","digital pathology","learned image compression","perceptual image quality","downstream task performance","task-agnostic metric"],"falsifier":"Take any compression scheme not in the paper, compress the RMS test tiles, compute the proposed cosine similarity, and run a downstream task not used here, such as biomarker prediction or tissue classification with a different model architecture. If the ranking of codecs by feature similarity disagrees with the ranking by task accuracy, the general claim fails; the paper currently reports alignment only for a segmentation and a classification task, and only qualitatively.","tokens_in":14903,"feed_emoji":"🔬","tokens_out":8248,"duration_ms":70305,"temperature":0.7,"pith_summary":"The paper tries to establish that the quality of a lossy compression scheme for whole-slide pathology images can be judged by a single task-agnostic number: the cosine similarity between deep features extracted from the original image and from its compressed version. If true, researchers and clinics could compare compression schemes without implementing and running every downstream task, such as classification, segmentation, or biomarker prediction, separately. The paper backs this by jointly benchmarking six codecs on perceptual metrics and two downstream tasks across four datasets, including a newly collected uncompressed rhabdomyosarcoma cohort. Its central empirical finding is that fine-tuned learned codecs win on their home domain but transfer poorly, while the feature-similarity ranking tracks actual downstream accuracy, with the stain-aware SQLC codec at the top.","feed_headline":"Cosine similarity ranks six pathology compression schemes","feed_subtitle":"A single similarity score between original and compressed images matches real classification and segmentation accuracy.","key_machinery":"The load-bearing object is a task-agnostic feature-similarity metric: for each tile, features are extracted from the original and compressed image by a histopathology-pretrained ResNet18, and the cosine similarity between the two feature vectors is computed after flattening at several network depths. The metric operationalizes the idea that a compressed image is downstream-safe when its representation still points in the same direction in feature space as the original. It is the only component of the study that is proposed as a general standard, and the paper argues it tracks the ordering of six codecs on real classification and segmentation tasks.","core_discovery":"On the paper's own terms, the discovery is that deep feature similarity is a reliable surrogate for downstream task performance on lossy compressed whole-slide images. The authors compute cosine similarity between embeddings of original and compressed tiles using a ResNet18 pretrained on histopathology, at six depths from early convolutional features to the final fully connected layer, and find that compression schemes with higher mean similarity are the same schemes that win classification and segmentation evaluations. The same benchmark shows that a codec fine-tuned on JPEG80 data for perceptual fidelity dominates perceptual metrics on JPEG80 data but loses that edge on uncompressed data, while the stain-aware SQLC codec, optimized for downstream performance, produces the highest feature similarity and the best downstream results. The paper therefore argues that feature similarity can replace per-task downstream evaluation as a general, standardized compression benchmark.","pith_inferences":["A testable extension the paper does not run: compress the same tiles with each codec at several bitrates, compute the proposed cosine similarity, and fit a monotone mapping to task accuracy for a third task such as mutation prediction; agreement would strengthen the claim that the metric is task-agnostic.","Because the metric depends on the feature extractor, adopting it would create a de facto standard extractor; different foundation models may rank codecs differently, especially if their training data already contained compression artifacts.","The appendix result where a large foundation model performed best on JPEG70-compressed input rather than on the original suggests that some feature extractors are adapted to compressed inputs; this could mean the metric measures match to the extractor's training distribution as much as information preserved by the codec."],"forward_implications":["If feature similarity is a valid proxy, compression benchmarking no longer needs a separate implementation of each downstream task; one frozen feature extractor and a cosine distance suffice.","The benchmark ranking implies that WebP, a conventional codec, is the most practical near-term successor to JPEG for whole-slide storage because it is nearly as good as the best learned codecs but far faster.","Learned codecs that are fine-tuned on one compression artifact distribution should be expected to degrade when applied to data compressed differently, so deployment requires matching training and deployment compression settings.","The long decoding times of learned codecs, roughly 100 seconds per whole slide for SQLC at low bitrate, are a concrete barrier to clinical use independent of quality gains."],"supporting_citations":[{"why":"Supplies the SPL2 perceptual-quality codec and the JPEG80 recompression test set used in the perceptual track.","marker":"[25]"},{"why":"Supplies the SQLC stain-aware, downstream-optimized codec and the Camelyon16 classification benchmark and splits.","marker":"[26]"},{"why":"Supplies the ResNet18 feature extractor on which the proposed task-agnostic metric is computed.","marker":"[45]"},{"why":"Supplies the bmshj-factorized architecture and pretraining used for the generic CAI learned-codec baseline.","marker":"[31]"},{"why":"Inspires the deep-feature similarity approach and supplies the LPIPS perceptual metric used as a baseline.","marker":"[43]"},{"why":"Provides the automatic segmentation framework that defines the PanNuke downstream nuclei segmentation task.","marker":"[47]"},{"why":"Provides the Camelyon16 lymph-node metastasis whole-slide dataset and the JPEG-quality-80 evaluation context.","marker":"[11]"},{"why":"Provides the uncompressed RMS whole-slide cohort used to evaluate initial compression without prior lossy steps.","marker":"[37]"}],"fun_headline_variants":["Feature similarity ranks WSI codecs accurately","One metric predicts downstream compression quality","Deep features judge WSI codecs without task runs","Cosine similarity replaces WSI per-task tests","Pathology compression: similarity score suffices"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the similarity of deep features between an original and a compressed image, measured by one pathology-trained feature extractor, predicts what any downstream model will need from that image; a second premise is that the RMS cohort really is uncompressed, which the paper asserts without verification.","fun_headline_variants_meta":{"raw":{"variants":["Feature similarity ranks WSI codecs accurately","One metric predicts downstream compression quality","Deep features judge WSI codecs without task runs","Cosine similarity replaces WSI per-task tests","Pathology compression: similarity score suffices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000131,"raw_usage":{"total_tokens":1122,"prompt_tokens":933,"completion_tokens":189,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":123}},"tokens_in":549,"tokens_out":189,"duration_ms":2561,"temperature":1.0,"reasoning_tokens":123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:23:22.393249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any compression scheme not in the paper, compress the RMS test tiles, compute the proposed cosine similarity, and run a downstream task not used here, such as biomarker prediction or tissue classification with a different model architecture. If the ranking of codecs by feature similarity disagrees with the ranking by task accuracy, the general claim fails; the paper currently reports alignment only for a segmentation and a classification task, and only qualitatively.","supporting_citations":[],"review_version":1}