REVIEW 3 major objections 5 minor 1 cited by
Explicit Residual-Based Scalable Image Coding for Humans and Machines
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that explicitly encoding the pixel residual between the machine reconstruction and the original image yields a scalable human-plus-machine codec that beats the previous feature-fusion design by up to 29.57% BD-rate.
desk verdict A sound, incremental residual-coding extension to the authors' own ICMH-FF framework, with a headline BD-rate gain that is plausible but not fully pinned down by the reported evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the explicit residual signal. At pixel level, the difference image $x_d = x - \hat{x}_m$ isolates exactly the information the machine codec discarded, so the additional learned codec needs to compress only that. At feature level, a feature subtraction network computes per-slice residuals $y_{ak} = y_k - \hat{y}_{mk}$, and the decoder adds them back to the quantized machine features. Both variants retain the slice-wise channel-autoregressive entropy structure of the base codec, so the number of residual slices $N_a$ directly trades model size against reconstruction quality.
What would settle it
Retrain PR-ICMH with the base SA-ICM codec jointly fine-tuned with the residual codec, or with the base operating at different quality levels, and measure BD-rate against ICMH-FF on COCO-val; if the 29.57% gain shrinks or reverses, the central claim fails.
Extended reading notes
Core claim
The central claim is that replacing implicit decoder-side feature fusion with an explicitly coded residual makes a scalable human-machine codec more efficient. In PR-ICMH, the machine-oriented reconstruction $\hat{x}_m$ from a fixed SA-ICM base is subtracted from the original image, and the difference image $x_d$ is compressed by an additional learned codec; the human-oriented image is $\hat{x}_m$ plus the decoded residual. In FR-ICMH, the residual is computed slice-wise on latents as $y_{ak} = y_k - \hat{y}_{mk}$, with only $N_a$ of $N_m$ slices transmitted. Because the base SA-ICM is untouched, machine-vision performance is exactly that of the base model while human reconstruction improves, and the reported result is a BD-rate reduction of up to 29.57% over ICMH-FF at $N_a = 3$, with consistent gains across $N_a = 3, 4, 5$.
Load-bearing premise
The result assumes that training only the residual codec on top of a frozen SA-ICM base is sufficient; if the base and residual layers must be jointly optimized to preserve the reported gains, the central claim weakens.
Editorial extensions
If this is right
- If PR-ICMH's reported gains hold, a scalable human-plus-machine codec can be built by freezing any task-agnostic machine codec and adding a separate learned residual codec, without retraining the recognition side.
- Because the machine-oriented stream is untouched, recognition performance is identical to the base SA-ICM model, so the enhancement layer can be dropped or replaced without harming machine tasks.
- Reducing the residual slice count from $N_a = 5$ to $N_a = 3$ cuts the residual model's parameters from 76.6M to 58.7M while sacrificing little BD-rate, so encoder complexity can be traded against compression performance.
- The coded signal in the enhancement layer is literally the perceptual detail omitted by the machine stream, which makes the scalable layer interpretable rather than a black-box feature fusion.
- The proposed residual mechanism is not tied to a particular downstream task, so the same frozen-base recipe should apply across recognition models that share a task-agnostic machine codec.
Reading between the lines
- Editorial inference: If the same explicit-residual principle transfers to video, a temporal residual between successive machine-oriented reconstructions could be coded instead of a full human-oriented stream, potentially yielding scalable machine-plus-human video at lower bitrate.
- Editorial inference: The reported 29.57% gain is measured with a fixed base at one quality level; a natural extension is to test how the residual codec behaves when the base operates at several quality levels, since the residual's entropy should fall as the base reconstruction improves.
- Editorial inference: The comparison is against ICMH-FF on COCO-val; applying PR-ICMH to base codecs optimized for detection or segmentation, with the same frozen-base recipe, would test whether the gains depend on SA-ICM's edge-preserving prior or transfer generically.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes two extensions of the scalable image coding framework ICMH-FF, in which a fixed machine-oriented codec (SA-ICM) provides a base layer and an additional learned codec supplies information for human-oriented reconstruction. FR-ICMH computes and compresses feature-level residuals between the SA-ICM latents and the latents of an additional LIC-TCM codec, while PR-ICMH compresses the pixel-level difference between the SA-ICM reconstruction and the original image. Only the additional codec is trained, using rate–distortion losses (5) and (6). Experiments on COCO-val report rate–distortion curves and BD-rate numbers versus ICMH-FF, with PR-ICMH giving up to 29.57% BD-rate savings at Na=3.
Significance. Strengths: The residual definitions are mathematically clean; equations (2), (3), and (6) are architectural identities that exactly reconstruct the intended targets, so there is no circularity between the training target and the reported metric. If the reported gains hold under a well-defined total-bitrate evaluation, the explicit-residual design is a simple and interpretable improvement over ICMH-FF's implicit feature fusion, and the slice-count ablation addresses model complexity. Weaknesses: The empirical evidence is a single run on a single dataset with no error bars, no code release, and an ambiguous BD-rate definition; the headline number is the best of three Na settings. The central claim therefore needs verification before the result can be considered established.
major comments (3)
- [Table I and Section IV.A] The BD-rate values in Table I do not indicate whether they are computed from the additional-information-only rate (solid curves in Fig. 6) or the total bitrate including the fixed SA-ICM base layer (dashed curves). Since the base layer is identical for PR-ICMH and ICMH-FF at a given image, the base bitrate is a constant offset in the total-rate comparison; a 29.57% saving on additional bitrate would translate into a smaller saving on total bitrate. The paper's central claim, 'PR-ICMH achieving up to 29.57% BD-rate reduction,' is therefore ambiguous and potentially unsupported unless the authors specify the rate definition and report both additional-only and total-bitrate BD-rate values.
- [Section IV.A, Table I] All results come from a single training run evaluated on COCO-val only, and no error bars, confidence intervals, or significance tests are provided. The differences in PR-ICMH BD-rate across Na settings (-29.57%, -26.06%, -26.78%) are small enough that they could reflect training noise, especially since Na=3 is selected as the headline. Without multiple seeds or a statistical analysis, the claims that PR-ICMH 'consistently' outperforms ICMH-FF and that Na=3 is the best configuration are not firmly established.
- [Section III and IV.A] The evaluation fixes the SA-ICM base layer at a single operating point (λ=0.05) and trains the residual codec independently. The authors do not analyze how the residual codec's rate-distortion trade-off interacts with the base layer's quality, nor do they report results at other base-layer λ values. Since the proposed methods are described as a scalable coding framework, the absence of evidence across base operating points leaves the generalizability of the reported gains open; the 29.57% figure may be specific to this one base configuration.
minor comments (5)
- [Section IV.A, last paragraph] The sentence 'PR-ICMH demonstrates consistently large gains regardless of the number of slices, achieving the best performance with a BD-rate reduction of 29.57% when Na=3' is internally inconsistent because Na=4 and Na=5 yield smaller savings; consider saying 'the gains are similar across slice counts' instead.
- [Figure 6] The caption should state explicitly that solid lines are additional-only bitrates and dashed lines are total bitrates; currently this information appears only in the body text.
- [Table I] Add a note or column header clarifying whether the BD-rate is computed on total bitrate or additional-only bitrate; this is essential for interpreting the headline result.
- [Section II.C] The phrase 'extended this principal' should be 'extended this principle'.
- [General] The paper does not state whether the evaluation code or trained models will be released; for reproducibility of the claimed BD-rate gains, code release would be valuable.
Circularity Check
No circularity: the residual codecs are trained on explicitly computed residual targets and evaluated on held-out COCO-val; all reported BD-rate gains are measured, not derived from the training targets.
full rationale
The derivation chain is self-contained and empirically measured. SA-ICM is fixed, and only the additional LIC is trained, with loss functions (5) and (6) targeting the explicitly computed feature residual ya or pixel residual xd. The reconstruction identities in (3) and the PR-ICMH addition of the decoded difference image to the machine-oriented reconstruction are architectural definitions, not fits that are later relabeled as predictions. The reported BD-rate savings in Table I are comparisons of measured rate-distortion points on COCO-val against the ICMH-FF baseline; no parameter is fitted to the BD-rate metric and then presented as a prediction. Self-citations to the authors' prior ICMH-FF and SA-ICM work are used as a baseline and base codec, respectively, not as load-bearing proof of the new method's validity. The only noteworthy issue is that Table I does not explicitly state whether BD-rate is computed on total bitrate or additional-only bitrate, and Figure 6 distinguishes these curves; that is an evaluation-reporting ambiguity that could affect the magnitude of the claimed gain, but it is not circularity. The central claim does not reduce to its inputs by construction.
Assumptions & free parameters
free parameters (3)
- Rate-distortion weight lambda (residual codec) =
0.005, 0.01, 0.02, 0.03, 0.05
- Number of residual slices Na =
3 (headline), 4, 5
- SA-ICM base quality lambda =
0.05
assumptions (4)
- domain assumption SA-ICM is task-agnostic and preserves all information needed by downstream recognition models.
- domain assumption Training the residual codec independently of the fixed base codec is sufficient to reach near-optimal joint rate-distortion.
- domain assumption COCO-train and COCO-val are a sufficient benchmark for the claimed BD-rate improvements.
- domain assumption SA-ICM features and LIC-TCM features are compatible enough for slice-wise subtraction and addition to be meaningful in FR-ICMH.
Cite this review
Pith. "Pith review of Explicit Residual-Based Scalable Image Coding for Humans and Machines." pith.science (2026). https://pith.science/paper/U3JIPZDK
@misc{pith2026250619297,
author = {Pith},
title = {Pith review of: Explicit Residual-Based Scalable Image Coding for Humans and Machines},
year = {2026},
howpublished = {\url{https://pith.science/paper/U3JIPZDK}},
note = {Machine review of arXiv:2506.19297}
}
read the original abstract
Scalable image compression is a technique that progressively reconstructs multiple versions of an image for different requirements. In recent years, images have increasingly been consumed not only by humans but also by image recognition models. This shift has drawn growing attention to scalable image compression methods that serve both machine and human vision (ICMH). Many existing models employ neural network-based codecs, known as learned image compression, and have made significant strides in this field by carefully designing the loss functions. In some cases, however, models are overly reliant on their learning capacity, and their architectural design is not sufficiently considered. In this paper, we enhance the coding efficiency and interpretability of ICMH framework by integrating an explicit residual compression mechanism, which is commonly employed in resolution scalable coding methods such as JPEG2000. Specifically, we propose two complementary methods: Feature Residual-based Scalable Coding (FR-ICMH) and Pixel Residual-based Scalable Coding (PR-ICMH). These proposed methods are applicable to various machine vision tasks. Moreover, they provide flexibility to choose between encoder complexity and compression performance, making it adaptable to diverse application requirements. Experimental results demonstrate the effectiveness of our proposed methods, with PR-ICMH achieving up to 29.57% BD-rate savings over the previous work.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Seed Selection for Human-Oriented Image Reconstruction via Guided Diffusion
Selecting the best of five random seeds for guided diffusion improves reconstructed image quality over a single random seed, and early-step selection recovers most of the gain at half the compute.
Reference graph
Works this paper leans on
-
[1]
Scalable Image Coding for Humans and Machines Using Feature Fusion Network,
T. Shindo, T. Watanabe, Y . Tatsumi and H. Watanabe, “Scalable Image Coding for Humans and Machines Using Feature Fusion Network,” 2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP), 2024, pp. 1-6
work page 2024
-
[2]
Scalable Image Coding for Humans and Machines,
H. Choi and I. V . Baji ´c, “Scalable Image Coding for Humans and Machines,” in IEEE Transactions on Image Processing, vol. 31, pp. 2739-2754, 2022
work page 2022
-
[3]
Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation,
H. Li et al. , “Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation,” Computer Vision – ECCV 2024. ECCV
work page 2024
-
[4]
Semantic Structured Image Coding Framework for Multiple Intelligent Applications,
S. Sun, T. He and Z. Chen, “Semantic Structured Image Coding Framework for Multiple Intelligent Applications,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 9, pp. 3631- 3642, 2021
work page 2021
-
[5]
SSSIC: Semantics-to-Signal Scalable Image Coding With Learned Structural Representations,
N. Yan et al., “SSSIC: Semantics-to-Signal Scalable Image Coding With Learned Structural Representations,” in IEEE Transactions on Image Processing, vol. 30, pp. 8939-8954, 2021
work page 2021
-
[6]
Scalable Video Coding for Humans and Machines,
H. Choi and I. V . Baji ´c, “Scalable Video Coding for Humans and Machines,” 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP), Shanghai, China, 2022, pp. 1-6
work page 2022
-
[7]
Bridging the Gap Between Image Coding for Machines and Humans,
N. Le et al. , “Bridging the Gap Between Image Coding for Machines and Humans,” 2022 IEEE International Conference on Image Processing (ICIP), 2022, pp. 3411-3415
work page 2022
-
[8]
Towards Analysis-Friendly Face Representation With Scalable Feature and Texture Compression,
S. Wang et al. , “Towards Analysis-Friendly Face Representation With Scalable Feature and Texture Compression,” in IEEE Transactions on Multimedia, vol. 24, pp. 3169-3181, 2022
work page 2022
Show all 43 references
-
[9]
Learned scalable video coding for humans and machines,
H. Hadizadeh and I. V . Bajic, “Learned scalable video coding for humans and machines,” arXiv preprint arXiv:2307.08978, 2023
2023 arXiv
-
[10]
VVC+M : Plug and Play Scalable Image Coding for Humans and Machines,
A. Harell, Y . Foroutan and I. V . Baji´c, “VVC+M : Plug and Play Scalable Image Coding for Humans and Machines,” 2023 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2023, pp. 200-205
2023
-
[11]
Conditional and Residual Methods in Scalable Coding for Humans and Machines,
A. de Andrade, A. Harell, Y . Foroutan and I. V . Baji ´c, “Conditional and Residual Methods in Scalable Coding for Humans and Machines,” 2023 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2023, pp. 194-199
2023
-
[12]
Semantically Struc- tured Image Compression via Irregular Group-Based Decoupling,
R. Feng, Y . Gao, X. Jin, R. Feng and Z. Chen, “Semantically Struc- tured Image Compression via Irregular Group-Based Decoupling,” 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 17191-17201
2023
-
[13]
TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception,
Y . -H. Chen et al. , “TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception,” 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 23240-23250
2023
-
[14]
ICMH-Net: Neural Image Compres- sion Towards both Machine Vision and Human Vision,
L. Liu, Z. Hu, Z. Chen and D. Xu, “ICMH-Net: Neural Image Compres- sion Towards both Machine Vision and Human Vision,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 8047-8056
2023
-
[15]
Scalable image coding with enhancement features for human and machine,
Y . Wu, P. An, C. Yang and X. Huang, “Scalable image coding with enhancement features for human and machine,” Multimedia Systems, vol. 30, no. 77, 2024
2024
-
[16]
Shindo, Y
T. Shindo, Y . Tatsumi, T. Watanabe and H. Watanabe, ”Refining Coded Image in Human Vision Layer Using CNN-Based Post-Processing,” 2024 IEEE 13th Global Conference on Consumer Electronics (GCCE), 2024, pp. 166-167
2024
-
[17]
Layered and scalable image coding with semantic features for human and machine,
J. Wei et al. , “Layered and scalable image coding with semantic features for human and machine,” Engineering Applications of Artificial Intelligence, vol. 155, 2025
2025
-
[18]
Semantic Prior-Guided Scalable Image Coding,
W. Shi, W. Yin, F. Tao and Y . Wen, “Semantic Prior-Guided Scalable Image Coding,” 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1-5
2025
-
[19]
Variational Image Compression with a Scale Hyperprior,
J. Ball ´e, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational Image Compression with a Scale Hyperprior,” International Conference on Learning Representations (ICLR), 2018, pp. 1-10
2018
-
[20]
Joint Autoregressive and Hi- erarchical Priors for Learned Image Compression,
D. Minnen, J. Ball ´e, and G. Toderici, “Joint Autoregressive and Hi- erarchical Priors for Learned Image Compression,” 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018, pp.10794- 10803
2018
-
[21]
Channel-Wise Autoregressive Entropy Models for Learned Image Compression,
D. Minnen and S. Singh, “Channel-Wise Autoregressive Entropy Models for Learned Image Compression,” 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 3339-3343
2020
-
[22]
Learned Image Compression with Mixed Transformer-CNN Architectures,
J. Liu, H. Sun and J. Katto, “Learned Image Compression with Mixed Transformer-CNN Architectures,” 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023, pp. 14388-14397
2023
-
[23]
End-to-end optimized image compression,
J. Balle, V . Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” arXiv preprint arXiv:1611.01704, 2016
2016 arXiv
-
[24]
Learned Image Com- pression With Discretized Gaussian Mixture Likelihoods and Attention Modules,
Z. Cheng, H. Sun, M. Takeuchi and J. Katto, “Learned Image Com- pression With Discretized Gaussian Mixture Likelihoods and Attention Modules,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7936-7945
2020
-
[25]
High Efficiency Compression for Object Detection,
H. Choi and I. V . Bajic, “High Efficiency Compression for Object Detection,” 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 1792-1796
2018
-
[26]
Visual Analysis Motivated Rate-Distortion Model for Image Coding,
Z. Huang, C. Jia, S. Wang and S. Ma, “Visual Analysis Motivated Rate-Distortion Model for Image Coding,” 2021 IEEE International Conference on Multimedia and Expo (ICME), 2021, pp. 1-6
2021
-
[27]
Region of Interest Enabled Learned Image Coding for Machines,
J. I. Ahonen, N. Le, H. Zhang, F. Cricri and E. Rahtu, “Region of Interest Enabled Learned Image Coding for Machines,” 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP), 2023, pp. 1-6
2023
-
[28]
Image Coding For Machines: an End-To-End Learned Approach,
N. Le, H. Zhang, F. Cricri, R. Ghaznavi-Youvalari and E. Rahtu, “Image Coding For Machines: an End-To-End Learned Approach,” 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 1590-1594
2021
-
[29]
Learned Image Coding for Machines: A Content-Adaptive Approach,
N. Le, H. Zhang, F. Cricri, R. Ghaznavi-Youvalari, H. R. Tavakoli and E. Rahtu, “Learned Image Coding for Machines: A Content-Adaptive Approach,” 2021 IEEE International Conference on Multimedia and Expo (ICME), 2021, pp. 1-6
2021
-
[30]
Image Coding For Machine Via Analytics-Driven Appearance Redundancy Reduction,
X. Shen, H. Ou and W. Yang, “Image Coding For Machine Via Analytics-Driven Appearance Redundancy Reduction,” 2024 IEEE Inter- national Conference on Image Processing (ICIP), 2024, pp. 1883-1889
2024
-
[31]
Image Coding For Machines With Edge Information Learning Using Segment Any- thing,
T. Shindo, K. Yamada, T. Watanabe and H. Watanabe, “Image Coding For Machines With Edge Information Learning Using Segment Any- thing,” 2024 IEEE International Conference on Image Processing (ICIP), 2024, pp. 3702-3708
2024
-
[32]
Image Coding for Machines with Object Region Learning,
T. Shindo, T. Watanabe, K. Yamada and H. Watanabe, “Image Coding for Machines with Object Region Learning,” 2024 IEEE 21st Consumer Communications & Networking Conference (CCNC), 2024, pp. 1040- 1041
2024
-
[33]
Shindo, T
T. Shindo, T. Watanabe, Y . Tatsumi and H. Watanabe, ”Delta-ICM: En- tropy Modeling with Delta Function for Learned Image Compression,” 2025 IEEE International Conference on Consumer Electronics (ICCE), 2025, pp. 1-6
2025
-
[34]
Image Coding for Machines with Omnipotent Feature Learning,
R. Feng et al. , “Image Coding for Machines with Omnipotent Feature Learning,” Computer Vision - ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol. 13697, 2022, pp 510-528
2022
-
[35]
Segment Anything,
A. Kirillov et al. , “Segment Anything,” 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3992-4003
2023
-
[36]
High Efficiency Video Coding, Standard ISO/IEC 23008-2, ISO/IEC JTC 1, 2013
2013
-
[37]
Versatile Video Coding, Standard ISO/IEC 23090-3, ISO/IEC JTC 1, 2020
2020
-
[38]
DVC: An End-To-End Deep Video Compression Frame- work,
G. Lu et al. , “DVC: An End-To-End Deep Video Compression Frame- work,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10998-11007
2019
-
[39]
Information Technology—JPEG 2000—Image Coding System—Part1: Core Coding System, ISO/IEC 15 444-1, 2000
2000
-
[40]
Learning-Based Scalable Image Compres- sion With Latent-Feature Reuse and Prediction,
Y . Mei, L. Li, Z. Li and F. Li, “Learning-Based Scalable Image Compres- sion With Latent-Feature Reuse and Prediction,” in IEEE Transactions on Multimedia, vol. 24, pp. 4143-4157, 2022
2022
-
[41]
COMPASS: High-Efficiency Deep Image Compression with Arbitrary-scale Spatial Scalability,
J. Park, J. Lee and M. Kim, “COMPASS: High-Efficiency Deep Image Compression with Arbitrary-scale Spatial Scalability,” 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 12780- 12789
2023
-
[42]
Microsoft COCO: Common Objects in Context,
T. Y . Lin et al., “Microsoft COCO: Common Objects in Context,” Com- puter Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol. 8693, pp.740-755, 2014
2014
-
[2024]
15109, 2024, pp
Lecture Notes in Computer Science, vol. 15109, 2024, pp. 382- 399
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.