Pith. sign in

REVIEW 4 major objections 6 minor 17 references

DeepFGS: Fine-Grained Scalable Coding for Learned Image Compression

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DeepFGS is a learned image codec that produces a single bitstream which can be truncated at nearly any channel boundary and decoded into an image whose quality grows smoothly with the number of channels received, while coming close to the…

desk verdict A genuinely fine-grained scalable learned codec with strong Kodak results, but the 'truncate at any position' claim needs a bitstream-syntax reality check. read the letter →

arxiv 2412.00437 v1 pith:ZMCB47DS submitted 2024-11-30 eess.IV cs.CV

Yongqi Zhai , Yi Ma , Luyang Tang , Wei Jiang , Ronggang Wang This is my paper · ORCID
classification eess.IVcs.CV
keywords learnedimagecompressionscalablecodingfine-grainedscalabilitybitstreamtruncationfeatureseparationmutualentropymodeldecodermultiplexingrate-distortionoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that learned image compression can be made finely scalable without the usual penalty: one encoding pass produces a bitstream whose prefixes are each useful, and decoding more of the stream (more channels of the latent representation) yields smoothly better reconstructions. The proposed framework, DeepFGS, separates image information into basic features and scalable features, re-orders the scalable channels by importance during training, and conditions the entropy model of the scalable part on the basic part. If correct, this means a network sender can serve many bandwidths from a single stored stream, and the rate–distortion loss relative to non-scalable codecs can be kept small. The paper reports that DeepFGS outperforms prior learned and traditional scalable codecs in PSNR and MS-SSIM and nearly closes the quality gap to a strong non-scalable learned codec.

What carries the argument

The carrying mechanism is the combination of a feature separation backbone and a channel-wise information rearrangement strategy: the backbone splits the image into basic features $y_b$ and scalable features $y_s$, and the rearrangement—implemented not by physically permuting features but by a sampled training loss over prefixes—makes each channel of $y_s$ depend only on earlier channels, turning the bitstream into a quality ladder that can be cut at each channel boundary. A second mechanism is the mutual entropy model, a conditional entropy coder in which $\hat{y}_s$ is coded using probabilities conditioned on $\hat{y}_b$, lowering the rate by the mutual information between the two latents. A third is decoder multiplexing with a feature fusion module, allowing the basic-only and full reconstructions to share one decoder.

What would settle it

Encode a set of held-out images with a trained DeepFGS model, decode every prefix of each stream, and record PSNR; if any prefix scores below the previous prefix (or below the basic-layer reconstruction), the forward-dependence ordering has failed for that image. A sharper test is to measure the per-channel entropies of the encoded scalable latents and compare their rank order with the fixed order learned during training: low rank correlation across images would show that the no-side-information transfer assumption does not hold universally.

Watch

Extended reading notes

Core claim

The central claim is that fine-grained scalability can be built into a learned codec by construction rather than by training multiple codecs or iterating a recurrent network. The authors claim that the feature separation backbone yields two latent sets, the basic features $y_b$ and the scalable features $y_s$; the information rearrangement strategy, driven by a sampled prefix loss, orders the channels of $y_s$ so that each prefix $y_s^{\le i}$ adds a smooth quality gain; and the mutual entropy model encodes $\hat{y}_s$ conditional on $\hat{y}_b$, using $H(\hat{y}_s|\hat{y}_b)=H(\hat{y}_s)-I(\hat{y}_s,\hat{y}_b)$ to reduce the rate. With decoder reuse through the feature fusion module, this single framework is claimed to outperform existing scalable codecs and to approach the rate–distortion performance of non-scalable learned models.

Load-bearing premise

The scheme rests on the assumption that the channel ordering and mutual-entropy model learned during training remain valid for every image seen at inference, so the decoder can reconstruct any prefixed subset of the bitstream without side information about which channels were used or how the arithmetic-coded segments were synchronized.

Editorial extensions

If this is right

  • A single DeepFGS bitstream can be cut at any channel boundary and still decode to a complete image, with quality rising smoothly from the basic layer up to the full stream.
  • Bandwidth adaptation requires no re-encoding and no storage of multiple versions of the same image, because one pass produces the entire scalable stream.
  • Coding the scalable latents conditioned on the basic latents saves rate to the extent that $I(\hat{y}_s,\hat{y}_b)$ is nonzero, and the reported experiments show this improves the rate–distortion curve.
  • The gap between scalable and non-scalable learned compression can be small: DeepFGS is reported near an attention-based non-scalable codec and close to its own single-rate variant.
  • Reusing the decoder with feature fusion keeps parameter and computation cost below that of multi-decoder scalable systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the truncated-stream property makes the codec a natural fit for network-adaptive delivery and edge caching, where a server can send a longer byte range when bandwidth allows; the paper does not implement such a protocol or measure end-to-end latency.
  • Editorial inference: the learned channel ordering is fixed after training and content-agnostic; a per-image or per-content-class ordering could improve the rate–distortion trade-off further, at the cost of side information that the current design deliberately avoids.
  • Editorial inference: the same basic-versus-scalable separation and mutual conditioning could be carried into learned video compression, using the previous decoded frame as the 'basic' condition for the current frame's scalable latents; the paper leaves this uncharted.
  • Editorial inference: the demonstration of 'any position' truncation is at channel boundaries; true byte-level truncation inside an arithmetic-coded segment would require an entropy-coding resynchronization mechanism that the paper does not specify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DeepFGS proposes a learned image compression framework that generates a single flexible bitstream whose prefixes can be decoded into images of progressively increasing quality. The method separates latent features into a basic part and a scalable part using a feature separation backbone, applies an information rearrangement strategy to order scalable channels by importance, employs a mutual entropy model that conditions scalable latents on basic latents, and reuses the decoder through a feature fusion module to reduce complexity. Experiments on the Kodak dataset report rate-distortion curves showing gains over traditional scalable codecs (SVC and SHVC) and prior learned scalable models (Jia et al., Mei et al., Guo et al., Zhang et al.) in both PSNR and MS-SSIM, and near-competitive performance with non-scalable learned codecs. The paper's central claim is that this is a fine-grained, one-pass scalable coding method where the bitstream can be truncated at any position and still decode to a complete image.

Significance. If validated, a one-pass bitstream that supports fine-grained truncation at byte granularity would be practically valuable for adaptive streaming and progressive transmission. The training strategy of sampling random truncation points is a sensible way to induce a channel ordering, and the decoder-multiplexing idea is a reasonable complexity reduction. The paper is empirical and does not claim formal derivations; its value rests on the strength of the experimental evidence. However, the manuscript currently lacks a specification of the actual bitstream syntax, which is essential for the central truncation claim, and the experimental evaluation does not include variance estimates or released artifacts.

major comments (4)
  1. [Introduction and Experiments (Fig. 1 and Fig. 6)] The central claim that the bitstream can be truncated at any position is not substantiated. The paper describes rates R(yb) and R(ys^{<=i}) in Eqs. (3)-(5) as sums of estimated entropies, but an entropy estimate is not a bitstream layout. The experiments decode pre-selected channel counts (e.g., 'Use channel 0-191' in Fig. 1 and the 8-channel or 1-channel intervals in Fig. 6) with k known in advance, which is a channel-selection protocol rather than byte-level truncation. Without specifying how arithmetic-coded channel segments are terminated, concatenated, and located after an arbitrary byte cut, a truncation can land inside a codeword and cause desynchronization. Please provide a precise bitstream syntax, clarify how the decoder determines the number of channels present, and report experiments that truncate the actual encoded bitstream at byte positions.
  2. [Information Rearrangement Strategy] The claim that no side information is needed because the channel ordering is learned during training is ambiguous. If the permutation is a fixed global ordering applied to every image, the statement is plausible, but the paper does not state this explicitly and the visualization in Fig. 3 is anecdotal. If the ordering is computed per image at inference time, the ordering index must be transmitted, and that overhead is not accounted for in the rate calculation. Please clarify whether the rearrangement is a fixed trained permutation and discuss how the decoder knows both the ordering and the number of decoded channels from the truncated bitstream.
  3. [Experiments and Fig. 6] The rate-distortion comparisons are presented as single curves with no error bars, multiple training runs, or statistical significance tests. This is particularly relevant for the ablation study in Fig. 7 and the comparisons to prior scalable methods, where the claimed improvements need to be robust to training seed and initialization. Please provide at least three runs per configuration with mean and standard deviation, or state clearly that the curves are from a single run and temper the corresponding claims.
  4. [Eqs. (4)-(5)] The sampling distribution of j in Eq. (5) is not specified, and Eq. (4) counts R(yb) once in every summand, which disproportionately weights the base-layer rate in the multi-term loss. The effect of this weighting on the learned channel ordering is not analyzed. Please state the sampling distribution (e.g., uniform over the range) and discuss how the choice of w(i) and the truncation schedule affect the granularity and rate range of the final RD curve.
minor comments (6)
  1. [Figure 6] The legend in Fig. 6 contains garbled citation strings such as '/s48 /s46 /s54/s48'; these should be replaced with readable labels that map to the references in the caption.
  2. [Eq. (5)] The notation 'j is sampled from [C1, C2] ∩ N' is confusing; it should be 'j is sampled uniformly from {C1, C1+1, ..., C2}' or the actual distribution should be stated.
  3. [Figure 4 and Mutual Entropy Model] The architecture of the mutual entropy model is shown in Fig. 4 but not described in the text; a brief explanation of how the hyperprior and prior fusion are combined would improve readability.
  4. [Figure 3 discussion] The statement that 'our model arranges the channels in descending entropy order' should be tempered because the entropy ordering is an emergent property of training rather than an explicit sorting operation; suggest phrasing such as 'the learned ordering tends to have decreasing entropy'.
  5. [General] There are minor language issues, including 'the two locations share the same decoder' grammar, and the phrase 'nearly continuously adjustable' should be defined (e.g., 'the RD curve is sampled at intervals of one channel').
  6. [Appendix] The Experiments section mentions 'more results on the high-definition data set will be given in the Appendix,' but no appendix is included in the manuscript.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the RD results are an optimized objective evaluated on a held-out set against external baselines; the only self-citation is contextual and not load-bearing.

full rationale

The paper is an empirical, benchmark-driven study rather than a formal derivation. Its key rate-distortion curves are obtained by training the network with the loss in Eqs. (3)-(5), which explicitly minimizes distortion at sampled prefix-channel counts, and then evaluating on the held-out Kodak set against external baselines (Jia, Mei, Toderici, Zhang, Guo, SVC, SHVC, BPG, JPEG2000, cheng2020, bmshj2018, mbt2018). Because the loss is the objective being optimized, reporting the resulting RD curve is a direct measurement of the fitted model, not an independent prediction; this is standard for learned compression and is not circular. The only self-citation is reference [1] (MLIC, with overlapping authorship), used in the introduction to state that learned codecs have surpassed traditional codecs; this claim is contextual and does not carry any load in the DeepFGS derivation or evaluation. No uniqueness theorem is imported, no ansatz is smuggled in via citation, and no known result is renamed as new. The paper's main weakness is that the bitstream-level claim, that the bitstream is truncated at any position and decodes to a complete image, is not backed by an explicit arithmetic-coding syntax; however, missing evidence about byte-prefix decodability is a correctness and evidence concern, not circularity. Therefore the circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no exotic physical or mathematical entities. It relies on standard machine learning components (channel attention, hyperprior entropy models, arithmetic coding) plus hand-chosen hyperparameters. The main unpaid assumptions are transferability of the learned channel ordering and the adequacy of the small Kodak benchmark for the central claim.

free parameters (4)
  • rate-distortion tradeoff lambda = 0.002 (MSE), 7.0 (MS-SSIM)
    Chosen by hand to set the minimum bitrate and balance rate against distortion; standard in learned image compression but a free design choice.
  • distortion weight schedule w(i) = w(i) = floor(i/8)
    Hand-designed function that assigns different weights to different truncation points in Eq. (4); it directly shapes the fine-grained RD curve.
  • latent channel counts C1 and C2 = 192 each
    Architecture choice determining the total number of decodable layers and the granularity of scalability, not derived from theory.
  • training truncation range k = integer sampled in [192, 384]
    During training the decoder receives a random number of channels; this sampling distribution determines how well each quality level is optimized.
assumptions (4)
  • standard math Arithmetic coding is lossless given the modeled probability distribution.
    The AE/AD modules in Figure 2 rely on this standard result; any mismatch between model probabilities and actual symbol frequencies would lose compression efficiency.
  • domain assumption PSNR and MS-SSIM on the 24 Kodak images are accepted measures of image quality.
    The central outperformance claim is made with respect to these metrics on a small test set; if the metrics or dataset are not representative, the claim weakens.
  • domain assumption Training on 256x256 COCO patches generalizes to 768x512 Kodak images.
    The model is trained exclusively on randomly cropped COCO patches and evaluated on full-resolution Kodak; good generalization is assumed but not demonstrated across datasets.
  • domain assumption A fixed channel ordering learned from training data is valid for all test images.
    The information rearrangement strategy relies on the network learning a global importance ordering, with no per-image side information; if the ordering varies across images, prefix decoding degrades.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepFGS: Fine-Grained Scalable Coding for Learned Image Compression." pith.science (2026). https://pith.science/paper/ZMCB47DS

@misc{pith2026241200437,
  author       = {Pith},
  title        = {Pith review of: DeepFGS: Fine-Grained Scalable Coding for Learned Image Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMCB47DS}},
  note         = {Machine review of arXiv:2412.00437}
}
read the original abstract

Scalable coding, which can adapt to channel bandwidth variation, performs well in today's complex network environment. However, most existing scalable compression methods face two challenges: reduced compression performance and insufficient scalability. To overcome the above problems, this paper proposes a learned fine-grained scalable image compression framework, namely DeepFGS. Specifically, we introduce a feature separation backbone to divide the image information into basic and scalable features, then redistribute the features channel by channel through an information rearrangement strategy. In this way, we can generate a continuously scalable bitstream via one-pass encoding. For entropy coding, we design a mutual entropy model to fully explore the correlation between the basic and scalable features. In addition, we reuse the decoder to reduce the parameters and computational complexity. Experiments demonstrate that our proposed DeepFGS outperforms previous learning-based scalable image compression models and traditional scalable image codecs in both PSNR and MS-SSIM metrics.

Figures

Figures reproduced from arXiv: 2412.00437 by the authors.

Figure 1
Figure 1. Illustration of fine-grained scalable Coding of the proposed DeepFGS. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our fine-grained scalable coding framework DeepFGS. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the entropy and the reconstruction quality of latent repre [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The architecture of our mutual entropy model for ˆy [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Observation of the process of feature fusion. Basic features, fusion fea [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of rate-distortion performance of our DeepFGS with existing [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation study on the proposed DeepFGS [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Visual quality comparisons with the learning-based scalable method Jia [2] [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

  1. [1]

    Mlic: Multi-reference entropy model for learned image compression,

    Wei Jiang, Jiayu Yang, Yongqi Zhai, Peirong Ning, Feng Gao, and Ronggang Wang, “Mlic: Multi-reference entropy model for learned image compression,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 7618–7627

    2023

  2. [2]

    Layered image compression using scalable auto-encoder,

    Chuanmin Jia, Zhaoyi Liu, Yao Wang, Siwei Ma, and Wen Gao, “Layered image compression using scalable auto-encoder,” in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2019, pp. 431–436

    work page 2019

  3. [3]

    Variable rate image compression with recurrent neural networks,

    George Toderici, Sean M O’Malley, Sung Jin Hwang, Damien Vincent, David Min- nen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar, “Variable rate image compression with recurrent neural networks,” arXiv preprint arXiv:1511.06085, 2015

    arXiv 2015

  4. [4]

    Full resolution image compression with recurrent neural networks,

    George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell, “Full resolution image compression with recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5306–5314

    work page 2017

  5. [5]

    Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks,

    Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici, “Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4385–4393

    work page 2018

  6. [6]

    Cbam: Convolu- tional block attention module,

    Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon, “Cbam: Convolu- tional block attention module,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19

    work page 2018

  7. [7]

    Joint autoregressive and hierarchical priors for learned image compression,

    David Minnen, Johannes Ball´ e, and George D Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in Advances in Neural Information Processing Systems, 2018, pp. 10771–10780

    work page 2018

  8. [8]

    Learning-based scalable image compression with latent-feature reuse and prediction,

    Yixin Mei, Li Li, Zhu Li, and Fan Li, “Learning-based scalable image compression with latent-feature reuse and prediction,” IEEE Transactions on Multimedia, 2021

    work page 2021

Show all 17 references
  1. [9]

    Microsoft coco: Common objects in context,

    Tsungyi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ra- manan, Piotr Dollar, and C Lawrence Zitnick, “Microsoft coco: Common objects in context,” pp. 740–755, 2014

    2014

  2. [10]

    Adam: A method for stochastic optimization,

    Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  3. [11]

    Multiscale structural similarity for image quality assessment,

    Zhou Wang, Eero P Simoncelli, and Alan C Bovik, “Multiscale structural similarity for image quality assessment,” vol. 2, pp. 1398–1402, 2003

    2003

  4. [12]

    Learned scalable image compression with bidirectional context disentanglement network,

    Zhizheng Zhang, Zhibo Chen, Jianxin Lin, and Weiping Li, “Learned scalable image compression with bidirectional context disentanglement network,” in 2019 IEEE In- ternational Conference on Multimedia and Expo (ICME). IEEE, 2019, pp. 1438–1443

    2019

  5. [13]

    Deep scalable image compression via hierarchical feature decorrelation.,

    Zongyu Guo, Zhizheng Zhang, and Zhibo Chen, “Deep scalable image compression via hierarchical feature decorrelation.,” in PCS, 2019, pp. 1–5

    2019

  6. [14]

    Learned image compression with discretized gaussian mixture likelihoods and attention modules,

    Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  7. [15]

    Variational image compression with a scale hyperprior,

    Johannes Ball´ e, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick John- ston, “Variational image compression with a scale hyperprior,” arXiv preprint arXiv:1802.01436, 2018

    2018 arXiv

  8. [16]

    Kodak lossless true color image suite (photocd pcd0992),

    Eastman Kodak, “Kodak lossless true color image suite (photocd pcd0992),” 1999

    1999

  9. [17]

    Text of iso/iec 14496-10: 2005/fdam 3 scalable video coding,

    T Wiegand, “Text of iso/iec 14496-10: 2005/fdam 3 scalable video coding,” ISO/IEC JTC1/SC29/WG11 N9197, 2007

    2005

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.