Pith. sign in

REVIEW 4 major objections 4 minor 25 references

Point Cloud-Assisted Neural Image Compression

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A neural codec that fuses lidar point clouds as auxiliary input cuts image bitrate by 54.518% on the KITTI dataset.

desk verdict Genuinely new application of point-cloud side information to learned image compression, but the headline bitrate gain is not yet causally attributed to the point cloud without a same-architecture control. read the letter →

arxiv 2412.11771 v1 pith:AMVNZY36 submitted 2024-12-16 eess.IV cs.CV

Ziqun Li , Qi Zhang , Xiaofeng Huang , Zhao Wang , Siwei Ma , Wei Yan This is my paper · ORCID
classification eess.IVcs.CV
keywords point-cloud-assistedimagecompressionlearnedmulti-modalfeaturefusionunifiedrepresentationdepthmapprojectionKITTIBD-ratechannelattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a lidar point cloud, projected into the camera image plane and fed to a neural codec as extra input, can substantially reduce the bitrate needed to store or transmit the image. The proposed codec PCA-NIC builds a shared $4 \times H \times W$ representation from the RGB image and a depth map derived from the point cloud, then extracts and fuses features from both modalities with a channel-attention fusion module. On the KITTI driving dataset, it reports a 54.518% BD-rate reduction and a 2.101 dB BD-PSNR improvement over the image-only learned codec baseline, with consistent gains in MS-SSIM. The authors further argue from an entropy decomposition that mixed-modality feature extraction gives the network strictly more information than image-only extraction, which is why the auxiliary depth helps.

What carries the argument

Two mechanisms carry the argument. The first is the unified representation: a point cloud is projected into the pixel coordinate system by $X_{2D}=P^{(2)}_{rect}R^{(0)}_{rect}T^{cam}_{velo}X_{3D}$, with depth retained as the pixel value, producing a $1 \times H \times W$ depth map that is concatenated with the $3 \times H \times W$ image into a $4 \times H \times W$ array. This turns unordered 3D points into an ordered 2D grid so existing CNN image transforms can process the point cloud without specialized operators. The second is MMFFT, the multi-modal feature fusion transform module: the image latent $y_{img}$ and the point-cloud latent $y_{pc}$ are concatenated along the channel dimension, projected to $M$ channels, and split into a residual path and a channel-attention path; adding the two paths produces the fused latent $y$ that is quantized and entropy-coded with a Gaussian mean-scale model. The attention path is what up-weights common image-point-cloud features and suppresses modality-specific noise.

What would settle it

Compress the point cloud and transmit it alongside the image, then compute the end-to-end BD-rate of PCA-NIC against the same image-only baseline; if including the point-cloud bits makes the 54.518% saving shrink to zero or reverse, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the 3D structure encoded in a point cloud is complementary to image texture and can be exploited before entropy coding rather than treated as a separate modality to be compressed. Projecting the point cloud onto the image with the KITTI calibration matrices yields a pixel-aligned depth map; concatenating it with the RGB image gives a $4 \times H \times W$ tensor that a conventional 2D neural codec can ingest. PCA-NIC runs separate analysis transforms for image and point cloud, then MMFFT concatenates the latents along the channel dimension and applies a channel-attention mechanism built from average and max pooling so that features shared by both modalities are up-weighted and redundant cross-modal information is pruned. The fused latent is quantized and entropy-coded, so the bitstream contains only image-side bits. Relative to the image-only baseline, the paper reports a 54.518% BD-rate reduction and 2.101 dB BD-PSNR improvement on KITTI.

Load-bearing premise

The point cloud is available to the codec without consuming any bitrate or latency budget, so all reported BD-rate and BD-PSNR gains count only image bits and never the cost of acquiring, encoding, or transmitting the lidar data.

Editorial extensions

If this is right

  • In systems that already collect lidar for other tasks, the image bitstream can be compressed with the point cloud as side information at no extra acquisition cost.
  • The unified representation lets existing 2D neural codec backbones be reused for multimodal input, avoiding the need for specialized point-based network operators.
  • The reported BD-rate and BD-PSNR gains count only image bits, so the method's real-world advantage depends on whether the point cloud is already available or must be transmitted as part of the total data budget.
  • The attention-based fusion between image and depth latents suggests that other pixel-aligned auxiliary modalities, such as stereo depth or radar maps, could yield similar compression gains.
  • The codec improves both PSNR and MS-SSIM, indicating that point-cloud assistance helps preserve texture and structural detail, not just low-frequency content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the 54.518% saving is best read as an upper bound for deployments that pay for the point cloud; a joint rate-distortion comparison that includes compressed lidar bits could shrink or erase the gap.
  • Inference: a direct test is to replace the lidar depth map with a monocular depth estimate at the encoder; preserving most of the BD-rate gain would show the method generalizes beyond lidar-equipped platforms.
  • Inference: the same fusion recipe could be applied to video compression, where temporally consistent depth from lidar or stereo may remove inter-frame redundancy more effectively than motion estimation alone.
  • Inference: the shared-feature up-weighting implies the method is really about suppressing cross-modal redundancy, so it may transfer to joint image-depth coding with a single bitstream serving both reconstruction and geometry estimation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PCA-NIC, a learned image codec that uses a co-captured LiDAR point cloud as auxiliary information to improve image compression on the KITTI dataset. The authors unify image and point cloud into a 4×H×W representation by projecting 3D points onto the image plane and using depth as an extra channel, then feed image and point cloud through separate analysis transforms, fuse the latents with a proposed MMFFT module, and encode the fused representation with an entropy model based on Cheng2020. They report BD-rate reductions of 54.518% in PSNR and 29.315% in MS-SSIM over Cheng2020, together with ablations of the attention mechanism in MMFFT and of replacing their entropy model with that of MLIC++. The paper claims to be the first to improve image compression using point cloud data.

Significance. If the reported gains are causally attributable to point cloud assistance, the result is significant for multimodal driving compression: it would show that an already-available depth stream can substantially reduce image bitrate. The paper has real strengths: it reports detailed training settings, multiple rate-distortion operating points, and comparisons against strong learned codecs, and the ablation results are directionally consistent with the main curves. However, the central attribution is not yet established, because no same-architecture control without the point cloud branch is provided, the information-theoretic motivation in Eq. (1) is formally incorrect, and the evaluation protocol for the auxiliary point cloud bits is not stated. These issues are fixable within the scope of the manuscript, so the appropriate decision is major revision rather than rejection.

major comments (4)
  1. [Experiments/Table 1] The central claim that point cloud assistance causes the large BD-rate reduction is not supported by the currently reported experiments. Table 1 compares PCA-NIC with Cheng2020, LIC TCM, and MLIC++, which differ in architecture, channel counts, attention mechanisms, and entropy model; the ablations PCA-NIC no attn and PCA-NIC MLIC++ still include the point cloud branch in every configuration. A same-architecture control 'PCA-NIC without point cloud' (same image analysis transform, synthesis transform, entropy model, and capacity, but no g_pc^a or MMFFT point-cloud input) is needed to isolate the contribution of the point cloud. Without it, the 54.518% BD-rate improvement over Cheng2020 cannot be causally attributed to point cloud assistance.
  2. [Proposed Method / Analysis, Eq. (1)] The information-theoretic motivation is formally incorrect. Eq. (1) writes H(I) = H(Iy) + H(In), which assumes that Iy and In are independent; in general H(I) = H(Iy) + H(In | Iy). Moreover, H(My) > H(Iy) does not imply lower coding cost, because the rate depends on the entropy of the quantized latent representation and on the resulting distortion; a larger or richer feature set may increase the coding cost. The theoretical argument should be corrected or removed, and the empirical comparison should be the basis for the claim.
  3. [Experiments / Rate-distortion evaluation] The reported BD-rate and BD-PSNR numbers count only image bits; the paper never states whether the point cloud is assumed to be free side information available at the encoder or must be transmitted or stored together with the image. If the lidar stream is part of the total compressed representation, the reported savings may shrink or disappear. The authors should state the assumption explicitly, and if joint transmission is intended, they should report rate-distortion results that include the point cloud bitrate.
  4. [Ablation Studies] The ablation naming and the description of the entropy model are ambiguous. The name 'PCA-NIC no attn' is used twice: once for MMFFT without attention and once for PCA-NIC with the MLIC++ entropy model. In addition, 'our entropy model' is never specified beyond saying it follows the context coding of [9] with a Gaussian mean-scale likelihood; given that the entropy model replacement is used as an ablation, its exact architecture and design choices must be described.
minor comments (4)
  1. [Introduction and Abstract] The text repeatedly cites 'cheng2020 [5]', but reference [5] is Ballé et al.'s end-to-end codec, while Cheng2020 is reference [9]. The citation markers should be corrected throughout.
  2. [Figure 1 caption] The caption defines Mimg twice in contradictory ways: it first says Mimg denotes the features belonging to the image in the mixed feature, then says features not belonging to the image are denoted as Mimg. This needs a clear notation, e.g., Mimg and Mnot-img.
  3. [Training Settings / Results] The paper gives inconsistent descriptions of the evaluation data: Training Settings says the testing dataset comprises randomly cropped 256×256 samples from the first 5000 KITTI test examples, while Results says the validation set comprises 256×1024-sized crops from regions with height 110 or above in the last 2518 test samples. The authors should clarify which set produced Table 1 and Fig. 5.
  4. [Throughout] There are several typos and awkward phrasings, including 'frature fusion transform' in the Fig. 4 caption, 'entopy model' in Ablation Studies, and 'increasement' in the analysis section. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the compression gains are measured against external codecs and no result reduces by construction to the point-cloud input or to a fitted parameter.

full rationale

The derivation chain in PCA-NIC is not circular. The central performance claims (BD-rate -54.518%, BD-PSNR +2.101 dB on KITTI) are benchmarked against Cheng2020, MLIC++, and LIC TCM, and these baselines are external codecs, not outputs of the paper's own fitting procedure. The point-cloud branch is encoder-side auxiliary input and is not defined in terms of the image bitrate it claims to improve; no parameter is fitted to the reported BD-rate and then renamed a prediction. The informal entropy analysis in Eq. (1) is mathematically flawed (it treats H(I) as H(Iy)+H(In) and H(My) as additive without conditioning), but that is an unsupported derivation rather than a circular one, and the experimental numbers do not depend on it. The absence of a same-architecture 'PCA-NIC without point cloud' ablation weakens causal attribution of the gain to point-cloud assistance, but this is a missing control, not a definitional equivalence or a self-citation substitution. There are no load-bearing self-citations or uniqueness claims imported from the authors' prior work. Accordingly, no circular step can be exhibited with the required quote-to-reduction standard.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper's main contribution is an empirical codec, so the ledger is dominated by domain assumptions about the auxiliary modality rather than fitted physical constants. The only hand-chosen numbers are standard training hyperparameters and architecture channel counts.

free parameters (2)
  • Rate-distortion Lagrange multipliers lambda = 0.0016, 0.0032, 0.0075, 0.015, 0.03, 0.045
    Standard rate-distortion tradeoff sweep used to produce the rate-distortion curve; chosen by hand, not fitted to derive a constant.
  • Latent channel counts N and M = N=192, M=288
    Architecture capacity inherited from Cheng2020 [9]; chosen by hand, with no sensitivity study reported.
assumptions (4)
  • domain assumption KITTI calibration matrices in Eq. (3) correctly project each 3D point onto the image pixel grid, and depth can serve as a valid extra image channel.
    The unified 4xHxW representation relies on this projection being accurate for all training and test samples; no handling of points falling outside the image or occluded pixels is described.
  • domain assumption Point cloud data is available to the codec without additional bitrate or latency cost.
    All BD-rate numbers count only image bits; if the point cloud must be transmitted or stored, total system rate is understated. This is the load-bearing assumption behind the headline saving.
  • domain assumption The entropy relationships in the analysis section imply that a neural network can extract more useful image features from mixed data.
    The paper treats H(My)>H(Iy) as evidence of improved compression, but this is an informal modeling assumption, not a proven rate-distortion theorem.
  • domain assumption The entropy model and context coding of Cheng2020 [9] can be adapted by changing channel counts without loss of correctness.
    The paper states the modules 'enhance the module of [9]' but gives no derivation or proof that the modified entropy model remains valid.
invented entities (1)
  • MMFFT module
    purpose: Fuses image and point cloud latent features by channel concatenation, convolution, and channel attention to emphasize shared information and suppress point-cloud-only content.
    The module is evaluated only through in-paper ablations and has no external falsifiable prediction outside the described experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Point Cloud-Assisted Neural Image Compression." pith.science (2026). https://pith.science/paper/AMVNZY36

@misc{pith2026241211771,
  author       = {Pith},
  title        = {Pith review of: Point Cloud-Assisted Neural Image Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AMVNZY36}},
  note         = {Machine review of arXiv:2412.11771}
}
read the original abstract

High-efficient image compression is a critical requirement. In several scenarios where multiple modalities of data are captured by different sensors, the auxiliary information from other modalities are not fully leveraged by existing image-only codecs, leading to suboptimal compression efficiency. In this paper, we increase image compression performance with the assistance of point cloud, which is widely adopted in the area of autonomous driving. We first unify the data representation for both modalities to facilitate data processing. Then, we propose the point cloud-assisted neural image codec (PCA-NIC) to enhance the preservation of image texture and structure by utilizing the high-dimensional point cloud information. We further introduce a multi-modal feature fusion transform module (MMFFT) to capture more representative image features, remove redundant information between channels and modalities that are not relevant to the image content. Our work is the first to improve image compression performance using point cloud and achieves state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2412.11771 by the authors.

Figure 1
Figure 1. Assuming all features of the image are I, the image features that can be extracted by the neural network are Iy, and the image features that cannot be extracted by the neural network are In. Similarly, regarding point cloud features, we have P, Py, and Pn. By using neural networks to simultaneously extract features from both image and point cloud, the total mixed feature that can be extracted is My, the shared featu… view at source ↗
Figure 2
Figure 2. The unified representation of image and point cloud. The projection is to trans [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The overall architecture of PCA-NIC. ↓ means down-sampling. ↑ means up￾sampling. RS is residual network. Attn is the attention module of [9]. N and M are channels, where N and M is equal to 192 and 288, respectively. As depicted in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The left part is the diagram of MMFFT, and the right is the frature fusion [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: PSNR-Bit-rate curve and MS-SSIM-Bit-rate curve. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the reconstructed 007517.png from the KITTI dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 22 canonical work pages

  1. [9]

    Learned image compression with discretized gaussian mixture likelihoods and attention modules,

    Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR) , 2020

    work page 2020

  2. [1]

    Jpeg: Still image data compression standard[m],

    WB Pennebaker and JL Mitchell, “Jpeg: Still image data compression standard[m],” Springer Science & Business Media , 1992

    work page 1992

  3. [2]

    Bpg image format,

    Fabrice Bellard, “Bpg image format,” https://bellard.org/bpg, 2015

    work page 2015

  4. [3]

    Web picture format,

    Google, “Web picture format,” 2010

    work page 2010

  5. [4]

    Overview of the versatile video coding (vvc) standard and its applications,

    Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens-Rainer Ohm, “Overview of the versatile video coding (vvc) standard and its applications,” IEEE Transactions on Circuits and Systems for Video Technology, 31(10), 3736-3764. , 2021

    work page 2021

  6. [5]

    End-to-end optimized image compression,

    Johannes Ball´ e, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimized image compression,” arxiv preprint arxiv:1611.01704 , 2016

    arXiv 2016

  7. [6]

    Variational image compression with a scale hyperprior,

    Johannes Ball´ e, David Minnen, Saurabh Singh, Sung Jin Hwang, Nick Johnston, and Google, “Variational image compression with a scale hyperprior,” arxiv preprint arxiv:1802.01436, 2018

    arXiv 2018

  8. [7]

    Joint autoregressive and hierarchical priors for learned image compression,

    David Minnen, Johannes Ball´ e, and George D Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” Advances in neural information processing systems 31, 2018

    work page 2018

Show all 25 references
  1. [8]

    Asym- metric gained deep image compression with continuous rate adaptation,

    Ze Cui, Jing Wang, Shangyin Gao, Tiansheng Guo, Yihui Feng, and Bo Bai, “Asym- metric gained deep image compression with continuous rate adaptation,” In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  2. [10]

    A unified end-to-end framework for efficient deep image compression,

    Jiaheng Liu, Guo Lu, Zhihao Hu, and Dong Xu, “A unified end-to-end framework for efficient deep image compression,” arXiv preprint arXiv:2002.03370 , 2020

    2002 arXiv

  3. [11]

    Causal contextual prediction for learned image compression,

    Z Guo, Z Zhang, R Feng, and Z Chen, “Causal contextual prediction for learned image compression,” IEEE Transactions on Circuits and Systems for Video Technology 32.4 , pp. 2329–2341, 2021

    2021

  4. [12]

    Checker- board context model for efficient learned image compression,

    Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin, “Checker- board context model for efficient learned image compression,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14771–14780, 2021

    2021

  5. [13]

    Channel-wise autoregressive entropy models for learned image compression,

    David Minnen and Saurabh Singh, “Channel-wise autoregressive entropy models for learned image compression,” In IEEE International Conference on Image Processing (ICIP), p. 3339–3343, 2020

    2020

  6. [14]

    Elic: Efficient learned image compression with unevenly grouped spacechannel contextual adaptive coding,

    Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang, “Elic: Efficient learned image compression with unevenly grouped spacechannel contextual adaptive coding,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , p. 5...

    2022

  7. [15]

    Mlic: Multi-reference entropy model for learned image compression,

    Wei Jiang, Jiayu Yang, Yongqi Zhai, Peirong Ning, Feng Gao, and Ronggang Wang, “Mlic: Multi-reference entropy model for learned image compression,” In Proceedings of the 31st ACM International Conference on Multimedia , 2023

    2023

  8. [16]

    Learned image compression via neighborhood-based attention optimization and context modeling with multi-scale guiding,

    Bo Li, Yongjun Li, Jincheng Luo, Xinru Zhang, Chaoyue Li, Zhimin Chenjin, and Yong Liang, “Learned image compression via neighborhood-based attention optimization and context modeling with multi-scale guiding,” Engineering Applications of Artificial Intelligence. 129: 107596 , 2024

    2024

  9. [17]

    Corner-to-center long-range context model for efficient learned image compres- sion[j],

    Yang Sui, Ding Ding, Xiang Pan, Xiaozhong Xu, Shan Liu, Bo Yuan, and Zhenzhong Chen, “Corner-to-center long-range context model for efficient learned image compres- sion[j],” Journal of Visual Communication and Image Representation. 98: 103990 , 2024

    2024

  10. [18]

    Deep residual learning for image recognition.,

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition.,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

    2016

  11. [19]

    Transformerbased transform coding,

    Yinhao Zhu, Yang Yang, and Taco Cohen, “Transformerbased transform coding,” In International Conference on Learning Representations , 2022

    2022

  12. [20]

    The devil is in the details: Window-based attention for image compression,

    Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang, “The devil is in the details: Window-based attention for image compression,” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022

    2022

  13. [21]

    Entroformer: A transformer-based entropy model for learned image compression,

    Yichen Qian, Xiuyu Sun, Zhiyu Tan Ming Lin, and Rong Jin, “Entroformer: A transformer-based entropy model for learned image compression,” In International Conference on Learning Representations, 2021

    2021

  14. [22]

    Contextformer: A trans- former with spatio-channel attention for context modeling in learned image compres- sion,

    A Burakhan Koyuncu, Han Gao, and Eckehard Steinbach, “Contextformer: A trans- former with spatio-channel attention for context modeling in learned image compres- sion,” In European Conference on Computer Vision , 2022

    2022

  15. [23]

    Learned image compression with mixed transformer-cnn architectures,

    Jinming Liu, Heming Sun, and Jiro Katto, “Learned image compression with mixed transformer-cnn architectures,” In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , 2023

    2023

  16. [24]

    Non-local attention optimized deep image compression,

    Haojie Liu, Tong Chen, Peiyao Guo, Qiu Shen, Xun Cao, Yao Wang, and Zhan Ma, “Non-local attention optimized deep image compression,” arXiv preprint arXiv:1904.09757, 2019

    1904 arXiv

  17. [25]

    Vision meets robotics: The kitti dataset[j],

    A Geiger, P Lenz, C Stiller, and et al., “Vision meets robotics: The kitti dataset[j],” The International Journal of Robotics Research , 2013

    2013

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.