{"id":"26d0de45-d3b9-42e1-8961-6a5ab433a1a4","arxiv_id":"2412.11771","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Point cloud depth projected onto the image, fused through a new attention module, improves learned image compression on KITTI by 54.5% BD-rate over Cheng2020.","lead":"This paper builds a neural image compressor that accepts a point cloud as extra input, projecting lidar depth onto the image plane and fusing it with image features. The authors report 54.5% bitrate savings over a strong learned codec on KITTI, which would matter for vehicles that already carry both cameras and lidar.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing same-architecture ablation without point cloud leaves the 54.5% BD-rate gain unattributed; the improvement over Cheng2020 may stem from added capacity or the entropy model, not from point cloud assistance.","rationale":"The reader's verdict is CONDITIONAL because of a suspected free-side-information bitrate problem. I find that concern less central: the architecture as described uses the point cloud only in the encoder analysis path, before quantization, so the point cloud does not need to be transmitted to the decoder. The stronger issue is causal attribution. The paper claims 'point cloud-assisted' compression and reports a large gain over image-only codecs, but all reported variants include the point cloud input. The gain could come from the higher-capacity analysis/synthesis transforms, the attention module, or the entropy model, none of which is specific to point cloud. A same-architecture ablation without point cloud is the minimal experiment needed to support the central claim. Because such an ablation is absent, the central claim is underdetermined; however, this does not prove the claim false, so a conditional verdict remains appropriate. The verdict should be CONDITIONAL (or UNCHANGED from the reader's CONDITIONAL), pending the requested ablation.","tokens_in":6899,"tokens_out":7321,"duration_ms":71165,"concrete_test":"Implement and train a 'PCA-NIC w/o point cloud' variant that uses the same encoder, decoder, and entropy model as PCA-NIC but replaces the point cloud input with a constant zero depth map (or removes the g_pc^a and MMFFT point-cloud path entirely), under identical training settings and validation set. Compare its BD-rate and BD-PSNR against Cheng2020 and against the full PCA-NIC. If the no-point-cloud variant retains most of the 54.518% BD-rate gain, the point cloud is not the cause and the central claim fails; if the gain drops substantially, the point cloud contribution is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that point cloud assistance causes the large BD-rate improvement, but no experiment isolates the contribution of the point cloud input. Table 1 compares PCA-NIC against Cheng2020 and MLIC++, which differ in architecture, channel widths (N=192, M=288), attention mechanisms, and entropy model. The ablations shown, PCA-NIC no attn and PCA-NIC MLIC++, still use point cloud in every row; they test the attention module and the entropy model, not the point cloud branch. Without a 'PCA-NIC w/o point cloud' baseline, the 54.518% BD-rate reduction over Cheng2020 could be due to the larger model capacity or the modified entropy model rather than to the point cloud itself. The information-theoretic motivation in Eq. (1) is also flawed: H(I) = H(Iy) + H(In) assumes independence between Iy and In, but generally H(I) = H(Iy) + H(In | Iy); this argument does not establish that adding point cloud features improves compression. The reader's concern about point cloud bitrate is less decisive because the decoder does not appear to need the point cloud: the point cloud branch (g_pc^a, MMFFT) operates before quantization, so point cloud is encoder-side auxiliary information and no point cloud bits are transmitted. The missing attribution is therefore the most load-bearing issue: the reported gain cannot be causally assigned to point cloud assistance without a same-architecture control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PCA-NIC, a learned image codec that uses a co-captured LiDAR point cloud as auxiliary information to improve image compression on the KITTI dataset. The authors unify image and point cloud into a 4×H×W representation by projecting 3D points onto the image plane and using depth as an extra channel, then feed image and point cloud through separate analysis transforms, fuse the latents with a proposed MMFFT module, and encode the fused representation with an entropy model based on Cheng2020. They report BD-rate reductions of 54.518% in PSNR and 29.315% in MS-SSIM over Cheng2020, together with ablations of the attention mechanism in MMFFT and of replacing their entropy model with that of MLIC++. The paper claims to be the first to improve image compression using point cloud data.","tokens_in":7196,"tokens_out":4999,"duration_ms":47371,"significance":"If the reported gains are causally attributable to point cloud assistance, the result is significant for multimodal driving compression: it would show that an already-available depth stream can substantially reduce image bitrate. The paper has real strengths: it reports detailed training settings, multiple rate-distortion operating points, and comparisons against strong learned codecs, and the ablation results are directionally consistent with the main curves. However, the central attribution is not yet established, because no same-architecture control without the point cloud branch is provided, the information-theoretic motivation in Eq. (1) is formally incorrect, and the evaluation protocol for the auxiliary point cloud bits is not stated. These issues are fixable within the scope of the manuscript, so the appropriate decision is major revision rather than rejection.","major_comments":[{"comment":"The central claim that point cloud assistance causes the large BD-rate reduction is not supported by the currently reported experiments. Table 1 compares PCA-NIC with Cheng2020, LIC TCM, and MLIC++, which differ in architecture, channel counts, attention mechanisms, and entropy model; the ablations PCA-NIC no attn and PCA-NIC MLIC++ still include the point cloud branch in every configuration. A same-architecture control 'PCA-NIC without point cloud' (same image analysis transform, synthesis transform, entropy model, and capacity, but no g_pc^a or MMFFT point-cloud input) is needed to isolate the contribution of the point cloud. Without it, the 54.518% BD-rate improvement over Cheng2020 cannot be causally attributed to point cloud assistance.","section":"Experiments/Table 1"},{"comment":"The information-theoretic motivation is formally incorrect. Eq. (1) writes H(I) = H(Iy) + H(In), which assumes that Iy and In are independent; in general H(I) = H(Iy) + H(In | Iy). Moreover, H(My) > H(Iy) does not imply lower coding cost, because the rate depends on the entropy of the quantized latent representation and on the resulting distortion; a larger or richer feature set may increase the coding cost. The theoretical argument should be corrected or removed, and the empirical comparison should be the basis for the claim.","section":"Proposed Method / Analysis, Eq. (1)"},{"comment":"The reported BD-rate and BD-PSNR numbers count only image bits; the paper never states whether the point cloud is assumed to be free side information available at the encoder or must be transmitted or stored together with the image. If the lidar stream is part of the total compressed representation, the reported savings may shrink or disappear. The authors should state the assumption explicitly, and if joint transmission is intended, they should report rate-distortion results that include the point cloud bitrate.","section":"Experiments / Rate-distortion evaluation"},{"comment":"The ablation naming and the description of the entropy model are ambiguous. The name 'PCA-NIC no attn' is used twice: once for MMFFT without attention and once for PCA-NIC with the MLIC++ entropy model. In addition, 'our entropy model' is never specified beyond saying it follows the context coding of [9] with a Gaussian mean-scale likelihood; given that the entropy model replacement is used as an ablation, its exact architecture and design choices must be described.","section":"Ablation Studies"}],"minor_comments":[{"comment":"The text repeatedly cites 'cheng2020 [5]', but reference [5] is Ballé et al.'s end-to-end codec, while Cheng2020 is reference [9]. The citation markers should be corrected throughout.","section":"Introduction and Abstract"},{"comment":"The caption defines Mimg twice in contradictory ways: it first says Mimg denotes the features belonging to the image in the mixed feature, then says features not belonging to the image are denoted as Mimg. This needs a clear notation, e.g., Mimg and Mnot-img.","section":"Figure 1 caption"},{"comment":"The paper gives inconsistent descriptions of the evaluation data: Training Settings says the testing dataset comprises randomly cropped 256×256 samples from the first 5000 KITTI test examples, while Results says the validation set comprises 256×1024-sized crops from regions with height 110 or above in the last 2518 test samples. The authors should clarify which set produced Table 1 and Fig. 5.","section":"Training Settings / Results"},{"comment":"There are several typos and awkward phrasings, including 'frature fusion transform' in the Fig. 4 caption, 'entopy model' in Ablation Studies, and 'increasement' in the analysis section. A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The missing same-architecture control is the main technical gap and is directly fixable with an additional experiment. The point-cloud bitrate issue is a matter of stating and, if necessary, reporting the evaluation protocol rather than a fundamental flaw. I recommend that the editor require the authors to add the control, correct or remove the entropy argument, and clarify the bitrate accounting before resubmission. The novelty claim 'first to improve image compression using point cloud' should also be checked against prior multimodal compression work during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely new application: no one has used point cloud as side information to a learned image codec, and the unified 4xHxW projection plus the MMFFT fusion are reasonable components. Second, the headline 54.5% BD-rate gain over Cheng2020 is not yet causally attributable to the point cloud, because every reported model still has point cloud input; there is no same-architecture control without point cloud.\n\nThe paper does several things well. The projection of depth onto the image plane follows the standard KITTI calibration and is clearly described. PCA-NIC is straightforward: separate analysis transforms for image and point cloud, then a fusion module that applies channel attention to the concatenated features. The ablations show that the attention in MMFFT and the custom entropy model improve on their variants. Experiments use a reasonable range of rate-distortion trade-offs and report both PSNR and MS-SSIM.\n\nThe load-bearing flaw is the missing attribution. PCA-NIC differs from Cheng2020 in capacity, attention, and entropy model; without a 'PCA-NIC without point cloud' baseline, the gain could come from added capacity. The reader's point-cloud bitrate concern is, I think, less decisive: the point cloud is used only at the encoder, before quantization, and no point-cloud bits are transmitted, so in a multimodal sensor suite this is legitimate auxiliary data. More serious is the entropy model: the paper never says what 'our entropy model' is beyond a Gaussian mean-scale with context from Cheng2020, and the ablation table lists two different variants under the same name 'PCA-NIC no attn'. The information-theoretic motivation in Eq. (1) assumes H(I)=H(Iy)+H(In), which is not generally true, and H(My)>H(Iy) does not by itself imply better compression. No error bars or code are provided, which matters for a 54% claim.\n\nWho this is for: researchers working on multimodal driving data and learned compression, or on auxiliary-data side information for codecs. It is a solid first step, not a definitive result. The direction deserves serious referee time.\n\nRecommendation: send to peer review, but require a same-architecture no-point-cloud ablation, a precise description of the entropy model, corrected ablation naming, and ideally code/data.","headline":"Genuinely new application of point-cloud side information to learned image compression, but the headline bitrate gain is not yet causally attributed to the point cloud without a same-architecture control.","tokens_in":7734,"tokens_out":2973,"would_cite":true,"duration_ms":30660,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural codec that fuses lidar point clouds as auxiliary input cuts image bitrate by 54.518% on the KITTI dataset.","keywords":["point-cloud-assisted image compression","learned image compression","multi-modal feature fusion","unified representation","depth map projection","KITTI","BD-rate","channel attention"],"falsifier":"Compress the point cloud and transmit it alongside the image, then compute the end-to-end BD-rate of PCA-NIC against the same image-only baseline; if including the point-cloud bits makes the 54.518% saving shrink to zero or reverse, the paper's central claim is falsified.","tokens_in":6712,"feed_emoji":"🚗","tokens_out":10922,"duration_ms":97567,"temperature":0.7,"pith_summary":"The paper tries to establish that a lidar point cloud, projected into the camera image plane and fed to a neural codec as extra input, can substantially reduce the bitrate needed to store or transmit the image. The proposed codec PCA-NIC builds a shared $4 \\times H \\times W$ representation from the RGB image and a depth map derived from the point cloud, then extracts and fuses features from both modalities with a channel-attention fusion module. On the KITTI driving dataset, it reports a 54.518% BD-rate reduction and a 2.101 dB BD-PSNR improvement over the image-only learned codec baseline, with consistent gains in MS-SSIM. The authors further argue from an entropy decomposition that mixed-modality feature extraction gives the network strictly more information than image-only extraction, which is why the auxiliary depth helps.","feed_headline":"Point-cloud side info cuts image bitrate by 54.5%","feed_subtitle":"A neural codec that fuses lidar depth as extra input beats image-only baselines on KITTI.","key_machinery":"Two mechanisms carry the argument. The first is the unified representation: a point cloud is projected into the pixel coordinate system by $X_{2D}=P^{(2)}_{rect}R^{(0)}_{rect}T^{cam}_{velo}X_{3D}$, with depth retained as the pixel value, producing a $1 \\times H \\times W$ depth map that is concatenated with the $3 \\times H \\times W$ image into a $4 \\times H \\times W$ array. This turns unordered 3D points into an ordered 2D grid so existing CNN image transforms can process the point cloud without specialized operators. The second is MMFFT, the multi-modal feature fusion transform module: the image latent $y_{img}$ and the point-cloud latent $y_{pc}$ are concatenated along the channel dimension, projected to $M$ channels, and split into a residual path and a channel-attention path; adding the two paths produces the fused latent $y$ that is quantized and entropy-coded with a Gaussian mean-scale model. The attention path is what up-weights common image-point-cloud features and suppresses modality-specific noise.","core_discovery":"The central discovery is that the 3D structure encoded in a point cloud is complementary to image texture and can be exploited before entropy coding rather than treated as a separate modality to be compressed. Projecting the point cloud onto the image with the KITTI calibration matrices yields a pixel-aligned depth map; concatenating it with the RGB image gives a $4 \\times H \\times W$ tensor that a conventional 2D neural codec can ingest. PCA-NIC runs separate analysis transforms for image and point cloud, then MMFFT concatenates the latents along the channel dimension and applies a channel-attention mechanism built from average and max pooling so that features shared by both modalities are up-weighted and redundant cross-modal information is pruned. The fused latent is quantized and entropy-coded, so the bitstream contains only image-side bits. Relative to the image-only baseline, the paper reports a 54.518% BD-rate reduction and 2.101 dB BD-PSNR improvement on KITTI.","pith_inferences":["Inference: the 54.518% saving is best read as an upper bound for deployments that pay for the point cloud; a joint rate-distortion comparison that includes compressed lidar bits could shrink or erase the gap.","Inference: a direct test is to replace the lidar depth map with a monocular depth estimate at the encoder; preserving most of the BD-rate gain would show the method generalizes beyond lidar-equipped platforms.","Inference: the same fusion recipe could be applied to video compression, where temporally consistent depth from lidar or stereo may remove inter-frame redundancy more effectively than motion estimation alone.","Inference: the shared-feature up-weighting implies the method is really about suppressing cross-modal redundancy, so it may transfer to joint image-depth coding with a single bitstream serving both reconstruction and geometry estimation."],"forward_implications":["In systems that already collect lidar for other tasks, the image bitstream can be compressed with the point cloud as side information at no extra acquisition cost.","The unified representation lets existing 2D neural codec backbones be reused for multimodal input, avoiding the need for specialized point-based network operators.","The reported BD-rate and BD-PSNR gains count only image bits, so the method's real-world advantage depends on whether the point cloud is already available or must be transmitted as part of the total data budget.","The attention-based fusion between image and depth latents suggests that other pixel-aligned auxiliary modalities, such as stereo depth or radar maps, could yield similar compression gains.","The codec improves both PSNR and MS-SSIM, indicating that point-cloud assistance helps preserve texture and structural detail, not just low-frequency content."],"supporting_citations":[{"why":"The image-only learned codec baseline that PCA-NIC extends and compares against for BD-rate and BD-PSNR.","marker":"[9]"},{"why":"Foundational end-to-end learned image compression architecture that supplies the transform-quantize-entropy-code template.","marker":"[5]"},{"why":"Hyperprior-based variational image compression that provides the side-information and Gaussian entropy-modeling basis.","marker":"[6]"},{"why":"The KITTI dataset and its calibration matrices that provide the image, point cloud, and projection parameters used in the unified representation.","marker":"[25]"},{"why":"The MLIC++ entropy model used in the ablation to test whether the proposed entropy model or the baseline drives the gains.","marker":"[15]"}],"fun_headline_variants":["Point cloud side info slashes image bitrate by 54.5%","Lidar depth cuts image compression bitrate 54.5%","Neural codec with point cloud beats image-only by 54.5%","Fusing point clouds improves image compression 54.5%","Point cloud-assisted codec saves 54.5% bitrate on KITTI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The point cloud is available to the codec without consuming any bitrate or latency budget, so all reported BD-rate and BD-PSNR gains count only image bits and never the cost of acquiring, encoding, or transmitting the lidar data.","fun_headline_variants_meta":{"raw":{"variants":["Point cloud side info slashes image bitrate by 54.5%","Lidar depth cuts image compression bitrate 54.5%","Neural codec with point cloud beats image-only by 54.5%","Fusing point clouds improves image compression 54.5%","Point cloud-assisted codec saves 54.5% bitrate on KITTI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1817,"prompt_tokens":896,"completion_tokens":921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":821}},"tokens_in":512,"tokens_out":921,"duration_ms":8300,"temperature":1.0,"reasoning_tokens":821,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:35:52.479115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compress the point cloud and transmit it alongside the image, then compute the end-to-end BD-rate of PCA-NIC against the same image-only baseline; if including the point-cloud bits makes the 54.518% saving shrink to zero or reverse, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"Learned image compression with discretized gaussian mixture likelihoods and attention modules,","cited_arxiv_id":null,"evidence_quote":"The image-only learned codec baseline that PCA-NIC extends and compares against for BD-rate and BD-PSNR."},{"cited_title":"Vision meets robotics: The kitti dataset[j],","cited_arxiv_id":null,"evidence_quote":"The KITTI dataset and its calibration matrices that provide the image, point cloud, and projection parameters used in the unified representation."},{"cited_title":"Mlic: Multi-reference entropy model for learned image compression,","cited_arxiv_id":null,"evidence_quote":"The MLIC++ entropy model used in the ablation to test whether the proposed entropy model or the baseline drives the gains."}],"review_version":1}