{"id":"b28f155f-760f-4ed2-8eca-311e2788f649","arxiv_id":"2412.15752","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Projecting sparse LiDAR depth into predicted structural features and injecting them into learned image codecs consistently improves rate-distortion performance on KITTI and Waymo.","lead":"This paper adds a LiDAR point cloud as free side information to learned image compression for autonomous driving, improving BD-Rate by 3 to 17 percent on KITTI and Waymo. It is a practical engineering contribution that could reduce camera-image bandwidth when point clouds are already transmitted for other reasons.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported BD-Rate gains charge zero bits for the point cloud; if its bitrate is counted, end-to-end benefit may vanish, and the G-PCC test omits the point-cloud rate.","rationale":"The central image-level claim is plausible and supported by reasonable controls: the parameter-augmentation control (ELIC-PA versus ELIC-PC, Table III) and the zero-tensor control (Table IV) indicate that point-cloud signal contributes beyond added parameters and architecture. The load-bearing gap is the unconditional phrasing of the claim in the abstract and conclusion. Because the paper explicitly assumes the point cloud is available at no counted bitrate (Secs. III-A and IV-A), this is a flagged limitation rather than an internal inconsistency. The G-PCC lossy experiment was the natural place to quantify the point-cloud rate, but the paper omits it, leaving the end-to-end benefit unverified. The one concrete check of adding point-cloud bitrate to the x-axis would settle whether the practical claim survives. This matches the reader's weakest assumption, so the CONDITIONAL verdict stands unchanged. I do not see a more load-bearing concern: the ablations are well designed, the datasets are appropriate, and the training details are stated. The main reservation is the missing end-to-end accounting, which the proposed test directly addresses.","tokens_in":15904,"tokens_out":9288,"duration_ms":83688,"concrete_test":"Take the exact G-PCC encoded point-cloud bitstreams already generated for Figure 10 (KITTI, ELIC-PC with lossy point clouds), record their byte sizes, convert to bpp using the same 1242x256 image dimensions, add these to the corresponding image bitrates, and recompute the BD-Rate of ELIC-PC versus ELIC on the total-rate axis. If the recomputed BD-Rate remains negative, the point-cloud bitrate does not erase the gain and the concern is resolved; if it becomes positive or within run-to-run noise, the end-to-end claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that incorporating point cloud assistance consistently enhances compression performance, but every reported BD-Rate (Table I, Figures 7-9) is computed from image bitrate only. Section III-A explicitly assumes 'priority transmission of the point cloud,' and Section IV-A states that point cloud frames are 'assumed to have been losslessly compressed.' This is a clearly flagged scope limitation, not a hidden error. However, the G-PCC robustness experiment (Figure 10) was the natural place to close the end-to-end gap: the authors encode point clouds lossily at the highest G-PCC bitrate and measure a 2.41% image-level RD degradation, but they never report the point-cloud byte count or add it to the x-axis. In the autonomous-driving scenario that motivates the paper, the LiDAR stream is part of the data to be stored or transmitted. If its bitrate exceeds the image bitrate savings, the method provides no end-to-end compression benefit. Without the point-cloud rate, the headline claim 'consistently enhances the performance' is only established for a subsystem, not for the compression system as a whole.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for using sparse LiDAR point clouds as auxiliary side information in learned image compression for autonomous driving. The method projects a point cloud to a sparse depth map (PCP), predicts dense structural features from it (PIP), extracts multi-scale context (MCM), and injects these features into existing learned codecs (HYPER, ELIC, MLIC) in both the transform and entropy-model branches. Evaluations on KITTI and Waymo report BD-Rate improvements of about 6.05%, 15.67%, and 2.80% for ELIC, HYPER, and MLIC on KITTI relative to ELIC, with similar gains on Waymo. Extensive ablations include removing PIP/MCM components, using point cloud information only at the encoder or decoder, a zero-tensor control that isolates architectural gain from point-cloud signal, a parameter-augmentation control, and a G-PCC lossy point-cloud robustness test.","tokens_in":16147,"tokens_out":5158,"duration_ms":49414,"significance":"If the reported gains hold, the work is a useful contribution to multi-modal compression for autonomous driving, where camera and LiDAR streams are naturally co-located. The paper is empirically careful in several respects: test scenes are separated from training scenes, multiple codec baselines are used, the zero-tensor control (Table IV) distinguishes signal gain from parameter gain, the parameter-augmentation control (Table III, Fig. 15) strengthens the attribution of the gain, and the G-PCC lossy test (Fig. 10) addresses robustness to imperfect point-cloud inputs. The main weakness is that all rate-distortion numbers charge zero bits to the point-cloud stream, so the headline claim of 'consistently enhancing performance' is currently established only for the image subsystem, not for the full sensor-compression system.","major_comments":[{"comment":"","section":"Sec. IV-B2 and Sec. IV-A"},{"comment":"","section":"Sec. III-C, Eq. (5)"}],"minor_comments":[{"comment":"","section":"Sec. IV-A"},{"comment":"","section":"Sec. IV-B2"},{"comment":"","section":"Sec. IV-D, Table V"},{"comment":"","section":"Sec. III-D, Fig. 6"},{"comment":"","section":"References"},{"comment":"","section":"Sec. III-C, Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the missing point-cloud bitrate, which I have flagged as a major comment. The paper's assumption of priority/lossless point-cloud transmission is explicitly stated, so this is not a hidden error, but for a systems-oriented journal like TCSVT the authors should either add the end-to-end measurement or clearly delimit the claim. The zero-tensor and parameter-augmentation controls are commendable and made the empirical evaluation stronger. The novelty relative to Gnutti et al. (Ref. [32]) is adequately argued via LiDAR sparsity, though the distinction could be sharpened in the text. I recommend major revision rather than rejection because the missing rate information is readily addable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuinely new and reasonably careful empirical paper. The idea is simple: in autonomous driving, LiDAR point clouds are often already available, so use them to help compress the camera images. As far as I know, this is the first work to do this with sparse LiDAR point clouds, and the reported gains are plausible: 3–6% BD-Rate on top of strong baselines like ELIC and MLIC, and up to 17% for the hyperprior baseline.\n\nThe paper earns its keep on the experimental side. The zero-tensor control in Table IV separates the gain from added model capacity from the gain from actual point cloud signal, and the parameter-augmentation control in Table III rules out the trivial explanation that the improvement is just extra parameters. The G-PCC lossy point cloud experiment shows the method is robust to imperfect inputs, which is a real deployment concern. The ablations of PIP and MCM are thoughtfully designed, and the finding that their gains are superadditive is interesting.\n\nThe soft spots are real but not hidden. The most important is that all BD-Rate numbers are computed on image bitrate only. The paper explicitly assumes the point cloud is transmitted with priority or losslessly compressed, so this is a stated scope limitation, not an error. But the G-PCC experiment was the natural place to close the end-to-end gap, and they don't report the point cloud byte count or add it to the rate axis. For a pure compression story, the end-to-end benefit is still open. Also, all results are single runs with no error bars or significance measures, and no code or checkpoints are provided, so independent verification is limited. These are fixable in a revision.\n\nThe citation pattern looks fine. The authors correctly separate [31], where images assist point cloud compression, and [32], which uses denser phone depth maps rather than sparse LiDAR projections.\n\nWho is this for? Anyone working on multi-modal compression for autonomous driving, and the learned image compression community more broadly. It's a useful engineering contribution with a clean experimental design, and the open end-to-end question is a reviewer discussion point rather than a fundamental flaw.\n\nRecommendation: send it to peer review. The central claim about image compression with free auxiliary sensor data is well supported within its stated scope, and the paper deserves referee time. I'd ask for code, error bars, and an end-to-end comparison with point cloud bitrate counted before accepting the practical benefit as fully established.","headline":"Genuinely new use of sparse LiDAR point clouds to improve learned image compression, with honest scoping and solid controls; the end-to-end bitrate question remains open.","tokens_in":16666,"tokens_out":2765,"would_cite":true,"duration_ms":21921,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding sparse LiDAR point clouds into learned image compression cuts bitrate by up to 17% on autonomous-driving benchmarks while preserving structures.","keywords":["learned image compression","sparse point cloud","LiDAR","multi-modal compression","autonomous driving","depth map","cross-modal prediction","rate-distortion optimization"],"falsifier":"Measure end-to-end rate-distortion performance on KITTI or Waymo while charging the compressed size of the LiDAR point cloud (lossless, or G-PCC at the tested configuration) against the total bitrate; if the point-cloud-assisted codec no longer beats the unassisted codec in BD-Rate under this accounting, the practical claim collapses.","tokens_in":15694,"feed_emoji":"🚗","tokens_out":7181,"duration_ms":56996,"temperature":0.7,"pith_summary":"This paper argues that in an autonomous vehicle, where a camera image and a sparse LiDAR point cloud capture the same scene, the point cloud can serve as free side information to make learned image compression more efficient. The authors propose a framework that projects the 3D point cloud into a 2D depth map, learns to predict a dense image-like structure from that sparse map, and injects the resulting multi-scale features into the encoder, decoder, and entropy model of an existing learned compression network. Across three compression backbones (ELIC, HYPER, MLIC) on the KITTI and Waymo datasets, they report consistent bitrate savings at matched visual quality, from about 2.8% to 15.7% on KITTI. A zero-tensor control shows part of the gain is just added model capacity, but about 3.7 percentage points of ELIC's gain comes from the point cloud signal itself, and the gain persists when the point cloud is lossy-compressed with G-PCC.","feed_headline":"Point clouds trim learned image compression rates by up to 17%","feed_subtitle":"A generic assist module plugs into ELIC, HYPER, and MLIC, cutting bitrate while keeping reconstructed structures intact.","key_machinery":"The load-bearing mechanism is the point-cloud feature injection path: Point Cloud Projection (PCP) maps 3D LiDAR points into a 2D depth map aligned with the camera view; Point-to-image Prediction (PIP) densifies that sparse depth map into an image-like prediction using randomized color transforms that decouple structure from photometry; Multi-scale Context Mining (MCM) generates three scales of features that are concatenated into the transform at matching scales and fused into a hyper-prior feature via a small Hyper Refiner (HR) that replaces the entropy model's mean and variance prediction. This path turns a sparse geometric hint into dense structural side information that the compression network can condition on at both encoder and decoder.","core_discovery":"The central claim is that sparse LiDAR point clouds can be turned into a useful conditional side channel for learned image compression, even though the point cloud is far sparser than the image and shares only structural, not photometric, information with it. The paper's proposed design—point cloud projection to a normalized, histogram-equalized depth map; Point-to-image Prediction (PIP) to densify it into a predicted image with randomized color targets; Multi-scale Context Mining (MCM) to extract multi-scale structural features; and a Hyper Refiner to fuse point-cloud features into the entropy model—consistently improves rate-distortion performance when plugged into HYPER, ELIC, and MLIC. The ablation studies attribute most of the gain to the decoding side, suggesting the side information mainly improves reconstruction rather than decorrelation. The paper further demonstrates robustness to lossy point clouds and shows that the point-cloud-specific gain exceeds what an equal number of added parameters achieves without the point cloud.","pith_inferences":["Editorial extension: the reported savings are image-only bitrate; if the LiDAR stream's own bits are charged, the end-to-end benefit is unmeasured and could vanish, since the paper assumes the point cloud is already available at both ends.","Editorial extension: the same projection-and-densification approach could be tested with other sparse geometric sensors such as radar, SLAM depth, or stereo disparity, and the multi-scale feature design suggests the gain should grow with point density and with how much structure the side channel shares with the image.","Editorial extension: a direct test of the structural-cue hypothesis would be to feed the network a point cloud with depth values randomly permuted inside objects; if the compression gain persists, the mechanism is not geometric structure but something else in the network."],"forward_implications":["The framework is not tied to one codec: the same point-cloud feature path improved HYPER, ELIC, and MLIC, and the injection points are generic enough to apply to other learned codecs with a hyperprior.","The assist is cheap at inference: decoding time is essentially unchanged from the base model and point-cloud processing costs only a few milliseconds per frame (6.4 ms for ELIC-PC).","The method remains useful when the point cloud is lossy-compressed: with G-PCC lossy input and no retraining, ELIC-PC still beats the unassisted ELIC on KITTI, though the low-bitrate end loses more than the high-bitrate end.","Because the gains appear mainly at the decoder side (4.32% vs 0.42% for encoder-only on ELIC), the side information acts primarily as a reconstruction aid rather than a decorrelation aid, suggesting the architecture mainly improves how the decoded image is synthesized."],"supporting_citations":[{"why":"Supplies the HYPER baseline, one of the three learned compression models the point-cloud branch is added to.","marker":"[14]"},{"why":"Supplies the ELIC baseline, which anchors all BD-Rate comparisons and is the second test model.","marker":"[20]"},{"why":"Supplies the MLIC baseline, the third test model the point-cloud branch is added to.","marker":"[22]"},{"why":"Supplies the KITTI dataset of paired camera images and LiDAR point clouds used for training and evaluation.","marker":"[35]"},{"why":"Supplies the Waymo dataset, the second paired image and point-cloud benchmark with more varied scenes.","marker":"[48]"},{"why":"Provides the G-PCC lossy point-cloud compression used to test the framework's robustness to imperfect point clouds.","marker":"[49]"}],"fun_headline_variants":["LiDAR side channel slashes learned image compression bitrate","Point clouds give learned image compression a 17% rate cut","Cross-sensor trick: LiDAR depth cuts image compression bitrate","Sparse LiDAR data boosts learned image compression efficiency","How sparse point clouds give learned compression a leg up"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The point cloud is assumed to be available at both encoder and decoder without any bitrate cost; if transmitting or storing that point cloud is charged against the system, the reported compression gains may shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["LiDAR side channel slashes learned image compression bitrate","Point clouds give learned image compression a 17% rate cut","Cross-sensor trick: LiDAR depth cuts image compression bitrate","Sparse LiDAR data boosts learned image compression efficiency","How sparse point clouds give learned compression a leg up"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2586,"prompt_tokens":917,"completion_tokens":1669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1587}},"tokens_in":533,"tokens_out":1669,"duration_ms":9705,"temperature":1.0,"reasoning_tokens":1587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:08:03.994104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure end-to-end rate-distortion performance on KITTI or Waymo while charging the compressed size of the LiDAR point cloud (lossless, or G-PCC at the tested configuration) against the total bitrate; if the point-cloud-assisted codec no longer beats the unassisted codec in BD-Rate under this accounting, the practical claim collapses.","supporting_citations":[{"cited_title":"Varia- tional image compression with a scale hyperprior,","cited_arxiv_id":null,"evidence_quote":"Supplies the HYPER baseline, one of the three learned compression models the point-cloud branch is added to."},{"cited_title":"Elic: Efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,","cited_arxiv_id":null,"evidence_quote":"Supplies the ELIC baseline, which anchors all BD-Rate comparisons and is the second test model."},{"cited_title":"Mlic: Multi- reference entropy model for learned image compression,","cited_arxiv_id":null,"evidence_quote":"Supplies the MLIC baseline, the third test model the point-cloud branch is added to."},{"cited_title":"Scalability in perception for autonomous driving: Waymo open dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the Waymo dataset, the second paired image and point-cloud benchmark with more varied scenes."},{"cited_title":"G-pcc codec description v5,","cited_arxiv_id":null,"evidence_quote":"Provides the G-PCC lossy point-cloud compression used to test the framework's robustness to imperfect point clouds."}],"review_version":1}