{"id":"75ed958f-4a2d-4bf1-8cd2-988f7761740e","arxiv_id":"2505.09562","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OffsetOcc adds DETR-style object queries with offset-based shape prediction to existing 3D occupancy models, producing panoptic scene completion from cameras alone.","lead":"This paper introduces OffsetOcc, a camera-only system that predicts 3D object shapes as sets of offset vectors from object centers, then fuses them with an occupancy grid to label individual object instances. It is a possible building block for autonomous vehicles that need a complete, instance-aware map of occluded areas.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Panoptic Module assigns instance IDs to baseline object voxels but never writes the Object Module's predicted occupancy into the grid, so differentiable object shapes cannot add occluded voxels; the claimed completion is untested.","rationale":"The reader correctly identified that the panoptic evaluation is a LIDAR proxy rather than a direct test of occluded-voxel instance completion, and that the offline pseudo-GT from bounding box / semantic voxel intersection is noisy. My read adds a more structural reason why the central claim is not demonstrated: the described Panoptic Module does not insert the Object Module's predicted occupancy into the final grid, so object-shape completion cannot improve occluded-region recall beyond what the frozen baseline already predicts. This makes the headline claim “panoptic scene completion through differentiable object shapes” internally weaker than the evaluation alone would suggest.\n\nI do not regard this as a fatal flaw or as evidence of misrepresentation. The paper is transparent about using a proxy and about its limitations; the architecture could still be a useful panoptic labeling extension to an SSC baseline. The right response is to require (a) confirmation from the code about the fusion behavior and (b) a full-grid or occluded-only panoptic evaluation, possibly with an OR-ablation of the object occupancy. These are concrete, runnable checks. Since the reader already assigned CONDITIONAL, my verdict remains CONDITIONAL in substance, so no change to the reader's verdict is needed.","tokens_in":15495,"tokens_out":6688,"duration_ms":74502,"concrete_test":"Inspect the released Panoptic Module code to confirm that the voxelized Object Module occupancy from Eq. (1) is not unioned into the baseline occupancy grid. Then compute, on the Occ3D-nuScenes validation split, panoptic recall of ground-truth object voxels in the final grid separately for visible and occluded voxels (using the Occ3D visibility mask), and compare with the frozen baseline's object-voxel recall on the same voxels. If the final-grid recall on occluded object voxels equals the baseline recall, the Object Module contributes zero completion. As a complementary check, ablate by OR-ing the Eq. (1) predicted object occupancy into the baseline before voting and report full-grid and occluded-only panoptic quality; this directly tests whether the differentiable shapes can improve completion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that “each voxel classified as an object by the baseline decoder is assigned an instance ID based on the Object Module prediction.” The Object Module's offset occupancy (Eq. 1) is therefore used only as voting evidence for labeling voxels that the baseline already predicts; there is no union or OR of the predicted object point cloud with the baseline occupancy grid. Consequently, any occluded object voxel that the baseline misses—precisely the case scene completion should fix—is absent from the final panoptic grid no matter how accurate the learned object shapes are. The central contribution, “object shapes learned differentiably,” is thus not actually a completion mechanism, only an instance-labeling mechanism.\n\nThe evaluation in Sec. 4.3 labels LIDAR points according to corresponding voxel labels. LIDAR points are observed surface points, so this metric is insensitive to whether occluded voxels are completed. The reader's concern about the LIDAR proxy is therefore compounded by an architectural reason why shape predictions cannot add occluded occupancy. The framework may still provide useful panoptic IDs on top of an SSC baseline, but the headline claim that differentiable object shapes enable 3D panoptic scene completion is unsupported by both the described fusion and the chosen metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OffsetOcc, a camera-only framework for 3D panoptic scene completion. It augments a standard semantic scene completion (SSC) baseline with an Object Module, a DETR-style decoder that predicts object centers and a set of offsets that describe each object's occupied voxels, and a Panoptic Module that merges these predictions with the baseline occupancy grid to produce a panoptic voxel output. The authors train the Object Module on Occ3D-nuScenes by generating panoptic labels from box-voxel intersection, and evaluate the full model with mIoU/IoU on the occupancy task plus a panoptic LIDAR segmentation metric as a proxy for panoptic performance. They report mIoU 28.0, IoU 43.9 on the visibility-masked setting, and panoptic LIDAR PQ 29.4, with ablations on loss decoupling and voting radius.","tokens_in":15758,"tokens_out":6311,"duration_ms":63008,"significance":"The idea of learning object shapes as differentiable offset sets is interesting and could be a useful component for panoptic scene understanding. The paper is clearly written and the code is released, which is commendable. However, the implementation as described does not actually use the learned object shapes to add occluded occupancy: the Panoptic Module only assigns instance IDs to voxels already classified as objects by the baseline, and the only panoptic evaluation is on LIDAR points, which are observed surfaces. Consequently, the central claim of 'panoptic scene completion through differentiable object shapes' is not supported by the architecture or the experiments. If the authors were to redesign the fusion so that predicted object occupancy is written into the grid and to evaluate with a voxel-level panoptic metric, the contribution could be more meaningful, but the present manuscript does not deliver the promised capability.","major_comments":[{"comment":"The Panoptic Module uses the Object Module's predicted point cloud only as voting evidence: 'each voxel classified as an object by the baseline decoder is assigned an instance ID based on the Object Module prediction.' There is no union or OR operation that inserts the predicted offset-derived voxels (Eq. 1) into the final panoptic grid. Therefore, any occluded object voxel that the baseline misses is absent from the output, regardless of how accurate the learned shapes are. This is a load-bearing flaw: the paper's central claim is that differentiable object shapes enable 3D panoptic scene completion, but in the described architecture they only enable instance labeling of baseline predictions. To support the claim, the fusion must actually add the Object Module's predicted occupancy to the grid, and the experiments must show that occluded voxels are completed better as a result.","section":"Sec. 3.3, Panoptic Module and Eq. (1)"},{"comment":"The panoptic evaluation labels LIDAR points according to their corresponding voxel labels. LIDAR points are located only on observed surfaces, so this metric cannot measure whether occluded voxels have been correctly completed. This is particularly serious because the paper defines scene completion as 'reconstruct[ing] regions that are currently occluded' (Sec. 1). Even if the architecture were modified to write Object Module voxels into the grid, the current metric would remain blind to that improvement. A voxel-level panoptic metric on the full occupancy grid, ideally reported both with and without the visibility mask, is required to test the claimed capability.","section":"Sec. 4.3, Panoptic LIDAR segmentation and Table 2"},{"comment":"The panoptic ground truth is generated 'by intersecting semantic voxel annotations with 3D bounding boxes.' For classes whose shapes are not well approximated by boxes, such as pedestrians, bicycles, and traffic cones, this intersection yields noisy or incorrect instance labels. Because the Object Module is trained to reconstruct exactly these voxel sets, the supervision may be too inaccurate to support the paper's claim that accurate object shapes are learned. The authors should quantify the error introduced by this label generation, for example by comparing against manually labeled instances on a subset, and discuss the impact on the object shape learning.","section":"Sec. 4.1, Datasets (panoptic ground truth)"}],"minor_comments":[{"comment":"The notation is inconsistent: the occupancy score is written as s_i^k in some places and s^k_i in others; please use a consistent subscript/superscript convention for the object index and offset index.","section":"Eq. (1)"},{"comment":"The x-axis label is 'd' while the text and Sec. 3.3 use 'r' for the voting radius; unify the notation to avoid confusion.","section":"Fig. 4"},{"comment":"The paper calls the metric 'panoptic LIDAR segmentation,' but the task under study is panoptic scene completion on a voxel grid; the relationship between the two should be clarified, and the proxy nature of the LIDAR metric should be stated more prominently.","section":"Sec. 4.3"}],"recommendation":"reject","confidential_remarks":"This manuscript currently does not support its central claim. The described fusion never writes the Object Module's predicted occupancy into the final panoptic grid, and the evaluation is a LIDAR-point proxy that cannot measure completion of occluded voxels. Even if the authors re-run experiments with a union fusion and a voxel-level panoptic metric, the paper would need substantial rewriting and new experimental evidence. I would recommend rejection, although a resubmission after such a major overhaul could be re-considered. The offset-based object representation itself has merit, but as presented it is not a scene completion mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the representation is genuinely new: modeling each object's occupied voxels as a set of DETR-style learned offsets is a clean idea, and the plug-in design is attractive for extending existing occupancy models. Second, the central claim—that these differentiable object shapes enable panoptic scene completion—does not hold as written. Section 3.3 says the Panoptic Module assigns an instance ID to each voxel that the baseline decoder already classifies as an object. There is no union or OR of the predicted point cloud with the baseline occupancy. So the Object Module's offsets only vote on which instance ID to assign to voxels the baseline already predicts; they never add occluded voxels. That means the 'completion' is done entirely by the frozen baseline SSC model, not by the differentiable shapes. The evaluation compounds this: the panoptic metric labels LIDAR points, which are observed surfaces, so it cannot detect whether occluded voxels were completed. The ablation does support the decoupling trick, and the overhead numbers are honest and useful. The paper is also commendably clear about its limitations, though it does not flag this particular gap between the architecture and the claim. The panoptic ground truth from box-voxel intersection is noisy for non-box classes, Panoptic-FlashOcc is not compared, and there is no variance reporting. The low PQ (29.4 vs PanoOcc's 62.1) is a symptom, not the main problem. Who gets value from this? Researchers working on object-centric occupancy representations or panoptic labeling of occupancy grids will want to read it, but they should not take the completion claim at face value. I would send it to peer review—the representation deserves discussion and the authors should be asked to either fix the fusion so predicted voxels actually enter the grid, or reposition the work as panoptic instance labeling on top of an SSC baseline. As is, I would not cite it as a completion method.","headline":"The offset-set representation is genuinely novel, but the paper's central claim of panoptic scene completion via differentiable object shapes is not implemented: predicted object voxels are never written into the final grid, so the completion part is untested.","tokens_in":16295,"tokens_out":1883,"would_cite":true,"duration_ms":20820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OffsetOcc models each object's shape as a differentiable set of center-relative offsets, enabling panoptic 3D scene completion from cameras alone.","keywords":["3D panoptic scene completion","occupancy prediction","differentiable object shape","offset representation","camera-only perception","autonomous driving","set prediction","voxel grid"],"falsifier":"Compute occupancy IoU restricted to camera-occluded thing voxels, the complement of the visibility mask, on a held-out set with reliable occluded-object labels: if a model trained without the visibility mask performs no better than a constant-empty baseline in these regions, the claimed completion of occluded object shapes is not happening. A second check would compare PQ against true panoptic voxel ground truth instead of the LIDAR point proxy.","tokens_in":15322,"feed_emoji":"🚗","tokens_out":6115,"duration_ms":56306,"temperature":0.7,"pith_summary":"This paper tries to establish that 3D panoptic scene completion from cameras alone can be solved by treating each object's shape as a differentiable set of 3D offsets from the object center, rather than decoding voxels bottom-up and clustering them into instances. The authors build OffsetOcc, a framework of two plug-in modules: an Object Module that predicts object class, center, and a set of occupied-voxel offsets for up to 900 queries in a set-prediction scheme, and a parameter-free Panoptic Module that fuses those instance shapes with a baseline occupancy grid by majority voting. The motivation is that scene completion needs to hallucinate occluded voxels, and distinguishing instances within a class matters for planning; current occupancy models trained with a visibility mask ignore occluded regions and can learn unrealistic object shapes. A sympathetic reader should care because the proposed representation is generic: it can sit on top of any camera-based occupancy decoder and adds only 2.3 million parameters, turning semantic occupancy into panoptic occupancy with modest latency overhead. The paper reports 28.0 mIoU on Occ3D-nuScenes under the standard visibility-masked evaluation and 29.4 PQ on a panoptic LIDAR proxy, while arguing that the masked metric hides poor completion of occluded regions.","feed_headline":"Objects as offset sets enable camera-only panoptic 3D scenes","feed_subtitle":"A small add-on to occupancy models separates instances and fills occluded voxels from six cameras.","key_machinery":"The load-bearing identity is the offset occupancy representation: an object is its center plus a set of vectors to occupied voxel centers, with per-offset occupancy scores, converted to a mask by voxelization. This replaces full per-object grid masks with a compact, differentiable description and gives a top-down, object-centric prior that bottom-up voxel decoders lack. The machinery also includes the two-level Hungarian assignment (objects, then voxels within each object) that lets the shape loss be computed against ground-truth voxel sets, and the location/offset decoupling that makes offset learning independent of center accuracy. The Panoptic Module's radius-r majority voting is the fusion mechanism that turns separate object masks plus a baseline grid into a single panoptic grid without learned parameters.","core_discovery":"The central claim is that object occupancy can be represented as a set prediction: for each detected object the model outputs a 3D center and a fixed-size set of K offset vectors, each offset aimed at the center of a voxel the object occupies and paired with an occupancy score in [0,1]; voxels whose score passes 0.5 are kept and voxelized into an object mask. Because this mask is generated from continuous vectors, the whole shape estimate is differentiable and can be trained end-to-end from the semantic voxel annotations already present in occupancy benchmarks, without needing panoptic labels at training time. The Object Module uses learnable object queries with 3D deformable cross-attention to the voxel feature grid, and training matches predictions to ground truth twice with the Hungarian algorithm, once at the object level and once at the voxel level. A supervised decoupling trick anchors the predicted offsets to the ground-truth object center during training, so the shape loss does not absorb localization error. The Panoptic Module then assigns baseline object voxels to the most-voted nearby instance ID, producing a complete panoptic voxel grid.","pith_inferences":["If object shapes are indeed learned as class-typical offset distributions, the same offset head could be conditioned on temporal features or depth estimates, letting a partially visible object trigger a complete shape prior without extra supervision.","The parameter-free voting fusion could be replaced by a learned refinement that estimates per-voxel instance confidence, which might close part of the gap to stronger panoptic baselines.","Because training is done without the visibility mask, the Object Module's losses may act as a shape regularizer that makes the baseline occupancy decoder more geometrically plausible even where no instance supervision exists; this is testable by ablating the frozen baseline.","A dataset with true panoptic voxel annotations would let the field decide whether the offset representation or bottom-up clustering is better for completing occluded instances."],"forward_implications":["Existing camera-based 3D scene completion models can be upgraded to panoptic output by adding the two modules, without retraining the base occupancy decoder.","Since offsets are learned per class from occupancy annotations, objects that are partially occluded can, in principle, receive completed voxel shapes rather than just a bounding box.","Panoptic occupancy metrics should be reported without the visibility mask, because masked evaluation discards exactly the occluded regions scene completion is meant to fill.","The overhead is small enough (2.3 million parameters, and 714 ms versus 500 ms per six-camera frame on an L40S) to fit real-time autonomous-driving pipelines."],"supporting_citations":[{"why":"Supplies the object-query set-prediction paradigm and the Hungarian matching used to assign predictions to ground-truth objects.","marker":"[3]"},{"why":"Provides the Occ3D-nuScenes semantic voxel occupancy benchmark, the visibility mask, and the voxel annotations from which panoptic labels are generated offline.","marker":"[34]"},{"why":"Defines the baseline semantic scene completion loss and the camera-only setting that OffsetOcc extends.","marker":"[2]"},{"why":"Provides the BEV-transformer encoder design that inspires the baseline uplifting encoder used with the proposed modules.","marker":"[20]"},{"why":"Supplies the deformable cross-attention and self-attention layers used inside the Object Module.","marker":"[51]"},{"why":"Prior camera-based panoptic occupancy method used as the main comparison baseline for panoptic performance.","marker":"[40]"},{"why":"Supplies the nuScenes images, LIDAR sweeps, and 3D bounding boxes used for annotation and evaluation.","marker":"[1]"}],"fun_headline_variants":["Camera-only panoptic 3D from differentiable object offsets","Set-predicted object shapes give camera-only panoptic 3D","No panoptic labels needed for camera-only 3D completion","Object offsets turn occupancy predictions into panoptic 3D","Learnable offset sets make 3D panoptic scenes camera-only"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that panoptic training targets can be manufactured by intersecting semantic voxel labels with 3D bounding boxes, which is reliable for box-shaped vehicles but noisy for pedestrians, cones, and bicycles, and that labeling LIDAR points with voxel labels adequately measures panoptic scene completion, which never tests whether occluded voxels were actually completed.","fun_headline_variants_meta":{"raw":{"variants":["Camera-only panoptic 3D from differentiable object offsets","Set-predicted object shapes give camera-only panoptic 3D","No panoptic labels needed for camera-only 3D completion","Object offsets turn occupancy predictions into panoptic 3D","Learnable offset sets make 3D panoptic scenes camera-only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3311,"prompt_tokens":964,"completion_tokens":2347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2260}},"tokens_in":580,"tokens_out":2347,"duration_ms":17115,"temperature":1.0,"reasoning_tokens":2260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:28:07.717218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute occupancy IoU restricted to camera-occluded thing voxels, the complement of the visibility mask, on a held-out set with reliable occluded-object labels: if a model trained without the visibility mask performs no better than a constant-empty baseline in these regions, the claimed completion of occluded object shapes is not happening. A second check would compare PQ against true panoptic voxel ground truth instead of the LIDAR point proxy.","supporting_citations":[{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the object-query set-prediction paradigm and the Hungarian matching used to assign predictions to ground-truth objects."},{"cited_title":"Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Provides the Occ3D-nuScenes semantic voxel occupancy benchmark, the visibility mask, and the voxel annotations from which panoptic labels are generated offline."},{"cited_title":"Monoscene: Monoc- ular 3d semantic scene completion","cited_arxiv_id":null,"evidence_quote":"Defines the baseline semantic scene completion loss and the camera-only setting that OffsetOcc extends."},{"cited_title":"Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers","cited_arxiv_id":null,"evidence_quote":"Provides the BEV-transformer encoder design that inspires the baseline uplifting encoder used with the proposed modules."},{"cited_title":"Temporal","cited_arxiv_id":null,"evidence_quote":"Supplies the deformable cross-attention and self-attention layers used inside the Object Module."},{"cited_title":"Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation","cited_arxiv_id":null,"evidence_quote":"Prior camera-based panoptic occupancy method used as the main comparison baseline for panoptic performance."}],"review_version":1}