{"id":"d74e12aa-1bb9-4167-832d-0e7d514ac239","arxiv_id":"2506.07419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"V2XGen automatically creates perspective-consistent V2X test scenes, finds more occlusion and long-range perception errors than random selection or CooTest, and improves detection accuracy after retraining.","lead":"This paper presents V2XGen, a tool that automatically generates realistic driving scenes for connected-vehicle systems by editing 3D sensor data from multiple vehicles in a consistent way. The scenes are used to find failures in cooperative perception systems and to improve them by retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-agent perspective consistency is asserted via Equation 3 but never directly validated; the acknowledged V2V4Real pose error (Sec. 5.3) could misalign inserted objects across views and undermine both scene realism and the retraining claim.","rationale":"The reader identifies the same load-bearing assumption: accurate extrinsic transformations between agents are necessary for V2XGen's perspective-consistent rendering. I agree this is the most central risk because it targets the paper's main technical novelty over single-agent test generation. The paper's own Section 5.3 concedes the V2V4Real dataset has a positioning error, yet the evaluation provides no direct measurement of cross-agent alignment of inserted objects. FRD and the user study only assess per-view realism, so they cannot validate joint consistency. A direct geometric alignment check on generated scenes would settle whether Equation 3's premise is satisfied in practice. The retraining experiments also lack a control condition, and the fitness-guided comparison has a partially circular element because the fitness function uses the same failure indicators as the effectiveness metric; however, those issues mainly affect the strength of secondary claims, whereas inconsistent multi-view geometry would undermine the core method itself. Since the concern is concrete and addressable with an additional validation experiment, the existing CONDITIONAL verdict remains appropriate: the paper should be revised to provide direct evidence of cross-agent consistency or to quantify and correct for the dataset pose error.","tokens_in":20311,"tokens_out":5649,"duration_ms":82475,"concrete_test":"For a sample of generated scenes (e.g., 100), take the inserted entity's point cluster from the ego cloud and from each cooperative cloud, transform both into the world coordinate system using the dataset-provided inverse extrinsics, and compute the centroid distance (or Chamfer distance) between them. If the median distance exceeds one LiDAR beam spacing or a small fraction of object size (e.g., >0.5 m), the claimed perspective consistency is not achieved. Report this metric for generated objects alongside real seed-scene objects as a baseline; if generated objects align significantly worse than real objects, Equation 3-based rendering inherits the pose error and the central realism claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"V2XGen's central differentiator over single-agent test generators is perspective-consistent multi-view synthesis. The mechanism is Equation 3: a single world-space object location is mapped into each agent's local LiDAR frame via T^t_world->Vx, and virtual LiDAR rendering is done per view. This is only physically correct if the extrinsic transforms in the seed dataset are accurate. Section 5.3 states that V2V4Real has a developer-confirmed positioning error and that it may adversely affect the retraining process. If that pose noise is non-negligible, an inserted object will appear at different physical locations in the ego and cooperative point clouds, so the generated scene violates the cross-view consistency the method claims to enforce. The paper never measures this alignment error: FRD (Eq. 10) is computed on per-agent point clouds via RangeNet++ activations, and the user study asked participants to judge each agent's frame separately. Neither metric can detect cross-agent inconsistency. Thus the evidence does not establish the key premise that generated scenes are realistic V2X scenes, as opposed to individually plausible but misaligned point clouds. This also threatens RQ3, because models trained on misaligned collaborative data may improve on the same erroneous geometry without generalizing to correctly registered scenes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents V2XGen, an automated test-scene generation tool for V2X cooperative perception systems. V2XGen applies five transformation operators (insertion, deletion, scale, rotation, translation) to multi-agent LiDAR scenes and claims to maintain perspective consistency across agents through a world-to-vehicle coordinate transformation (Eq. 3). Generated scenes are selected through a fitness-guided strategy (Eqs. 4-6) that targets occluded and long-range objects. The authors evaluate the tool on six cooperative detection systems with different fusion schemes using the V2V4Real dataset. They report (i) realism via FRD and a 22-participant user study; (ii) error-detection effectiveness against CooTest and a non-guided variant; and (iii) improvements in average precision and reduced perception errors after retraining.","tokens_in":20585,"tokens_out":6421,"duration_ms":76984,"significance":"If the results hold, V2XGen would be a useful contribution to testing and robustness improvement for cooperative perception systems. The paper addresses a real gap: single-agent test generators do not automatically produce multi-view consistent scenes, and the existing CooTest baseline does not support object-level manipulation. The tool and generated scenes are released, and the evaluation covers early, late, and intermediate fusion systems. The main limitations are that the central realism claim is not directly validated for cross-agent consistency, and the efficiency claims are partly measured with the same error definitions used in the selection objective. These issues are addressable with additional validation and statistical rigor, but they are load-bearing for the paper's central claims.","major_comments":[{"comment":"The paper's central differentiator is perspective consistency across agents, but this property is never directly measured. Equation (3) maps a single world-space point to each agent's local LiDAR frame, so if the seed dataset's extrinsic transforms are not accurate, an inserted object will appear at different physical locations in different agents' point clouds and the generated scene will not be a valid V2X scene. The realism metrics used in RQ1 do not detect this failure: FRD in Eq. (10) is computed per agent on RangeNet++ activations, and the user study asks participants to judge ego and cooperative data frames separately. Moreover, Section 5.3 acknowledges a developer-confirmed positioning error in the V2V4Real dataset that may adversely affect retraining. The authors should quantify cross-agent alignment directly, for example by rendering a known object into multiple views, transforming the resulting point clouds back to world coordinates, and measuring the dispersion of the object center, and they should report how pose error or misalignment tolerance affects scene realism and retraining results.","section":"Section 3.2.1 and Section 5.1"},{"comment":"The fitness-guided selection objective and the RQ2 evaluation metrics are aligned by construction. The scores FOP and FLP in Eqs. (4) and (5) include the indicator IGT(Bgt, Bp), which is true when the system under test fails to detect a ground-truth object, and Algorithm 1 retains only high-fitness scenes. Tables 2 and 3 then count occlusion and long-range perception errors on the retained set, so higher error counts compared with random retention are a partial reflection of the optimization objective rather than an independent demonstration of testing effectiveness. To support the claim of improved testing efficiency, the authors should use an independent evaluation, for example measuring detection of previously unknown failures on a held-out set of transformed scenes, or evaluating whether scenes selected using one system expose errors in a different system without re-optimizing the fitness function.","section":"Section 3.3 and Section 5.2 (Eqs. 4-6; Tables 2-3)"},{"comment":"All quantitative results are reported as averages over three or five runs without standard deviations, confidence intervals, or significance tests. Since the generation process involves random operator selection, random seeds, and randomly selected subsets, the reader cannot determine whether the reported differences are reliable. The only statistical test reported is the Wilcoxon rank-sum test for the user study in Section 5.1. The authors should report per-run results or variance for Tables 1-4 and apply appropriate significance tests or effect-size measures; without this, claims such as the 3.3%-15.5% retraining improvements in Table 4 cannot be distinguished from random variation.","section":"Section 5 (Tables 1-4)"}],"minor_comments":[{"comment":"The tool name is rendered inconsistently as V2XGen and V2XGEN; please unify the notation.","section":"Throughout"},{"comment":"The SAGen baseline is described as single-agent operators combined with Eq. (3) to enforce cross-view consistency; if this is the case, the comparison isolates operator-level fidelity while still assuming the same coordinate transforms, so the baseline does not test the full multi-agent consistency claim. Please clarify what is being ablated.","section":"Section 4.4"},{"comment":"The metamorphic relations in Eqs. (7) and (8) are written with notation such as zeta{CPJsK union GTins, CPJsigma(s, eins)K}, but the intended equality criterion is only described in words as soft equality derived from AP; a precise formal definition would improve reproducibility.","section":"Section 3.4"},{"comment":"The FRD values in Table 1 are reported without units or an explanation of the scale, making it difficult to interpret whether a reduction from 194.3 to 184.2 is practically meaningful; please state the range of the metric and report the real-vs-real baseline if available.","section":"Section 5.1"},{"comment":"Table 2 reports absolute OE and LE counts without giving the total number of objects or scenes in the retained test sets; the error rates in Table 3 are more interpretable, but the denominator should be stated explicitly for both tables.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a competent software-testing paper with a relevant tool and a broad evaluation. My main reservation is that the two central claims—realistic multi-agent scenes and efficient error detection—are supported by metrics that are either per-agent or directly aligned with the selection objective. I would be willing to accept after a revision that adds a direct cross-agent consistency check, an independent evaluation for the fitness-guided strategy, and statistical reporting of variance. The retraining result is promising but also needs a clearer description of the data split used for final evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before anything else: V2XGen is a genuine engineering step forward for V2X test-scene generation, and the authors have shipped the tool and generated data. That is real evidence and should count. But the load-bearing claim—that inserted objects are perspective-consistent across agents—rests on an equation and a hope, not on a measurement. If that claim falls, the tool becomes a collection of single-agent operators bolted together.\n\nWhat is actually new: five transformation operators (insert, delete, scale, rotate, translate) designed around multi-agent coordinate transforms, plus a fitness-guided search that retains scenes likely to trigger occlusion and long-range perception errors. The operator design is thoughtful, especially the occlusion-aware insertion and the virtual LiDAR completion after deletion. The evaluation covers six fusion schemes and three baselines, which is solid coverage. The released artifacts make the work reproducible.\n\nNow the soft spots, in order of severity. First, cross-agent consistency is never validated. Equation 3 maps world coordinates into each agent's local LiDAR frame, but the paper itself notes in Section 5.3 that V2V4Real has a developer-confirmed positioning error that may affect retraining. If the poses are noisy, an inserted object lands in different physical places for different agents, and the generated scene is not a coherent V2X scene. The realism metrics used—FRD on per-agent point clouds and a 22-participant user study where each agent's frame is rated separately—cannot detect this misalignment. I do not think this kills the paper, but it needs to be measured, e.g., by comparing the rendered object's world-space position across agents or by checking IoU between agents' rendered bounding boxes.\n\nSecond, the fitness-guided comparison is partly circular. The fitness metric in Equations 4–6 is built from occlusion and long-range error scores, and Algorithm 1 retains high-fitness scenes; Tables 2 and 3 then count those same error types on the retained set. That the retention strategy finds more of the errors it was designed to maximize is less surprising than the paper implies. This is addressable by reporting detection of errors not in the fitness definition, or by using held-out error types.\n\nThird, the quantitative evidence is thinner than the prose. Averages over three to five runs without standard deviations or confidence intervals make it hard to know whether the improvements are meaningful. And the retraining experiment (RQ3) lacks a control condition: compare retraining on V2XGen data against retraining on the original dataset, random augmentation, or CooTest scenes. Without that, the robustness improvement claim is not isolated.\n\nThe authors are clearly serious—they identify the pose error themselves, which is honest—and the artifact release is commendable. The paper deserves peer review, but it needs a major revision that directly validates cross-agent consistency, de-circularizes the fitness evaluation, and tightens the retraining experiment. I would bring it to a reading group if anyone in the group works on ADS testing; I would cite it if I were building on V2X test generation, but only after the revision addresses the alignment question.","headline":"A plausible, useful V2X test-generation tool whose central claim—cross-agent perspective consistency—is asserted but never directly measured; needs one focused revision.","tokens_in":21132,"tokens_out":2018,"would_cite":true,"duration_ms":27681,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"V2XGen claims that automatically transforming real multi-vehicle LiDAR scenes — inserting, deleting, scaling, rotating, and translating objects while keeping every agent's viewpoint consistent — yields realistic V2X test scenes that…","keywords":["V2X cooperative perception","automated test generation","metamorphic testing","LiDAR point clouds","occlusion perception error","long-range perception error","fitness-guided testing","autonomous driving"],"falsifier":"Have annotators label a random sample of V2XGen-generated scenes: if the human-labeled bounding boxes of inserted objects fall below the IoU 0.5 threshold against V2XGen's metadata-derived labels in any cooperating vehicle's view, the perspective-consistency assumption breaks. A sharper test is to perturb the extrinsic pose matrices by the noise levels documented in V2V4Real and measure how much cross-view object alignment and FRD realism degrade; if realistic noise already misaligns the rendered object across agents, the residual perception errors after retraining are explained by the assumption failing rather than by system weaknesses.","tokens_in":20118,"feed_emoji":"🚗","tokens_out":8406,"duration_ms":79426,"temperature":0.7,"pith_summary":"V2XGen is an automated tool for generating test scenes for Vehicle-to-Everything cooperative perception systems, in which vehicles share LiDAR point clouds to detect objects that a single car would miss due to occlusion or distance. The paper's claim is that by transforming real multi-vehicle scenes with five operators (insertion, deletion, scaling, rotation, and translation) while preserving viewpoint consistency through coordinate transforms and virtual LiDAR rendering, V2XGen produces realistic test data that exposes two V2X-specific failure classes: occlusion perception errors and long-range perception errors. A fitness-guided strategy scores candidate scenes by their predicted ability to trigger those failures and keeps the highest-scoring ones, which the authors show finds more errors than random selection or the existing CooTest approach. The paper further claims that retraining cooperative perception systems on the generated scenes improves average detection precision while reducing occlusion and long-range errors. If correct, this replaces costly manual collection and labeling with an automated loop for testing and improving cooperative driving perception.","feed_headline":"Automated scenes expose V2X perception errors and lift retraining","feed_subtitle":"The fitness-guided scenes uncover occlusion and long-range failures; retraining on them raises detection precision.","key_machinery":"Three mechanisms carry the argument. The multi-agent perspective transformation (Equation 3) maps world coordinates into each participant's local coordinate system so that any inserted or moved entity occupies the same physical location in every agent's point cloud. A set of virtual LiDAR sensors, initialized to the real sensor's configuration (Velodyne VLP-32), uses ray-casting to render inserted entities, delete occluded points, and complete ground and objects behind deleted entities, so scenes follow physical occlusion laws. The fitness metric (Equation 6) combines an occlusion-error score and a long-range-error score that weight missed objects by their occlusion rate and distance, guiding scene selection toward cases where the ego vehicle needs help most; metamorphic relations (input-output properties that must still hold after transformation) built on insertion and deletion supply the test oracle, so labels come from the transformation itself rather than manual annotation.","core_discovery":"The central claim is that V2XGen generates realistic V2X test scenes, detects erroneous behaviors in cooperative perception systems operating under different fusion schemes, and that retraining the systems under test with the generated scenes enhances average detection precision while reducing occlusion and long-range perception errors. The realism is carried by perspective consistency: each transformation operator renders an entity through per-agent virtual LiDARs and intersection with background geometry, so a manipulated object appears from every cooperative vehicle's viewpoint as it would in the world. The targeted errors are formalized as occlusion perception error (the system misses an object that is occluded from the ego vehicle but visible to a cooperative vehicle) and long-range perception error (the system misses an object beyond the ego vehicle's range even though a cooperative vehicle could assist). The paper supports the claim with FRD realism scores and a user study favoring V2XGen over single-agent baselines, higher occlusion and long-range error rates than CooTest and unguided generation on six systems, and before/after retraining AP improvements across all six systems.","pith_inferences":["The perspective-consistency machinery could transfer to vehicle-infrastructure (V2I) cooperative perception, where one sensor has a fixed elevated pose; the insertion operator's validity checks would need to handle different road-surface and occlusion geometry.","The fitness metric is open-ended: adding new score terms (e.g., for misclassification of rescaled vehicle shapes or for weather-degraded views) could steer the same rendering pipeline toward additional failure classes without retraining the generator.","The measured realism and retraining gains are tied to V2V4Real's Velodyne VLP-32 configuration; re-initializing the virtual LiDAR to other sensor models (solid-state, different beam counts) would test whether the rendering approach transfers.","Because the fitness scores favor scenes where the ego vehicle is occluded or short-ranged, a retrained model's behavior on ordinary balanced traffic may shift; measuring per-scenario AP distributions, not just averages, would reveal such trade-offs."],"forward_implications":["V2X test data can be generated and labeled automatically from real seed scenes, eliminating manual data collection and annotation for the covered scenarios.","Fitness-guided selection concentrates the test budget on occlusion and long-range cases, detecting more perception failures per generated test than random or CooTest-guided selection.","Retraining cooperative perception systems on generated scenes improves detection precision on both transformed and original test sets, with reported average occlusion-error reductions of 54.9% and long-range-error reductions of 30.2%.","Because the operators work on any multi-view LiDAR scene with pose annotations, the approach extends across early, late, and intermediate fusion architectures."],"supporting_citations":[{"why":"Supplies the V2V4Real real-world multi-vehicle scenes used as transformation seeds and as test and validation data.","marker":"[13]"},{"why":"CooTest, the only prior V2X cooperative-perception testing tool, serves as the baseline for error-detection effectiveness.","marker":"[15]"},{"why":"TauLim provides the single-agent point-cloud insertion method combined into the SAGen baseline for realism comparison.","marker":"[16]"},{"why":"LiRTest supplies the metamorphic-relation oracle, the soft AP-based equality criterion, and the baseline scale, rotation, and translation transformations.","marker":"[17]"},{"why":"SalsaNext segments road points so inserted entities are only placed on valid road surfaces in every view.","marker":"[27]"},{"why":"ShapeNet's car models form the 3483-entity asset pool used for insertion operations.","marker":"[33]"},{"why":"V2X-ViT is one of the four intermediate-fusion systems under test, grounding the claim that the method works across fusion schemes.","marker":"[34]"},{"why":"The Frechet Range Distance metric quantifies realism of synthetic LiDAR data and supplies the RQ1 comparison against baselines.","marker":"[37]"}],"fun_headline_variants":["V2XGen's realistic scenes expose V2X perception blind spots","Automated realistic V2X scenes uncover errors and boost retraining","Fitness-guided V2X scenes reveal occlusion failures and lift detection","Realistic V2X test scenes improve cooperative perception accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the coordinate transformations between vehicles are accurate enough for a rendered object to land in the same physical position in every vehicle's point cloud; the paper itself reports in Section 5.3 that V2V4Real has a developer-confirmed positioning error that may adversely affect retraining.","fun_headline_variants_meta":{"raw":{"variants":["V2XGen's realistic scenes expose V2X perception blind spots","Automated realistic V2X scenes uncover errors and boost retraining","Fitness-guided V2X scenes reveal occlusion failures and lift detection","Realistic V2X test scenes improve cooperative perception accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2679,"prompt_tokens":1028,"completion_tokens":1651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1577}},"tokens_in":644,"tokens_out":1651,"duration_ms":14008,"temperature":1.0,"reasoning_tokens":1577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:34:56.287644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have annotators label a random sample of V2XGen-generated scenes: if the human-labeled bounding boxes of inserted objects fall below the IoU 0.5 threshold against V2XGen's metadata-derived labels in any cooperating vehicle's view, the perspective-consistency assumption breaks. A sharper test is to perturb the extrinsic pose matrices by the noise levels documented in V2V4Real and measure how much cross-view object alignment and FRD realism degrade; if realistic noise already misaligns the rendered object across agents, the residual perception errors after retraining are explained by the assumption failing rather than by system weaknesses.","supporting_citations":[{"cited_title":"Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds,","cited_arxiv_id":null,"evidence_quote":"SalsaNext segments road points so inserted entities are only placed on valid road surfaces in every view."}],"review_version":1}