{"id":"37603b24-b82b-4673-a102-8e93b07a6194","arxiv_id":"2510.16923","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"UnDREAM enables optimization of adversarial textures on arbitrary 3D objects inside Unreal Engine by bridging the simulator to the differentiable renderer Mitsuba.","lead":"UnDREAM connects Unreal Engine 5's photorealistic simulator with Mitsuba, a differentiable renderer, so adversarial patterns on 3D objects can be optimized while the object stays inside a realistic scene. It matters because adversarial attacks that ignore lighting, weather, and materials often fail when transferred to the physical world.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated proxy gradients undermine 'end-to-end': Algorithm 1 backprops via Mitsuba twin, not Unreal; only a white-texture check is reported, so texture updates may optimize a rendering that diverges from the simulation.","rationale":"The reader's weakest_assumption is exactly the concern I would raise, so I set agreement to agree. The paper has real assets: code repository, UV mapping, demonstrations of objects, and large attack success rates. Those count as evidence that the pipeline runs. But they do not establish the key scientific claim. The load-bearing condition is not merely that the Unreal and Mitsuba scenes look similar; it is that the gradient of the loss through Mitsuba is aligned with the gradient one would obtain through Unreal's rendering (which is unavailable). Algorithm 1's loss is computed on Unreal images, while the texture update uses Mitsuba gradients; if these two image-formation processes differ, the optimizer is solving the wrong problem. The only reported verification is a white-texture render, which checks that the transformed geometry appears similar but cannot check photometry under weather/lighting/material changes. Since the paper's experiments include exactly those environmental variations, the gap is central. A concrete finite-difference gradient-alignment test would settle it. My verdict remains CONDITIONAL: the framework is promising and reproducible, but the headline claim should be held conditional on gradient fidelity evidence.","tokens_in":9933,"tokens_out":3975,"duration_ms":36339,"concrete_test":"Implement a gradient-fidelity check for one scene (park, sunny). (1) Render identical texture and camera in Unreal and Mitsuba; compute per-pixel absolute difference and SSIM. Repeat under cloudy/rainy/night. (2) Pick a small texture patch; compute the Mitsuba analytic gradient of the victim model's classification loss, and compare with a finite-difference estimate of the same loss gradient using Unreal renders (render the scene with texture+ε and texture−ε, recompute loss). Report cosine similarity and whether following Mitsuba gradients decreases the Unreal loss over, say, 20 iterations. If image SSIM is low or gradient cosine similarity is below ~0.7, the twin is not a faithful proxy and the central claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that UnDREAM enables end-to-end optimization of adversarial textures inside photorealistic simulation. For that to hold, the gradient used in Line 8 of Algorithm 1 must be a valid gradient of the loss on the Unreal-rendered images (Lines 3–5). But Unreal rendering is non-differentiable, so the actual update is computed through the separately generated Mitsuba twin. The paper only validates this twin with a plain-white-texture visual check (§3.1.2 step 4), which can confirm geometry/coordinate conversion but not appearance under the lighting, weather, material, and post-processing effects that are central to the photorealistic claim. No image-level or gradient-level comparison between Unreal and Mitsuba is reported, and the transformation section (Appendix A) describes only poses/coordinates, not conversion of lights, weather, or materials. If Mitsuba's image and gradients diverge from Unreal's, the attack will optimize a rendering that differs from the one scored by the loss, and the claimed 'end-to-end' property is not established. This is a surrogate-gradient pipeline unless validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents UnDREAM, a software framework that couples Unreal Engine 5 (UE5) photorealistic simulation with Mitsuba differentiable rendering. The framework converts UE5 LevelSequence scenes into Mitsuba XML scenes through coordinate/scale/rotation transforms, then runs an adversarial attack loop: render frames in UE5, compute a victim model's loss, backpropagate the loss through the Mitsuba twin scene, and update the 3D object texture before re-inserting it into UE5. The paper claims this is the first framework to enable end-to-end optimization of adversarial perturbations on arbitrary 3D objects inside photorealistic simulation. Experiments demonstrate PGD and Auto-PGD attacks on DETR-ResNet-50 for classification and detection tasks in park/city environments under different lighting and weather conditions, with a small set of objects (sphere, bin, car, umbrella).","tokens_in":10216,"tokens_out":3634,"duration_ms":32978,"significance":"If the central claim holds, UnDREAM is a useful open-source systems contribution: it lowers the barrier to optimizing 3D adversarial textures in a high-fidelity simulator, and the single-line attack integration is a practical advantage. The paper ships code and assets, which is commendable. However, the load-bearing 'end-to-end' claim depends on an unvalidated proxy-gradient assumption: the loss is scored on UE5 renderings while gradients are backpropagated through a separately constructed Mitsuba scene. The manuscript provides only a white-texture visual check for the twin scene and no quantitative alignment evidence. Until that is supplied, the framework is better described as a surrogate-gradient pipeline than an end-to-end differentiable simulation pipeline.","major_comments":[{"comment":"The central claim of end-to-end optimization is undercut by an unvalidated proxy gradient. In Algorithm 1, lines 3–5 compute the loss on Unreal-rendered frames, but lines 7–9 backpropagate through the Mitsuba twin scene. For this to be a true end-to-end optimization, the Mitsuba rendering must be a faithful differentiable analogue of the Unreal rendering under the same lighting, weather, material, and post-processing conditions. The only validation described is a plain-white-texture visual check (§3.1.2 step 4), which can confirm geometry/coordinate conversion but not appearance or gradient fidelity. Appendix A documents only coordinate/rotation/scale conversion, not conversion of lights, weather, or materials. No image-level or gradient-level comparison between UE5 and Mitsuba is reported. Please add quantitative alignment experiments (e.g., pixel-wise error or gradient cosine similarit","section":"§3.2, Algorithm 1; §3.1.2 step 4"},{"comment":"The attack evaluation is presented as a single demonstration: one person walking in a park, one victim model (DETR-ResNet-50), and no indication of the number of frames, scenes, or random seeds. The 100% benign accuracy/mAP in Table 3 suggests a small or possibly single-frame evaluation set. For a framework whose contribution is to enable systematic evaluation of adversarial attacks, please report results across multiple scenes, sequences, and seeds, with mean and standard deviation. Also clarify the attack budget: what norm and bound do the values 0.78 and 1.00 refer to, and how is the budget applied to a detection mAP metric? Without this information, the quantitative claim in §4.3 is not reproducible.","section":"§4.3, Table 3"},{"comment":"The paper argues that existing pipelines optimize textures outside simulation and therefore fail under environmental interactions, but no experiment compares UnDREAM against such baselines. To support the claim that in-simulation optimization 'preserves lighting, perspective, and physical material interactions' and leads to better threat modeling, the manuscript should include a quantitative comparison with at least one prior approach (e.g., a 2D patch superimposed on UE5 frames, or a texture optimized in Mitsuba alone and then transferred to UE5). Such baselines are essential to show that the UnDREAM pipeline provides an actual improvement in attack effectiveness or transferability rather than merely a convenient implementation.","section":"§1, §4"}],"minor_comments":[{"comment":"The sentence 'UnDREAM eliminates the need for calculating the bounds of the adversarial object across This leads to better alignment' is incomplete. Please revise.","section":"Contribution 2 (§1)"},{"comment":"The notation and control flow are unclear. Line 3 uses x_{i-n} without defining the index range, and the 'RENDERING JOB FINISH CALLBACK' procedure appears nested inside the for loop. Clarify the asynchronous rendering semantics and the relationship between attack iterations and rendering jobs.","section":"Algorithm 1"},{"comment":"The 'only framework' claim in the caption and text is a strong novelty claim. Please define the comparison scope more precisely, and consider citing or discussing prior differentiable-simulation bridges if any exist, to avoid overclaiming.","section":"Table 1"},{"comment":"The statement that 'Mitsuba, due to its differentiable ray tracing method, has increased photorealism compared to alternatives such as PyTorch3D' is not substantiated. Rephrase as an architectural difference or provide a citation.","section":"§2.2"},{"comment":"Typo: 'hte free assets' should be 'the free assets'. The reproducibility statement is otherwise a strength.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a promising systems contribution, but the end-to-end claim needs to be backed by validation of the Mitsuba twin as a faithful differentiable surrogate for UE5. I would also like to see a baseline comparison and a more rigorous evaluation protocol before this is publishable as a research contribution. The 'first framework' claim in Table 1 should be checked carefully against the broader differentiable-simulation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nUnDREAM is a genuine new software artifact and probably useful to people who want to optimize adversarial textures inside a high-fidelity simulator. The authors build an automatic pipeline that takes an Unreal Engine 5 LevelSequence, converts object/camera poses and UV maps into Mitsuba XML scenes, runs an attack like PGD against a victim model, and feeds the texture gradients back into Unreal. The coordinate-system conversion in Appendix A is real work, and the promise of downloadable code and assets makes this a reproducible engineering contribution. The proof-of-concept numbers are striking (classification accuracy from 100% to 9.5%), though they are single-run and unbounded.\n\nWhere the paper is soft is exactly where the reader's stress test lands. The loss is computed from Unreal frames, but gradients are backpropagated through a separately constructed Mitsuba scene. For that to be end-to-end, Mitsuba's output has to be a good enough stand-in for Unreal's under the same lighting, weather, and materials. The only validation is a white-texture visual check, which confirms geometry/coordinate alignment but tells you nothing about whether specular highlights, shadows, rain, or sky matching carry over. Appendix A describes only pose and scale conversions, not material or light conversion. If Mitsuba diverges, the update is optimizing the wrong rendering and the 'end-to-end' claim is just surrogate-gradient optimization with extra engineering.\n\nI want to be clear: this is not a fatal flaw, because approximate gradients are often fine in practice. But the paper needs to supply evidence. Two concrete additions would do: (1) image-level and gradient-level agreement metrics between Unreal and Mitsuba renderings across the tested conditions; (2) a baseline comparison against standard 2D patch-superimposition attacks, plus error bars over multiple runs and scenes. Without those, Table 3 is proof of concept, not proof of the bridge.\n\nCitations look reasonable and the related work is honest about prior platforms. Table 1's 'only framework' claim is self-assessed, but it's a minor overstatement rather than a serious issue.\n\nBottom line: this deserves a serious referee, not a desk reject. It's a useful tool paper for the adversarial-ML community, and the central idea is sound if the validation gap is closed. I'd want to see a revision with agreement checks before trusting the results.","headline":"Useful new simulation-to-differentiable-rendering bridge, but the 'end-to-end' label outruns the evidence: gradients come from a Mitsuba twin whose agreement with Unreal is never quantitatively checked.","tokens_in":10671,"tokens_out":1964,"would_cite":true,"duration_ms":19074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UnDREAM claims a framework that optimizes adversarial textures end-to-end inside photorealistic 3D simulation by routing gradients through a differentiable twin scene.","keywords":["adversarial attacks","differentiable rendering","photorealistic simulation","3D object textures","end-to-end optimization","object detection robustness","autonomous driving"],"falsifier":"Take a fixed texture and a fixed frame, render it in the simulator and in the differentiable twin, and compare the two images pixel by pixel under several lighting and weather conditions; any substantial mismatch means the gradient path is optimizing a different rendering than the one the victim model sees. A tighter test would measure whether a texture gradient computed through the differentiable renderer points in the same direction as a finite-difference perturbation of the texture in the simulator.","tokens_in":9823,"feed_emoji":"🎯","tokens_out":6714,"duration_ms":52032,"temperature":0.7,"pith_summary":"The paper introduces UnDREAM, a framework that couples a photorealistic, non-differentiable simulator with a differentiable renderer so adversarial perturbations on arbitrary 3D objects can be optimized with real gradients instead of 2D patches superimposed afterward. The simulator produces the images the victim model sees; a twin scene in the differentiable renderer reproduces the object-camera geometry and lets image-loss gradients flow back to the texture. The updated texture is then reinserted into the simulator, closing an end-to-end optimization loop. If this works as described, it lets attacks be designed and evaluated under weather, lighting, camera motion, and object animation, which existing pipelines approximate or ignore.","feed_headline":"Adversarial textures now optimize inside photorealistic simulations","feed_subtitle":"A twin-scene gradient bridge lets 3D attacks adapt to weather, lighting, and motion, not flat 2D patches.","key_machinery":"The load-bearing component is the twin-scene bridge: an automatic transformation of each simulator frame into a differentiable scene with the adversarial texture attached as a differentiable texture. The transformation reconciles the coordinate systems, rotation conventions, and units between the two renderers, so that gradients computed in the differentiable scene can be attributed to texture pixels that appear in the simulator's image. This is what lets the optimizer update the texture as it actually appears under the simulator's lighting, materials, and motion, without requiring the simulator itself to be differentiable.","core_discovery":"The paper's central claim is that the non-differentiability of photorealistic simulation need not block gradient-based attacks. UnDREAM automatically converts a simulator scene—including camera and object poses, rotation conventions, and units—into a differentiable renderer scene for every frame. Each attack iteration renders images in the simulator, computes the victim model's loss, backpropagates through the differentiable twin to update the adversarial texture, and writes the updated texture back into the simulator. The authors demonstrate this on classification and object detection with PGD and Auto-PGD, reducing accuracy and mAP dramatically, and show that the optimized texture changes","pith_inferences":["If the twin-scene gradient flow is accurate enough, the same bridge could be used for non-adversarial inverse problems such as designing textures or materials to match a target appearance under simulated weather and lighting.","Since the differentiable renderer supports differentiation with respect to scene parameters beyond textures, the framework could extend to optimizing object pose, camera path, or lighting rather than only the adversarial texture.","A decisive but unperformed check is comparing a texture gradient computed through the twin renderer against a finite-difference perturbation in the simulator itself; the paper's white-texture verification alone does not rule out divergence under weather or material effects.","The framework could plausibly generate physically varying adversarial training data, where perturbations are optimized in simulation before being used to robustify detectors."],"forward_implications":["Adversarial attacks can be optimized directly inside scenes with weather, lighting, moving cameras, and animated objects, instead of superimposing patches onto pre-rendered frames.","Any 3D object with a UV mapping—sphere, bin, car, umbrella—can serve as an attack surface, removing the restriction to 2D rectangular patches.","Switching between attack algorithms reduces to changing a single line that updates the texture, making the pipeline easy to extend to PGD, Auto-PGD, or other gradient-based attacks.","Because the texture is embedded in the scene rather than pasted per frame, attacks are temporally consistent across consecutive frames, which matters for tracking and video-based evaluation.","The same loop can be turned around to evaluate defenses under realistic, high-fidelity conditions, giving a unified testbed for attacks and countermeasures."],"fun_headline_variants":["End-to-end adversarial attacks in photorealistic sims","Optimize 3D adversarial textures in any simulated scene","First bridge from photorealistic sims to differentiable attacks","Gradient attacks that adapt to weather, lighting, and motion"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The loss is computed from the photorealistic simulator's images, but gradients are backpropagated through a separate differentiable re-rendering of the same scene, so the two renderings must match closely enough that texture updates truly improve the simulator's output.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end adversarial attacks in photorealistic sims","Optimize 3D adversarial textures in any simulated scene","First bridge from photorealistic sims to differentiable attacks","Gradient attacks that adapt to weather, lighting, and motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1155,"prompt_tokens":676,"completion_tokens":479,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":420,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":420,"tokens_out":479,"duration_ms":4796,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T09:07:54.332042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed texture and a fixed frame, render it in the simulator and in the differentiable twin, and compare the two images pixel by pixel under several lighting and weather conditions; any substantial mismatch means the gradient path is optimizing a different rendering than the one the victim model sees. A tighter test would measure whether a texture gradient computed through the differentiable renderer points in the same direction as a finite-difference perturbation of the texture in the simulator.","supporting_citations":[],"review_version":1}