{"id":"1b9bd2f4-a043-4a17-ad24-bb8ca72b911f","arxiv_id":"2501.18590","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single video diffusion system both estimates scene properties from video and renders photorealistic images from those properties, enabling relighting, material editing, and object insertion.","lead":"DiffusionRenderer trains two video AI models: one that reads a video and estimates scene properties like shape, color, roughness, and lighting, and one that re-renders new images from those properties. It offers a practical route to relighting and editing real-world videos without building detailed 3D geometry or running slow physics simulations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world claims rest on an unvalidated pseudo-label loop: the inverse renderer labels DL3DV10k, the forward renderer is trained on those labels, and no real-world ground truth tests either stage; if the synthetic-trained inverse renderer is biased on real videos, the forward renderer inherits…","rationale":"The reader's weakest assumption is exactly the point that matters most: the synthetic-to-real generalization of the inverse renderer is the foundation for both the real-world forward-renderer training data and the real-world application claims. My stress-test finds no additional flaw that would overturn the paper's synthetic results; those are internally consistent and the ablations are informative. The concern is not that the idea is wrong, but that the central real-world claim is supported only by a self-referential pipeline: inverse renderer labels real data, forward renderer trains on those labels, and the paper's real-world evaluation is qualitative and produced by the same models. The proposed concrete test would break the circularity by measuring pseudo-label fidelity against physical ground truth from controlled real captures. Because the paper provides neither such validation nor code/data to reproduce the loop, the appropriate verdict remains conditional: accept only if the pseudo-labels are shown to be quantitatively reliable on real scenes. This does not change the reader's conditional verdict, but it sharpens the condition that must be met.","tokens_in":19999,"tokens_out":3076,"duration_ms":34289,"concrete_test":"Acquire a small controlled real-world test set: roughly 20–40 scenes with a video or multi-view images, a calibrated HDR environment map measured by a chrome-ball capture, and actual photographs under at least one additional known HDR environment map. Run the inverse renderer to predict G-buffers, then path-trace those G-buffers with the paper's own OptiX renderer under the second HDR map. If the path-traced result using predicted G-buffers does not match the real photograph under the second light better than a trivial baseline (e.g., a constant albedo or the input image), the inverse renderer's real-world pseudo-labels are not trustworthy. Alternatively, compare predicted albedo/roughness/metallic/normal against measured values from the controlled capture; high error would invalidate the DL3DV10k auto-labeling premise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the forward renderer can act as a neural approximation of path-traced shading on real-world scenes, and that the inverse renderer accurately estimates G-buffers from real video. The load-bearing link is Sec. 4.3's use of the inverse renderer to auto-label 150,000 real-world clips from DL3DV10k, followed by Sec. 4.4's joint training of the forward renderer on those pseudo-labels. There is no independent real-world measurement of albedo, roughness, metallic, normals, or lighting to validate either the inverse renderer's outputs or the forward renderer's conditioning faithfulness. The evidence offered is qualitative (Fig. 5, Fig. 7) and comes from the same model family that produced the labels. If the inverse renderer has systematic biases on real scenes—e.g., due to the synthetic-to-real gap in sensor noise, tonemapping, complex outdoor illumination, or albedo-light ambiguity—those biases are baked into the forward renderer's training signal. The LoRA term in Eq. 4 can absorb domain shift in appearance, but it cannot correct a systematically wrong supervision signal. The absence of error bars on the quantitative tables and the lack of real-world quantitative evaluation make this concern unresolvable from the paper as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiffusionRenderer, a unified framework with two video diffusion models: a neural inverse renderer that estimates per-pixel G-buffers (normals, depth, albedo, roughness, metallic) from RGB video, and a neural forward renderer that synthesizes photorealistic video from G-buffers and an HDR environment map, replacing explicit light transport. The inverse model is trained on synthetic data plus public intrinsics datasets, then applied to auto-label 150,000 real-world clips from DL3DV10k; the forward model is jointly trained on synthetic data and these pseudo-labels with a LoRA adapter for real data. Experiments report strong quantitative results on synthetic benchmarks for forward rendering, inverse rendering, and relighting, plus qualitative real-world relighting, material editing, and object insertion.","tokens_in":20335,"tokens_out":4837,"duration_ms":45888,"significance":"If the real-world claims hold, this is a substantial contribution: it extends image-based neural deferred shading to a video-diffusion setting, proposes a lighting conditioning method via multi-resolution environment-map features in cross-attention, and demonstrates a single framework for relighting, material editing, and object insertion from a single video. Strengths include a large curated synthetic dataset (150,000 videos with ground-truth G-buffers), systematic ablations of video vs. image models and of lighting encoders, and supplementary evaluation with ColorVideoVDP temporal consistency and a human/GPT-4V preference study. The synthetic quantitative evidence is consistent and often large-margin across all three tasks.","major_comments":[{"comment":"The forward renderer's real-world supervision is produced by the paper's own inverse renderer and the off-the-shelf DiffusionLight estimator, and no independent real-world ground truth is used to validate either stage. The claim in Sec. 4.3 that 'our inverse rendering model generalizes to real-world videos' is supported only by qualitative examples (Fig. 5), and the same model is then used as the labeler for the forward renderer. Because systematic errors in the pseudo-labels are inherited by the forward renderer, and the LoRA term f_{θ+Δθ} can only adapt appearance rather than correct a systematically wrong supervision signal, the real-world relighting and editing claims in Secs. 5.4 and 5.5 are not squarely supported. Please add a quantitative real-world evaluation on a dataset with ground-truth target lighting (e.g., MIT Multi-Illumination or a multi-view relighting benchmark) and/or a real-world intrinsic-image benchmark with human annotations (e.g., IIW or SAW), and report agreement between the inverse renderer's pseudo-labels and an independent estimator or a set of manual annotations.","section":"4.3, 4.4, Eq. (4)"},{"comment":"No variance estimates are reported. PSNR, SSIM, and LPIPS are single numbers, and several comparisons are relatively close (e.g., Table 1 SyntheticObjects: Ours 28.3 vs. SSRT 29.4 and SplitSum 28.7; Table 2 SyntheticObjects: Ours 27.50 vs. Neural Gaffer 26.39). Without error bars, it is impossible to assess whether the claimed improvements are statistically significant. Please report mean and standard deviation over at least three runs, or bootstrap confidence intervals over test scenes, for all quantitative tables.","section":"Tables 1-4"},{"comment":"All quantitative evaluation for forward rendering, inverse rendering, and relighting is performed on synthetic test sets rendered by the authors' own OptiX/AgX pipeline, which may be similar in distribution to the training synthetic data. The real-world evaluation is qualitative only. The abstract and conclusion claim the method 'consistently outperforming the state-of-the-art' without scope qualification. Please either add quantitative real-world metrics or explicitly restrict the headline quantitative claims to synthetic benchmarks, with a clear statement that real-world performance is currently supported only qualitatively.","section":"5.1, 5.2, 5.4"}],"minor_comments":[{"comment":"The phrase 'approximates inverse and forwards rendering' contains a typo; it should be 'forward rendering'.","section":"Abstract"},{"comment":"The paper defines normals in camera space but does not explain how the inverse renderer's output normal maps are transformed or kept in the same coordinate frame as the forward renderer's condition; please clarify the normal-space convention and its normalization.","section":"4.1"},{"comment":"The caption references 'SSRT side view' and 'SSRT relit' but does not clearly describe what each subfigure displays; please expand the caption so the failure cases are understandable without the main text.","section":"Fig. 2"},{"comment":"Several GPT-4V preference scores are close to chance (e.g., 52-57% in the Objects rows); please discuss whether these differences are meaningful or present confidence intervals for the preference percentages.","section":"Supplement Table S2"},{"comment":"The environment map auto-encoder objective reconstructs LDR and log-space maps but not the directional encoding; a brief sentence explaining why Edir is only an input to the direction query encoder and not a reconstruction target would improve clarity.","section":"Eq. (7)"},{"comment":"Eq. (3) uses g0 without explicitly stating that it is the VAE-encoded latent of the attribute map; please add a sentence defining the notation for g0 and z0 in both Eq. (3) and Eq. (4).","section":"4.2"}],"recommendation":"major_revision","confidential_remarks":"The synthetic results are strong and the framework is well motivated, but the real-world claims rest on a self-labeling loop that lacks independent validation. I would be willing to accept after the authors provide a credible quantitative real-world evaluation or explicitly scope the claims to synthetic data. The fit with the journal's scope is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth a serious look. It unifies inverse and forward rendering in a single video-diffusion framework, and the synthetic benchmarks show real gains over strong baselines, especially for relighting on complex scenes. The environment-map cross-attention encoder is a clean design choice, and the ablation shows it matters.\n\nThe good parts first. The forward renderer trained on G-buffers plus an HDR env map gets 26.0 PSNR on SyntheticScenes versus 24.8 for SSRT and 18.5 for RGB↔X. Relighting jumps to 24.63 PSNR versus 20.75 for Neural Gaffer. That is a significant margin, and it holds up across PSNR, SSIM, LPIPS, plus the CVVDP scores in the supplement. The inverse renderer is also strong on synthetic data, with a big drop in metallic/roughness error relative to RGB↔X.\n\nThe soft spots are real but need to be sized correctly. First, no variance in any table, so we don't know if those gains are stable. That's minor, and fixable. Second, the real-world evaluation is qualitative only. That is a bigger deal because the abstract says 'consistently outperforming' without qualification. Third, and this is the one the stress-test flags: the forward model is trained on pseudo-labels produced by the paper's own inverse renderer on DL3DV10k (Sec 4.3). The LoRA can absorb appearance shift, but it can't fix systematically wrong G-buffers. The stress test is right that there is no independent real-world ground truth to break the loop.\n\nThat said, I would not call it a fatal flaw. The synthetic experiments give the forward renderer an independent check with clean G-buffers. The circularity mainly weakens the real-world generalization claims, and those are presented qualitatively anyway. What the paper needs is a real-world quantitative benchmark, or at least a careful caveat in the abstract, and some error bars. If those are added, the central system claim holds up fine.\n\nWho is this for? Anyone working on relighting, video intrinsics, or neural rendering should know it. It deserves a serious referee. I'd send it to review, probably with a request for the real-world caveat and variance reporting. I'd also bring it to the reading group, because the pseudo-label loop is a good discussion point.","headline":"A genuinely new diffusion-based renderer with strong synthetic results, but real-world claims rest on qualitative evidence and a self-labeling loop that deserves a caveat.","tokens_in":20843,"tokens_out":2944,"would_cite":true,"duration_ms":28145,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video diffusion models do inverse and forward rendering from one video","keywords":["video diffusion models","inverse rendering","forward rendering","relighting","G-buffers","neural rendering","material editing","object insertion"],"falsifier":"Take a set of real indoor and outdoor scenes, capture each under several known HDR lighting conditions, have the inverse renderer predict G-buffers from a single video, then run the forward renderer with the captured target light and measure PSNR, SSIM, and LPIPS against the true relit frames. If the synthetic-trained inverse estimates deviate strongly from measured normals and albedo, or the relit output is no closer to ground truth than per-frame image baselines, the central generalization claim would fail.","tokens_in":19822,"feed_emoji":"🎬","tokens_out":4791,"duration_ms":42334,"temperature":0.7,"pith_summary":"The paper proposes a pair of video diffusion models that together replace the two halves of physically based rendering: an inverse renderer that estimates geometry and material buffers (G-buffers) from ordinary video, and a forward renderer that turns those buffers plus a target environment map into a photorealistic relit video. The claim is that a video diffusion prior, conditioned on per-pixel scene attributes and lighting, can stand in for explicit light transport simulation, including shadows and reflections, without requiring 3D geometry or path tracing. The authors argue this matters because real-world scenes rarely come with the exact geometry, materials, and lighting that classic PBR needs, so a data-driven approximation trained on synthetic data plus auto-labeled real videos can make relighting, material editing, and object insertion practical from a single video input.","feed_headline":"Video diffusion models do inverse and forward rendering from one video","feed_subtitle":"A neural renderer learns shading from G-buffers and lighting, enabling relighting and object insertion from real video.","key_machinery":"The load-bearing object is the video diffusion model. The forward renderer concatenates five VAE-encoded G-buffer latents (normals, depth, base color, roughness, metallic) as pixel-aligned conditioning to the denoising UNet, while an environment-map encoder converts HDR lighting into multi-resolution cross-attention features so lighting is queried at each UNet level rather than concatenated as flat pixels. The inverse renderer reuses the same architecture, concatenating the input video latent and using five optimizable domain embeddings to generate one buffer at a time. A synthetic data pipeline of 150,000 path-traced videos with ground-truth G-buffers and environment maps supplies supervision, and a LoRA is added during training on auto-labeled real data.","core_discovery":"On the paper's own terms, DiffusionRenderer establishes that a video diffusion model can act as a neural approximation of path-traced shading: conditioned on camera-space normals, normalized depth, base color, roughness, metallic maps, and HDR environment-map lighting, the forward model generates photorealistic images and temporally consistent videos without explicit light transport. The companion inverse model, the same video diffusion architecture with a per-attribute domain embedding, estimates those G-buffers from RGB video, and although it is trained only on synthetic data it is claimed to generalize to real-world video. That generalization is exploited: auto-labeled real videos are used as training data for the forward renderer, with a LoRA adapter absorbing the domain difference between synthetic and real data. The result is a three-step editing pipeline—estimate buffers, edit buffers or lighting, re-render—that supports relighting, material editing, and object insertion from a single video.","pith_inferences":["Editorial: If the synthetic-trained inverse renderer generalizes in less curated settings, the same auto-labeling loop could extend to arbitrary online video, turning unlabeled footage into a scalable source of render-training data.","Editorial: Because the forward renderer is conditioned on noisy G-buffers, a testable extension is to corrupt G-buffers at training time in a controlled way to quantify robustness and build error-aware editing tools.","Editorial: The paper's quantitative evidence is strongest on synthetic benchmarks; a natural next experiment is measuring relighting fidelity on real scenes with a captured ground-truth target light, which would directly stress the weakest assumption."],"forward_implications":["If the central claim holds, relighting a real scene requires only a single input video: G-buffer estimation plus a target environment map, with no 3D reconstruction needed.","The forward renderer's tolerance for imperfect G-buffers means editing can happen directly in buffer space, making material edits and object insertions that would break a classic PBR pipeline feasible.","Because both models are video models, output is temporally consistent across frames, addressing a common artifact of per-frame image relighting.","Training with auto-labeled real video expands the effective training distribution beyond synthetic data, improving real-world rendering without expensive ground-truth capture.","A one-step deterministic variant of the inverse renderer is competitive for photometric accuracy, suggesting the approach can be made much cheaper at inference."],"supporting_citations":[{"why":"Stable Video Diffusion; the pretrained image-to-video diffusion model both renderers are fine-tuned from.","marker":"[8]"},{"why":"EDM; supplies the noise schedule and denoising score-matching objective used for training.","marker":"[33]"},{"why":"RGB↔X; the image-diffusion G-buffer synthesis and rendering baseline this work extends to video.","marker":"[83]"},{"why":"Neural Gaffer; provides the HDR environment-map lighting representation and a conditioning baseline compared against.","marker":"[30]"},{"why":"DL3DV10k; the real-world video dataset auto-labeled and used to train the forward renderer.","marker":"[47]"},{"why":"DiffusionLight; off-the-shelf environment map estimation used to label real-world lighting.","marker":"[61]"},{"why":"LoRA; the low-rank adaptation used to absorb the synthetic/real domain gap during real-data training.","marker":"[27]"},{"why":"Disney BRDF; defines the base color, roughness, and metallic material buffers used as conditions.","marker":"[12]"},{"why":"InteriorVerse; part of the co-training data for the inverse renderer.","marker":"[91]"},{"why":"HyperSim; part of the co-training data for the inverse renderer.","marker":"[64]"}],"fun_headline_variants":["Video diffusion does both inverse and forward rendering","One video in, relightable scenes out via diffusion","Diffusion model swaps G-buffers for photoreal video","Neural renderer turns video into editable scene buffers","From video to G-buffers and back: diffusion rendering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that the inverse renderer, trained only on synthetic data, produces accurate G-buffer estimates on real-world video, because those estimates become the training labels for the forward renderer; the paper supports this assumption with qualitative examples rather than real-world ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Video diffusion does both inverse and forward rendering","One video in, relightable scenes out via diffusion","Diffusion model swaps G-buffers for photoreal video","Neural renderer turns video into editable scene buffers","From video to G-buffers and back: diffusion rendering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1428,"prompt_tokens":896,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":467}},"tokens_in":512,"tokens_out":532,"duration_ms":4769,"temperature":1.0,"reasoning_tokens":467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:52:52.433475+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of real indoor and outdoor scenes, capture each under several known HDR lighting conditions, have the inverse renderer predict G-buffers from a single video, then run the forward renderer with the captured target light and measure PSNR, SSIM, and LPIPS against the true relit frames. If the synthetic-trained inverse estimates deviate strongly from measured normals and albedo, or the relit output is no closer to ground truth than per-frame image baselines, the central generalization claim would fail.","supporting_citations":[{"cited_title":"Elucidating the design space of diffusion-based generative models","cited_arxiv_id":null,"evidence_quote":"EDM; supplies the noise schedule and denoising score-matching objective used for training."},{"cited_title":"RGB ↔X: image decomposition and synthesis using material-and lighting-aware diffusion models","cited_arxiv_id":null,"evidence_quote":"RGB↔X; the image-diffusion G-buffer synthesis and rendering baseline this work extends to video."},{"cited_title":"Neural gaffer: Relighting any object via diffusion","cited_arxiv_id":null,"evidence_quote":"Neural Gaffer; provides the HDR environment-map lighting representation and a conditioning baseline compared against."},{"cited_title":"DL3DV-10K: a large-scale scene dataset for deep learning-based 3D vision","cited_arxiv_id":null,"evidence_quote":"DL3DV10k; the real-world video dataset auto-labeled and used to train the forward renderer."},{"cited_title":"DiffusionLight: light probes for free by painting a chrome ball","cited_arxiv_id":null,"evidence_quote":"DiffusionLight; off-the-shelf environment map estimation used to label real-world lighting."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"LoRA; the low-rank adaptation used to absorb the synthetic/real domain gap during real-data training."},{"cited_title":"What is in this image? Describe the materials. Be concise and produce an answer with a few sentences, no more than 50 words","cited_arxiv_id":null,"evidence_quote":"InteriorVerse; part of the co-training data for the inverse renderer."},{"cited_title":"Susskind","cited_arxiv_id":null,"evidence_quote":"HyperSim; part of the co-training data for the inverse renderer."}],"review_version":1}