{"id":"a2af56ed-1934-4a61-bff1-fccff0532a6e","arxiv_id":"2607.08016","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A video relighting system that refines a physically based re-render of the input under target lighting with a fine-tuned video diffusion model and reports state-of-the-art results on synthetic and real benchmarks.","lead":"LightCrafter relights videos by first using physics to render the scene under new lighting, then having an AI video model turn that rough render into photorealistic footage. The design separates lighting control from artifact cleanup, which is what keeps shadows and colors stable over long clips.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world evaluation may reward pipeline-specific artifact correction rather than true relighting: DL3DV is a round-trip/reconstruction task and pseudo-pair training optimizes source lighting to fit the input, so the claimed real-world SOTA is not yet established by captured relit video ground trut","rationale":"The paper's central idea—refine a PBR proxy rather than translating pixels directly—is coherent, and the synthetic benchmark provides genuine ground-truth evidence (Table 1 synthetic: LightCrafter 24.16 PSNR vs 20.92 for the best prior) that the hybrid pipeline improves relighting under controlled conditions. The artifact-matched synthetic curation in Sec. A.1 is a real strength: training on renders from the actual reconstruction stack rather than clean buffers is the right way to avoid train/test mismatch. I also credit the ablations showing that both synthetic and real data contribute (Table 2), and the attempt to evaluate on MIT. The load-bearing weakness is the real-world evaluation and the pseudo-pair assumption. Because both the real-world training pairs and the DL3DV evaluation use the same inverse-rendering/PBR pipeline, and because the envmap optimization in Eq. (1) is free to compensate for reconstruction errors, the model can learn and be rewarded for a pipeline-specific correction that is near-identity in lighting. This does not invalidate the method's potential, but it means the claim of state-of-the-art on real-world video relighting is not yet supported by captured relit video ground truth. The reader's CONDITIONAL verdict already captures this; my stress test does not move it, though it sharpens the condition: demonstrate direct relighting to captured multi-illumination video and verify the optimized source lighting matches the true capture.","tokens_in":15147,"tokens_out":9646,"duration_ms":101795,"concrete_test":"Construct or use a captured multi-illumination video benchmark (e.g., extend MIT Multi-Illumination with camera motion, or use a newly captured set) and evaluate LightCrafter by relighting a source video directly to a held-out target illumination, comparing against the captured target video. If the PSNR/LPIPS on this direct test is substantially worse than the reported DL3DV round-trip numbers, the round-trip evaluation is rewarding reconstruction/artifact-correction rather than true relighting. As a secondary check, compare the optimized Lsource from Eq. (1) to the known captured envmap on MIT scenes; large deviations that track albedo/geometry would confirm that the pseudo-pair assumption is violated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4's real-world pseudo-pair construction optimizes Lsource in Eq. (1) with recovered S held fixed to minimize ||R(S,Lsource) - I||. This is underconstrained: with imperfect albedo, normals, or geometry, the optimizer can absorb scene-dependent appearance into the envmap, e.g., shifting light direction to compensate for wrong normals or boosting intensity to compensate for dark albedo. The resulting render Î is therefore close to I by construction, and the pair (Î, I) teaches the refiner to undo this specific pipeline's artifacts rather than to relight under novel illumination. At test time, the DL3DV evaluation is a round-trip: a LaV-relit video is inverse-rendered, re-rendered, and refined back to the original, with metrics computed against that original. A model that has learned the pipeline's render-to-input mapping can score highly on this task without performing general relighting. The MIT Multi-Illumination result is real captured ground truth, but it is static-image and does not exercise long-form video or the same artifact distribution as the video pipeline. Thus the load-bearing evidence for 'outperform prior SOTA on real-world relighting benchmarks' rests on a metric that can be satisfied by reconstruction, not relighting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LightCrafter proposes a hybrid video relighting pipeline that first recovers scene intrinsics via inverse rendering, renders a physically-based rendering (PBR) proxy under the target illumination, and then fine-tunes a video diffusion model (CogVideoX-5B) to refine the proxy into a photorealistic relit video. The key idea is to bake the illumination target into the PBR render, so the diffusion model only needs to correct rendering artifacts rather than learn lighting concepts. Training uses synthetic paired videos (with ground-truth relit targets) and real-world pseudo-pairs, where an environment map is optimized so the PBR render matches the input video. The paper also contributes a synthetic benchmark, a long-video overlap-fusion inference scheme, and reports improvements over existing baselines on synthetic and real-world benchmarks.","tokens_in":15533,"tokens_out":6071,"duration_ms":57643,"significance":"If the claims are validated, the hybrid reformulation—translation of a PBR proxy rather than direct video-to-video translation—is a conceptually appealing way to combine explicit light control with generative realism. The artifact-matched data curation idea is useful, and the release of dataset, code, and benchmark promises reproducibility. The synthetic paired evaluation is a genuine contribution with ground-truth relit targets, and the full method does improve on it. However, the real-world video evaluation rests on a round-trip reconstruction proxy rather than captured relit videos, and the pseudo-pair training relies on an unproven assumption that optimized lighting does not absorb inverse-rendering errors. These issues affect the strength of the central 'real-world SOTA' claim.","major_comments":[{"comment":"The 'Real-World' results average MIT Multi-Illumination (captured ground truth, but static images) with DL3DV (a round-trip reconstruction proxy). The DL3DV protocol relights via Light-A-Video and then asks the method to map back to the original input; metrics are computed against the original frames. A model trained on pseudo-pairs (Sec. 3.4) that has learned its own pipeline's render-to-input mapping can score highly on this task without performing general relighting. Since the abstract claims to 'outperform prior SOTA on existing real-world relighting benchmarks,' the video-relighting claim is not supported by the DL3DV numbers. Report MIT and DL3DV separately and either add a captured multi-illumination video benchmark or explicitly scope the claim to image relighting plus a reconstruction proxy.","section":"§4.1, Table 1"},{"comment":"The pseudo-pair construction optimizes a 64×128 HDR environment map to minimize ||R(S,L_source)−I|| with S held fixed. With imperfect normals, albedo, or geometry, this underconstrained optimization can absorb scene-dependent appearance errors into the lighting (e.g., shifting light direction to compensate for wrong normals, boosting intensity for dark albedo). The three scalar alignment parameters only correct global yaw/pitch/exposure. Thus the assertion that 'any discrepancy between Î and I comes from inverse rendering errors rather than lighting change' is an unsupported axiom. The pair (Î,I) may instead teach a pipeline-specific identity/correction mapping. Validate by comparing optimized envmaps to ground-truth lighting in synthetic scenes, or by testing on held-out scenes with known lighting.","section":"§3.4, Eq. (1)"},{"comment":"All quantitative claims are point estimates without error bars, number of test videos, or significance tests. The real-world differences (e.g., LightCrafter 19.91 vs. PCRP-video 17.97 PSNR) could be within noise given the small evaluation set (10 MIT scenes, unstated DL3DV clip count). Report standard errors or confidence intervals across scenes and illuminations, and state the number of DL3DV test clips used.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The claim of outperforming prior SOTA on real-world benchmarks should be qualified given the proxy nature of the DL3DV evaluation. Section 5 also contains a typo: 'This division of yields higher fidelity' appears to be missing a word (likely 'labor').","section":"Abstract / §5"},{"comment":"The captions of Figures 9–13 contain garbled strings such as '/uni0000002c/uni00000051/...' that appear to be accidental text; please clean these up.","section":"Figures 9–13"},{"comment":"Please specify the number of DL3DV test clips and report MIT and DL3DV metrics separately. Also define T-CLIP, which is used but not described.","section":"§4.1"},{"comment":"The conditioning noise augmentation is described as applied in pixel space before encoding; clarify whether the noise is added before or after tonemapping, and whether the same noise schedule is used at inference.","section":"§A.1, Eq. (9)"},{"comment":"PCRP-video is an author re-implementation since the original PCRP is image-only and has no public code. State clearly that this baseline is a video extension that may not reflect the original method's full performance, and consider labeling it 'PCRP-video (our impl.)' in tables.","section":"§A.3"}],"recommendation":"major_revision","confidential_remarks":"The main weakness is the real-world evaluation: the DL3DV round-trip protocol rewards reconstructing the input through the same pipeline used for pseudo-pair training, so the 'real-world SOTA' claim is not yet established. The authors should be encouraged to present MIT and DL3DV separately, add a true captured multi-illumination video benchmark if possible, and empirically validate the pseudo-pair assumption. The paper also has unusual future-dated references (2026) and an arXiv ID from July 2026; this does not affect the technical content but may need editorial attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LightCrafter does two things well. First, it reformulates video relighting as translation of a PBR proxy: instead of asking a video diffusion model to infer lighting from pixels, it renders the proxy under the target light and fine-tunes CogVideoX to fix the proxy's artifacts. That is a sensible division of labor. Second, the data curation is a real contribution: the artifact-matched synthetic pairs and the real-world pseudo-pair pipeline directly attack the train/test mismatch that plagues most generative relighting. The overlap-fused temporal tiling, fusing noise predictions across windows before each scheduler step, is also a clean idea for long-form consistency. The synthetic benchmark with ground-truth relit targets is genuine evidence, and the full method beats the baselines on it; the ablations showing that both synthetic and real data matter are useful.\n\nThe soft spot is the real-world evaluation. The pseudo-pair training optimizes a per-scene environment map so the PBR render matches the input, plus three alignment parameters (yaw, pitch, exposure). The paper claims that any residual discrepancy is inverse-rendering error rather than lighting change. That is plausible but unverified. With imperfect geometry or albedo, the optimizer can absorb scene-dependent appearance into the lighting, making the real-world training branch a pipeline-specific correction loop. The DL3DV evaluation is a round-trip: relight with LaV, then map back to the original. A model that has learned to invert its own render-to-input mapping can score well on that without performing general relighting. The MIT Multi-Illumination result is genuine captured ground truth, but it is static imagery, so it does not exercise the video claims. On top of that, Tables 1 and 2 report point estimates without error bars, and the PCRP-video baseline is author-implemented.\n\nNone of this sinks the core idea. The synthetic results and the ablations give good evidence that the hybrid works; what is not established is the \"state-of-the-art on real-world benchmarks\" claim. That is a validation gap, not a load-bearing flaw. The paper also has an honest limitations section, acknowledging noise on specular and transparent surfaces and that errors can exceed what the refiner corrects.\n\nThis deserves a serious referee. It is for researchers in video relighting, inverse rendering, and generative video editing, and it makes a useful contribution even if the real-world numbers are weaker than claimed. I would want referees to ask for code and data release, error bars, a captured multi-illumination video evaluation if one exists, and a clear explanation of why the DL3DV round-trip does not reward an identity mapping.","headline":"A solid hybrid idea for video relighting — PBR proxy translation plus a diffusion refiner — with good synthetic evidence, but the real-world evaluation is the weak link: the DL3DV round-trip can be satisfied by pipeline-specific correction rather than true relighting.","tokens_in":16034,"tokens_out":3368,"would_cite":true,"duration_ms":33324,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LightCrafter relights video by translating a PBR re-render of the input, not the raw footage, and beats prior state-of-the-art on real benchmarks.","keywords":["video relighting","physically based rendering","diffusion models","inverse rendering","temporal consistency","artifact-matched training","generative video translation","long-form video"],"falsifier":"Take a real multi-illumination video with captured ground-truth light B and input light A, run the full pipeline from A to B, and compare the refined output to the captured B frames and to the PBR proxy alone. If the refinement does not clearly beat the proxy, or if a round-trip (A→B→A) succeeds mainly by restoring a learned pipeline-specific correction, the pseudo-pair assumption and the central hybrid claim are undermined.","tokens_in":15013,"feed_emoji":"💡","tokens_out":12914,"duration_ms":96599,"temperature":0.7,"pith_summary":"LightCrafter tries to establish that video relighting is best treated as refinement of a physically based proxy: first recover a scene's intrinsics, re-render it under the target light with a forward renderer, and let a video diffusion model clean up that noisy render into a photorealistic output. The point is that the PBR proxy already carries the structured lighting change — shadows, highlights, shading — and is temporally consistent by construction, so the diffusion model only needs to correct inverse-rendering artifacts instead of learning what an environment map is. The authors argue this hybrid outperforms both inverse-rendering-only and direct generative translation on real-world relighting benchmarks, and provides long-form consistency without chunk-boundary drift. If right, it moves the hard part of relighting from the generative model back to the renderer, making relighting control a matter of rendering rather than prompting.","feed_headline":"Relighting video as proxy refinement tops direct generative relighting","feed_subtitle":"Baking the target light into a PBR render gives control and long-form consistency.","key_machinery":"The central object is the PBR proxy video: a physically based rendering of the recovered scene state under the target illumination, computed with the rendering equation using a Lambertian diffuse lobe and a Cook-Torrance microfacet specular term, with visibility from meshified depth. Because incident illumination is additive, the target can encode environment maps, point lights, area lights, or combinations, and the rendering is frame-aligned and temporally consistent by construction. The diffusion refiner takes the latent of the proxy concatenated with the noisy target latent and predicts noise; it learns artifact correction, not relighting. The artifact-matched data curation — synthetic pa","core_discovery":"LightCrafter's central claim is that a noisy physically based rendering of a video under target illumination is a better conditioning signal for generative relighting than the input video or intrinsics alone. The pipeline first recovers scene intrinsics — depth, camera pose, albedo, normals, roughness/metalness — via off-the-shelf inverse renderers, optimizes a source environment map so a PBR render matches the input, then forward-renders the recovered scene under the target illumination using a physically based renderer with a Cook-Torrance BRDF and explicit visibility. It then fine-tunes a pretrained video diffusion transformer to translate that PBR proxy into a photorealistic video, train","pith_inferences":["If the pseudo-pair assumption doesn't hold — i.e., the optimized source light absorbs scene-dependent appearance rather than leaving only inverse-rendering error — the real-world training could teach the refiner a pipeline-specific correction, making the round-trip evaluation partially a test of that learned mapping rather than of true relighting.","The proxy-translation recipe likely transfers to other video edits where a cheap forward model exists (material swapping, weather simulation, object compositing), since the diffusion model only needs residual artifact correction.","The overlap-fused temporal tiling is a general drop-in for extending any windowed video diffusion model to arbitrary length when a consistent conditioning signal is available; it could be tested independently of relighting.","A cleaner evaluation would use captured real multi-illumination video pairs; if the refiner's gain over the PBR proxy shrinks on such pairs, the contribution of generative refinement would need to be rebalanced."],"forward_implications":["Relighting control becomes a rendering problem: any target light that a renderer can express — environment maps, point lights, area lights, combinations — is baked into the proxy, so the diffusion model never has to interpret illumination concepts.","Long-form relighting is stabilized: because the proxy is consistent across all frames and the refiner uses overlap-fused temporal tiling, sequences far longer than the training window can be relit without chunk-boundary drift or independent interpretation of the lighting.","Scene editing reduces to re-rendering: material edits, light insertion, and object compositing are applied to the recovered intrinsics or lighting, re-rendered, and refined by the same trained refiner with no retraining.","Training-data composition matters: the ablations show that removing synthetic pairs hurts relighting control while removing real-world pseudo-pairs hurts photorealism, implying the artifact-matched pairing is what makes the refiner generalize."],"fun_headline_variants":["PBR proxy video translation beats direct relighting","Relight long videos by editing a PBR render, not the video","Proxy PBR render enables control and consistency in relighting","PBR proxy conditioning: new state of the art in video relighting","Translate a PBR render under target light, tops relighting benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is the real-world pseudo-pair assumption: after optimizing the source environment map so the PBR render matches the input, any remaining discrepancy is treated as inverse-rendering error rather than lighting mismatch, so pairing the render with the input teaches artifact correction without ground-truth relit videos; if the optimized lighting absorbs scene-dependent appearance instead, the refiner may learn a pipeline-specific correction rather than tr","fun_headline_variants_meta":{"raw":{"variants":["PBR proxy video translation beats direct relighting","Relight long videos by editing a PBR render, not the video","Proxy PBR render enables control and consistency in relighting","PBR proxy conditioning: new state of the art in video relighting","Translate a PBR render under target light, tops relighting benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001069,"raw_usage":{"total_tokens":4361,"prompt_tokens":837,"completion_tokens":3524,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":3447}},"tokens_in":581,"tokens_out":3524,"duration_ms":22966,"temperature":1.0,"reasoning_tokens":3447,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:56:17.985988+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real multi-illumination video with captured ground-truth light B and input light A, run the full pipeline from A to B, and compare the refined output to the captured B frames and to the PBR proxy alone. If the refinement does not clearly beat the proxy, or if a round-trip (A→B→A) succeeds mainly by restoring a learned pipeline-specific correction, the pseudo-pair assumption and the central hybrid claim are undermined.","supporting_citations":[],"review_version":2}