{"id":"818895fa-87e4-4a06-af56-725916efd8da","arxiv_id":"2412.03225","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper adapts a text-to-video diffusion transformer so that material maps are generated as video frames, enabling material creation from a photo crop and/or text with automatic distortion correction.","lead":"MaterialPicker is a system that turns a photo of a surface, or a short text description, into a full set of 3D material maps such as color, roughness, height, and metalness. It works even when the photo is distorted, shot at an angle, or partially blocked, which makes picking materials out of everyday pictures much easier and faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's CLIP-I/DINO metrics are geometry-insensitive, so 'distortion correction'—a headline capability—is never quantitatively measured; a spatial-alignment metric on the known-warp synthetic benchmark is needed.","rationale":"I read the paper as claiming two interlocking things: a practical capability (a rough crop or text yields rectified, high-quality material maps) and a mechanism (the pretrained video DiT's temporal coherence becomes cross-channel material consistency). The capability is supported by abundant qualitative results and by Table 1 as an appearance-fidelity comparison against Material Palette; that support is non-trivial and should count. The load-bearing gap I see is narrower: the quantitative metric used for the flagship capability is insensitive to the geometric rectification that defines the capability. CLIP-I/DINO compare global embeddings, which by design tolerate translation, scale, and perspective distortions, so the numbers in Table 1 cannot distinguish 'rectified the crop' from 'generated a similar-looking flat texture.' The paper's own synthetic warped benchmark is the natural place to settle this, because the unwarped ground truth and warp parameters are known. This is not an attack on the authors' results; it is an identification of the specific measurement that would make the central claim falsifiable. It differs from the reader's weakest assumption, which concerns whether the video prior rather than the DiT/data does the work; that is an attribution question about the mechanism. The metric problem directly affects whether the headline capability is demonstrated. Both point to the same overall verdict: conditional pending stronger evidence.","tokens_in":18232,"tokens_out":8965,"duration_ms":94374,"concrete_test":"Run the synthetic distorted-texture benchmark described in Section 4.4.5 (TexSD materials with known homography and TPS warps) through MaterialPicker and Material Palette, and include a no-rectification baseline that feeds the warped texture as-is. Score each method with (1) a spatial rectification metric: after global color normalization only, PSNR/SSIM between the output albedo and the known unwarped ground truth; and (2) a residual-warp estimate showing how much of the known homography/TPS parameters remain in the output. If the Table 1 CLIP-I/DINO advantage is not accompanied by a clear improvement on these spatial metrics, the 'better distortion correction' headline is not established; if it is, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline capability is that a photo crop taken at an angle, with distortion or occlusion, is rectified into fronto-parallel material maps. The only quantitative evaluation of this capability, Table 1, uses CLIP-I and DINO embedding similarities between generated maps and ground truth. These embeddings are intentionally robust to geometric transformations, so a model that ignores the input perspective and produces a plausible texture with similar global appearance can still score well. Section 4.1.3 explicitly rejects per-pixel metrics, but distortion correction is inherently a spatial/geometric property, so that choice leaves the headline claim unmeasured. The synthetic homography/TPS distortion test in Section 4.4.5 is shown only qualitatively; no number records how much of the known warp is removed. Consequently, the central claim of 'better distortion correction' rests on visual inspection, not on the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MaterialPicker, a multi-modal material generator that fine-tunes a pretrained DiT-based text-to-video model (architecturally similar to HunyuanVideo) to produce five PBR material maps (albedo, normal, roughness, height, metallic) from an input photograph crop and/or a text prompt. The key design is to treat the material maps as video frames, with the input image as the first frame and a predicted dominant-material mask as the second frame, so the video prior enforces cross-channel consistency. The authors build a large synthetic dataset of 800K rendered scene crops with known material ground truth plus an 800K text-material pair set, and train the model at 256x256 resolution. They report quantitative CLIP-I/DINO comparisons with Material Palette on a synthetic benchmark, qualitative real-photo comparisons with Material Palette, ControlMat, and Hao et al., and a series of ablations covering modality combination, dataset mixing, mask prediction, input scale, distortion robustness, lighting robustness, and tileability. The central claims are more diverse generation, better distortion correction, and faster inference relative to prior material generation methods.","tokens_in":18434,"tokens_out":4603,"duration_ms":43950,"significance":"If the central claims hold, this is a practically useful contribution: a user can outline a rough crop around a material in an arbitrary photograph and obtain aligned, perspective-rectified PBR maps without controlled capture, masks, or a lengthy per-image optimization. The maps-as-frames adaptation of a video DiT is conceptually elegant and minimally invasive, and the paper provides a plausible mechanism—temporal consistency in the video prior transferring to spatial cross-map consistency—even though it is not proven analytically. The paper also honestly documents its limitations (shading/albedo confusion, holes/cutouts, text preservation, transparency) in Section 5. The evaluation is mixed: the synthetic comparison with Material Palette is properly quantified with confidence intervals on CLIP-I and DINO, and the ablations support several design choices, but the headline capability of distortion correction is not measured by any geometry-sensitive metric. This gap is substantive because the central novelty is precisely the robustness to perspective, distortion, and occlusion.","major_comments":[{"comment":"The quantitative evaluation of the paper's headline capability, distortion correction, is absent. CLIP-I and DINO are global embedding-similarity metrics that are intentionally robust to geometric transformations; a model that ignores the input perspective and generates a plausible fronto-parallel texture with similar global appearance can score well. The synthetic homography/TPS robustness test in Sec. 4.4.5 (Fig. 11) is shown only qualitatively. Because the central claim is that the model rectifies distorted or angled inputs, the authors should add a quantitative metric on this known-warp benchmark. For example, with the known ground-truth texture and the known warp, one can measure perceptual similarity (LPIPS) or pixel-level alignment (SSIM after inverse warping) between the generated albedo/normal maps and the undistorted ground truth, or measure keypoint/line-straightness errors. Without such a metric, the reported numbers in Table 1 cannot substantiate 'better distortion correction.'","section":"Sec. 4.1.3, Table 1"},{"comment":"The comparisons with ControlMat and Hao et al. are purely qualitative. In the text, the authors state that ControlMat 'fails to handle imperfect perspectives' and that Hao et al. 'does not generalize well to real-world photographs,' but no quantitative evidence is provided for either comparison. Since these are load-bearing claims about the advantage of the proposed approach over prior work, the authors should quantify at least a subset of these comparisons, e.g., on the synthetic dataset or on a small set of real photos with manually annotated rectification accuracy. Without numbers, the reader cannot assess whether the visual difference is consistent or representative.","section":"Sec. 4.2, Figs. 5 and 17"},{"comment":"The text-to-material comparison reports CLIP scores on three examples only, with no variance or full-dataset statistics. The claim that the model's text-to-material capability is 'comparable to other state-of-the-art approaches' is under-supported. A quantitative evaluation on a larger set of prompts with confidence intervals would be needed to fairly compare with MatGen and MatFuse, particularly because the paper's main contribution is image-conditioned generation but the text-conditioned mode is still claimed as a benefit.","section":"Sec. 4.3, Fig. 6"}],"minor_comments":[{"comment":"The statement that 'per-pixel metrics cannot be used for our results' is too categorical. For the synthetic distortion benchmark in Sec. 4.4.5, the ground truth and the warp are known, so pixel-wise or alignment-based metrics are in fact possible; they are only inapplicable to the uncontrolled real-photo setting.","section":"Sec. 4.1.3"},{"comment":"The sentence 'with the exception of the Albedo for which the intervals overlap' is ambiguous because there are two metrics: for CLIP the albedo interval overlaps but the mean favors the proposed method, whereas for DINO the albedo mean is lower for the proposed method. Please specify which metric is being discussed and clarify what 'overlap' implies for the comparison.","section":"Table 1"},{"comment":"The claim that 'the number of frames it is able to produce is not fixed' could be misleading: the model is trained with a fixed seven-frame layout, and inference uses the same layout. What the authors mean is that the transformer token sequence can accommodate a different number of frames in principle; this should be phrased as a design flexibility rather than a property of the trained model.","section":"Sec. 3.3"},{"comment":"The mask-output variant is claimed to 'perform comparably' to the mask-input variant, but no quantitative metric is given for the mask quality or the material quality. A simple IoU or appearance metric on the synthetic test set would make this ablation more convincing.","section":"Sec. 4.4.3"},{"comment":"The multi-modality ablation in Fig. 7 convincingly shows a single case where text helps disambiguate metallic vs non-metallic appearance, but it is anecdotal; consider adding a second or third example to demonstrate the effect is systematic.","section":"Sec. 4.4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is generally well written and the central idea is interesting, but the evaluation gap around the headline claim of distortion correction is significant. I would ask the authors to add a quantitative geometry-sensitive metric on the known-warp synthetic benchmark and to quantify at least one of the comparisons with ControlMat or Hao et al. before considering this for acceptance. The lack of a code/model release is not a blocker, but the dataset creation script is a useful step."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely new idea—treating a text-to-video DiT as a multi-map material generator by stacking albedo, normal, roughness, height, metallic, and mask as video frames, with the input photo as the first frame. The visual results, especially on in-the-wild crops, are convincing. The quantitative evaluation, however, does not actually measure the headline \"distortion correction\" claim. The only numbers in Table 1 are CLIP-I and DINO similarities, which are roughly invariant to geometric warp. The stress-test note is on target: a model that ignores perspective and emits a plausible texture can still score fine. This is not a fatal flaw, because the synthetic homography/TPS test in Sec. 4.4.5 shows qualitative rectification, but it means the paper's central claim rests on eyeballing figures plus an appearance metric that is insensitive to the geometry being corrected.\n\nCredit where due. The 800K crop-pair Scenes dataset is a real contribution, and the detail of rescaling material maps by UV coordinates so input and output share texture scale is careful. The mixed-dataset ablation (Sec. 4.4.2) supports the design choice honestly. The mask-as-output instead of mask-as-input saves a segmentation step and is shown to be quality-neutral. The 15x speedup over Material Palette is meaningful. The limitations section is candid about shading confusion, missing opacity, text, and transparency. This is a serious systems paper, not a toy.\n\nSoft spots, in proportion: (1) The metric gap is the main one. A spatial-alignment measure (e.g., warp error or correspondence-based accuracy) on the known-warp synthetic benchmark would directly test the distortion-rectification claim and should be added. (2) Comparisons to ControlMat and Hao et al. are qualitative only; given that ControlNet-based ControlMat is a natural baseline, some numbers would help. (3) No code, weights, or data are released; the paper promises a dataset creation script, but until that exists the 800K-pair claim is not independently checkable. (4) The \"temporal consistency transfers to spatial alignment\" premise is empirically motivated, not proven, but the paper does not oversell it—it is a design hypothesis validated by results.\n\nThe central argument holds up. The method exists and works; the evaluation is incomplete in one specific, fixable way. The reader's conditional verdict is fair.\n\nWho is this for? Researchers in material appearance, inverse rendering, and content creation. It deserves a serious referee. My recommendation: send to peer review, and require the spatial-alignment metric and preferably quantitative comparison with ControlMat before acceptance.","headline":"Genuinely new maps-as-frames adaptation of a video DiT for material generation, with strong visuals and a real dataset contribution, but the headline distortion-correction claim is not quantified by the reported metrics.","tokens_in":18969,"tokens_out":1834,"would_cite":true,"duration_ms":20324,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuning a pretrained text-to-video Diffusion Transformer with material maps as video frames yields a multi-modal generator that corrects distortion and occlusion in photo crops and outputs albedo, normal, roughness, height, and…","keywords":["material generation","PBR material maps","diffusion transformer","video diffusion model","single-image material capture","text-to-material generation","surface rectification","inverse rendering"],"falsifier":"Train the same model with the five material-map frames randomly shuffled at both training and inference, or with a U-Net video backbone in place of the DiT, and compare cross-map consistency and distortion rectification on the synthetic evaluation set. If shuffled-order or U-Net variants produce equally consistent, equally rectified materials, the claim that the DiT's video prior enforces cross-channel coherence would be undermined.","tokens_in":18075,"feed_emoji":"🎨","tokens_out":7273,"duration_ms":60517,"temperature":0.7,"pith_summary":"The paper claims that a pretrained text-to-video Diffusion Transformer can be turned into a practical material generator by treating the five PBR maps—albedo, normal, roughness, height, and metallicity—as the frames of a video sequence and the user's photo crop as the first, clean frame. After finetuning on 1.6 million paired examples (rendered synthetic-scene crops plus text-material pairs), the model outputs all five maps plus a dominant-material mask from an image crop, a text prompt, or both, correcting perspective distortion while ignoring occlusion and lighting in the input. This matters because it moves material authoring and inverse rendering away from controlled capture: a rough crop around any material in an ordinary smartphone photo is enough to obtain render-ready maps in about twelve seconds, without a mask or a special lighting setup. The paper argues that the video model's temporal-consistency prior becomes spatial consistency across material channels, and that this is why the approach outperforms image-diffusion baselines on distortion correction and diversity.","feed_headline":"Retrained video model turns rough photo crops into PBR materials","feed_subtitle":"It outputs albedo, normal, roughness, height and metallic maps in 12 seconds from a smartphone photo, with no controlled capture or masks.","key_machinery":"The load-bearing mechanism is the maps-as-video-frames adaptation of a Diffusion Transformer: five material maps and a mask are treated as video frames with temporal positional embeddings at fps=1, the input photo is the first clean frame, and the training loss denoises only the six generated frames. Because DiT operates on tokenized latents rather than fixed tensors, the model inherits the video prior's temporal consistency as spatial coherence across maps, tolerates misalignment between input photo and output maps, and can be extended to extra channels by appending frames. This setup also makes the mask a predicted output rather than a user input, and it keeps compatibility with test-time tricks like noise rolling for seamless tiling.","core_discovery":"On its own terms, the paper's discovery is that a video generator's prior can be re-purposed without architectural surgery: the material maps are stacked as fixed 'keyframes' at fps=1, giving a seven-frame input of photo, mask, and five maps, with noise applied only to the last six frames. The DiT's token-based attention then reasons jointly over the photo and the desired maps while tolerating pixel misalignment, which the paper hypothesizes conv-based U-Net backbones cannot do because they prefer pixel-aligned input-output pairs. The result is a multi-modal generator that rectifies distorted or angled photos of materials, handles occlusion, outputs a mask for the dominant material, matches the texture scale of the input, and produces more diverse materials than training-from-scratch material models, because the pretrained video prior is preserved.","pith_inferences":["If the temporal-to-spatial transfer is the real source of the gain, then material quality should improve automatically as base video-generation models improve, making the recipe a durable adapter rather than a one-off architecture.","A testable extension would be to vary crop purity (the training data requires the dominant material to cover at least 70% of the crop) and measure whether mask accuracy and map quality degrade gracefully on mixed-material crops.","Because the paper deliberately targets appearance similarity rather than pixel-aligned measurement, the method is better positioned for authoring and rendering than for metrology; quantifying true map accuracy would require a test set with known geometry and lighting.","Another extension is to ask whether frame order matters: shuffling the five map frames during finetuning and inference should degrade cross-map consistency if the temporal prior is doing the work; if it does not, the consistency could come from the text or image prior instead."],"forward_implications":["A user can outline a rough crop around a material in a smartphone photo and receive albedo, normal, roughness, height, and metallic maps in about 12 seconds, with no mask and no flash-only capture.","Because texture scale is aligned during dataset construction, generated maps match the scale of the input photo, and noise rolling makes them tileable at 512 or 1024 resolution without retraining.","Adding a text prompt reduces ambiguity in image-conditioned generation, for example distinguishing metallic from non-metallic surfaces, while text-only generation stays competitive with dedicated text-to-material models.","The same finetuning recipe could generate additional output channels, such as opacity, specular, coat, or subsurface-scattering maps, by appending new frames to the token sequence.","The approach opens a route to other multi-channel prediction tasks, such as intrinsic image decomposition, by reusing the video prior."],"supporting_citations":[{"why":"Supplies the pretrained DiT-based text-to-video model whose architecture and prior are finetuned into the material generator.","marker":"[Weijie et al. 2024]"},{"why":"Defines the Diffusion Transformer backbone and VAE tokenization that make frame count and channel count flexible.","marker":"[Peebles and Xie 2023]"},{"why":"The DiT-based video generation model whose strong video prior motivates re-purposing a video generator for material maps.","marker":"[Brooks et al. 2024]"},{"why":"Source of the 800K text-material pairs, the ControlMat baseline for material capture, and the noise-rolling technique for tileable outputs.","marker":"[Vecchio et al. 2024a]"},{"why":"The Material Palette baseline that the material-extraction comparison must beat, and the source of the TexSD texture test set.","marker":"[Lopes et al. 2024]"},{"why":"Baseline for texture rectification against which the paper compares distortion handling on real photographs.","marker":"[Hao et al. 2023]"},{"why":"Provides the additional Materials dataset that is augmented to 800,000 cropped material maps for generation diversity.","marker":"[Martin et al. 2022]"},{"why":"CLIP embeddings are used both as the text encoder in the model and as the CLIP-I evaluation metric.","marker":"[Radford et al. 2021]"}],"fun_headline_variants":["Video prior DiT turns rough photo crops into PBR material maps","No masks needed: re-tasked video model outputs material maps","Material generation from photo or text via a video model prior","DiT repurposed from video generation to fix material distortion","Multi-modal material maker: one DiT from video prior to PBR maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire maps-as-frames design rests on the assumption that the temporal-consistency prior learned by a text-to-video model transfers to spatial consistency across material maps when the maps are arranged as fixed keyframes at fps=1, a transfer the paper validates only empirically in its experiments.","fun_headline_variants_meta":{"raw":{"variants":["Video prior DiT turns rough photo crops into PBR material maps","No masks needed: re-tasked video model outputs material maps","Material generation from photo or text via a video model prior","DiT repurposed from video generation to fix material distortion","Multi-modal material maker: one DiT from video prior to PBR maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001075,"raw_usage":{"total_tokens":4456,"prompt_tokens":857,"completion_tokens":3599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":3510}},"tokens_in":473,"tokens_out":3599,"duration_ms":27096,"temperature":1.0,"reasoning_tokens":3510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:37:38.944752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model with the five material-map frames randomly shuffled at both training and inference, or with a U-Net video backbone in place of the DiT, and compare cross-map consistency and distortion rectification on the synthetic evaluation set. If shuffled-order or U-Net variants produce equally consistent, equally rectified materials, the claim that the DiT's video prior enforces cross-channel coherence would be undermined.","supporting_citations":[],"review_version":1}