Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Fine-Grained Spatially Varying Material Selection in Images

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a two-resolution vision-transformer selector separates whole textures from their repeating components and stays consistent as clicks, zoom, or lighting change.

desk verdict Solid texture-level material selection with a genuinely useful dataset, but the advertised subtexture novelty is validated mainly against the authors' own operationalization and a 20-image self-annotated test set. read the letter →

arxiv 2506.09023 v2 pith:6AZMBN75 submitted 2025-06-10 cs.GR cs.CV

Julia Guerrero-Viu , Michael Fischer , Iliyan Georgiev , Elena Garces , Diego Gutierrez , Belen Masia , Valentin Deschaintre This is my paper
classification cs.GRcs.CV
keywords materialselectionsubtextureDINOv2visiontransformermulti-resolutionfeaturessyntheticdatasetimageeditingconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Material selection—marking every pixel that shares a clicked material—is a common first step in image editing, but previous methods blur fine boundaries and cannot separate the repeating elements of a pattern. The paper proposes a model that selects at two granularities: the whole texture, such as all the squares of a chessboard, and the subtexture, such as the black squares only. Its method feeds a downscaled image and high-resolution tiles through the DINOv2 vision transformer, concatenates the resulting features, and compares them with the clicked pixel to produce two similarity maps. The authors train on a new synthetic dataset of more than 800,000 images with dense annotations at both levels, and report higher accuracy and consistency than prior methods, especially on thin structures and across changes of illumination. If the claim is right, fine edits such as recoloring one flower on a dress reduce to a single click.

What carries the argument

The load-bearing mechanism is multi-resolution feature aggregation. From each of four transformer blocks of DINOv2, the method resamples the feature maps of the downscaled full image and of the tiled high-resolution image to a common target resolution and concatenates them along the channel axis, preserving spatial layout; this is different from earlier scaling approaches that downsample the high-resolution features. The aggregated features are compared, through a cross-similarity layer, with the image patch centered on the user's clicked pixel, producing query-conditioned features that are upsampled and fused by a residual CNN into two sigmoid output channels. The subtexture level is carried by the dataset construction: a binary mask combines two reflectance maps into one texture, and the subtexture ID is the reflectance map underneath the mask.

What would settle it

Have independent annotators segment individual pattern elements on the Two-Level Test images and compare their boundaries with the published ground truth; if annotators disagree strongly with each other or with the mask-derived labels, the subtexture granularity is not a stable quantity.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that material selection becomes both finer and more stable when the selector is driven by features from DINOv2—a self-supervised vision transformer pre-trained on natural images—extracted at two resolutions. The full image is downscaled to the encoder's native resolution, the original image is split into four tiles, and the feature maps from four transformer blocks are resampled and concatenated along the channel dimension, so the selection head sees broad context and fine detail simultaneously. The model outputs two per-pixel similarity maps, one for texture and one for subtexture, trained jointly with binary cross-entropy loss. Subtexture is defined in the new DuMaS dataset by alpha-compositing two reflectance maps under a binary mask, so the subtexture label is the constituent reflectance map. On the Materialistic Test, the new Two-Level Test, and the Multi-Illumination datasets, the paper reports the best L1, IoU, and F1 among the compared methods, and roughly 1.8 times lower consistency error than Materialistic under pixel, zoom, and lighting changes.

Load-bearing premise

The load-bearing premise is that 'subtexture' is well defined by binary-mask combinations of two reflectance maps, and that the authors' manual annotations on 20 real images are a faithful ground truth for that definition.

Editorial extensions

If this is right

  • Users can edit individual repeating pattern elements, such as one flower on a dress or one deer on a cushion, without manually masking every instance.
  • A single jointly trained model covers both selection granularities, so an editing tool does not need separate texture and subtexture models.
  • Because DuMaS is rendered as video, the same data can fine-tune video selection models like SAM2 to track material selections across frames.
  • Selection predictions are less sensitive to the threshold used to turn similarity scores into masks, reducing the need for manual threshold adjustment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mask-based definition of subtexture transfers to real materials, then material selection could be extended to video with temporal consistency by fine-tuning on the rendered camera trajectories, a direction the paper itself mentions.
  • The method's reliance on a binary subtexture definition suggests it will treat continuous material gradients, like a rainbow wall, as ambiguous; a graded similarity output would be a natural follow-up.
  • Because the multi-resolution tiling is encoder-agnostic, the same concatenation strategy could be applied to other ViT-based selection and segmentation tasks beyond material editing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents a method for two-level material selection in images, where a user click on a pixel yields per-pixel similarity maps at both texture and subtexture granularities. The method builds on the Materialistic architecture, replacing the encoder with DINOv2, adding a multi-resolution feature aggregation scheme, a two-channel selection head, and multi-query sampling during training. The authors introduce DuMaS, a synthetic dataset of over 800,000 rendered images with dense texture and subtexture annotations, and evaluate on the existing Materialistic Test set (50 images) plus a new manually annotated Two-Level Test set (20 images), comparing against Materialistic and SAM2 fine-tuned on DuMaS. Quantitative results, robustness experiments, and ablations are reported in support of the claimed improvements.

Significance. If the results hold, the paper would advance material selection by enabling finer-grained selections and improving consistency, and the DuMaS dataset could be a valuable resource for future work. The architectural modifications are reasonable and the ablations consistently favor the full method. The promise to release code and part of the dataset is a strength. However, the subtexture-level evaluation rests on the authors' own synthetic operationalization and self-annotated test set, so the central novelty is not yet independently grounded; this is the main factor limiting confidence in the contribution as presented.

major comments (2)
  1. [Sections 3.2 and 4.1, Table 1] The subtexture level is defined by the authors' synthetic construction in Section 3.2, where a texture is a binary-mask combination of two reflectance maps and the subtexture label is the constituent reflectance map ID. The only real-world validation is the Two-Level Test set of 20 images, manually annotated by the authors (Section 4.1), with no stated annotation protocol or inter-annotator agreement. Since subtexture selection is the main novel claim, the quantitative subtexture results in Table 1 and Table 2 may partly measure agreement with the authors' operationalization rather than an externally grounded notion of 'subtexture.' The paper's own Section 5 acknowledges the definition may not transfer to continuous surfaces, but the central claim of 'two-level selection' still needs stronger independent grounding. I recommend adding a detailed annotation protocol with inter-annotator agreement statistics, or an independent evaluation such as a user study, to substantiate the subtexture contribution.
  2. [Abstract, Tables 1 and 2] The paper repeatedly claims to 'significantly outperform' previous work, but no variance estimates or significance tests are reported. The real-image test sets are small (50 and 20 images, with 10 queries per image), and the samples are not independent because multiple queries come from the same image. The reported differences, while large in several metrics, cannot be assessed for statistical reliability. Please report per-image standard deviations or confidence intervals and perform a paired significance test (e.g., Wilcoxon signed-rank on per-image IoU or F1) for the main comparisons, so that the 'significant' claim is supported.
minor comments (5)
  1. [Table 3] The 'Challenging Subset' used in the rightmost columns is not defined in the text; please specify its source (e.g., a subset of the two test sets) and composition so readers can interpret the ablation results.
  2. [Section 3.1.2] The multi-resolution pyramid construction is unclear for the 1024x1024 training images: the text says each I_i is split into 2^i non-overlapping tiles of resolution r x r, but 1024 is not divisible by the DINOv2 patch size 14. Please clarify how the tiles are resized or padded to the native encoder resolution.
  3. [Section 4.3] For the pixel and zoom consistency experiments, the number of images and materials used is not reported; please provide these counts so the reader can gauge the coverage of the evaluation.
  4. [Section 1 and 4.1] The paper states it will release 'a significant subset' of the dataset and the evaluation framework; please specify which subset and under what license, since this affects the reproducibility value of the contribution.
  5. [Caption of Figure 8] The caption reads 'Input & GT' for the first column, but the abbreviation GT is not expanded in that caption; consider writing 'ground truth' for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the selection pipeline is trained on synthetic annotations and tested on held-out real-world datasets, with no prediction reducing to its inputs by construction.

full rationale

The paper's central claims are empirical: a ViT-based model is trained on the DuMaS synthetic dataset and evaluated on held-out real-world data. The texture-level comparison uses the pre-existing Materialistic Test set (50 images), which is external to this work, and the illumination-robustness evaluation uses the external Multi-Illumination dataset. The subtexture level is operationalized in Section 3.2 by construction (a texture map is a binary-mask combination of two reflectance maps, and the subtexture ID is the constituent reflectance map ID), but the evaluation of that level uses a separate, manually annotated 20-image test set (Section 4.1); it is not generated by the same automated construction, so the reported IoU/L1/F1 numbers are not equal to the training labels by construction. Section 5 explicitly acknowledges that the subtexture definition 'may not easily translate to continuously varying surfaces' and that material selection is inherently ambiguous, which is a construct-validity caveat rather than a circular step. Self-citations (Materialistic, SAMa, MatSynth) are used as prior methods or baselines, not as the sole justification of the paper's claims; DINOv2 features and S2 scaling are external and independently validated. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. Under the strict standard of exhibiting an equation or fitted value that reduces to its own input, no circular step is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central method depends on the transferability of frozen DINOv2 features and of synthetic renders to real images, and on the validity of the subtexture operationalization. These are stated as design choices rather than proven, and are the main items a reader should test when reproducing the work.

free parameters (4)
  • Number of multi-resolution levels n = 2
    Chosen by experimentation; three levels did not improve results materially and increased compute (Section 3.1.2).
  • Number of query pixels per image during training = 10
    Uniformly sampled 10 pixels per image crop to stabilize training (Section 3.3); ablation shows improvement.
  • Binarization threshold = 0.5
    IoU and F1 metrics computed by thresholding similarity scores at 0.5 (Section 4.2); threshold sensitivity is reported as robustness, not optimized.
  • Training hyperparameters = lr 1e-4, 10 epochs, batch size 4 per GPU
    Adam optimizer settings listed in Section 3.3, standard choices not derived from first principles.
assumptions (3)
  • domain assumption DINOv2 self-supervised features, frozen, contain sufficient material information for selection.
    The paper relies on pretrained DINOv2 features and only trains a lightweight head (Section 3.1.1); no analysis of what the features encode is provided.
  • domain assumption DuMaS synthetic renders are a sufficient training distribution for real-world material selection.
    The method is trained exclusively on synthetic Blender renders (Section 3.2) and tested on real photos; the transfer assumption is standard for this line of work but unproven beyond the small test sets.
  • ad hoc to paper Subtexture is a well-defined, consistent annotation target across different materials and scenes.
    Subtexture is defined via binary mask composition of two reflectance maps (Section 3.2); this operationalization may not match perceptual grouping for continuous or non-composite materials, as the paper itself notes for the rainbow wall in Section 5.
invented entities (1)
  • Subtexture selection level
    purpose: Defines a finer granularity than texture-level selection, grouping only pixels within the same constituent reflectance component of a spatially varying texture.
    The level is defined and labeled in the authors' synthetic dataset and their manual test set; no external benchmark exists yet to validate the perceptual reality of this grouping.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-Grained Spatially Varying Material Selection in Images." pith.science (2026). https://pith.science/paper/6AZMBN75

@misc{pith2026250609023,
  author       = {Pith},
  title        = {Pith review of: Fine-Grained Spatially Varying Material Selection in Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6AZMBN75}},
  note         = {Machine review of arXiv:2506.09023}
}
read the original abstract

Selection is the first step in many image editing processes, enabling faster and simpler modifications of all pixels sharing a common modality. In this work, we present a method for material selection in images, robust to lighting and reflectance variations, which can be used for downstream editing tasks. We rely on vision transformer (ViT) models and leverage their features for selection, proposing a multi-resolution processing strategy that yields finer and more stable selection results than prior methods. Furthermore, we enable selection at two levels: texture and subtexture, leveraging a new two-level material selection (DuMaS) dataset which includes dense annotations for over 800,000 synthetic images, both on the texture and subtexture levels.

Figures

Figures reproduced from arXiv: 2506.09023 by the authors.

Figure 1
Figure 1. Our proposed method allows fine-grained material selection in images on two different levels of granularity, significantly outperforming previous work [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Model architecture. Our model extracts ViT features at different resolutions, for the full image and for each separate tile, leading to a multi-resolution feature encoding. The subsequent spatial processing layers upscale the features before the cross-similarity computes the attention with respect to the clicked image pixel and patch. We exploit the information encoded at different depths by repeating this process a… view at source ↗
Figure 3
Figure 3. Fine-grained material selection. Our multi-resolution aggregation allows to recover thin structures, overcoming a limitation acknowledged in previous work [Sharma et al. 2023]. 2024]. This strategy is similar in spirit to using a higher effective batch size, re-using the image encoder computation and making the optimization more stable. We show in Section 4 how the multiple query sampling benefits the robustness and… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: DuMaS training dataset creation. Starting from twelve base noise patterns from Substance Designer, we vary their parameters to create 1,571 binary masks which we use to randomly combine 3,026 stationary reflectance maps to generate 10,881 materials (see text for more d…
Figure 5
Figure 5. Figure 5: Qualitative results. We show results of our method (two-level selection) for images in our real-world test datasets. For each example and level, we show the predicted, binarized selection masks as green image overlays and the similarity score before thresholding in fal…
Figure 6
Figure 6. Figure 6: Qualitative comparison. Texture-level selection results for Materialistic, SAM2 fine-tuned on our DuMaS dataset, and our method. The white squares highlight areas where the methods struggle. Materialistic fails to select relevant areas, especially in the presence of sm…
Figure 7
Figure 7. Figure 7: Robustness. Robustness evaluation of our method with respect to the clicked pixel (left subset), the image crop with increasing zoom levels (middle subset), and illumination changes (right subset). The images are challenging due to similar albedo (left), strong shading…
Figure 8
Figure 8. Figure 8: Ablations. We ablate parts of our method (multi-resolution and multi-sampling, respectively) on two challenging examples containing thin structures (net and zebra stripes) and albedo entanglement (net has a similar albedo to the white parts of the basketball, in front …
Figure 9
Figure 9. Figure 9: Editing. We use our method’s selection masks at subtexture level to perform fine-granular edits of the image’s materials in Photoshop. (Ours, Single Level). Notably, training with all data concurrently does not negatively impact performance, allowing to have a single m…
Figure 10
Figure 10. Figure 10: Limitations. Our method struggles with clicks on out-of-focus regions (top) and long-horizon imagery with changing frequencies (middle). Textures without individual components (bottom) stretches our definition of subtexture. 6 Conclusion We present a novel method for …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. {\Phi}eat: Physically Grounded Material Feature Representation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Φeat fine-tunes a DINOv3 ViT with renderings of the same material under varied geometry and lighting, yielding features that improve material selection and material k-NN classification.

Reference graph

Works this paper leans on

5 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [3]

    InProceedings of the IEEE/CVF International Conference on Computer Vision

    Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026. Peter Kocsis, Vincent Sitzmann, and Matthias Niessner. 2024. Intrinsic Image Diffusion for Indoor Single-view Material Estimation.Conference on Computer Vision and Pattern Recognition (CVPR). Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Di...

    arXiv 2003

  2. [4]

    In International Conference on Machine Learning

    Hiera: A hierarchical vision transformer without the bells-and-whistles. In International Conference on Machine Learning. PMLR, 29441–29454. Lavanya Sharan. 2009.The perception of material qualities in real-world images. Ph. D. Dissertation. Massachusetts Institute of Technology. Lavanya Sharan, Ce Liu, Ruth Rosenholtz, and Edward H Adelson. 2013. Recogni...

    work page 2013

  3. [2017]

    In2017 international conference on 3D vision (3DV)

    Segcloud: Semantic segmentation of 3d point clouds. In2017 international conference on 3D vision (3DV). IEEE, 537–547. Paul Upchurch and Ransen Niu. 2022. A dense material segmentation dataset for indoor and outdoor scene parsing. InEuropean conference on computer vision. Springer, 450–466. A Vaswani. 2017. Attention is all you need.Advances in Neural Inf...

    work page 2017

  4. [2023]

    Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion

    Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion.arXiv preprint arXiv:2306.04633(2023). Daniel Bolya, Chaitanya Ryali, Judy Hoffman, and Christoph Feichtenhofer. 2023. Window Attention is Bugged: How not to Interpolate Position Embeddings.arXiv preprint arXiv:2311.05613(2023). Hussein Boyaci, Lawrence T Maloney, and Sarah ...

    work page Pith review arXiv 2023

  5. [2024]

    https://evermotion.org/shop/cat/397/archinteriors

    Evermotion Arch Interior. https://evermotion.org/shop/cat/397/archinteriors. Miika Aittala, Tim Weyrich, Jaakko Lehtinen, et al. 2015. Two-shot SVBRDF capture for stationary materials.ACM Trans. Graph.34, 4 (2015), 110–1. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A richly annotated catalog of surface appearance.ACM Trans...

    work page 2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.