Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

MRD uses physically based differentiable rendering to construct 3D scenes that are physically different but produce identical activations in a vision model, revealing which shape and material cues the model truly encodes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:40 UTC pith:WOPPOUGG

load-bearing objection New combination of differentiable path tracing and model-metamer optimization, honestly executed, but the metamer label rests on a self-calibrated cosine-similarity threshold rather than behavioral equivalence — so treat the success counts as directional. the 4 major comments →

arxiv 2512.12307 v5 pith:WOPPOUGG submitted 2025-12-13 cs.CV cs.GR

MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding

Benjamin Beilharz , Thomas S. A. Wallis This is my paper
classification cs.CV cs.GR
keywords model metamersdifferentiable rendering3D scene understandinginverse renderingvision model interpretabilityBRDF reconstructionshape reconstructionrepresentational similarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces MRD (metamers rendered differentiably), a method that uses physically based differentiable rendering to find 3D scenes that are physically different from a target scene yet produce the same activations in a given vision model—model metamers. The authors show this works for a range of networks: under their criterion, 14 of 24 material reconstructions and 15 of 71 shape reconstructions reach the baseline-level latent similarity that defines a metamer. The pattern of successes and failures is the point: it tells which physical scene parameters—geometry (shape) or bidirectional reflectance (material)—each model is sensitive to or ignores. If the approach holds, MRD offers a physically grounded alternative to pixel-based feature visualization for probing the implicit 3D understanding of image-computable models.

Core claim

Using gradients of a physically based differentiable renderer with respect to scene parameters—vertex positions for geometry, and BRDF parameters such as roughness, metallicity, and albedo for materials—the authors optimize a scene so that its rendered images produce latent codes matching those of a target scene. They define a model metamer as a reconstruction whose cosine similarity to the target in a chosen network layer reaches or exceeds the similarity achieved by a baseline pixel-loss reconstruction. On this definition they report metameric scenes for multiple models and scene types (e.g., LPIPS and VGG for materials; LPIPS, VGG, and ResNet for some shapes) and they interpret near-basel

What carries the argument

The central mechanism is physically based differentiable rendering (PBDR) coupled with a metamerism loss. PBDR evaluates the rendering equation (a recursive integral governing light transport) and backpropagates gradients through the simulation with respect to scene parameters; the authors rely on boundary-aware gradient estimation (projective sampling) so that silhouette and shadow edges contribute stable derivatives. This lets them fix all scene properties except one (e.g., geometry or BRDF) and update the free parameters to minimize a loss between the rendered images' latent codes and those of the target images, all while keeping the network weights frozen. The metamerism criterion is set

Load-bearing premise

The entire metamer claim rests on equating 'representationally indistinguishable' with reaching the baseline-level cosine similarity in a single chosen network layer; if that baseline is too low, or if layer-level similarity does not track the model's downstream behavior, the counts and the shape-versus-material conclusions would change.

What would settle it

Re-run the baseline pixel-loss reconstruction with extra camera views and higher sample counts. If the resulting baseline similarity rises above the values used as the metamer threshold, the scenes the paper labels metamers would no longer qualify, falsifying the specific claim that those scenes are representationally indistinguishable.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The method provides a per-model, per-parameter diagnostic: for any scene property that can be parameterized in a renderer, MRD can test whether a network is invariant to it by asking whether a physically different setting yields the same latent code.
  • Because the geometry experiments show that some networks (e.g., ResNet) treat visually very different shapes as near-equivalent, the results imply those models have larger equivalence classes for shape than for material—a property that could affect robustness and transfer.
  • The observation that perceptual metrics (LPIPS, VGG) and shape-biased networks (ResNet-SIN) frequently reach baseline-level similarity suggests that representations trained with perceptual or style-transfer objectives encode physical scene parameters more directly.
  • The approach can be extended to probe other scene parameters such as lighting, camera pose, or object arrangement, since the optimization framework already separates physical causes.
  • If applied to deeper layers or to human perceptual models, MRD could generate 'semantic' metamers—different scenes that are functionally equivalent for a given concept—potentially clarifying what high-level features a model uses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An editorial extension would be to test whether the paper's metamers also pass behavioral tests (e.g., downstream classifiers or human oddity paradigms); if they do not, the method would need a stricter equivalence criterion that tracks functional behavior rather than layer-level cosine similarity.
  • Because MRD attributes equivalence to specific physical parameters, it could be used to audit generative models: if a renderer's latent space collapses distinct shapes, that may predict when generated 3D content will be perceptually or semantically ambiguous.
  • The large shape equivalence classes observed for some networks may have implications for adversarial robustness: they suggest that small geometric perturbations invisible to a human observer might be invisible to the model as well, which could either help (stability) or hurt (missed detail).
  • A testable extension is to run MRD with multiple initializations and measure the variance of metameric success; the control experiment hints that seeds cluster tightly, but a systematic analysis across seeds and scenes would establish confidence intervals for the reported counts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MRD (metamers rendered differentiably), which uses physically based differentiable rendering to optimize 3D scene parameters (object shape or BRDF) so that multi-view renderings match the latent representations of frozen vision models. The authors define a reconstruction as a model metamer if its peak cosine similarity to the target latent code reaches or exceeds the peak similarity achieved by a pixel-loss baseline run. They apply the method to two settings: material reconstruction with four BSDF classes and six models, and shape reconstruction with four meshes, three environment maps, and six models. They report 14/24 material metamers and 15/71 shape metamers, and use these counts plus RSA to draw conclusions about which networks encode shape vs. material information.

Significance. If the central claim is correct, the paper would provide a novel, physically grounded interpretability tool: reconstructing 3D scenes that are representationally indistinguishable from a target for a given network, while explicitly varying only one physical parameter class. The use of a true path-traced differentiable renderer (Mitsuba 3) and the projection-sampling treatment of geometry boundaries is technically sound, and the authors are honest in reporting many failures and marginal cases. The method is potentially useful for probing invariances of vision models and for generating physically valid test stimuli. However, the strength of the empirical conclusions—the metamer counts and the derived model sensitivities—rests entirely on the operational criterion for metamerism, and that criterion is not validated against the models' actual behavior. If the criterion is too permissive, the counts overstate model invariance; if too strict, they understate it. The contribution is therefore best viewed as a promising proof-of-concept that requires additional validation before its interpretative claims can be accepted.

major comments (4)
  1. [§2, §2.5, §3.2, §4.2] The metamer criterion is self-referential. The paper defines a successful reconstruction as one whose peak latent similarity matches the pixel-loss baseline (§2: 'we define stimuli whose latent similarity matches this reference level as metameric'), and the baseline is the authors' own optimization run using fixed hyperparameters. This threshold is not an independent benchmark, and the control in Fig. 6 only separates random meshes from reconstructions; it does not show that reaching the baseline level makes the stimulus behaviorally indistinguishable. The paper does not verify that metameric scenes yield the same labels, logits, or any task output, despite asserting in §2 that 'any downstream behavior is fully determined by these latent codes.' Since the central counts (14/24 and 15/71) are defined relative to this threshold, the qualitative conclusions about model sensitivity to shape
  2. [Tables 3 and 4] The metamer/non-metamer dichotomy is fragile. Many non-metamers are within 0.001 of the baseline (e.g., Table 4: Dragon Skybox ResNet-SIN -0.000, Lion Statue Hallstatt ResNet-SIN -0.001, Suzanne Garden ResNet -0.001), and several metamers exceed it by similarly tiny amounts (e.g., Dog Hallstatt LPIPS +0.000). No per-condition variance or confidence intervals are reported for peak similarity differences, so these classifications may reflect optimization noise rather than representational equivalence. The single-run-per-condition design (except for the seed ECDF in Fig. 6, which is averaged across all scenes) is insufficient to support the sharp thresholding used in the paper.
  3. [Table 3 and §2.6] The RSA analysis for material reconstructions is underpowered. With N=4 views, the representational similarity matrix has only 6 off-diagonal entries, and Kendall's tau on 6 pairs has extremely low statistical power; the p-values reported in Table 3 (e.g., Aurora CLIP RSA=1.000, p<0.05) are not meaningful evidence of representational geometry. The authors acknowledge at one point that material RSA is based on 4 views, but this limitation is not reflected in the interpretation of the RSA columns. The RSA-based claims about relational alignment in the material experiments should be removed or re-run with more views.
  4. [Fig. 4, Tables 3–4] The LPIPS similarity metric is never precisely defined. LPIPS is originally a distance (lower = more similar), yet the tables report high 'Top Sim' values for LPIPS (e.g., Dog Garden LPIPS 0.991). Is this negative LPIPS, 1−LPIPS, or a similarity transform? Without the definition, the quantitative comparisons involving LPIPS—which are central to the claim that perceptual losses generate metamers—are uninterpretable. The same ambiguity applies to the 'Hyperspherical similarity/LPIPS' label in Fig. 4.
minor comments (5)
  1. [§5.1] Typo: 'PBRD is an active field' should be 'PBDR'.
  2. [§2.2] The description of ResNet50-SIN should give the exact checkpoint/version (e.g., Geirhos et al. 2019) and clarify whether the weights are the released Stylized-ImageNet weights.
  3. [Table 4 caption] The columns 'Seed Null Diff' and 'Seed Base Diff' are not clearly explained. The caption mentions approximate Gaussian kernel density and normalization, but the table header is cryptic.
  4. [§2.5] The Dual-Buffer loss in Eq. (10) is introduced for translucent materials, but the text does not explain why this loss is needed beyond the reference; a brief intuition would help.
  5. [§3.2] The statement 'perceptual metrics and scale-invariant CNN features (e.g. LPIPS, ResNet-SIN) frequently reach baseline-level similarity' is not backed by a statistical comparison; consider adding a summary statistic or test.

Circularity Check

1 steps flagged

Metamerism is defined by reaching an internally calibrated baseline-similarity threshold, so the headline metamer counts are partly self-definitional; the per-model comparison retains empirical content.

specific steps
  1. self definitional [§2 'Model Metamers'; operationalized in §2.5, §3.2, §4.2 and Tables 3–4]
    "By establishing the similarity levels achieved in the baseline reconstruction run, we obtain a criterion for declaring metamers in subsequent experiments: here, we define stimuli whose latent similarity matches this reference level as metameric with respect to the model."

    A reconstruction is declared a model metamer exactly when its peak cosine similarity reaches the peak similarity of the authors' own pixel-loss baseline run (Top Sim − Top Base ≥ 0; Tables 3–4). The baseline is generated with the same renderer, scene, optimizer, and similarity metric, so it is an internal calibration, not an independent benchmark of model equivalence. The headline counts (14/24 material, 15/71 shape) are arithmetic consequences of that self-set threshold: several entries differ by 0.000–0.001, so a slightly different pixel-loss run would reclassify them. The later claim that metameric results are 'representationally indistinguishable' (e.g. §3.2) restates this operational definition; no behavioral checks (labels, logits, or held-out views) independently verify functional e

full rationale

The core rendering machinery (Mitsuba 3, Large Steps, projective sampling) is external and is not reduced to the paper's own definitions, so the method itself is not circular. I find no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result. The one notable circular element is the metamer criterion itself: the paper defines 'model metamer' as 'latent similarity matches the baseline reconstruction run' and then reports counts of metamers under that definition. This makes the counts relative to an internally calibrated threshold rather than an independent test of representational indistinguishability. The paper is transparent about this ('by our definition'), and the empirical variation across networks, the ECDF control, and the physical grounding of reconstructions provide independent content, so the circularity is partial rather than total. The main weakness is construct validity—whether cosine similarity at one layer matches the baseline implies functional equivalence—but that is a limitation of the operationalization rather than a hidden derivation of the result from its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method introduces no new physical entities or forces. The free parameters are dominated by the self-calibrated metamer threshold and the baseline-dependent hyperparameters (λ, tesselation epochs, LR); the axioms are standard physics plus the paper's specific operational choices about what 'representationally indistinguishable' means and how optimization is assumed to behave.

free parameters (4)
  • metamer threshold (baseline peak similarity) = per scene and per model, e.g. 0.983 (Dog Garden LPIPS), 0.999 (ResNet-SIN)
    The central criterion: a reconstruction is declared metameric if its peak similarity meets or exceeds the peak similarity of the authors' pixel-loss baseline run (§2.5, §3.2, §4.2). All metamer counts depend on this fitted threshold.
  • shape Laplacian regularization weight λ = 15 (Dragon, Lion Statue, Dog), 25 (Suzanne)
    Introduced by Nicolet et al. [48]; values were adjusted for Mitsuba and chosen per scene after testing the baseline (§4.1, Table 2).
  • tesselation epochs and learning-rate decay = tessellate at epochs like [5, 25, 50, 100, 150, 250, 350, 450]; LR multiply by 8e-3 after remeshing
    Remeshing epochs were 'picked after testing our baseline' (§4.1), i.e., tuned to the pixel-reconstruction run; this affects achievable similarity for the model runs.
  • learning rates, samples per pixel, early-stopping patience = e.g. LR 1e-1 (shape), 1e-1/1e-2 (materials); SPP not fully tabulated; patience 50 epochs
    Optimization hyperparameters chosen by the authors (Tables 1–2); not fitted to produce metamers but strongly influence which runs reach baseline.
axioms (5)
  • standard math The rendering equation, as approximated by Mitsuba 3's path tracer and projective-sampling integrator, provides unbiased image and gradient estimates (§2.1).
    The inverse-rendering machinery rests on this physics/math background; the paper cites [30,65,74] but does not re-derive it.
  • domain assumption Cosine similarity between ℓ2-normalized latent vectors (penultimate layer, or aggregated VGG features for LPIPS) is an appropriate measure of representational indistinguishability (§2.6).
    This definition of 'model metamer' is the paper's own operational choice; if downstream behavior diverges for same-cosine inputs, the conclusions about invariance do not transfer.
  • domain assumption Latent equivalence in the probed layer implies functional equivalence for downstream behavior (§2).
    The paper asserts 'Inputs that map to the same point... are functionally indistinguishable to the model' — an assumption about the sufficiency of one layer's geometry, not verified behaviorally.
  • domain assumption Gradient descent through the differentiable renderer reliably reaches the metameric equivalence class rather than a local minimum (§2.4, §4).
    The authors acknowledge local minima ('anamorphous blobs') and use early stopping; convergence to 'a' metamer rather than 'the' metamer is assumed.
  • domain assumption Rendered, tonemapped, sRGB-converted images are sufficiently in-distribution for ImageNet/CLIP/DINO trained networks (§2.3, §5.1).
    The paper itself lists this as a limitation: 'rendered images may represent an out-of-distribution problem for the models we tested' (§5.1).

pith-pipeline@v1.3.0-alltime-deepseek · 25502 in / 12107 out tokens · 123887 ms · 2026-08-03T16:40:11.735958+00:00 · methodology

0 comments
read the original abstract

While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.

Figures

Figures reproduced from arXiv: 2512.12307 by Benjamin Beilharz, Thomas S. A. Wallis.

Figure 1
Figure 1. Figure 1: An overview of metamers rendered differentiably (MRD). Starting from a scene with known parameters [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Top: Hyperspherical similarity between reconstructed and ground-truth latents across optimization, shown for all feature representations. These curves define the expected similarity levels obtainable when the BSDF is known, providing the reference with which later, unknown-material reconstructions are compared. Shaded regions show one standard deviation across views. Middle: Reconstructed images at selecte… view at source ↗
Figure 3
Figure 3. Figure 3: Reconstruction similarity compared to the ground-truth baseline for brushed-metal objects using VGG (top) and ResNet (bottom). [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Top: Hyperspherical similarity/LPIPS between reconstructed latents and ground-truth latent codes over the course of training. This run serves as a diagnostic reference, illustrating how rapidly and how well different network feature spaces can be aligned under our chosen optimization settings. Shaded regions show one standard deviation across views. Middle: Reconstructed images at selected iterations (5, 2… view at source ↗
Figure 5
Figure 5. Figure 5: Evaluation of reconstruction fidelity using LPIPS similarity and ResNet hypersphere similarity. We compare the convergence of our scene parameter [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The mean empirical cumulative distributions of across all experi [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space

    cs.CV 2026-05 unverdicted novelty 7.0

    MAPS provides 2618 validated 3D meshes and a controllable rendering pipeline to attribute vision model recognition failures to specific scene parameters, finding camera distance and elevation as the dominant failure f...

  2. Scene Parameter Saliency via Differentiable Light Transport

    cs.CV 2026-07 conditional novelty 4.0

    Gradients of a scalar image metric through a differentiable renderer act as metric-specific saliency maps over scene parameters, revealing which materials and surfaces drive the metric.

Reference graph

Works this paper leans on

81 extracted references · 12 canonical work pages · cited by 2 Pith papers · 5 internal anchors

  1. [1]

    Gulsum Alicioglu and Bo Sun. 2022. A Survey of Visual Analytics for Explainable Artificial Intelligence Methods.Computers & Graphics102 (Feb. 2022), 502–520. doi:10.1016/j.cag.2021.09.002

    doi:10.1016/j.cag.2021.09.002 2022

  2. [2]

    Balas, L

    B. Balas, L. Nakano, and R. Rosenholtz. 2009. A Summary-Statistic Representation in Peripheral Vision Explains Visual Crowding.Journal of Vision9, 12 (Nov. 2009), 13–13. doi:10.1167/9.12.13

    doi:10.1167/9.12.13 2009

  3. [3]

    Benjamin J. Balas. 2006. Texture Synthesis and Perception: Using Computational Models to Study Texture Representations in the Human Visual System.Vision Research46, 3 (Feb. 2006), 299–309. doi:10.1016/j.visres.2005.04.013

    work page doi:10.1016/j.visres.2005.04.013 2006

  4. [4]

    Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. 2017. Eigen-Distortions of Hierarchical Representations. InNeural Information Process- ing Systems (NIPS). 10

    2017

  5. [5]

    Lukas Boehm, Jonas Leo Mueller, Christoffer Loeffler, Leo Schwinn, Bjoern Es- kofier, and Dario Zanca. 2025. Understanding Cross-Model Perceptual Invari- ances Through Ensemble Metamers. arXiv:2504.01739 [cs] doi:10.48550/arXiv. 2504.01739

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.01739 2025

  6. [6]

    Judy Borowski, Roland S Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. 2021. Exemplary Natural Im- ages Explain CNN Activations Better than State-of-the-Art Feature Visualizations. InInternational Conference on Learning Representations

    2021

  7. [7]

    Mario Botsch and Leif Kobbelt. 2004. A Remeshing Approach to Multiresolution Modeling. InProceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing. ACM, Nice France, 185–192. doi:10.1145/1057432.1057457

    arXiv 2004

  8. [8]

    Broderick, Gizem Rufo, Jonathan Winawer, and Eero P

    William F. Broderick, Gizem Rufo, Jonathan Winawer, and Eero P. Simoncelli

  9. [9]

    Brent Burley. 2015. Extending Disney’s Physically Based BRDF with Integrated Subsurface Scattering.SIGGRAPH’15 Courses(2015)

    2015

  10. [10]

    Brent Burley and Walt Disney Animation Studios. 2012. Physically-based shading at disney. InAcm siggraph, Vol. 2012. vol. 2012, 1–7

    2012

  11. [11]

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging Properties in Self-Supervised Vision Transformers. In2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Montreal, QC, Canada, 9630–9640. doi:10.1109/ICCV48922. 2021.00951

    arXiv 2021

  12. [12]

    Alla Chaitanya, Anton S

    Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder.ACM Transactions on Graphics36, 4 (Aug. 2017), 1–12. doi:10.1145/ 3072959.3073601

    arXiv 2017

  13. [13]

    Xi Deng, Fujun Luan, Bruce Walter, Kavita Bala, and Steve Marschner. 2022. Reconstructing Translucent Objects using Differentiable Rendering. InACM SIGGRAPH 2022 Conference Proceedings

    2022

  14. [14]

    Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W

    Adrien Doerig, Rowan P. Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W. Lindsay, Konrad P. Kording, Talia Konkle, Marcel A. J. van Gerven, Nikolaus Kriegeskorte, and Tim C. Kietzmann. 2023. The Neuroconnectionist Research Programme.Nature Reviews Neuroscience24, 7 (2023), 431–450. doi:10. 1038/s41583-023-00705-w

    2023

  15. [15]

    Jonathan Dupuy and Wenzel Jakob. 2018. An Adaptive Parameterization for Efficient Material Acquisition and Rendering.ACM Transactions on Graphics37, 6 (2018)

    2018

  16. [16]

    Pavithra Elumalai, Mohammad Bashiri, Goirik Chakrabarty, Suhas Shrinivasan, and Fabian H. Sinz. 2025. Beyond Pixels: A Differentiable Pipeline for Probing 16•Beilharz et al. Neuronal Selectivity in 3D. arXiv:2510.13433 [cs.CV] https://arxiv.org/abs/2510. 13433

    arXiv 2025

  17. [17]

    Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visu- alizing higher-layer features of a deep network.University of Montreal1341, 3 (2009), 1

    2009

  18. [18]

    Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. 2019. Metamers of Neural Networks Reveal Divergence from Human Perceptual Sys- tems. InProceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA

    2019

  19. [19]

    McDermott

    Jenelle Feather, Guillaume Leclerc, Aleksander Mądry, and Josh H. McDermott

  20. [20]

    Jenelle Feather, David Lipshutz, Sarah E Harvey, Alex H Williams, and Eero P Si- moncelli. 2025. Discriminating Image Representations with Principal Distortions. InInternational Conference on Learning Representations

    2025

  21. [21]

    2023), 2017–2034

    Model Metamers Reveal Divergent Invariances between Biological and Artificial Neural Networks.Nature Neuroscience26, 11 (Nov. 2023), 2017–2034. doi:10.1038/s41593-023-01442-0

    doi:10.1038/s41593-023-01442-0 2023

  22. [22]

    Funke, Judy Borowski, Karolina Stosio, Wieland Brendel, Thomas S

    Christina M. Funke, Judy Borowski, Karolina Stosio, Wieland Brendel, Thomas S. A. Wallis, and Matthias Bethge. 2021. Five Points to Check When Comparing Visual Perception in Humans and Machines.Journal of Vision21, 3 (2021), 16. doi:10.1167/jov.21.3.16

    work page doi:10.1167/jov.21.3.16 2021

  23. [23]

    Simoncelli

    Jeremy Freeman and Eero P. Simoncelli. 2011. Metamers of the Ventral Stream. Nature Neuroscience14, 9 (Sept. 2011), 1195–1201. doi:10.1038/nn.2889

    doi:10.1038/nn.2889 2011

  24. [24]

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. 2019. IMAGENET-TRAINED CNNS ARE BI- ASED TOWARDS TEXTURE; INCREASING SHAPE BIAS IMPROVES ACCU- RACY AND ROBUSTNESS. (2019)

    2019

  25. [25]

    Gatys, Alexander S

    Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015. A Neural Algorithm of Artistic Style. arXiv:1508.06576 [cs.CV]

    Pith/arXiv arXiv 2015

  26. [26]

    Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. 2017. Unsupervised monocular depth estimation with left-right consistency. InProceedings of the IEEE conference on computer vision and pattern recognition. 270–279

    2017

  27. [27]

    Don't trust your eyes: on the (un)reliability of feature visualizations

    Robert Geirhos, Roland S. Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. 2024. Don’t Trust Your Eyes: On the (Un)Reliability of Feature Visual- izations. arXiv:2306.04719 [cs] doi:10.48550/arXiv.2306.04719

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2306.04719 2024

  28. [28]

    Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 2018. A papier-mâché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 216–224

    2018

  29. [29]

    Raju, and Nikolaus Kriegeskorte

    Tal Golan, Prashant C. Raju, and Nikolaus Kriegeskorte. 2020. Controversial Stim- uli: Pitting Neural Networks against Each Other as Models of Human Cognition. Proceedings of the National Academy of Sciences117, 47 (Nov. 2020), 29330–29337. doi:10.1073/pnas.1912334117

    doi:10.1073/pnas.1912334117 2020

  30. [30]

    Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022. Mitsuba 3 Renderer

    2022

  31. [31]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs]

    Pith/arXiv arXiv 2015

  32. [32]

    James T Kajiya. 1986. THE RENDERING EQUATION. 20, 4 (1986)

    1986

  33. [33]

    Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. 2022. Dr. Jit: A just-in-time compiler for differentiable rendering.ACM Transactions on Graphics (TOG)41, 4 (2022), 1–19

    2022

  34. [34]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Opti- mization. arXiv:1412.6980 [cs] doi:10.48550/arXiv.1412.6980

    Pith/arXiv arXiv doi:10.48550/arxiv.1412.6980 2017

  35. [35]

    Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry V. Dylov

  36. [36]

    Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Ren- dering. arXiv:2011.03277 [cs] doi:10.48550/arXiv.2011.03277

    Pith/arXiv arXiv doi:10.48550/arxiv.2011.03277 2020

  37. [37]

    Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. 2024. GS-IR: 3D Gaussian Splatting for Inverse Rendering. arXiv:2311.16473 [cs] doi:10.48550/ arXiv.2311.16473

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.16473 2024

  38. [38]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifi- cation with deep convolutional neural networks.Advances in neural information processing systems25 (2012)

    2012

  39. [39]

    Loper and Michael J

    Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differ- entiable Renderer. InComputer Vision – ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 154–169. doi:10.1007/978-3-319-10584-0_11

    doi:10.1007/978-3-319-10584-0_11 2014

  40. [40]

    Ivan Lopes, Jean-François Lalonde, and Raoul de Charette. 2024. Material Trans- forms from Disentangled NeRF Representations. arXiv:2411.08037

    Pith/arXiv arXiv 2024

  41. [41]

    Jingwang Ling, Ruihan Yu, Feng Xu, Chun Du, and Shuang Zhao. 2024. NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering. arXiv:2402.04829 [cs] doi:10.48550/arXiv.2402.04829

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.04829 2024

  42. [42]

    James Clerk Maxwell. 1857. XVIII.—Experiments on Colour, as Perceived by the Eye , with Remarks on Colour-Blindness.Transactions of the Royal Society of Edinburgh21, 2 (1857), 275–298. doi:10.1017/S0080456800032117

    work page doi:10.1017/s0080456800032117

  43. [43]

    Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2022. Text2mesh: Text-driven neural stylization for meshes. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 13492–13502

    2022

  44. [44]

    Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding Deep Image Representations by Inverting Them. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR). arXiv, Boston, MA, USA. arXiv:1412.0035 [cs] doi:10. 1109/CVPR.2015.7299155

    Pith/arXiv arXiv 2015

  45. [45]

    Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. 2022. Clip-mesh: Generating textured meshes from text using pretrained image-text models. InSIGGRAPH Asia 2022 conference papers. 1–8

    2022

  46. [46]

    Peter Neri. 2022. Deep Networks May Capture Biological Behavior for Shallow, but Not Deep, Empirical Characterizations.Neural Networks152 (2022), 244–266. doi:10.1016/j.neunet.2022.04.023

    work page doi:10.1016/j.neunet.2022.04.023 2022

  47. [47]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.arXiv:2003.08934 [cs](2020). arXiv:2003.08934 [cs]

    Pith/arXiv arXiv 2020

  48. [48]

    Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry.ACM Transactions on Graphics40, 6 (Dec. 2021), 1–13. doi:10.1145/3478513.3480501

    arXiv 2021

  49. [49]

    Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering.ACM Transactions on Graphics39, 4 (Aug. 2020), 146:146:1–146:146:15. doi:10.1145/3386569.3392406

    arXiv 2020

  50. [50]

    Anh Nguyen, Jason Yosinski, and Jeff Clune. 2019. Understanding Neural Net- works via Feature Visualization: A Survey. InExplainable AI: Interpreting, Explain- ing and Visualizing Deep Learning, Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (Eds.). Vol. 11700. Springer International Publishing, Cham, 55–76. do...

    work page doi:10.1007/978-3-030-28954-6_4 2019

  51. [51]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Lab...

    Pith/arXiv arXiv 2024

  52. [52]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, Hi...

    2019

  53. [53]

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature Visual- ization.Distill2, 11 (2017), 10.23915/distill.00007. doi:10.23915/distill.00007

    doi:10.23915/distill.00007 2017

  54. [54]

    Javier Portilla and Eero P Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients.International journal of computer vision40, 1 (2000), 49–70

    2000

  55. [55]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Nat- ural Language Supervision. arXiv:2103.00020 [cs] doi:10.48550/arXiv.2103.00020

    Pith/arXiv arXiv doi:10.48550/arxiv.2103.00020 2021

  56. [56]

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. 2022. Dreamfusion: Text-to-3d using 2d diffusion.arXiv preprint arXiv:2209.14988(2022)

    Pith/arXiv arXiv 2022

  57. [57]

    Why Should I Trust You?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). Association for Computing Machinery, New York, NY, USA, 1135–1144. doi:10.1145/2939672.2939778

    arXiv 2016

  58. [58]

    Rosenholtz, J

    R. Rosenholtz, J. Huang, A. Raj, B. J. Balas, and L. Ilie. 2012. A Summary Statistic Representation in Peripheral Vision Explains Visual Search.Journal of Vision12, 4 (April 2012), 14–14. doi:10.1167/12.4.14

    doi:10.1167/12.4.14 2012

  59. [59]

    Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. 2020. Accelerating 3D Deep Learning with PyTorch3D. arXiv:2007.08501 [cs] doi:10.48550/arXiv.2007.08501

    Pith/arXiv arXiv doi:10.48550/arxiv.2007.08501 2020

  60. [60]

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra. 2017. Grad-Cam: Visual Explanations from Deep Networks via Gradient-Based Localization. InProceedings of the IEEE Inter- national Conference on Computer Vision. 618–626

    2017

  61. [61]

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv:1312.6034 [cs] MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding•17

    Pith/arXiv arXiv 2014

  62. [62]

    Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. 2022. Clip-forge: Towards zero-shot text-to-shape generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18603–18613

    2022

  63. [63]

    1998.Robust Monte Carlo methods for light transport simulation

    Eric Veach. 1998.Robust Monte Carlo methods for light transport simulation. Stanford University

    1998

  64. [64]

    V. S. R. Veeravasarapu, Constantin Rothkopf, and Ramesh Visvanathan. 2017. Adversarially Tuned Scene Generation. arXiv:1701.00405 [cs] doi:10.48550/arXiv. 1701.00405

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1701.00405 2017

  65. [65]

    Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556(2014)

    Pith/arXiv arXiv 2014

  66. [66]

    Thomas S. A. Wallis, Matthias Bethge, and Felix A. Wichmann. 2016. Testing Models of Peripheral Encoding Using Metamerism in an Oddity Paradigm.Journal of Vision16, 2 (March 2016), 4. doi:10.1167/16.2.4

    work page doi:10.1167/16.2.4 2016

  67. [67]

    Thomas S. A. Wallis, Christina M Funke, Alexander S Ecker, Leon A Gatys, Felix A Wichmann, and Matthias Bethge. 2019. Image Content Is More Important than Bouma’s Law for Scene Metamers.eLife8 (April 2019), e42512. doi:10.7554/eLife. 42512

    doi:10.7554/elife 2019

  68. [68]

    Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backprop- agation: Differentiating Light Paths Using Constant Memory and Linear Time. ACM Transactions on Graphics40, 4 (July 2021), 108:1–108:14. doi:10.1145/3450626. 3459804

    doi:10.1145/3450626 2021

  69. [69]

    Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV). 52–67

    2018

  70. [70]

    Z Wang and Eero P Simoncelli. 2008. Maximum Differentiation (MAD) Com- petition: A Methodology for Comparing Computational Models of Perceptual Quantities.Journal of Vision8, 12 (2008), 8

    2008

  71. [71]

    David R Walton, Rafael Kuffner Dos Anjos, Sebastian Friston, David Swapp, An- thony Steed, and Tobias Ritschel. 2021. Beyond Blur: Real-Time Ventral Metamers for Foveated Rendering.SIGGRAPH(2021), 14

    2021

  72. [72]

    Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering.ACM Transactions on Graphics39, 4 (Aug. 2020), 143:143:1–143:143:19. doi:10.1145/3386569.3392383

    arXiv 2020

  73. [73]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang

  74. [74]

    Wichmann and Robert Geirhos

    Felix A. Wichmann and Robert Geirhos. 2023. Are Deep Neural Networks Ade- quate Behavioral Models of Human Visual Perception?Annual Review of Vision Science9, 1 (2023), 501–524. doi:10.1146/annurev-vision-120522-031739

    doi:10.1146/annurev-vision-120522-031739 2023

  75. [75]

    Ziyi Zhang, Nicolas Roussel, and Wenzel Jakob. 2024. Many-Worlds Inverse Rendering. arXiv:2408.16005 doi:10.48550/arXiv.2408.16005

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.16005 2024

  76. [76]

    Tingting Zhu, Bo Peng, Jifan Liang, Tingchen Han, Hai Wan, Jingqiao Fu, and Junjie Chen. 2024. How to Evaluate Semantic Communications for Images with ViTScore Metric? arXiv:2309.04891

    Pith/arXiv arXiv 2024

  77. [77]

    Roland Simon Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. 2021. How Well Do Feature Visu- alizations Support Causal Understanding of CNN Activations?. InThirty-Fifth Conference on Neural Information Processing Systems (NeurIPS). 18•Beilharz et al. Table 4. Shape reconstruction results across all sha...

    2021

  78. [78]

    Ziyi Zhang, Nicolas Roussel, and Wenzel Jakob. 2023. Projective Sampling for Differentiable Rendering of Geometry.ACM Transactions on Graphics42, 6 (Dec. 2023), 1–14. doi:10.1145/3618385

    doi:10.1145/3618385 2023

  79. [2018]

    arXiv:1801.03924 [cs] doi:10.48550/arXiv.1801.03924

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv:1801.03924 [cs] doi:10.48550/arXiv.1801.03924

    Pith/arXiv arXiv doi:10.48550/arxiv.1801.03924

  80. [2022]

    arXiv:2208.14818 [eess] doi:10.48550/arXiv.2208.14818

    PyTorch Image Quality: Metrics for Image Quality Assessment . arXiv:2208.14818 [eess] doi:10.48550/arXiv.2208.14818

    Pith/arXiv arXiv doi:10.48550/arxiv.2208.14818

Showing first 80 references.