{"id":"0509da81-85ce-4c40-b781-705f8f25e499","arxiv_id":"2411.16148","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A new probing method renders intermediate DNN features as graphics elements and claims evidence for a 2D-2.5D-3D construction sequence in face perception networks.","lead":"This paper introduces a probing method that decodes internal features of a neural network into depth, lighting, and viewpoint, then renders the input image back. The authors claim the decoded geometry shows networks build 2D, then 2.5D, then 3D representations, matching David Marr's theory of vision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2D→2.5D→3D progression may be an artifact of the probe's increasing receptive field and the reconstruction objective; a raw-pixel control is missing.","rationale":"The reader's weakest_assumption identifies the same load-bearing vulnerability: the flat low-level depth map may reflect the probe's limited receptive field and the reconstruction training objective rather than a genuine 2D representation. My stress-test concurs and sharpens the point. The paper's evidence—cross-architecture consistency in Table 1 and training dynamics in Figure 4—is suggestive but does not resolve the confound, because those architectures also have increasing receptive fields and are trained under the same reconstruction loss. The single-viewpoint experiment in Section 4.6 actually strengthens the concern by showing that 3D emergence is driven by viewpoint variation in the data, making the claim task-dependent. A raw-pixel or fixed-receptive-field control is the minimal experiment that would distinguish a Marr-like internal progression from a trivial consequence of the probing setup. Without such a control, the central claim remains unsupported, so the REJECT verdict is appropriate and unchanged.","tokens_in":14026,"tokens_out":3472,"duration_ms":33840,"concrete_test":"Run the graphics probe pipeline on a null input: feed raw image patches (or a randomly initialized, untrained WinT) at the same three receptive-field sizes (1/4, 1/2, full image) through the identical probe decoders and reconstruction loss. If the decoded depth-variance profile reproduces the 2D→2.5D→3D pattern, the progression is an artifact of receptive field size and the objective, not of learned internal representation. As a complementary check, train a WinT variant with full-image receptive fields at all layers under the same loss; if low-level depth becomes non-flat (3D-like), the original low-level flatness is caused by limited receptive field.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that DNNs encode 2D, then 2.5D, then 3D representations across layers—rests on interpreting the graphics probe's decoded depth as a faithful readout of internal geometry. This interpretation is undercut by a confound: the probe's receptive field grows from 1/4 to 1/2 to full image across the three probed levels (Section 4.2, Figure 3), and the backbone is trained from scratch with an unsupervised reconstruction loss (Section 4.1, Eq. 5). At the low level, each probe sees only a 1/4-image window; with no global context, a flat depth map plus albedo/texture is the most parsimonious way to reconstruct local appearance. At the middle level, 1/2-image windows allow local normal variation but not global depth. At the high level, full-image context makes full 3D recovery possible. The observed depth-variance pattern (flat → shallow depth/high normal variance → deep) is therefore exactly what the receptive-field hierarchy would predict, without invoking Marr stages. The paper provides no control applying the same probe to raw pixels or to a network with full-image receptive fields at all layers. Section 4.6 further shows that 3D emergence depends on viewpoint variation in training data, indicating the progression is task-driven rather than a universal property of DNNs. The 3D threshold of 15×10^-3 in Figure 8 is also set post hoc without justification. Thus the paper's headline claim is not supported by the current evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a \"graphics probe,\" a sub-network inserted into intermediate layers of a window-based transformer, that decodes the layer's features into depth, albedo, camera viewpoint, and lighting, and then re-renders the input image via a differentiable renderer. The authors train the network and probes jointly with a reconstruction loss and report that low-level layers decode flat depth maps (2D), middle layers decode shallow depth with strong normal variation (2.5D), and high layers decode full 3D geometry. They also report that middle-level probes are view-tuned while high-level probes are part-tuned, and that a single-view training set prevents 3D emergence. The paper interprets these observations as evidence that deep neural networks implement Marr's 2D-2.5D-3D sequence.","tokens_in":14335,"tokens_out":5256,"duration_ms":50270,"significance":"If the central claim is established, the paper would provide a concrete empirical bridge between DNN internal representations and Marr's classical framework, with implications for interpretability, face perception, and the conditions under which 3D representations emerge. The graphics probe is an original and potentially useful tool, and the paper includes several positive features: quantitative depth/normal variance distributions, a controlled single-view training experiment (Section 4.6), and cross-architecture replication (Section 4.7). However, the central claim is not yet supported because of two confounds: the probed levels have different receptive-field sizes, and the probes are trained jointly with the network rather than applied as readouts of a frozen representation. The paper would need additional controls to substantiate the strong Marr-alignment conclusion.","major_comments":[{"comment":"The central 2D-2.5D-3D claim is confounded by receptive-field size. The probed stages use window sizes of 1/4, 1/2, and the full image, as stated in Section 3.1 and Table 1 of the supplement. A probe seeing only 1/4 of the image cannot estimate global 3D structure, and a flat depth map plus albedo/texture is the most parsimonious way to reconstruct a local patch. The observed monotonic increase in depth variance with level is therefore exactly what the architecture predicts, independent of whether the layer itself stores a 3D representation. To support the claim, the authors should provide a control that applies the same graphics probe to raw pixels at the same three receptive-field sizes, or that probes features at all layers with a fixed full-image receptive field. Without such a control, the low-level '2D' result cannot be attributed to the representation rather than to limited context.","section":"Section 3.1, Section 4.2, Figure 3"},{"comment":"The network and the graphics probes are trained jointly with the reconstruction loss in Eq. (5). The decoded depth, albedo, and viewpoint are optimized parameters of the same learning process, not independent readouts of a pre-existing representation. The observation that high-level layers 'build 3D' may reflect the probe's role in explaining viewpoint variation, and Section 4.6 itself shows that 3D depth emerges only when training data contain pose variation. This makes the finding task-dependent rather than a general property of DNNs. The authors should probe a pretrained, fixed network (for example, a face-recognition backbone) or train a linear probe on frozen features, and compare against the joint-training result.","section":"Section 4.1, Eq. (5), Section 4.6"},{"comment":"The 3D threshold of 15x10^-3 in Figure 8 is introduced post hoc and without justification or sensitivity analysis. The claim that the single-view model 'does not reach the threshold necessary for 3D geometry' is not quantitative evidence unless the threshold is derived independently from the data or from a principled criterion. The authors should report the full distributions of depth variance for both models, include a statistical comparison, and either justify the threshold or replace it with a continuous measure that does not rely on a hand-set cutoff.","section":"Section 4.6, Figure 8"},{"comment":"The cross-architecture results in Table 1 are not accompanied by error bars or significance tests, and the low-level VGG16 and ResNet18 depth variances (0.21 and 0.26 x10^-3) are nonzero. The supplementary text acknowledges these maps are 'not entirely flat' and reclassifies them as 2D because they lack semantic structure. This reclassification makes the 2D/2.5D/3D taxonomy difficult to falsify. The authors should formalize the criteria for each stage and apply them consistently, or report the classification as a categorical judgment rather than a quantitative finding.","section":"Section 4.7, Table 1, Supplementary Section 3"}],"minor_comments":[{"comment":"The abstract contains the typo 's sur normals' and the conclusion contains 'phenomenons'; both should be corrected.","section":"Abstract and Conclusion"},{"comment":"In Eq. (3), the concatenation list repeats 'theta_v_k' twice; the second entry should be 'theta_l_k'. Also, 'theta_b_k' is used for the albedo component but the symbol 'b' is not defined.","section":"Eq. (3)"},{"comment":"Section 4.2 states that lower layers with 1/8 image receptive fields were investigated, while Section 3.1 and the supplementary Table 1 state that the bottom stage is not probed. Please clarify which stages were actually probed.","section":"Section 4.2 vs. Section 3.1"},{"comment":"Figure 4 shows distributions and mean variations without confidence intervals or sample sizes; adding error bars or box plots would strengthen the quantitative comparison.","section":"Figure 4"},{"comment":"The hardmax operation in Eq. (2) is not defined; please specify whether it is a one-hot argmax over each row and discuss its effect on gradient flow.","section":"Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline claim is stronger than the evidence supports: the result is currently about a jointly trained reconstruction model with increasing receptive fields, not about natural DNN representations. However, the confounds are addressable with additional control experiments (raw-pixel probes, fixed-receptive-field probing, and frozen-network probing), so I do not see this as an unfixable rejection. If the authors can supply those controls, the paper could be a useful contribution to interpretability; otherwise, the framing should be substantially narrowed to a method demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know before you read it: the graphics probe is a genuinely new and flexible interpretability tool, but the paper's headline claim—that DNNs build 2D, then 2.5D, then 3D representations—is not actually supported by the current experiments, for a fairly structural reason.\n\nWhat it does well: the probe itself is a nice idea. It decodes intermediate features into depth, albedo, camera, and lighting, and renders them back to the image through a differentiable renderer. That is a creative and potentially reusable way to visualize what a layer encodes. The mid-level \"low-relief\" result—rich normal variation within a narrow depth band—is genuinely evocative, and the view-direction shift from viewer-centered to object-centered is a thoughtful observation. They also test several architectures (VGG, ResNet, Swin, ViT) and report consistent depth/normal variance patterns, and they include a single-view training control in Section 4.6 that shows 3D does not emerge without viewpoint variation. Those are real contributions.\n\nThe soft spots are not minor. First, this is not probing in the usual sense: the network is trained from scratch with the reconstruction loss, and the probes are the output heads. So the depth maps are fitted values, shaped by the probe and the loss, not neutral readouts of a pre-existing representation. Second, the receptive-field confound is exactly as bad as it looks. The low-level probe sees a 1/4-image window, the mid-level sees 1/2, the high-level sees the full image. A flat depth map is the parsimonious way to reconstruct a local texture patch; full-image context is what makes full 3D recovery possible. Without a control that applies the same probe to raw pixels or to a network with matched receptive fields at all layers, the observed depth variance progression is exactly what the architecture predicts, and Marr doesn't need to be invoked. The 15e-3 threshold in Figure 8 is set post hoc, and the single-view result actually undercuts the universal claim by showing the progression is task-driven.\n\nThe abstract says \"DNNs\" do this; the evidence is about reconstruction-trained face networks. That overgeneralization is not fatal to the tool, but it is fatal to the paper's title claim as currently written.\n\nBottom line: the tool deserves a serious referee and a revision that adds controls. I would not cite the Marr result in its current form, but I'd happily discuss the method at a reading group. My recommendation: send to review, ask for raw-pixel and fixed-network controls, and a more careful framing.","headline":"A clever and reusable graphics-probe tool, but the headline 2D–2.5D–3D result is confounded by the probe's growing receptive field and by training the network to reconstruct through the probes.","tokens_in":14898,"tokens_out":3173,"would_cite":false,"duration_ms":29814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep face networks construct representations in the order Marr's theory predicts: flat 2D, low-relief 2.5D, then full 3D.","keywords":["Marr's theory of vision","2D sketch","2.5D sketch","3D model","graphics probe","deep neural network interpretability","face perception","analysis-by-synthesis"],"falsifier":"Train the same Window Transformer with a classification head instead of the reconstruction loss and re-run the graphics probes; if the 2D–2.5D–3D depth pattern vanishes, the sequence is a product of the reconstruction objective rather than of deep perception itself.","tokens_in":13826,"feed_emoji":"🧠","tokens_out":6851,"duration_ms":57880,"temperature":0.7,"pith_summary":"This paper tries to establish that deep neural networks trained to reconstruct or perceive faces organize their internal representations according to the classic 2D–2.5D–3D construction theory of vision. The authors insert a graphics probe into intermediate layers; the probe reads out what a layer encodes by rendering the input image from the layer's features using depth, albedo, camera view, and lighting. Readouts show flat geometry with no depth variation in low-level layers, shallow depth with rich surface-normal variation in middle layers, and full 3D shape at high layers. The same probing reveals mid-level features are tuned to viewpoints while high-level features are tuned to facial parts, and training on single-view faces prevents the 3D stage from appearing. If correct, the paper gives empirical support for a much-debated stage theory inside deep learning models, and a tool for seeing what those models actually compute.","feed_headline":"Face networks build 2D, 2.5D, then 3D representations","feed_subtitle":"Decoded depth maps show hidden layers building flat 2D, then low-relief 2.5D, then full 3D.","key_machinery":"The graphics probe is the object that carries the argument: a probe token is inserted into each window of a transformer or appended to a CNN block, gathers information from the layer's tokens through self-attention, and is decoded into four computer-graphics elements—a depth map, an albedo map, a 6DoF camera view, and lighting. The probe is trained unsupervised, by analysis-by-synthesis, to reconstruct the input image from those elements. Because the probe can produce either flat or sculpted depth and either viewer-centered or object-centered views, the choices it settles on expose the layer's preferred representation; multiple probes compete through a Z-buffer, revealing which probe is tuned to which image or face part. The depth map and its statistics, depth variation and normal variation, are the direct evidence for classifying a layer as 2D, 2.5D, or 3D.","core_discovery":"On the paper's own terms, the central discovery is that a vision-oriented deep network spontaneously develops the three representational stages proposed in classical computational vision. Low-level layers encode a genuinely 2D representation, as shown by decoded depth maps that are flat planes with normals pointing straight at the viewer. Middle layers build a 2.5D representation: depth remains shallow but surface normals vary enough to create shading, like a low-relief sculpture. High layers construct a full 3D model, with depth variations large enough to represent three-dimensional shape from any viewpoint. The same three-stage pattern appears across several architectures, and the final 3D stage only emerges when training data contain viewpoint variation.","pith_inferences":["If the flat low-level readout is a true representation rather than an artifact of the probe's receptive field, then removing the reconstruction objective should preserve flatness; this is a testable ablation the paper does not run.","The sudden jump in high-level depth variance around epoch 15 suggests a phase-transition-like moment in training; probing per epoch could reveal whether the 3D stage appears abruptly or gradually.","The same graphics-probe method could be applied to non-face object categories to test whether the 2D–2.5D–3D ladder is a general property of deep visual representation or specific to faces with strong pose variation.","Because training on single-view faces still yields a 2.5D shading representation, the 2.5D stage may be a minimal requirement for any reconstruction-based network, while full 3D is an extra adaptation to viewpoint diversity."],"forward_implications":["Low-level layers of trained face networks should be read as flat 2D sketches: probing them yields no depth variation and no normal variation.","Middle layers should be read as 2.5D low-relief representations: they produce shading through surface normals while keeping depth nearly constant.","High-level layers should be read as full 3D object-centered models, with depth variation large enough to render profile views of the face.","Viewpoint variation in the training data is a prerequisite for the 3D stage; networks trained on a single yaw angle stay in the 2.5D regime.","The 2D–2.5D–3D ordering generalizes across VGG16, ResNet18, ViT, SwinT, and the proposed Window Transformer."],"supporting_citations":[{"why":"Supplies the 2D–2.5D–3D theory that the paper sets out to test.","marker":"[1]"},{"why":"Provides the differentiable renderer that lets reconstructed images be compared with the input for training.","marker":"[37]"},{"why":"The Swin transformer architecture that the probed Window Transformer modifies and that is probed as another architecture.","marker":"[35]"},{"why":"The Vision Transformer architecture probed to show the results generalize beyond window-based transformers.","marker":"[36]"},{"why":"The unconstrained face dataset (CelebA) used for training and testing the probes.","marker":"[39]"},{"why":"The laser-scanned face dataset rendered at 13 yaw angles, providing the viewpoint variation needed for the experiments.","marker":"[40]"},{"why":"Precedent for 2.5D-sketch-then-3D reconstruction from a single image, which the probing results connect to.","marker":"[29]"},{"why":"The analysis-by-synthesis reconstruction loss used to train the graphics probe without explicit 3D supervision.","marker":"[38]"},{"why":"Prior work on hierarchical 3D face representation from 2D images, which the template and probe design extends.","marker":"[26]"}],"fun_headline_variants":["Deep nets echo Marr: 2D to 3D in three steps","Neural networks follow Marr's vision stages","DNNs build flat, relief, then full 3D","Marr's theory seen in deep vision nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats the decoded depth maps as faithful readouts of what each layer actually encodes, rather than as artifacts of the probe's receptive-field size or its reconstruction objective.","fun_headline_variants_meta":{"raw":{"variants":["Deep nets echo Marr: 2D to 3D in three steps","Neural networks follow Marr's vision stages","DNNs build flat, relief, then full 3D","Marr's theory seen in deep vision nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1200,"prompt_tokens":1016,"completion_tokens":184,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":114}},"tokens_in":632,"tokens_out":184,"duration_ms":2653,"temperature":1.0,"reasoning_tokens":114,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:29:19.396377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same Window Transformer with a classification head instead of the reconstruction loss and re-run the graphics probes; if the 2D–2.5D–3D depth pattern vanishes, the sequence is a product of the reconstruction objective rather than of deep perception itself.","supporting_citations":[{"cited_title":"Dosovitskiy, L","cited_arxiv_id":null,"evidence_quote":"Provides the differentiable renderer that lets reconstructed images be compared with the input for training."},{"cited_title":"Degenerate Swin to Win: Plain Window-based Transformer without Sophisticated Operations","cited_arxiv_id":"2211.14255","evidence_quote":"The Swin transformer architecture that the probed Window Transformer modifies and that is probed as another architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The unconstrained face dataset (CelebA) used for training and testing the probes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The laser-scanned face dataset rendered at 13 yaw angles, providing the viewpoint variation needed for the experiments."},{"cited_title":"Grossman, G","cited_arxiv_id":null,"evidence_quote":"Precedent for 2.5D-sketch-then-3D reconstruction from a single image, which the probing results connect to."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The analysis-by-synthesis reconstruction loss used to train the graphics probe without explicit 3D supervision."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on hierarchical 3D face representation from 2D images, which the template and probe design extends."}],"review_version":1}