{"id":"98eaa067-5daf-43f5-8245-1a30dde8ae0b","arxiv_id":"2602.03531","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Masked Autoencoders build class-separable representations across depth and keep their embeddings directionally stable under blur and occlusion, which tracks their robust classification.","lead":"This paper analyzes how Masked Autoencoders organize class information in their internal representations and why they stay accurate under blur and occlusion. It finds class-separable structure emerging in deeper layers and introduces two metrics for measuring representation robustness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unreported rank k in the subspace and feature-retention analyses makes Figures 4, 6–8 non-reproducible; a k-sweep is needed to validate the central claims.","rationale":"The reader's weakest_assumption identified the unreported rank k in Section III-A3 as a key gap, and the manuscript text confirms this: no k is given in the subspace analysis or in the feature-retention analysis. This is load-bearing because the central quantitative claims—progressive class-subspace divergence and head-wise feature retention—are expressed as figures derived from these analyses. Without the value of k, the reader cannot verify the trends, and the analysis is not reproducible. The concrete test I propose directly checks whether the qualitative trend is robust to k. If it is, the concern lands only as a reporting deficiency; if it is not, the central structural claim collapses. I do not think this moves the verdict from CONDITIONAL—it is the same verdict the reader reached—but it sharpens the condition: the authors must report k and demonstrate stability over a range. My agreement_with_reader is 'agree' because we identify the same weakest assumption, although I also emphasize that the feature-retention analysis suffers from the same hidden k, reinforcing the need for a sweep.","tokens_in":10913,"tokens_out":6101,"duration_ms":65870,"concrete_test":"Reproduce the Section III-A3 principal-angle analysis on the same 10-class, 50-image subset for k ∈ {1, 2, 5, 10, 20, 50, 100, 200, 500}. Generate the Figure 4 box plots for each k. The central claim is supported only if the median θ1 increases monotonically with layer depth for a wide range of k (or for a clearly justified, pre-specified k). If the trend disappears or reverses for any intermediate k (e.g., k=100), the conclusion that class subspaces 'systematically rotate away' is not robust. Additionally, perform the analogous k-sweep for the feature-retention heatmaps (Figs. 6–8); if the retention pattern depends strongly on the chosen top-k, the proposed robustness indicator is not well-defined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The principal quantitative evidence for the central structural claim—that class-specific subspaces rotate apart with depth—is Figure 4, computed per Section III-A3. That section defines each class subspace using 'the top-k singular vectors' but never states the value of k. This matters because principal angles between k-dimensional subspaces in R^768 change sharply with k: for k large enough that 2k > 768, generic subspaces may intersect, making the smallest principal angle θ1 = 0 and destroying the increasing box-plot trend; for small k, the angles are noisy and may not reflect class structure. The same omission occurs in Section III-B2 for the feature-retention indicator: 'top-k active features' appears only in an illustrative example ('k=10'), and the actual k used for Figures 6–8 is not reported. Without k (or a k-sweep), Figures 4, 6, 7, and 8 are not reproducible, and the trends they display could be artifacts of a single favorable k. The qualitative t-SNE visualizations (Figs. 1–2) are suggestive, but the paper's contribution 2 (a 'geometric characterization') rests on an unreported hyperparameter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes a pretrained MAE ViT-B/16 on ImageNet and claims that (i) the pretrained encoder progressively builds class-separable structure across depth, visible in CLS, mean-patch, and raw patch token embeddings; (ii) class-specific subspaces rotate apart with depth; (iii) after fine-tuning, MAE remains accurate under Gaussian blur and attention-guided occlusion, with supporting ImageNet-C results; and (iv) two proposed indicators—directional alignment and head-wise active-feature retention—quantify representation robustness and track classification performance. The evidence includes t-SNE visualizations, attention-distance measurements, principal-angle box plots, accuracy tables, cosine-similarity tables, and feature-retention heatmaps.","tokens_in":11196,"tokens_out":5819,"duration_ms":66061,"significance":"If substantiated, this would be a useful empirical contribution to MAE interpretability, especially the layer-wise subspace geometry and the attention-guided occlusion protocol. The paper uses a standard public architecture, evaluates on a broad corruption set (ImageNet-C), and introduces two simple, interpretable robustness indicators. However, the central geometric claim and the quantitative robustness results are currently under-specified: the analysis depends on unreported rank/feature-count hyperparameters, the robustness comparison lacks any baseline model, and the accuracy numbers have no error bars. These gaps are fixable but need to be addressed before the paper can support its conclusions.","major_comments":[{"comment":"The subspace rank k is never reported. Section III-A3 says class subspaces retain 'the top-k singular vectors' and calls the restriction 'more for convenience of analysis,' but no k is given for Figure 4. This is load-bearing because principal angles between k-dimensional subspaces of R^768 are highly sensitive to k: for 2k > 768 generic subspaces intersect, forcing the smallest principal angle theta_1 to 0, while for very small k the angles are noisy. The observed increasing trend could be an artifact of one favorable k. Please report k and provide a k-sweep (e.g., k = 1, 2, 4, 8, 16, 32, 64) showing that Figure 4's trend is stable. The same problem affects the feature-retention analysis: in Section III-B2, 'top-k active features' is only illustrated with k=10, and the actual k used for Figures 6–8 is not stated, making those heatmaps non-reproducible. A sensitivity analysis for the 60%","section":"III-A3, III-B2, IV-A2, Figs. 4, 6–8"},{"comment":"The paper repeatedly contrasts MAE with 'ViTs' and concludes that MAE maintains robust classification performance, but no supervised ViT or other SSL baseline is evaluated under the same perturbations. Tables I–II and Figure 5 report absolute accuracies for a single fine-tuned MAE on ten ImageNet classes. Without a supervised ViT-B/16 (or another SSL method) trained and evaluated under the identical blur/occlusion/ImageNet-C protocol, the comparative claim 'robust compared to ViTs' is unsupported. The authors should either add such baselines or explicitly weaken the claims to state that MAE retains high absolute accuracy under these degradations.","section":"Introduction, IV-B, V"},{"comment":"No error bars, confidence intervals, repetition counts, or seeds are reported for the accuracy numbers. The evaluation appears to use 10 classes with 50 images each (approximately 500 images), so adjacent accuracy values such as 87.6% vs. 87.4% in Table I are within binomial sampling error. The claim that accuracy 'remains stable' across a wide perturbation range needs support from bootstrap confidence intervals or multiple evaluation runs; otherwise the observed stability may be an artifact of a small sample.","section":"IV-B, Tables I–II, Fig. 5"},{"comment":"The classifier used to produce the accuracy results is under-specified. The text says predictions are made 'based on mean patch embedding \\bar z^{(l)}' but does not state which layer l is used, how the classifier is trained, what head architecture is used, or whether the fine-tuned MAE model is taken from the original MAE repository with its default classification head. Without these details, Tables I–II and Figure 5 cannot be reproduced, and the claimed link between representational robustness and classification accuracy cannot be independently checked.","section":"IV-B"}],"minor_comments":[{"comment":"The class subsets and image sampling are not fully specified: Section IV-A1 uses 90 images from three ImageNet-1K classes, while Section IV-A2 uses ten classes with 50 images per class, but the class identities and random seed are not reported. Please provide them for reproducibility.","section":"III-A3, IV-A1, IV-A2"},{"comment":"Directional robustness, as defined by Eq. (1), is essentially a statement that perturbed embeddings remain in the same direction as clean embeddings. Calling this 'robustness' is partly a restatement of the metric. The paper should soften this framing and rely more on the classification-accuracy link, which is currently only anecdotal (the correct-vs-misclassified cosine comparison in Section IV-C has no sample sizes or statistical test).","section":"III-B2, IV-C"},{"comment":"There are several typos and infelicities: 'prinicpal angles' (III-A3), 'covenience' (III-A3), 'perturbated' (V), and 'Imagenet-1K' should be 'ImageNet-1K' for consistency. Please also clarify in Section III-A1 whether the stated masking ratio (0.75, N_v=49) applies to pretraining only or also to inference/perturbation experiments.","section":"Throughout"},{"comment":"The attention-distance figure for standard ViT is reproduced or adapted from [9]. Please confirm that permission/reproduction guidelines are followed and clearly state how attention distances are computed for MAE when 'no masking' is used, since the pretrained MAE was trained with 75% masking.","section":"IV-A1, Fig. 3"},{"comment":"No code or model checkpoints are provided. Given the number of unstated hyperparameters (k, top-k, thresholds, seeds), releasing evaluation code would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the unreported rank k is well-founded and is the main technical obstacle to the paper's central geometric claim. The missing baselines and missing error bars are also significant but addressable within the scope of a revision. I do not see grounds for rejection if the authors provide the requested k-sweep, baseline comparisons, and statistical uncertainty quantification; without them, the paper is not yet reproducible enough for a journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: if the layer-wise subspace-separation result holds, it is a genuinely useful addition to the MAE-interpretability literature, and the head-wise feature-retention analysis is a nice complement that goes beyond cosine similarity. The paper is clearly written, internally consistent, and honest about what is already known — the random-masking invariance finding explicitly defers to [20], [21]. Credit where due: the qualitative t-SNE story, the early-global-attention observation with the attention-distance plots, and the ImageNet-C robustness table are solid, reproducible-in-spirit pieces of work. The subspace trend (Figure 4) is the real novelty, and it is plausible.\n\nThe soft spots are real but fixable. First is the unreported rank k in the subspace analysis and in the top-k active feature retention. The paper literally says 'top-k' and never gives k for Figures 4, 6–8; the illustrative k=10 in the text is not tied to the experiments. That makes the central quantitative claims non-reproducible, and as the stress-test notes, the principal-angle trend can change sharply with k. A k-sweep is needed. Second, the robustness section compares MAE's behavior 'compared to ViTs' in the framing, but no supervised ViT or other SSL baseline appears in the experiments — the comparison is implicit. Third, 10 classes and 50 images per class with no error bars is a thin base for strong claims, though the consistency with the t-SNE and attention-distance results mitigates that. Fourth, the directional robustness metric is admittedly close to circular, but the paper does not lean on it alone; the accuracy curves and feature-retention counts carry the argument, so I would call that a minor concern, not a fatal one. Also, no code/data artifacts are provided, which compounds the k problem.\n\nWho is this for? People working on SSL interpretability, robustness of vision transformers, and anyone trying to understand why MAE fine-tunes well. The paper deserves a serious referee — conditional acceptance with requests for a k-sweep, a ViT baseline, error bars, and code. It is not ready as-is, but the core observations are worth engaging with.","headline":"Worth a serious referee, but only after the authors disclose the subspace rank k and add baselines; the central class-separation trend is plausible but currently rests on unreported knobs.","tokens_in":11649,"tokens_out":565,"would_cite":false,"duration_ms":7815,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that masked autoencoders develop class-separable latent geometry during label-free pretraining, and that this geometry is why fine-tuned MAE stays accurate under blur and occlusion.","keywords":["Masked Autoencoders","representation learning","self-supervised learning","robustness","subspace analysis","principal angles","attention mechanisms","Vision Transformer"],"falsifier":"Repeat the principal-angle analysis at several values of k (for example, 1, 10, 50, 100, and 200) on the same 10 classes; if the layer-wise increase in the smallest principal angle is not monotonic or disappears for some k, the claim that class subspaces progressively diverge is an artifact of the chosen k. A second decisive test: run the same subspace analysis on all 1000 ImageNet classes or on a larger per-class sample; if the trend flattens, the 10-class result does not reflect global MAE structure.","tokens_in":10795,"feed_emoji":"🖼️","tokens_out":3932,"duration_ms":39138,"temperature":0.7,"pith_summary":"The paper asks why Masked Autoencoders, trained without labels to reconstruct missing image patches, classify so well and remain accurate when images are degraded. It argues that the MAE encoder progressively builds class-aware structure during pretraining: token embeddings for different classes become increasingly separable with depth, and their class-specific subspaces rotate apart. After fine-tuning, MAE keeps stable accuracy under Gaussian blur and attention-guided occlusion because its latent representations stay directionally aligned and its attention heads preserve their active features, until perturbations become extreme. The paper offers two quantitative sensitivity indicators that connect representation stability to classification robustness, giving a mechanistic explanation for a behavior previously observed only empirically.","feed_headline":"Masked autoencoders learn class structure without labels","feed_subtitle":"Layer-wise subspace analysis ties MAE's strong accuracy under blur and occlusion to a latent space that barely changes under perturbation.","key_machinery":"The central objects are class-specific subspaces built by SVD of patch-token matrices for each class at each encoder layer; principal angles between pairs of subspaces quantify how far classes rotate apart with depth. The robustness analysis rests on two indicators: cosine similarity between clean and perturbed mean-patch embeddings, and a common-feature count that tracks how many of the top-k active dimensions per attention head survive perturbation. Together they map accuracy changes to representation-level changes, and attention-distance measurements across layers explain why global context is available from the start.","core_discovery":"On the paper's own terms: a pretrained MAE encoder organizes its latent space by class even though it never saw labels. Class clusters appear in CLS, mean-patch, and raw patch token embeddings around the seventh through ninth layers and grow stronger in deeper layers. SVD-based class subspaces diverge progressively with depth, and MAE attention is global from the first layer, unlike standard Vision Transformers. After fine-tuning, top-1 accuracy stays above 80 percent under heavy blur and above 60 percent even when 90 percent of the most attended patches are occluded. The two proposed indicators—cosine alignment between clean and perturbed mean-patch embeddings, and per-head retention of com","pith_inferences":["The subspace-divergence measure could be run on other masked-image models to test whether class-separable geometry is unique to MAE or a general property of masked reconstruction; the paper does not make that comparison.","If directional alignment is a reliable proxy for correctness, the same cosine-similarity indicator could serve as a test-time uncertainty signal: when a perturbed embedding rotates away from the clean direction, confidence should be discounted.","The pronounced collapse at roughly 60 percent occlusion, visible in both accuracy and feature retention, suggests a capacity limit of the learned representation that could be compared across models or training recipes.","Because MAE attends globally from early layers, attention-rollout-based occlusion may be inherently less damaging for MAE than for a standard ViT; the paper's attention-distance result implies this, though it does not directly compare the two under identical occlusion."],"forward_implications":["Fine-tuned MAE keeps top-1 accuracy above 80 percent under the strongest Gaussian blur tested, even as PSNR drops from 28 to 20 dB, implying the mean-patch embedding retains class information when local detail is destroyed.","Attention-guided occlusion of the 50 percent most-attended patches leaves accuracy high, and even 90 percent occlusion retains 60.8 percent accuracy, implying the model does not depend on any single attended patch.","ImageNet-C results extend the robustness claim beyond blur and occlusion: weather, digital corruption, defocus blur, and motion blur show stable accuracy until high severity, with noise-based corruptions declining more steeply.","Correctly classified images have high cosine similarity between clean and perturbed embeddings (about 0.915) while misclassified images have much lower similarity (about 0.289), so directional alignment is a representation-level correlate of correctness.","The layer at which class clusters emerge (layers 7-9) is where MAE's latent space first becomes class-aware, providing a natural depth marker for future transfer or probing."],"fun_headline_variants":["Unsupervised MAE reveals class structure in latent space","MAE latent space self-organizes by class, resistant to blur","Class-aware subspaces emerge in MAE without any supervision","Masked autoencoders stay sharp under blur without supervised labels","No labels needed: MAE builds separable class subspaces"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The main load-bearing assumption is that the unstated rank k used for the SVD class subspaces, together with the 10 ImageNet classes and 50 images per class, is representative of MAE's behavior across all classes; if the divergence trend depends strongly on k or on the selected classes, the central geometric claim would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised MAE reveals class structure in latent space","MAE latent space self-organizes by class, resistant to blur","Class-aware subspaces emerge in MAE without any supervision","Masked autoencoders stay sharp under blur without supervised labels","No labels needed: MAE builds separable class subspaces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001182,"raw_usage":{"total_tokens":4693,"prompt_tokens":693,"completion_tokens":4000,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":3916}},"tokens_in":437,"tokens_out":4000,"duration_ms":30954,"temperature":1.0,"reasoning_tokens":3916,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:55:03.852923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the principal-angle analysis at several values of k (for example, 1, 10, 50, 100, and 200) on the same 10 classes; if the layer-wise increase in the smallest principal angle is not monotonic or disappears for some k, the claim that class subspaces progressively diverge is an artifact of the chosen k. A second decisive test: run the same subspace analysis on all 1000 ImageNet classes or on a larger per-class sample; if the trend flattens, the 10-class result does not reflect global MAE structure.","supporting_citations":[],"review_version":1}