{"id":"ef211763-c95b-48f9-ac4b-1d85e836807f","arxiv_id":"2607.10391","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Intermediate ViT layers recover 18–76% of last-layer errors, and VFusion’s learned vertical fusion closes about 45% of the remaining gap to a layer-wise oracle.","lead":"Middle layers of Vision Transformers often classify images that the final layer gets wrong. A small learned fusion head can harvest that internal signal and improve accuracy without running multiple models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The redundancy-vs-diversity mechanism remains correlational and is the softest support for VFusion's design rationale, even though the accuracy and recoverability numbers themselves are solid.","rationale":"The reader correctly isolated the weakest link: the mechanism claim is correlational, not causal, while the recoverability counts, multi-dataset accuracy tables, OOD results, backbone-scale ablations, capacity control, and code release make the empirical performance claim credible. That combination already justifies CONDITIONAL rather than unconditional ACCEPT; nothing stronger (e.g., data leakage, oracle mis-definition, or capacity confounds) appears after re-reading Tables 1–5 and the appendices. The proposed test directly probes whether the design rationale is load-bearing or merely post-hoc. No change to the reader’s verdict or confidence is warranted.","tokens_in":24064,"tokens_out":626,"duration_ms":32095,"concrete_test":"On the 9 non-saturated datasets of Table 4, retrain VFusion once with the existing L_orth term and once with an explicit diversity-promoting regularizer (e.g., maximize average pairwise cosine distance or JS divergence among the selected layer logits inside the fusion objective, same capacity and λ schedule). If the diversity-regularized variant matches or exceeds the original VFusion accuracy while the original L_orth still yields lower latent off-diagonal correlation, the redundancy rationale is not required for the gains; if it underperforms, the compression design is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim packages two things: (1) substantial recoverability (Table 1: 18–76 % of last-layer errors fixed by some intermediate probe) plus VFusion closing ~45 % of the best-layer-to-oracle gap (Table 2 averages 89.5 → 91.4 vs oracle 93.7), and (2) the mechanistic assertion that this is driven by a “redundancy-correctness correspondence” rather than predictive diversity (§3.2, Fig. 1). The second part rests only on two correlations—recovery rate vs correction entropy (r = 0.87) and vs average pairwise logit cosine distance on jointly misclassified samples (r = −0.77)—plus a parallel JS-divergence check in Appendix A.1. These are observational; they do not isolate whether the shared signal is what the low-dimensional compressor actually exploits, nor rule out that residual local complementarity (Appendix A.2) still functions as a weak form of diversity. If the correlations mis-attribute the source, the explicit design choice of aggressive compression + orthogonalization (Eqs. 7–10, λ = 2) loses its theoretical justification even while the empirical gains remain. The capacity-controlled Final-only ablation (Appendix C.2) and MoE comparisons show multi-layer features help, but they do not causally confirm the redundancy story.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that intermediate Vision Transformer layers contain substantial corrective signal for multi-class classification: independent layer-wise probes recover 18%–76% of last-layer errors across 16 datasets (Table 1; recovery rate in Eq. 2). It attributes this primarily to a “redundancy-correctness correspondence” rather than predictive diversity, based on correlations of recovery rate with correction entropy and (negatively) with pairwise logit disagreement (Fig. 1; §3.2). Building on that analysis, it proposes VFusion: select multi-layer [CLS] tokens, concatenate them, and compress them with a supervised MLP encoder into a low-dimensional latent (Eqs. 6–7) trained with cross-entropy plus an orthogonalization regularizer (Eqs. 8–10). Empirically, VFusion outperforms non-parametric and parametric aggregation baselines (Average, Majority, Super Learner, NLC, MoE, sw-MoE) on 16 ID datasets (Table 2; avg. 91.4% vs best-layer 89.5%, closing ~45% of the best-layer–oracle gap) and five shift benchmarks (Table 3), with ablations on layer selection, backbone family/scale, horizontal fusion, capacity, efficiency, and latent size.","tokens_in":24442,"tokens_out":1551,"duration_ms":27883,"significance":"If the empirical results hold, the work is a useful and practical contribution to frozen-backbone ViT deployment: it quantifies internal recoverability at scale, shows that simple last-layer probing leaves unused signal, and provides a lightweight vertical fusion head that improves ID and OOD accuracy without multi-backbone ensembles. Strengths include broad public-benchmark evaluation (21 datasets), multiple established baselines, capacity-controlled Final-only comparison (Appendix C.2), efficiency vs horizontal Super Learner (Appendix C.1), generalization across supervised/self-supervised/CLIP backbones and scales (Fig. 3), and released code. The recoverability framing and layer-selection guideline (contiguous depth preferred over striding) are actionable for practitioners. The mechanistic “redundancy-correctness” story is softer than the accuracy claims and should be treated as interpretive rather than established; even so, the method and measurement contributions remain significant for the field.","major_comments":[{"comment":"§3.2 and Fig. 1 (also Abstract/Introduction): the claim that recoverability is “not primarily driven by predictive diversity, but by a redundancy-correctness correspondence” is supported only by observational correlations (recovery rate vs H_corr, r=0.87; vs average pairwise cosine distance on jointly misclassified samples, r=−0.77; JS check in Appendix A.1). These do not causally isolate shared-signal compression from residual local complementarity (Appendix A.2 documents non-zero unique neighbor recoveries at every depth). Please rephrase causal language to correlational evidence, and clarify that VFusion’s compression design is motivated by—not proven by—this analysis. The accuracy claims do not require a stronger mechanism proof, but the current wording overstates what Fig. 1 establishes.","section":"§3.2, Fig. 1, Abstract"},{"comment":"§4.2.1 / Eqs. (6)–(7): the decomposition h^(ℓ)=s^(ℓ)+r^(ℓ) and the assertion that the low-dimensional encoder “discards layer-specific residuals” are motivational, not measured. No ablation isolates shared vs layer-private components (e.g., reconstruction of s, mutual information across layers, or comparison of low-dim fusion vs a capacity-matched high-dim concat MLP without aggressive bottleneck). Appendix C.2 shows multi-layer input helps vs Final-only, and MoE uses the same concat features, but neither confirms noise-suppression of r^(ℓ). Either add a targeted diagnostic or present the encoder as supervised multi-layer dimensionality reduction without claiming residual filtering as established.","section":"§4.2.1, Eqs. (6)–(7)"},{"comment":"Table 2 / §5.2: VFusion approaches or exceeds the layer-wise oracle on Cars, Flow, IN1k, and ESAT. The oracle (Eq. 1) marks a sample correct if any single-layer probe is correct; surpassing it implies the fused representation creates new decision boundaries, not only selection among layers. This is interesting but under-discussed. Please quantify how often VFusion is correct when all layer probes fail (or when only a minority succeed), and state clearly that the oracle is not an upper bound on feature-level fusion—only on discrete layer selection—so the “45% of headroom closed” framing does not imply proximity to an absolute ceiling.","section":"Table 2, §5.2, Eq. (1)"}],"minor_comments":[{"comment":"Tables consistently typeset “ESA T” / “ESA T” with a space (EuroSAT). Fix throughout Tables 1–2 and related captions.","section":"Tables 1–2"},{"comment":"§5.1.3: probe hyperparameter grid and VFusion settings are clear; please also state whether MoE/sw-MoE expert MLPs match the probe head capacity and whether routing is trained jointly with experts under the same early-stopping protocol.","section":"§5.1.3"},{"comment":"Figure 2 right panel is effective; ensure the “45%” annotation is defined in the caption as (a_VFusion − a_best)/(a_oracle − a_best) using the Table 2 averages so the figure is self-contained.","section":"Figure 2"},{"comment":"§6.3 / Table 5: HFusion is introduced as horizontal feature fusion; a one-sentence architectural parallel to VFusion (same encoder family on concatenated last-layer features) would help readers map vertical vs horizontal settings.","section":"§6.3, Table 5"},{"comment":"Appendix A.2 Table A1 is informative; consider a brief pointer in main §3.2 that local unique recoveries exist at all depths, so redundancy is imperfect—this would balance the low-disagreement narrative without new experiments.","section":"Appendix A.2"},{"comment":"Limitations (§7.1) correctly note supervised fusion and frozen backbones; a short remark that recoverability is measured with trained probes (not zero-shot) would prevent over-reading for CLIP-style settings already discussed in §2.3.","section":"§7.1"}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical methods paper with released code and unusually broad evaluation for a fusion-head contribution. The main risk is overstated mechanism language, not broken numbers; minor revision to tone and a short oracle-surpassing discussion should suffice. Fit is appropriate for a CV/ML journal that values practical representation use of foundation models. No integrity concerns from the manuscript as provided."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core is simple: intermediate ViT [CLS] probes recover a large share of last-layer mistakes (18–76% across 16 datasets, Table 1), and a lightweight learned compressor over concatenated multi-layer features (VFusion) beats standard aggregation baselines and closes about 45% of the best-layer-to-oracle gap (89.5 → 91.4 vs 93.7). That is a real, practical result for frozen-backbone classification, especially where multi-model ensembles are expensive or unavailable.\n\nWhat is new is not “use intermediate layers” — OOD layer work, Super Learner, NLC, and MoE are already in the literature — but the systematic multi-class recoverability audit plus a feature-level vertical fusion recipe that is evaluated broadly: ID, covariate/subpopulation OOD, layer-selection ablations, backbone family/scale (DINOv2/CLIP/IN21k, S/B/L), capacity-controlled Final-only, efficiency vs horizontal SL, and a code link. Gains are modest on average and larger on hard/fine-grained sets (GTS, Cars, FGVC). Contiguous deeper layers work; striding hurts more. Horizontal extension of the same idea also works. Math is standard (concat → MLP bottleneck + optional orth loss); citations are appropriate; no circular scoring against the oracle.\n\nThe soft spot is the mechanism pitch. Figure 1’s correlations (recovery vs correction entropy r=0.87; vs logit cosine distance r=−0.77, plus JS in the appendix) support “not pure ensemble diversity,” but they do not isolate that the compressor exploits shared redundancy rather than residual local complementarity (Appendix A.2). Free knobs (dz, λ, encoder widths) are ablated enough to show robustness, not uniqueness. Scope is frozen backbone + classification only — stated as limitations, not hidden.\n\nThis is for people who ship frozen ViT classifiers and care about cheap robustness. It deserves a serious referee. I would engage: cite the recoverability tables and try VFusion as a baseline; treat the redundancy story as provisional framing.","headline":"Solid multi-dataset recoverability numbers and a practical vertical fusion head; the diversity-vs-redundancy story is correlational, not causal, but the accuracy claims hold without it.","tokens_in":25051,"tokens_out":539,"would_cite":true,"duration_ms":6855,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Intermediate Vision Transformer layers correct 18–76% of last-layer mistakes, and a learned vertical fusion of those layers closes 45% of the gap to an any-layer oracle.","keywords":["Vertical Ensembling","Internal Representations","Recoverability","Vision Transformers","Feature Fusion","Layer Aggregation","Out-of-Distribution Classification"],"falsifier":"On a held-out collection of classification datasets, measure layer-wise recovery rates and train VFusion against best-layer and aggregation baselines; if recovery rates collapse toward zero and VFusion no longer beats the best single layer or the oracle gap remains largely unclosed, the central claim fails.","tokens_in":24948,"feed_emoji":"🧩","tokens_out":959,"duration_ms":17830,"temperature":0.7,"pith_summary":"Vision Transformers are almost always used as black boxes that expose only their final layer for classification. This paper shows that is wasteful: when a lightweight probe is trained on every intermediate layer across 16 datasets, those probes recover between 18% and 76% of the samples the final layer gets wrong. The authors argue the gains come less from predictive diversity than from a redundancy-correctness pattern—layers act as slightly different probes of a shared discriminative signal. They therefore introduce VFusion, a small learned encoder that concatenates selected layer tokens, compresses them into a low-dimensional latent, and trains a classifier on that latent while the backbone stays frozen. The result beats standard aggregation baselines on both in-distribution and shifted data, and works across model sizes and pre-training styles, offering a cheap vertical alternative to multi-model ensembles.","feed_headline":"Middle ViT layers fix up to 76% of final errors","feed_subtitle":"A tiny learned head fuses one frozen transformer's depth and closes 45% of the oracle gap.","key_machinery":"VFusion: a learnable fusion encoder that concatenates selected intermediate [CLS] tokens, maps the concatenation into a low-dimensional latent (with optional orthogonalization), and trains a lightweight classifier on that latent while the backbone remains frozen; together with the recoverability rate (fraction of final-layer errors corrected by at least one intermediate probe).","core_discovery":"Intermediate representations inside a single frozen Vision Transformer contain substantial corrective signal: independent layer-wise probes recover 18–76% of last-layer errors across 16 datasets. That recoverability is better explained by distributed, redundant probes of a shared decision boundary than by ensemble-style disagreement. A lightweight supervised fusion head (VFusion) that compresses the multi-layer hierarchy into one low-dimensional token captures a large fraction of this unused signal, closing 45% of the accuracy gap between the best single layer and a theoretical any-layer oracle, and outperforming established aggregation methods in both in-distribution and out-of-distribution","pith_inferences":["The same recoverability pattern may appear in other transformer stacks (language, multimodal) that keep relatively uniform representations across depth, inviting vertical fusion heads beyond vision classification.","If the redundancy-correctness account holds, unsupervised or self-supervised pre-training of the fusion encoder could remove the need for task labels at fusion time.","Dense prediction tasks such as segmentation may benefit even more, because intermediate layers already encode spatial detail that the final [CLS] token discards."],"forward_implications":["A frozen off-the-shelf ViT can be made more accurate by attaching only a small fusion head instead of training or running multiple backbones.","Horizontal ensembles become less necessary when the vertical hierarchy inside one model already supplies recoverable signal.","Deeper contiguous layers preserve most of the fusion benefit, so practitioners can safely drop early layers if compute is tight.","The same compression principle extends to horizontal fusion across heterogeneous pre-trained backbones when multiple models are available.","Gains appear larger on fine-grained and degraded-image tasks, suggesting the method is most useful where last-layer features are fragile."],"fun_headline_variants":["Middle layers rescue 18-76% of final ViT errors","VFusion condenses ViT depth to fix last-layer failures","Internal ViT hierarchy recovers most final misclassifications","Learnable vertical fusion closes 45% of ViT oracle gap","Single frozen ViT layers correct last-layer mistakes"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the corrective power of intermediate layers comes mainly from them being stable, redundant probes of one shared signal rather than from genuine predictive diversity, a claim resting on correlations with correction entropy and logit disagreement rather than a direct causal test.","fun_headline_variants_meta":{"raw":{"variants":["Middle layers rescue 18-76% of final ViT errors","VFusion condenses ViT depth to fix last-layer failures","Internal ViT hierarchy recovers most final misclassifications","Learnable vertical fusion closes 45% of ViT oracle gap","Single frozen ViT layers correct last-layer mistakes"]},"model":"grok-4.5","effort":"low","cost_usd":0.001806,"raw_usage":{"total_tokens":951,"prompt_tokens":879,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":18060000,"prompt_tokens_details":{"text_tokens":879,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":879,"tokens_out":72,"duration_ms":1064,"temperature":1.0,"reasoning_tokens":0,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T12:04:52.310621+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out collection of classification datasets, measure layer-wise recovery rates and train VFusion against best-layer and aggregation baselines; if recovery rates collapse toward zero and VFusion no longer beats the best single layer or the oracle gap remains largely unclosed, the central claim fails.","supporting_citations":[],"review_version":1}