{"id":"78d1af75-24d5-4fe0-8d28-917abf982038","arxiv_id":"2511.12723","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LAYA replaces the last-layer classifier with an input-conditioned attention mixture of all hidden representations, yielding small accuracy gains and per-sample layer-attribution scores.","lead":"A new output-head module, LAYA, uses input-dependent attention to combine features from all hidden layers before classification. On small vision and text benchmarks it matches or slightly beats the standard last-layer head, and its attention weights are presented as built-in explanations of which layers matter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attention weights are mixture weights, not attribution scores; alpha_i alone cannot quantify a layer's contribution to the logit, and the paper never validates this central interpretability claim.","rationale":"The reader identifies the lack of validation of attention weights as the weakest assumption. I agree that validation is missing, but I go further: the formulation itself makes it implausible that alpha_i alone is a faithful attribution, because the actual effect on the logit is modulated by W z_i. This is an internal inconsistency, not just an empirical gap. Nevertheless, the paper could be salvaged by redefining interpretability as 'learned mixture weights' rather than 'layer attribution', or by imposing normalization and demonstrating that alpha tracks true contribution. The reader's conditional verdict is appropriate: the architecture contribution may stand if the interpretability claim is either validated under a precise definition or dropped. My read does not change the verdict, but it strengthens the condition: empirical validation alone is insufficient unless it tests the mathematical relationship between alpha and output contribution. Thus I recommend keeping the conditional accept, with the requirement that the authors either prove alpha is a valid attribution or explicitly downgrade the claim.","tokens_in":15457,"tokens_out":4105,"duration_ms":44173,"concrete_test":"Evaluate a trained LAYA model on a test set. For each sample x, compute the logit contribution of layer i to the predicted class c as c_i(x) = alpha_i(x) * (W_c . z_i(x)). Rank all layers by |c_i(x)| and separately by alpha_i(x). Compute the Spearman rank correlation per sample and the aggregate correlation across all samples. If the correlation is consistently low (e.g., < 0.7), alpha does not reflect actual contribution. Additionally, perform a leave-one-layer-out ablation (zeroing z_i in the aggregation) and measure the change in predicted probability for the true class; compare these ablation importances with alpha_i using rank correlation. If either comparison fails, the interpretability claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central, differentiating claim is that LAYA's attention coefficients alpha_i(x) provide 'explicit and quantitative attribution' and 'quantify the contribution of each representation to the final decision' (Abstract, Section 3). This is not merely unvalidated; it is mathematically unjustified as stated. In the model, h_agg = sum_i alpha_i z_i, then logits = W h_agg + b. The contribution of layer i to the logit of class c is alpha_i * (W_c . z_i), not alpha_i alone. Because z_i are not normalized to comparable scales and W is not constrained, a layer with large alpha_i but small projection magnitude can contribute less than a layer with small alpha_i and large projection magnitude. Thus the attention weights are interpolation weights in a mixture, not measures of causal influence or attribution. The paper never defines what 'contribution' means, nor does it compare alpha_i to any ground truth (e.g., ablation, gradient-based attribution, or leave-one-out). The interpretability experiments in Section 4.3 only describe the learned alpha distributions; they never test whether those distributions agree with the actual effect of each layer on predictions. Consequently, the core interpretability contribution collapses unless the authors either impose and verify conditions under which alpha_i equals (or at least correlates with) the true per-layer contribution, or explicitly reframe alpha as mere mixture weights and abandon the attribution claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LAYA, an output head that replaces the standard final-layer classifier with an input-conditioned weighted sum of all hidden-layer representations z_i = g_i(h_i), where the weights α_i(x) are produced by a small attention network over depth. The authors claim two main benefits: consistent accuracy/F1 gains over LASTLAYER, CONCAT, and SCALARMIX on Fashion-MNIST, CIFAR-10, and IMDB, and an intrinsic, quantitative layer-attribution signal given by α_i(x). Experiments also include an artwork-classification illustration with a frozen ViT. The manuscript is clearly written and the method is simple to describe, but the evidence for both headline claims is currently weak.","tokens_in":15760,"tokens_out":4563,"duration_ms":45139,"significance":"If the attribution claim were validated, LAYA would be a useful architecture-agnostic output head: it adds modest parameters, leaves the backbone untouched, and could provide per-sample depth explanations without post-hoc tools. The comparison controls the backbone across heads, the code is promised, and the method is easy to retrofit. However, the central interpretability claim is not currently established: α_i(x) is asserted to measure layer contribution without a formal definition or any ground-truth validation. The accuracy gains are also small and the reported 95% confidence intervals overlap across methods. The idea is worth pursuing, but the manuscript needs substantial additional evidence or a careful reframing before the claims can be accepted.","major_comments":[{"comment":"The central interpretability claim equates the attention coefficients with quantitative attribution. From the model, h_agg = Σ_i α_i(x) z_i and the logit for class c is Σ_i α_i(x)(W_c · z_i) + b_c. The actual contribution of layer i to that logit is α_i(x)(W_c · z_i), not α_i(x) alone. Since the adapters g_i are unconstrained projections and W is unrestricted, a layer with large α_i can have a small effect if |W_c · z_i| is small. The paper never defines 'contribution' nor tests α_i against any ground truth (e.g., leave-one-layer-out, input perturbation, or gradient-based attribution). Section 4.3 only reports descriptive statistics of α_i. This is a load-bearing gap: without validation or an explicit convention under which α_i is the contribution, the interpretability contribution collapses to a statement about mixture weights.","section":"Section 3 (Formulation) and Section 4.3"},{"comment":"The claim that LAYA yields 'consistent improvements' and that 'the differences can be attributed directly to the aggregation mechanism' is not supported by the reported statistics. On CIFAR-10, LAYA's 95% CI [0.7138, 0.7286] overlaps CONCAT [0.7036, 0.7274], SCALARMIX [0.6932, 0.7356], and LASTLAYER [0.6819, 0.7405]. On Fashion-MNIST and IMDB the CIs also overlap, and the mean differences are a few tenths of a percentage point. With five seeds and no paired test or effect-size analysis, the accuracy advantage is not established. The statement that overlapping CIs indicate 'stability' is also not a valid statistical inference.","section":"Table 1 and Section 4.2"},{"comment":"The hyperparameter protocol appears asymmetric. LAYA is selected by a per-dataset grid search over d*, τ, ψ, and scorer width using validation accuracy, while the baselines are described only as undergoing 'parallel ablations over d*'. This gives LAYA additional model-selection opportunities, and the extent of baseline tuning is not reported. Even if the procedure is fair, the manuscript should state exactly how each baseline was tuned and how many configurations were considered, otherwise the small observed accuracy gap may reflect selection bias rather than the aggregation mechanism.","section":"Section 4.1 (Training details)"}],"minor_comments":[{"comment":"The heading 'Ablation study' is misleading. Table 2 lists the best configuration found by grid search; it does not ablate components such as ψ, τ, or scorer width. Consider renaming it 'Selected hyperparameters' or adding actual ablation experiments over the proposed components.","section":"Table 2"},{"comment":"The artwork classification experiment uses a single ViT+LAYA run with no baseline, no runs over seeds, and no comparison to alternative explanations. It is presented as illustrative, but statements about 'style fingerprints' and 'deviations indicate uncertainty' go beyond what the evidence can support.","section":"Section 4.3, artwork experiment"},{"comment":"The abstract says experiments cover 'vision and language benchmarks', but the only language benchmark is IMDB. This is fine, but the claim in the abstract that gains are 'up to about one percentage point in accuracy' should be read against the overlapping confidence intervals in Table 1; the current wording overstates the result.","section":"Abstract and Section 1"},{"comment":"Some references are incomplete: 'D Hendrycks. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415, 2016' has an odd initial, and 'Alexey Dosovitskiy, 2020' in the ViT citation is missing coauthors. Please normalize all reference entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable submission for an applied ML venue, but the reviewer should insist that the attribution claim be either validated or explicitly retracted and reframed. The accuracy comparison also needs proper statistical treatment. If the author can add a validation study for α_i (or clearly label it as mixture weights) and temper the performance claims, the paper could be publishable; I do not see the current evidence as sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know two things about the LAYA preprint. First, the core architecture idea is simple and legitimate: instead of classifying from the last hidden layer, it learns input-dependent attention weights over all hidden representations and combines them via adapters before the final projection. That is a genuine variation on ScalarMix, which uses global scalar weights, and it is cheap and easy to retrofit. Second, the paper's main selling point — that the attention coefficients α_i(x) are \"intrinsic layer-attribution scores\" that \"quantify the contribution of each representation to the final decision\" — does not hold up. The coefficients are mixture weights, not attribution scores. Since the logit for class c is W_c^T (sum_i α_i z_i) + b_c, the actual contribution of layer i to that logit is α_i (W_c^T z_i), not α_i alone. Without normalizing the z_i or constraining W, a layer with large α_i but small projection can contribute less than a layer with small α_i and large projection. The paper never defines what it means by \"contribution\" and never validates the attention weights against any ground truth, such as layer ablation, perturbation tests, or comparison with post-hoc attribution methods. The interpretability experiments are descriptive only: they plot mean attention per layer and per class, but they never test whether those distributions actually reflect the effect of each layer on predictions.\n\nThe accuracy results are also weaker than the abstract suggests. On all three benchmarks, the 95% confidence intervals for LAYA overlap with LASTLAYER and the other baselines. On Fashion-MNIST and IMDB the mean differences are under half a percentage point. The claim of \"consistent improvements\" is too strong unless the authors report paired significance tests or confidence intervals on the differences. The hyperparameter search is also somewhat asymmetric: LAYA gets a full grid search per dataset, while the baselines only get a parallel sweep over d*. That is a minor but real imbalance.\n\nWhat the paper does well: the writing is clear, the method is described precisely, the ablations over temperature, projection size, and pre-transform are useful, and the experiments are reasonably controlled with fixed backbones and multiple seeds. The artwork classification experiment adds a nice qualitative demonstration of task-dependent depth usage, even if it is not a head-to-head comparison.\n\nThe paper deserves a serious referee because the idea is worth discussing and the central overclaim needs to be fixed. The authors should either (1) impose and verify conditions under which α_i can be interpreted as a contribution score, or (2) drop the attribution claim and present LAYA as an adaptive layer-mixture head with potential applications in compression and early exiting. If the explanation claim is reframed as hypothesis-generating, the architecture contribution can stand on its own. For now, I would not cite the attribution claim, but I would bring the paper to a reading group to talk about why attention weights are not explanations.","headline":"LAY A is a clean but modest output-head variation; the interpretability claim that attention weights are layer attributions is not justified by the math or the experiments.","tokens_in":16274,"tokens_out":3048,"would_cite":false,"duration_ms":28419,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing the last-layer classifier head with input-conditioned attention over all hidden layers yields both small consistent accuracy gains and intrinsic layer-attribution scores.","keywords":["layer-wise attention","output head","depth-aware aggregation","interpretability","layer attribution","attention over depth","deep learning","representation learning"],"falsifier":"On a held-out set, remove or perturb individual layers (e.g., zero out a layer's contribution or retrain without that layer) and compare the resulting prediction changes to LAYA's attention weights; if layers with near-zero attention still cause large prediction changes, or high-attention layers have no effect, the attribution claim is falsified.","tokens_in":15290,"feed_emoji":"🧠","tokens_out":2584,"duration_ms":28514,"temperature":0.7,"pith_summary":"The paper argues that the standard practice of making predictions from only the final hidden layer discards useful intermediate information, and it introduces LAYA, a new output head that weights all hidden layers with input-dependent attention. It claims that LAYA consistently matches or slightly improves accuracy across vision and language benchmarks, while its learned attention coefficients double as intrinsic layer-attribution scores. Because interpretability emerges from the model's own computation, no post-hoc explanation method is needed. The broader point is that the output stage can be an active depth-aware reasoning module rather than a passive projection.","feed_headline":"Attention over every layer beats last-layer-only heads","feed_subtitle":"A drop-in output module puts per-input weights on network depth, matching or beating standard heads and explaining which layers matter.","key_machinery":"Attention over the depth dimension of a network, rather than over tokens or spatial locations. The mechanism works by making layers comparable through lightweight adapters, reading all layer representations with a scoring MLP, and normalizing the resulting logits with a temperature-scaled softmax. This turns the output head from a static projection into an input-conditioned aggregator and produces the interpretability signal as a byproduct.","core_discovery":"LAYA generalizes the standard classifier head. Each hidden layer hi is first projected by a learnable adapter gi, then aggregated as h_agg = sum_i αi(x) gi(hi), where the coefficients αi(x) are generated by a scoring MLP over the concatenated layer features and normalized by a temperature-scaled softmax. The final prediction is made from h_agg. Because the attention weights are learned jointly with the model and depend on the input, they serve as explicit, per-sample attribution scores over network depth, revealing which abstraction levels contribute to each decision without requiring external explanation tools.","pith_inferences":["The interpretability claim stands only if the attention weights faithfully reflect true layer importance; a natural test is to compare LAYA's weights against leave-one-layer-out ablations, and if they diverge the intrinsic-explanation claim would need qualification.","The reported gains are small (roughly 0.1 to 1 percentage point), which suggests depth-aware aggregation matters most where intermediate features are decisive; larger effects might appear on fine-grained classification or document-style tasks not tested here.","Because the scoring network consumes all layer features, the attention mechanism could be pushed toward true layer selection through temperature scheduling or sparsity regularization, a testable extension that might sharpen both accuracy and interpretability.","The attention distribution's entropy could be explored as an uncertainty or out-of-distribution detector, since the paper's own analysis shows that error cases often have more diffuse attention profiles."],"forward_implications":["Existing backbones can be retrofitted with LAYA at the output stage without modifying feature learning, making depth-aware prediction available even for frozen pretrained models.","The per-sample, per-layer attention weights allow analysis of which abstraction levels drive decisions for each class and how attention shifts between correct and incorrect predictions.","Layers that consistently receive negligible attention become candidates for pruning or removal, providing a principled basis for model compression.","Attention profiles that concentrate on early layers can justify early-exit inference, since sufficient evidence may accumulate before the deepest layers.","Differences in attention between correct and misclassified samples could serve as a diagnostic signal for uncertain or out-of-distribution inputs."],"fun_headline_variants":["Layer-wise attention replaces last-layer shortcut","Attention over hidden layers explains every decision","Per-input layer weights unlock interpretable depth","New output head turns depth into explanation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The attention coefficients αi(x) are assumed to faithfully quantify how much each layer truly contributes to the prediction, but the paper never verifies this against ground-truth layer importance, so if attention is not a faithful explanation the interpretability claim falls even if the accuracy result holds.","fun_headline_variants_meta":{"raw":{"variants":["Layer-wise attention replaces last-layer shortcut","Attention over hidden layers explains every decision","Per-input layer weights unlock interpretable depth","New output head turns depth into explanation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1265,"prompt_tokens":731,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":482}},"tokens_in":475,"tokens_out":534,"duration_ms":7365,"temperature":1.0,"reasoning_tokens":482,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:44:53.629745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set, remove or perturb individual layers (e.g., zero out a layer's contribution or retrain without that layer) and compare the resulting prediction changes to LAYA's attention weights; if layers with near-zero attention still cause large prediction changes, or high-attention layers have no effect, the attribution claim is falsified.","supporting_citations":[],"review_version":1}