{"id":"7cb6d832-4d21-420d-bb10-8f2a75eca0ea","arxiv_id":"2507.18681","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A layer-selection rule based on mutual information and linear-probe accuracy identifies where a concept can be probed most accurately in a neural network.","lead":"This paper proposes a fast way to pick which layer of a neural network to probe for a human-defined concept, using how informative and how regular the layer's activations are. The chosen layers let simple probes reach near-oracle accuracy, without training probes on every layer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The informativeness term in Eq. 1 relies on mutual-information estimates on very high-dimensional representations with an undisclosed manual adjustment; if these estimates are unstable, layer selection could be driven by noise, and no reported ablation shows the term is actually needed.","rationale":"The reader's weakest assumption correctly identifies the mutual-information estimator as the least secure load-bearing component of Eq. 1. The paper explicitly acknowledges limitations and a manual gamma adjustment, and it provides no sensitivity analysis. This is exactly the kind of internal evidence that should be flagged: the informativeness half of the proposed criterion rests on an estimate whose reliability in 6.3M-dimensional spaces with ~770 samples is unknown. My concrete test would settle the concern directly: if removing U or perturbing the estimator leaves selections and accuracies essentially unchanged, then the central empirical claim survives regardless of MI quality; if not, the paper's reported numbers depend on an unvalidated estimate. I agree with the reader's CONDITIONAL verdict because the concern is real but not demonstrated to be fatal. The secondary issues of lambda tuning on the benchmark and missing confidence intervals are real but less central, since the method's performance could still be valid even if the reported point estimate is optimistic. The proposed test is a single, feasible check that distinguishes these possibilities.","tokens_in":16977,"tokens_out":7249,"duration_ms":89204,"concrete_test":"Re-run the full layer-selection pipeline from Table 2 with lambda = 0 (regularity only), using the reported R values, and also recompute U for fCUB and fImageNet under default and varied gamma settings. Compare the selected layers and resulting probe test accuracies against Table 2. If the average accuracy stays within about one percentage point and layer selections rarely change, the mutual-information estimates are not load-bearing; if selections shift materially, the paper needs either a validated MI estimator or a U-free criterion to support its central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that combining informativeness U and regularity R in Eq. 1 identifies layers that yield accurate concept probes. The regularity term R is a logistic-regression accuracy, which is straightforward to estimate. The informativeness term U, however, is estimated with the Noshad et al. (2019) estimator on full layer representations of up to 6,322,176 units using at most 1,000 samples. The paper itself cites the formal limitations of mutual-information measurement (McAllester and Stratos, 2020) in the Section 3 footnote, and Appendix C reports that the gamma smoothness parameter was adjusted for fCUB and fImageNet 'to allow for a more accurate estimation' without giving the adjusted values or any sensitivity analysis. If those estimates are biased or unstable in these high-dimensional regimes, the argmax in Eq. 1 can select the wrong layer. The concern is partly mitigated by the weighting lambda = 0.26, which gives U only 26% weight, and by Figure 4, which shows results are relatively stable for lambda below 0.6. Yet the paper never reports what happens when U is removed entirely or replaced by an alternative estimator, so the actual contribution of the informativeness term to the headline 98.6%-of-oracle result is unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic layer-selection method for concept probing. For each layer of a neural network and a given human-defined concept, the method computes an informativeness score U (normalized mutual information between the layer's activations and concept labels) and a regularity score R (logistic-regression accuracy on the concept labels), then selects the layer maximizing the weighted combination in Eq. (1). The method is evaluated on four datasets (XTRAINS, GTSRB, CUB, ImageNet) and six original models, training five types of probes and comparing against a layer-average baseline, an exhaustive validation-based search, a prior input-reduction method, and an oracle that selects the layer with the highest test accuracy. The headline results report that the proposed method reaches on average 90.1--90.2% probe accuracy, roughly 98.6% of the oracle's accuracy, while taking about 9 minutes per concept instead of several hours for exhaustive validation. The authors also present qualitative observations about how concept informativeness and regularity evolve across layers, arguing that the two characteristics jointly identify suitable probing layers.","tokens_in":17199,"tokens_out":6339,"duration_ms":66837,"significance":"If the central claim holds, the paper provides a practically useful tool: it reduces layer selection for concept probing from an expensive exhaustive search to a fast screening procedure, and the reported accuracy is close to that of a test-oracle upper bound. The empirical study is unusually broad for the probing literature in terms of coverage (four datasets, six original models, thirty concept-model pairs, five probe architectures) and includes a runtime comparison, which is a strength. The conceptual separation between informativeness and regularity is a useful framing and leads to interesting qualitative findings about how different concepts are encoded at different depths. However, the central claim is currently supported by an evaluation in which the global hyperparameter λ in Eq. (1) is selected on the same benchmark used to report the final accuracies, and the mutual-information estimates underlying the informativeness term are not validated in the high-dimensional regime where the largest models operate. These issues need to be addressed before the quantitative claims can be fully trusted.","major_comments":[{"comment":"The global weight λ is set to 0.26 because it 'results in the best average performance' on the benchmark (Section 5, after Figure 4), and the headline results in Table 1 are then reported using this value. Unless the average performance used for tuning is computed on a separate validation split, this is test-set fitting of a hyperparameter on the same data used to report the final accuracies, which makes the 98.6%-of-oracle figure optimistic. The comparison with the oracle is also asymmetric: the oracle is evaluated per concept using the test labels, while λ is a single value tuned on the aggregate test results. Please clarify the tuning protocol explicitly, and either (a) select λ on a nested validation split and report test results with that fixed λ, (b) report results averaged over a range of stable λ values (e.g., 0.1--0.5), or (c) show that the conclusions in Table 1 are unchanged when λ is chosen independently of the test data. The sentence 'the cost of fine-tuning λ is low -- for a given concept, it leads to considering only, on average, 3.7 different layers' should be explained, since in the paper λ is not actually fine-tuned per concept.","section":"Section 5, Eq. (1), Figure 4, Table 1"},{"comment":"The mutual-information estimates for fCUB and fImageNet use an adjusted γ smoothness parameter whose values are not reported, and the ensemble size is increased to 15 without any sensitivity analysis. Given the dimensionality of the representations (up to 6,322,176 units) and the sample size (at most 1,000), the U term in Eq. (1) may be dominated by estimation noise, especially because the authors themselves cite the formal limitations of mutual-information measurement (McAllester and Stratos, 2020). The paper never reports an ablation with λ = 0 (i.e., the informativeness term removed) or a comparison with an alternative MI estimator. Such an experiment is necessary to establish that the U term contributes to layer selection rather than merely adding noise; if the results without U are similar, the claim that informative and regular layers are jointly selected should be revised.","section":"Appendix C and Section 3"},{"comment":"The entire empirical evaluation uses a single fixed data split (random state 0 in train_test_split, Appendix C), and the reported differences between the proposed method and the best-validation baseline are often small (e.g., fB: 85.6 vs 85.4; several rows in Table 2 are identical or differ by less than one point). Without repeated splits, bootstrap, or another measure of variability, it is difficult to know whether the observed superiority over 'Best Validation' is a reliable effect or split noise. Please add variance estimates over multiple random splits, or at least report per-concept confidence intervals, for the comparisons in Tables 1 and 2.","section":"Section 5, Tables 1--3 and Appendix C"}],"minor_comments":[{"comment":"The sentence 'Others have studied how the representations of the output concepts of a model evolve across its layers (Noshad et al., 2019; Alain and Bengio, 2017)' appears to attribute a study of output-concept evolution to the mutual-information estimator paper of Noshad et al. (2019); please verify and replace with the correct reference.","section":"Related Work, Section 6"},{"comment":"The rendered form of Eq. (1) is ambiguous: the term involving (1−λ) reads as 'k R(c|fl(x)) − 1/k − 1' in the text, which is not dimensionally consistent. If the intended expression is (1−λ) * (R(c|f_l(x)) − 1/k)/(k − 1), please write it explicitly, since this normalization is central to the method.","section":"Equation (1)"},{"comment":"The y-axis of Figure 4 is labeled 'Accuracy' but the figure is used to justify the choice of λ; please state explicitly whether the plotted accuracy is test accuracy or validation accuracy, and how the 'average' over concepts and models is computed.","section":"Figure 4 and Section 5"},{"comment":"The affiliation text contains a spacing artifact ('NOV A LINCS' and 'NOV A School of Science and Technology'), which should be corrected to 'NOVA LINCS' in the final version.","section":"Authors' affiliation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extended version of a NeSy 2025 paper, but the introduction does not state which content is new to the extended version and what the incremental contribution over the conference version is. The editor may wish to ask the authors for a clear statement of novelty relative to de Sousa Ribeiro et al. (2025b). Also, no code or data release is mentioned; given the breadth of the empirical study, a public implementation would substantially increase reproducibility and the usefulness of the method to the probing community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the core method is the authors' own NeSy'25 criterion, so this paper is not a new proposal. What it contributes is the broadest empirical validation of that heuristic I've seen: six models, four datasets, thirty concept-model pairs, and sensible baselines. The headline — about 98.6% of oracle probe accuracy — is credible and practically useful.\n\nThe evaluation is the strong part. They compare against the layer average, an oracle upper bound, exhaustive validation, and a unit-selection method. Their method is substantially faster than exhaustive search and beats it slightly in accuracy. The per-concept tables in the appendix are consistent with the aggregate, which suggests the effect is real.\n\nSoft spots, in proportion:\n\n- λ is chosen as the value that maximizes average performance on the same benchmark used for the headline result. That is a mild form of fitting. It is tempered by Figure 4, which shows the accuracy surface is flat for λ<0.6.\n\n- The mutual-information term is estimated on representations up to 6.3M units from at most 1,000 samples. Appendix C says the γ smoothness parameter was manually adjusted for the two largest models, but the values are not reported and there is no sensitivity analysis. This is the weakest part of the method's foundation. The impact is limited by λ=0.26, which gives U only 26% weight.\n\n- On the ablation point: the stress-test worried that no ablation removes U entirely. That's not quite right — Figure 4 sweeps λ down to 0, which is exactly the R-only case, and the average accuracy appears similar for λ below 0.6. The paper doesn't quote the λ=0 number or compare it directly to λ=0.26, so the U term's marginal contribution is unquantified. I'd call this a soft spot, not a fatal flaw.\n\n- No repeated seeds or confidence intervals anywhere. With training sets capped at 1,000 samples, that matters.\n\nThe central claim holds up: a weighted combination of informativeness and regularity selects layers close to the oracle. The method itself isn't novel, but the evidence is solid enough to make this a useful empirical reference.\n\nWho it's for: anyone doing concept probing who wants a practical layer-selection heuristic with multi-model evidence.\n\nRecommendation: send it to peer review. The referee should ask for the gamma values, the λ=0 comparison in tabular form, and some repeated runs. But it deserves the time.","headline":"A solid empirical validation of an existing layer-selection heuristic for concept probing; the core method is not new but the evaluation is broad and useful, with tune-λ and MI-estimator caveats.","tokens_in":17759,"tokens_out":5574,"would_cite":true,"duration_ms":55216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"To probe a neural network for a human-defined concept, pick the layer whose representations are both informative and regular; a weighted score of the two selects that layer in minutes and reaches about 98.6% of oracle accuracy.","keywords":["concept probing","layer selection","mutual information","uncertainty coefficient","logistic regression accuracy","neural network interpretability","representation analysis","probe efficiency"],"falsifier":"Recompute the layer ranking for one of the large ResNet50 models (CUB or ImageNet) using a different mutual-information estimator, or the same estimator with several smoothness values, and train probes on the layers that win under each ranking. If the ranking flips to a layer whose probe accuracy is substantially below the reported about-90-percent level, or if the reported layer no longer beats the layers-average baseline, the selection criterion is not robust.","tokens_in":16679,"feed_emoji":"🔍","tokens_out":10545,"duration_ms":91716,"temperature":0.7,"pith_summary":"Concept probes—small classifiers trained on a network's internal activations to detect human-defined concepts—work only if you probe the right layer. This paper claims that the right layer is the one whose representations are both informative about the concept (high normalized mutual information) and regular with respect to it (high logistic-regression accuracy), and it proposes selecting the layer that maximizes a weighted combination of the two. Across six vision models and five concepts each, probes trained from the selected layer reach about 90 percent average accuracy, roughly 98.6 percent of an oracle that knows the best layer in hindsight, while costing about nine minutes of computation instead of hours. The same characterization also reveals where in a model a concept is formed, when it is discarded, and whether a concept is encoded at all.","feed_headline":"A 10-minute score picks the best layer for concept probes","feed_subtitle":"Layers that are both informative and regular yield simple, accurate probes—about 98.6% of oracle performance.","key_machinery":"The machinery is a two-part layer signature plus a weighted selection rule. Informativeness is measured by the uncertainty coefficient $U(c|f_l(x)) = I(f_l(x);c)/H(c)$, estimated from samples with a mutual-information estimator designed for high-dimensional variables; this says whether the concept's value is present in the activations at all. Regularity is measured by $R(c|f_l(x))$, the five-fold cross-validated accuracy of a logistic regression on the layer's activations; this says how directly and simply the concept can be read out, which in turn predicts how little data a probe needs and how simple the probe can be. Equation (1) combines the two into a single score, with the $k$-normalization of $R$ making the regularity term comparable across concepts with different numbers of values. The claim is that maximizing this score is a proxy for choosing the layer that would give the best probe.","core_discovery":"The paper's central claim is that for any given human-defined concept, a network's layers can be ranked by a single criterion that predicts where a concept probe will succeed, and that no expensive probe training is needed to compute that ranking. A layer is characterized by $U(c|f_l(x))$, the uncertainty coefficient (mutual information normalized by the concept's entropy), and $R(c|f_l(x))$, the cross-validated accuracy of a logistic regression trained on that layer's activations to predict the concept. The selected layer is the one maximizing $\\lambda U(c|f_l(x)) + (1-\\lambda) k (R(c|f_l(x))-1)/(k-1)$, with $k$ the number of concept values and $\\lambda$ controlling the information-regularity trade-off; the paper sets $\\lambda=0.26$ after an ablation. Empirically, probes trained on this layer outperform the average over all layers, match or slightly beat an exhaustive validation-based search while being far cheaper, and come within about 1.3 points of the oracle. The paper also shows that the two measures trace interpretable trajectories across layers: task-unrelated concepts stay low in regularity and lose information early, low-level concepts peak before the high-level concepts built on them, and uniformly simple concepts suggest the model itself could be simpler.","pith_inferences":["An unstated extension is whether the same two-term criterion transfers to transformer and text models; the paper tests only convolutional vision models, so attention-layer behavior remains open.","Because the mutual-information estimator is the least controlled component, a cheaper proxy such as a small probe's validation accuracy on a subsample might give nearly the same layer ranking at lower cost; the paper does not test this.","The layer trajectories could serve as a shortcut-learning diagnostic: a task-relevant concept that is discarded early, or whose regularity drops in the final layers, would flag representations the model may not actually rely on.","The paper fixes $\\lambda=0.26$, but its own ablation shows the optimal weight is concept-dependent (Wooden and Rectangular prefer different ranges), so an adaptive per-concept $\\lambda$ could improve further."],"forward_implications":["Layer selection becomes a minutes-long precomputation: about 9 minutes per model-concept pair instead of the 334 minutes needed to train probes on every layer.","Probes from the selected layer reach about 98.6 percent of oracle accuracy and slightly beat an exhaustive validation-based search, so cheaper selection does not cost accuracy.","Because the selected layers are regular, simpler probe models (logistic regression and ridge classifiers) suffice more often, reducing training time and the amount of labeled data needed.","The informativeness–regularity trajectories are themselves interpretable: they distinguish concepts the model does not encode, show when low-level concepts are reused to build high-level ones, and reveal when a task is simple enough that the whole network could be smaller.","The method also beats a unit-selection baseline (input reduce) that searches from the last layer and stops early, missing useful internal representations."],"supporting_citations":[{"why":"Supplies the mutual-information estimator used to compute every informativeness score $U(c|f_l(x))$.","marker":"Noshad et al. (2019)"},{"why":"Cited for the formal limitations of mutual-information measurement, motivating the use of a high-dimensional estimator.","marker":"McAllester and Stratos (2020)"},{"why":"Establishes linear classifier probes and the premise that probe accuracy reflects what representations encode.","marker":"Alain and Bengio (2017)"},{"why":"Supplies the minimum-description-length view that simpler regularities require less data and support simpler probes.","marker":"Voita and Titov (2020)"},{"why":"Grounds the regularity term in the minimum description length principle.","marker":"Rissanen (1978)"},{"why":"Contributes the input-reduce unit-selection baseline and mapping-network probe used in the comparison.","marker":"de Sousa Ribeiro and Leite (2021)"},{"why":"Provides the revised CUB labels and GTSRB ontology labels the evaluation relies on.","marker":"de Sousa Ribeiro et al. (2025a)"}],"fun_headline_variants":["One score finds the best layer for concept probes","Skip probe training: score picks the right layer","Auto-pick the layer where concepts shine","Informativeness + regularity = layer choice","Select probe layers without training probes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the estimated mutual-information scores rank layers correctly even for very high-dimensional representations; if those estimates are noisy or biased, the selected layer can be wrong even when the regularity term is measured well.","fun_headline_variants_meta":{"raw":{"variants":["One score finds the best layer for concept probes","Skip probe training: score picks the right layer","Auto-pick the layer where concepts shine","Informativeness + regularity = layer choice","Select probe layers without training probes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1260,"prompt_tokens":931,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":547,"tokens_out":329,"duration_ms":3404,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:11:19.705334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the layer ranking for one of the large ResNet50 models (CUB or ImageNet) using a different mutual-information estimator, or the same estimator with several smoothness values, and train probes on the layers that win under each ranking. If the ranking flips to a layer whose probe accuracy is substantially below the reported about-90-percent level, or if the reported layer no longer beats the layers-average baseline, the selection criterion is not robust.","supporting_citations":[],"review_version":2}