{"id":"31bf4cac-900c-4c94-8dee-a9ea107e9f5e","arxiv_id":"2508.19613","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ALSA learns anchors in logit space and uses their influence on unlabeled samples to estimate model accuracy under distribution shift.","lead":"This paper introduces ALSA, a method that estimates a machine learning model's accuracy on unlabeled data under distribution shift by learning reference points, or anchors, in the model's logit space. If the estimates are reliable, it gives practitioners a way to monitor deployed models without collecting labels on new test data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ALSA's target estimate inherits source-fitted p(correct|z); no benchmark isolates a shift that breaks this conditional, so the 'wide range' claim is under-tested.","rationale":"The reader's weakest assumption and my concern coincide. The method's entire transfer step is the fit of anchors on source labels applied to target logits; if p(correct|z) is not invariant, there is no mechanism in ALSA to correct for it. I do not think this invalidates the empirical contribution; the paper is honest about the assumption and has released code. But the abstract/conclusion claim of robustness across a 'wide range of distribution shifts' overreaches until the boundary is mapped. A label-shift experiment is the cheapest way to map it. I also checked for internal inconsistencies (e.g., the rectification condition in Eq. (8) vs the wording in Appendix E), but those are secondary; this conditional-invariance issue is the load-bearing one. Because the paper already discloses the assumption and the empirical trends are consistent with it, I recommend keeping the reader's CONDITIONAL verdict rather than upgrading to acceptance.","tokens_in":19034,"tokens_out":7816,"duration_ms":93460,"concrete_test":"Use a CIFAR-10 ResNet18 trained on the paper's imbalanced source, then build target sets (a) containing only classes 0-4 and (b) reweighting class frequencies 10:1, keeping labels hidden from ALSA. Compute ALSA-G MAE vs ATC/AC on these targets. If ALSA's MAE rises substantially above its CIFAR-10.1/10.2 values while a conditional-shift-respecting method does not, the p(correct|z) invariance is the operative boundary. For a direct mechanism check, on the same targets bucket logits by the learned anchors and compare per-bucket empirical correct rates in source validation vs target; a mismatch in buckets where ALSA overestimates confirms the bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ALSA's target estimate in Eq. (7) is a plug-in for E_{P_T}[p_S(correct|Z)]: anchors are fit on source validation via Eq. (9), and Eqs. (4)/(8) map target logits through that source-fitted function. This is unbiased only if p_T(correct|z) = p_S(correct|z) on the target logit support. The paper acknowledges this in Appendix I.3 as an assumption that 'may become less valid when corruption levels are extremely severe,' but it is the method's core invariance, not a peripheral caveat. The reported benchmarks mostly change p(x) (CIFAR-10C, MNIST-M, Office-31) or change group frequencies while keeping the labeling function stable (WILDS); the mild class-imbalance training in H.2 is the closest to label shift, at only 1:3. No experiment forces a genuine p(y|z) change, e.g., a target whose label support or prior is very different from source. Under such a shift, anchors learned from source class-balance encode the wrong conditional, so the average in Eq. (7) is biased even if the marginal logit distribution is perfectly matched. Appendix I.3's observed overestimation on extreme corruptions is exactly this bias. The central claim should therefore be scoped to shifts with approximate conditional alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ALSA estimates the accuracy of a classifier on an unlabeled target dataset by placing k learnable anchors in logit space, each with a peak value and variance, and defining the per-sample correctness probability as the sigmoid of the sum of Gaussian-like or exponential influence functions. The anchors are trained on a labeled source validation set by binary cross-entropy (Eq. (9)); the target accuracy estimate is the average of these probabilities over target logits (Eq. (7)), with a rectification for low-influence regions. The method is evaluated on vision, language, and graph benchmarks—CIFAR-10/100C, CIFAR-10.1/10.2, ImageNetV2, MNIST-M, Office-31, Waterbirds/Amazon WILDS, and ogbn-arxiv—across multiple architectures, reporting MAE and correlation against baselines (AC, DoC, IM, ATC, COT, GNNEval). Code is released.","tokens_in":19377,"tokens_out":6915,"duration_ms":74110,"significance":"If the empirical results hold, ALSA is a practical, unified OOD accuracy estimator with a favorable accuracy/efficiency trade-off. The paper's strengths include a broad multi-modal evaluation, ablations of learnable components, sensitivity analyses for alpha and anchor count, and a complexity analysis; the release of code is a plus. The main limitation is that the method's core assumption—that p(correct|z) transfers from source to target—is only tested indirectly, and one proposition is not proven for trained networks. The contribution is significant for the accuracy-estimation subfield if that assumption and the reported results are made explicit and properly scoped.","major_comments":[{"comment":"The target estimator is a plug-in for E_{P_T}[p_S(correct|Z)]: anchors are fit on source validation via Eq. (9), and Eq. (7) evaluates that source-fitted function on target logits. The method is therefore unbiased only if p_T(correct|z)=p_S(correct|z) on the support of the target logits. The paper acknowledges this in Appendix I.3 but only as a caveat for extreme corruptions. None of the main benchmarks induces a genuine conditional shift: CIFAR-C, MNIST-M, and Office-31 are covariate shifts; WILDS changes group frequencies with a stable labeling function; the class-imbalance training in H.2 (1:3) is mild. Thus the claim of robustness 'across a wide range of distribution shifts' is under-tested. I recommend adding an experiment that changes the label prior or labeling function (e.g., label shift with rebalanced target classes, or a synthetic shift that changes class confusability) and sc","section":"Section 3.3 / 3.4, Appendix I.3, H.2"},{"comment":"Proposition 1 derives a bandwidth bound for a randomly initialized linear layer with W ~ N(0, sigma_W), b=0, and Xavier initialization. The text then argues that training updates are confined to directions orthogonal to the all-ones vector, but no proof shows that the magnitude bound (Bandwidth < 4z sqrt(1/(m+c))) persists after training. Since ALSA is applied to trained networks, the proposition as stated does not cover the setting of interest. It should either be explicitly labeled as an initialization-only observation or supplemented by an empirical/theoretical argument for trained networks.","section":"Section 2.2 / Appendix B"},{"comment":"There is an inconsistency in the rectification condition. Eq. (8) defines insufficient influence as |Infl(z_i, A)| < t, where Infl(z_i, A) is the total influence from all anchors, while Appendix E defines it as |Infl(z_i, (a_j,p_j,v_j))| < t for all j individually. These are different conditions and can lead to different predictions. In addition, Section 3.3 leaves t in terms of a per-anchor p, while Appendix E fixes p=6 to obtain a unified threshold; anchors with smaller learned p are then treated with the same threshold. The paper should state which condition is implemented and justify the unified p=6 threshold given that p is a learned parameter.","section":"Section 3.3 vs Appendix E"},{"comment":"The paper does not report the alpha value used for Tables 1 and 2. This hyperparameter controls when low-influence logits are replaced by 1/c and is shown in Appendix J to affect MAE substantially at extreme values. For reproducibility, the alpha values (and anchor counts) for each benchmark should be listed in the main text or in an experiment-setting table. The code release is helpful, but the paper itself should specify the default configuration.","section":"Appendix J / main experiments"}],"minor_comments":[{"comment":"Eq. (6) defines an 'initial' accuracy estimate before rectification, and Eq. (7) the final one. Section 3.4's stopping criterion uses p_true from Eq. (6) but does not explicitly state whether p_rec from Eq. (7) is used during training. Please clarify.","section":"Section 3.3 / 3.4"},{"comment":"ALSA obtains R^2 = -0.32 and Pearson r = 0.46 on GIN, which is lower than several baselines' correlation coefficients. The sentence 'consistently achieves high correlation scores across models' is too strong and should be qualified, or the authors should explain why R^2 is negative (e.g., small variance in true accuracy across test years).","section":"Table 3, GIN row"},{"comment":"Office-31 results have very large standard deviations (e.g., 15.29 for AC, 33.25 for ATC). Consider reporting per-domain results separately or discussing this instability, since the averaged MAE may be dominated by a few seeds/domains.","section":"Table 1, Office-31"},{"comment":"Figure 11 reports MSE (mean squared error) while Tables 1 and 2 use MAE. Please use one metric consistently to avoid confusion.","section":"Figure 11"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about conditional alignment is real and should be addressed in revision. The paper is a solid empirical contribution if the claims are scoped to shifts where p(correct|z) is approximately stable, the missing alpha values are reported, and the inconsistency between Eq. (8) and Appendix E is resolved. I recommend major revision rather than rejection because the central methodology is defensible and the reported results are strong on the benchmarks considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: ALSA is worth taking seriously. It learns a set of anchors in logit space to estimate per-sample correctness probability, then averages them to predict accuracy on an unlabeled target set. That mechanism is not in the cited prior work, and it works well across vision, language, and graph benchmarks. Code is released, standard deviations are reported, and ablations are honest. The practical value is real: a single, modality-agnostic tool for monitoring deployed models under shift is something people need, and ALSA is simple enough to actually use.\n\nThe main contribution is empirical. On Waterbirds and Amazon-WILDS the gains over softmax baselines are large, and even on CIFAR-10C and CIFAR-100C it stays competitive while the baselines degrade. The anchor visualization and the ablation making all parameters learnable do help build the case. I also appreciate that the authors disclose the key assumption in Appendix I.3: the method requires approximate alignment of the source and target conditional distributions p(y|z). That is exactly the load-bearing point, and they put it on the table, even if they bury it in an appendix.\n\nNow the soft spots, in proportion. First, the theoretical motivation is shaky. Proposition 1 is proven for random initialization only, then asserted to hold for trained networks without a rigorous argument. That is a real gap, though the empirical structure is clearly there in the figures. Second, the main results never state the default value of the confidence interval alpha; Appendix J sweeps alpha but does not say which value produced Tables 1 and 2. That is a genuine reproducibility omission. Third, several strong baselines from the related work—ProjNorm, Agreement-on-the-line, AutoEval—are not compared. Fourth, the text claims \"consistently high correlation\" but Table 3 shows ALSA with rho=0.46 on GIN, worse than AC's 0.85. That overclaim should be trimmed.\n\nOn the stress-test point: I agree that no benchmark isolates a shift that genuinely breaks p(y|z). The corruption experiments only degrade p(x); the WILDS shifts reweight groups with a stable labeling function; the imbalance experiments are mild label priors. So the \"wide range\" claim is under-tested, exactly as the appendix's observed overestimation on extreme corruptions predicts. This is a scoping issue, not a fatal one—most real deployments are closer to covariate shift than to arbitrary conditional shift.\n\nFor whom is this? Anyone working on OOD accuracy estimation, model monitoring, or reliability under distribution shift. It deserves a serious referee: the method is clean, the evaluation is broad, and the weaknesses are fixable. I would accept it for review with a request to tighten the theory and report the missing hyperparameter.\n\nRead it, cite it if you work in the area, and bring it to reading group if you want a concrete example of an empirical paper that is honest about its core assumption.","headline":"ALSA is a practical, empirically strong accuracy estimator that is genuinely new in mechanism, but its theory is thin and its 'wide range' claim runs ahead of what the benchmarks actually test.","tokens_in":19834,"tokens_out":1937,"would_cite":true,"duration_ms":22563,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A handful of learnable anchors in logit space can estimate a model's accuracy on unlabeled, shifted data without any target labels.","keywords":["out-of-distribution accuracy estimation","logit space","learnable anchors","distribution shift","unlabeled model evaluation","softmax information loss","class imbalance"],"falsifier":"Construct a target set by taking source validation logits and relabeling a contiguous region of logit space as incorrect while leaving the marginal logit distribution unchanged, so that p(correct|z) differs between source and target. ALSA, whose anchors were trained on the unaltered source labels, would then estimate accuracy by averaging correctness probabilities from the source surface, and the deviation between its estimate and the relabeled target accuracy would be roughly the probability mass of the relabeled region.","tokens_in":18950,"feed_emoji":"🎯","tokens_out":4632,"duration_ms":50974,"temperature":0.7,"pith_summary":"This paper tries to establish that a model's accuracy on an unlabeled, out-of-distribution dataset can be estimated by looking at where its logits fall in a low-dimensional band of logit space. The key claim is that logits carry more information about whether a prediction is correct than softmax scores do, because softmax throws away magnitude and class-specific structure. The proposed method, ALSA, places learnable reference points, called anchors, in logit space; each anchor exerts a distance-decaying influence, and the total influence is converted into a per-sample probability of correctness. Averaging those probabilities over the unlabeled target set yields an accuracy estimate. The paper reports consistent gains over softmax-based and similarity-based baselines across vision, language, and graph benchmarks under several kinds of distribution shift.","feed_headline":"Logit anchors forecast model accuracy on shifted data","feed_subtitle":"ALSA works directly in logit space, beating softmax and similarity baselines on vision, language, and graph shifts.","key_machinery":"The central object is a set of anchor triplets (position a, peak influence p, variance v) in logit space, combined through a Gaussian-like influence function that decays with cosine distance from the anchor. The total influence from all anchors is mapped by a sigmoid into a per-sample probability of correct prediction, and the average of these probabilities is the accuracy estimate. The band-width proposition justifies the geometric assumption that logits concentrate near a hyperplane, while the rectification threshold, derived from a confidence interval through the inverse error function, handles low-influence regions where the estimate would otherwise revert to an uninformative 50%.","core_discovery":"The paper claims that a classifier's logits lie in a narrow band around a (c-1)-dimensional hyperplane perpendicular to the all-ones vector, and that within that band a logit's position carries reliable information about whether the prediction will be correct. ALSA formalizes this by initializing k learnable anchors in logit space, each defined by a position vector, a signed peak influence, and a variance controlling how far its influence reaches. The total influence from all anchors is passed through a sigmoid to give the probability that the corresponding sample is correctly predicted. The estimated accuracy on an unlabeled target set is the average of these per-sample probabilities, with","pith_inferences":["Beyond the paper, the learned anchor surface could double as a principled confidence or selective-classification score, since it assigns a calibrated correctness probability to every logit region rather than a single global threshold.","The disclosed assumption that p(correct|z) aligns between source and target suggests ALSA should be stress-tested under explicit label shift, where the same logit region may have different correctness rates; the paper's benchmarks cover covariate-like shifts more than pure label shifts.","If anchor positions and peak values converge to the same distribution regardless of initialization, as the paper argues, then the anchor set is effectively approximating an underlying correctness surface, which would make ALSA a candidate building block for other logit-based analyses such as OOD detection.","The influence-based aggregation could be extended to produce per-class accuracy estimates, which the current average-based estimate does not directly expose, potentially aiding debugging of skewed classifiers."],"forward_implications":["If ALSA's central claim holds, deployed models can be monitored on unlabeled test data by averaging per-sample correctness probabilities derived from logit positions, without needing target labels.","Avoiding softmax compression preserves logit magnitude information, which should improve estimates in settings where class distributions are imbalanced and where accuracy differs across regions of logit space.","The same anchor mechanism transfers across vision, language, and graph models, suggesting a general-purpose estimator rather than a domain-specific similarity score.","Because inference scales linearly with the number of samples and anchors, the method could be used in production monitoring with modest compute overhead.","ALSA is reported to remain accurate even when true accuracy drops by over 30% under synthetic corruptions, and to keep mean absolute error below the levels where several baselines fail on at least one dataset."],"supporting_citations":[{"why":"Defines ATC and the general unlabeled accuracy-estimation task; ALSA's main softmax-based baseline and the comparison point for threshold-based estimation.","marker":"[10]"},{"why":"Defines Average Confidence and Difference of Confidence baselines that operate on softmax scores, motivating ALSA's logit-space approach.","marker":"[16]"},{"why":"Provides the Confidence Optimal Transport baseline that estimates OOD error from optimal transport cost on softmax distributions.","marker":"[34]"},{"why":"Exemplifies similarity-based accuracy estimation via Fréchet distance, the computationally expensive alternative ALSA aims to avoid.","marker":"[5]"},{"why":"Provides a rotation-prediction similarity-based baseline for accuracy estimation, used as a comparison approach.","marker":"[6]"},{"why":"Supplies the graph-specific GNNEvaluator baseline that ALSA must outperform on ogbn-arxiv.","marker":"[52]"},{"why":"Supplies the Xavier initialization condition used in Proposition 1 to bound the width of the logit band.","marker":"[13]"},{"why":"Provides the WILDS benchmark framework and the subpopulation shift datasets Amazon-WILDS and Waterbirds-WILDS used for evaluation.","marker":"[28]"},{"why":"Provides the CIFAR-10C and CIFAR-100C synthetic corruption benchmarks used to test estimation under progressive corruption severity.","marker":"[21]"},{"why":"Provides the ImageNetV2 natural shift benchmark used to evaluate accuracy estimation on non-simulated distribution changes.","marker":"[40]"}],"fun_headline_variants":["Logit anchors forecast accuracy on shifted, unlabeled data","Anchors in logit space beat softmax for OOD accuracy","ALSA: Estimate model accuracy from logit geometry alone","Forget softmax: logit anchors estimate shifted-data accuracy","Logit position predicts correctness—ALSA turns it into accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The anchors are learned on the source validation set and applied to the target set, so ALSA works only if the probability of a correct prediction given a logit vector is approximately the same in source and target; the paper notes this can break down under extremely severe corruptions.","fun_headline_variants_meta":{"raw":{"variants":["Logit anchors forecast accuracy on shifted, unlabeled data","Anchors in logit space beat softmax for OOD accuracy","ALSA: Estimate model accuracy from logit geometry alone","Forget softmax: logit anchors estimate shifted-data accuracy","Logit position predicts correctness—ALSA turns it into accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":1813,"prompt_tokens":757,"completion_tokens":1056,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":970}},"tokens_in":501,"tokens_out":1056,"duration_ms":11580,"temperature":1.0,"reasoning_tokens":970,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:38:05.191157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a target set by taking source validation logits and relabeling a contiguous region of logit space as incorrect while leaving the marginal logit distribution unchanged, so that p(correct|z) differs between source and target. ALSA, whose anchors were trained on the unaltered source labels, would then estimate accuracy by averaging correctness probabilities from the source surface, and the deviation between its estimate and the relabeled target accuracy would be roughly the probability mass of the relabeled region.","supporting_citations":[{"cited_title":"Leveraging unlabeled data to predict out-of-distribution perfor- mance","cited_arxiv_id":null,"evidence_quote":"Defines ATC and the general unlabeled accuracy-estimation task; ALSA's main softmax-based baseline and the comparison point for threshold-based estimation."},{"cited_title":"Predicting with confidence on unseen distributions","cited_arxiv_id":null,"evidence_quote":"Defines Average Confidence and Difference of Confidence baselines that operate on softmax scores, motivating ALSA's logit-space approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Confidence Optimal Transport baseline that estimates OOD error from optimal transport cost on softmax distributions."},{"cited_title":"Are labels always necessary for classifier accuracy evaluation? In CVPR, 2021","cited_arxiv_id":null,"evidence_quote":"Exemplifies similarity-based accuracy estimation via Fréchet distance, the computationally expensive alternative ALSA aims to avoid."},{"cited_title":"What does rotation prediction tell us about classifier accuracy under varying testing environments? In ICML, 2021","cited_arxiv_id":null,"evidence_quote":"Provides a rotation-prediction similarity-based baseline for accuracy estimation, used as a comparison approach."},{"cited_title":"Gnnevaluator: Evaluating GNN performance on unseen graphs without labels","cited_arxiv_id":null,"evidence_quote":"Supplies the graph-specific GNNEvaluator baseline that ALSA must outperform on ogbn-arxiv."},{"cited_title":"Understanding the difficulty of training deep feed- forward neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the Xavier initialization condition used in Proposition 1 to bound the width of the logit band."},{"cited_title":"Haque, Sara M","cited_arxiv_id":null,"evidence_quote":"Provides the WILDS benchmark framework and the subpopulation shift datasets Amazon-WILDS and Waterbirds-WILDS used for evaluation."},{"cited_title":"Dietterich","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-10C and CIFAR-100C synthetic corruption benchmarks used to test estimation under progressive corruption severity."},{"cited_title":"Do ima- genet classifiers generalize to imagenet? In ICML, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the ImageNetV2 natural shift benchmark used to evaluate accuracy estimation on non-simulated distribution changes."}],"review_version":1}