{"id":"465c865e-c3b5-4d04-b589-6d8db43fae2a","arxiv_id":"2506.06048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"TRUST computes confidence as the angular distance between a test image and a slightly modified, maximally-confident version of it, and claims this ranks predictions monotonically.","lead":"This paper introduces TRUST, a test-time confidence score that measures how far an image is from the nearest typical example of its predicted class, by slightly modifying the image until the classifier is very sure. The authors argue this score ranks predictions more reliably than dropout-based uncertainty and can flag out-of-distribution inputs, but the method is slow and lacks a rigorous proof that the modified image is truly a cluster center.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline monotonicity claim is tested on selected easier classes for TinyImageNet and ImageNet, not on the full test populations, so the population-level claim is not yet supported.","rationale":"The reader's conditional verdict is appropriate: the paper offers an interesting test-time confidence method and shows monotonic accuracy on CIFAR-10 and CAMELYON-17, but the evidence is not complete. I do not think the mode-projection assumption is the single most load-bearing issue, because even if the optimization does not provably land on a feature-space mode, the empirical ranking could still be useful. The more direct threat to the stated claim is that the two large-scale benchmark results are reported on selected, easier class subsets rather than full test populations. The abstract and contribution sections claim 'population accuracy consistently increases' and 'four benchmark datasets,' which is stronger than what the experiments actually support. A concrete full-dataset rerun would settle this. I therefore keep the reader's CONDITIONAL verdict unchanged, but I flag the class-selection issue as the condition that most urgently needs verification.","tokens_in":16566,"tokens_out":7971,"duration_ms":84210,"concrete_test":"Re-run the TinyImageNet and ImageNet evaluations on the complete test sets (all classes) using the same models and the same Dropout, ViM, SIRC, and LogitNorm baselines, and recompute the top-k accuracy curves and AURC/AUSE. Also state explicitly whether the class-selection filter was applied to every baseline. If monotonicity fails on any full test set, or a baseline achieves lower AURC/AUSE than CrossEntro+TRUST, the central claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TRUST defines a monotonic subset-selection function over the test population and outperforms baselines on 'four benchmark datasets' is only tested on subpopulations for two of those datasets. Section 4.1.1 states: 'For TinyImageNet, we selected classes with >=60% accuracy (overall: 65.19%) for further analysis. For ImageNet, we randomly chose 100 classes with moderate accuracy (80-90%).' Table 1 then reports TinyImageNet and ImageNet rows built from these subsets but presents them as full-dataset results. Selecting only classes where the model already has moderate-to-high accuracy removes exactly the hard, low-confidence samples that a reliability ranking must handle, so the observed monotonic accuracy and favorable AURC/AUSE on these rows may not survive on the full test population. This is load-bearing because two of the four datasets in the paper's empirical support depend on it, and the reported numbers are not comparable to baselines evaluated on full test sets unless the same class filter was applied to every method, which is not stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TRUST, a test-time reliability score computed as the cosine distance between a test sample's penultimate-layer feature and the feature of an optimized counterfactual point obtained by minimizing cross-entropy plus an L1 input perturbation. The authors argue that this optimization projects the test sample to the nearest mode of its predicted class's micro-cluster, mitigating the effect of noisy classifier weights on uncertainty estimates. They report that the score produces monotonic accuracy stratification, improves AURC/AUSE over baselines on CIFAR-10, CAMELYON-17, TinyImageNet, and ImageNet, detects OOD inputs, and exposes differences between CNN and ViT feature geometries. Theoretical results are provided on high-dimensional concentration, a noise-robustness bound for cosine scoring, and ablations on temperature, regularization, and convergence.","tokens_in":16782,"tokens_out":7358,"duration_ms":65649,"significance":"If the central geometric assumption is valid, the proposal is a novel and practically interesting alternative to sampling- or retraining-based uncertainty estimators: it adds computation at test time rather than modifying training, and it is orthogonal to training losses such as LogitNorm. The paper is honest about its main limitation (per-sample cost in seconds; effectiveness diminishes for small architectures). A valuable positive is that the evaluation is post-hoc with fixed hyperparameters, so the reported gains are not fitted to the benchmark metrics. However, the current evidence does not yet support the population-level monotonicity claim because two of the four datasets are evaluated on accuracy-filtered class subsets, and the theoretical link between Eq. (1) and a feature-space mode is asserted rather than proved. The paper would be strengthened by full-population evaluation and by a synthetic or low-dimensional verification of the mode-projection property.","major_comments":[{"comment":"The TinyImageNet and ImageNet rows of Table 1 are computed on restricted class subsets (classes with ≥60% accuracy; a random subset of 100 classes with 80–90% accuracy), yet Section 5.1 and the abstract present these as evidence that TRUST defines a monotonic subset-selection function over the test population and outperforms baselines on four benchmark datasets. This filter removes hard, low-confidence samples—exactly the cases a reliability score must rank—so the observed monotonicity and favorable AURC/AUSE on those rows may not extend to the full test population. Moreover, unless the same class filter was applied to all baselines (which is not stated), the reported numbers are not comparable to the baselines' full-test-set results.","section":"4.1.1, Table 1"},{"comment":"The paper's central mechanistic claim is that minimizing Eq. (1) 'projects x_test to its nearest mode' (Section 3.1), so that the cosine distance between x_test and the optimized point equals epistemic uncertainty. No proof is given that the stationary point of Eq. (1) is a mode of the predicted class's feature distribution, nor is it shown that the high-temperature softmax preserves micro-cluster membership. Since the TRUST score's meaning depends entirely on this equivalence, the claim is unsupported as it stands. A concrete fix would be to validate the mode-projection property on a synthetic mixture of Gaussians or on a low-dimensional learned subspace, and to compare the optimized features with the actual density maxima of the training features.","section":"3.1, Eq. (1)"},{"comment":"The theoretical noise-robustness result is not connected to the actual algorithm. Theorem 3 assumes a Gaussian perturbation on the angle ω and shows that cosine scoring has lower effective variance than direct noise, but the paper does not bound the estimation error of the mode obtained from Eq. (1), nor does it show that this error follows the assumed noise model. The proof also relies on a first-order Taylor approximation and a variance comparison; without an additional argument, a variance ordering does not by itself establish the claimed ordering of sorting-error probabilities. Consequently, the theoretical section does not yet substantiate the paper's claim that test-time optimization provides superior robustness over direct scoring.","section":"3.2.1, Theorem 3"},{"comment":"The claim of superior performance over state-of-the-art baselines ('state-of-the-art performance in identifying reliable predictions' in Section 1) is only partially supported by the experiments. On CAMELYON-17 only Dropout and CrossEntro+TRUST are reported; on ImageNet no baseline is reported; and on TinyImageNet only Dropout is reported. Without comparing against ViM, SIRC, or LogitNorm (or at least strong softmax/MSP baselines) on the full test sets of those datasets, the cross-dataset superiority claim is not established.","section":"5.1, Table 1"}],"minor_comments":[{"comment":"The manuscript states 'Code is available at LINK'; the placeholder 'LINK' is not a functional repository URL, which harms reproducibility and should be replaced with a real link.","section":"1"},{"comment":"In Step 3 of the proof, solving Φ(t√d) ≈ 2/n² yields t√d ≈ -√(2 ln(n²/2)) ≈ -2√(ln n) for large n, not -√(2 ln n) as stated. The final formula is therefore off by a factor of √2; since the theorem is used only as a qualitative motivation for micro-cluster separation, the main conclusions are unaffected, but the proof should be corrected.","section":"Supplementary, Theorem 5 proof"},{"comment":"The phrase 'defines a monotonic subset-selection function' suggests a proven property, but the paper provides only empirical evidence at a few selected percentile thresholds (20%, 40%, 60%, 80%, 100%). Consider rephrasing to 'exhibits monotonic accuracy stratification at the evaluated thresholds'.","section":"Abstract, 5.1"},{"comment":"The notation in Theorem 3 is confusing: δθ is introduced but δω is used in the statement and proof, and the sentence 'For an equivalent score functions' is grammatically incorrect. Clarifying the noise model would improve readability.","section":"3.2.1, Theorem 3"},{"comment":"The sentence 'For TinyImageNet, we selected classes with ≥60% accuracy (overall: 65.19%)' should specify whether 65.19% is the model's full-test accuracy or the subset's accuracy, to avoid ambiguity when interpreting Table 1.","section":"4.1.1"}],"recommendation":"major_revision","confidential_remarks":"The subpopulation evaluation on TinyImageNet and ImageNet is the most serious empirical issue, because the headline population-level claim rests on it and the reported numbers are presented as full-dataset results. The unsupported equivalence in Eq. (1) is a fundamental theoretical gap that will require either a proof or a substantially weakened claim. These are fixable within the manuscript's scope, so major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely new idea: at test time, perturb the input to raise the predicted class's softmax probability under a high temperature, then use the cosine distance in feature space between the original and perturbed point as a reliability score. That's distinct from prior distance-to-training-density measures, and the CIFAR-10 results are clean enough to take seriously. The ablations on T and lambda show the score is not overly sensitive to hyperparameters, and the MMD correlation between accuracy drop and TRUST-score shift under corruptions is a nice observation.\n\nThe problems start with the evaluation. Table 1 reports TinyImageNet and ImageNet rows as if they were full test sets, but Section 4.1.1 says TinyImageNet was filtered to classes with >=60% accuracy and ImageNet to 100 classes with 80-90% accuracy. That removes exactly the hard, low-confidence samples a reliability score is supposed to rank. The abstract's claim of a \"monotonic subset-selection function over the population\" is therefore not supported for two of the four datasets. This is load-bearing, not cosmetic.\n\nThe theoretical story is also weaker than the writing suggests. Equation 1 is an adversarial perturbation toward the predicted class, not a derived projection to a feature-space mode. No proof shows the stationary point is a mode or that high-temperature softmax preserves micro-cluster membership. Theorem 3 bounds sorting errors under generic Gaussian noise but never connects that noise model to the actual optimization error. The supplementary proof of Theorem 5 contains a factor-of-sqrt(2) slip in the extreme-value calculation; minor, since it's only motivation, but it's there. Code is promised only as a \"LINK\" placeholder, and the method costs several seconds per sample.\n\nSome of the comparison is also thin: TinyImageNet and ImageNet have at most one baseline (Dropout) alongside TRUST, so the SOTA claims rest heavily on CIFAR-10 and CAMELYON-17. And on CIFAR-10, the Dropout baseline is arguably also monotonic, undercutting the contrast they draw.\n\nWho is this for? Researchers working on test-time uncertainty or selective classification will want to know about the recipe, but they should not cite the numbers without re-running on full test sets. The idea is interesting enough that a serious referee should see it, and the flaws are addressable in revision.\n\nMy recommendation: send to peer review, but insist on full test-set evaluation, released code, and a softened interpretation of the mode-projection claim.","headline":"A promising test-time uncertainty idea whose headline claim is only demonstrated on easy class subsets for two of four datasets, and whose mode-projection interpretation is asserted, not proven.","tokens_in":17308,"tokens_out":4891,"would_cite":false,"duration_ms":46296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRUST ranks predictions by the distance of each test image to its predicted class's feature-space center, and accuracy rises as low-scoring samples are removed.","keywords":["uncertainty estimation","test-time optimization","selective classification","epistemic uncertainty","out-of-distribution detection","confidence scoring","feature-space geometry","neural collapse"],"falsifier":"Take a trained model and a held-out test set; compute TRUST scores, then independently estimate each predicted class's true feature-space mode (for example by density estimation over training features or by averaging near neighbors). If the cosine distance from test samples to their TRUST-optimized points does not rank-match the distance to the independently estimated nearest mode for a substantial fraction of samples, the score is not measuring what the derivation assumes.","tokens_in":16339,"feed_emoji":"🎯","tokens_out":6589,"duration_ms":59163,"temperature":0.7,"pith_summary":"This paper claims that a reliability score computed at test time—by slightly perturbing each test image until it lands on the nearest dense center ('mode') of its predicted class in the model's feature space, then measuring the cosine distance to that mode—ranks predictions by true accuracy. The authors argue that classifier weights carry noise that corrupts standard confidence estimates, but that this noise can be bypassed with extra computation per sample. On CIFAR-10, TinyImageNet, CAMELYON-17, and ImageNet, they report that accuracy rises monotonically as lower-scoring samples are removed, and that TRUST improves on dropout, ViM, SIRC, LogitNorm, and density-aware baselines under risk-based metrics. They also report that the gap between training and test TRUST distributions tracks accuracy drops under corruption and detects out-of-distribution data. If true, the score gives a way to decide which predictions to trust without retraining the model.","feed_headline":"New score ranks AI predictions by true reliability at test time","feed_subtitle":"Per-image optimization toward a class's feature center flags misclassified and out-of-distribution samples before deployment.","key_machinery":"The score's engine is the test-time optimization in Eq. 1: $\\arg\\min_{\\Delta x} L(x_{\\mathrm{test}}+\\Delta x, y_{\\mathrm{test}})+\\lambda\\|\\Delta x\\|_1$, run with an elevated softmax temperature $T$, whose converged input $x_{\\mathrm{test}}^{\\mathrm{mode}}=x_{\\mathrm{test}}+\\Delta x$ is taken to be the nearest mode of the predicted class's micro-cluster. The TRUST score is then the cosine distance between the final-layer features $f^l_\\theta(x_{\\mathrm{test}})$ and $f^l_\\theta(x_{\\mathrm{test}}^{\\mathrm{mode}})$. The surrounding argument uses hypersphere concentration (Theorem 1), near-orthogonality of high-dimensional points (Theorem 2 and Corollary 1), and a noise-robustness bound (Theorem 3) showing that cosine-distance noise is damped by a factor $\\sin^2(\\omega)$ relative to direct score noise.","core_discovery":"TRUST (Test-time Resource Utilization for Superior Trustworthiness) is a sample-specific reliability score defined as the cosine similarity between the final-layer feature vector of a test image and the feature vector of that image after a constrained test-time optimization. The optimization minimizes cross-entropy plus an L1 penalty on the input perturbation, with a high softmax temperature, to move the image to its predicted class's nearest feature-space mode while keeping the change sparse. The paper argues that under high-dimensional concentration of measure, training points form well-separated micro-clusters around class modes, so the angular distance to the nearest mode is a proxy for epistemic uncertainty; it proves that noisy estimation of this cosine distance causes no more sorting errors than the same noise applied directly to an uncertainty score. Empirically, TRUST defines a monotonic subset-selection function: on every tested dataset and architecture, accuracy increases as lower-scored samples are removed, and it outperforms the compared baselines on AURC and AUSE while also separating in-distribution from out-of-distribution data and exposing train-test misalignment.","pith_inferences":["Because the paper observes early convergence within a few hundred epochs, the per-sample computational cost may be reducible well below the reported several seconds per image, making batch or interactive deployment more practical than the stated limitation suggests.","The score is a pure geometric distance on learned features, so the same construction may transfer to other modalities with learned feature spaces, even though the paper only tests image classifiers.","The monotonicity claim suggests an internal consistency check: if a model's TRUST-sorted accuracy is flat or decreasing, that may indicate its feature space lacks the assumed micro-cluster geometry.","The paper's own observation that TRUST's effectiveness diminishes with smaller architectures points to a testable boundary: the method should be expected to help only once a model's feature space is high-dimensional enough for the concentration arguments to apply."],"forward_implications":["On all four datasets tested, retaining only the top 20% of TRUST-ranked samples lifts accuracy well above full-set accuracy, so TRUST can be used for selective classification.","Because TRUST improves LogitNorm-trained models further, it complements training-time confidence methods rather than replacing them.","The train-test TRUST distribution gap (measured by MMD) correlates with accuracy drop, giving a signal that a model or test distribution has shifted.","TRUST scores separate in-distribution from out-of-distribution samples without needing OOD data or retraining.","TRUST's monotonicity holds across CNN and ViT architectures and across feature layers, with the final layer giving the best stratification."],"supporting_citations":[{"why":"Supplies the neural-collapse observation that final-layer representations concentrate per class, motivating the micro-cluster topology.","marker":"[37]"},{"why":"Establishes that distance-to-high-density-region measures capture epistemic uncertainty better than standard scores, the basis TRUST extends.","marker":"[24]"},{"why":"Provides the LogitNorm training method that TRUST builds on and compares against, and on which TRUST gives further gains.","marker":"[46]"},{"why":"Provides the Monte Carlo dropout baseline used as the main comparison for uncertainty-based ranking.","marker":"[16]"},{"why":"Provides the ViM baseline whose OOD and stratification performance TRUST is compared against.","marker":"[9]"},{"why":"Provides the SIRC baseline for selective classification that TRUST is compared against.","marker":"[10]"},{"why":"Provides the density-aware evidential baseline for confidence estimation compared in the main results.","marker":"[11]"},{"why":"Defines AURC, the risk-coverage metric used to evaluate ranking quality.","marker":"[47]"},{"why":"Defines AUSE, the area-under-sparse-error metric used to evaluate ranking quality.","marker":"[32]"}],"fun_headline_variants":["Test-time tweak sharpens AI confidence scores","Reliability score for AI predictions, refined at test time","Optimize at test time for reliable AI confidence","Adjust AI confidence at inference for trustworthy scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole score rests on the assumption that the test-time optimization moves each test image to the nearest dense center of its predicted class in the model's feature space, rather than to some other point.","fun_headline_variants_meta":{"raw":{"variants":["Test-time tweak sharpens AI confidence scores","Reliability score for AI predictions, refined at test time","Optimize at test time for reliable AI confidence","Adjust AI confidence at inference for trustworthy scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001104,"raw_usage":{"total_tokens":4575,"prompt_tokens":886,"completion_tokens":3689,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":3629}},"tokens_in":502,"tokens_out":3689,"duration_ms":24348,"temperature":1.0,"reasoning_tokens":3629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T06:01:35.597154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained model and a held-out test set; compute TRUST scores, then independently estimate each predicted class's true feature-space mode (for example by density estimation over training features or by averaging near neighbors). If the cosine distance from test samples to their TRUST-optimized points does not rank-match the distance to the independently estimated nearest mode for a substantial fraction of samples, the score is not measuring what the derivation assumes.","supporting_citations":[{"cited_title":"Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay","cited_arxiv_id":"2309.04644","evidence_quote":"Supplies the neural-collapse observation that final-layer representations concentrate per class, motivating the micro-cluster topology."},{"cited_title":"To trust or not to trust a classifier.Advances in neural information processing systems, 31, 2018","cited_arxiv_id":null,"evidence_quote":"Establishes that distance-to-high-density-region measures capture epistemic uncertainty better than standard scores, the basis TRUST extends."},{"cited_title":"Mitigating neural network overconfidence with logit normalization","cited_arxiv_id":null,"evidence_quote":"Provides the LogitNorm training method that TRUST builds on and compares against, and on which TRUST gives further gains."},{"cited_title":"Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning","cited_arxiv_id":null,"evidence_quote":"Provides the Monte Carlo dropout baseline used as the main comparison for uncertainty-based ranking."},{"cited_title":"ViM: Out-of-Distribution With Virtual-Logit Matching","cited_arxiv_id":null,"evidence_quote":"Provides the ViM baseline whose OOD and stratification performance TRUST is compared against."},{"cited_title":"Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data","cited_arxiv_id":null,"evidence_quote":"Provides the SIRC baseline for selective classification that TRUST is compared against."},{"cited_title":"Uncertainty Estimation by Density Aware Evidential Deep Learning.International Conference on Machine Learning, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the density-aware evidential baseline for confidence estimation compared in the main results."},{"cited_title":"Uncertainty Quantification Metrics for Deep regression.Pattern Recognition Letters, 186, 2024","cited_arxiv_id":null,"evidence_quote":"Defines AUSE, the area-under-sparse-error metric used to evaluate ranking quality."}],"review_version":1}