{"id":"099fd9f3-4dce-4bf5-82a4-5d5fcaa9345d","arxiv_id":"2412.15311","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Class-specific score scaling at test time lets a vanilla ERM model match or surpass debiasing methods and yields a scalar robust-average accuracy trade-off metric.","lead":"This paper shows that a simple post-training adjustment, scaling each class's prediction score by a factor chosen on a small labeled validation set, lets a plain ERM model match or beat specialized debiasing methods on group robustness benchmarks. It also introduces a metric that summarizes the trade-off between average and worst-group accuracy as a single number.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-set overfitting of the greedy scaling search likely inflates ERM+RS gains; FMoW's reported robust coverage drop is direct evidence that selected scaling does not transfer.","rationale":"The paper's central contribution is a training-free post-hoc scaling that, when applied to an ERM model, matches or outperforms debiasing methods. The mechanism is to select scaling factors on a group-annotated validation set by maximizing the evaluation metric. For this claim to hold, the selected factors must generalize to the test set. The reader flagged this as the weakest assumption, and the paper's own FMoW results provide a concrete illustration: the robust coverage of ERM+RS is lower than that of ERM, indicating the scaling frontier deteriorates even though the single selected point improves. This is consistent with the high-capacity greedy search over 401 values per class overfitting the validation set. The authors do provide useful sanity checks (Figure A6, validation-size ablation in Table 5), and the latter actually confirms the sensitivity: IRS degrades substantially with 1% validation. The concern does not refute the method on CelebA/Waterbirds, but it does mean the reported gains, especially for multi-class and time-shifted datasets, may be optimistic unless out-of-sample validation is demonstrated. A two-fold validation split would settle this cleanly. I therefore agree with the reader's conditional verdict and do not move it.","tokens_in":19758,"tokens_out":8293,"duration_ms":73668,"concrete_test":"Select the full greedy scaling vector on one half of the FMoW validation set; evaluate worst-group coverage and accuracy on the other half and on the test set. Compare with in-sample values and with plain ERM (coverage 34.5). If out-of-sample coverage is below 34.5 while in-sample is above, the reported gains are validation overfitting. Also rerun the CelebA greedy search with the grid restricted to n in [-20,20] and check whether the +47.7 worst-group gain persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the class- and cluster-specific scaling vectors obtained by maximizing the target metric on the validation set remain near-optimal on the test set. The paper's transferability evidence (Figure A6) is a 1D sweep on binary CelebA; it does not cover the full C-dimensional greedy vector, the K=20 cluster centroids used by IRS, or time-shifted splits. FMoW-WILDS, where validation and test come from different years, provides a direct internal check: Table 4 reports worst-group robust coverage of 32.9 for ERM+RS versus 34.5 for plain ERM, even though the single selected worst-group accuracy improves (35.7 vs 34.5). This is the signature of validation overfitting—the selected point transfers partially but the trade-off frontier does not. The greedy search over 401 values per class is unregularized, and Table 5 shows IRS is highly sensitive to validation size (worst-group gain drops from +54.2 to +25.9 when validation is cut to 1%), confirming the mechanism. Since the central claim is that ERM+RS matches debiasing methods, inflated validation selection would bias the comparison in favor of ERM+RS.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, post-hoc class-specific logit scaling technique (RS) and an instance-wise variant (IRS) that scales logits per feature cluster, along with a scalar 'robust coverage' metric that summarizes the robust-versus-average accuracy Pareto frontier. Experiments on CelebA, Waterbirds, CivilComments-WILDS, and FMoW-WILDS show that RS applied to an ERM model recovers or exceeds many recent debiasing methods' worst-group accuracy, and IRS improves both robust and average accuracy in several settings. The paper interprets these results as evidence that much of the apparent robustness gain of training-time debiasing methods can be reproduced by test-time score rescaling, and argues that robust accuracy should be evaluated jointly with average accuracy.","tokens_in":20018,"tokens_out":3925,"duration_ms":33863,"significance":"If the claims hold, this is a useful and inexpensive post-processing technique, and the paper's re-evaluation of existing debiasing algorithms is a worthwhile caution for the group-robustness community. The paper ships standard-benchmark experiments across vision and text, reports means and standard deviations over three runs, and includes helpful ablations on validation-set size and cluster count. The robust coverage metric, though dependent on the proposed scaling procedure, is a conceptually useful scalar summary of the robust-average trade-off. The central claims are falsifiable and the experiments are reproducible in principle, which I regard as strengths.","major_comments":[{"comment":"The transferability assumption stated in Section D ('the optimal scaling factor identified in the validation set can be used in the test set') is supported only by the binary-CelebA 1D sweep in Figure A6, not by the full C-dimensional greedy vector or by the K=20 cluster-specific vectors used by IRS. FMoW-WILDS, where validation and test come from different years, provides a direct internal check in Table 4: ERM+RS has worst-group robust coverage 32.9 versus 34.5 for plain ERM, even though the selected worst-group accuracy improves. This is the signature of a validation/test mismatch in the fitted scaling frontier and directly contradicts the sentence in Section 4.2 that robust scaling 'does not suffer from any performance degradation' on FMoW. The authors should report the full FMoW frontier realized on test, and either restrict the transferability claim to datasets without temporal shift or qualify it explicitly.","section":"§3.2, §D, Table 4"},{"comment":"The greedy coordinate search over 401 scaling values per class is an unregularized selection on the validation split, and Table 5 shows the consequence: IRS worst-group gain falls from +54.2 to +25.9 when the validation set is cut to 1%, with the standard deviation rising to 14.4. This is direct evidence that the selection procedure can overfit the validation split. Since the headline comparison (ERM+RS matching debiasing methods) uses validation-selected scaling factors, the paper should report whether the ERM+RS advantage over LfF, JTT, and CR is statistically significant across seeds and, ideally, compare against a baseline that is given an analogous post-hoc selection budget. Without this, the central claim risks being partly an artifact of selection on validation rather than a property of the underlying ERM features.","section":"§3.2, Table 5"},{"comment":"The definition of robust coverage in Eq. (4) takes a max over scaling vectors, but the text does not make explicit whether the reported coverage numbers are computed by selecting s on the validation split and then evaluating the resulting frontier on the test split, or by optimizing s directly on the test split. Table A8 distinguishes 'Robust Coverage' from 'Realized Robust Coverage' but the main tables report only one number. This matters because if the max is evaluated on test, the coverage is an oracle quantity that does not measure what a practitioner would obtain; if it is evaluated on validation, the reported values inherit the transfer problem raised above. The authors should define the split protocol for every coverage number reported in Tables 1–4.","section":"§3.4, Eq. (4), Table A8"}],"minor_comments":[{"comment":"The notation (s ⊙ ŷ)_c is used before the paper defines whether ŷ is a probability vector or a logit vector; please state the convention explicitly, since multiplying probabilities versus logits changes the effect of scaling.","section":"§3.2, Eq. (3)"},{"comment":"The complexity entry 'O(n9)' is presumably meant to be O(n^9) or exponential in the class count; as printed it is unclear, and a polynomial-time claim for full grid search would be inconsistent with searching 401 values per class.","section":"Table A9"},{"comment":"Several display equations have minor formatting issues (e.g., missing spaces around 'where * indicates', and 'na¨ıve' instead of 'naïve'); these should be cleaned up.","section":"Appendix A"},{"comment":"In Figure 5, the claim that validation coverage is 'almost consistent' with test robust accuracy is supported only visually; consider reporting the numerical correlation or per-K values.","section":"§4.3, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core observation is interesting, but the validity of the headline comparison hinges on validation-to-test transfer of the fitted scaling vectors, and the FMoW result in Table 4 is an internal contradiction that needs to be addressed head-on. I would not reject, but the revision needs more than cosmetic changes: the transfer evidence must be extended beyond the CelebA 1D sweep, the robust coverage protocol must be specified per split, and the comparison to baselines should be accompanied by significance or selection-bias checks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new things here are the robust coverage metric and the empirical claim that a vanilla ERM model plus validation-tuned class-specific scaling matches or beats recent debiasing methods. Both are worth taking seriously. The robust coverage is a sensible scalar summary of the robust-average trade-off frontier, and the experiments span four datasets with consistent results. The greedy search is cheap, and the validation-size ablation is a good sanity check.\n\nThe soft spots are real but not fatal. The main concern is the validation-fitting protocol: the scaling vector and cluster centroids are chosen by an unregularized greedy search on the validation set with group labels, so ERM+RS gets a post-hoc optimization step that the baselines do not get in exactly the same way. The paper claims the selected scaling transfers to the test set, but Figure A6 only shows a 1D sweep on binary CelebA, not the full high-dimensional search or the K=20 cluster assignment used by IRS. The FMoW results are the most direct internal check, and they are mixed: worst-group accuracy improves from 34.5 to 35.7 under ERM+RS, but the robust coverage for ERM+RS is 32.9, which is lower than plain ERM's worst-group accuracy. That comparison is not apples-to-apples, since ERM's robust coverage is not reported, but it does hint that the scaling frontier does not transfer cleanly to time-shifted data. The stress-test's framing that this is \"direct evidence\" of validation overfitting is a bit too strong; it is suggestive, not conclusive.\n\nA more obvious omission is the absence of the simplest control: scaling logits by inverse class frequency as a post-hoc baseline. The claim that ERM+RS matches debiasing methods would be much more convincing if it beat that trivial baseline. Also, no code is released, which matters because the main contribution is an empirical finding that should be easy to reimplement. The robust coverage metric is not model-independent; it depends on the scaling search itself, so it cannot be treated as an unbiased yardstick without further discussion.\n\nWho is this for? People benchmarking group-robustness algorithms and anyone comparing trade-offs between robust and average accuracy. The paper deserves a serious referee: the core finding, if it holds, changes how we interpret existing debiasing results. I would send it to review but ask for the class-prior scaling baseline, significance checks on gains, and code before accepting.","headline":"Useful post-hoc scaling benchmark with a new scalar metric; the headline claim is plausible but the validation-fitting protocol and missing simple baseline need scrutiny before relying on it.","tokens_in":20502,"tokens_out":4156,"would_cite":true,"duration_ms":34158,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Class-specific logit scaling lets a naive ERM model match or beat debiasing methods.","keywords":["group distributionally robust optimization","spurious correlation","worst-group accuracy","unbiased accuracy","class-specific scaling","test-time post-processing","robust-average accuracy trade-off","robust coverage"],"falsifier":"On a benchmark with a genuine distribution shift between validation and test, such as FMoW-WILDS whose splits come from different years, select the scaling vector that maximizes worst-group accuracy on validation, apply it to test, and compare with a scaling vector selected directly on test. If the validation-selected vector is substantially worse, the transfer assumption that carries the method fails.","tokens_in":19586,"feed_emoji":"⚖️","tokens_out":8240,"duration_ms":64257,"temperature":0.7,"pith_summary":"This paper seeks to show that the gains reported by training-time group-debiasing methods are mostly a reallocation of accuracy along a robust-versus-average trade-off, not a sign that those methods learn better features. It demonstrates that multiplying a trained model's per-class prediction scores by class-specific constants, a post-processing step that requires no training, moves a plain empirical-risk-minimization (ERM) model to the same worst-group or unbiased accuracy points as recent debiasing algorithms, and sometimes beyond them. An instance-wise variant scales scores per cluster of similar examples and, the paper reports, improves both worst-group and average accuracy at once. The paper also introduces a scalar 'robust coverage' metric that integrates the Pareto frontier of robust-versus-average accuracy, so that algorithms can be compared on the whole trade-off rather than on a single robust-accuracy number. If the claims hold, many reported robustness gains are reproducible at test time for free, and evaluation of debiasing methods should be re-centered on the trade-off.","feed_headline":"Training-free logit scaling matches or beats debiasing methods","feed_subtitle":"A plain ERM model rescored per class reproduces robust-accuracy gains without additional training.","key_machinery":"The load-bearing object is class-specific score scaling: replace the prediction $\\arg\\max_c \\hat{y}_c$ with $\\arg\\max_c (s \\odot \\hat{y})_c$, where $s \\in \\mathbb{R}^C$ is a per-class multiplier selected by greedy coordinate search on a validation set that carries group labels. Because rescaling only reorders existing class scores, it needs no training and leaves the feature extractor untouched. The instance-wise variant clusters validation features into $K$ groups ($K=20$ in the experiments) with K-means, estimates a separate scaling vector per cluster, and applies the vector of the nearest centroid to each test example. The paper's evaluation object, robust coverage, is $\\int_0^1 \\max_s \\{RA_s \\mid AA_s \\ge c\\}\\,dc$ (discretized with $D=10^3$ slices), the area under the best robust accuracy achievable at each average-accuracy threshold.","core_discovery":"The central claim is that, on standard spurious-correlation benchmarks, a naive ERM baseline matches or outperforms recent debiasing methods once its logits are rescaled class-wise, and that an instance-wise version of the same rescaling can improve robust and average accuracy simultaneously. The prediction rule is $\\arg\\max_c (s \\odot \\hat{y})_c$, where $s \\in \\mathbb{R}^C$ is a per-class multiplier found by a greedy search on a group-annotated validation set; sweeping $s$ traces the trade-off curve between worst-group (or unbiased) accuracy and average accuracy. The instance-wise extension clusters validation features with K-means, assigns a separate scaling vector to each cluster, and applies the vector by cluster membership at test time. From this curve the paper derives robust coverage, the area under the Pareto frontier of the robust-average accuracy trade-off, and uses it to argue that evaluations reporting only robust accuracy give an incomplete picture.","pith_inferences":["A testable implication beyond the paper's experiments: if rescaling reproduces the gains of debiasing methods, those methods may be shifting decision boundaries rather than learning fundamentally different features; comparing feature-space clusterings of ERM and debiased models would check this directly.","The per-cluster scaling idea suggests a cheap deployment-time adaptation recipe: any pretrained model plus a small group-labeled validation set could be re-targeted to a new subpopulation mix without retraining.","Robust coverage values depend on how densely the scaling search approximates the Pareto frontier; reporting the search grid and cluster count alongside the scalar would make the metric reproducible.","The transfer assumption could be stress-tested with more than one spurious attribute or with continuously shifting attributes, where K-means clusters may not align with the groups that determine fairness."],"forward_implications":["Robust accuracy alone is an incomplete summary; debiasing methods should be reported together with average accuracy or with robust coverage.","A naive ERM model plus class-specific scaling is a competitive baseline that future debiasing methods should be required to beat.","The same trained model can be steered to different target metrics, worst-group, unbiased, or average accuracy, by choosing different scaling vectors without retraining.","Instance-wise scaling can push the trade-off outward, improving both worst-group and average accuracy on the same model."],"supporting_citations":[{"why":"Defines Group DRO, worst-group accuracy, and the Waterbirds setup that the paper uses as its main comparison target and benchmark.","marker":"[24]"},{"why":"Supplies the JTT baseline whose two-stage training-time reweighting is contrasted with the paper's training-free scaling.","marker":"[18]"},{"why":"Supplies the LfF baseline and the relative-difficulty reweighting scheme that ERM plus robust scaling is claimed to match or beat.","marker":"[21]"},{"why":"Provides the SUBY and SUBG baselines and the argument that validation sets with group annotations are needed for model selection.","marker":"[12]"},{"why":"Provides the CivilComments-WILDS and FMoW-WILDS benchmarks used for the large-scale and distribution-shift experiments.","marker":"[16]"},{"why":"Supplies the BPA baseline and the feature-clustering idea that the instance-wise scaling extends.","marker":"[27]"},{"why":"Supplies the George baseline and an earlier demonstration that clustering features can uncover hidden subgroups.","marker":"[28]"},{"why":"Provides the last-layer retraining baseline, a post-processing competitor that still needs training, in contrast to the paper's approach.","marker":"[15]"}],"fun_headline_variants":["Logit scaling: ERM matches debiasing","Class scaling improves robust accuracy with no training","Simple rescaling beats complex debiasing methods","Adaptive class scaling: robust gains without retraining","ERM + class scaling: a new baseline for group robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's scaling factors and cluster centroids are chosen on a group-annotated validation set and then applied to the test set unchanged; if the validation and test sets differ in group composition or in how features relate to labels, the chosen scaling is no longer optimal.","fun_headline_variants_meta":{"raw":{"variants":["Logit scaling: ERM matches debiasing","Class scaling improves robust accuracy with no training","Simple rescaling beats complex debiasing methods","Adaptive class scaling: robust gains without retraining","ERM + class scaling: a new baseline for group robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1552,"prompt_tokens":939,"completion_tokens":613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":538}},"tokens_in":555,"tokens_out":613,"duration_ms":5995,"temperature":1.0,"reasoning_tokens":538,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:43:34.620584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a benchmark with a genuine distribution shift between validation and test, such as FMoW-WILDS whose splits come from different years, select the scaling vector that maximizes worst-group accuracy on validation, apply it to test, and compare with a scaling vector selected directly on test. If the validation-selected vector is substantially worse, the transfer assumption that carries the method fails.","supporting_citations":[{"cited_title":"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case gen- eralization","cited_arxiv_id":null,"evidence_quote":"Defines Group DRO, worst-group accuracy, and the Waterbirds setup that the paper uses as its main comparison target and benchmark."},{"cited_title":"Just train twice: Improving group robustness without training group information","cited_arxiv_id":null,"evidence_quote":"Supplies the JTT baseline whose two-stage training-time reweighting is contrasted with the paper's training-free scaling."},{"cited_title":"Learning from failure: Training debiased classifier from biased classifier","cited_arxiv_id":null,"evidence_quote":"Supplies the LfF baseline and the relative-difficulty reweighting scheme that ERM plus robust scaling is claimed to match or beat."},{"cited_title":"Simple data bal- ancing achieves competitive worst-group-accuracy","cited_arxiv_id":null,"evidence_quote":"Provides the SUBY and SUBG baselines and the argument that validation sets with group annotations are needed for model selection."},{"cited_title":"Wilds: A benchmark of in-the-wild distribution shifts","cited_arxiv_id":null,"evidence_quote":"Provides the CivilComments-WILDS and FMoW-WILDS benchmarks used for the large-scale and distribution-shift experiments."},{"cited_title":"Unsupervised learning of debiased representations with pseudo-attributes","cited_arxiv_id":null,"evidence_quote":"Supplies the BPA baseline and the feature-clustering idea that the instance-wise scaling extends."},{"cited_title":"No subclass left behind: Fine-grained robustness in coarse-grained classifica- tion problems","cited_arxiv_id":null,"evidence_quote":"Supplies the George baseline and an earlier demonstration that clustering features can uncover hidden subgroups."}],"review_version":1}