{"id":"cb05be84-8550-468b-82fd-5d4e78921d13","arxiv_id":"2412.11072","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A data selection method that replaces a clean holdout set with a zero-shot predictor plus peer prediction to pick training samples that reduce label bias and improve fairness.","lead":"This paper proposes a data selection method that uses a pre-trained zero-shot model (CLIP) as a stand-in for a clean holdout set, augmented with a peer-prediction term, to filter training data for fairer classifiers. The authors report accuracy and fairness gains on two face-attribute datasets under injected label noise, but the derivation of the selection function contains a significant algebraic error.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (13) is not a consequence of Eq. (11) and Eq. (12): the zero-shot-loss coefficient changes from −α to +(1−α) and the peer penalty changes from +αγ to −γ, so the central selection objective is underived.","rationale":"The reader's rationale already identifies the Eq. (11)/(12) to Eq. (13) derivation as the central error and uses it as the main reason for rejection, so my stress-test agrees with the final verdict. However, the reader's explicitly stated weakest_assumption is the zero-shot posterior approximation in Eq. (10), which is a different, though related, vulnerability. My read is that the derivation flaw is more load-bearing: even if Eq. (10) were validated and the zero-shot predictor were perfectly aligned with a clean fair holdout, Eq. (13) still would not follow from the preceding equations. The sign and coefficient changes invert the intended dependence on the proxy model's confidence and change the peer-prediction penalty, so the method's actual selection behavior is not the one derived. This is independent of the empirical tables; it means the central theoretical claim is unsupported as written. A symbolic re-derivation is a cheap, decisive check. I therefore recommend no change to the reader's rejection.","tokens_in":18178,"tokens_out":7702,"duration_ms":69556,"concrete_test":"Use a symbolic algebra system (e.g., SymPy) to substitute Eq. (12) into Eq. (11), simplify under Σ_s C_s/m = 1, and compare the resulting expression term-by-term with Eq. (13). A mismatch in any coefficient or in the presence of the group sum confirms the derivation error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on Eq. (13) being a valid fair selection objective. That equation does not follow from the derivation. Starting from Eq. (11), Σ_s (C_s/m)[L[y|x,D_t,s] − α L[y, f~(x),s]], and replacing the zero-shot loss with the peer-prediction expression in Eq. (12), L[y,f~(x),s] − γ E_{Y|D_{s'}}[L[Y,f~(x),s]], gives Σ_s (C_s/m)L[y|x,D_t,s] − α Σ_s (C_s/m)L[y,f~(x),s] + αγ Σ_s (C_s/m)E_{Y|D_{s'}}[L[Y,f~(x),s]]. Even using Σ_s C_s/m = 1, this is not Eq. (13), which reads L[y|x,D_t,s] + (1−α)L[y,f~(x),s] − γ E_{Y|D_{s'}}[L[Y,f~(x),s]]. The coefficient of the zero-shot term changes from −α to +(1−α), the peer-penalty coefficient changes from αγ to γ, and the sum over s is dropped without justification. Because Eq. (13) is the method's selection function, the claimed connection to fair holdout-loss maximization is unsupported. The empirical results may still indicate a useful selection heuristic, but the paper's stated theoretical foundation does not hold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data selection method for fair learning under label bias. It extends the RHO-LOSS criterion of Mindermann et al. (2022) by using a zero-shot predictor (CLIP) as a proxy for a clean holdout model, and augments this proxy with a peer prediction mechanism intended to correct label bias in the selection score. The final selection objective is Eq. (13), which is used in Algorithm 1 to rank training instances. Experiments on CelebA and LFW+a with synthetic symmetric label bias report improved accuracy and lower demographic parity violation relative to uniform sampling, gradient norm selection, and RHO-LOSS, together with ablation studies over the zero-shot backbone, target backbone, and hyperparameters.","tokens_in":18533,"tokens_out":12764,"duration_ms":105056,"significance":"If the derivation were sound, the paper would offer a practical contribution: a modality-agnostic data selection principle that mitigates label bias without requiring a clean holdout set or noise rate estimation. The empirical study is reasonably broad (two datasets, multiple fairness metrics, ablations, convergence analysis) and the reported gains over the baselines are consistent across settings. However, the central theoretical derivation is invalid, and the empirical results cannot by themselves establish the paper's stated claim that Eq. (13) simulates training on a clean holdout set under a fair distribution. The paper also deserves credit for comparing several zero-shot backbones and for reporting selected-data statistics (Fig. 1), which directly probe the method's intended behavior.","major_comments":[{"comment":"Eq. (13) does not follow from Eqs. (11) and (12). Substituting Eq. (12) into Eq. (11) for the zero-shot loss term gives Σ_s (C_s/m)[L[y|x,D_t,s] − αL[y,f~(x),s] + αγE_{Y|D_{s'}}[L[Y,f~(x),s]]], not the expression in Eq. (13). The coefficient of the zero-shot term changes from −α to +(1−α), the peer-penalty coefficient changes from αγ to γ, and the sum over s is dropped even though E_{Y|D_{s'}} depends on s through the choice of the opposing group. Since Eq. (13) is the selection criterion used in Algorithm 1 and the paper's stated connection to fair holdout-loss maximization rests on this derivation, the central theoretical claim is unsupported.","section":"Fair Data Selection with Peer Prediction Mechanism (Eqs. 11-13)"},{"comment":"Eq. (10) assumes that the zero-shot predictor f~(x) is the posterior mean of a model trained on the clean fair holdout distribution. This is a strong approximation: a pretrained model may carry its own label bias or be misaligned with the task distribution, in which case the 'holdout loss' term in the selection criterion is biased rather than a fair reference. The paper justifies the assumption only by the heuristic argument that the posterior is narrow over a large training set; the ablation over three zero-shot backbones (Table 3) tests sensitivity to the backbone but does not validate the approximation against a true clean holdout model. Because this approximation makes Eq. (9) tractable and underlies the claim that no clean holdout set is needed, it is load-bearing and requires direct validation.","section":"Method, Eq. (10)"}],"minor_comments":[{"comment":"In the derivation of the expectation version of the peer loss, the step from the double sum over i and i′≠i to (1/N_s)Σ_i E_{Y|D_{s'}}[γL[Y,f~(x_i),s]] drops the finite-sample factor (N_s−1)/N_s; the equality should be stated as an approximation or include this factor.","section":"Appendix, derivation of Eq. (12)"},{"comment":"The notation in Eq. (14) is inconsistent with the appendix: the first term uses L[Y,f~(X),S] while the appendix's Eq. (24) uses L[Z,f~(X),S], and the third term writes L[j,f~(X)] rather than L[j,f~(X),s]; these should be harmonized.","section":"Eq. (14) and appendix Eq. (24)"},{"comment":"The hardware description 'NVIDIA GeForce RTX 3090 with 86GB memory' appears to be a typo, as the RTX 3090 has 24GB of memory.","section":"Experimental setup"},{"comment":"The loop header 'for t in 0, · · ·, Tdo' contains a typo and should read 'for t in 0, · · ·, T do'.","section":"Algorithm 1"},{"comment":"The implementation details for the baselines, especially RHO-LOSS's use of a holdout set and the hyperparameter settings for the gradient norm variants, are not described in enough detail to allow replication.","section":"Baselines"}],"recommendation":"reject","confidential_remarks":"The core problem is the invalid derivation of Eq. (13); since this equation is the actual selection rule used in the experiments, the theoretical framing of the method collapses. The empirical section is comparatively solid, and a revised manuscript that presents Eq. (13) as a heuristic selection rule with a corrected analysis could be reconsidered, but as written the central claim is not supportable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper puts forward a fairness-aware data selection objective that combines RHO-LOSS, a zero-shot predictor as a holdout proxy, and a peer-prediction fairness penalty. That combination is new, and the resampling step to address selection bias is a sensible addition. The empirical results on CelebA and LFW+a are consistent: the method improves accuracy and reduces demographic parity violation compared to uniform sampling, RHO-LOSS, and gradient-norm baselines, and the ablation across zero-shot predictors and backbones suggests the effect is not tied to one model choice.\n\nThe soft spot is not a detail: the central derivation does not hold up. Substituting Eq. (12) into Eq. (11) gives a term of the form L[y|x,D_t,s] − α L[y,f~(x),s] + αγ E[L], with the sum over s still present. The paper instead writes Eq. (13) as L[y|x,D_t,s] + (1−α)L[y,f~(x),s] − γ E[L], dropping the sum and changing both coefficients. Since Eq. (13) is the actual selection function used in training, the method is not what the derivation claims it is. This is a load-bearing error, not a typo in a minor expression.\n\nThere are additional concerns. The approximation in Eq. (10) — that the zero-shot predictor's log-posterior equals the posterior expectation of a clean holdout-trained model — is strongly assumed and never validated. The peer-prediction term only penalizes cross-group disagreement; it does not correct the zero-shot predictor's own misclassifications. Experiments are limited to two image datasets, binary labels, and a small hyperparameter grid for α and γ; the paper does not state whether the grid was evaluated on a validation split, so tuning on the test metric is a real possibility. The reproducibility checklist claims code in the appendix, but no code appears in the provided manuscript.\n\nWho should read this? It is most useful to someone working on data selection for fairness who wants a concrete idea for a heuristic: pick training points that have low zero-shot loss plus a cross-group disagreement penalty. That heuristic may be worth exploring, but the theoretical justification as written is wrong.\n\nRecommendation: reject in current form. The authors should redo the derivation (or honestly present Eq. (13) as a heuristic), validate the zero-shot approximation on a held-out clean set, and report a proper hyperparameter-selection protocol. If that happens, the idea deserves another look.","headline":"The fairness-aware data selection idea is new, but Eq. (13) is underived; as written the central claim fails, though the heuristic may be salvageable with a corrected derivation.","tokens_in":19064,"tokens_out":4464,"would_cite":false,"duration_ms":37681,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a data selection rule scoring training points by their expected impact on a fair distribution—using a zero-shot predictor as a proxy for a clean holdout set and a peer prediction penalty—can pick instances less…","keywords":["fair data selection","label bias","zero-shot predictor","peer prediction","demographic parity","data selection","fairness","online batch selection"],"falsifier":"On a benchmark with known clean labels, construct a version of the task where CLIP has group-dependent errors (e.g., its predictions for female images are systematically shifted toward the majority label), then run Eq. (13) selection and measure the fraction of selected instances whose true clean labels differ from their observed biased labels. If that fraction is not materially lower than uniform sampling, or if training on the selected set does not reduce demographic parity violation, the central claim fails.","tokens_in":17969,"feed_emoji":"⚖️","tokens_out":9401,"duration_ms":69760,"temperature":0.7,"pith_summary":"The paper tries to establish that a data selection rule, which scores each training example by its expected effect on a fair predictive distribution, can train fair classifiers even when the labels themselves are biased against a sensitive group. It claims the score can be computed without a clean holdout set: a zero-shot pretrained model such as CLIP stands in for the validation model, and a peer prediction penalty keeps that proxy from leaking group bias into the scores. The resulting objective is a drop-in filter for any log-likelihood or cross-entropy based classifier, needs no noise-rate estimation, and in experiments on CelebA and LFW+a with 20% and 40% symmetric label bias it improves both test accuracy and demographic parity relative to baselines while converging faster.","feed_headline":"Data selection rule boosts accuracy and fairness on biased labels","feed_subtitle":"Zero-shot proxy plus peer prediction picks fair samples without a clean holdout set.","key_machinery":"The engine of the method is the selection objective in Eq. (13), which assigns each candidate sample $(x,y,s)$ the score $L[y|x,D_t,s] + (1-\\alpha) L[y,\\tilde{f}(x),s] - \\gamma\\,\\mathbb{E}_{Y|D_{s'}}[L[Y,\\tilde{f}(x),s]]$. Here $L[y|x,D_t,s]$ is the current model's training loss; the zero-shot predictor $\\tilde{f}$ (CLIP in the experiments) supplies the holdout-loss term that replaces a clean validation set; and the peer prediction expectation, drawn from a different demographic group $s'$, penalizes the proxy's group-dependent loss. The companion decomposition of the expected score into a fair-model term, a noisy-loss penalty, and a demographic-disagreement penalty is what the authors use to argue that maximizing the score prefers instances less affected by label bias.","core_discovery":"The central claim is that the reducible holdout loss (RHO-LOSS) selection criterion can be re-derived with a fair label distribution in place of the observed biased one, and that the resulting selection function is tractable without a holdout set. The ideal selection objective is written as a group-weighted score, and the intractable term involving the holdout posterior is lower-bounded and then approximated by a zero-shot predictor $\\tilde{f}$ under the assumption that the predictor's training data make its posterior narrow. Adding a cross-group peer loss yields the final selection score $L[y|x,D_t,s] + (1-\\alpha)L[y,\\tilde{f}(x),s] - \\gamma\\,\\mathbb{E}_{Y|D_{s'}}[L[Y,\\tilde{f}(x),s]]$, which separates into a clean fair-model loss, a penalty on noisy label transitions, and a penalty on cross-group disagreement in the proxy loss. The paper claims this is why the selected instances are less influenced by label bias, and the experiments support higher accuracy and lower fairness violation than uniform sampling, gradient-norm selection, and RHO-LOSS.","pith_inferences":["If the zero-shot predictor itself carries label bias, the approximated holdout term is biased; the peer-prediction penalty only punishes cross-group disagreement, so a predictor that makes the same confident mistake for both groups would evade the penalty. A sensitivity test swapping in predictors with known group bias would reveal this failure mode.","The decomposition assumes label flips depend on (Z,S) but not on X; under instance-dependent label noise the noisy-loss penalty may misalign, so an experiment with within-group flip rates would test robustness beyond symmetric bias.","The derivation targets demographic parity; equalized odds or calibration would require a different group-conditional penalty, which the same derivation can yield but is not pursued."],"forward_implications":["Training on the selected subset removes the need for a clean holdout set, so the method applies where fair labels are unavailable.","The approach is compatible with any log-likelihood or cross-entropy based classifier and needs no noise-rate estimation, so it can be added to existing training pipelines.","It addresses both label bias and selection bias (via the resampling step), improving demographic parity without sacrificing accuracy in the tested settings.","Because it selects only a fraction of each batch, training converges faster than with uniform sampling or RHO-LOSS, per the reported epochs-to-target-accuracy results."],"supporting_citations":[{"why":"Supplies the RHO-LOSS criterion and baseline that the selection principle is derived from and compared with.","marker":"(Mindermann et al. 2022)"},{"why":"Provides the Bayesian lower-bound approximation and the zero-shot predictor strategy used to eliminate the holdout set.","marker":"(Deng, Cui, and Zhu 2023)"},{"why":"Supplies the peer prediction / peer loss mechanism that the fairness penalty is built from.","marker":"(Liu and Guo 2020)"},{"why":"Defines label bias as group-dependent label flips and motivates the fair-label distribution used in the derivation.","marker":"(Wick, panda, and Tristan 2019)"},{"why":"Supports the resampling step that corrects selection bias in the selected batches.","marker":"(Kamiran and Calders 2012)"}],"fun_headline_variants":["Zero-shot proxy plus peer prediction selects fair data, no holdout","Holdout-free data selection using zero-shot proxy and peer loss","Fair data selection with zero-shot predictor, no clean set needed","Peer prediction helps zero-shot proxy avoid holdout for fairness","Data selection that fixes label bias without a clean holdout"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire selection score rests on the assumption that the zero-shot predictor $\\tilde{f}$ approximates the posterior predictive of a model trained on a clean fair holdout set; if CLIP carries label bias or does not fit the task, the approximated holdout-loss term is biased and the selection can favor the wrong points.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot proxy plus peer prediction selects fair data, no holdout","Holdout-free data selection using zero-shot proxy and peer loss","Fair data selection with zero-shot predictor, no clean set needed","Peer prediction helps zero-shot proxy avoid holdout for fairness","Data selection that fixes label bias without a clean holdout"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000403,"raw_usage":{"total_tokens":2080,"prompt_tokens":908,"completion_tokens":1172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1087}},"tokens_in":524,"tokens_out":1172,"duration_ms":10768,"temperature":1.0,"reasoning_tokens":1087,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:19:31.228223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a benchmark with known clean labels, construct a version of the task where CLIP has group-dependent errors (e.g., its predictions for female images are systematically shifted toward the majority label), then run Eq. (13) selection and measure the fraction of selected instances whose true clean labels differ from their observed biased labels. If that fraction is not materially lower than uniform sampling, or if training on the selected set does not reduce demographic parity violation, the central claim fails.","supporting_citations":[{"cited_title":"M.; Razzak, M","cited_arxiv_id":null,"evidence_quote":"Supplies the RHO-LOSS criterion and baseline that the selection principle is derived from and compared with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian lower-bound approximation and the zero-shot predictor strategy used to eliminate the holdout set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the peer prediction / peer loss mechanism that the fairness penalty is built from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines label bias as group-dependent label flips and motivates the fair-label distribution used in the derivation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the resampling step that corrects selection bias in the selected batches."}],"review_version":1}