{"id":"b90e7434-fb8e-45dc-a3b5-e14027960834","arxiv_id":"2506.01532","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Face-recognition models trained on data rebalanced with a continuous ethnicity score are fairer, and often as accurate, as models trained on conventionally balanced data.","lead":"This paper proposes treating a person's ethnicity as a continuous score instead of a single category, and uses that score to remove less representative identities from face-recognition training data. The authors report that models trained on such 'continuously balanced' data become fairer across ethnic groups and sometimes do not lose accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The continuous ethnicity score is a classifier trained in-sample on the very dataset being balanced; the reported gains may reflect deleting classifier-hard or atypical identities rather than continuous ethnicity. An out-of-sample score test is needed before the central claim is accepted.","rationale":"Agreeing with the reader's weakest assumption. The empirical work is broad and the artifacts are promised, which gives the paper independent support. But the most load-bearing condition for the central claim is not the sampling arithmetic—the protocols are simple and correctly implemented—it is that the numerical score used to rank identities measures continuous ethnicity rather than in-sample classifier confidence. The paper's own Section 4.4 concedes that the score only measures ease of classification. That concession is not fatal by itself: a classifier trained on ethnicity can still capture real phenotypic variation. However, because the classifier is trained and evaluated on the same identities, the low scores that drive removal are confounded with training-set atypicality, label noise, and image difficulty. The only evidence separating ethnicity from typicality is Section 4.3's CR-FIQA scatter, but that controls for one quality measure, not for the full family of confounds. The proposed out-of-sample split directly addresses this: if the score's effect transfers to disjoint identities, the sampling method is not merely memorizing the training set; if it does not, the central claim is an artifact. The reader's other concerns—missing theoretical validation and lack of error bars—are real but secondary to the construct-validity issue. The recommended verdict is unchanged: CONDITIONAL, because the paper is a plausible and useful empirical contribution whose central interpretation still needs this validation.","tokens_in":20963,"tokens_out":4293,"duration_ms":47663,"concrete_test":"Compute out-of-sample identity scores: split BUPT-BalancedFace identities into two disjoint halves (14k and 14k). Train the Section 4.4 ethnicity classifier on half A only; freeze it and score all identities in half B using the same protocol (define IDS_A by averaging image scores per identity). Re-run Protocol A and B on half B, removing 3.5k and 14k identities, and train the same iResNet-34/ElasticArcFace configuration on the resulting sets. The decisive comparison is at the 14k-removal level: if the STD and SER gains over random sampling seen in Table 3 (STD 0.55 vs 1.62, SER 1.16 vs 1.62) largely vanish, the effect is an in-sample classifier-confidence artifact; if they persist, the score transfers across identities and the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that continuous ethnicity balancing outperforms discrete balancing—rests entirely on the construct validity of the pseudo-continuous ethnicity score. Section 4.4 defines the score as the softmax confidence of a four-class ethnicity classifier built on an ElasticArc ResNet-100 backbone, trained on BUPT-BalancedFace (only the classification layer is updated), and then applied to the same BUPT-BalancedFace identities to compute IDS. The authors themselves state that a high probability does not mean that the image is representative of the definition of a group, only that the image is easily classified as belonging to that group of images in the dataset. Therefore the score is, by construction, a measure of in-sample classifiability and typicality relative to the training distribution. The three protocols (Algorithms 1-3) remove the lowest-score identities from the currently lowest- or highest-scoring ethnicity, which is exactly a data-cleaning operation on hard, outlier, or mislabeled images. Table 2's Min versus Max gap (African 89.95 vs 92.78) is the expected behavior of any classifier-confidence utility, not evidence that the axis of selection is ethnicity. Section 4.3's comparison with CR-FIQA rules out one quality confound but not pose, blur, identity difficulty, or training-set membership; and the relabeling R variants relabel by the same classifier's argmax, so they cannot break the circularity. If the score is primarily a typicality or utility proxy, the fairness gains in Tables 3-5 and the released datasets encode that artifact, and the paper's central conceptual conclusion does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that ethnicity labels for face-recognition training data should be treated as continuous rather than discrete, and that balancing a training set in this continuous space yields fairer and often more accurate models than balancing by discrete ethnicity counts. The authors construct what they call an ethnicity score by training a four-class softmax ethnicity classifier on top of a fixed ElasticArcFace ResNet-100 feature extractor, using the BUPT-BalancedFace training set itself, and then compute per-image, per-identity, and per-ethnicity scores. They propose three sampling protocols (A, B, C) that iteratively remove low-scoring identities from a selected ethnicity group, plus relabeled variants A(R), B(R), C(R), and compare the resulting models against models trained on the full dataset and on randomly sampled subsets preserving discrete balance. Experiments are carried out with iResNet-34/50/100 and ElasticArcFace/CurricularFace losses, with evaluation on RFW, reporting per-ethnicity accuracy, STD, and SER. The central empirical claim is that continuous-space balancing consistently outperforms discrete-space balancing, with Protocol A giving the largest fairness gains and Protocol B retaining accuracy best. The authors also release code and sampled dataset versions.","tokens_in":21253,"tokens_out":3193,"duration_ms":33802,"significance":"If the central claim holds, the paper would make a useful conceptual contribution: it challenges the common assumption that equalizing discrete demographic counts is sufficient for dataset balance, and it provides a concrete, reproducible recipe for score-guided identity removal that appears to improve fairness metrics without sacrificing overall accuracy. The breadth of the experimental work is a strength: more than 65 trained models, three architectures, two losses, multiple removal fractions, and publicly released code and sampled datasets. The main risk is construct validity: the continuous ethnicity score is produced by a classifier trained on the very dataset being rebalanced, and the paper itself acknowledges (Section 4.4) that a high score means only that the image is easily classified into that group of images in the dataset, not that it is representative in an objective sense. If the score mainly captures in-sample classifiability, typicality, or image utility, then the reported gains could be a data-cleaning artifact rather than evidence about the continuous nature of ethnicity.","major_comments":[{"comment":"The ethnicity score is generated by a classifier trained on BUPT-BalancedFace and then applied to the same BUPT-BalancedFace identities to compute IDS and drive the removal protocols. This is an in-sample scoring procedure. The authors themselves state in Section 4.4 that a high probability does not mean the image is representative of the definition of a group, only that it is easily classified as belonging to that group of images in the dataset. Under this definition, the score is a measure of classifiability/typicality relative to the training distribution, and removing the lowest-score identities is equivalent to removing hard or atypical training samples. The reported advantage over random sampling could therefore be explained by data cleaning rather than by any continuous property of ethnicity. A necessary control is an out-of-sample validation of the score: for example, train the ethnicity classifier on a different dataset (e.g., BUPT-GlobalFace or an external ethnically labeled dataset) and apply it to BUPT-BalancedFace, or evaluate on held-out identities to show that the score generalizes as a measure of group representativeness rather than memorization of training identities. Without such a test, the central claim that continuous ethnicity balancing outperforms discrete balancing is not established.","section":"Section 4.4 and Section 3.1"},{"comment":"The single-ethnicity analysis in Table 2 shows that retaining the highest-scoring African identities gives a 92.78% African accuracy versus 89.95% for the lowest-scoring ones, which is consistent with the paper's assumption. However, this pattern is exactly what any classifier-confidence or utility-based selection would produce: removing the least classifiable identities of a group improves the group's training signal. The comparison with CR-FIQA in Section 4.3 is only a qualitative scatter plot and reports no correlation coefficient; it rules out one quality confound but not pose, blur, identity difficulty, or training-set membership. To separate the ethnicity-representativeness effect from a generic utility effect, the authors should include a control protocol driven by an alternative utility score (e.g., CR-FIQA, image norm, or an unsupervised typicality measure) and show that the ethnicity-guided selection produces different or better fairness outcomes than utility-guided selection at the same removal level. As it stands, Table 2 and the CR-FIQA plot do not distinguish between 'ethnically atypical' and 'hard to classify' as the operative selection axis.","section":"Section 4.3 and Table 2"},{"comment":"All reported results appear to come from a single training run per condition. Many of the claimed improvements are small in absolute terms: for example, at 27k identities in Table 3, Protocol A has STD 1.17 and SER 1.76 versus 1.27 and 1.86 for Random, and in Table 5 Protocol C at 21k has STD 1.60 versus 1.44 for Random, i.e., worse. Without multiple seeds, confidence intervals, or a significance test, the 'consistently outperform' claim in the abstract is not statistically supported. I recommend either reporting results over at least three seeds per condition with means and standard deviations, or conducting a statistical test over bootstrap resamples of the evaluation set, or explicitly stating and discussing the single-seed limitation. This is load-bearing because the central claim is a comparative statement across many conditions, and the magnitude of some differences is comparable to typical training noise.","section":"Section 5.2 and Tables 3-5"},{"comment":"Protocol C removes the lowest-scoring identity from the ethnicity group with the highest total ES score, which pushes that group's total score down. The paper describes this as balancing in the continuous space, but the procedure is not symmetric with Protocols A and B: it equalizes the sum rather than the mean, and the paper does not justify why sum-balancing is the right continuous notion of balance for the fairness claim. This matters because Protocol C is reported as having the weakest fairness gains (Table 5). Please clarify the theoretical target of Protocol C and, if it is meant to balance sums, discuss why that target is appropriate; otherwise the comparison among protocols is hard to interpret.","section":"Section 5.2.3 and Algorithm 3"}],"minor_comments":[{"comment":"There are typos in 'Balacing' and 'continuos' in the title/abstract; these should be corrected.","section":"Title and Abstract"},{"comment":"The ethnicity classifier training details are incomplete: no number of epochs, learning rate, batch size, or validation accuracy is reported for the classification layer. Since the entire paper depends on this classifier, these details are necessary for reproducibility.","section":"Section 4.4"},{"comment":"The sentence 'these latter strategies seem to hold significant advantages with respect to random sampling' appears in table captions but is not supported by any statistical significance test; please either remove 'significant' or add the appropriate test.","section":"Section 5.2.2"},{"comment":"The text says 'removing exactly half of the identities in a given ethnicity,' and Table 2 reports 3,500 retained identities, but it is not immediately clear that each ethnicity in BUPT-BalancedFace contains 7,000 identities; please state the group sizes explicitly in the table caption.","section":"Section 5.1"},{"comment":"The figures are referenced out of order: Figure 6 is referred to in Section 4.4 before Figure 5 is introduced. Please renumber or reorder so that figures appear in the order they are cited.","section":"Figure 2 and Figure 6 numbering"},{"comment":"The claim that the ethnicity score is 'uncorrelated' with CR-FIQA is based on visual inspection of a scatter plot; report the actual correlation coefficient (e.g., Pearson and Spearman) for the comparison.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's own Section 4.4 caveat essentially concedes that the ethnicity score measures in-sample classifiability rather than objective group representativeness. In my view, the out-of-sample control and the utility-control experiment are not optional additions; they are necessary to distinguish the paper's central claim from a data-cleaning artifact. The single-seed issue is also important given the small reported differences. I would not reject the manuscript, because the idea is interesting and the experimental infrastructure is valuable, but the current evidence is not sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a substantial empirical study that introduces a plausible and useful idea—balancing face-recognition training data by continuous ethnicity scores instead of discrete counts—but the central claim is not yet established, because the scores are learned on the very dataset being balanced. The consistent-looking gains over random sampling may be mostly a data-cleaning artifact.\n\nWhat's genuinely new: the three sampling protocols (A, B, C) plus relabeling variants, the systematic comparison against discrete-balanced random sampling across four removal rates, and the release of the created datasets and code. Training more than 65 models across three architectures and two losses is real work, and the single-ethnicity analysis in Table 2 is a nice sanity check that the score captures something about representativeness. The authors also deserve credit for being candid in Section 4.4: they explicitly say a high probability only means easy classification, not representativeness. That is the right caveat, but it cuts against their own strong conclusion.\n\nThe soft spots are serious and in one case load-bearing. First, the ethnicity classifier is trained on BUPT-BalancedFace and then applied to BUPT-BalancedFace to compute identity scores. Removing the lowest scores removes the images that are hardest or most atypical relative to the training distribution. The improvement over random sampling is therefore expected even if the score has nothing to do with ethnicity as a continuous trait. The CR-FIQA comparison rules out one quality confound but not pose, blur, or intra-class typicality. Second, the abstract claims theoretical validation, but the theory section is notation and algorithms only—no formal statement or proof. That is overclaiming. Third, there are no multiple seeds or error bars, and many differences in the tables are well under one percentage point; 'consistently outperform' is stronger than the data can support. Protocol C in particular shows bias metrics close to or worse than random at the heavier removal rates (Table 5). Fourth, the relabeling variants use the same classifier's argmax, so they cannot break the circularity.\n\nNone of this kills the practical potential: a cheap pseudo-labeling method that guides identity removal and cuts training cost by 25% with little accuracy loss is worth having. But the paper's conceptual conclusion—that continuous ethnicity is the true balancing axis—needs out-of-sample score validation (e.g., train the classifier on BUPT-GlobalFace, or compare against an external continuous annotation) and variance reporting.\n\nMy recommendation: send it to peer review, but with a request for major revision. The empirical scaffolding is strong enough to merit referee time; the central claim just needs the right control before it can be taken at face value.","headline":"A genuinely useful empirical study of continuous-ethnicity balancing, but the in-sample pseudo-label makes the core comparison over random sampling look like data cleaning.","tokens_in":21861,"tokens_out":3028,"would_cite":false,"duration_ms":31963,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating ethnicity as a continuous score per identity, and balancing face datasets on that score, yields fairer models than balancing discrete group counts.","keywords":["face recognition","demographic bias","fairness","continuous ethnicity labels","dataset balancing","identity representativeness","sampling protocols","bias mitigation"],"falsifier":"Train the same three sampling protocols with a continuous ethnicity score obtained from human-annotated ancestry proportions instead of classifier confidence, and compare against randomly downsampled discrete-balanced subsets on the per-ethnicity benchmark. If the fairness and accuracy advantage over random sampling disappears, then the reported gains came from removing hard or in-sample images rather than from the continuous nature of ethnicity; a cheaper confound check is to inspect whether identities removed by Protocol A are systematically low-quality according to a face quality score.","tokens_in":20696,"feed_emoji":"⚖️","tokens_out":8209,"duration_ms":81264,"temperature":0.7,"pith_summary":"This paper argues that ethnicity in face recognition should be treated as a continuous spectrum rather than a discrete category. The authors build a classifier on top of a face recognition backbone to score how strongly each image, and each identity, represents its labeled ethnic group, and they show that identities within the same category vary widely on this score. They propose three sampling protocols that remove identities so that the remaining data is balanced in this continuous space, and they train over sixty-five models to compare those subsets with randomly downsampled subsets that keep discrete counts equal. Their central claim is that continuous-space balancing consistently outperforms discrete balancing: lower cross-ethnicity accuracy gaps (STD and SER) with equal or better average accuracy, even when up to half of all identities are removed. If correct, this reframes dataset balance, because equal counts per ethnicity no longer count as balanced when individuals differ in representativeness.","feed_headline":"Ethnicity is a spectrum: continuous labels beat discrete balancing","feed_subtitle":"Balancing by how representative each face is, not by group counts, narrows accuracy gaps across ethnicities.","key_machinery":"The carrying mechanism is a four-dimensional continuous ethnicity score produced by a linear classifier layer trained on top of a frozen face recognition backbone, which assigns each image a probability vector over four ethnic groups. Three aggregation levels link images to identities to ethnicities: the image score $ies_{j,i}$, the identity score $IDS_j$ (in Protocol A the mean of its image scores, in Protocols B and C the sum), and the ethnicity score $ES_y$ (mean over identities in A and B, sum in C). Three removal protocols repeatedly delete the least representative identity from the ethnicity with the lowest score (A and B) or the highest score (C), creating a sequence of continuously balanced subsets; a relabeling variant reassigns each identity to its maximum-scoring ethnicity to show the effect is not merely the removal of mislabeled samples.","core_discovery":"The central discovery is that the discrete ethnicity label attached to an identity does not measure that identity's contribution to demographic balance. The paper defines a continuous ethnicity score at three levels: per-image score $ies$, per-identity score $IDS$, and per-ethnicity score $ES$, computed by a classifier attached to a pretrained face recognition model. It then establishes three results: removing the highest-scoring identities from one ethnicity degrades accuracy on that ethnicity much faster than removing the lowest-scoring ones; a dataset with equal identity counts per ethnicity is still visibly imbalanced in the continuous space; and models trained on subsets balanced by continuous scores sit on or near the Pareto frontier of error versus fairness, dominating randomly sampled discrete-balanced subsets of the same size. The authors take this as evidence that balancing should equalize representativeness scores rather than category counts, and that this reframing holds across several network sizes and two training losses.","pith_inferences":["If the ethnicity score is accepted as a measure of representativeness, the same balancing recipe should transfer to other coarse demographic attributes such as age or gender presentation, where discrete bins are also rough proxies for continuous variation.","The identity-level score can be reused as a general data-pruning utility: pairing it with a quality score, which the paper shows to be weakly correlated, could select subsets that are simultaneously fair and high-quality.","The paper's own caveat, that the score measures how easily an image is classified into a group, suggests a strong test: replace the classifier score with human-annotated continuous ancestry proportions and see whether the balancing advantage survives; if it does not, the effect is about image difficulty rather than ethnicity."],"forward_implications":["A training set with an equal number of identities per ethnicity can still be demographically imbalanced, so future dataset construction should report continuous-score distributions, not just counts.","Under Protocol B, removing 25% of identities costs almost no accuracy while clearly improving fairness, meaning fairer models can be trained on smaller, cheaper datasets.","Protocol A produces the largest fairness gains at extreme removal levels, cutting STD from 1.62 to 0.55 when half of the identities are removed while keeping average accuracy at the random-sampling level.","The advantage of continuous balancing persists across iResNet-34, iResNet-50, and iResNet-100 and across ElasticArcFace and CurricularFace losses, so it is not tied to one architecture or loss.","The released subsets give other researchers ready-made training data with characterized fairness and accuracy trade-offs, so new losses or architectures can be tested on top of them."],"supporting_citations":[{"why":"supplies the discretely balanced training set used as the starting point for all sampling protocols and as the discrete-balance baseline.","marker":"[38]"},{"why":"provides the per-ethnicity evaluation benchmark on which accuracy and fairness metrics are computed.","marker":"[37]"},{"why":"provides the ElasticArc loss and the pretrained backbone whose latent space is reused for the ethnicity classifier; ElasticArcFace is also one of the two training losses.","marker":"[2]"},{"why":"provides the second training loss used to show the continuous-balancing advantage holds beyond a single loss.","marker":"[20]"},{"why":"supplies the CR-FIQA quality scores used to argue that the ethnicity score is not merely a proxy for image quality.","marker":"[3]"},{"why":"is the prior work whose assumption that discrete balance is sufficient is directly reformulated and extended by the paper.","marker":"[17]"},{"why":"provides the residual network architectures used to train the face recognition models in the experiments.","marker":"[18]"}],"fun_headline_variants":["Continuous ethnicity labels beat discrete balancing","Balancing by representativeness, not ethnicity counts","Discrete balancing fails; continuous scores improve fairness","Ethnicity as spectrum: better face recognition fairness","Representativeness scores outperform discrete group balance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the classifier's confidence score actually measures how representative an identity is of its ethnic group; the paper itself notes the score only captures how easily an image is classified into that group, so if the score is really measuring image difficulty or training-set memorization, the fairness gains could be a data-cleaning artifact rather than evidence about continuous ethnicity.","fun_headline_variants_meta":{"raw":{"variants":["Continuous ethnicity labels beat discrete balancing","Balancing by representativeness, not ethnicity counts","Discrete balancing fails; continuous scores improve fairness","Ethnicity as spectrum: better face recognition fairness","Representativeness scores outperform discrete group balance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1415,"prompt_tokens":867,"completion_tokens":548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":483,"tokens_out":548,"duration_ms":6077,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:38:47.337945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same three sampling protocols with a continuous ethnicity score obtained from human-annotated ancestry proportions instead of classifier confidence, and compare against randomly downsampled discrete-balanced subsets on the per-ethnicity benchmark. If the fairness and accuracy advantage over random sampling disappears, then the reported gains came from removing hard or in-sample images rather than from the continuous nature of ethnicity; a cheaper confound check is to inspect whether identities removed by Protocol A are systematically low-quality according to a face quality score.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the discretely balanced training set used as the starting point for all sampling protocols and as the discrete-balance baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the per-ethnicity evaluation benchmark on which accuracy and fairness metrics are computed."},{"cited_title":"Boutros, N","cited_arxiv_id":null,"evidence_quote":"provides the ElasticArc loss and the pretrained backbone whose latent space is reused for the ethnicity classifier; ElasticArcFace is also one of the two training losses."},{"cited_title":"Huang, Y","cited_arxiv_id":null,"evidence_quote":"provides the second training loss used to show the continuous-balancing advantage holds beyond a single loss."},{"cited_title":"Boutros, M","cited_arxiv_id":null,"evidence_quote":"supplies the CR-FIQA quality scores used to argue that the ethnicity score is not merely a proxy for image quality."},{"cited_title":"Gwilliam, S","cited_arxiv_id":null,"evidence_quote":"is the prior work whose assumption that discrete balance is sufficient is directly reformulated and extended by the paper."}],"review_version":1}