{"id":"51566632-4282-4874-9c02-5d9f3fda4d2a","arxiv_id":"2412.14724","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FROC transports the upper group ROC onto L1-distance boundaries around the lower group ROC to impose threshold-invariant fairness with claimed minimal AUC loss.","lead":"This paper introduces FROC, a post-processing method that bends the ROC curve of one protected group toward the other so a classifier stays fair at every decision cutoff while losing little AUC. The authors prove an optimality result under restrictive geometric assumptions and test the method on Adult, COMPAS, and CelebA classifiers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FROC's optimality theorem assumes the shifted ROC points are jointly realizable as a scoring function, but the paper never proves feasibility or monotonicity; without this, Theorem 4.2's constructed h need not exist.","rationale":"The central claim is that FROC solves Problem 2 under Assumption 4.2. The most load-bearing condition for that claim is not just the shape of the optimal fair points but whether those points can be assembled into a feasible scoring function h. The geometric transport in Algorithm 1 moves each query point independently; the appendix realizes individual points by randomization but does not construct a full scoring function whose ROC is the output set. The shifted points may leave the hypograph of ROCup or violate monotonicity, and the proof of CutShift optimality is a duplicated text block rather than a proof. This is a gap in the existence of the alleged maximizer, so the theorem's conclusion is unsupported even if Assumption 4.2 holds. The reader's weakest_assumption pointed to the same unproven feasibility/monotonicity issue, so I agree with the rejection. The finite-query gap between sampled thresholds and the code/algorithm mismatch are additional independent problems, but the realizability gap is sufficient by itself to block the optimality theorem.","tokens_in":25743,"tokens_out":10324,"duration_ms":89096,"concrete_test":"Enumerate pairs of monotone piecewise-linear ROC curves with vertices on a small grid (k=4 to 10, epsilon in {0.01, 0.05, 0.1}) that satisfy Assumption 4.2 and keep ROCup above ROCdown. For every run of Algorithm 1, check (i) each returned eQ^up_i lies in the hypograph of ROCup, and (ii) the sequence eQ^up_i is nondecreasing in both FPR and TPR. For any case that passes these checks, construct h using the Theorem C.1 randomization at each threshold and compare h's empirical ROC and AUC with the FairROC and with the true optimum of Problem 2. A single violation of (i) or (ii) falsifies Theorem 4.2 as stated.","verdict_should_be":"REJECT","load_bearing_attack":"Theorem 4.2 claims the ROCs returned by FROC represent a classifier solving Problem 2. That requires the transported points eQ^up_i to be the ROC of some actual scoring function h in S|_s. The paper only proves (Theorem C.1) that a single point in the triangle with (0,0), (1,1), and Q^up_i can be realized by randomizing three classifiers. It never proves that the entire sequence of UpShift, LeftShift, or CutShift points produced by Algorithm 1 lies in the hypograph of ROCup, nor that the sequence is monotone in FPR and TPR so it can be realized as one threshold-parameterized ROC curve. Algorithm 1 chooses U_i or L_i solely by area comparison, with no feasibility check, and the proof of Theorem D.3 is a copy of Theorem D.2 rather than an argument for CutShift optimality. If some U_i lies above ROCup, or if the shifted list is non-monotone, then no h in the feasible set has that ROC: the object FROC outputs is not a classifier, the AUC-loss analysis compares phantom curves, and Theorem 4.2 does not follow. Assumption 4.2 does not rule out these failures, since it only constrains FPR ordering between the original curves and the number of intersections with norm boundaries.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes εp-Equalized ROC, a threshold-invariant group fairness notion for binary classifiers, and a post-processing algorithm FROC that transports the ROC of the disadvantaged group toward the other group's ROC so that the L1 distance between the two ROCs is at most ε at all thresholds. The central claim (Theorem 4.2) is that, under piecewise-linearity and Assumption 4.2, FROC returns the classifier that maximizes AUC among all transformations of the given scoring function satisfying ε1-Equalized ROC. The paper also reports experiments on Adult, COMPAS, and CelebA showing fairness improvements with small accuracy loss.","tokens_in":26055,"tokens_out":6045,"duration_ms":51544,"significance":"If the optimality theorem were valid, the paper would contribute a useful post-processing method with a provable AUC-optimality guarantee for a threshold-invariant fairness notion, backed by released code and empirical validation. The notion of εp-Equalized ROC is a reasonable generalization of equalized odds to all thresholds, and the geometric transport perspective is appealing. However, the central theoretical result is not established: the appendix proof of CutShift optimality is a copy of the Norm Boundary proof, and the paper never proves that the transported points are jointly realizable as the ROC of a single classifier in the ROC-space of s. These gaps block the main claim and also leave the fairness guarantee for all thresholds unverified.","major_comments":[{"comment":"The proof of Theorem D.3 (CutShift optimality) is verbatim identical to the proof of Theorem D.2 (Norm Boundary optimality) except for cosmetic figure references; it does not contain any argument specific to CutShift, such as why the closer of the two intersection points plef t and pright is AUC-optimal. Consequently Theorem 4.4 is unproved, and Theorem 4.2, which depends on it, is unsupported.","section":"Appendix D.3 / Theorem 4.4"},{"comment":"The paper proves only that a single point in the triangle with vertices (0,0), (1,1), and Q^up_i can be realized by randomizing three classifiers (Theorem C.1). It never proves that the sequence of points eQ^up_i produced by Algorithm 1 lies in the feasible ROC-space S|_s and is monotone in FPR and TPR. In fact, Algorithm 1 chooses UpShift or LeftShift purely by area comparison, so an UpShift at index i (FPR = FPR(Q^down_i)) followed by a LeftShift at index i+1 (FPR = FPR(Q^down_{i+1}) - ε) can decrease FPR when ε is large relative to the query spacing. Without monotonicity and feasibility, the polyline through eQ^up_i is not the ROC of any classifier h ∈ S|_s, and the AUC-loss analysis in Section 4 compares curves that need not correspond to valid classifiers.","section":"Section 3.3-3.4 and Theorem C.1"},{"comment":"Theorem 4.3 states that all optimal fair points must lie on some norm boundary, but Algorithm 1 itself outputs points Q^up_i that are already in the hypograph of ROCdown (line 12-14) and are not on any norm boundary. Moreover, if ||Q^up_i - Q^down_i||_1 < ε, moving to the norm boundary would strictly reduce AUC without improving fairness, contradicting the claimed necessity. The proof of Theorem D.2 also considers only a local quadrilateral area and does not account for global interactions among the transported points, so the contradiction argument is not rigorous.","section":"Theorem 4.3 and Algorithm 1"},{"comment":"Problem 2 requires ε1-Equalized ROC for all α ∈ (0,1), but FROC enforces the constraint only at the k query thresholds. Because the FPR coordinates of the transported points eQ^up_i generally differ from those of the corresponding Q^down_i, the piecewise-linear curves between query points are not guaranteed to stay within L1 distance ε; the manuscript provides no argument or bound for the intermediate thresholds, so the output may not satisfy the stated fairness constraint.","section":"Sections 2.3 and 3.3, fairness over all thresholds"},{"comment":"Even accepting the PLA bound, Theorem 4.2 is stated for the original piecewise-linear classifier, but the analysis mixes L_PLA (approximation loss) and L_AUC (transport loss). No bound on L_AUC is given, and the optimality claim is not tied to a quantitative total loss bound. This makes it difficult to interpret the sense in which FROC is provably optimal with respect to the original Problem 2.","section":"Section 4.1 and Theorem 4.2"}],"minor_comments":[{"comment":"The paper says randomization can represent any point in the hypograph of ROC, but the feasible set is actually the ROC convex hull; the distinction matters for points that are below the ROC curve but outside the convex hull of (0,0), (1,1), and the ROC.","section":"Section 3.3"},{"comment":"The empirical results are reported only through figures and informal statements (e.g., '7-8% improvement in fairness'); the paper should include a table with the actual ε1 fairness violation before/after, AUC before/after, and standard deviations for each classifier and dataset.","section":"Section 5.3"},{"comment":"The claim that FROC 'beats' FNNC because FNNC overachieves fairness is not a controlled comparison; the fairness targets of the two methods are different, so the comparison does not support the stated superiority.","section":"Section 5.2"},{"comment":"The implementation uses a hard-coded FPR range [init, fin] = [0.2, 1] that is not described in Algorithm 1 or in the theoretical analysis; this restricts the transported region and should be disclosed as a free parameter.","section":"Appendix F"},{"comment":"There are several typos and incomplete references: the placeholder '[? ]' in Section 1.1, the undefined variable 'label' in the Appendix F code, and inconsistent notation between ε1 and εp in Definition 2.2 and the theorems.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript's central theorem rests on appendix proofs that are not merely informal but contain a verbatim copy-paste (D.3 is D.2) and on an unproven feasibility/monotonicity property of the transported ROC points. These are load-bearing, not cosmetic. The algorithm also appears to contradict its own Norm Boundary theorem for already-fair query points. Fixing these would require substantial new analysis and possibly algorithm changes, so I do not see this as a minor-revision case. I would not encourage the editor to send this to a second round without a major methodological overhaul."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a good idea with an undercooked proof, and the central claim as written does not hold. The problem is real: practitioners want a single post-processing step that keeps a score-based classifier fair at every threshold, without retraining. The geometric transport of one group's ROC onto the L1 norm boundary of the other's is a workable intuition, and a linear-time algorithm is worth having. The epsilon1-Equalized ROC notion is useful, though very close to Chen and Wu's threshold-invariant fairness.\n\nWhat the paper does well: the problem formulation is honest, the PLA loss bound is correct, and the empirical plots on Adult, COMPAS, and CelebA show the code does pull the ROCs closer together. That is evidence the approach has practical legs.\n\nThe soft spots are in the center of the paper. The optimality theorem (Theorem 4.2) rests on appendix proofs that are informal and, in one place, broken: Appendix D.3 is a literal copy of D.2, so CutShift optimality is never actually proved. More importantly, no argument shows the shifted points produced by Algorithm 1 lie in the feasible region (the hypograph) of the original ROC, or that the sequence preserves monotone FPR/TPR ordering. Without that, the constructed classifier may fail to exist, and the AUC-loss comparison is against phantom curves. Assumption 4.2 does not fix this. The stress-test note is on target.\n\nTwo smaller issues. Definition 2.2 compares ROC curves at the same FPR (parameter alpha), while the abstract and algorithm promise fairness at the same score threshold; those differ and the paper never reconciles them. Also, fairness is only checked at k query points, with no bound on violation between queries, so the all-thresholds guarantee is not rigorous. Both are fixable.\n\nThe shipped code implements a simpler algorithm than Algorithm 1, with hardcoded FPR range and different upshift/cut logic. That is a reproducibility problem. The experiments are thin: baseline comparisons lack error bars, and the FNNC \"overachieves fairness\" explanation is asserted, not shown.\n\nOverall, the paper deserves a serious referee, but the submission should be rejected or, at best, returned for major revision. The idea is salvageable; the proofs and code are not ready.","headline":"A clever and practical post-processing idea, but the optimality proof misses the key feasibility step, and the shipped code does not match the algorithm.","tokens_in":26579,"tokens_out":3964,"would_cite":false,"duration_ms":44210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A post-processing algorithm called FROC transforms any trained scoring function into a randomized classifier whose per-group ROC curves lie within ε of each other at every threshold, and proves this is done with minimal AUC loss when the…","keywords":["fairness","ROC curve","post-processing","AUC optimization","equalized odds","threshold-invariant fairness","randomized classifier","ROC transport"],"falsifier":"Construct or find a piecewise-linear classifier whose upper ROC intersects some norm boundary B_i in more than two points (or violates the FPR ordering), run FROC, and compare its AUC against a brute-force optimum among ε1-Equalized ROC classifiers in the same ROC-space; any gap would falsify Theorem 4.2's optimality claim for that case, although the paper's own statement is that the guarantee would not hold there.","tokens_in":25467,"feed_emoji":"⚖️","tokens_out":4310,"duration_ms":37979,"temperature":0.7,"pith_summary":"The paper attacks a practical mismatch: fairness definitions like equalized odds are checked at one threshold, but practitioners read classifiers off the whole ROC and change thresholds freely. It proposes ε1-Equalized ROC, requiring the FPR/TPR curves of the two protected groups to stay within L1 distance ε for every threshold. It then gives FROC, a post-processing algorithm that transports the higher ROC down toward the lower one by randomizing between existing classifiers, and proves that under a piecewise-linearity and regularity assumption this transport is AUC-optimal. If the proofs hold, practitioners get a plug-in fix that guarantees threshold-independent fairness without retraining.","feed_headline":"FROC provably equalizes ROC curves with minimal AUC loss","feed_subtitle":"Post-processing moves the higher group's ROC into the lower one's ε-diamond at every threshold, without retraining.","key_machinery":"The key mechanism is ROC-space randomization: any point inside the convex hull of three classifiers in (FPR, TPR) coordinates is realized by a randomized classifier that mixes the three with weights given by barycentric coordinates. FROC's operation is transport of the upper ROC onto the norm-boundary rhombus B_i = {x : ||x − Q^down_i||_1 = ε} around each lower-curve query point, choosing between UpShift (to the top vertex U_i), LeftShift (to the left vertex L_i), or CutShift (to an intersection point of ROC_up with B_i) by comparing quadrilateral areas via Heron's formula. The optimality argument runs through three lemmas: optimal fair points lie on the norm boundary; boundary cuts must be CutShift; otherwise the area comparison picks UpShift or LeftShift.","core_discovery":"The central claim is that FROC solves Problem 2: among all transformations h of a given scoring function s that can be realized in the ROC-space of s and satisfy ε1-Equalized ROC, FROC returns one with maximum AUC. The algorithm queries each group's ROC at k equally spaced thresholds, forms piecewise-linear approximations, and moves every query point of the 'up' curve to the boundary of the ε-diamond around the corresponding 'down' point, choosing the vertex or intersection point that minimizes the local AUC loss. Theorem 4.2 states this is optimal provided s is piecewise linear and Assumption 4.2 holds.","pith_inferences":["The optimality guarantee depends on the non-intersection assumption on ROCs and on Assumption 4.2, so the guarantee should not be expected to carry over to arbitrary classifiers; the paper itself leaves intersecting ROCs as future work.","Violations of Assumption 4.2 are plausible for wiggly ROC curves; FROC would still produce a fair classifier, but the practical cost is bounded only by how far the optimal fair point can deviate from the norm boundary in those cases.","Because FROC only needs ROC query access, it is model-agnostic and could be applied to any score-producing system, including ensembles or commercial APIs, as long as per-group ROC curves can be estimated.","A direct empirical test of the optimality claim would be a brute-force search over randomized mixtures in the ROC-space on a small dataset, comparing FROC's AUC loss against the true minimum among ε1-Equalized ROC classifiers."],"forward_implications":["A practitioner can post-process an already-trained classifier once and then choose any threshold later, because fairness holds at all thresholds.","For piecewise-linear scoring functions satisfying Assumption 4.2, no other fair ROC-transport can achieve the same AUC; FROC's AUC loss is the minimum possible (up to the piecewise-linear approximation loss, which shrinks as k grows).","The approximation loss is bounded by L_PLA ≤ (1/2) u_T u_F / k, so increasing the number of ROC queries k reduces the loss to zero in the limit.","Empirically, on Adult, COMPAS, and CelebA with neural, logistic, and random-forest classifiers, FROC improves fairness (e.g., disparate impact and mean equalized odds) with at most about a 2% accuracy drop, and for small ε it can beat in-processing baselines that over-achieve the target fairness."],"supporting_citations":[{"why":"Provides the fact that any point in the hypograph or convex hull of the ROC can be realized by randomization, which underpins FROC's classifier construction and Theorem C.1.","marker":"[38]"},{"why":"Supplies the ROC-based fairness framework and the slope-boundedness assumption that FROC relaxes in Assumption 4.1.","marker":"[26]"},{"why":"Introduces the threshold-invariant fairness notion that ε1-Equalized ROC generalizes.","marker":"[27]"},{"why":"Defines equalized odds, the error-rate fairness concept that FROC extends to all thresholds.","marker":"[19]"},{"why":"Provides the FNNC in-processing baseline and its hyperparameters, which FROC is compared against and can beat in accuracy for small ε.","marker":"[10]"},{"why":"Supplies the FairProjection-CE and FairProjection-KL post-processing baselines used for empirical comparison.","marker":"[41]"}],"fun_headline_variants":["FROC: fair ROC curves with optimal AUC retention","Equalize ROC curves, minimize AUC loss","Post-hoc fairness for any classifier's ROC","Optimal fair ROC transformation without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The optimality guarantee holds only when the upper ROC is well-separated and regular: Assumption 4.2 requires the FPR of the upper curve at each query point to lie between the FPRs of neighboring lower-curve points, and the upper curve to meet every ε-diamond boundary at most twice; the paper concedes that without it FROC remains fair but is not proven optimal.","fun_headline_variants_meta":{"raw":{"variants":["FROC: fair ROC curves with optimal AUC retention","Equalize ROC curves, minimize AUC loss","Post-hoc fairness for any classifier's ROC","Optimal fair ROC transformation without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2355,"prompt_tokens":990,"completion_tokens":1365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":1308}},"tokens_in":606,"tokens_out":1365,"duration_ms":7950,"temperature":1.0,"reasoning_tokens":1308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:58:56.900676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or find a piecewise-linear classifier whose upper ROC intersects some norm boundary B_i in more than two points (or violates the FPR ordering), run FROC, and compare its AUC against a brute-force optimum among ε1-Equalized ROC classifiers in the same ROC-space; any gap would falsify Theorem 4.2's optimality claim for that case, although the paper's own statement is that the guarantee would not hold there.","supporting_citations":[{"cited_title":"Fairness and Machine Learning","cited_arxiv_id":null,"evidence_quote":"Provides the fact that any point in the hypograph or convex hull of the ROC can be realized by randomization, which underpins FROC's classifier construction and Theorem C.1."},{"cited_title":"Learning fair scoring functions: Bipartite ranking under roc-based fairness constraints","cited_arxiv_id":null,"evidence_quote":"Supplies the ROC-based fairness framework and the slope-boundedness assumption that FROC relaxes in Assumption 4.1."},{"cited_title":"Towards threshold invariant fair classification","cited_arxiv_id":null,"evidence_quote":"Introduces the threshold-invariant fairness notion that ε1-Equalized ROC generalizes."},{"cited_title":"Equality of opportunity in supervised learning","cited_arxiv_id":null,"evidence_quote":"Defines equalized odds, the error-rate fairness concept that FROC extends to all thresholds."},{"cited_title":"Fnnc: Achieving fairness through neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the FNNC in-processing baseline and its hyperparameters, which FROC is compared against and can beat in accuracy for small ε."},{"cited_title":"Beyond adult and compas: Fair multi-class prediction via information projection","cited_arxiv_id":null,"evidence_quote":"Supplies the FairProjection-CE and FairProjection-KL post-processing baselines used for empirical comparison."}],"review_version":1}