{"id":"aa276394-ad50-42db-9427-7bf9358d68f2","arxiv_id":"2505.24097","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"An iterative threshold calibration method, Performative Risk Control, gives finite-sample guarantees that risk stays controlled under performative (self-influencing) distribution shifts.","lead":"This paper proposes Performative Risk Control (PRC), an iterative calibration procedure that tunes a decision threshold so that a blackbox model's risk stays below a user-specified level even when the model's own decisions change the data distribution. The contribution is a finite-sample statistical guarantee for risk control when predictions are performative, which matters for safely deploying models in credit, pricing, and other decision systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.3's proof applies Proposition A.4 to bλ_t, but bλ_t is not the plain UCB threshold; without an augmented-risk counterfactual for f(λ)=R(bλ_{t−1},λ)+τ(bλ_{t−1}−λ), the anytime-safety and tightness guarantees are unsupported as written.","rationale":"The paper's central assertion is Theorem 3.3: with probability 1−δ, Algorithm 1 simultaneously enjoys iterative safety, anytime safety, and tightness. The proof of that theorem rests on Theorem A.5, and the load-bearing step in Theorem A.5 is the application of Proposition A.4 to the selected threshold bλ_t. That step is not justified by the text. Proposition A.4 is a restatement of the standard UCB calibration lemma, whose selected threshold is defined purely from the UCB of the true risk; Algorithm 1's threshold also contains the linear performativity guard τ(bλ_{t−1}−λ), so bλ_t solves a different equation. Consequently, the event on which Proposition A.4 gives coverage is not the event needed at bλ_t. The missing argument is the well-known counterfactual augmentation: define f(λ)=R(bλ_{t−1},λ)+τ(bλ_{t−1}−λ) and its UCB, then bλ_t is the canonical UCB threshold for f. With this repair, the inequalities in Theorem A.5 likely close, so the framework may well be correct. I therefore do not recommend a verdict change from the reader's CONDITIONAL. I do not select Assumption 3.2 as the single most load-bearing concern: sensitivity assumptions are legitimate modeling conditions, and the proof gap is a correctness issue that bites regardless of whether the assumption holds. The reader's rationale mentions the same augmentation gap, so my read partially agrees with the reader; the weakest_assumption field, however, points to a different (albeit related) condition.","tokens_in":21035,"tokens_out":9231,"duration_ms":93716,"concrete_test":"Re-derive Theorem A.5 Case 2 with f(λ)=R(bλ_{t−1},λ)+τ(bλ_{t−1}−λ) and \\hat f^+(λ)=\\hat R_n^+(bλ_{t−1},λ)+τ(bλ_{t−1}−λ). Verify that bλ_t equals inf{λ : \\hat f^+(λ) ≤ α}, that \\hat f^+ is continuous and non-increasing, and that the hypotheses of Proposition A.4 hold for f (including existence of a λ with f(λ)≤α and \\hat f^+(λ)≤α, e.g. λsafe). If the repaired proof goes through, Theorem 3.3 stands; if only f(bλ_t)≤α is obtained, confirm that the final conversion to R(bλ_t,bλ_t)≤α uses τ≥γM and the union bound exactly as claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition A.4 in App. A.4 proves safety only for the standard UCB threshold λ*_+ := inf{λ : \\hat R_n^+(bλ_{t−1},λ;δ') < α}. In Algorithm 1, however, bλ_t is the minimizer of V(bλ_{t−1},λ,δ) = \\hat R_n(bλ_{t−1},λ) + c(n,δ/T̃) + τ(bλ_{t−1}−λ), so the selected threshold is not λ*_+. The proof of Theorem A.5, Case 2, contains the chain R(bλ_t) ≤ R(bλ_{t−1},bλ_t) + γM|bλ_t−bλ_{t−1}| ≤ \\hat R_n^+(bλ_{t−1},bλ_t,δ/T̃) + γM|bλ_t−bλ_{t−1}|, citing Proposition A.4 for the second inequality. That citation is not available as stated: Proposition A.4's probability statement concerns the risk at λ*_+, not at the data-dependent minimizer of V. The standard repair is to run the UCB argument on the counterfactual augmented risk f(λ) = R(bλ_{t−1},λ) + τ(bλ_{t−1}−λ), whose natural UCB is \\hat f^+(λ) = \\hat R_n^+(bλ_{t−1},λ) + τ(bλ_{t−1}−λ). Then bλ_t is exactly the plain UCB threshold for f, and a Bates-style argument would give f(bλ_t) ≤ α with probability 1−δ/T̃. This is a specific, addressable derivation gap rather than a demonstrated counterexample, but until it is supplied, Theorem 3.3 is not proven by the submitted text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Performative Risk Control (PRC), an iterative post-hoc calibration procedure for a threshold parameter λ so that the performative risk R(λ)=E_{z∼D(λ)}ℓ(z,λ) is controlled. Algorithm 1 samples from the distribution induced by the current threshold and sets the next threshold as the infimum of the set {λ : \\hat R_n(\\hat λ_{t−1},λ)+c(n,δ/\\tilde T)+τ(\\hat λ_{t−1}−λ)≤α}, stopping when the progress is below Δλ. Theorem 3.3 claims that, under a Wasserstein-Lipschitz condition on D(·) with constant γ and τ≥γ, with probability 1−δ all iterates satisfy safety of the deployed pair R(\\hat λ_{t−1},\\hat λ_t)≤α, safety at anytime R(\\hat λ_t)≤α, and tightness of the final threshold R(\\hat λ_T)≥α−Δα. The paper also extends the procedure to quantile-based risk measures (Theorem 4.2) and reports experiments on credit default risk for expected type-II error and 90%-CVaR.","tokens_in":21472,"tokens_out":8001,"duration_ms":76545,"significance":"If the main theorem is established, this is a genuinely useful bridge between conformal risk control and performative prediction: it gives an anytime-safe, iteratively refined calibration procedure with a clean finite-sample guarantee in a setting where the data distribution reacts to the deployed decision rule. The paper has concrete strengths: it provides several explicit confidence-width constructions (Hoeffding, Bernstein, Hoeffding-Bentkus, CLT), a nontrivial quantile-risk extension, and experiments that simulate the performative shift and attempt to verify the sensitivity constant. The central proof is, however, incomplete as written: the key step applies a UCB-calibration proposition to an iterate that is not the plain UCB threshold, and the pointwise confidence statement in Eq. (1) is data-dependent. These are specific, addressable gaps rather than demonstrated counterexamples, so the contribution is promising but not yet proven.","major_comments":[{"comment":"The citation of Proposition A.4 in the second inequality of Theorem A.5, Case 2 is not available as written. Proposition A.4 proves safety at λ_*^+:=inf{λ′: \\hat R_n^+(λ,λ′;δ′)<α}, whereas Algorithm 1 selects \\hat λ_t as the infimum over {λ: V(\\hat λ_{t−1},λ,δ)≤α}, with V containing the additional term τ(\\hat λ_{t−1}−λ). The proof therefore jumps from R(\\hat λ_t) ≤ R(\\hat λ_{t−1},\\hat λ_t)+γM|\\hat λ_t−\\hat λ_{t−1}| to ≤ \\hat R_n^+(\\hat λ_{t−1},\\hat λ_t,δ/\\tilde T)+γM|\\hat λ_t−\\hat λ_{t−1}| without a valid probabilistic statement at the data-dependent point \\hat λ_t. A standard repair is to run the UCB argument on the augmented risk f(λ)=R(\\hat λ_{t−1},λ)+τ(\\hat λ_{t−1}−λ), whose natural upper confidence bound is \\hat f^+(λ)=\\hat R_n^+(\\hat λ_{t−1},λ,δ/\\tilde T)+τ(\\hat λ_{t−1}−λ), and to prove that the inf-threshold of \\hat f^+ controls f. Until this argument is supplied, the anytime-safety and tightness guarantees in Theorem 3.3 are unsupported.","section":"App. A.4, Theorem A.5 and Proposition A.4"},{"comment":"The pointwise confidence statement in Eq. (1) is formulated for \"bλ_{t−1} and λ′≥bλ_t for all t encountered in the procedure,\" so the event itself depends on the algorithm's data-dependent iterates. This is not a well-defined pointwise coverage statement, and it cannot be used as a hypothesis in Proposition A.4 without an explicit union bound over the thresholds that may be visited. The construction of c(n,δ′) in App. A.2 uses worst-case variance but does not prove a uniform-in-λ bound with failure probability δ/\\tilde T when the iterates are stopping times. The proof should either state a simultaneous confidence band over a fixed grid or over all λ′∈Λ with total error δ/\\tilde T, or justify the stopping-time argument directly.","section":"Eq. (1), §3.2 and App. A.2"},{"comment":"The empirical claims are conditioned on an estimated sensitivity constant: γ is bounded by p·C with C read off a 20-bin histogram of the balanced subset, and the same balanced subset is used to evaluate the algorithm. The theorem requires the exact γ, and if the estimated γ is smaller than the true value, the anytime guarantee can fail. This is not a circularity in the theorem, because τ is not fitted to the target guarantee; however, it is a mismatch between the strength of the claimed finite-sample guarantee and the experimental verification. The paper should either state the empirical results as heuristic or provide a valid confidence upper bound for γ derived from a separate calibration set.","section":"App. B.1 and §5"}],"minor_comments":[{"comment":"There are typos in \"initate\" (Section 1) and \"Hoeffding-Benkus\" (App. A.2 heading and text; should be Hoeffding-Bentkus).","section":"Section 1 and App. A.2"},{"comment":"The shorthand R(λ) is used in Theorem 3.3 without a formal definition; the paper should define R(λ):=R(λ,λ) at first use, since both R(λ) and R(λ,λ′) appear throughout.","section":"Notation, Theorem 3.3"},{"comment":"The proof of Theorem 4.2 is one sentence saying it follows from App. A.4; given the additional CDF confidence bands and the Hölder conditions, a more detailed proof would help the reader verify that the quantile confidence width satisfies the same union-bound requirements.","section":"Theorem 4.2"}],"recommendation":"major_revision","confidential_remarks":"The proof gap in Theorem 3.3 is central but likely repairable with an augmented-risk UCB argument and a simultaneous confidence statement. I recommend major revision rather than rejection: the framework and experiments are valuable, but the submitted text does not yet prove its main finite-sample guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing you should know: this paper has a genuinely new idea—finite-sample risk control under performative distribution shift—but the proof of the main theorem has a concrete gap that needs repair before the claims are fully supported. I think the gap is fixable, so it's a conditional revise, not a reject.\n\nWhat's new: PRC extends the UCB calibration framework of Bates et al. to a setting where the distribution depends on the deployed threshold. The iterated procedure with a Lipschitz-sensitivity assumption on the distribution map is a natural and useful generalization, and the anytime-safety guarantee—risk controlled at every iterate, not just the final one—is a nice property for policy deployment. The quantile-risk extension (CVaR et al.) is a plus. The experiments on credit scoring are consistent with the claims and reproducible.\n\nNow the soft spots, in order of importance. The gap: in Theorem A.5, Case 2, the proof bounds R(λhat_t) using Proposition A.4, but that proposition only certifies the standard UCB threshold λ*_+ = inf{λ' : \\hat R_n^+(λ_prev, λ') < α}. Algorithm 1 selects a different object—the minimizer of \\hat R_n(λ_prev,λ) + c + τ(λ_prev−λ). The citation therefore doesn't apply as written. The standard repair is to run the UCB argument on the augmented risk f(λ) = R(λ_prev,λ) + τ(λ_prev−λ), whose natural upper confidence bound has the same form, and then τ(λ_prev−λ) is already inside the UCB. That's a local fix but it's load-bearing: without it, the anytime-safety and the proof of part (i) are not rigorous. I checked the rest of the chain; part (iii) uses pointwise bounds directly and looks fine.\n\nSecond, Assumption 3.2 (γ-sensitivity of the loss distribution) is simply assumed. In the experiments, γ is estimated from the same balanced dataset used to evaluate the algorithm, so the reported failure probabilities aren't a pure test of the method. That's not a flaw in the theory, just a caveat on the empirical claims. The paper does say γ is a domain-knowledge parameter, which is honest.\n\nThird, the 'first to study risk control under performativity' claim should be delimited against strategic conformal prediction (Csillag et al.) and distribution-shift-robust conformal methods. The current related work mentions them but doesn't draw the boundary sharply; a referee should ask for that.\n\nWho is this for: anyone working on calibration, conformal risk control, or performative prediction. It's a solid subfield contribution, not a breakthrough, but the framework is likely to be cited. My recommendation: send to peer review. The gap is specific and addressable; a serious referee could get the authors to fix it in a revision. I'd accept it as a conditional, expecting a corrected proof and a sharper comparison.","headline":"Genuinely new framework for risk control under performativity, but the main proof has a fixable gap; send to peer review.","tokens_in":21974,"tokens_out":4296,"would_cite":true,"duration_ms":36952,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new calibration procedure proves finite-sample risk control even when predictions reshape the data.","keywords":["performative prediction","risk control","finite-sample guarantees","conformal prediction","anytime safety","quantile risk measures","credit scoring","threshold calibration"],"falsifier":"Run Algorithm 1 on a constructed distribution map where a small threshold decrease causes a mass of loss values to jump from 0 to 1 (a super-Lipschitz shift with no finite γ), with τ set to any finite value; if the middle iterations or final threshold ever show empirical risk above α at a rate exceeding δ, the anytime-safety claim is falsified. Alternatively, on any real dataset, measure W1(D_l(λ1), D_l(λ2)) for nearby thresholds and check whether it exceeds the assumed γ|λ1−λ2|.","tokens_in":20830,"feed_emoji":"🛡️","tokens_out":7932,"duration_ms":67415,"temperature":0.7,"pith_summary":"Safely calibrating a black-box model usually assumes the calibration data look like the deployment data. This paper treats the harder case where the act of deploying a threshold changes the distribution of future data — a phenomenon called performativity, common in credit scoring and other policy settings. It introduces Performative Risk Control, an iterative procedure that at each round uses samples drawn under the current threshold to pick a less conservative threshold, adding a 'performativity guard' that anticipates how the distribution will shift. Under the assumption that the loss distribution moves Lipschitz-continuously with the threshold (speed at most γ), choosing the guard τ ≥ γ yields a finite-sample guarantee, holding with probability 1−δ, that every intermediate threshold and the final threshold keep the expected risk at or below the user level α, while the final threshold is never much more conservative than needed. The same guarantees extend to quantile-based risk measures such as CVaR.","feed_headline":"Calibration that stays safe even when predictions reshape the data","feed_subtitle":"Guarantees expected risk stays ≤ α at every iteration, with high probability, under performative shifts.","key_machinery":"The engine is the update rule $\\lambda_t = \\inf\\{\\lambda \\in \\Lambda : \\hat{R}_n(\\lambda_{t-1}, \\lambda) + c(n, \\delta/\\tilde{T}) + \\tau(\\lambda_{t-1} - \\lambda) \\le \\alpha\\}$, together with a stopping rule that terminates when $\\lambda_{t-1} - \\lambda_t < \\Delta\\lambda$. Here $\\hat{R}_n$ is the empirical loss evaluated at the candidate threshold on samples drawn under the previous threshold, $c(n, \\delta/\\tilde{T})$ is a pointwise confidence width from a concentration bound (Hoeffding, Bernstein, Hoeffding–Bentkus, or CLT), and the term $\\tau(\\lambda_{t-1} - \\lambda)$ is the performativity guard. The guard is what converts the assumption that the loss distribution moves at speed at most $\\gamma$ into an upper bound on the performative error, so that $\\tau \\ge \\gamma$ makes each deployed threshold safe against the distribution shift it induces. The stopping rule and the joint choice of $\\tilde{T}$ and $\\Delta\\lambda$ ensure the process terminates and the final threshold is tight.","core_discovery":"The paper's central claim is that a simple, iterative, defensively conservative threshold update converts static-distribution risk control into performative risk control without sacrificing finite-sample guarantees. Starting from a guaranteed-safe threshold λ_safe, each round samples data from the distribution induced by the previously deployed threshold, forms an empirical upper confidence bound for the loss evaluated at candidate thresholds, and adds the term τ(λ_{t−1}−λ) — the performativity guard — to bound the error that arises because deploying λ moves the distribution itself. The procedure then moves to the most aggressive threshold whose guarded upper bound still lies at or below α, and stops when further progress would be smaller than a precomputed increment Δλ. Provided the distribution map is (γ,1,ℓ(·,λ))-sensitive for all λ (loss distributions shift by at most γ in 1-Wasserstein distance per unit threshold change), the loss is continuous and monotone in λ, and τ ≥ γ, Algorithm 1 proves with probability 1−δ that the risk stays at or below α at every step along the trajectory and that the returned threshold satisfies R(λ_T) ≥ α − Δα. The proof rests on a lemma bounding the performative error by γ times the threshold gap, a UCB-calibration argument adapted from the static risk-control literature, and a union bound over the at most T̃ iterations.","pith_inferences":["The conservative choice $\\tau \\ge \\gamma$ with $\\gamma$ estimated from a single balanced dataset may be loose in practice; estimating $\\gamma$ per iteration or from streaming deployment data could yield smaller $\\Delta\\alpha$ while preserving the proof structure — a modification the authors do not investigate.","The monotonicity assumption on the loss confines the method to parameters where larger values are uniformly safer; for non-monotone post-processing knobs such as temperature with an optimum in the interior, a two-sided or signed guard would be needed.","The sensitivity assumption is stated in Wasserstein-1 distance on losses, so a practical validation route is to estimate the Lipschitz constant empirically from repeated deployments and check whether observed distribution shifts stay within the assumed bound."],"forward_implications":["An institution can deploy a sequence of increasingly aggressive thresholds, each guaranteed (with probability $1-\\delta$) to keep expected risk at or below $\\alpha$, even when applicants or agents adapt their behavior to the deployed threshold.","The user-specified tightness $\\Delta\\alpha$ can be driven to zero as the sample size grows, so the final threshold is asymptotically as aggressive as the risk budget allows.","The anytime-safety property means the guarantee holds for every intermediate deployment, not just the final one, which is essential for monitoring policies that are rolled out incrementally.","Because the framework only post-processes a black-box predictor and never refits it, the same procedure applies to any threshold-indexed prediction rule with a monotone loss, including prediction sets and standard classifiers."],"supporting_citations":[{"why":"Supplies the risk-controlling prediction-set framework and the UCB-calibration validity proposition that PRC restates in the performative setting.","marker":"Bates et al. (2021)"},{"why":"Extends risk control to general bounded losses and provides the conformal risk control setup that this work generalizes to performative distributions.","marker":"Angelopoulos et al. (2022b)"},{"why":"Introduces performative prediction and the ε-sensitivity assumption, which the paper generalizes to sensitivity of the induced loss distribution.","marker":"Perdomo et al. (2020)"},{"why":"Provides the quantile risk control machinery used for the CVaR extension of the algorithm.","marker":"Snell et al. (2022)"},{"why":"Supplies the Hoeffding–Bentkus p-values used as one option for the confidence width.","marker":"Angelopoulos et al. (2022a)"},{"why":"Supplies the empirical Bernstein bound used as another confidence-width option.","marker":"Maurer and Pontil (2009)"},{"why":"Supplies the functional CLT used to derive the quantile-based confidence width in the CVaR experiment.","marker":"Vaart (1998)"}],"fun_headline_variants":["Risk control that holds even when predictions shift data","Finite-sample risk bounds despite performative feedback","Calibrate safely when predictions reshape the input distribution","Guaranteed risk control under performative model shifts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the loss distribution changes at a bounded rate — at most γ per unit change in the threshold, measured in 1-Wasserstein distance — so the shift induced by any deployment step is no larger than the guard τ times the step size.","fun_headline_variants_meta":{"raw":{"variants":["Risk control that holds even when predictions shift data","Finite-sample risk bounds despite performative feedback","Calibrate safely when predictions reshape the input distribution","Guaranteed risk control under performative model shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001233,"raw_usage":{"total_tokens":5104,"prompt_tokens":1021,"completion_tokens":4083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":4022}},"tokens_in":637,"tokens_out":4083,"duration_ms":24749,"temperature":1.0,"reasoning_tokens":4022,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:06.777220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on a constructed distribution map where a small threshold decrease causes a mass of loss values to jump from 0 to 1 (a super-Lipschitz shift with no finite γ), with τ set to any finite value; if the middle iterations or final threshold ever show empirical risk above α at a rate exceeding δ, the anytime-safety claim is falsified. Alternatively, on any real dataset, measure W1(D_l(λ1), D_l(λ2)) for nearby thresholds and check whether it exceeds the assumed γ|λ1−λ2|.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the risk-controlling prediction-set framework and the UCB-calibration validity proposition that PRC restates in the performative setting."},{"cited_title":"and Pontil, M","cited_arxiv_id":null,"evidence_quote":"Supplies the empirical Bernstein bound used as another confidence-width option."}],"review_version":1}