{"id":"eef42e89-4368-4bb2-b77c-14e20e78c88d","arxiv_id":"2607.24562","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HG-CRC enforces simultaneous per-group risk guarantees in LLM selective prediction via Bonferroni correction over hierarchy nodes and a leaf-first threshold policy, reaching 0% empirical violations and WGER=0 on ARC Challenge for two models.","lead":"This paper introduces a post-hoc calibration method that gives per-group error guarantees for language models that can abstain from answering. It tests the method on multiple-choice benchmarks and shows it can keep every group within a 10% error budget, at the cost of answering fewer questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-sample default, not the formally guaranteed split variant, produces the headline zeros; Lemma 1's worst-case slack ε_n≈0.34 makes the deployed guarantee vacuous. Verify the split variant at 500 trials.","rationale":"The reader's weakest assumption is exactly the in-sample slack under Assumption 1, and I agree that this is the most load-bearing concern. The only unconditional formal guarantee, Proposition 1, attaches to the split variant, which is not the procedure used in the main experiments. The deployed in-sample variant is covered only by Lemma 1, and the paper's own Section 8.3 concedes that the worst-case measured slack ε_n≈0.34 makes that guarantee materially weaker than the headline numbers suggest. This is a correctness risk rather than a stylistic issue: the abstract's '0% violation and WGER=0' are empirical upper bounds from 500 bootstrap trials and are not backed by a non-vacuous simultaneous bound for the deployed procedure. The paper deserves credit for flagging this limitation, for implementing and partially checking the split variant in Remark 2, and for the honest equal-participation comparison in Section 8.2, which narrows but does not falsify the method's value as an automatic safe operating-point selector. Because the split variant already exists and differs in participation by only 0.6–5.5pp in Remark 2, the missing 500-trial split comparison is a tractable, decisive test rather than a fundamental obstacle. A conditional verdict remains appropriate: the method is promising and mostly sound, but the central empirical claim should be re-verified under the formally guaranteed procedure before acceptance.","tokens_in":24388,"tokens_out":12203,"duration_ms":126902,"concrete_test":"Run the split variant (Algorithm 1 with mode=split, δ̂=δ/|H|) on ARC Challenge for Qwen3-4B and Llama-3.1-8B using the exact 500-trial bootstrap protocol of Section 4.5, reporting violation rate, WGER, and participation for the configurations of Tables 2 and 5. If the split variant attains 0% violations and WGER=0 with participation within the 0.6–5.5pp cost measured in Remark 2, the central claim is supported by the unconditional Proposition 1. If the split variant shows violations or much lower participation, the reported in-sample safety is not certified and the abstract's claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's unconditional simultaneous guarantee is Proposition 1, which holds only for split calibration. All headline experiments deploy the in-sample variant, whose only formal support is Lemma 1, adding ε_n to each node's bound and |H|·ε_n to the simultaneous bound. Section 8.3 reports a worst-case jackknife slack ε_n≈0.34 on the small ARC pool, so for the five-node ARC hierarchy δ+|H|·ε_n≈1.75: the deployed variant has no meaningful worst-case guarantee. The reported mean slack below 1% does not certify the worst case, and Assumption 1 (threshold regularity with margin γ≫1/n) is an unverified condition on the score distributions. Remark 2's split-variant check uses 200 paired resamples and reports participation cost, not a full WGER/violation comparison at the paper's 500-trial resolution. The headline 0% violation and WGER=0 are therefore empirical properties of the in-sample procedure, not consequences of the formal simultaneous guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HG-CRC, a conformal risk control method that calibrates thresholds for every node of a user-specified group hierarchy and applies a Bonferroni correction to provide simultaneous per-node risk guarantees in selective prediction with abstention. A leaf-first policy routes each example to the deepest certified ancestor whose score threshold it satisfies. Three theoretical results are offered: Proposition 1 gives an unconditional simultaneous bound for a split-calibration variant, Proposition 2 gives an exact bound for an in-sample variant with test-point augmentation, and Lemma 1 covers the deployed plain in-sample variant up to an additive slack ε_n. The empirical study covers three LLMs and two benchmarks, reporting a 0% empirical violation rate and WGER=0 on ARC Challenge for Qwen3-4B and Llama-3.1-8B, with participation cost of 22–37 pp relative to global CRC, plus ablations, robustness checks, and an equal-participation comparison that shows a coverage-matched global threshold controls per-group risk about as well as HG-CRC in most settings.","tokens_in":24520,"tokens_out":10481,"duration_ms":94930,"significance":"If the formal guarantees and empirical results hold, HG-CRC is a useful post-hoc calibration framework for group-conditional selective prediction in LLMs, requiring no retraining and providing simultaneous certificates over hierarchical groups. The paper is unusually transparent: it explicitly caveats that the headline zeros are empirical upper bounds at 500 bootstrap trials, that difficulty bins are endogenous to the deployed score, that worst-case slack for the deployed guarantee is ε_n≈0.34 on the small ARC pool, and that at equal participation a single global threshold matches HG-CRC on per-group safety in most measured settings. The split-variant guarantee (Proposition 1) is the paper's main formal contribution and its proof is coherent. The study also ships a 500-trial evaluation with rule-of-three error bars, an exogenous-difficulty robustness check, and an honest equal-participation analysis, which are strengths.","major_comments":[{"comment":"The stated bound \"P(R_v(τ*_v)>α | ρ_v(x)=1) ≤ δ̂+ε_n\" is not what the proof establishes. The proof defines ε_n as the marginal disagreement probability P(â(x)≠â+(x)), but then derives a conditional slack of ε_n/π_v, where π_v=P(g_τ(x)=1,ρ_v(x)=1), and the text immediately after the proof concedes that \"the conditional slack the lemma adds to δ̂ is ε_n/π_v\". For sparse nodes (small π_v, e.g., Gemma's leaf cells), ε_n/π_v can greatly exceed ε_n. The simultaneous bound should therefore be δ + |H|·sup_v(ε_n/π_v) (or an analogous expression), not δ + |H|·ε_n. With the measured worst-case marginal slack ε_n≈0.34 on the small ARC pool (Section 8.3), and |H|=5, the bound δ+|H|ε_n is already vacuous; using the correct conditional slack makes the deployed variant's worst-case guarantee even weaker. This is a load-bearing error because Lemma 1 is the only formal support for the deployed default procedure used in all experiments. The lemma must be restated with the correct conditional slack, and the numerical discussion in Section 8.3 must be revised to report conditional worst-case slack at the nodes that actually answer.","section":"Section 3.5, Lemma 1"},{"comment":"The formal simultaneous guarantee (Proposition 1) holds only for the split variant, yet the split variant is never evaluated at the paper's headline 500-trial resolution. Remark 2 reports a 200-resample comparison of participation costs and a McNemar test on node-violation indicators, but it does not report the split variant's WGER or violation rate at 500 trials, nor does it provide the split-variant analog of the Table 2 headline numbers. Since the abstract and Section 5.2 present the 0% violation and WGER=0 as the main empirical result for the deployed in-sample variant, and since the in-sample variant's guarantee is only approximate (Lemma 1, with the issue above), the paper should report the split variant's WGER and violation rate at the same 500-trial resolution, ideally as a primary result or as a clearly labeled alternative. Without this, the reader cannot verify that the variant carrying the rigorous guarantee actually realizes the claimed empirical behavior in the regimes of interest.","section":"Section 3.5, Remark 2 and Section 8.3"},{"comment":"The equal-participation analysis shows that, at matched coverage, a single global threshold controls per-group risk about as well as HG-CRC in five of six cells, with the exception of Gemma under mixture shift. This is an honest and important qualification of the empirical contribution, but it should be reflected in the abstract and introduction: the headline \"0% violation and WGER=0\" could be read as evidence that group-conditional calibration inherently protects subgroups better than any single threshold, whereas the paper's own data indicate the effect is mostly a participation effect. The authors do state this in Section 8.2, but the framing in the Abstract and Section 1 (\"enforcing simultaneous risk guarantees\") should explicitly acknowledge that the empirical per-group safety margin over a coverage-matched global threshold is small, and that the value of HG-CRC lies in the automatic selection of a safe operating point and in the formal simultaneous certificate of the split variant.","section":"Section 8.2, Table 11"}],"minor_comments":[{"comment":"The sentence \"Nodes below this threshold are pruned from H, reducing |H| and accordingly tightening the per-node budget δ̂\" is backwards: reducing |H| increases δ̂=δ/|H|, i.e., loosens the per-node budget. The sentence should say \"increasing\" or \"loosening\".","section":"Section 3.2"},{"comment":"It is unclear whether the threshold grid T in split mode is computed on the full calibration set, on the node's own fold, or on the residual cell. The proof of Proposition 1 assumes T is either fixed a priori or a permutation-invariant function of the node's calibration fold F_ℓ, but Section 4.2 says the grid consists of 100 quantiles of the NLL scores in the calibration set. Please clarify the exact computation of T in split mode and reconcile it with the proof.","section":"Section 4.2 and Algorithm 1"},{"comment":"In the sentence \"since the disagreement set has probability at most ε_n and enters both the numerator and the conditioning event of R_v = E[ℓ|g_τ(x)=1,ρ_v(x)=1], the conditional risks differ by |R_deployed−R_aug| ≤ ε_n/π_v\", the variable π_v is defined only in the preceding sentence. Please ensure the notation is introduced before use, and consider defining the conditional disagreement probability explicitly.","section":"Section 3.5, Lemma 1 proof"},{"comment":"For Gemma, the exogenous-difficulty row reports a groupwise violation of 0.07, which is above δ=0.05, while the endogenous row reports 0.11. The text says the effect \"persists\" under exogenous difficulty, but the exogenous groupwise violation for Gemma is actually lower and still above the target; please state this explicitly so the reader can see the quantitative contrast.","section":"Section 5.3, Table 4"},{"comment":"The limitation paragraph on the in-sample guarantee says the worst-case slack \"is materially weaker than the headline numbers suggest\"; given that δ+|H|ε_n≈1.75 on the ARC pool, the worst-case guarantee is vacuous (the probability bound exceeds 1). The wording should say \"vacuous\" or \"has no meaningful worst-case content\" for that configuration, rather than only \"materially weaker\".","section":"Section 8.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and the split-variant guarantee appears sound, but the formal statement of Lemma 1 has an error that affects the only guarantee covering the deployed default procedure. The authors should correct the lemma and report the split variant at the paper's full 500-trial resolution. The equal-participation result in Section 8.2 is a significant qualification that should be surfaced earlier. I recommend major revision rather than rejection because the core framework is plausible and the empirical claims are carefully qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nThe part worth knowing: the paper gives a clean Bonferroni-corrected hierarchy construction for group-conditional selective prediction, with a genuinely unconditional formal guarantee for a split-calibration variant, and then honestly reports that the deployed in-sample variant does not inherit that guarantee without slack. The headline 0% violations on ARC are real at 500 bootstrap trials, but they are empirical properties of the in-sample procedure, not consequences of Proposition 1. The paper says as much in Section 8.3. The stress-test note is basically right: Lemma 1 adds ε_n to the simultaneous bound, and the worst-case ε_n ≈ 0.34 on the small ARC pool makes δ + |H|·ε_n vacuous there. What saves the paper is that it discloses this and provides a formally valid split variant; what is missing is split-variant results at the same 500-trial resolution, not merely 200 paired resamples reporting participation cost.\n\nWhat is new: HG-CRC is a plausible and clearly specified post-hoc calibration method — leaf-first routing, residual calibration, Bonferroni over nodes, min-node-size pruning. Proposition 1's split proof is coherent: deeper folds define residuals, shallower folds calibrate, exchangeability is restored. Proposition 2's augmentation is a neat bridge but is not deployed. The evaluation is unusually honest: the equal-participation comparison shows a coverage-matched global threshold does about as well on worst-group risk, which I read as deflating but credible; they also flag the endogeneity of NLL-derived difficulty bins and test with an exogenous score; WGER's dependence on abstention is acknowledged; and the 50-to-500 trial artifact is disclosed.\n\nSoft spots, in proportion. First, the formal guarantee for the deployed default is not established; the mean slack is small, but worst-case slack could matter for sparse nodes, as they concede. The fix is straightforward: run the split variant at full resolution and release code. Second, the equal-participation result removes much of the empirical case for the hierarchy; the remaining case is the automatic safe operating point plus the formal simultaneous guarantee. That is still useful, but narrower than the abstract's first reading. Third, code is promised only upon acceptance, so independent verification is currently impossible. The citation pattern looks solid; the related work is appropriate and the incremental claim is stated carefully.\n\nWho this is for: people deploying LLMs with abstention in high-stakes or fairness-sensitive settings, and conformal-prediction researchers. It deserves a serious referee; with split-variant results at 500 trials and released code, it could become a solid paper. My recommendation: engage with it, but require the missing experiment before accepting the headline guarantee as something more than empirical.","headline":"A clearly specified hierarchical group-conditional CRC with a sound split-variant guarantee, an honest but deflating empirical story, and a real gap between the headline zeros and the formally guaranteed procedure.","tokens_in":25126,"tokens_out":2906,"would_cite":true,"duration_ms":29376,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HG-CRC, a hierarchical group-conditional calibration method, achieves an empirical zero violation rate and worst-group excess risk 0 on ARC Challenge for Qwen3-4B and Llama-3.1-8B.","keywords":["selective prediction","conformal risk control","group-conditional guarantees","hierarchical calibration","Bonferroni correction","abstention","large language models","worst-group excess risk"],"falsifier":"Run the split variant of HG-CRC on ARC Challenge with exogenous human difficulty labels and 500 bootstrap trials, and count trials where any node's realized risk exceeds $\\alpha$; a violation rate meaningfully above $\\delta$ would refute the simultaneous claim. Alternatively, construct a synthetic score distribution with an atom at the selected grid threshold so that Assumption 1 fails, and check whether the deployed in-sample violation rate exceeds $\\delta+|\\mathcal{H}|\\varepsilon_n$.","tokens_in":1862,"feed_emoji":"🛡️","tokens_out":5996,"duration_ms":110692,"temperature":0.7,"pith_summary":"The paper argues that the standard marginal guarantee of conformal risk control is the wrong target for selective prediction in language models: a model can meet its overall error budget while a subgroup that becomes over-represented at test time absorbs most of the errors. HG-CRC fixes this by assigning one calibrated abstention threshold to every node of a user-defined group hierarchy, applying a Bonferroni correction so all nodes are certified simultaneously, and routing each example to the deepest certified node that will answer it. The method needs only a held-out calibration set and no retraining, and it is tested on three LLMs and two benchmarks across eight distribution-shift and robustness settings. The headline result is that on ARC Challenge, for models with sufficiently high base accuracy, HG-CRC attains an empirical violation rate of 0% and worst-group excess risk 0, at a participation cost of 22 to 37 percentage points relative to global CRC; the paper is explicit that these zeros are empirical upper bounds of about 0.6% at 500 bootstrap trials, and that the result does not carry over to MMLU-Pro.","feed_headline":"Zero violations on ARC: per-group thresholds with Bonferroni","feed_subtitle":"Marginal risk control fails under shift; this hierarchy certifies every group and abstains the rest.","key_machinery":"The load-bearing object is a rooted group hierarchy $\\mathcal{H}$ of nested membership masks, combined with a leaf-first selective predictor $v^*(x)=\\arg\\max_{v\\in\\Pi(x)}\\{\\mathrm{depth}(v): v \\text{ certified}, s(x)\\le \\tau_v\\}$, which sends each test point to the deepest certified ancestor whose threshold it satisfies. Each node's threshold is chosen from a Clopper-Pearson upper bound $U_v(\\tau)$ at per-node level $\\hat\\delta=\\delta/|\\mathcal{H}|$, and the key twist is residual calibration: a parent node is calibrated on exactly the examples that reach it after deeper certified nodes claim their own, via the residual map $\\rho_v(x)=m_v(x)\\prod_{c\\in\\mathrm{desc}(v)}(1-m_c(x)g_{\\tau_c}(x))$. This aligns each node's calibration population with its deployment population. The guarantees form a cascade: split calibration gives an unconditional simultaneous bound (Proposition 1), in-sample calibration with test-point augmentation is exact (Proposition 2), and the deployed plain in-sample procedure is valid up to a measured jackknife slack $\\varepsilon_n$ (Lemma 1) under a threshold-regularity assumption.","core_discovery":"The paper's central claim is that the marginal risk guarantee of conformal risk control is not enough for selective prediction in language models, because a population-level budget can be met while subgroup risk exceeds it under shifts in group composition. HG-CRC replaces the single threshold with one per node of a user-defined hierarchy, calibrates each node on its residual population after deeper certified nodes claim their own examples, and answers a test point if its deepest certified ancestor's threshold is satisfied. The formal machinery gives a simultaneous guarantee: with probability at least $1-\\delta$, every node's deployed risk is at most $\\alpha$ in the split variant, with the in-sample variant holding the same bound up to a measured slack $\\varepsilon_n$. On ARC Challenge the procedure reaches an empirical violation rate of $0$ and worst-group excess risk $0$ for Qwen3-4B and Llama-3.1-8B at 500 bootstrap trials, which the paper treats as an upper bound of about $0.6\\%$; on MMLU-Pro it either abstains entirely or retains a worst-group excess risk of $0.014$.","pith_inferences":["The equal-participation comparison suggests that an oracle global threshold matched to HG-CRC's coverage would protect per-group risk about as well in most settings, so the method's practical value lies in automatically finding a safe per-group operating point and in holding a simultaneous certificate, not in a large per-group safety margin.","A direct test of the endogenous-difficulty concern would use human grade-level difficulty labels from the original ARC release; the paper's own exogenous-model check suggests the mixture-shift failure would persist.","Because participation collapses for models whose accuracy is near or below $1-\\alpha$, HG-CRC can serve as a deployability diagnostic: near-zero participation under the hierarchy is evidence that no threshold can meet the budget for that model and task.","The measured jackknife slack $\\varepsilon_n$ could be reported alongside empirical violation rates in production, turning the in-sample guarantee into a transparent worst-case bound."],"forward_implications":["On ARC Challenge, HG-CRC achieves an empirical violation rate of 0 and WGER 0 for Qwen3-4B and Llama-3.1-8B across 500 bootstrap trials, which the paper reads as an upper bound of about 0.6% rather than a certified zero.","Under a mixture shift that over-represents hard examples, global CRC violates the risk budget in 47% of trials for Qwen and 32% for Llama, while groupwise thresholds hold it; HG-CRC keeps the simultaneous version of that protection.","Label noise makes the method abstain more rather than violate: at 10% label noise participation drops to 8.5% with a 0% violation rate.","The difficulty level of the hierarchy is what clears the budget on ARC: without it, violations return to the 11% level of global CRC.","Bonferroni correction is required for the formal simultaneous guarantee, and its empirical effect appears only when many nodes are tested, as on MMLU-Pro where removing it raises the violation rate from 0.6% to 14.0%."],"supporting_citations":[{"why":"Supplies the Clopper-Pearson-based risk control that HG-CRC extends node-by-node.","marker":"Angelopoulos et al. 2023"},{"why":"Defines the selective prediction framework of predictor plus selector that HG-CRC operates within.","marker":"Geifman & El-Yaniv, 2017"},{"why":"Provides ARC Challenge, the benchmark where the zero-violation results are obtained.","marker":"Clark et al., 2018"},{"why":"Provides MMLU-Pro, the second benchmark where the guarantee is not met or participation collapses.","marker":"Wang et al., 2024"},{"why":"Provides Llama-3.1-8B-Instruct, one of the two models with zero empirical violations.","marker":"Dubey et al., 2024"},{"why":"Provides Qwen3-4B, the other model with zero empirical violations.","marker":"Team, 2025b"},{"why":"Provides Gemma-3-4B, the poorly calibrated model that degrades by abstaining.","marker":"Team, 2025a"},{"why":"Supplies the group-conditional conformal background that flat groupwise CRC inherits.","marker":"Vovk et al., 2009"}],"fun_headline_variants":["Hierarchical risk control eliminates subgroup violations on ARC","Group-conditional calibration: zero violations, not just average risk","Marginal guarantees fail under shift; HG-CRC certifies each group","Per-node thresholds for LLM abstention: 0% violations on ARC","From population to group: hierarchical conformal risk control"],"cache_read_input_tokens":27264,"weakest_assumption_plain":"The deployed in-sample guarantee relies on the threshold-regularity assumption that each calibration curve crosses the risk budget cleanly at the chosen threshold, so a one-point change in calibration data never flips the selected threshold; on the small ARC pool the measured worst-case slack is about 0.34, and if that assumption fails, the simultaneous guarantee degrades.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical risk control eliminates subgroup violations on ARC","Group-conditional calibration: zero violations, not just average risk","Marginal guarantees fail under shift; HG-CRC certifies each group","Per-node thresholds for LLM abstention: 0% violations on ARC","From population to group: hierarchical conformal risk control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1527,"prompt_tokens":1158,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":774,"completion_tokens_details":{"reasoning_tokens":283}},"tokens_in":774,"tokens_out":369,"duration_ms":4101,"temperature":1.0,"reasoning_tokens":283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:26:12.977072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the split variant of HG-CRC on ARC Challenge with exogenous human difficulty labels and 500 bootstrap trials, and count trials where any node's realized risk exceeds $\\alpha$; a violation rate meaningfully above $\\delta$ would refute the simultaneous claim. Alternatively, construct a synthetic score distribution with an atom at the selected grid threshold so that Assumption 1 fails, and check whether the deployed in-sample violation rate exceeds $\\delta+|\\mathcal{H}|\\varepsilon_n$.","supporting_citations":[],"review_version":2}