{"id":"5c332314-4023-4325-aa24-d879de43b001","arxiv_id":"2606.29054","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Derives a closed-form impossibility bound and feasibility test for conformal risk control on structured LLM outputs, with empirical comparison of bounds and adaptive inference across models and tasks.","lead":"The paper proves an impossibility result showing that distribution-free conformal risk control must abstain on at least ((μ-α)/(1-α)) fraction of examples when base risk exceeds the target, and provides a feasibility test plus empirical validation of adaptive methods under shift. A smart generalist should read it to understand the hard limits on certifying LLM structured outputs before deployment.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest_assumption concerns the validity conditions for the concentration inequalities used in the certification hierarchy. The strongest_claim, however, is the impossibility result, which is logically independent of those inequalities and rests only on the definition of conditional risk. Because that derivation is elementary and correct, the central claim has no identified load-bearing vulnerability.","tokens_in":1864,"tokens_out":314,"duration_ms":41602,"concrete_test":"Locate the section deriving the impossibility bound; confirm it uses the conditional-risk definition P(error | output) ≤ α and reproduces the inequality chain above. If the derivation matches, the bound is tight and the feasibility test is sound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The impossibility result follows directly from the definition of conditional risk control. Let a be the abstention fraction and μ the base error probability. Then P(error and output) ≥ μ − a. For conditional risk P(error | output) ≤ α we require P(error and output) ≤ α(1 − a). Combining yields a ≥ (μ − α)/(1 − α). This is a deterministic, distribution-free lower bound that holds for any method achieving the target; it requires neither concentration inequalities nor exchangeability of nonconformity scores. The feasibility test is therefore valid on its own terms. The concentration assumptions identified by the reader apply only to the upper-bound certification results (Hoeffding, Bernstein, e-CRC), not to this necessary-condition argument.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proves an impossibility result for any distribution-free method (including CRC) certifying structured LLM outputs: when base risk μ exceeds target α, abstention must be at least ((μ-α)/(1-α)), yielding a closed-form pre-check for feasibility. It ranks certification bounds (Hoeffding, empirical Bernstein, betting-based e-CRC) with quantified gains (+37% certified configurations from Hoeffding to Bernstein; e-CRC useful at small calibration sizes), validates ACI under cross-dataset shift (violations drop from 71% to 21%), and reports experiments across six models (3B-72B), eight datasets, four tasks, and six nonconformity scores, showing practical certification at α=0.30-0.40 for NER/QA/CLS while hard settings at α=0.10 remain uncertifiable. A three-step deployment recipe is provided.","tokens_in":2012,"tokens_out":478,"duration_ms":23032,"significance":"If the impossibility bound and bound hierarchy hold, the work supplies a concrete, distribution-free feasibility test and practical guidance on bound selection and shift adaptation for LLM reliability, directly addressing the 7.5-12.5% shortfall of heuristics. The closed-form abstention lower bound, explicit comparison of concentration inequalities with regime-specific gains, and multi-model/multi-task empirical coverage are strengths that would make the contribution substantial for deployment of certified structured generation.","major_comments":[],"minor_comments":[{"comment":"§3 (nonconformity scores): explicit formulas or pseudocode for the six scores would improve reproducibility, as the current description leaves the exact definitions of some scores implicit.","section":"§3"},{"comment":"Table 2 or equivalent results table: the +37% gain from Hoeffding to Bernstein should be accompanied by the precise number of configurations, datasets, and α values underlying the percentage to allow direct verification.","section":"Table 2"},{"comment":"Figure 4 (ACI results): axis labels and legend should explicitly state the value of α and the shift type (cross-dataset) to avoid ambiguity when interpreting the 71% to 21% reduction.","section":"Figure 4"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the positive assessment, accurate summary of our contributions, and recommendation for minor revision. The report correctly identifies the impossibility bound, certification hierarchy, ACI validation under shift, and multi-model empirical results as core strengths.","responses":[],"tokens_in":1431,"tokens_out":65,"duration_ms":16622,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's central result is a simple lower bound: if the base error rate mu sits above the target alpha, any distribution-free method must abstain on at least ((mu - alpha)/(1 - alpha)) of the cases to have any chance of controlling conditional risk. This yields an immediate pre-check before running conformal risk control on structured LLM outputs.\n\nThey then compare three certification bounds (Hoeffding, empirical Bernstein, betting-based e-CRC) and show Bernstein delivers the largest practical gain, certifying 37% more configurations in their runs. e-CRC helps mainly with small calibration sets. Adaptive conformal inference under cross-dataset shift drops risk violations from 71% to 21%, with the remaining failures lining up exactly where the bound predicts they must occur.\n\nThe experiments cover six models, eight datasets, four tasks, and six nonconformity scores, which is reasonable coverage for this kind of work. The impossibility argument itself is deterministic and does not lean on the concentration inequalities used for the upper bounds.\n\nThe softer part is the empirical section, which reports gains from counting certified configurations after the fact; it would be clearer to see more detail on score selection and whether the observed abstention rates match the predicted minimum. The shift mitigation is promising but still leaves some residual failures.\n\nThis is aimed at practitioners who need risk guarantees on LLM structured generation and want a concrete way to decide when it is even feasible. The theoretical piece is clean enough and the practical recipe follows directly, so it deserves a serious referee.","headline":"The impossibility bound giving a closed-form feasibility test is the main new piece and holds up on its own terms.","tokens_in":2475,"tokens_out":375,"would_cite":true,"duration_ms":23858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"When base risk exceeds the target, any distribution-free method must abstain on at least ((μ-α)/(1-α)) examples.","keywords":["conformal risk control","LLM structured generation","impossibility bounds","abstention","adaptive conformal inference","distribution-free certification","NER QA classification"],"falsifier":"A single run in which base risk μ > α yet the method abstains on strictly fewer than ((μ - α)/(1 - α)) examples while still keeping empirical risk at or below α would falsify the impossibility claim.","tokens_in":2748,"feed_emoji":"","tokens_out":716,"duration_ms":22525,"temperature":0.7,"pith_summary":"The paper proves that conformal risk control for structured LLM outputs faces a hard limit: if the model's inherent error rate μ sits above the allowed level α, certification requires abstaining on a calculable minimum share of cases. This yields an immediate pre-check that tells whether CRC can succeed on a given task and dataset. The work then ranks three concentration bounds by how many configurations they certify, shows that adaptive conformal inference cuts violations under shift, and maps which NER, QA, and classification setups become feasible only after relaxing α. Experiments on six models and eight datasets confirm the bound's predictions in practice.","feed_headline":"Base risk above target forces CRC to abstain on fixed fraction of cases","feed_subtitle":"A simple pre-check predicts whether distribution-free certification can meet the risk target for LLM outputs.","key_machinery":"The impossibility abstention bound ((μ - α)/(1 - α)), which acts as a distribution-free feasibility test for whether conformal risk control can certify outputs at the chosen risk level.","core_discovery":"When the base risk μ exceeds the target α, any distribution-free certification method must abstain on at least ((μ - α)/(1 - α)) fraction of examples. This bound supplies a closed-form feasibility test that can be evaluated before running CRC. Tighter bounds (empirical Bernstein over Hoeffding, then e-CRC) expand the set of certifiable tasks, with the largest gain coming from the Bernstein step; adaptive conformal inference further reduces target violations under cross-dataset shift.","pith_inferences":["Tasks whose base risk lies far above α may remain impractical to certify until the underlying model improves.","The feasibility check could be inserted as a gate before any production deployment of CRC for LLMs.","If nonconformity scores violate exchangeability in deployment, the bound and all certification guarantees lose their justification."],"forward_implications":["The closed-form test lets practitioners reject CRC on hard configurations without running the procedure.","Switching from Hoeffding to empirical Bernstein increases the fraction of certifiable configurations by 37 percent.","e-CRC enables certification with only 20 percent calibration data where Hoeffding yields none.","Adaptive conformal inference lowers risk-target violations from 71 percent to 21 percent under dataset shift.","Raising the target from 0.10 to 0.30-0.40 makes 47 percent of NER, 40 percent of QA, and 60 percent of classification cases certifiable."],"fun_headline_variants":["Base risk excess dictates fixed CRC abstention fraction","Pre-check reveals when CRC certifies structured LLM outputs","Impossibility result sets minimum abstention for distribution-free CRC","Bernstein bound expands certifiable tasks over Hoeffding","Adaptive inference halves violations from distribution shift"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The nonconformity scores must satisfy the conditions required for the chosen concentration inequalities to apply, including the needed exchangeability or independence structure.","fun_headline_variants_meta":{"raw":{"variants":["Base risk excess dictates fixed CRC abstention fraction","Pre-check reveals when CRC certifies structured LLM outputs","Impossibility result sets minimum abstention for distribution-free CRC","Bernstein bound expands certifiable tasks over Hoeffding","Adaptive inference halves violations from distribution shift"]},"model":"grok-4.3","cost_usd":0.003984,"raw_usage":{"total_tokens":2099,"prompt_tokens":796,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":39837000,"prompt_tokens_details":{"text_tokens":796,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1230,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":796,"tokens_out":73,"duration_ms":10380,"temperature":1.0,"reasoning_tokens":1230,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T09:11:49.948145+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A single run in which base risk μ > α yet the method abstains on strictly fewer than ((μ - α)/(1 - α)) examples while still keeping empirical risk at or below α would falsify the impossibility claim.","supporting_citations":[],"review_version":1}