{"id":"134ca103-9538-4635-9922-9cabc8b9fd57","arxiv_id":"2504.17671","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Split conformal prediction on multiple-choice visual question answering produces answer sets whose empirical miscoverage stays below a user-set alpha across eight vision-language models and two benchmarks.","lead":"This paper wraps vision-language models in a standard statistical wrapper, split conformal prediction, to output a set of answer options instead of a single guess. The claimed payoff is a user-set error ceiling that holds across eight models and two question-answering benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection bias from discarding unsupported samples invalidates the claimed marginal coverage: true miscoverage is at least the discarded fraction, so Eq. (5) is unverified unless that fraction is reported.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue: discarding samples without the correct answer breaks exchangeability and invalidates the marginal guarantee over the original test set. My analysis strengthens this by observing an algebraic lower bound: discarded samples are guaranteed miscoverage because the correct answer cannot appear in any prediction set derived from the generated candidates. Therefore, unless the discard rate D is reported and shown to be small enough, the empirical results cannot establish Eq. (5) over the benchmark distribution. Since the underlying SCP theory is standard and the manuscript may be salvageable with a transparent report of discard rates and full-population error rates, I would keep the reader's CONDITIONAL verdict, hence no change is needed.","tokens_in":9112,"tokens_out":6566,"duration_ms":69564,"concrete_test":"For each LVLM, benchmark, alpha, and split ratio, compute D = (# test samples whose 36 generated responses do not contain the correct answer) / (# original test samples). Then compute the original-test error rate by treating every discarded test sample as a miscoverage event and adding it to the retained-sample error rate. If this combined rate exceeds alpha anywhere, Eq. (5) fails on the original distribution. Additionally, re-run calibration and evaluation without discarding any samples and report both retained-only and full-population error rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, Eq. (5), is a marginal coverage guarantee over the original test distribution. Section 4.1 (Implementation Details) states: 'Samples are discarded if no correct answer exists in the generated set.' Let D be the fraction of original test samples discarded. For any discarded (x, y*), y* is absent from the generated candidate set; since the nonconformity score S(x, y*) is derived from candidate frequencies, y* cannot be included in C_alpha(x). Thus the true test miscoverage rate on the original distribution is at least D plus the miscoverage rate among retained samples. If D > alpha for any tested alpha, Eq. (5) is algebraically violated. Even when D <= alpha, the reported error rates in Table 1 and Figure 1 are conditional on the filtered, easier subset, not on the benchmark distribution claimed in the abstract. The paper never reports D, so the central empirical claim is not supported as stated. This is not a flaw in SCP theory itself; it is a mismatch between the implemented pipeline and the guarantee asserted for the original benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript applies split conformal prediction (SCP) to closed-form visual question answering with large vision-language models. For each question, the LVLM is sampled multiple times without a prompt; the sampled answers are semantically clustered and converted to option-only sets, and the cluster-frequency distribution defines a nonconformity score S(x,y)=1-f(y|x). After calibrating the (1-alpha) quantile of the scores on a calibration split, prediction sets are formed by thresholding and are evaluated on ScienceQA and MMMU with eight LVLMs. The paper claims that empirical miscoverage is controlled by Eq. (5) for all tested alpha values and split ratios, and that prediction set sizes decrease as alpha increases. A central implementation detail is that samples are discarded when no generated answer matches the ground truth, and the discard rate is never reported.","tokens_in":9301,"tokens_out":7414,"duration_ms":75160,"significance":"The mathematical core is a standard split-conformal quantile argument, and the paper correctly recognizes that no fitted parameters enter the coverage calculation; the theorem itself is not in dispute. If the empirical claims were supported, the paper would be a useful demonstration that a sampling-frequency score can be wrapped in SCP to provide finite-sample coverage on widely used LVLM benchmarks, together with a robustness check across split ratios. However, the empirical demonstration as reported is not sufficient to establish the stated benchmark-level guarantee, because the pipeline filters the data after observing ground truth and evaluates error rates on a selected, potentially easier subset. The central claim in Eq. (5) is therefore not verified for the original test distribution, and the paper's second claim about prediction-set dynamics is additionally undermined by a section of text that reverses the meaning of alpha.","major_comments":[{"comment":"The discard step invalidates the stated marginal coverage claim. The implementation states: 'Samples are discarded if no correct answer exists in the generated set,' but the paper never reports how many samples are discarded. Let D be the discard fraction on the test portion. A discarded test sample has its true answer absent from the candidate set over which scores are defined, so the true answer cannot enter the prediction set produced by Eq. (2) under the calibration distribution used; the empirical miscoverage on the original test distribution is therefore at least D plus the miscoverage on the retained subset. If D exceeds alpha for any tested alpha, Eq. (5) is algebraically violated; even when D is below alpha, the rates in Table 1 and Figure 1 are conditional on a filtered, easier subset and do not establish the benchmark-level guarantee claimed in the abstract and Section 4.2. The authors must report D for each benchmark, model, and alpha value, or restate the guarantee as applying only to the retained population, or modify the pipeline so that the coverage statement covers the original test distribution.","section":"4.1, Eq. (5)"},{"comment":"The quantile definition does not match the standard split-conformal construction. The text selects the '⌈(1−α)(n+1)/n⌉-th order statistic' after arranging the calibration scores in descending order; the usual finite-sample guarantee uses an ascending empirical quantile with index ⌈(1−α)(n+1)⌉ (or the equivalent n/(n+1) correction), and the upper bound in Eq. (4) depends on that index. As written, the procedure is underspecified and appears to select a different order statistic than the one the theorem requires. Please state the sorting direction, give the exact order-statistic index, and confirm that the implementation and the theorem use the same definition; otherwise the finite-sample bounds in Eqs. (3)-(5) are not connected to the described algorithm.","section":"3.2, step 2, and Eq. (4)"},{"comment":"The discussion of alpha in the first paragraph of Section 4.3 is inverted. The text says 'Lower α values correspond to higher permissible error rates' and 'higher α values impose stricter error control,' while Section 3.2 correctly states the opposite and the figures show that higher alpha produces smaller prediction sets. This passage misstates the paper's second central claim and should be corrected, because as written it would lead a reader to the wrong conclusion about the direction of the trade-off between alpha and prediction set size.","section":"4.3, first paragraph"}],"minor_comments":[{"comment":"The title on the first page contains typographical corruption ('DATA-D RIVEN', 'VISION -L ANGUAGE') and should be cleaned up.","section":"Title page"},{"comment":"Section 4.3 refers to 'Section 3.2.2,' but Section 3.2 has no numbered subsections; the cross-reference should be updated.","section":"4.3, 'Section 3.2.2'"},{"comment":"The notation for the number of calibration samples uses n and N interchangeably across Sections 3.1 and 3.2; one symbol should be used consistently.","section":"3.1 and 4.1"},{"comment":"The 'bidirectional discrimination' and 'semantic clustering' steps performed by Qwen2.5-3B-Instruct are not described; without these details the score construction is not reproducible, especially regarding how aliases are merged into canonical option labels.","section":"4.1, Implementation Details"},{"comment":"The description 'averaging results over 100 rounds' is ambiguous: it should state whether the calibration/test split is redrawn each round and whether the reported error rates are averaged over rounds or pooled across rounds.","section":"4.1, 'averaging results over 100 rounds'"},{"comment":"The caption refers to 'the α values' when the table cells are measured empirical error rates at the fixed level α=0.2; the caption should explicitly say that the entries are empirical miscoverage rates.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the unreported sample filtering in Section 4.1, which separates the theory from the empirical claim. This is fixable within the manuscript's scope by reporting discard rates and either narrowing the claim to the retained population or adding an abstention-aware correction. I also recommend that the editor ask for a careful consistency check between the text and the figures; the inverted-alpha passage in Section 4.3 suggests the prose has not been fully checked against the experimental results. The contribution is otherwise a straightforward application of existing conformal methodology, so the novelty rests on the quality and honesty of the empirical demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical idea here is reasonable, but the headline empirical claim is not supported as written. The paper applies standard split conformal prediction to multiple-choice VQA, using a nonconformity score built from the frequency of semantically clustered answers across repeated samples. That is a sensible way to get a distribution over options from free-form LVLM output, and the theory section is a correct restatement of Vovk et al. The breadth of the evaluation—eight LVLMs across two benchmarks—is decent, and the authors cite the relevant conformal literature, including ConU and Sample-then-Identify, so the application context is not misrepresented.\n\nThe problem is in Section 4.1: \"Samples are discarded if no correct answer exists in the generated set.\" The stress-test note is right. Any discarded sample is by construction a miscoverage case—the true answer is not in the candidate set, so it cannot be in the prediction set. The true test miscoverage on the original distribution is therefore at least the discard fraction D, and the paper never reports D. Reported error rates in Table 1 and Figure 1 are conditional on the filtered, easier subset. Equation (5) is asserted for the original benchmark distribution, but the experiments only validate it on whatever subset survives the filter. This is not a subtle statistical nuance; it is a direct mismatch between the implemented pipeline and the claimed guarantee. The authors could fix this by reporting D, treating discarded samples as errors, or making an explicit conditional-coverage claim. None of that is present.\n\nThe other weaknesses are smaller. The pipeline is under-specified: no code, no details on the entailment LLM or clustering, and the quantile formula in Section 3.2 has an unusual finite-sample correction that is not clearly the standard one. The novelty is genuinely incremental—this is a known method with a locally engineered score, and the authors already cite work that is very close (ConU, Sample-then-Identify). There are also cosmetic issues like the title spacing and the odd label \"Romano upper bound.\"\n\nWho is this for? Practitioners wanting a ready-made uncertainty wrapper for VQA might find the recipe useful, but only after the discard-rate problem is addressed. As a methodological contribution, it breaks no new ground. I would bring it to a reading group as a cautionary example of conformal prediction gone sloppy, but I would not cite it in its current form.\n\nRecommendation: send it to peer review only if the authors can be pushed to report discard rates and rerun the analysis accordingly. The flaw is fixable, and the application is of practical interest, so a serious referee could help turn this into a modest but valid paper. If the discard rates turn out high, the paper's empirical basis collapses.","headline":"The paper's central empirical claim is unverified because the pipeline discards every sample whose true answer never appears in the generated candidate set, and that discard rate is never reported.","tokens_in":9844,"tokens_out":3178,"would_cite":false,"duration_ms":32075,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A calibration wrapper keeps the error rate of vision-language answer sets under a preset cap across eight models.","keywords":["conformal prediction","large vision-language models","visual question answering","hallucination mitigation","uncertainty quantification","marginal coverage","prediction sets","split conformal prediction"],"falsifier":"Rerun the pipeline on a fixed test set but count every discarded question (one where no sampled answer contains the correct option) as an automatic error; if the empirical error rate on the full original set exceeds $\\alpha$ whenever the discard rate is non-negligible, the guarantee only holds for the filtered subset, not the original distribution.","tokens_in":8904,"feed_emoji":"🎯","tokens_out":13876,"duration_ms":112039,"temperature":0.7,"pith_summary":"The paper aims to establish that a wrapper method called Split Conformal Prediction can take the raw answers of a large vision-language model on multiple-choice visual question answering and turn them into a prediction set with a finite-sample guarantee: for any user-chosen error rate $\\alpha$, the chance that the true answer is missing from the set is at most $\\alpha$, assuming the calibration and test data are exchangeable. This matters because vision-language models can be confidently wrong, and a plug-in layer that controls the error rate without retraining would give safety-critical applications a formal basis for deciding when to trust a model's answer. The paper reports experiments on ScienceQA and MMMU with eight LVLMs, where the empirical error rate stays below $\\alpha$ for every tested value and the average prediction set shrinks as $\\alpha$ grows. The practical payoff claimed is a hallucination filter: raise $\\alpha$ when risk is acceptable, and the model returns smaller, cleaner answer sets.","feed_headline":"Eight vision-language models stay under a set error cap","feed_subtitle":"Calibrated prediction sets shrink as allowed error grows, filtering low-confidence answers without retraining.","key_machinery":"The load-bearing object is the nonconformity score $S(x,y) = 1 - \\hat{f}(y\\mid x)$, where $\\hat{f}(y\\mid x)$ is the relative frequency of answer option $y$ after semantic clustering of many sampled generations. The method orders the calibration scores, takes the $\\lceil (1-\\alpha)(n+1)/n \\rceil$-th order statistic as the threshold $\\tau$, and returns every option whose score is at most $\\tau$. This one-threshold rule is what converts a heuristic confidence estimate into a set with a distribution-free coverage guarantee; because the threshold rises with $\\alpha$, the returned set necessarily shrinks as the allowed error rate increases.","core_discovery":"The central claim is that the answer set $\\hat{C}_\\alpha(x_{\\mathrm{test}}) = \\{y : S(x_{\\mathrm{test}}, y) \\le \\tau\\}$, built from the nonconformity score $S(x,y) = 1 - \\hat{f}(y\\mid x)$ and the calibration quantile $\\tau = Q_{1-\\alpha}(\\{S_i\\})$, satisfies $P[Y_{\\mathrm{test}} \\notin \\hat{C}_\\alpha(X_{\\mathrm{test}})] \\le \\alpha$ when calibration and test samples are exchangeable. The paper implements this for multiple-choice MMMU and ScienceQA questions by sampling 36 answers per question, clustering semantically equivalent answers into options, and using a small consistency-checking model to produce the score distribution. Across eight LVLMs from four model families, the empirical error rate is reported to stay strictly below $\\alpha$ at every tested risk level, including $\\alpha \\ge 0.6$, while prediction set size decreases as $\\alpha$ increases. The split-ratio experiments add the claim that this coverage holds for calibration fractions from 0.1 to 0.9, not just at a fixed 1:1 split.","pith_inferences":["An implication the authors leave implicit is that the guarantee is marginal over the whole test distribution; per-question or per-subgroup coverage can fail even when marginal coverage holds, so users needing guarantees on rare or high-risk subgroups would require a conditional version.","The inverse relation between $\\alpha$ and prediction set size is a near-mechanical consequence of quantile thresholding; the empirically interesting finding is that the score distributions of these models are regular enough for the relation to appear, with at least one anomaly being InternVL2-1B at $\\alpha = 0.1$.","A testable extension is to apply the same wrapper to open-ended generation using a semantic-entailment scorer as the conformity measure; the open question is whether the returned sets stay small enough to be useful, since the paper only treats multiple-choice options."],"forward_implications":["For the eight tested LVLMs on ScienceQA and MMMU, a user can set a target error rate $\\alpha$ and obtain answer sets whose empirical error stayed below $\\alpha$ in every reported configuration.","The method works on a model as a black box: no retraining, no internal logits, and no distributional assumptions beyond exchangeability, so it can be added to models that only expose sampled text.","Setting a larger $\\alpha$ yields smaller prediction sets, giving a direct lever for filtering low-confidence or confusable options and reducing hallucination-related noise.","The split-ratio results imply that even a small calibration set (10 percent of data) can enforce the marginal coverage claim in the reported settings."],"supporting_citations":[{"why":"Supplies the exchangeability-based coverage theorem that the paper's guarantee rests on.","marker":"Vovk et al. (2005)"},{"why":"Provides the conformal prediction formulation and the nonconformity-score quantile construction the method adapts.","marker":"Romano et al. [2019]"},{"why":"Shows conformal prediction sets improve human decision making, motivating the VQA application.","marker":"Cresswell et al. [2024]"},{"why":"Extends correctness-coverage guarantees to multiple-choice QA, the closest prior setup to this paper.","marker":"Ke [2025]"},{"why":"Surveys conformal prediction for NLP and supports applying SCP to language-generation outputs.","marker":"Campos et al. [2024]"},{"why":"Supplies the statement of split conformal validity and quantile definitions the method uses.","marker":"Angelopoulos et al. [2023]"},{"why":"Applies conformal uncertainty to LLMs with correctness-coverage guarantees, a direct basis for the pipeline.","marker":"Wang et al. [2024]"},{"why":"Extends risk control to multimodal LLMs, supporting the paper's multimodal applicability claim.","marker":"Wang et al. [2025b]"},{"why":"Develops selective conformal uncertainty in LLMs, providing a comparator for coverage behavior.","marker":"Wang et al. [2025c]"}],"fun_headline_variants":["Calibrated answer sets hold error below a user-set cap","Prediction sets shrink as allowed error grows, no retraining","Eight vision-language models pass error cap test","Conformal calibration keeps VQA errors under alpha","Data-driven calibration for trustworthy VQA answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that, after the pipeline discards questions where none of the sampled answers contains the correct option, the retained calibration and test questions are exchangeable with the original test distribution; if hard questions are discarded at a higher rate, the stated coverage over the original test set does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated answer sets hold error below a user-set cap","Prediction sets shrink as allowed error grows, no retraining","Eight vision-language models pass error cap test","Conformal calibration keeps VQA errors under alpha","Data-driven calibration for trustworthy VQA answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1662,"prompt_tokens":1050,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":537}},"tokens_in":666,"tokens_out":612,"duration_ms":5840,"temperature":1.0,"reasoning_tokens":537,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:33:37.624661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the pipeline on a fixed test set but count every discarded question (one where no sampled answer contains the correct option) as an automatic error; if the empirical error rate on the full original set exceeds $\\alpha$ whenever the discard rate is non-negligible, the guarantee only holds for the filtered subset, not the original distribution.","supporting_citations":[{"cited_title":"Conu: Conformal uncertainty in large language models with correctness coverage guarantees","cited_arxiv_id":null,"evidence_quote":"Applies conformal uncertainty to LLMs with correctness-coverage guarantees, a direct basis for the pipeline."}],"review_version":1}