{"id":"c555e475-916d-4e35-99b2-52374b49bc05","arxiv_id":"2608.08709","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper introduces Verification-Cost Errors and a verification-aware benchmarking protocol, arguing that reliability depends on whether errors can be detected within a deployment budget.","lead":"This paper argues that AI evaluation should measure how hard it is to verify a model's output, not just whether the output is correct. It defines Verification-Cost Errors and proposes a six-step benchmarking protocol that records human verification effort against a deployment budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VCE classification inherits the completeness of the ex ante acceptance relation A: under an incomplete A (e.g., a test suite that misses edge cases), the paper's motivating failure mode is counted as correct and never reaches the VCE denominator.","rationale":"The paper's core argument is careful and largely sound: the separation of latent verification cost from observed success probability and budget consumption is honest, and the protocol addresses verifier dependence, premature closures, and statistical thinness. The reader's CONDITIONAL verdict is appropriate. The load-bearing soft spot is not the abstract availability of ground truth, which benchmark evaluation routinely assumes, but the completeness and independence of the acceptance relation A on which the VCE definition rests. The paper's own Section 2 example shows why this matters: if A is the provided test suite, an output that passes all tests but fails on edge cases is, by definition, not incorrect under A and can never be a VCE, even though it is precisely the kind of output the paper says is dangerous. Step 3 requires A to be independent of the verification episode but offers no way to verify or bound A's completeness. This does not refute the central claim that correctness alone is insufficient; it does mean the headline construct is conditional on an adequately complete A, and the paper should either prescribe A-validation or report VCE rates as A-relative with sensitivity analysis. Because the paper already presents the constructs as conceptual instruments and leaves instantiation to future work, this concern supports the existing CONDITIONAL verdict rather than moving it.","tokens_in":17352,"tokens_out":14313,"duration_ms":177104,"concrete_test":"Take a code-generation dataset whose reference test suite plays the role of A, and construct a held-out set of edge-case tests (empty inputs, duplicates, extreme values) defining an enriched acceptance relation A'. Run the Section 6 protocol on the same model outputs and the same verifier panel under both A and A'. For outputs that pass A but fail A', compute how many are classified as correct versus confirmed VCEs under each relation. If any such output is hard to verify and is counted as correct under A but becomes a VCE under A', the confirmed VCE rate changes with the completeness of A; that would demonstrate that the framework must either require A-validation or report sensitivity of VCE rates to A before the headline measure is interpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 defines a Verification-Cost Error only for pairs (x, yhat) with (x, yhat) not in A, with A fixed a priori (Step 1) and required to be independent of the verification episode (Step 3). The motivating example of Section 2 is a sorting function that passes the provided tests but fails on empty inputs and duplicates. If the declared A is the provided test suite, that pair is in A and therefore cannot be a VCE no matter how many verifiers fail to expose the edge-case bug. The framework therefore silently inherits the completeness of A: a weak reference layer makes the very failure mode motivating the paper invisible to its headline construct. Step 1 says GT ensures comparability, not completeness, and Step 3 gives no procedure for certifying or bounding A's completeness (e.g., hidden tests, formal specifications, or adversarial review). This is distinct from the mere absence of ground truth for open-ended tasks: even with ex ante GT, an incomplete A breaks the mapping from verification difficulty to VCE classification, so reported VCE rates are not a stable indicator of deployment risk unless A's adequacy is validated and reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that AI evaluation should measure verification cost in addition to output correctness, formalizing this through the concept of Verification-Cost Errors (VCEs): incorrect input-output pairs that a declared fraction of the verifier population fails to identify within a deployment-specific budget. The paper motivates the need with examples from code generation, document understanding, and RAG systems, reviews why existing frameworks (hallucination, calibration, benchmarking, explanations, tool augmentation) omit the cost dimension, and proposes a six-step verification-aware benchmarking protocol with explicit treatment of censoring, premature closures, verifier control, and stratified reporting. The authors position the contribution as conceptual rather than empirical, explicitly leaving concrete instantiations to future work.","tokens_in":17578,"tokens_out":10522,"duration_ms":107660,"significance":"If the formalization succeeds, the paper provides a principled, deployment-relative way to quantify the frequently discussed but ill-defined 'verification burden' of AI outputs. The definitions are operational, do not depend on fitted parameters, and the protocol attends carefully to censoring, premature closures, uncertainty intervals, and verifier heterogeneity—issues that are easy to overlook but essential for a measurable construct. The proposed VCE rate and observed-budget-consumption indicators are falsifiable in principle, and the paper's explicit separation of latent verification cost from observed indicators is a methodological strength. However, the central construct depends on an acceptance relation A whose completeness is not addressed, which limits the applicability of VCE rates as deployment-risk indicators and is a load-bearing gap in the current formalization.","major_comments":[{"comment":"The VCE construct inherits the completeness of the acceptance relation A. In the motivating example of Section 2, a sorting function passes the provided test suite but fails on empty inputs and duplicates. If A is instantiated as that test suite (which the Section 6 protocol permits, since Step 1 only fixes GT for comparability and Step 3 only requires independence from the verification episode), then the pair (x, ŷ) belongs to A, so it is not in IncorrectPairs and cannot be a VCE no matter how many verifiers fail to expose the bug. The paper provides no procedure for certifying or bounding A's completeness (e.g., hidden tests, formal specifications, or adversarial review). Consequently, reported VCE rates are not a stable indicator of deployment risk unless A's adequacy is validated and reported. Please either require a complete acceptance specification and explain how to construct one for open-ended tasks, or introduce a completeness-aware reporting scheme that makes the dependence of VCE rates on A explicit.","section":"Section 5 (VCE definition) and Section 6 (Steps 1 and 3)"}],"minor_comments":[{"comment":"The phrase 'Multi-modal documentsunderstanding' should read 'Multi-modal document understanding' (missing space and incorrect noun form).","section":"Section 2, paragraph on multi-modal documents"},{"comment":"The Stopping Rule text contains a typographical artifact: 'self-reported confidence≥τ 1 (pre-registered, typically0 .9)' should read 'self-reported confidence ≥ τ (pre-registered, typically 0.9)'.","section":"Section 6, Step 3"},{"comment":"The notation 'ŷ = g(x)' is slightly ambiguous because g is described as a realization rather than a deterministic map; clarify that this means 'ŷ is an output produced by the system for input x under the fixed configuration'.","section":"Section 5, definition of IncorrectPairs"},{"comment":"Spelling is inconsistent between British forms ('hypothesised', 'behaviour') and American forms ('operationalize'); standardize to one convention.","section":"Throughout"},{"comment":"Krippendorff's α is mentioned without a reference; adding a citation would help readers who are not familiar with the measure.","section":"Section 6, Step 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position piece with no original measurements; this is transparently stated, and the contribution is conceptual. The heavy reliance on self-cited work (Immediato 2025, 2026) for the JORABP taxonomy, Cognitive Verification, and governance integration may raise novelty concerns, but the formal VCE definition itself appears independent of those works. The main technical issue is the completeness of the acceptance relation, which is fixable but requires substantive additions to the protocol as described in the major comment. If the authors address that issue, the paper could be a valuable contribution to the discussion of AI evaluation methodology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time: it takes a real gap in AI evaluation and gives it a precise, measurement-oriented vocabulary. The central claim—correctness alone doesn’t capture reliability; verification cost within a declared budget should be reported alongside accuracy—is defensible, and the formalization is more careful than most position pieces. The VCE definition is operational, the handling of censoring and premature closures is genuinely thoughtful, and the three-state classification with an inconclusive rate and envelope is honest about thin verifier coverage. The six-step protocol is detailed enough to run, and the paper explicitly labels its novelty as narrow. That deserves credit.\n\nThe main soft spot is exactly the one in the stress-test note. VCE is defined only for pairs outside the acceptance relation A, but A is fixed a priori and no procedure is given to certify or bound its completeness. The motivating example—a sorting function that passes the supplied tests but fails on empty inputs and duplicates—is not a VCE if A is the test suite, because the pair is in A. So the framework inherits whatever blind spots A has. This is distinct from the separate problem of missing ground truth for open-ended tasks; even with perfect ex ante GT, an incomplete A breaks the mapping from verification difficulty to VCE classification. The authors would need to require a completeness certificate or report A’s coverage as part of the protocol.\n\nTwo smaller concerns. First, no original instantiation: the protocol is not run, and the “evidence” in the abstract is cited from prior work. The body is transparent about this (the figures are hallucination rates, not VCE rates), but the abstract slightly overclaims. Second, the conceptual foundations lean heavily on self-cited work (Immediato 2025/2026). This is not circular and is flagged, but a referee should check the load.\n\nWho is this for? Anyone working on evaluation methodology, benchmark design, or human-AI oversight in high-stakes deployment. It won’t settle the empirical question, but it gives a shared language for asking it. I’d send it to peer review; a good referee can push the authors on A-completeness and ask for at least a small pilot. The paper is honest enough to benefit from that.","headline":"A careful, honest position piece that formalizes verification cost as a measurable evaluation dimension; the main gap is that the framework silently inherits whatever completeness the chosen acceptance relation has.","tokens_in":18129,"tokens_out":3131,"would_cite":true,"duration_ms":33643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Correctness alone cannot measure AI reliability; verification cost is the missing metric.","keywords":["verification cost","Verification-Cost Errors","AI evaluation","correctness","benchmarking","human oversight","hallucination","reliability"],"falsifier":"Run the proposed protocol on two models with identical accuracy on a fixed dataset: if the probability of a correct verdict within budget and the observed budget consumption are statistically indistinguishable across models, the added dimension adds no predictive power beyond accuracy. At the mechanism level, if verifiers detect errors in fluent, authority-styled outputs at the same rate and within the same budget as in plain outputs, the self-masking hypothesis fails.","tokens_in":17124,"feed_emoji":"⏱️","tokens_out":8233,"duration_ms":66749,"temperature":0.7,"pith_summary":"This paper argues that a model's reliability is not captured by how often it is right; what matters is whether a user can detect when it is wrong within the time and effort a deployment allows. The authors define a Verification-Cost Error (VCE) as an incorrect input-output pair that at least a fraction $q$ of verifiers fails to identify within a budget $B$. They propose two observable indicators — the probability of a correct verdict within budget and the observed budget consumption — and a six-step benchmarking protocol that records verification traces under a pre-registered measurement specification. If the paper is right, evaluation tables should carry verification-outcome columns next to accuracy, and systems should be designed to be cheap to check, not merely to be right.","feed_headline":"Verification cost belongs on AI report cards beside accuracy","feed_subtitle":"Two models can score alike while one quietly drains human oversight time hiding its errors.","key_machinery":"The load-bearing construct is the VCE condition, defined over a declared measurement specification $u$ (cost unit), $V$ (verifier population), $B$ (budget), $\\tau$ (confidence threshold), and $q$ (failure fraction), together with an acceptance relation $A$ fixed ex ante. The VCE definition is deliberately outcome-based: it is decidable from the verification trace because it asks what the protocol observes (whether a correct verdict was reached within budget), not what it cannot observe (how long a correct verdict would have taken). Two supporting definitions do the work: the failure probability $\\Pr[S_B=0]$, decomposed into wrong verdict, budget exhaustion, and declared unverifiability, and the observed budget consumption, which measures burden rather than latent cost. A six-step protocol (ground truth, verification trace, stopping rule, verifier controls, stratified reporting, model-level aggregation) carries the construct into practice.","core_discovery":"The central claim is that reliability decomposes into correctness and the cost of establishing correctness, and only the second explains deployment risk when errors are plausible. The paper formalizes this with the VCE condition: an incorrect pair $(x,\\hat y)$ is a Verification-Cost Error under measurement specification $(u,V,B,\\tau,q)$ when $\\Pr_{v\\sim V}[S_B(x,\\hat y;v)=0]\\ge q$, i.e., when at least a fraction $q$ of the verifier population fails to reach a ground-truth-consistent verdict within budget $B$. Because a protocol that halts at the budget never observes how much longer a correct verdict would have taken, the latent verification cost is replaced by two observable quantities: the probability of success within budget and the observed budget consumption $E[E^u_d]/B$. These are reported separately for correct and incorrect outputs, and the paper argues that the resulting indicators, not accuracy alone, should be the basis of AI evaluation.","pith_inferences":["The framework turns verification cost into a monitoring indicator: tracking observed budget consumption across model releases would reveal whether capability gains are buying cheaper oversight or merely hiding failures behind fluency, without changing the protocol.","A direct causal test of the self-masking hypothesis is within reach: randomly present the same correct and incorrect content in plain and authority-styled formats, and measure whether the styled presentations raise time-to-verdict and lower success-within-budget under a fixed $B$.","The VCE envelope could be read as a cost-of-knowledge measure for a benchmark, supplementing accuracy with how much human audit a model's errors demand.","For agentic systems, the verification trace is already generated by the agent's own tool calls; recording it against a budget would let VCE-style indicators be measured without dedicated human panels."],"forward_implications":["Evaluation tables would report verification-outcome indicators beside accuracy: the probability of a correct verdict within budget and the observed budget consumption.","Benchmarks that report only accuracy are incomplete, because a model can score high on a test suite while its errors silently consume large amounts of human oversight.","System design should target verification burden, not only error rate: outputs that expose checkable intermediate representations are preferred, and explanations count as valuable only when they reduce the cost of checking.","VCE results are deployment-relative: the same model output may be a VCE under a 5-minute review budget and not under a 60-minute one, so reported rates must carry the full measurement specification.","A VCE rate is not a single number; the confirmed rate and the inconclusive rate form a classification envelope that should be reported over the denominator of incorrect outputs."],"supporting_citations":[{"why":"Supplies the motivating empirical asymmetry: experienced developers took 19% longer with AI while estimating they were 20% faster.","marker":"Becker et al. [2025]"},{"why":"Preregistered evaluation showing RAG legal tools hallucinate in 17% to 33% of cases, evidence that grounding restructures rather than removes verification burden.","marker":"Magesh et al. [2025]"},{"why":"Profiles legal hallucinations with counterfactual bias and overconfidence, the conditions the paper ties to high verification cost.","marker":"Dahl et al. [2024]"},{"why":"Shows AI explanations induce overreliance and do not improve human-AI team performance, grounding the explanatory-tax argument.","marker":"Bansal et al. [2021]"},{"why":"Holistic evaluation of language models that moves beyond accuracy but omits verification cost; the position the paper extends.","marker":"Liang et al. [2023]"},{"why":"Dynabench adversarial benchmarking; used to argue that adversarial evaluation does not prioritize high-verification-cost outputs.","marker":"Kiela et al. [2021]"},{"why":"Semantic entropy detects only a subclass of hallucinations, showing detectability varies with failure type, which VCEs generalize.","marker":"Farquhar et al. [2024]"},{"why":"Introduces the Banana Peel taxonomy and cognitive verification, which the paper formalizes as VCEs within the plausibility-focused subclass.","marker":"Immediato [2025]"},{"why":"Documents benchmark contamination and re-entry of evaluation artefacts, supporting the paper's co-adaptive dynamics risk.","marker":"Deng et al. [2024]"}],"fun_headline_variants":["Correctness alone can't gauge AI reliability","Verification cost is the missing AI metric","Accuracy hides the real cost of verifying AI","Put verification cost on AI report cards","AI evaluation needs a verification budget check"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The definitions collapse if ground truth cannot be fixed independently of the verification episode: the same bounded process cannot both detect an error and certify that one exists, so outputs whose correctness is not settled in advance fall outside the framework.","fun_headline_variants_meta":{"raw":{"variants":["Correctness alone can't gauge AI reliability","Verification cost is the missing AI metric","Accuracy hides the real cost of verifying AI","Put verification cost on AI report cards","AI evaluation needs a verification budget check"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1594,"prompt_tokens":935,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":609}},"tokens_in":551,"tokens_out":659,"duration_ms":6828,"temperature":1.0,"reasoning_tokens":609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:25:39.726968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed protocol on two models with identical accuracy on a fixed dataset: if the probability of a correct verdict within budget and the observed budget consumption are statistically indistinguishable across models, the added dimension adds no predictive power beyond accuracy. At the mechanism level, if verifiers detect errors in fluent, authority-styled outputs at the same rate and within the same budget as in plain outputs, the self-masking hypothesis fails.","supporting_citations":[],"review_version":1}