{"id":"aabb19b9-f241-4d95-9fa5-7bf1c7a084de","arxiv_id":"2608.13326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A protocol-level identifiability audit over finite behavioral policy classes shows that base-only observation under-identifies the selective-response estimand, full support identifies it, and base accuracy diverges from intervention-response fidelity in two instruction-tuned models.","lead":"This paper introduces a formal audit that checks whether an LLM evaluation protocol's observations can distinguish the behavioral property it claims to measure, before any model is run. In a controlled reasoning setting, it shows that base-only accuracy hides a much lower selective-response fidelity, and that a two-cell observation plan can identify the target property where a 36-cell tensor is overkill.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2-cell minimum support O* omits base observations; a policy that is base-incorrect but target- and sham-correct is observationally equivalent to the ideal updater on O*, so the 'two cells suffice' result is an artifact of excluding base-incorrect behavior from H.","rationale":"The reader's weakest assumption is that the declared finite policy class adequately spans plausible model behaviors; my concern is a concrete instance of that assumption failing at the exact point where the paper makes its strongest synthetic claim. The audit machinery and Theorem 1 are sound; the issue is the interpretation of the minimum-support synthesis. The absence of a base cell in every 2-cell minimum is not an oversight within H—all seven policies are base-correct—but it makes the minimum support sensitive to a very basic omitted behavior. The paper's own empirical numbers show base-incorrectness is common, so the omission is material, not hypothetical. I do not recommend REJECT because the paper repeatedly frames results as relative to the frozen class and the central theorem is conditional by design; the concern changes the strength of the contribution, not its logical validity. The verdict should remain CONDITIONAL, with an explicit condition that the authors either augment H with base-incorrect (and other empirically attested) policies and recompute the synthesis, or state prominently that O* is only a minimum for a class in which every policy is base-correct. This is a partial agreement with the reader: same general weakness, but the reader did not identify that the omission has a specific structural signature (no base cell) that can be tested by a single added policy.","tokens_in":22398,"tokens_out":16251,"duration_ms":169553,"concrete_test":"Add to H the policy h_B defined on the 36-cell tensor as: incorrect on base for every mapping/readout; on target, output the oracle-correct label; on sham, output the oracle-correct base label (identical to ideal on target-gen and sham-score cells). Re-run Algorithm 1 and the minimum hitting-set synthesis over H ∪ {h_B}. If the output reports h_B ≡_{O*} ideal_semantic_updater with τ differing, the published O* no longer identifies the estimand on the expanded class; the minimum-support size and composition should then be recomputed. As a robustness variant, add a battery of 'intervention-correct but base-incorrect' policies with different readout/mapping patterns and check whether any 2-cell support still hits all cross-estimand distinguishing sets. If no 2-cell support survives, the abstract's two-cell claim requires the caveat that base-incorrect policies are excluded from H.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing step is the synthesis claim in Section 4.5 that a 2-cell support O* = {target|m1|gen, sham|m0|score} identifies τfull on the frozen seven-policy class. O* contains no base cell. For the seven declared policies this is harmless because every policy in Table 8 is base-correct by construction, so base responses carry no separating information. But τfull is defined to require base-correctness (Section 2.1), and the empirical study itself finds base accuracy far below 1 (0.403 in Table 3, 0.620 in §5.5). Add a single deterministic policy h_B: on base it returns an oracle-incorrect label; on target it returns the oracle-correct target label; on sham it returns the oracle-correct base label (so it matches ideal_semantic_updater on exactly the two O* cells). Then τfull(h_B)=0 while τfull(ideal)=1, yet h_B ≡_{O*} ideal, so O* does not point-identify τfull on H ∪ {h_B}. This is not a contradiction of Theorem 1, which is explicitly relative to the declared H, but it shows the headline 'two cells instead of 36' depends on excluding base-incorrect behavior—the same behavior the empirical section documents. The Limitations sentence about the frozen class is accurate but does not appear in the abstract, where the 2-cell result is presented as the payoff of the audit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a protocol-level identifiability audit for LLM evaluation: given a declared finite policy class H, an observation support O, and a target estimand τ, the audit checks whether O separates every pair of policies with different τ values. Theorem 1 (Section 2.2) states that point identification is equivalent to the absence of cross-estimand observational equivalences, and the proof is correct. The audit is applied to a solver-grounded three-valued reasoning setting with a frozen class of seven deterministic policies. The paper finds that base-only support collapses all seven policies into one equivalence class, full support identifies the estimand, and a minimum identifying support of two cells exists (Section 4.5). Empirically, the paper reports a dissociation between base accuracy and selective-response fidelity on two instruction-tuned LLMs, with cluster-bootstrap confidence intervals and a reproduction on a second deterministic source. The paper explicitly limits the structural results to the frozen policy class.","tokens_in":22588,"tokens_out":8639,"duration_ms":84990,"significance":"The main contribution is a structural, model-free criterion for diagnosing under-identification before model inference. The enumeration is exhaustive and machine-checkable in principle, and the theorem is correct. The empirical work is careful: cluster-level bootstrap, explicit confidence intervals, a second source, and an attention-mask reproducibility check. If the identification audit is adopted as a complement to construct-validity arguments, it could be useful. However, the significance of the specific minimum-support result is limited by the hand-selected policy class, which is the paper's own stated limitation.","major_comments":[{"comment":"The 2-cell minimum identifying support O* = {target|m1|gen, sham|m0|score} contains no base cell. Because every policy in the frozen class H is base-correct by construction (Table 8), base responses carry no separating information for the class. The paper's own empirical results, however, show base accuracy far below 1 (0.403 in Table 3 and 0.620 in §5.5), so a policy that is base-incorrect but target- and sham-correct on the two O* cells would be observationally equivalent to the ideal updater on O* while having τfull=0. The Limitations section acknowledges this in general terms ('additional policies may require more support'), but the abstract and Section 8 present the two-cell result as the headline payoff without this caveat. The authors should either expand H to include base-incorrect behavior (and report whether any two-cell support survives) or qualify the abstract and discussion explicitly.","section":"§4.5 and Abstract"},{"comment":"The headline comparison of 0.620 base accuracy versus 0.324 selective response is between a marginal rate and a joint rate (base correct, target following, sham stability). The gap is therefore partly mechanical. The paper's own conditional statistic (0.138 local response among base-correct units) is the more direct evidence for the dissociation claim, but the abstract and Section 5.5 should make clear that the 0.324 rate is a joint probability that cannot be directly contrasted with a marginal accuracy rate.","section":"§5.5 and Table 3"}],"minor_comments":[{"comment":"The abstract states '0.620 versus 0.324 across six balanced oracle-transition directions', while Table 3 lists base accuracy 0.403 for the original 24-cluster pilot; the abstract should note that the 0.620 figure comes from the balanced 48-cluster set.","section":"Abstract"},{"comment":"The symbol θlocal is introduced but not used consistently in Section 5.2; aligning the notation between the formal definition and the empirical section would improve clarity.","section":"§2.1"},{"comment":"The phrase 'These authors contributed equally as second authors' is ungrammatical; it should read 'contributed equally' without the word 'second'.","section":"Footnote 2"},{"comment":"The |O| column for O0 is 1; since the full tensor has 36 cells, it would help to state explicitly that O0 projects onto the base world only, ignoring the mapping and readout axes.","section":"Table 1"},{"comment":"Panels (b) and (c) use shading for 'number of minima that use the cell' and 'load', but no colorbar or numeric legend is provided; adding one would make the figure interpretable.","section":"Figure 9"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely acceptable after revision if the authors address the O* caveat. The central theorem is sound and the empirical work is careful. The main risk is overselling the two-cell result as a general finding when it is specific to a base-correct policy class, which the paper's own empirical data contradict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nWorth a look: this paper gives evaluation designers a pre-inference, zero-model-call check for whether a protocol can separate the behaviors it claims to measure. Theorem 1 is just the finite-class point-identification condition, but the application is genuinely useful: base-only support collapses all seven frozen policies into one equivalence class, full support separates them, and every leave-one-out support leaves a constructive collision. The exhaustive 16-subset lattice and the witness construction are clean, and the empirical section is unusually careful — cluster bootstrap, a second deterministic source, an attention-mask rerun, and explicit separation of local versus full-support estimands. The authors are honest about the target/sham confound and the frozen-class limitation.\n\nThe soft spot is the headline synthesis result. The 2-cell minimum support O* omits the base cell, which is harmless only because all seven policies are base-correct by construction. The paper's own data show real models are base-correct only 40–62% of the time. Add one policy that is base-incorrect but matches the ideal updater on target and sham: it is observationally equivalent to the ideal on O*, yet τfull = 0. So \"two cells instead of 36\" is an artifact of excluding the most common real failure mode, and the abstract presents it as the payoff. The Limitations sentence covers this, but it belongs in the framing of the synthesis claim.\n\nThe other issue is verification: code and data are withheld until acceptance, so the empirical numbers cannot be checked. The synthetic results are enumerable from the paper, but the headline dissociation rests on frozen responses that should be released with the preprint.\n\nVerdict: this deserves a serious referee. The audit framework is a real methodological addition, and the failure analysis is mostly sound. I would push for revision: release the frozen artifacts, and either expand the policy class to include base-incorrect policies before claiming minimal support, or clearly state that O* is the minimum for a class that assumes base correctness. The target/sham confound should be reframed as protocol discrimination, which the authors already half-do.\n\nWho it is for: evaluation designers and anyone building benchmarks who wants a structural check before burning inference budget. I would cite the audit framework; I would not cite the 2-cell number without the caveat.","headline":"The audit idea is sound and the empirical work is careful, but the 'two cells suffice' payoff is an artifact of a seven-policy class that omits the base-incorrect behavior the paper itself documents.","tokens_in":23202,"tokens_out":2339,"would_cite":true,"duration_ms":23939,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a benchmark score should not be interpreted until the protocol's observation support is audited for point identification, and that a finite-class criterion makes this audit structural and free of any…","keywords":["protocol-level identifiability audit","LLM evaluation validity","point identification","observation equivalence","intervention-response fidelity","minimum identifying support","frozen policy class","solver-grounded reasoning"],"falsifier":"Insert an eighth deterministic policy into the frozen class that matches the ideal updater on a candidate two-cell support but inverts elsewhere, and rerun the enumeration: if no two-cell support separates every cross-estimand pair, the minimum-support conclusion is an artifact of the chosen policy class. On the empirical side, a third model family whose base accuracy and selective-response fidelity coincide under the same six-direction balanced-transition protocol would refute the reported dissociation.","tokens_in":22119,"feed_emoji":"🔍","tokens_out":14662,"duration_ms":121139,"temperature":0.7,"pith_summary":"This paper argues that a benchmark score should not be interpreted until a prior question is answered: given only the observations the protocol collects, is the behavioral quantity the score claims to measure point-identified at all? It formalizes a finite-class identifiability audit in which a target estimand is identified exactly when no two policies with different estimand values are observationally equivalent on the protocol's support, and the check runs with zero model calls. On a solver-grounded reasoning testbed, the audit shows that base-only observation collapses seven frozen policies into a single equivalence class with six collisions, while full support separates all seven, and every leave-one-out support keeps at least one collision. Empirically, base accuracy and selective-response fidelity diverge under pair-valid generation (0.620 versus 0.324, and 0.646 versus 0.331 on a second source), so local correctness does not determine intervention-response fidelity. A reader should care because the same structural check can be run before any inference budget is spent, and it shows why two precise-sounding scores can answer entirely different questions.","feed_headline":"Precise scores can hide a benchmark that fails to identify its target","feed_subtitle":"Base accuracy and response fidelity diverge on real models; a zero-model-call audit exposes why before inference.","key_machinery":"The load-bearing object is the finite-class point-identification criterion of Theorem 1: on a declared policy class H, an estimand τ is point-identified by the observation support O exactly when no two policies with different τ values share an observation profile on O. Around this sits the Interventional Response Tensor $R_{w,\\pi,r}$, whose 36 cells index three worlds (base, target, sham), the six $S_3$ label permutations mapping TRUE/FALSE/UNKNOWN onto A/B/C, and two readouts (generated choice and candidate scoring). Collision structure doubles as a synthesis engine: for each cross-estimand pair the distinguishing cells form a set, and a support identifies τ precisely when it hits every such set, so finding the cheapest identifying protocol becomes an exact minimum-hitting-set enumeration over the 16-subset lattice of support components.","core_discovery":"The paper's central claim is that evaluation-design validity is an identification problem, not an estimation problem. For a finite behavioral policy class H, an observation support O, and a binary estimand τ, Theorem 1 proves that τ is point-identified on O if and only if every pair of policies with different τ values is observationally distinguishable on O; any estimator reading only O must give equal values to observationally equivalent policies, so a collision between different-τ policies is a structural obstruction that more data cannot cure. Applied to the frozen class of seven deterministic policies, the criterion certifies that base-only support is under-identified (one equivalence class, six cross-estimand collisions), that the full 36-cell support identifies the contract-stable selective-response property (seven classes, zero collisions), and that all four support components are essential. The same collision structure synthesizes a minimum identifying support of two cells for the frozen class, and on frozen model responses base accuracy and selective-response fidelity diverge in every model–readout stratum and across six balanced oracle-transition directions.","pith_inferences":["The minimum-hitting-set formulation is the portable part of the paper: any benchmark with a declared finite policy class and binary estimand inherits the same machinery, so the two-cell answer is case-specific while the synthesis procedure generalizes.","A natural follow-up is to ship an 'identifiability certificate' with released benchmarks: a witness list of collisions showing which claims the published support can and cannot separate, computed before any model inference.","Applied to standard reasoning benchmarks, the audit predicts that observed accuracy differences across prompt or edit variants mix genuine estimand gaps with under-identification artifacts, a hypothesis a re-scoring study could test.","The granularity bound on the stochastic mixture suggests that finitely many observations can separate behavior classes but not mixture parameters, a distinction that would also apply to claims about 'how often' a model uses a strategy."],"forward_implications":["A protocol that is not point-identified cannot be rescued by more data; only a larger support can fix it, so identification must be checked before estimation.","Base-only observation under-identifies the full-support selective-response property on the frozen class, so base accuracy alone cannot certify intervention-response fidelity.","All four support components—target world, matched sham, paired readout, and full $S_3$ mapping—are essential; removing any one reintroduces at least one cross-estimand collision.","The minimum identifying support $O^*$ has two cells (26 distinct minima), so the audit does not demand the full 36-cell tensor; identification has a precise price.","Base correctness and selective-response fidelity are distinct estimands that diverge empirically (0.620 versus 0.324 across six balanced transitions, and 0.646 versus 0.331 on a second source), so local accuracy is not a proxy for intervention-response fidelity."],"supporting_citations":[{"why":"Supplies the first frozen instruction-tuned model whose responses ground the empirical accuracy–fidelity dissociation.","marker":"Qwen Team, 2024"},{"why":"Supplies the second deterministic source on which the same gap recurs (0.646 versus 0.331).","marker":"Llama Team, AI @ Meta, 2024"},{"why":"Documents the motivating fragility: accuracy on a reasoning benchmark collapses under symbolic re-instantiation even when added clauses do not change the answer.","marker":"Mirzadeh et al., 2025"},{"why":"The construct-validity audit of many benchmarks that this paper extends from construct operationalization to protocol-level point identification.","marker":"Bean et al., 2025"},{"why":"Grounds the opening premise that capability claims can outrun what benchmark measurements support.","marker":"Raji et al., 2021"},{"why":"Shows chain-of-thought prompting lifts local accuracy on GSM8K, the high-local-accuracy backdrop the paper contrasts with fragility.","marker":"Wei et al., 2022"}],"fun_headline_variants":["Zero-call audit exposes benchmark blind spots","Precise scores, failed identification: protocol audit","Base accuracy ≠ fidelity: audit proves why","Structural check: identify before you estimate LLMs","Collision witness: why benchmark precision misleads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The audit's verdicts are relative to the hand-declared seven-policy class, and the paper's Limitations section concedes that if real models exhibit behaviors outside that class, the structural results—including the two-cell minimum support—need not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Zero-call audit exposes benchmark blind spots","Precise scores, failed identification: protocol audit","Base accuracy ≠ fidelity: audit proves why","Structural check: identify before you estimate LLMs","Collision witness: why benchmark precision misleads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":3060,"prompt_tokens":1008,"completion_tokens":2052,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":1981}},"tokens_in":624,"tokens_out":2052,"duration_ms":19796,"temperature":1.0,"reasoning_tokens":1981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:34:24.823152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Insert an eighth deterministic policy into the frozen class that matches the ideal updater on a candidate two-cell support but inverts elsewhere, and rerun the enumeration: if no two-cell support separates every cross-estimand pair, the minimum-support conclusion is an artifact of the chosen policy class. On the empirical side, a third model family whose base accuracy and selective-response fidelity coincide under the same six-direction balanced-transition protocol would refute the reported dissociation.","supporting_citations":[],"review_version":1}