Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation

Data Science; Economics); Junhao Luo; Ning Huang; Southwestern University of Finance; Wei Deng (School of Statistics; Wenxuan Tang; Ziqi Sha

REVIEW 2 major objections 5 minor 14 references

Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that a benchmark score should not be interpreted until the protocol's observation support is audited for point identification, and that a finite-class criterion makes this audit structural and free of any…

desk verdict The audit idea is sound and the empirical work is careful, but the 'two cells suffice' payoff is an artifact of a seven-policy class that omits the base-incorrect behavior the paper itself documents. read the letter →

arxiv 2608.13326 v1 pith:KIV5QELS submitted 2026-08-13 cs.CL

Junhao Luo , Ning Huang , Ziqi Sha , Wenxuan Tang , Wei Deng (School of Statistics , Data Science , Southwestern University of Finance , Economics) This is my paper · ORCID

classification cs.CL

keywords protocol-levelidentifiabilityauditLLMevaluationvaliditypointidentificationobservationequivalenceintervention-responsefidelityminimumidentifyingsupportfrozenpolicyclasssolver-groundedreasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a benchmark score should not be interpreted until a prior question is answered: given only the observations the protocol collects, is the behavioral quantity the score claims to measure point-identified at all? It formalizes a finite-class identifiability audit in which a target estimand is identified exactly when no two policies with different estimand values are observationally equivalent on the protocol's support, and the check runs with zero model calls. On a solver-grounded reasoning testbed, the audit shows that base-only observation collapses seven frozen policies into a single equivalence class with six collisions, while full support separates all seven, and every leave-one-out support keeps at least one collision. Empirically, base accuracy and selective-response fidelity diverge under pair-valid generation (0.620 versus 0.324, and 0.646 versus 0.331 on a second source), so local correctness does not determine intervention-response fidelity. A reader should care because the same structural check can be run before any inference budget is spent, and it shows why two precise-sounding scores can answer entirely different questions.

What carries the argument

The load-bearing object is the finite-class point-identification criterion of Theorem 1: on a declared policy class H, an estimand τ is point-identified by the observation support O exactly when no two policies with different τ values share an observation profile on O. Around this sits the Interventional Response Tensor $R_{w,\pi,r}$, whose 36 cells index three worlds (base, target, sham), the six $S_3$ label permutations mapping TRUE/FALSE/UNKNOWN onto A/B/C, and two readouts (generated choice and candidate scoring). Collision structure doubles as a synthesis engine: for each cross-estimand pair the distinguishing cells form a set, and a support identifies τ precisely when it hits every such set, so finding the cheapest identifying protocol becomes an exact minimum-hitting-set enumeration over the 16-subset lattice of support components.

What would settle it

Insert an eighth deterministic policy into the frozen class that matches the ideal updater on a candidate two-cell support but inverts elsewhere, and rerun the enumeration: if no two-cell support separates every cross-estimand pair, the minimum-support conclusion is an artifact of the chosen policy class. On the empirical side, a third model family whose base accuracy and selective-response fidelity coincide under the same six-direction balanced-transition protocol would refute the reported dissociation.

Watch

Extended reading notes

Core claim

The paper's central claim is that evaluation-design validity is an identification problem, not an estimation problem. For a finite behavioral policy class H, an observation support O, and a binary estimand τ, Theorem 1 proves that τ is point-identified on O if and only if every pair of policies with different τ values is observationally distinguishable on O; any estimator reading only O must give equal values to observationally equivalent policies, so a collision between different-τ policies is a structural obstruction that more data cannot cure. Applied to the frozen class of seven deterministic policies, the criterion certifies that base-only support is under-identified (one equivalence class, six cross-estimand collisions), that the full 36-cell support identifies the contract-stable selective-response property (seven classes, zero collisions), and that all four support components are essential. The same collision structure synthesizes a minimum identifying support of two cells for the frozen class, and on frozen model responses base accuracy and selective-response fidelity diverge in every model–readout stratum and across six balanced oracle-transition directions.

Load-bearing premise

The audit's verdicts are relative to the hand-declared seven-policy class, and the paper's Limitations section concedes that if real models exhibit behaviors outside that class, the structural results—including the two-cell minimum support—need not transfer.

Editorial extensions

If this is right

A protocol that is not point-identified cannot be rescued by more data; only a larger support can fix it, so identification must be checked before estimation.
Base-only observation under-identifies the full-support selective-response property on the frozen class, so base accuracy alone cannot certify intervention-response fidelity.
All four support components—target world, matched sham, paired readout, and full $S_3$ mapping—are essential; removing any one reintroduces at least one cross-estimand collision.
The minimum identifying support $O^*$ has two cells (26 distinct minima), so the audit does not demand the full 36-cell tensor; identification has a precise price.
Base correctness and selective-response fidelity are distinct estimands that diverge empirically (0.620 versus 0.324 across six balanced transitions, and 0.646 versus 0.331 on a second source), so local accuracy is not a proxy for intervention-response fidelity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The minimum-hitting-set formulation is the portable part of the paper: any benchmark with a declared finite policy class and binary estimand inherits the same machinery, so the two-cell answer is case-specific while the synthesis procedure generalizes.
A natural follow-up is to ship an 'identifiability certificate' with released benchmarks: a witness list of collisions showing which claims the published support can and cannot separate, computed before any model inference.
Applied to standard reasoning benchmarks, the audit predicts that observed accuracy differences across prompt or edit variants mix genuine estimand gaps with under-identification artifacts, a hypothesis a re-scoring study could test.
The granularity bound on the stochastic mixture suggests that finitely many observations can separate behavior classes but not mixture parameters, a distinction that would also apply to claims about 'how often' a model uses a strategy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—

Worth a look: this paper gives evaluation designers a pre-inference, zero-model-call check for whether a protocol can separate the behaviors it claims to measure. Theorem 1 is just the finite-class point-identification condition, but the application is genuinely useful: base-only support collapses all seven frozen policies into one equivalence class, full support separates them, and every leave-one-out support leaves a constructive collision. The exhaustive 16-subset lattice and the witness construction are clean, and the empirical section is unusually careful — cluster bootstrap, a second deterministic source, an attention-mask rerun, and explicit separation of local versus full-support estimands. The authors are honest about the target/sham confound and the frozen-class limitation.

The soft spot is the headline synthesis result. The 2-cell minimum support O* omits the base cell, which is harmless only because all seven policies are base-correct by construction. The paper's own data show real models are base-correct only 40–62% of the time. Add one policy that is base-incorrect but matches the ideal updater on target and sham: it is observationally equivalent to the ideal on O*, yet τfull = 0. So "two cells instead of 36" is an artifact of excluding the most common real failure mode, and the abstract presents it as the payoff. The Limitations sentence covers this, but it belongs in the framing of the synthesis claim.

The other issue is verification: code and data are withheld until acceptance, so the empirical numbers cannot be checked. The synthetic results are enumerable from the paper, but the headline dissociation rests on frozen responses that should be released with the preprint.

Verdict: this deserves a serious referee. The audit framework is a real methodological addition, and the failure analysis is mostly sound. I would push for revision: release the frozen artifacts, and either expand the policy class to include base-incorrect policies before claiming minimal support, or clearly state that O* is the minimum for a class that assumes base correctness. The target/sham confound should be reframed as protocol discrimination, which the authors already half-do.

Who it is for: evaluation designers and anyone building benchmarks who wants a structural check before burning inference budget. I would cite the audit framework; I would not cite the 2-cell number without the caveat.

Referee Report

2 major / 5 minor

Summary. The paper proposes a protocol-level identifiability audit for LLM evaluation: given a declared finite policy class H, an observation support O, and a target estimand τ, the audit checks whether O separates every pair of policies with different τ values. Theorem 1 (Section 2.2) states that point identification is equivalent to the absence of cross-estimand observational equivalences, and the proof is correct. The audit is applied to a solver-grounded three-valued reasoning setting with a frozen class of seven deterministic policies. The paper finds that base-only support collapses all seven policies into one equivalence class, full support identifies the estimand, and a minimum identifying support of two cells exists (Section 4.5). Empirically, the paper reports a dissociation between base accuracy and selective-response fidelity on two instruction-tuned LLMs, with cluster-bootstrap confidence intervals and a reproduction on a second deterministic source. The paper explicitly limits the structural results to the frozen policy class.

Significance. The main contribution is a structural, model-free criterion for diagnosing under-identification before model inference. The enumeration is exhaustive and machine-checkable in principle, and the theorem is correct. The empirical work is careful: cluster-level bootstrap, explicit confidence intervals, a second source, and an attention-mask reproducibility check. If the identification audit is adopted as a complement to construct-validity arguments, it could be useful. However, the significance of the specific minimum-support result is limited by the hand-selected policy class, which is the paper's own stated limitation.

major comments (2)

[§4.5 and Abstract] The 2-cell minimum identifying support O* = {target|m1|gen, sham|m0|score} contains no base cell. Because every policy in the frozen class H is base-correct by construction (Table 8), base responses carry no separating information for the class. The paper's own empirical results, however, show base accuracy far below 1 (0.403 in Table 3 and 0.620 in §5.5), so a policy that is base-incorrect but target- and sham-correct on the two O* cells would be observationally equivalent to the ideal updater on O* while having τfull=0. The Limitations section acknowledges this in general terms ('additional policies may require more support'), but the abstract and Section 8 present the two-cell result as the headline payoff without this caveat. The authors should either expand H to include base-incorrect behavior (and report whether any two-cell support survives) or qualify the abstract and discussion explicitly.
[§5.5 and Table 3] The headline comparison of 0.620 base accuracy versus 0.324 selective response is between a marginal rate and a joint rate (base correct, target following, sham stability). The gap is therefore partly mechanical. The paper's own conditional statistic (0.138 local response among base-correct units) is the more direct evidence for the dissociation claim, but the abstract and Section 5.5 should make clear that the 0.324 rate is a joint probability that cannot be directly contrasted with a marginal accuracy rate.

minor comments (5)

[Abstract] The abstract states '0.620 versus 0.324 across six balanced oracle-transition directions', while Table 3 lists base accuracy 0.403 for the original 24-cluster pilot; the abstract should note that the 0.620 figure comes from the balanced 48-cluster set.
[§2.1] The symbol θlocal is introduced but not used consistently in Section 5.2; aligning the notation between the formal definition and the empirical section would improve clarity.
[Footnote 2] The phrase 'These authors contributed equally as second authors' is ungrammatical; it should read 'contributed equally' without the word 'second'.
[Table 1] The |O| column for O0 is 1; since the full tensor has 36 cells, it would help to state explicitly that O0 projects onto the base world only, ignoring the mapping and readout axes.
[Figure 9] Panels (b) and (c) use shading for 'number of minima that use the cell' and 'load', but no colorbar or numeric legend is provided; adding one would make the figure interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the structural audit is explicitly relative to a declared policy class, and the empirical dissociation is a direct measurement rather than a fitted prediction.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. Theorem 1 is a formal statement of the definition of point identification on a finite policy class, proven directly from observational equivalence; it is not a substantive empirical claim smuggled in as a theorem. The seven-policy class is declared, not fitted to the observed accuracy or fidelity numbers, so the synthetic recovery results (base-only under-identification, full-support identification, leave-one-out collisions) are exact enumerations over an explicit H. The minimum identifying support O* is likewise computed as a hitting set over distinguishing cells of that same declared class, and the paper explicitly hedges the numerical outcome as case-specific: 'the numerical minimum is case-specific, while the synthesis procedure is the methodological contribution.' The empirical accuracy–fidelity dissociation (0.620 vs. 0.324, and the replicated 0.646 vs. 0.331) is a measurement result on frozen model responses, not a value predicted from a fitted parameter. The only candidate concern is that the 2-cell support O* contains no base cell, and a base-incorrect policy added to H would break identification on O*. But the paper explicitly limits its structural claims to the frozen class: 'The structural results are relative to a frozen seven-policy class, not an exhaustive model of natural-language behavior; additional policies may require more support.' That is an honest external-validity limitation, not a circular derivation. There are no self-citations, no imported uniqueness theorems, and no fitted inputs renamed as predictions. The audit is a worked formal example plus an empirical pilot, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The seven behavioral policies are formal modeling devices used to define the audit, not postulated entities requiring independent evidence. The interventional response tensor is a data structure. The only free parameters in the empirical work are measured rates, not fitted constants; the illustrative cost model (candidate scoring at three times generated choice) is an assumption for the cost calculation, not a fitted parameter.

assumptions (5)

domain assumption The declared finite behavioral policy class H is a sufficient proxy for the space of plausible model behaviors.
Section 4.1 defines seven deterministic policies plus one stochastic mixture; all structural identifiability and synthesis conclusions are relative to this class. The Limitations section concedes that additional policies may require more support.
domain assumption Oracle labels are fixed by an independent solver under open-world signed-Horn three-valued semantics.
Section 3.1 states that the solver removes experimenter judgment from the oracle. This constrains the empirical setting to a controlled formal domain and does not guarantee transport to natural-language evaluation.
standard math Theorem 1 relies on standard finite-set point-identification reasoning.
The proof in Section 2.2 uses only the definition of observational equivalence and the requirement that estimators measurable with respect to O assign the same value to equivalent policies.
domain assumption The four support components (target, sham, paired readout, full S3 mapping) are the only observation axes relevant for the audit.
Section 3.3-3.4 define these as the protocol requirements; the 16-subset lattice and leave-one-out analysis enumerate only these components. Other potential axes, such as prompt wording or temperature, are not audited.
domain assumption Target and sham edits are assumed to exercise the intended protocol-discrimination role despite differing in edit type and surface form.
Section 3.2 acknowledges that target uses support replacement while sham adds a query-disjoint fact, so edit type perfectly distinguishes the roles. The paper explicitly refrains from claiming a pure semantic-edit effect, which limits the empirical interpretation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation." pith.science (2026). https://pith.science/paper/KIV5QELS

@misc{pith2026260813326,
  author       = {Pith},
  title        = {Pith review of: Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KIV5QELS}},
  note         = {Machine review of arXiv:2608.13326}
}

abstract

LLM benchmark scores can be precise even when the observation protocol does not identify the behavioral property they are intended to measure. In a controlled, solver-grounded setting, we formalize a protocol-level identifiability audit over a finite behavioral policy class: given policies H, observation support O, and estimand $\tau$, we test whether O separates every pair with different $\tau$. The audit requires zero model calls and resolves our diagnostic case: base-only observation collapses seven frozen deterministic policies into one equivalence class; full support yields seven classes and no cross-estimand collisions; every leave-one-out support retains a constructive collision witness. Empirically, both constrained-generation variants have pair-validity 1.0, yet base accuracy and selective-response fidelity diverge - 0.620 versus 0.324 across six balanced oracle-transition directions (cluster-bootstrap 95% CI [0.600, 0.642] vs. [0.304, 0.345]) - and the gap recurs on a second deterministic source (0.646 vs. 0.331). The audit also synthesizes a minimum identifying support $O^*$ for the frozen policy class: two cells instead of the full 36-cell tensor. This case shows how evaluation-design validity can be checked structurally before model inference and why base correctness does not determine intervention-response fidelity.

Figures

Figures reproduced from arXiv: 2608.13326 by the authors.

**Figure 1.** Protocol design overview. (1) Audited policies: seven frozen deterministic behaviors H1–H7 (ideal semantic updater, target inertia, arbitrary edit reactor, fixed-label responder, mapping-conditional shortcut, generationselection failure, candidate-scoring success) with their base/target/sham response rules. (2) Observation geometry: the world×mapping×readout tensor with 3×6×2 = 36 cells; orange marks the selected … view at source ↗

**Figure 2.** Support-component essentiality: the full power-set lattice over target ( [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Accuracy–response gaps across six balanced oracle-transition directions with synchronized cluster [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Mapping audit: rates before (n = 288) and after validity filtering (n = 176). Numerators remain fixed for semantic equivariance, fixed-label attachment, and equivariance plus correctness, so each rate rises by 2.4–4.0 percentage points; denominator choice changes rates…

**Figure 5.** Figure 5: Paired readout transition from generated [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Robustness readout contracts (equal validity) and bounded source transport: the accuracy–selective gap [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Target–sham isolation audit: the formal oracle passes on 144 worlds; the surface audit flags material [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Six mutually exclusive diagnoses over 576 paired units (counts in bars, cumulative share in red). Three archetypes—reportability failure, baseknowledge failure, and target inertia—account for 91.3% of units. These post-result labels do not imply internal causes. Algor…

**Figure 9.** Figure 9: Protocol synthesis on the 36-cell tensor. (a) Full support: each cell is shaded by its [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

[3]

InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages 375–385, New York, NY , USA

Measure- ment and fairness. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages 375–385, New York, NY , USA. Asso- ciation for Computing Machinery. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil˙e Lukoši...

work page 2021
[6]

InFindings of the Association for Com- putational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA

Making reasoning matter: Measur- ing and improving faithfulness of chain-of-thought reasoning. InFindings of the Association for Com- putational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA. Association for Com- putational Linguistics. Qwen Team

work page 2024
[7]

Inioluwa Deborah Raji, Emily Denton, Emily M

Qwen2.5 technical report.arXiv preprint arXiv:2412.15115. Inioluwa Deborah Raji, Emily Denton, Emily M. Ben- der, Alex Hanna, and Amandalynne Paullada

arXiv
[9]

InAdvances in Neu- ral Information Processing Systems 36 (NeurIPS 2023)

Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. InAdvances in Neu- ral Information Processing Systems 36 (NeurIPS 2023). Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, and Yonatan Belinkov

work page 2023
[10]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9935–9960, Suzhou, China

Measuring chain of thought faithfulness by unlearning reasoning steps. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9935–9960, Suzhou, China. Association for Computational Linguistics. 9 Hanna Wallach, Meera Desai, A. Feder Cooper, An- gelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Ch...

work page 2025
[11]

Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico. Association for Computational ...

work page 2024
[13]

arXiv preprint arXiv:2405.15092

Disso- ciation of faithful and unfaithful reasoning in LLMs. arXiv preprint arXiv:2405.15092. Kerem Zaman and Shashank Srivastava

arXiv
[14]

InProceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29425–29449, Suzhou, China

A causal lens for evaluating faithfulness metrics. InProceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29425–29449, Suzhou, China. Association for Computational Lin- guistics. A Model–Readout Behavioral Composition Contract Sel. Nonsel. Perv. Inert. Inval. Total Qwen·scoring 17 2 18 107 0 144 Qwen·choice 9 2 1...

work page 2025

Show all 14 references

[2020]

InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online

Evaluating models’ local decision bound- aries via contrast sets. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics. Abigail Z. Jacobs and Hanna Wallach

2020
[2021]

Association for Computational Linguistics

What will it take to fix benchmarking in natural language under- standing? InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4843–4855, Online. Association for Computational...

2021
[2023]

Llama Team, AI @ Meta

Measuring faithful- ness in chain-of-thought reasoning.arXiv preprint arXiv:2307.13702. Llama Team, AI @ Meta

arXiv
[2024]

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar

The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar

arXiv
[2025]

Charlotte Siska, Katerina Marazopoulou, Melissa Ailem, and James Bono

Measurement to meaning: A validity- centered framework for AI evaluation.arXiv preprint arXiv:2505.10573. Charlotte Siska, Katerina Marazopoulou, Melissa Ailem, and James Bono

arXiv
[2026]

Accepted at COLM

Compared to what? baselines and metrics for counterfactual prompting.arXiv preprint arXiv:2605.01048. Accepted at COLM

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [3]

InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages 375–385, New York, NY , USA

Measure- ment and fairness. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages 375–385, New York, NY , USA. Asso- ciation for Computing Machinery. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil˙e Lukoši...

work page 2021

[2] [6]

InFindings of the Association for Com- putational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA

Making reasoning matter: Measur- ing and improving faithfulness of chain-of-thought reasoning. InFindings of the Association for Com- putational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA. Association for Com- putational Linguistics. Qwen Team

work page 2024

[3] [7]

Inioluwa Deborah Raji, Emily Denton, Emily M

Qwen2.5 technical report.arXiv preprint arXiv:2412.15115. Inioluwa Deborah Raji, Emily Denton, Emily M. Ben- der, Alex Hanna, and Amandalynne Paullada

arXiv

[4] [9]

InAdvances in Neu- ral Information Processing Systems 36 (NeurIPS 2023)

Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. InAdvances in Neu- ral Information Processing Systems 36 (NeurIPS 2023). Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, and Yonatan Belinkov

work page 2023

[5] [10]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9935–9960, Suzhou, China

Measuring chain of thought faithfulness by unlearning reasoning steps. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9935–9960, Suzhou, China. Association for Computational Linguistics. 9 Hanna Wallach, Meera Desai, A. Feder Cooper, An- gelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Ch...

work page 2025

[6] [11]

Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico. Association for Computational ...

work page 2024

[7] [13]

arXiv preprint arXiv:2405.15092

Disso- ciation of faithful and unfaithful reasoning in LLMs. arXiv preprint arXiv:2405.15092. Kerem Zaman and Shashank Srivastava

arXiv

[8] [14]

InProceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29425–29449, Suzhou, China

A causal lens for evaluating faithfulness metrics. InProceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29425–29449, Suzhou, China. Association for Computational Lin- guistics. A Model–Readout Behavioral Composition Contract Sel. Nonsel. Perv. Inert. Inval. Total Qwen·scoring 17 2 18 107 0 144 Qwen·choice 9 2 1...

work page 2025