Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

T0 review · 3 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Perturbation audits meant to validate AI benchmarks can silently manufacture their own conclusions through pipeline bugs that leave no trace in the reported numbers.

desk verdict Honest case study of silent audit-pipeline bugs with a usable gate and chronology template; scoped tightly enough that the main risk is over-portability by readers, not overclaim by authors. read the letter →

arxiv 2607.02586 v1 pith:Y3W2ULBW submitted 2026-07-01 cs.LG cs.SE

Yanhang Li , Zhichao Fan , Zexin Zhuang This is my paper
classification cs.LGcs.SE
keywords benchmarkvalidityperturbationauditsconstructAIgovernanceevidencepipelinefailuremodesdue-diligencegatesafetybenchmarksself-auditchronology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Governance frameworks ask AI providers for documented evaluation evidence, and a common form is a perturbation-based construct-validity audit: edit benchmark items in controlled ways and check that scores move only when the construct flips. This paper argues those audits are themselves fragile measurement systems. Their headline numbers can be produced by implementation failures that a reader cannot detect from the reported scores alone. In a self-audit of five safety benchmarks against two open-weight instruction-tuned models, the authors encounter five classes of such silent pipeline failures and run every cell through a unified six-point due-diligence gate; every cell lands non-confirmatory and none reaches confirmatory. The reusable contribution is a withholding-and-disclosure protocol—taxonomy, gate, and self-audit chronology—so assurance-grade evidence cannot hide silent bugs behind clean numbers.

What carries the argument

The F1–F5 taxonomy of silent, realized, gateable pipeline failures (silent no-op perturbations, regex-extraction artefacts, non-faithful scoring with three subtypes, broken bootstrap pairing, metric-archetype mismatch) together with the hierarchical G1–G6 due-diligence gate that assigns each cell exactly one status and withholds confirmatory verdict unless all six checks pass.

What would settle it

A multi-codebase survey of perturbation-audit pipelines in which none of F1–F5 or close analogues appear after the same G1–G6 gate is applied, or a replication of this ten-cell panel that reaches confirmatory status under fully implemented G5 and G6 with no residual silent defects.

Watch

Extended reading notes

Core claim

Perturbation-based benchmark-audit pipelines are fragile measurement systems whose conclusions can be silently manufactured by pipeline-level bugs invisible in the reported numbers. Five classes of such failure (F1–F5) are realized in a single two-model, five-benchmark self-audit; under a hierarchical six-point due-diligence gate, every cell of the ten-cell panel is non-confirmatory and no cell reaches confirmatory status.

Load-bearing premise

That five failure modes found in one two-model, five-benchmark, single-seed case study form a useful starting taxonomy for other audit pipelines, even though generality across models, benchmarks, and codebases is untested.

Editorial extensions

If this is right

  • Any audit producing benchmark-based governance evidence should publish a self-audit chronology (the eight fields of Box 1) alongside its numbers.
  • A clean scalar ratio without archetype disclosure and scorer validation is not fit for assurance-grade evidence.
  • Evidence consumers should receive the four non-confirmatory buckets (ineligible, scorer-unvalidated, failed numerical gates, exploratory) rather than a single collapsed ‘inconclusive’ verdict.
  • Repair of one pipeline bug can introduce another that only per-cell scorer-output inspection catches.
  • The gate is a withholding protocol supplementary to classical construct-validity evidence, not a route to benchmark-validity verdicts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Similar silent multi-stage pipeline failures likely affect neighbouring evaluation families that also emit governance evidence from code (retrieval reliability, agent safety, scoring harnesses).
  • Making the self-audit chronology a default annex for regulated model evaluations would raise the cost of silent failure without mandating any particular benchmark.
  • The three selection criteria used for F1–F5 (silent, realized, gateable) can be reused to grow the taxonomy across other audit codebases without claiming completeness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that perturbation-based construct-validity audits used as governance evidence are themselves fragile measurement pipelines: their conclusions can be silently manufactured by implementation details invisible in reported numbers. It introduces an illustrative five-class taxonomy (F1–F5) of silent pipeline failures, split into software-assurance and measurement-faithfulness layers, and a six-point due-diligence gate (G1–G6) with hierarchical status precedence. In a self-audit of two 7B instruction-tuned models against five safety benchmarks (10 cells; 200 items; single seed/template), each failure class is realized at least once (including a repair-introduced F3c), and under the gate every cell is non-confirmatory (3 ineligible, 3 scorer-unvalidated, 2 failed G2–G4, 2 exploratory, 0 confirmatory). The authors explicitly scope the work as a case study, position F1–F5 as non-exhaustive, refuse benchmark-validity verdicts, and offer the gate plus a self-audit chronology template (Box 1) as a withholding/disclosure protocol supplementary to classical construct-validity evidence.

Significance. If the case study is taken on its own terms, the contribution is practically useful for AI evaluation and governance: it makes concrete how audit pipelines can emit plausible numbers while silently measuring the wrong object (silent no-ops, regex coverage, inverted conventions, harness ordering, top-k truncation, unpaired bootstrap, archetype mismatch). Strengths include (i) explicit refusal of overclaim (no construct-(in)validity verdicts; F1–F5 non-exhaustive; single-panel scope), (ii) per-class realizations with symptom/fix/governance lesson (Table 2), (iii) honest disclosure of a self-introduced regression (F3c) and prereg deviations (Appendix K), (iv) a hierarchical four-bucket status scheme that separates consumer actions rather than collapsing to one “inconclusive,” and (v) a minimal self-audit chronology template (Box 1) that is immediately actionable. The reusable artefacts—taxonomy, gate, status algorithm, chronology—are more valuable than any scalar CSR ranking. Generality across other audit codebases remains untested, so significance is as a carefully scoped methods/case-study paper, not as a calibrated standard.

major comments (3)
  1. [Abstract, §5, §6] Abstract, §5, and §6: the headline “0 confirmatory / every cell non-confirmatory” is partly procedural. Under the §3 algorithm, confirmatory requires G5 and G6 to be IMPLEMENTED and passed; both remain PROPOSED, so no cell can reach confirmatory by construction. The paper states this, but the abstract and contribution list lead with the zero as if it were primarily an empirical finding about the panel. Please restructure the abstract and §5 so the primary empirical result is the four-way breakdown among non-confirmatory statuses (and the two TruthfulQA cells that would upgrade only after G5/G6), and demote the zero-confirmatory count to an explicitly procedural consequence of the gate design.
  2. [§4, Table 2 (F4)] §4 / Table 2, F4: unlike F1–F3 and F5, F4 is not demonstrated by a controlled in-panel before/after. The text states F4 is “evidenced by out-of-panel/legacy numbers plus a methodological argument” (legacy-Llama CI width 26.4) and that a like-for-like in-panel ablation is unavailable because unpaired bootstrap was removed outright. For a paper whose central claim is that each of five classes was realized and would have altered headline findings, this is the weakest load-bearing cell. Either (a) restore a controlled unpaired-vs-paired comparison on at least one in-panel cell, or (b) reclassify F4 as “methodologically motivated / partially evidenced” and adjust the “demonstrate each” claim and Table 2 accordingly.
  3. [§2, Contributions, Scope] §2 (C1–C3) and Contributions (3)–(4): the gate is positioned as a “withholding and disclosure protocol for assurance-grade evidence,” but admission to F1–F5 requires failures that are silent, realized in this pipeline, and gateable (C1–C3). Combined with a single two-model/five-benchmark/one-seed harness—including a self-introduced F3c—this selection rule can overstate how often silent manufacturing occurs in typical assurance pipelines, or how portable G1–G6 thresholds are. The Scope and Appendix A already disclaim transfer; the contribution statement and governance framing should match that strength of evidence (e.g., “protocol candidate calibrated on one pipeline; transfer untested”) so readers do not treat F1–F5 or the 3/3/2/2/0 breakdown as a portable standard.
minor comments (5)
  1. [Figure 2, §5] Figure 2 caption and §5: CSR values for ineligible and scorer-unvalidated cells are plotted and then labelled “not interpretable as benchmark-validity evidence.” Consider greying those points or moving them to an appendix panel so the figure does not invite ratio comparison across ineligible cells.
  2. [Table 1] Table 1: G1 is PARTIAL and G5/G6 PROPOSED; a one-line “gate maturity” column or footnote would make the procedural zero-confirmatory result easier to see without reading §3 in full.
  3. [§3, Appendix E] CSR definition (§3): the engineering floor ε=0.01 and G3 threshold 0.02 are disclosed in Appendix E with a sensitivity sweep; a short forward pointer in the main-text CSR paragraph would help readers who stop at §3.
  4. [Appendix J, §4] Appendix J shared-prefix MCQ tokens and add_instruction contamination are marked UNFIXED, ACKNOWLEDGED; a single sentence in §4 or Limitations cross-referencing that they do not change any confirmatory status (because none is issued) would close the loop for readers who skip the appendix.
  5. [Abstract, §1, footnote 1] Terminology: “Construct Sensitivity Ratio” → “Contrast Selectivity Ratio” rename is well documented in footnote 1 and Appendix K; ensure the abstract and intro never reintroduce construct-validity language for CSR itself (currently mostly clean).

Circularity Check

1 steps flagged · score 2.0 of 10

Empirical self-audit with one mild procedural self-definition: zero confirmatory cells is true by construction of PROPOSED G5/G6, fully disclosed and not load-bearing for the fragility claim.

  1. self definitional [§3 status-assignment algorithm; also Abstract, §5 Table 3, §6 Discussion]
    "Because G5 and G6 are PROPOSED in this submission, no cell can reach confirmatory; the case study contains no confirmatory cells by construction, a procedural fact rather than an empirical verdict about benchmarks. ... Zero confirmatory cells is procedural, not an empirical null. ... even a cell whose CI is wholly above 1 remains exploratory while G5 or G6 is PROPOSED"

    The abstract and eligibility breakdown present 'no cell reaches confirmatory' / '0 confirmatory' as a result of applying the gate to the panel. Under the paper's own hierarchical algorithm, confirmatory requires G1–G6 all implemented and passed; with G5 and G6 left PROPOSED, zero confirmatory is true by definition of the status labels, independent of the numerical CSR values. Two TruthfulQA cells already satisfy the implemented numerical gates with paired-bootstrap CIs wholly above 1 and are held exploratory only because of that definitional choice. The paper discloses the procedural nature, so this is mild and does not force the fragility claim about F1–F5.

full rationale

This paper is a methodological case study, not a first-principles derivation that reduces a prediction to a fitted constant or a uniqueness theorem. F1–F5 are realized pipeline bugs (including a self-introduced F3c caught by meta-logprob inspection), documented with concrete symptoms, fixes, and before/after effects; that is ordinary empirical demonstration, not circular proof. Adjacent same-author arXivs are cited only for evidence-pipeline framing and are explicitly disclaimed as not supporting the F1–F5 taxonomy. The CSR rename after de-scoping E3 is a pre-committed prereg fallback with construct-validity language retracted, not a renaming of a known result as a new discovery. The sole mild circularity is that the headline count of zero confirmatory cells is forced by the status-assignment algorithm once G5 and G6 remain PROPOSED—two TruthfulQA cells already pass all implemented gates with CIs wholly above 1 and would upgrade if those gates were implemented. The paper states this is procedural rather than an empirical null about benchmarks, so the reduction is disclosed and does not underwrite the central fragility claim. Score 2 reflects that single non-load-bearing procedural step; no fitted-input-as-prediction, uniqueness import, or self-citation chain supports the main argument.

Assumptions & free parameters 6 free parameters · 5 assumptions · 5 invented entities

The paper is a methodological case study. Its load-bearing content is a set of engineering thresholds, domain assumptions about what counts as silent evidence failure, and invented taxonomy/gate entities justified only by realization inside one pipeline. No physical constants or formal theorems are claimed; the ledger therefore concentrates on free thresholds, selection axioms, and the new named objects (F1–F5, G1–G6, CSR, status buckets).

free parameters (6)
  • G3 non-trivial denominator threshold = 0.02
    max(|Δfmt|,|Δsem|) ≥ 0.02 before CSR is reported; chosen as an engineering floor, explicitly not equal to 2-SE of a p=0.5 n=200 proportion (≈0.07).
  • CSR epsilon floor = 0.01
    ε = 0.01 inside CSR = |Δattr| / max(|Δfmt|,|Δsem|,ε) to keep the ratio finite; engineering choice, not a statistical threshold.
  • G1 silent-no-op rate cap = 20%
    Silent-no-op rate against scorer-consumed prompt must be <20%; pragmatic cap motivated by observed 0–83% range.
  • G1 parseability floor = 0.95
    Parseability ≥ 0.95 for any regex-based scorer; engineering disclosure threshold.
  • G2 above-baseline margin = ≥2 SE
    Unperturbed score must exceed trivial baseline by ≥2 SE; standard-ish but still a chosen gate width.
  • Panel size and seed = 200 items, seed 42
    N=200 items per benchmark, single seed=42, single template; all headline counts depend on this fixed design.
assumptions (5)
  • ad hoc to paper A failure is in scope for the taxonomy only if it is silent (invisible in reported numbers), realized in this audit, and gateable by a concrete disclosure (C1–C3).
    §2 states these three selection criteria in priority order; they define F1–F5 and explicitly exclude F6–F10 that fail C2.
  • domain assumption Perturbation-based construct-validity audits are a common and governance-relevant form of documented evaluation evidence.
    Introduction frames EU AI Act Art. 55, NIST AI RMF, and GB/T 45654 as demanding documented evidence; the paper studies the perturbation family because it is cheap and directly emitted as such evidence.
  • ad hoc to paper Unsigned magnitude comparison |Δattr| > max(|Δfmt|,|Δsem|) is a necessary (not sufficient) condition for construct sensitivity under CSR.
    §3 defines CSR on unsigned family deltas and explicitly defers signed-effect validation; confirmatory-selective therefore certifies contrast magnitude only.
  • domain assumption Safety benchmarks split into diagnostic vs invariance archetypes under construct flips, and a single ratio metric cannot be interpreted without archetype disclosure.
    F5 / G5; BBQ is treated as invariance, TruthfulQA/ToxiGen as diagnostic; this reading is load-bearing for the F5-ineligible bucket.
  • domain assumption Item-level paired bootstrap is the correct uncertainty estimator; independent resampling of originals and perturbations is prohibited.
    G4 / F4; standard bootstrap practice (Efron & Tibshirani) applied to paired item differences.
invented entities (5)
  • F1–F5 pipeline failure taxonomy (silent no-ops, regex artefacts, non-faithful scoring F3a–c, broken bootstrap pairing, metric-archetype mismatch)
    purpose: Name the silent, realizable, gateable failure modes that can manufacture audit conclusions.
    Introduced in §2 as a deliberately non-exhaustive starting list selected by C1–C3; independent evidence is only the authors' own realizations.
  • G1–G6 six-point due-diligence gate with hierarchical status precedence
    purpose: Withholding and disclosure protocol that maps cells to ineligible / scorer-unvalidated / failed G2–G4 / exploratory / confirmatory.
    Table 1 and §3 algorithm; G5 and G6 remain PROPOSED, so confirmatory is unreachable by construction in this submission.
  • Contrast Selectivity Ratio (CSR)
    purpose: Cell statistic |Δattr| / max(|Δfmt|,|Δsem|,ε) used as the numerical object of the gate.
    Defined in §3; renamed from 'Construct Sensitivity Ratio' after E3 human audit was de-scoped (prereg fallback). Magnitude-only by design.
  • Four non-confirmatory status buckets (ineligible, scorer-unvalidated, failed G2–G4, exploratory)
    purpose: Force assurance reports to separate different consumer actions instead of collapsing to one inconclusive verdict.
    §3 status-assignment algorithm and Table 3; mechanical mapping from raw cell_validity.csv.
  • Minimal self-audit chronology template (Box 1)
    purpose: Require eight fields per failure so repair is distinguishable from never-caught bugs.
    §6 Box 1; worked instance in Appendix J.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits." pith.science (2026). https://pith.science/paper/Y3W2ULBW

@misc{pith2026260702586,
  author       = {Pith},
  title        = {Pith review of: Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3W2ULBW}},
  note         = {Machine review of arXiv:2607.02586}
}
read the original abstract

Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence. We argue the audits are themselves fragile: their conclusions can be silently manufactured by implementation details that readers cannot see in the reported numbers. We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models. Under a unified six-point due-diligence gate, every cell lands in a non-confirmatory bucket, and no cell reaches confirmatory. The evidence here is a single two-model, five-benchmark case study, and F1--F5 is an illustrative, deliberately non-exhaustive starting taxonomy -- not a comprehensive partition of audit failures. We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence, and not as a route to benchmark-validity verdicts.

Figures

Figures reproduced from arXiv: 2607.02586 by the authors.

Figure 1
Figure 1. Overview of our benchmark-audit pipeline; a roadmap figure—the gate definitions and the status-assignment algorithm are formalised in §3 ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Canonical-pipeline CSR per cell with 1000-resample paired-bootstrap 95% CI. Colour encodes the gate verdict (§3). Mistral × XSTest fails G3 (max(|∆fmt|, |∆sem|) = 0.012 < 0.02); no cell reaches confirmatory-selective under the full G1–G6 gate because G5, G6 remain PROPOSED. CSR values for ineligible and scorer-unvalidated cells are shown descriptively only and are not interpretable as benchmark-validity evidence (se… view at source ↗
Figure 3
Figure 3. Legacy-pipeline CSR (x-axis) versus canonical-pipeline CSR (y-axis) per cell, on symlog axes (linear in [−1, 1], log outside) so both the near-zero BBQ cells and the Mistral × XSTest outlier at CSR ≈ 85 are readable on one panel. Points on the dotted identity line are unchanged by repair; points far from it reflect the joint effect of F1–F3 fixes. Seven of ten cells change verdict (selective ↔ non-selective or denom… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Per-cell |∆fmt|, |∆sem|, |∆attr| under the canonical pipeline. Cells where |∆fmt| or |∆sem| < 0.02 drive the denominator below the ε floor and trigger G3. |∆sem| = 0 on CrowS-Pairs reflects F1-type silent no-ops on the question field, which the PLL scorer does not read…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 5 linked inside Pith

  1. [1]

    URL https: //arxiv.org/abs/2602.19555

    doi: 10.48550/arXiv.2602.19555. URL https: //arxiv.org/abs/2602.19555. Kiela, D., Bartolo, M., Nie, Y ., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stene- torp, P., Jia, R., Bansal, M., Potts, C., and Williams, A. Dynabench: Rethinking benchmarking in NLP. InPro- ceedings o...

    arXiv doi:10.48550/arxiv.2602.19555 2021

  2. [2]

    naacl-main.324

    URL https://aclanthology.org/2021. naacl-main.324. Lalor, J. P., Wu, H., and Yu, H. Learning latent parameters without human response patterns: Item response theory with artificial crowds. InProceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (...

    arXiv doi:10.18653/v1/2025 2021

  3. [3]

    acl-long.229

    URL https://aclanthology.org/2022. acl-long.229. Luo, H., Deng, Z., Chen, R., and Liu, Z. FAIntbench: A holistic and precise benchmark for bias evaluation in text-to-image models.arXiv preprint arXiv:2405.17814, 2024a. URL https://arxiv.org/abs/2405. 17814. Luo, H., Huang, H., Deng, Z., Li, X., Wang, H., Jin, Y ., Liu, Y ., Xu, W., and Liu, Z. BIGbench: A...

    arXiv doi:10.48550/arxiv.2604.11934 2022

  4. [4]

    findings-acl.165

    URL https://aclanthology.org/2022. findings-acl.165. Qian, P., Wang, S., Wang, X., Chen, Y ., Xu, W., Yu, Q., Lin, S., Zhang, S., You, J., and Wei, X. Relevant is not war- ranted: Evidence-force calibration for cited RAG, 2026. URLhttps://arxiv.org/abs/2605.28044. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., and Hanna, A. AI and the everything in...

    arXiv 2022

  5. [5]

    acl-main.442

    URL https://aclanthology.org/2020. acl-main.442. R¨ottger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. XSTest: A test suite for identify- ing exaggerated safety behaviours in large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T...

    doi:10.18653/v1/2024.naacl-long.301 2020

  6. [6]

    State Administration for Market Regulation and Stan- dardization Administration of China

    URL https://openreview.net/forum? id=RIu5lyNXjT. State Administration for Market Regulation and Stan- dardization Administration of China. GB/T 45654– 2025, cybersecurity technology—basic security requirements for generative artificial intelligence service. National Standard of the People’s Repub- lic of China, 2025. URL https://openstd. samr.gov.cn/bzgk/...

    arXiv 2025

  7. [7]

    Two models, five benchmarks, 200 items per cell, one seed, one prompt template.Our 10-cell grid (Qwen-2.5-7B and Mistral-7B-v0.3 × five benchmarks) cannot support any claim that a class of 7–8B open-weight models exhibits any pattern

  8. [8]

    Post-prereg exploratory status.Our preregistration (§K) was committed before we discovered failures F3b and F3c; analyses after those discoveries are formally exploratory

Show all 22 references
  1. [9]

    Their numbers illustrate the two failure classes but are not benchmark-validity claims

    Structurally-ineligible cells.CrowS-Pairs is archetype-ambiguous under our PLL + demographic-swap configuration and additionally hits F1 scorer-path mismatch; BBQ is an invariance benchmark penalised by our diagnostic ratio metric (F5). Their numbers illustrate the two failure...

  2. [10]

    Neither scorer is a human-judged label; both cells are reported in thescorer-unvalidatedbucket

    Unvalidated scorers for ToxiGen and XSTest.ToxiGen scoring uses targeted first-token log-probabilities over a fixed token set (after the F3c repair); XSTest refusal classification uses a ten-pattern regex over generations that we did not separately audit for false-negatives ag...

  3. [11]

    CSR is a contrastive-magnitude metric, not a signed-direction metric.CSRuses unsigned family-deltas, so a paired-bootstrap CI wholly above 1 only certifies that |∆attr| exceeds surface-family magnitudes. It doesnotcertify that attribute-flip responses are in the construct-cons...

  4. [12]

    CrowS- Pairs PLL on paired sentences (Salazar et al., 2020) and BBQ option log-probabilities split by context condition are reference-faithful

    Canonical scorers are audit-hardened, not reference-faithful across the board.TruthfulQA MC1 with deterministic per-item shuffle, ToxiGen targeted-token first-token logprobs, and XSTest’s 10-pattern refusal regex are audit-robust sub- stitutes rather than reference-faithful im...

    2020

  5. [13]

    Canonical

    The six-point checklist is a proposed tool, not a standard.It does not validate thresholds against external reference data, does not assign institutional ownership, and does not include anti-gaming provisions. Three further benchmark-specific limitations (shared-prefix MCQ tok...

    2020

  6. [14]

    Fix: parametriseformat mcq prompt(label style)

    F1a (fixed): format change labels wrote item[‘label style’]; prompt builder did not read it. Fix: parametriseformat mcq prompt(label style)

  7. [15]

    Fix: parametrise the canonical builder

    F1b (fixed): Fix landed only in evaluate.py; scoring canonical.py:: format mcq prompt remained hard-coded. Fix: parametrise the canonical builder

  8. [16]

    Fix: addparaphrase template,synonym substitution

    F1c (fixed): Rule-based semantic sentence restructure had no-op rates of 57–93% due to trigger-string mismatch on Q&A benchmarks. Fix: addparaphrase template,synonym substitution

  9. [17]

    Fix: canonical option-logprob scoring sets parseability to 1.0; legacy numbers are withdrawn

    F2a (partially fixed): Qwen × TruthfulQA legacy parseability 0.44. Fix: canonical option-logprob scoring sets parseability to 1.0; legacy numbers are withdrawn

  10. [18]

    Canonical pipeline switches to PLL; CrowS legacy numbers are withdrawn

    F3a (disclosed, not repaired in data): CrowS-Pairs legacy accuracy inverts fairness interpretation. Canonical pipeline switches to PLL; CrowS legacy numbers are withdrawn. 6.F3b (fixed): TruthfulQA MC2 correct-first ordering artefact. Fix: MC1 + per-item shuffle

  11. [19]

    Fix: targeted-token first-token log-probabilities

    F3c (fixed, self-introduced): Top-k truncation caused majority-class predictions on ToxiGen. Fix: targeted-token first-token log-probabilities

  12. [20]

    Fix: item-level paired bootstrap

    F4 (fixed): Broken bootstrap pairing. Fix: item-level paired bootstrap. Out-of-panel legacy-Llama at CSR≈9.5 showed CI width 26.4 under unpaired resampling

  13. [21]

    Not fixed; G3 gate flags denominator-dominated cells

    F5 (disclosed): BBQ / CrowS archetype mismatch under CSR. Not fixed; G3 gate flags denominator-dominated cells

  14. [22]

    (A)”, “(B)

    Shared-prefix collapse on MCQ letter tokens(UNFIXED,ACKNOWLEDGED): first token of “ (A)”, “(B)” can be identical. 11.TEXTCLFadd instructioncontent contamination on ToxiGen(UNFIXED,ACKNOWLEDGED). 17 Auditing the Audit: Five Failure Modes Containment of unfixed issues.The remain...

    2026

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.