Pith. sign in

REVIEW 3 major objections 5 minor 16 references

The metasignal package computes all 17 standard metacognitive measures in one call and reproduces the reference MATLAB pipeline to numerical precision for non-model-based measures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:46 UTC pith:4O4LPBGO

load-bearing objection A genuinely useful integration of the full metacognition battery with strong internal validation, though the external anchor to the Rahnev benchmark is thinner than the abstract implies — still deserves peer review and likely acceptance. the 3 major comments →

arxiv 2607.29093 v1 pith:4O4LPBGO submitted 2026-07-31 q-bio.NC cs.AIstat.AP

metasignal: A Python Package for Comprehensive Metacognitive Analysis and Decision-Making

Saurabh Ranjan , Mukesh Makwana , Konstantina Sokratous , Brian Odegaard This is my paper
classification q-bio.NC cs.AIstat.AP
keywords metacognitionsignal detection theorymeta-d'confidenceType-2 measuresPython packagereproducible researchdecision-making
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents metasignal, an open-source Python package that implements the full battery of 17 metacognitive measures compared in the 2025 benchmark assessment, plus sensitivity d', criterion c, and mean confidence. The central claim is that a single function, compute_all_measures, computes the entire set from trial-level arrays, and that the non-model-based measures reproduce the reference MATLAB pipeline's values to numerical precision across six benchmark datasets. Validation reports perfect correlations (r=1.000) with published values for task-performance, metacognitive-bias, and response-bias profiles, and near-perfect alignment on test–retest reliability. If true, researchers can substitute the Python package for the MATLAB pipeline without losing comparability to published norms. The package is designed to remove the fragmentation that forced researchers to use separate codebases for different measures.

Core claim

The paper's discovery is that one implementation can faithfully span the full set of metacognitive measures used in the field. metasignal converts trial-level stimulus, response, and confidence arrays into Type-2 count arrays, then computes all 17 measures — three meta-d' family estimates, four nonparametric Type-2 measures plus their eight SDT-normalized forms, and two model-based measures — through a single entry point. Validation against six datasets distributed with the benchmark pipeline shows that the 18 non-model-based measures agree with the MATLAB reference to numerical precision at the subject level (maximum systematic bias below 1.5×10−3, per-analysis correlations of 1.000), and t

What carries the argument

The central object is compute_all_measures, a single function that takes four trial-level arrays (stimulus category, response, integer confidence rating, and number of rating levels) and returns a fixed 26-element vector: d', c, mean confidence, the 17 metacognitive measures, and six meta-d' fit diagnostics. The computation relies on converting trials into Type-2 response-count matrices, using equal-variance SDT for Type-1 quantities, direct counts for nonparametric Type-2 measures, maximum-likelihood estimation for meta-d' (with the same golden-section search and precomputed integral-table interpolation as the MATLAB reference), and the lognormal confidence-noise model for meta-noise. The p

Load-bearing premise

The validation treats the MATLAB reference implementation and the 2025 benchmark's published values as ground truth; if that reference contains errors, the Python package inherits them because its goal is exact fidelity to that reference, not independent correctness of the measures.

What would settle it

Run metasignal and an independently coded implementation of the benchmark paper's measure definitions (written without reference to the MATLAB or Python code) on the same trial-level data from the six validation datasets; any systematic deviation above the reported tolerance (e.g., systematic bias above 1.5×10−3, or a correlation below 1.000) for the non-model-based measures would falsify the claim of numerical precision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any lab using the benchmark's measures can now compute all of them with one Python call and no proprietary software.
  • Results computed with metasignal remain directly comparable to the published 2025 benchmark norms, because the non-model-based measures match the reference pipeline to numerical precision.
  • The bundled validation workflow makes it possible to audit exactly where residual differences come from (optimizer choice, flat likelihood surfaces, sparse bins) instead of treating them as hidden failures.
  • The single 26-value output vector with stable ordering enables straightforward batch and group analyses, bootstrap intervals, and permutation tests to be standardized across studies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The package's focus on fidelity to one reference implementation means it is a conservative tool: it inherits the benchmark's definitions and quirks, so it will not alert users if those definitions are later revised.
  • The documented per-participant differences in meta-d' estimates (up to ~0.14) between optimizers suggest that when reporting meta-d' researchers may want to state which optimizer was used, since the difference is larger than many effect sizes of interest.
  • Because the package exposes the validation data and reproducibility scripts, it could serve as a baseline for future attempts to standardize metacognitive measurement beyond the current benchmark, for example by adding generative-model ground-truth checks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents metasignal, an open-source Python package that implements the 17 metacognitive measures benchmarked by Rahnev (2025), together with d', response criterion c, mean confidence, and a set of auxiliary inferential tools (bootstrap, permutation tests, optional Bayesian and information-theoretic extensions). The authors validate the package against (i) values reported in Rahnev (2025), (ii) a MATLAB pipeline bundled with the repository, and (iii) Python-generated arrays, reporting perfect correlations for non-model-based measures, matched 19 supplementary t-tests, and numerical agreement to within stated tolerances. They claim that researchers can adopt metasignal in place of the MATLAB pipeline without sacrificing comparability to the published benchmark, while disclosing residual differences in model-based fits and an unmatched precision protocol.

Significance. If the numerical fidelity claim is correct, metasignal fills a genuine gap: no single native-Python package previously implemented the full set of 17 metacognitive measures compared in Rahnev (2025). The manuscript's strengths include an unusually thorough validation workflow (reproducible scripts, JSON/CSV outputs, 19 matched t-tests, identity plots), clear documentation of residual differences, and a standalone command-line interface. The contribution is practical and likely to be adopted by labs that need an accessible, MATLAB-free implementation. The main weakness is that the external anchor of the validation is not fully independent: the bundled MATLAB pipeline is itself derived by the same authors from the original OSF code, and the reported comparison to published values relies primarily on correlation coefficients, which are insensitive to systematic offsets. These gaps are fixable and do not undermine the internal correctness demonstrated for the non-model-based measures, but they bear directly on the central claim of 'comparability' with the published benchmark.

major comments (3)
  1. [§5, Code Availability] The external validation anchor is not fully independent. The MATLAB reference pipeline is bundled in the repository and is described as 'derived from' the original OSF code, and both the Python and bundled MATLAB implementations were produced by the same authors (Code Availability). Agreement between Python and this bundled pipeline therefore does not establish that either matches the original OSF implementation. Since §9 claims 'without sacrificing comparability to the published benchmark,' the validation should include a direct comparison against the original OSF MATLAB code or its recorded outputs, using the same error metrics applied in §4.2. Without this, a systematic transcription error common to both implementations cannot be excluded.
  2. [§5, Fig. 2] The comparison to values reported in Rahnev (2025) is presented in Figure 2 only as Pearson correlations (r = 1.000 for the main panels). Pearson correlation is invariant to affine transformations, so it cannot detect a constant offset or multiplicative scaling in the Python values relative to the published values. Although §4.2 states that MAE, RMSE, signed bias, and maximum absolute error were examined, the results section does not report an absolute agreement metric against the published values; the 'maximum absolute difference of 10^-2' in §5 appears to concern MATLAB–Python agreement. To support the comparability claim, please report per-measure MAE and maximum absolute error against the published values (or clearly point to the corresponding table in ANALYSIS_REPORT.md), and state whether any subjects or measures were excluded from those calculations.
  3. [§8.1, §9] The Limitations section states that precision was recomputed under a 'bin-instance cap' and that the resulting precision figures are 'illustrative rather than an exact protocol match to Rahnev (2025).' Yet §9 concludes that researchers can adopt metasignal 'without sacrificing comparability to the published benchmark.' These statements are in tension. If the reliability/precision protocol is not matched exactly, the conclusion should explicitly restrict the comparability claim to the core measure profiles and supplementary t-tests, or the exact protocol should be implemented. As written, the final claim overstates the demonstrated scope of 'comparability' with the full Rahnev (2025) benchmark.
minor comments (5)
  1. [Title (compiled text)] The compiled title lacks spaces: 'A PYTHONPACKAGE FORCOMPREHENSIVE METACOGNITIVEANALYSIS ANDDECISION-MAKING' should be 'A Python Package for Comprehensive Metacognitive Analysis and Decision-Making'. The keywords line similarly reads 'KeywordsPython' in the compiled text.
  2. [§6.3] The reproducibility check using np.random.default_rng(2025) is useful, but note that NumPy's random stream may change across versions; consider pinning the NumPy version in the reproducibility note.
  3. [Figure 2 caption] The caption says 'MAT and PY report the Pearson correlation between each replication and the published values', but panel (d) reports r = 0.980/0.979, not 1.000. Consider clarifying that panel (d) is the test–retest reliability panel and that the lower r reflects the different protocol, not a validation failure.
  4. [References] In the Meyen et al. reference, 'V olker' should be 'Volker'. Also check that all author names and journal titles are rendered with consistent spacing after the journal-name compression.
  5. [§4.2] The sentence listing agreement metrics (Pearson correlation, MAE, RMSE, signed bias, max error, missing-value agreement) would be easier to interpret if the paper stated in §5 which of these are reported in the main text and which are only in ANALYSIS_REPORT.md.

Circularity Check

0 steps flagged

No circularity: validation-based package paper, not a derivation; no load-bearing self-citations.

full rationale

This is a software/validation paper rather than a derivation of new scientific results. The package implements published measures (Rahnev 2025; Maniscalco and Lau 2012) and validates them against external published values, an independent Python implementation, and a bundled MATLAB pipeline. No equation or result in the paper is shown to reduce to its own inputs. The central claim—that metasignal reproduces the reference MATLAB pipeline to numerical precision—is an explicitly stated target, and the agreement is tested empirically rather than assumed by construction. The closest concern is that the bundled MATLAB reference and the Python package share provenance, and the original OSF code is not directly compared; this is a limitation on validation independence, not a circular step. Self-citations (Lee et al., 2025; Rahnev et al., 2022) are motivational only and do not support the central claim. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 0 invented entities

The central claim (correct implementation of 17 known measures) rests on domain assumptions inherited from SDT and the reference benchmark, not on new free parameters or invented constructs. The only hand-chosen number is a computational bin cap in the precision validation, which the paper explicitly marks as illustrative.

free parameters (1)
  • bin-instance cap = unspecified (chosen by hand)
    Introduced in §8.1 to make precision recomputation tractable; it increases sampling noise and can invert signs for Gamma-Ratio, so precision figures are described as illustrative rather than exact protocol matches.
axioms (6)
  • domain assumption Type-1 sensitivity and criterion are computed under the equal-variance SDT model (Green and Swets, 1966).
    Invoked in §4.1; all d'/c outputs and the SDT-normalized measures depend on this model choice.
  • domain assumption The 17 measures defined by Rahnev (2025) are the authoritative benchmark for metacognitive measurement.
    The package's purpose and validation are defined against this single benchmark (§1, §4.2); if the benchmark's measure definitions are flawed, the package inherits the flaw.
  • domain assumption Meta-noise is estimated with the lognormal confidence-noise model used by the MATLAB benchmark.
    §4.1; the meta-noise estimator is not derived in the paper but ported from the benchmark's model.
  • standard math Maximum-likelihood estimation is appropriate for meta-d' fitting.
    §4.1; follows Maniscalco & Lau (2012); standard statistical approach, though bounded optimizers can yield small subject-level differences.
  • domain assumption For the optional hierarchical Bayesian subpackage, Stan's default priors and sampling are valid.
    §2/§3 mention sdtbayes with Stan-based estimation; priors/samplers are not detailed in the paper.
  • domain assumption Information-theoretic metacognition definitions in itmc follow Dayan (2023) and Meyen et al. (2025).
    §2; itmc is experimental and not part of the core benchmark.

pith-pipeline@v1.3.0-daily-deepseek · 7136 in / 12038 out tokens · 114448 ms · 2026-08-03T13:46:48.217876+00:00 · methodology

0 comments
read the original abstract

Metasignal is an open-source Python package for signal detection theory (SDT) and metacognitive measurement. It implements the 17 metacognitive measures evaluated by Rahnev (2025), together with the reference variables d' (perceptual sensitivity), response criterion c (response bias), and mean confidence. The 17 measures comprise three meta-d' family estimates, meta-d', M-ratio, and M-difference; four nonparametric Type-2 measures, the Type-2 area under the receiver-operating-characteristic curve (AUC2), Gamma, Phi, and delta confidence, together with their eight SDT-normalized ratio and difference forms; and two model-based measures, meta-noise and meta-uncertainty. A single function computes the complete set from trial-level stimulus, response, and confidence arrays. `metasignal` currently supports binary (two-alternative) discrimination tasks, in which each trial's stimulus and response are coded with exactly two categories. The package also provides a command-line interface, group summaries, bootstrap confidence intervals, permutation tests, optional hierarchical Bayesian models, and information-theoretic measures. `metasignal` unifies these measures in a single platform to encourage broader metacognition research and adoption in decision-making studies.

Figures

Figures reproduced from arXiv: 2607.29093 by Brian Odegaard, Konstantina Sokratous, Mukesh Makwana, Saurabh Ranjan.

Figure 1
Figure 1. Figure 1: Architecture of metasignal. Trial-level stimulus, response, confidence, and rating-scale arrays (stim, resp, conf, n_ratings) enter the stable stdpy core, which computes Type-1 SDT quantities (d ′ , criterion c) and all 17 metacognitive measures; compute_all_measures packages these into a single 26-element array (20 measures plus 6 meta-d ′ fit diagnostics). This output feeds the analysis layer (bootstrap … view at source ↗
Figure 2
Figure 2. Figure 2: Cross-implementation validation against Rahnev (2025). In each panel, published values (x-axis, “Paper”) are plotted against replication values (y-axis) from the MATLAB pipeline (blue squares) and the Python metasignal package (red triangles); the dashed line marks exact identity. Published values in all five panels are the per-measure averages reported in Rahnev (2025)’s [PITH_FULL_IMAGE:figures/full_fig… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 8 canonical work pages

  1. [3]

    Marie Hoven, Maël Lebreton, Jan B

    doi: 10.1098/ rstb.2011.0416. Marie Hoven, Maël Lebreton, Jan B. Engelmann, Damiaan Denys, Judy Luigjes, and Ruth J. van Holst. Abnormalities of confidence in psychiatry: An overview and future perspectives.Translational Psychiatry, 9(1):268,

    arXiv 2011

  2. [6]

    Maxime Griot, Coralie Hemptinne, Jean Vanderdonckt, and Demet Yuksel

    doi: 10.1177/09637214251391158. Maxime Griot, Coralie Hemptinne, Jean Vanderdonckt, and Demet Yuksel. Large language models lack essen- tial metacognition for reliable medical reasoning.Nature Communications, 16(1):642,

    work page doi:10.1177/09637214251391158

  3. [7]

    Dobromir Rahnev, Tarryn Balsdon, Lucie Charles, Vincent de Gardelle, Rachel Denison, Kobe Desender, Nathan Faivre, Elisa Filevich, Stephen M

    doi: 10.1093/pnasnexus/pgaf133. Dobromir Rahnev, Tarryn Balsdon, Lucie Charles, Vincent de Gardelle, Rachel Denison, Kobe Desender, Nathan Faivre, Elisa Filevich, Stephen M. Fleming, Janneke Jehee, Hakwan Lau, Alan L. F. Lee, Shannon M. Locke, Pascal Mamassian, Brian Odegaard, Megan Peters, Gabriel Reyes, Marion Rouault, Jerome Sackur, Jason Samaha, Clair...

    work page doi:10.1093/pnasnexus/pgaf133

  4. [9]

    Brian Maniscalco and Hakwan Lau

    doi: 10.1016/j.concog.2011.09.021. Brian Maniscalco and Hakwan Lau. A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings.http://www.columbia.edu/~bsm2105/type2sdt/,

    doi:10.1016/j.concog.2011.09.021 2011

  5. [12]

    Peter Dayan

    doi: 10.21105/joss.06966. Peter Dayan. Metacognitive information theory.Open Mind, 7:392–411,

    work page doi:10.21105/joss.06966

  6. [14]

    Charles R

    doi: 10.1162/OPMI.a.40. Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Géra...

    work page doi:10.1162/opmi.a.40

  7. [16]

    doi: 10.1038/s41592-019-0686-2. David M. Green and John A. Swets.Signal Detection Theory and Psychophysics. Wiley, New York,

    doi:10.1038/s41592-019-0686-2

  8. [2008]

    Mark Steyvers and Megan A

    doi: 10.1037/0033-295X.115.2.502. Mark Steyvers and Megan A. K. Peters. Metacognition and uncertainty communication in humans and large language models.Current Directions in Psychological Science,

    doi:10.1037/0033-295x.115.2.502

  9. [2012]

    Nick Yeung and Christopher Summerfield

    doi: 10.1098/rstb.2011.0417. Nick Yeung and Christopher Summerfield. Metacognition in human decision-making: confidence and error monitoring. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1594):1310–1321,

    arXiv 2011

  10. [2017]

    Manuel Rausch, Sascha Meyen, and Sebastian Hellmann

    doi: 10.1093/nc/nix007. Manuel Rausch, Sascha Meyen, and Sebastian Hellmann. statConfR: An R package for static models of decision confidence and metacognition.Journal of Open Source Software, 10(106):6966,

    work page doi:10.1093/nc/nix007

  11. [2019]

    doi: 10.1038/s41398-019-0602-7. Don A. Moore and Paul J. Healy. The trouble with overconfidence.Psychological Review, 115(2):502–517,

    work page doi:10.1038/s41398-019-0602-7

  12. [2020]

    Pauli Virtanen, Ralf Gommers, Travis E

    doi: 10.1038/s41586-020-2649-2. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ...

    doi:10.1038/s41586-020-2649-2

  13. [2021]

    Stephen M. Fleming. Hmeta-d: hierarchical bayesian estimation of metacognitive efficiency from confidence ratings. Neuroscience of Consciousness, 2017(1):nix007,

    2017

  14. [2022]

    Brian Maniscalco and Hakwan Lau

    doi: 10.1177/17456916221075615. Brian Maniscalco and Hakwan Lau. A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings.Consciousness and Cognition, 21(1):422–430,

    work page doi:10.1177/17456916221075615

  15. [2023]

    Sascha Meyen, Felix Göppert, Constantin Schrenk, Ulrike von Luxburg, and V olker H

    doi: 10.1162/opmi_a_00091. Sascha Meyen, Felix Göppert, Constantin Schrenk, Ulrike von Luxburg, and V olker H. Franz. Information-theoretic measures of metacognition: Bounds and relation to group performance.Open Mind, 9:1728–1762,

    work page doi:10.1162/opmi_a_00091

  16. [2025]

    Stephen M

    doi: 10.1038/s41467-025-56117-0. Stephen M. Fleming and Raymond J. Dolan. The neural basis of metacognitive ability.Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1594):1338–1349,

    doi:10.1038/s41467-025-56117-0