How far to trust a Pith review.
Pith measured its own reviews on 4 Oct 2026 and prints what it found here, the misses first.
Judged by two other models, not by people, against Pith's own copy of each paper's text.
18 of 105
errors that authors later admitted were found by Pith's review of the earlier version
measured 4 Oct 2026
76 in 100
sampled criticisms were judged accurate by two other models reading Pith's copy of the paper's text (113 of 149)
measured 4 Oct 2026
Errors the authors later admitted
Papers on Pith carry a review written by a model: major comments, minor comments and a recommendation. On 4 Oct 2026 we checked the reviews two ways, with two independent model families (gpt-5.6-luna and grok-4.7) that never saw each other's answers. The first way is here; the second is in the next section.
Sometimes authors withdraw a paper, or post a new version that says what was wrong with the old one. If Pith had reviewed the old version, its review either raised that error or it did not. The judges agreed on 105 such papers. Each mark below is one of them.
The review named the error itself in 17.1% of them (95% interval 11% to 26%) and found it or something closely related in 44.8% (36% to 54%). measured 4 Oct 2026 · judged by gpt-5.6-luna and grok-4.7
How the papers were chosen
We took arXiv's own record of every paper updated between 1 September 2025 and 2 October 2026 that has more than one version: 200,630 papers. Pith had reviewed an earlier version of 35,326 of them. In 201 of those, the authors withdrew the paper or posted a version whose comment says what was wrong and where, such as a gap in the proof of a theorem or a bug in the code behind a table. We judged 150 of these, drawn at random. Both judges read the review and the authors' own statement of the error and said whether the review found it, raised something closely related, or missed it. They agreed on 105 papers.
| Authors' statement | Agreed on | Found | Close | Missed |
|---|---|---|---|---|
| Withdrew or retracted the paper | 25 | 6 | 12 | 7 |
| Posted a corrected version | 80 | 12 | 17 | 51 |
Checks on the check: given a review of a different paper, the judges credited it with the error 1 of 30 times. When we wrote the authors' own statement into the paper's review as one more comment, gpt-5.6-luna recognised it 20 of 25 times and grok-4.7 recognised it 10 of 25 times. A review counts as finding an error only when both judges say so, so the found rate above is a floor.
Are a review's criticisms true of the paper?
By lane: how often a lane's major criticisms are right
A lane is the model that wrote a review's first pass, with the kind of text the review was given. Each lane is measured by itself: a sample of its published reviews, one major criticism from each, judged by models of other families that read the paper's LaTeX source, or its pages as images when there is no source. They never read Pith's text copy.
| Lane: model, and what it read | Reviews, last 14 days | Criticisms judged accurate | 95% interval | Judged, and agreed on | Judged by, reading what | When |
|---|---|---|---|---|---|---|
| selfhosted/deepseek-v4.1-flash, not recorded | 45,063 | Not yet measured. | ||||
| deepseek/deepseek-v4-flash, not recorded | 16,266 | Not yet measured. | ||||
| grok-4.5, not recorded | 57 | Not yet measured. | ||||
| grok-4.5, the abstract only | 2 | Not yet measured. | ||||
The audit by lane is built and has not been run yet, so no lane has a figure. Until it has, the only measured figure for reviews is the one below, which covers every lane together as they stood on 4 Oct 2026.
All lanes together, 4 Oct 2026: are a review's criticisms true of the paper?
We drew 200 reviews at random from the published record and took one major comment from each. Both families read the same paper text the reviewer read and said whether the comment is accurate, inaccurate, or impossible to settle from the paper. That text is Pith's own copy: text extracted from the PDF, the first 90,000 characters. Extraction flattens superscripts and fractions, and a long paper is cut. A criticism that is wrong because of the copy (it says a section is missing that lies past the cut, or reads a product as a power) was checked against the same copy and counts here as accurate. So this figure measures agreement with Pith's copy of the paper, not with the paper. The reviews sampled were written by several models over several months, some from the abstract alone, and the sample's rows do not record which model wrote each review, so the figure cannot be split by lane. They gave the same answer on 151 comments, and agreed that 149 could be settled. Each mark below is one of those.
75.8% were judged accurate (95% interval 68% to 82%). Left out of the marks: 2 could not be settled from the paper alone, and 49 where the two judges disagreed. measured 4 Oct 2026 · judged by gpt-5.6-luna and grok-4.7
Checks on the check: a major comment taken from a review of a different paper was judged accurate 0 of 30 times. Given an error the authors themselves later admitted, checked against the version that had it, the families confirmed it in 1 of 21 papers and called it inaccurate in 12. They confirm what can be seen on the page far more readily than a subtle technical error, so a comment they call inaccurate may still be right. Asked the same question twice, gpt-5.6-luna gave the same answer 100% and grok-4.7 gave the same answer 83% of the time.
The criticisms judged wrong, case by case
The audit kept 25 of the 36 criticisms that both judges called inaccurate. They are printed here in full, so a reader can check the judges as well as the reviews: what Pith's review said, why each judge called it wrong, and the words of the paper the judges pointed to.
-
Case 1 of 25 · arXiv:2604.11416 · version 1Exact Certification of Neural Networks and Partition Aggregation Ensembles against Label Poisoning
The paper says
For a finite-width neural network however, where w denotes the smallest layer width of the network, the output difference of the network to the SVM is bounded by O(ln w/√w) with probability p=1−exp(−Ω(w)).
Pith's review said
Abstract and ScaLabelCert section: the claim that ScaLabelCert yields an 'exact' certificate rests on the NTK equivalence for 'sufficiently wide' networks. No rigorous error bound is supplied quantifying the O(1/width) finite-width corrections to the predictor; because certification solves a worst-case optimization over adversarial label vectors, even small deviations can enlarge the set of successful flips and invalidate the kernel-derived radius for any concrete finite-width network. This dire… (the audit kept the first 500 characters)
Place named by the review: Abstract and ScaLabelCert derivation
Why the judges called it wrong
gpt-5.6-luna: The paper does provide a finite-width error bound, although it qualifies the certificate as asymptotically exact rather than exact for concrete finite-width networks.
grok-4.7: Section 5 does supply a finite-width output-error bound and explicitly qualifies the certificates as asymptotically exact, so the claim that no such bound is given is contradicted.
-
Case 2 of 25 · arXiv:2508.06249 · version 3In-Training Defenses against Emergent Misalignment in Language Models
The paper says
We investigate five training regularization interventions: (i) KL-divergence regularization... (v) inoculation prompting. ... Finally, we use λKL = 0.1, λLDIFS = 1.0, α = 5.0 ... and 5% additional benign data
Pith's review said
The five interventions are only named, not defined. For reproducibility and for the ranking to be meaningful, the paper must specify the KL-regularization coefficient, the layer(s) used for l2 feature-space distance, how the evil-persona steering vector is obtained, the interleaving ratio and selection criterion (including the direction of the perplexity gap and its normalization), and the inoculation prompting template. Without these, 'best results' is ambiguous.
Place named by the review: Abstract (intervention specificity)
Why the judges called it wrong
gpt-5.6-luna: The paper defines all five interventions in Section 3, specifies the KL and LDIFS coefficients, persona-vector construction and layer, interleaving selection and ratios, and gives the inoculation prompt template.
grok-4.7: The paper defines all five interventions, including the KL and LDIFS coefficients, every-5th-layer feature distance, persona-vector construction, interleaving ratio and normalized perplexity-gap criterion, and the inoculation prompt template.
-
Case 3 of 25 · arXiv:2604.22590 · version 1Strong solutions and stability for a thin-film equation of shear-thinning fluids with contact line in partial wetting
The paper says
In §4 ... this requires suitable estimates on the right-hand side terms ... where f=N(u) ... Proposition 4.1 ... we have the inequality
Pith's review said
The control of higher-order nonlinear terms in the time-discretization scheme and the precise compactness arguments used to pass to the limit while preserving the perturbation structure are load-bearing for the central existence claim. The abstract indicates these are handled by carefully tracking singular terms, but without the explicit estimates or the details of how the variational solution for the leading-order problem is obtained, it is difficult to confirm that the nonlinear remainders are… (the audit kept the first 500 characters)
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly provides higher-order nonlinear estimates in Proposition 4.1 and detailed time-discretization and compactness arguments in §5, so these details are not absent.
grok-4.7: The paper supplies the higher-order nonlinear estimates and the time-discretization/compactness argument that dominate the remainders, and the abstract ties singular-term tracking to contact-line velocity rather than to that domination.
-
Case 4 of 25 · arXiv:2605.11176 · version 1From raw data to neutrino candidates: a neural-network pipeline for Baikal-GVD
The paper says
The central element of this approach is a gradient reversal layer (GRL) positioned between the encoder and the domain classifier.
Pith's review said
Abstract and domain-adaptation description: the statement that domain adaptation yields 'improved agreement between the two domains' provides neither the adaptation method (adversarial, MMD, etc.), a quantitative shift metric, nor performance deltas on held-out real Baikal-GVD data versus MC. This directly affects transfer of the speedup and accuracy claims to experimental events.
Place named by the review: Abstract
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly describes adversarial domain adaptation and reports quantitative Wasserstein-distance reductions, although it does not provide held-out real-data performance deltas.
grok-4.7: Section 4.2 specifies adversarial domain adaptation via a gradient-reversal domain classifier, and the results report quantitative MC–data shift reductions (Wasserstein W1 from 0.0013 to 0.0009 and from 0.019 to 0.008), so those elements are not absent.
-
Case 5 of 25 · arXiv:2605.17117 · version 2Geometric Observables for Financial Regime Detection
The paper says
Features: log returns, 5/20-day rolling volatility, 5/20-day momentum, cross-correlation, and cross-dispersion (13 raw features), enriched to 52 features... then reduced to 15 PCA components
Pith's review said
Methods section: full methodological detail on the learned spectral embedding construction (kernel choice, dimension, hyperparameters beyond the free list) and data handling for equity-index returns is absent; without these the claim that the embedding preserves geometric properties meaningful for regime detection cannot be verified, directly affecting support for the central performance claims.
Place named by the review: Methods
Why the judges called it wrong
gpt-5.6-luna: The paper specifies the embedding equations, operator construction, dimensions, hyperparameter search, data source, features, preprocessing, and causal handling, although full gradient-trained operator learning is left for future work.
grok-4.7: Data handling and the QCML embedding (Hamiltonian, operators, dimensions, and hyperparameter spaces) are specified, so those methodological details are not absent.
-
Case 6 of 25 · arXiv:2508.08279 · version 1XFMNet: Decoding Cross-Site and Nonstationary Water Patterns via Stepwise Multimodal Fusion for Long-Term Water Quality Forecasting
The paper says
We introduce XFMNet, a stepwise multimodal fusion network... XFMNet first aligns temporal resolutions... followed by locally adaptive decomposition... A cross-attention gated fusion module... Through progressive and recursive fusion
Pith's review said
The full text of the submission is the manuscript 'Probabilistic Emissivity Retrieval from Hyperspectral Data via Physics-Guided Variational Inference' (arXiv:2508.08291v2), which has no overlap with the XFMNet abstract. None of the claimed components—adaptive downsampling, locally adaptive decomposition, cross-attention gated fusion, progressive and recursive fusion—appears in the supplied text, and there are no equations, algorithms, or architecture diagrams for XFMNet. The central claim that… (the audit kept the first 500 characters)
Place named by the review: Full text
Why the judges called it wrong
gpt-5.6-luna: The supplied manuscript is explicitly XFMNet and contains the named components, equations, architecture figures, and experimental tables showing improvements over baselines.
grok-4.7: The supplied text is the XFMNet manuscript itself and explicitly describes those components, equations, architecture figures, and baseline improvements, so the statement misidentifies the paper and falsely claims they are absent.
-
Case 7 of 25 · arXiv:2605.19757 · version 1On a Multiphase Vectorial Bernoulli Free Boundary Problem
The paper says
Proposition 3.7... there exist ν∈S^{d−1} and Cj... such that U1,0(y)=C1(y·ν)+ and U2,0(y)=C2(y·ν)−.
Pith's review said
The main regularity theorem (near the end of the manuscript, following the blow-up analysis): the C^{1,η} graph representation at branching points is obtained via an improvement-of-flatness argument that requires a complete classification of homogeneous blow-up limits compatible with the vectorial structure and disjoint-support constraint. The manuscript does not provide an independent enumeration or ruling-out of all possible limits (e.g., configurations with more than two phases meeting along… (the audit kept the first 500 characters)
Place named by the review: Blow-up analysis and classification of homogeneous solutions (preceding the main regularity theorem)
Why the judges called it wrong
gpt-5.6-luna: The manuscript explicitly classifies the relevant homogeneous blow-ups as two-phase half-space profiles and separately proves the absence of triple points before applying improvement of flatness.
grok-4.7: The paper explicitly excludes three-or-more-phase meetings and classifies the relevant two-phase blow-ups as flat half-space profiles, then obtains C^{1,η} via Reifenberg reduction and [18], so the claimed missing classification and failed flatness step are contradicted.
-
Case 8 of 25 · arXiv:2508.00280 · version 1WMAS: A Multi-Agent System Towards Intelligent and Customized Wireless Networks
The paper says
We propose a metric CWMAS that jointly considers the task performance and the conversation overhead as the objective function.
Pith's review said
The statement that a reinforcement learning algorithm optimizes the adjacency matrix of the directed acyclic graph lacks any formulation of the RL problem. The state space, action space, reward function, training procedure, and evaluation/evaluation split are not described. Without these, one cannot assess whether the reported performance is on tasks used to train the policy, which would constitute a training/evaluation leakage risk. The authors should specify the reward design and how generaliz… (the audit kept the first 500 characters)
Place named by the review: Abstract, 'reinforcement learning-based algorithm'
Why the judges called it wrong
gpt-5.6-luna: The paper does formulate the optimization objective and policy-gradient procedure, although it does not fully specify state/action spaces or an evaluation split for unseen task types.
grok-4.7: The paper does formulate a policy-gradient objective with utility ϕ as the performance signal and specifies the heatmap optimization procedure in Algorithm 2, so the claim that the RL problem and training procedure are entirely undescribed is contradicted.
-
Case 9 of 25 · arXiv:2604.08464 · version 1Formulae for indices of holomorphic foliations via reduction of singularities
The paper says
The Baum–Bott index is given by BB0(F) = ∑p∈Sing(¯F) BBp(¯F) +⟨−A−1SF,SF⟩−2⟨SF,ι⟩−⟨Aι,ι⟩.
Pith's review said
Theorem 5.1 (Baum-Bott index formula): the expression obtained by summing saddle-node terms over the resolved configuration is presented as general, but it is unclear whether the proof accounts for possible dependence on the global structure of the exceptional divisor when multiple saddle-nodes are present; this directly affects the asserted extension from the generalized-curve case.
Place named by the review: Theorem 5.1
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly gives the formula for arbitrary foliations and derives it recursively using all resolved singularities and the global exceptional-divisor intersection data.
grok-4.7: There is no Theorem 5.1; the Baum-Bott formula is Theorem 3.3(4), derived from the global intersection form and the discrepancy vector of Theorem 2.10, which already includes all saddle-node contributions via S_F.
-
Case 10 of 25 · arXiv:1906.11186 · version 1A Triptych Approach for Reverse Stress Testing of Complex Portfolios
The paper says
We present an Extended RST (ERST) triptych approach with three variables: level of plausibility, level of loss and scenario. In our approach, any two of these variables can be derived by providing the third as the input.
Pith's review said
Abstract: the central claim that 'any two of these variables can be derived by providing the third as the input' and that the updated LM algorithm 'reliably derives' the missing variable for non-linear payoffs rests on an unstated mathematical mapping and solver modification; no equations, Jacobian definition, or damping schedule are supplied to support this.
Place named by the review: Abstract
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly states the triptych mapping and supplies optimization equations for the scenario-, plausibility-, and P&L-driven cases, including equations (8), (12), and (14)–(21), although it does not provide a damping schedule.
grok-4.7: The paper states the triptych mapping and supplies explicit equations plus a bisection procedure for the adapted Levenberg–Marquardt method, so the claim that the derivation is unstated and unsupported by equations is contradicted.
-
Case 11 of 25 · arXiv:2606.08395 · version 1Prime Event Languages: An Information-Theoretic Investigation of Twin-Prime Event Structure
The paper says
For each prime gap g(n), a binary event is defined by E_twin(n) = 1 if g(n) = 2 and E_twin(n) = 0 otherwise.
Pith's review said
Event-language construction (abstract and §2–3): the manuscript supplies no explicit alphabet or deterministic mapping from primes to symbols (e.g., binary twin/non-twin flag, binned gap size, or joint encoding). Because the headline result is a lag-1 dependence four orders of magnitude smaller than the per-symbol entropy, even a modest definitional coupling (a twin event forcing the next gap symbol) would produce exactly the observed signature. Without the precise construction rule and a null-m… (the audit kept the first 500 characters)
Place named by the review: Event-language construction (abstract and §2–3)
Why the judges called it wrong
gpt-5.6-luna: The manuscript explicitly specifies a binary alphabet and deterministic twin-event mapping, so the statement’s central claim that these are absent is contradicted.
grok-4.7: Sections 2.1–2.2 and 3.2 explicitly define a binary alphabet and the deterministic twin/non-twin map E_twin(n)=1 iff g(n)=2, so the claimed missing construction rule is present.
-
Case 12 of 25 · arXiv:2606.06893 · version 1Workflow-to-Skill: Skill Creation via Routing-Workflow-Semantics-Attachments Decomposition
The paper says
Baseline. We use Anthropic Skill Creator (ASC) as the baseline. Metrics. We evaluate reconstructed skills using replay-based behavioral fidelity.
Pith's review said
Experiments section: the central claim of a 10.5% improvement in behavioral replay consistency provides no details on the consistency metric, exact baselines, error bars, statistical tests, or data exclusion rules. This directly affects verifiability of the reported result.
Place named by the review: Experiments
Why the judges called it wrong
gpt-5.6-luna: The experiments do provide a baseline and a description of the behavioral-fidelity metric, although they omit error bars, statistical tests, and data-exclusion rules.
grok-4.7: The Experiments section names ASC and describes the replay-based fidelity score, so the claim that the metric and exact baseline have no details is contradicted, even though error bars, tests, and exclusion rules are unreported.
-
Case 13 of 25 · arXiv:2604.02503 · version 1Nonlinear System Identification of Variable-Pitch Propellers Using a Wiener Model
The paper says
After 350 iterations, the cost function (16) was reduced from 5.19×10−4 to 3.2×10−4, which means a 38.3% accuracy increase.
Pith's review said
§4 (Results and Validation): the central claim that the model 'reproduces the measured dynamics with good accuracy' is unsupported by any quantitative metrics (RMSE, NRMSE, R², or cross-validation error statistics) or by explicit validation plots showing residuals; only qualitative statements appear.
Place named by the review: §4
Why the judges called it wrong
gpt-5.6-luna: The paper reports quantitative validation metrics and measured-versus-model comparison plots, although it does not provide residual plots or cross-validation statistics.
grok-4.7: §4 reports quantitative error (the MSE cost in (16)–(17)), an adjusted R² of 0.996, and measured-vs-model comparison plots (Figs. 5, 6, and 8), so the accuracy claim is not supported only by qualitative statements.
-
Case 14 of 25 · arXiv:2605.24165 · version 1PeerBTS: Incentivizing Effort in Strategyproof Peer Selection
The paper says
In all cases the mean lottery share consistently discriminates between agents based on prediction quality... the gap in lottery share between highly accurate predictors and random or low skill predictors is≈ 0.2, meaning about a20%increase in reward.
Pith's review said
Simulations section: the reported experiments are explicitly non-strategic and supply no quantitative performance numbers, effort levels, or strategic deviation analysis; this leaves the validation of the claimed effort incentives and BNIC properties unsupported by the presented evidence.
Place named by the review: Simulations
Why the judges called it wrong
gpt-5.6-luna: Although the simulations do not model strategic behavior, they report quantitative performance results, including lottery-share gaps and recall effects, so the statement incorrectly says they supply no quantitative numbers.
grok-4.7: Although the simulations are explicitly non-strategic, the paper does report quantitative performance numbers such as lottery-share gaps and predictor shares, so the claim that none are supplied is contradicted.
-
Case 15 of 25 · arXiv:2605.07776 · version 1Tracing Uncertainty in Language Model "Reasoning"
The paper says
The epistemic component is estimated using the squared L2 norm of the gradient... Committal aleatoric uncertainty is estimated as the Bernoulli variance... Distributional aleatoric uncertainty is computed as the predictive entropy
Pith's review said
The method section does not provide explicit equations or pseudocode for per-token uncertainty computation (e.g., whether it is negative log-probability, entropy, or another measure) or for extracting slope/linearity features from the trace; without these, it is impossible to determine whether the reported AUROC reflects reasoning quality or merely correlates with token rarity, length, or local predictability.
Place named by the review: Method
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly gives equations for all three per-token uncertainty measures and defines slope, r², and mean features, so the claimed absence of methodological specification is incorrect.
grok-4.7: Section 3.1 gives explicit equations for per-token epistemic, committal (Bernoulli variance), and distributional (entropy) uncertainty, so the claim that those computations are unspecified is false.
-
Case 16 of 25 · arXiv:2606.23195 · version 2Memory Contagion: Cross-Temporal Propagation of Evaluator Bias via Agent Memory
The paper says
Consistency. All evaluations in this phase use the same At (temperature=0, identical prompt template) and the same Γtemporal metric.
Pith's review said
The experimental design does not isolate memory content as the cause of observed bias propagation. The abstract and described setup report effects only when varying evaluator bias but do not hold retrieval mechanisms, prompt templates, and model fixed while varying solely the consolidated memory trajectories; observed differences could therefore arise from model-specific processing of memory-augmented inputs rather than the memory store itself. This directly undermines the claim that biased inpu… (the audit kept the first 500 characters)
Place named by the review: Abstract / Experimental phases
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly describes controlled mechanism-decomposition experiments that hold the target agent, temperature, prompt template, and metric fixed while varying memory content and retrieval.
grok-4.7: The paper compares the same target agent on biased versus clean memory under oracle consolidation and, in Phase 3, holds the agent, temperature, and prompt template fixed while isolating stored-memory content from retrieval.
-
Case 17 of 25 · arXiv:2605.11356 · version 1RankGuardPolar Private Public Finite Length Polar Codes with Rank-Certified Leakage
The paper says
Theorem1 (Leakage certificate): ... L(P)≜I(u A;x p) = rank(G P)−rank(G F,P).
Pith's review said
The central claim rests on an exact algebraic characterization of leakage via the rank of submatrices of the polar generator matrix under the BEC time-sharing model, together with the construction of the linear extractor R. The abstract and high-level description assert completeness and efficiency, but without the explicit derivation, the rank identity, or the proof that R recovers all leaked combinations for every P, it is impossible to confirm the claim is load-bearing and free of gaps (see re… (the audit kept the first 500 characters)
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly derives the rank-based leakage identity and proves in Theorem 2 that an extractor R recovers all r leaked independent linear combinations for every published set P.
grok-4.7: The paper explicitly states and proves the rank-difference leakage identity (Lemma 1, Theorem 1) and constructs R that recovers all leaked linear combinations for every P (Theorem 2), so those derivations are not missing.
-
Case 18 of 25 · arXiv:2609.13144 · version 1Type Diversity Enables Transformers to Generalise Compositionally
The paper says
There are 5x6x4=120 different training sets, and we train models with 14 different random seeds for each training set and report the mean and standard deviation.
Pith's review said
No mention is made of replication details, such as the number of independent dataset variants, the use of multiple random seeds, or whether the observed correlation is robust across different architectural choices (e.g., layer count, embedding size). These are standard controls needed to strengthen a correlational claim. Without them, the degree of support for the hypothesis remains unclear.
Place named by the review: Abstract
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly reports the number of dataset variants and multiple random seeds, although it does not establish robustness across architectural choices.
grok-4.7: The paper explicitly reports many dataset variants and multiple random seeds (12 in COGS; 14 per SCAN split), so the claim that these replication details are never mentioned is false.
-
Case 19 of 25 · arXiv:2606.31959 · version 1AnyBokeh: Physics-Guided Any-to-Any Bokeh Editing with Optical Fingerprint Transfer
The paper says
This formulation reveals a linear relationship between signed CoC and disparity difference, where the slope κ is shared by all pixels of the same source image.
Pith's review said
Abstract: the central claim that a linear relation between signed circle-of-confusion and disparity difference enables reliable single-image extraction of a transferable optical fingerprint is stated without derivation, error bounds, or ablation on robustness to lens aberrations, depth errors, or non-paraxial effects. This assumption is load-bearing for the fingerprint-transfer pipeline and the claim of avoiding all-in-focus reconstruction.
Place named by the review: Abstract
Why the judges called it wrong
gpt-5.6-luna: Although the abstract does not provide the derivation or robustness analyses, the paper derives the linear relation in Eqs. (1)–(4) and includes related ablations, so the claim that it is stated without derivation is incorrect.
grok-4.7: Section 3.1 derives the signed CoC–disparity linearity from the thin-lens model under S1≫f (Eqs. 1–4), so the claim is not stated without a derivation.
-
Case 20 of 25 · arXiv:2508.09697 · version 4Label-Noise Resistant Learning via Optimal Brain Damage Masking
The paper says
Proposition 1. Under the standard CE loss and the OBD assumptions, the estimated loss perturbation δL induced by masking a classifier connection is upper-bounded by one eighth of the empirical second moment of its edge activation.
Pith's review said
The bridge from OBD to low-activation masking is not established. OBD saliency is approximately (1/2) H_ii w_i^2, depending on the second-order curvature and the weight magnitude, not on the activation magnitude. Masking low-activation edges does not generally minimize the OBD loss perturbation: a low-activation edge can sit on a high-curvature coordinate, and a high-activation edge can carry label noise. The abstract needs a derivation or a controlled experiment showing that activation magnitud… (the audit kept the first 500 characters)
Place named by the review: Abstract
Why the judges called it wrong
gpt-5.6-luna: The paper explicitly derives an OBD-based activation-magnitude bound for loss perturbation and uses it to justify masking low-activation edges.
grok-4.7: The paper does derive the OBD-to-activation bridge: under CE the OBD perturbation equals a term proportional to p(1−p)a² and is upper-bounded by the activation second moment, so low activation cannot yield large OBD saliency in this setting.
-
Case 21 of 25 · arXiv:2606.16349 · version 3From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning
The paper says
These controls address several carrier-selection artifacts, but they are not raw activation-delta norm-matched controls. We therefore phrase the causal result as fixed-protocol sensitivity relative to matched unit-direction controls
Pith's review said
§3.4 and Table 4: Causal claims are framed as fixed-protocol sensitivity relative to matched unit-direction controls, which is appropriate, but the intervention-level follow-up shows high empty/nonsensical rates and low benign helpfulness on selected reopening rows. The manuscript should more clearly separate “reopens ASR under the protocol” from any implication of selective harmful-usefulness reopening, and state that raw activation-delta norm-matched controls remain future work so that scale a… (the audit kept the first 500 characters)
Place named by the review: §3.4, Table 4
Why the judges called it wrong
gpt-5.6-luna: The manuscript already explicitly limits causal claims, notes the high empty/nonsensical rates and low helpfulness of selected rows, and identifies norm-matched controls as future work.
grok-4.7: The paper already separates protocol ASR reopening from harmful-usefulness reopening and already treats raw activation-delta norm-matched controls as absent and future work, so the claimed gap is not missing.
-
Case 22 of 25 · arXiv:2604.24856 · version 1The Bulge Cluster Origin (BulCO) survey with CRIRES at the ESO-VLT: a chemical screening of the Globular Cluster NGC 6553
The paper says
Table A.1: Chemical abundances and corresponding errors of the observed stars in NGC 6553.
Pith's review said
Abstract: No tables of individual stellar abundances, no error budgets, and no discussion of potential systematics or test calibration/validation are provided. This omission prevents assessment of whether the reported spreads (e.g., in Na, N, C) and the chemical DNA test results are robust enough to support the in-situ conclusion.
Place named by the review: Abstract
Why the judges called it wrong
gpt-5.6-luna: The paper provides an individual-abundance table and an error budget, including formal errors and global uncertainties of 0.10–0.15 dex, so the claimed omission is incorrect.
grok-4.7: The paper provides Table A.1 of individual stellar abundances with errors and discusses formal and global error budgets plus Arcturus line calibration, so the claimed omissions are not present.
-
Case 23 of 25 · arXiv:2607.14972 · version 1Synergistic Effects of Phosphorus Doping and Oxygen Vacancies on Formaldehyde Oxidation over CeO₂(111): A First Principles Investigation
The paper says
The adsorption energy of HCHO is found to be -2.43 eV at 300 K... an adsorption energy of -1.32 eV is observed... At room temperature... O2 adsorbs exothermically with an adsorption energy of -1.59 eV.
Pith's review said
The finite-temperature adsorption energies are not reconciled with the static values. The AIMD values are HCHO: -2.43 eV (static -2.65 eV), O2 on defective surface: -1.32 eV, and O2 co-adsorption: -1.59 eV (static -1.77 eV). Since the relative adsorption strengths of HCHO and O2 determine which co-adsorption sequence is favored, the origin of the 0.5 eV differences (thermal sampling, finite-size, definition of the reference state) should be stated explicitly.
Place named by the review: Section III.C/III.D
Why the judges called it wrong
gpt-5.6-luna: Although the paper reports these finite-temperature values without explaining their differences from static calculations, the stated discrepancies are about 0.18–0.22 eV where comparable static values are given, not 0.5 eV.
grok-4.7: The paper's static and 300 K values differ by about 0.2 eV (HCHO −2.65 vs −2.43 eV; co-adsorbed O2 −1.77 vs −1.59 eV), not the claimed 0.5 eV, so the statement misdescribes the discrepancy.
-
Case 24 of 25 · arXiv:2401.10166 · version 4VMamba: Visual State Space Model
The paper says
VMamba exhibits linear growth in FLOPs while maintaining comparable performance. This demonstrates its state-of-the-art input scalability.
Pith's review said
§4 (experimental section): the headline claim of 'superior input scaling efficiency' is asserted without visible tables reporting FLOPs-vs-accuracy curves, resolution scaling plots, or error bars across multiple runs; without these quantitative details the central efficiency advantage cannot be independently verified from the manuscript.
Place named by the review: §4
Why the judges called it wrong
gpt-5.6-luna: The manuscript includes resolution-scaling results in Figures 4 and 5 and Appendix G, including quantitative accuracy, FLOPs, throughput, and memory comparisons, although it does not report error bars.
grok-4.7: Experiments are in §5, not §4, and the manuscript includes resolution-scaling and FLOPs/throughput plots (Figures 4–5) plus Appendix Table 9, so the claim is not left without those quantitative details.
-
Case 25 of 25 · arXiv:2502.07307 · version 1CreAgent: Towards Long-Term Evaluation of Recommender System under Platform-Creator Information Asymmetry
The paper says
the feedback memory ... will be updated based on the partial user feedback information ... (∑ E, ∑ Y)
Pith's review said
The creator feedback loop in Eqs. (1)–(2) and the PPO reward in §4.5 depend entirely on the user agent's clicks. Table 6 reports at most 0.630 accuracy and 0.740 recall, and the text explicitly acknowledges that the agents tend to hallucinate positive clicks, identifying some negative items as positive. With a positive-click bias, the feedback signal z_i(n) is systematically inflated, so the creator beliefs (Eq. (4)), PPO rewards (Eq. (7)), and the Section 7 conclusions about fairness- and diver… (the audit kept the first 500 characters)
Place named by the review: Appendix C and §5 (User agents)
Why the judges called it wrong
gpt-5.6-luna: The paper’s feedback utility z_i(n) uses both exposure and clicks in Eq. (2), so the creator loop and PPO reward do not depend entirely on user-agent clicks, although the paper does acknowledge false-positive click behavior.
grok-4.7: Eqs. (1)–(2) and the PPO reward use a β-weighted mix of exposure and clicks, not clicks alone, and Table 6 accuracy reaches 0.653 rather than at most 0.630, so the claimed systematic click-bias chain is not what the paper describes.
Labels on abstracts and citations
Pith labels every sentence of a paper's abstract and every sentence that cites another paper. A label is only useful if you know how often it is right, so we measured it. On 2 Oct 2026 we drew fresh random samples from the public record, had two independent model families (gpt-5.6-luna and grok-4.7) label them without seeing Pith's label, and counted how often Pith's published label matched the two families when they agreed with each other.
92.0% of sentence roles, 90.8% of main-claim picks and 93.3% of typed citations matched. measured 2 Oct 2026 · judged by gpt-5.6-luna and grok-4.7
Agreement with two other models measures consistency, not truth. Where the two families disagree with each other, the item is left out. The 95% intervals below are Wilson intervals on the number of items checked.
Abstract records
200 randomly drawn papers, 1,444 sentences. The record API returns these labels for each abstract sentence, with a probability for each.
| Label | Matches consensus | 95% interval | Items | The two families agree with each other |
|---|---|---|---|---|
| What a sentence does (background, method, result, and so on) | 92.0% | 90% to 93% | 1,292 | 90% |
| Whether the sentence reports the authors' own work | 92.2% | 91% to 93% | 1,429 | 99% |
| Whether the sentence states a number | 96.1% | 95% to 97% | 1,298 | 90% |
| How strongly the sentence is stated | 94.9% | 94% to 96% | 1,318 | 91% |
| Which sentence states the main claim | 90.8% | 85% to 94% | 152 papers | 76% |
The probability on each label is honest
Each label carries a probability. Sentence roles with a probability above 95% matched the consensus 99.6% of the time. Labels below 60% matched only 60%. Weigh a label by its probability.
| Probability shown (sentence role) | Items | Matches consensus | 95% interval |
|---|---|---|---|
| 0 to 0.6 | 87 | 59.8% | 49% to 69% |
| 0.6 to 0.8 | 160 | 74.4% | 67% to 80% |
| 0.8 to 0.95 | 251 | 90.0% | 86% to 93% |
| 0.95 to 1 | 794 | 99.6% | 99% to 100% |
Typed citations
1,429 citing sentences from random papers, published at Pith's confidence thresholds (intent probability 0.8, 0.95 for support, contest probability 0.7). 1,219 had a consensus. Overall 93.3% matched (95% interval 92% to 94%). A random assignment of the same labels would match 23% of the time.
| Published as | Checked | With consensus | Matched | 95% interval |
|---|---|---|---|---|
| background | 574 | 531 | 98% | 96% to 99% |
| uses the method or tool | 261 | 230 | 93% | 89% to 96% |
| uses the data | 97 | 77 | 95% | 87% to 98% |
| compares results | 279 | 217 | 87% | 82% to 90% |
| supports the citing paper | 89 | 69 | 94% | 86% to 98% |
| contests | 47 | 29 | 69% | 51% to 83% |
| extends | 82 | 66 | 88% | 78% to 94% |
Contest links are rare, so only a handful were in a random sample and their interval is wide. A separate sample drawn only from contest links agreed with the consensus 93% of the time.
What this does not show
- Two model families agreeing with Pith is evidence of consistency. It is not a human expert panel, and where the models share a blind spot, so does the check.
- The review figure of 4 Oct 2026 was judged against Pith's text copy of each paper, and it is one number for every lane. It is not the figure for a review written today.
- A review comment judged accurate agrees with the text its judges were given. Whether it matters to the paper's conclusions is a further question these checks do not score.
- The admitted-error papers are ones whose authors said what went wrong. Errors nobody has admitted are not in that sample, and no human expert has yet scored the reviews.
- The abstract sample is arXiv papers drawn at random across fields; a single field can differ.
- Only abstracts were labelled for most papers. A record's
coverageblock says what was and was not read.
Use the numbers
The same figures, and the 25 cases, are available as JSON at /api/v1/accuracy. Every paper record at /api/v1/records/<arXiv id> carries a short summary and a link back to this page.
A larger study with papers chosen by an outside curator is designed and has not been run. What has been measured, when and by whom, and what has not been run: the accuracy study page. What a review costs to make and how many are made a day: receipts. How a review is made: methodology. How much of a paper a program checked: verification tiers.