A 500-paper accuracy study: designed, not yet run
We have designed an accuracy study of about 500 independently curated papers. It has not been run, and no papers have been chosen for it.
This page is not a score. Below: what has been measured about Pith's reviews, when and by whom; what has not been run; and where the study stands.
What has been measured
| What | When | Judged by | On how many | Result |
|---|---|---|---|---|
| Whether a review's major criticisms are accurate about the paper (method and tables) | 4 October 2026 | Two other model families (gpt-5.6-luna and grok-4.7), reading Pith's own text copy of each paper (text extracted from the PDF, the first 90,000 characters). Not people. | 200 published reviews drawn at random, one major criticism from each; the two families gave the same answer on 151 and agreed 149 could be settled | 113 of 149 judged accurate (75.8%; 95% interval 68% to 82%) One figure for every model that had written reviews by then; it cannot be split by lane. A criticism that was wrong because of Pith's text copy was checked against the same copy. |
| Whether the review of an earlier version found the error its authors later admitted (method and tables) | 4 October 2026 | The same two model families (gpt-5.6-luna and grok-4.7), reading the review beside the authors' own statement of the error. Not people. | 150 papers whose authors withdrew or corrected them and said what was wrong; the two families gave the same answer on 105 | found in 18, a related concern in 29, missed in 58 (17.1% found; 95% interval 11% to 26%) Only errors that authors themselves admitted are in this sample. |
| Whether the labels Pith puts on abstract sentences and on citations match two other models (method and tables) | 2 October 2026 | Two other model families (gpt-5.6-luna and grok-4.7), labelling the same sentences without seeing Pith's label. Not people. | 1,444 sentences of 200 papers; 1,429 citing sentences, 1,219 with a consensus | sentence roles matched 92.0% of the time; typed citations 93.3% (95% interval 92% to 94%) These are labels, not reviews. Agreement with two models measures consistency, not truth. |
| How many of six defects planted in six papers a review catches (method and tables) | 31 July 2026 | One judge, blind: the judge saw anonymized review packets, and which model wrote which packet was unsealed only after the final judgment. The record published here does not name the judge. | 6 planted defects in six papers seeded with them, reviewed by each of 4 model configurations | 5 of 6 for DeepSeek V4 Flash, with the screening clause as worded on 31 July 2026; the other configurations are in the table below The 5 of 6 figure was measured on 31 July 2026 with the wording of the screening clause then in use. The last sentence of the clause was reworded on 13 November 2026, and the set of six has not been run again with the new wording. |
No figure on this page is a forecast, and none was judged by a person unless its row says so. Two models agreeing that a criticism is accurate is evidence that it is, not proof.
What has not been run
| What | Where it stands |
|---|---|
| The accuracy study of about 500 independently curated papers | Designed in July 2026. Not run. No papers have been chosen for it, and the choice of papers has not been approved. |
| A check of each lane's reviews against the paper itself | Built in November 2026. Not run. It would give one figure for each model that writes reviews, judged by models of other families that read the paper's LaTeX source or its pages and not Pith's text copy. No lane has a figure yet. |
| A check of the reviews by people | Not run. No human expert has scored a sample of Pith's reviews. Every figure on this page was judged by models, except where the row says otherwise. |
| The six planted defects, with the review instructions as they are worded today | Not run. The 5 of 6 figure was measured on 31 July 2026 with the wording of the screening clause then in use. The last sentence of the clause was reworded on 13 November 2026, and the set of six has not been run again with the new wording. |
Where the 500-paper study stands
| Question | Answer |
|---|---|
| Study | pith-error-rate-v1, version 1.0.0 |
| Design | designed |
| Run | not yet run |
| Planned size | about 500 independently curated papers |
| Papers chosen | no |
| Selection of papers approved | no |
| Results | none yet |
| Changes what the site publishes | no |
| Model the design was written for | Grok 4.5 at high effort, which wrote Pith's reviews when the design was fixed in July 2026. Other models write them now (the lanes). Run as designed, the study would measure a pipeline that is no longer in service; measuring today's would need a new design. |
Check that the design has not changed
The design is a fixed set of files. Their fingerprints:
manifest SHA-256 f3c1c40b22b3f13f0e0d906078f170b7f488376a5e8f428bd57e4889b0b0cd39
Merkle root 514ca5adcba6610735199a43083da85f22c2bae060f560d4cc4f2f5eed48ec89
What the study will not claim
This benchmark does not estimate or measure the truth rate of science.
What it will measure, once the papers are chosen: Pith classification, abstention, and graded-risk behavior on the frozen v1 benchmark corpus under the frozen v1 protocol.
The six planted defects, model by model
Measured 2026-07-31, n=6. One judge, blind: the judge saw anonymized review packets, and which model wrote which packet was unsealed only after the final judgment. The record published here does not name the judge.
| Model | Planted defects caught | Note |
|---|---|---|
| GPT-5.6 Luna | 6 of 6 | highest catch rate, higher cost |
| DeepSeek V4 Flash, with the screening clause as worded on 31 July 2026 | 5 of 6 | the production configuration on 31 July 2026; the last sentence of the clause was reworded on 13 November 2026 and the set has not been run again |
| DeepSeek V4 Flash, base | 3 of 6 | noticed defects but excused them; fixed by one prompt clause |
| Grok 4.5 high | 2 of 6 | previous production model |
The 5 of 6 figure was measured on 31 July 2026 with the wording of the screening clause then in use. The last sentence of the clause was reworded on 13 November 2026, and the set of six has not been run again with the new wording. These rates are six planted defects under a frozen rubric, judged blind. They are not a 500-paper confusion matrix. That study has not been run. Method and receipts: /receipts. The audit of published labels and review criticisms is at /accuracy.