RegCheck: A tool for structured comparisons between study registrations and papers

Beth Clarke; Ian Hussey; Jamie Cummins; Malte Elson

REVIEW 3 major objections 5 minor 6 references

RegCheck: A tool for structured comparisons between study registrations and papers

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RegCheck is an LLM-assisted pipeline that pulls the most relevant text from a study registration and its paper along user-chosen dimensions, returning candidate deviation judgments for a human to verify.

desk verdict A useful, honestly-caveated tool paper for registration–paper checks; the top-k retrieval problem behind the 'missing' judgment is the main technical concern to resolve before publication. read the letter →

arxiv 2601.13330 v2 pith:NQVZW7UT submitted 2026-01-19 cs.CL

Jamie Cummins , Beth Clarke , Ian Hussey , Malte Elson This is my paper · ORCID

classification cs.CL

keywords studyregistrationpreregistrationregistration-papercomparisonlargelanguagemodelsembeddingsdeviationdetectionresearchtrustworthinesshuman-in-the-loop

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents RegCheck, a modular tool that makes registration-paper consistency checks far less laborious by automating the search for relevant evidence. For each user-defined comparison dimension, RegCheck embeds both documents, retrieves the most pertinent passages, and has a large language model summarize the passages and issue a yes/no/missing deviation judgment. The human remains the decision-maker: the tool's output is an orienting aid with direct quotes, not a verdict. The authors argue that lowering the effort barrier could make registration checks a routine part of peer review and self-review, where they are currently rare. The paper explicitly does not claim formal validation; it describes the workflow, design principles, and an ongoing evaluation agenda.

What carries the argument

The central mechanism is the dimension-driven retrieval-and-judgment loop. A user-supplied dimension label and definition are embedded and used to rank sentence-aware text chunks from both documents by cosine similarity; the top excerpts are sent to a user-selected LLM, which produces concise summaries and a yes/no/missing deviation judgement. This loop turns an unstructured document pair into a structured report, and its modularity (choice of parser, model, and dimensions) is what lets the tool claim discipline-agnostic applicability.

What would settle it

A formal evaluation on a corpus with expert-established ground-truth deviations, comparing RegCheck's agreement with expert reviewers against expert-expert agreement, would settle the claim; if RegCheck's agreement falls significantly below the human-human baseline, or if it misses injected inconsistencies in artificially generated texts at high rates, the tool's orienting value would be undermined.

Watch

Extended reading notes

Core claim

RegCheck's central claim is that a modular pipeline of parsing, embedding, retrieval, and LLM judgment can convert the open-ended task of comparing a registration with its paper into a structured, auditable report. The user specifies or chooses dimensions; the system retrieves the most relevant excerpts from each source, produces LLM summaries with citation to excerpt IDs, and encodes a deviation judgment as yes, no, or missing. The design keeps human expertise in the loop by design—users define what matters and verify the evidence—and standardizes prompts and comparison logic across calls and users, countering the variability of ad hoc LLM use. The authors present this as infrastructure for

Load-bearing premise

The load-bearing premise is that RegCheck's retrieval and LLM judgments are reliable enough that the passages and yes/no/missing labels it returns orient a human reviewer correctly rather than misdirect attention—something the paper explicitly states is not yet formally validated.

Editorial extensions

If this is right

A reviewer could compare a registration and paper in minutes rather than hours, focusing on the exact passages relevant to each dimension.
Standardized prompting across calls and users would reduce the output variability that plagues ad hoc LLM comparisons.
The shareable report format with unique IDs would allow multiple reviewers to audit the same comparison, adding transparency to integrity checks.
Because human judgment is the final step, even imperfect automation could increase the frequency of registration checks relative to the current near-zero baseline.
If accuracy holds across disciplines, the same pipeline could be extended to other consistency checks, such as comparing analysis code to planned analyses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the 'missing' judgment could itself become a meta-science signal, flagging registrations that are too underspecified to be checked—a problem the paper notes in passing.
Inference: if such tools become common, the social cost of undisclosed deviations rises, which might push authors to report deviations proactively and reduce the stigma around legitimate changes of plan.
Inference: the same retrieval-plus-judgment architecture could generalize to other verification tasks such as checking whether a paper's claims are supported by its methods section, or whether a data appendix matches the text.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

RegCheck is a real, working system that fills a real gap: registrations go unexamined because manual comparison is laborious, and this tool gives reviewers an orienting signal via user-defined dimensions, dense retrieval, per-dimension LLM judgments, and shareable reports. That combination is genuinely new and plausibly useful. The authors are also refreshingly honest that formal evaluation is pending and that they provide no fidelity guarantees. For a tool paper, that candor goes a long way.

The strongest part is the design philosophy: human-in-the-loop, non-prescriptive, with independent LLM calls per dimension so one dimension's outcome doesn't contaminate another. The privacy section is thoughtful. The anecdotal claims about catching subtle deviations are plausible and appropriately framed as impressions.

The soft spot that matters most is the retrieval bottleneck. Deviations are judged only on the top-k chunks for each dimension. If a relevant passage doesn't rank in the top-k, it never reaches the LLM, and the resulting 'no deviation' or 'missing' verdict is an artifact of retrieval, not an exhaustive search. The 'missing' category in particular risks conflating 'not retrieved' with 'not present.' The paper gives no recall@k evaluation and no way for users to see what was excluded. This doesn't sink the tool — it is still useful for orienting attention — but it undermines the stronger reading of the accuracy claims. The authors should either bound the recall problem explicitly, surface unretrieved candidates, or change the language so 'missing' is understood as 'not found in the retrieved evidence.'

Two smaller issues: the abstract promises an example use case that never appears in the body, and the text says the code is on GitHub but gives no repository URL or commit hash. Both are easy to fix.

The citation pattern looks fine. The background on registration prevalence and non-checking is solid. The self-citation about prompt sensitivity is relevant, not load-bearing. This is not a circular or overclaimed paper; it is an honest engineering report awaiting its evaluation.

Bottom line: a serious referee should engage this. The architecture is coherent, the domain gap is real, and the planned evaluation strategy — including comparison to human-human agreement — is sensible. I'd want the retrieval-recall issue addressed and the missing example added before accepting, but I'd send it to review today.

For the reading group: worth a session, mostly to discuss what 'validated' should mean for a tool like this.

Recommendation: accept for peer review with requests for the example, the repo link, and a direct treatment of retrieval recall.

Referee Report

3 major / 5 minor

Summary. The paper introduces RegCheck, an open-source, LLM-assisted tool designed to help researchers, reviewers, and editors compare study registrations with corresponding papers. The tool's pipeline consists of six stages: ingestion, extraction, embedding, user-specified dimension definition, analysis (dense retrieval of relevant excerpts, LLM summarization, and a yes/no/missing deviation judgment), and reporting of structured, shareable outputs. The authors emphasize a human-in-the-loop design: users choose the comparison dimensions and are presented with direct quotes and LLM summaries for verification, rather than having the LLM replace human judgment. The paper describes the system's workflow, design principles, privacy considerations, availability, and outlines an ongoing program of formal evaluation, while also making anecdotal claims about the tool's accuracy.

Significance. If validated, RegCheck could substantially reduce the effort required for registration–paper consistency checks in peer review and metaresearch, potentially increasing the frequency of such checks. The system's modularity, discipline-agnostic framing, and transparent design are strengths, as is the authors' explicit acknowledgment that formal evaluation is pending. The open-source code and public app make the work immediately usable and falsifiable. However, the paper currently offers no empirical evidence for the tool's central functional claims of retrieval completeness, summarization fidelity, or judgment accuracy; those claims rest on unsystematic anecdotal impressions. The significance of the contribution therefore depends on whether those claims survive the ongoing evaluation.

major comments (3)

[§2, Stage 5] The retrieval step is load-bearing: the LLM judge sees only the top-k chunks returned by dense embedding retrieval, and the 'missing' category is inferred from the absence of retrieved evidence, not from an exhaustive search. This design cannot distinguish 'not present' from 'not retrieved,' so false 'missing' or false 'no deviation' outcomes can arise from retriever misses. The paper does not report a recall@k figure, specify the value of k, or provide any sensitivity analysis with respect to chunk size/overlap. A retrieval-recall evaluation, or at minimum a mechanism to flag low-confidence retrieval, is necessary to support the claim that RegCheck reliably orients users to relevant passages.
[§5] The paper states that 'our impressions of RegCheck, based on extensive-but-unsystematic testing, are that it is generally very accurate in its extractions and is quite adept at catching genuine deviations,' and that it 'frequently catches deviations that members of our research team missed.' No data are presented to substantiate these claims, and they are in tension with the adjacent disclaimer that 'we currently do not provide any guarantees about the fidelity of RegCheck’s output.' These are central functional claims; either they should be supported by a systematic evaluation (even a small pilot) or rephrased as provisional hypotheses pending the formal evaluation described in §7.
[§7] The authors correctly state that 'anecdotal testing is not a substitute for formal evaluation,' and they describe an evaluation plan. However, the manuscript as submitted contains no formal or structured evaluation of any component—retrieval, summarization, or deviation judgment. For a paper whose central contribution is a tool claiming to facilitate registration–paper comparisons, the absence of any measured accuracy data (e.g., inter-rater agreement on a small set of documents, or a pilot with injected inconsistencies) leaves the central claim unvalidated. At minimum, a proof-of-concept evaluation on a small corpus would make the paper self-contained and allow readers to assess whether the design is fit for purpose.

minor comments (5)

[§2, Stage 4] The description of 'pre-set dimensions' is clear, but the list of default dimensions is not given in the text (only described as 'sample size, primary outcomes,' etc.). Including an explicit list or reference to the repository would help readers understand the intended scope.
[§6] Minor typo: 'text served to its models' should likely be 'text sent to its models.'
[§7] Typo: 'solicling' should be 'soliciting.'
[§4] The privacy statement says 'no user information is logged.' If the web app keeps server logs (even transiently) or uses analytics, this should be clarified to avoid an overbroad claim.
[Figure 1] The screenshot is informative, but the color coding (red/blue/yellow) is described in the caption. Consider adding a legend within the figure itself for accessibility in grayscale printing.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: RegCheck is an engineering tool paper with explicit fidelity disclaimers; the one background self-citation is not load-bearing.

full rationale

RegCheck is a software/workflow description rather than a derivation or prediction. Its central claim—that it extracts relevant text and produces deviation judgments along user-specified dimensions—is an architectural design claim, not a mathematical consequence derived from its inputs. The main potential weakness (retrieval may miss relevant passages, and 'missing' may conflate 'not retrieved' with 'not present') is an accuracy limitation, not a circularity: the paper explicitly disclaims fidelity in Section 5 ('we currently do not provide any guarantees about the fidelity of RegCheck's output') and Section 7 ('anecdotal testing is not a substitute for formal evaluation'). The only self-citation by a co-author (Cummins, 2025) is used in Section 1 to support the background observation that unstandardized prompt content causes variability in LLM outputs; it is co-cited with independent work (Sclar et al., 2023), does not define RegCheck's outputs, and is not the basis of any claimed result. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The planned evaluation against human raters and against artificially injected inconsistencies is an external validation design, further confirming that the paper is not circular. The score of 1 reflects only the presence of a minor, non-load-bearing self-citation, not any circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new scientific entities are postulated; RegCheck is a software artifact. The free parameters are design choices in the retrieval pipeline. The axioms are unvalidated assumptions about retrieval, LLM reliability, human oversight, and cross-reference resolution — all load-bearing for the tool's claimed functionality.

free parameters (3)

chunk_size = 200 tokens
Sentence-aware token chunking size chosen by hand in Section 2; no sensitivity analysis is reported, yet it directly affects retrieval granularity.
chunk_overlap = 30 tokens
Overlap between chunks chosen by hand to preserve sentences in Section 2; no sensitivity analysis is reported.
top_k excerpts = not specified
Section 2 says retrieved candidates are 'truncated to the final top-k,' but the value of k is not reported; it is a key retrieval parameter without which exact behavior cannot be reproduced.

assumptions (4)

domain assumption Dense embedding retrieval with cosine similarity surfaces the passages genuinely most relevant to an arbitrary user-defined dimension.
The entire evidence-extraction step in Section 2 depends on this; no evaluation of retrieval quality is reported.
domain assumption A user-selected LLM produces faithful summaries and reliable deviation judgments from the retrieved excerpts.
Section 2 Analysis sends excerpts to an LLM and treats its yes/no/missing output as informative; the authors state in Sections 5 and 7 that formal validation is still underway.
domain assumption Human users will verify the tool's output and catch false positives and false negatives.
The human-in-the-loop design (Sections 3 and 6) assumes oversight is feasible and effective, but no user study supports this assumption.
domain assumption An LLM can resolve cross-references between studies in multi-study papers (e.g., 'our procedure was the same as the previous experiment') and extract relevant details from prior studies.
Section 2 Extraction relies on an unvalidated 'LLM flow' for this step; failure here would remove important context from the comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RegCheck: A tool for structured comparisons between study registrations and papers." pith.science (2026). https://pith.science/paper/NQVZW7UT

@misc{pith2026260113330,
  author       = {Pith},
  title        = {Pith review of: RegCheck: A tool for structured comparisons between study registrations and papers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQVZW7UT}},
  note         = {Machine review of arXiv:2601.13330}
}

read the original abstract

Across the social and medical sciences, researchers recognize that specifying planned research activities (i.e., 'registration') prior to the commencement of research has benefits for both the transparency and rigour of science. Despite this, evidence suggests that study registrations frequently go unexamined, minimizing their effectiveness. In a way this is no surprise: manually checking registrations against papers is labour- and time-intensive, requiring careful reading across formats and expertise across domains. The advent of AI unlocks new possibilities in facilitating this activity. We present RegCheck, a modular LLM-assisted tool designed to help researchers, reviewers, and editors from across scientific disciplines compare study registrations with their corresponding papers. Importantly, RegCheck keeps human expertise and judgement in the loop by (i) ensuring that users are the ones who determine which features should be compared, and (ii) presenting the most relevant text associated with each feature to the user, facilitating (rather than replacing) human discrepancy judgements. RegCheck also generates shareable reports with unique RegCheck IDs, enabling them to be easily shared and verified by other users. RegCheck is designed to be adaptable across scientific domains, as well as registration and publication formats. In this paper we provide an overview of the motivation, workflow, and design principles of RegCheck, and we discuss its potential as an extensible infrastructure for reproducible science with an example use case.

Figures

Figures reproduced from arXiv: 2601.13330 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages

[1]

(2025, August)

Adam, D. (2025, August). The peer-review crisis: how to fix an overloaded system.Nature,644(8075), 24–27. Retrieved 2025-11-24, fromhttps:// www.nature.com/articles/d41586-025-02457-2doi: 10.1038/d41586 -025-02457-2 Bakker, M., Veldkamp, C. L. S., Van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., ... Wicherts, J. M. (2020, December)....

arXiv 2025
[2]

Mahtani, K

doi: 10.48550/ARXIV.2509.13397 Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., ... Mahtani, K. R. (2019, December). COMPare: a prospective co- hort study correcting and monitoring 58 misreported trials in real time. Trials,20(1),

arXiv doi:10.48550/arxiv.2509.13397 2019
[5]

P., Nelson, L

doi: 10.48550/ARXIV.2310.11324 Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychol- ogy: Undisclosed flexibility in data collection and analysis allows present- ing anything as significant.Psychological science,22(11), 1359–1366. doi: 10.1177/0956797611417632 14 Spitzer, L., & Mueller, S. (2023a, March). Registered report: Survey...

arXiv doi:10.48550/arxiv.2310.11324 2011
[9]

(2013, April)

Retrieved 2025-10- 17, fromhttps://bmcmedresmethodol.biomedcentral.com/articles/ 10.1186/s12874-017-0465-7doi: 10.1186/s12874-017-0465-7 Mathieu, S., Chan, A.-W., & Ravaud, P. (2013, April). Use of Trial Reg- ister Information during the Peer Review Process.PLoS ONE,8(4), e59910. Retrieved 2026-01-16, fromhttps://dx.plos.org/10.1371/ journal.pone.0059910d...

arXiv 2025
[74]

(2023, October)

Retrieved 2025-10-16, fromhttps://linkinghub.elsevier.com/ retrieve/pii/S0895435619300435doi: 10.1016/j.jclinepi.2019.05.009 TARG Meta-Research Group & Collaborators. (2023, October). Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses.BMJ Open,13(10), e076264. Retrieved 2025-08-27...

work page doi:10.1016/j.jclinepi.2019.05.009 2025
[118]

Retrieved 2025-10-16, fromhttps://trialsjournal .biomedcentral.com/articles/10.1186/s13063-019-3173-2doi: 10 .1186/s13063-019-3173-2 GROBID. (2025, September).GROBID Documentation.Retrieved 2025-11-24, fromhttps://grobid.readthedocs.io/en/latest/ Hahn, L., Glöckner, A., Gollwitzer, M., Hellmann, J., Lange, J., Schindler, S., & Sassenberg, K. (2025, July)....

arXiv 2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

(2025, August)

Adam, D. (2025, August). The peer-review crisis: how to fix an overloaded system.Nature,644(8075), 24–27. Retrieved 2025-11-24, fromhttps:// www.nature.com/articles/d41586-025-02457-2doi: 10.1038/d41586 -025-02457-2 Bakker, M., Veldkamp, C. L. S., Van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., ... Wicherts, J. M. (2020, December)....

arXiv 2025

[2] [2]

Mahtani, K

doi: 10.48550/ARXIV.2509.13397 Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., ... Mahtani, K. R. (2019, December). COMPare: a prospective co- hort study correcting and monitoring 58 misreported trials in real time. Trials,20(1),

arXiv doi:10.48550/arxiv.2509.13397 2019

[3] [5]

P., Nelson, L

doi: 10.48550/ARXIV.2310.11324 Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychol- ogy: Undisclosed flexibility in data collection and analysis allows present- ing anything as significant.Psychological science,22(11), 1359–1366. doi: 10.1177/0956797611417632 14 Spitzer, L., & Mueller, S. (2023a, March). Registered report: Survey...

arXiv doi:10.48550/arxiv.2310.11324 2011

[4] [9]

(2013, April)

Retrieved 2025-10- 17, fromhttps://bmcmedresmethodol.biomedcentral.com/articles/ 10.1186/s12874-017-0465-7doi: 10.1186/s12874-017-0465-7 Mathieu, S., Chan, A.-W., & Ravaud, P. (2013, April). Use of Trial Reg- ister Information during the Peer Review Process.PLoS ONE,8(4), e59910. Retrieved 2026-01-16, fromhttps://dx.plos.org/10.1371/ journal.pone.0059910d...

arXiv 2025

[5] [74]

(2023, October)

Retrieved 2025-10-16, fromhttps://linkinghub.elsevier.com/ retrieve/pii/S0895435619300435doi: 10.1016/j.jclinepi.2019.05.009 TARG Meta-Research Group & Collaborators. (2023, October). Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses.BMJ Open,13(10), e076264. Retrieved 2025-08-27...

work page doi:10.1016/j.jclinepi.2019.05.009 2025

[6] [118]

Retrieved 2025-10-16, fromhttps://trialsjournal .biomedcentral.com/articles/10.1186/s13063-019-3173-2doi: 10 .1186/s13063-019-3173-2 GROBID. (2025, September).GROBID Documentation.Retrieved 2025-11-24, fromhttps://grobid.readthedocs.io/en/latest/ Hahn, L., Glöckner, A., Gollwitzer, M., Hellmann, J., Lange, J., Schindler, S., & Sassenberg, K. (2025, July)....

arXiv 2025