{"id":"72cec19a-3815-4ac9-9094-5bbcf85789c2","arxiv_id":"2601.13330","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RegCheck is an open-source, human-in-the-loop LLM tool that retrieves and compares registration/paper text along user-defined dimensions and issues yes/no/missing deviation judgments.","lead":"This preprint introduces RegCheck, an LLM-assisted tool that pulls the most relevant passages from a study registration and its published paper and flags whether they match. It aims to make routine registration checks cheap enough for reviewers, editors, and meta-researchers to actually do them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim assumes retrieval surfaces all dimension-relevant passages, but judgments are made only on top-k chunks; retrieval misses produce false 'no deviation' or 'missing' verdicts, and no recall evaluation is provided.","rationale":"The reader's weakest assumption—retrieval and LLM judgments must be reliable enough to orient human attention—is the same load-bearing point I identify. My attack sharpens it: because the LLM judge receives only top-k retrieved excerpts, any retrieval miss is invisible to the judgment. The 'missing' label is therefore not evidence of absence but absence of evidence in a truncated set. This makes the central claim in Section 1, that RegCheck 'automatically extracts the most relevant text' and produces deviation judgments, contingent on an unvalidated retrieval recall performance.\n\nThe paper is honest about this: Section 5 explicitly disclaims fidelity guarantees and Section 7 states anecdotal testing is insufficient. Those admissions weigh in the authors' favor, but they do not make the central claim supported. For a tool paper whose entire utility is providing a trustworthy orienting signal, the lack of any systematic retrieval evaluation is a real soft spot.\n\nI considered whether this should push the verdict to REJECT, but the architectural description is clear, the human-in-the-loop design mitigates downstream risk, and the authors state formal evaluation is underway. The paper can be conditionally accepted pending the promised evaluation or a clear statement that it is an unvalidated demo. Thus the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.\n\nCredit is due for the explicit caveats, the modular open-source plan, and the concrete evaluation methodology outlined in Section 7 (human-comparator agreement and artificial injected inconsistencies). Those are genuine steps toward validation. The concern is not that the authors are unaware of the issue, but that the central claim is currently unsupported and will remain so until recall-based evidence is presented.","tokens_in":10741,"tokens_out":3416,"duration_ms":36666,"concrete_test":"Construct a gold standard of 30 registration-paper pairs in which human annotators mark every passage relevant to the default dimensions and label any deviations. Run RegCheck with default settings. Measure recall@k of the retrieved excerpts per dimension against the annotated relevant passages, and the agreement of RegCheck's yes/no/missing labels with the human labels. Set a pass criterion: recall@k ≥ 0.90 and RegCheck-human agreement not worse than the human-human agreement measured on the same pairs. If retrieval recall falls below the threshold, the tool's orienting function fails in exactly the cases it is meant to catch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RegCheck's deviation judgments are generated from a bounded evidence set: for each dimension, dense retrieval selects top-k chunks (Section 2, stage 5) and the LLM sees only those excerpts. There is no completeness guarantee: if a passage that matters for the dimension scores below the retrieval cutoff, it never reaches the judge. A false 'no deviation' or 'missing' then results not from the documents but from the retriever. The 'missing' category is especially dangerous: it is an inference from absent retrieved evidence, not from an exhaustive search, so it will systematically conflate 'not retrieved' with 'not present.' The paper provides no recall@k evaluation, no top-k specification, and no mechanism to flag un-retrieved candidate passages, so the central orienting function—helping a human spot relevant text—is unsecured. Section 5 concedes no fidelity guarantees; Section 7 says anecdotal testing is not a substitute. This is not an internal inconsistency, but it means the central functional claim in Section 1 is currently an unsupported assertion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RegCheck, an open-source, LLM-assisted tool designed to help researchers, reviewers, and editors compare study registrations with corresponding papers. The tool's pipeline consists of six stages: ingestion, extraction, embedding, user-specified dimension definition, analysis (dense retrieval of relevant excerpts, LLM summarization, and a yes/no/missing deviation judgment), and reporting of structured, shareable outputs. The authors emphasize a human-in-the-loop design: users choose the comparison dimensions and are presented with direct quotes and LLM summaries for verification, rather than having the LLM replace human judgment. The paper describes the system's workflow, design principles, privacy considerations, availability, and outlines an ongoing program of formal evaluation, while also making anecdotal claims about the tool's accuracy.","tokens_in":10981,"tokens_out":2940,"duration_ms":33973,"significance":"If validated, RegCheck could substantially reduce the effort required for registration–paper consistency checks in peer review and metaresearch, potentially increasing the frequency of such checks. The system's modularity, discipline-agnostic framing, and transparent design are strengths, as is the authors' explicit acknowledgment that formal evaluation is pending. The open-source code and public app make the work immediately usable and falsifiable. However, the paper currently offers no empirical evidence for the tool's central functional claims of retrieval completeness, summarization fidelity, or judgment accuracy; those claims rest on unsystematic anecdotal impressions. The significance of the contribution therefore depends on whether those claims survive the ongoing evaluation.","major_comments":[{"comment":"The retrieval step is load-bearing: the LLM judge sees only the top-k chunks returned by dense embedding retrieval, and the 'missing' category is inferred from the absence of retrieved evidence, not from an exhaustive search. This design cannot distinguish 'not present' from 'not retrieved,' so false 'missing' or false 'no deviation' outcomes can arise from retriever misses. The paper does not report a recall@k figure, specify the value of k, or provide any sensitivity analysis with respect to chunk size/overlap. A retrieval-recall evaluation, or at minimum a mechanism to flag low-confidence retrieval, is necessary to support the claim that RegCheck reliably orients users to relevant passages.","section":"§2, Stage 5"},{"comment":"The paper states that 'our impressions of RegCheck, based on extensive-but-unsystematic testing, are that it is generally very accurate in its extractions and is quite adept at catching genuine deviations,' and that it 'frequently catches deviations that members of our research team missed.' No data are presented to substantiate these claims, and they are in tension with the adjacent disclaimer that 'we currently do not provide any guarantees about the fidelity of RegCheck’s output.' These are central functional claims; either they should be supported by a systematic evaluation (even a small pilot) or rephrased as provisional hypotheses pending the formal evaluation described in §7.","section":"§5"},{"comment":"The authors correctly state that 'anecdotal testing is not a substitute for formal evaluation,' and they describe an evaluation plan. However, the manuscript as submitted contains no formal or structured evaluation of any component—retrieval, summarization, or deviation judgment. For a paper whose central contribution is a tool claiming to facilitate registration–paper comparisons, the absence of any measured accuracy data (e.g., inter-rater agreement on a small set of documents, or a pilot with injected inconsistencies) leaves the central claim unvalidated. At minimum, a proof-of-concept evaluation on a small corpus would make the paper self-contained and allow readers to assess whether the design is fit for purpose.","section":"§7"}],"minor_comments":[{"comment":"The description of 'pre-set dimensions' is clear, but the list of default dimensions is not given in the text (only described as 'sample size, primary outcomes,' etc.). Including an explicit list or reference to the repository would help readers understand the intended scope.","section":"§2, Stage 4"},{"comment":"Minor typo: 'text served to its models' should likely be 'text sent to its models.'","section":"§6"},{"comment":"Typo: 'solicling' should be 'soliciting.'","section":"§7"},{"comment":"The privacy statement says 'no user information is logged.' If the web app keeps server logs (even transiently) or uses analytics, this should be clarified to avoid an overbroad claim.","section":"§4"},{"comment":"The screenshot is informative, but the color coding (red/blue/yellow) is described in the caption. Consider adding a legend within the figure itself for accessibility in grayscale printing.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is essentially a systems/software description with no empirical validation. Its acceptance hinges on whether the venue regards the paper as an infrastructure announcement or as a scientific claim about tool performance. The authors' candor about pending evaluation is commendable, but the unsupported accuracy claims in §5 are problematic and should either be removed or dramatically qualified. The retrieval-completeness issue is not merely a future-evaluation detail; it is a structural limitation that the authors should address, even if only by specifying top-k and providing recall estimates. With a focused revision that adds a small evaluation and tempers the claims, the paper could be suitable; in its current form it is not."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RegCheck is a real, working system that fills a real gap: registrations go unexamined because manual comparison is laborious, and this tool gives reviewers an orienting signal via user-defined dimensions, dense retrieval, per-dimension LLM judgments, and shareable reports. That combination is genuinely new and plausibly useful. The authors are also refreshingly honest that formal evaluation is pending and that they provide no fidelity guarantees. For a tool paper, that candor goes a long way.\n\nThe strongest part is the design philosophy: human-in-the-loop, non-prescriptive, with independent LLM calls per dimension so one dimension's outcome doesn't contaminate another. The privacy section is thoughtful. The anecdotal claims about catching subtle deviations are plausible and appropriately framed as impressions.\n\nThe soft spot that matters most is the retrieval bottleneck. Deviations are judged only on the top-k chunks for each dimension. If a relevant passage doesn't rank in the top-k, it never reaches the LLM, and the resulting 'no deviation' or 'missing' verdict is an artifact of retrieval, not an exhaustive search. The 'missing' category in particular risks conflating 'not retrieved' with 'not present.' The paper gives no recall@k evaluation and no way for users to see what was excluded. This doesn't sink the tool — it is still useful for orienting attention — but it undermines the stronger reading of the accuracy claims. The authors should either bound the recall problem explicitly, surface unretrieved candidates, or change the language so 'missing' is understood as 'not found in the retrieved evidence.'\n\nTwo smaller issues: the abstract promises an example use case that never appears in the body, and the text says the code is on GitHub but gives no repository URL or commit hash. Both are easy to fix.\n\nThe citation pattern looks fine. The background on registration prevalence and non-checking is solid. The self-citation about prompt sensitivity is relevant, not load-bearing. This is not a circular or overclaimed paper; it is an honest engineering report awaiting its evaluation.\n\nBottom line: a serious referee should engage this. The architecture is coherent, the domain gap is real, and the planned evaluation strategy — including comparison to human-human agreement — is sensible. I'd want the retrieval-recall issue addressed and the missing example added before accepting, but I'd send it to review today.\n\nFor the reading group: worth a session, mostly to discuss what 'validated' should mean for a tool like this.\n\nRecommendation: accept for peer review with requests for the example, the repo link, and a direct treatment of retrieval recall.","headline":"A useful, honestly-caveated tool paper for registration–paper checks; the top-k retrieval problem behind the 'missing' judgment is the main technical concern to resolve before publication.","tokens_in":11441,"tokens_out":1616,"would_cite":true,"duration_ms":18081,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RegCheck is an LLM-assisted pipeline that pulls the most relevant text from a study registration and its paper along user-chosen dimensions, returning candidate deviation judgments for a human to verify.","keywords":["study registration","preregistration","registration-paper comparison","large language models","embeddings","deviation detection","research trustworthiness","human-in-the-loop"],"falsifier":"A formal evaluation on a corpus with expert-established ground-truth deviations, comparing RegCheck's agreement with expert reviewers against expert-expert agreement, would settle the claim; if RegCheck's agreement falls significantly below the human-human baseline, or if it misses injected inconsistencies in artificially generated texts at high rates, the tool's orienting value would be undermined.","tokens_in":10634,"feed_emoji":"🔍","tokens_out":6488,"duration_ms":56035,"temperature":0.7,"pith_summary":"The paper presents RegCheck, a modular tool that makes registration-paper consistency checks far less laborious by automating the search for relevant evidence. For each user-defined comparison dimension, RegCheck embeds both documents, retrieves the most pertinent passages, and has a large language model summarize the passages and issue a yes/no/missing deviation judgment. The human remains the decision-maker: the tool's output is an orienting aid with direct quotes, not a verdict. The authors argue that lowering the effort barrier could make registration checks a routine part of peer review and self-review, where they are currently rare. The paper explicitly does not claim formal validation; it describes the workflow, design principles, and an ongoing evaluation agenda.","feed_headline":"Tool flags deviations between study registrations and papers","feed_subtitle":"RegCheck extracts key passages from registration and paper, flagging possible mismatches for a human to verify.","key_machinery":"The central mechanism is the dimension-driven retrieval-and-judgment loop. A user-supplied dimension label and definition are embedded and used to rank sentence-aware text chunks from both documents by cosine similarity; the top excerpts are sent to a user-selected LLM, which produces concise summaries and a yes/no/missing deviation judgement. This loop turns an unstructured document pair into a structured report, and its modularity (choice of parser, model, and dimensions) is what lets the tool claim discipline-agnostic applicability.","core_discovery":"RegCheck's central claim is that a modular pipeline of parsing, embedding, retrieval, and LLM judgment can convert the open-ended task of comparing a registration with its paper into a structured, auditable report. The user specifies or chooses dimensions; the system retrieves the most relevant excerpts from each source, produces LLM summaries with citation to excerpt IDs, and encodes a deviation judgment as yes, no, or missing. The design keeps human expertise in the loop by design—users define what matters and verify the evidence—and standardizes prompts and comparison logic across calls and users, countering the variability of ad hoc LLM use. The authors present this as infrastructure for","pith_inferences":["Inference: the 'missing' judgment could itself become a meta-science signal, flagging registrations that are too underspecified to be checked—a problem the paper notes in passing.","Inference: if such tools become common, the social cost of undisclosed deviations rises, which might push authors to report deviations proactively and reduce the stigma around legitimate changes of plan.","Inference: the same retrieval-plus-judgment architecture could generalize to other verification tasks such as checking whether a paper's claims are supported by its methods section, or whether a data appendix matches the text."],"forward_implications":["A reviewer could compare a registration and paper in minutes rather than hours, focusing on the exact passages relevant to each dimension.","Standardized prompting across calls and users would reduce the output variability that plagues ad hoc LLM comparisons.","The shareable report format with unique IDs would allow multiple reviewers to audit the same comparison, adding transparency to integrity checks.","Because human judgment is the final step, even imperfect automation could increase the frequency of registration checks relative to the current near-zero baseline.","If accuracy holds across disciplines, the same pipeline could be extended to other consistency checks, such as comparing analysis code to planned analyses."],"fun_headline_variants":["RegCheck: LLM tool for structured registration-paper checks","Human-in-the-loop tool checks registrations vs papers","RegCheck: AI-assisted audit of study registrations vs papers","Structured registration-paper comparisons with human verification"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that RegCheck's retrieval and LLM judgments are reliable enough that the passages and yes/no/missing labels it returns orient a human reviewer correctly rather than misdirect attention—something the paper explicitly states is not yet formally validated.","fun_headline_variants_meta":{"raw":{"variants":["RegCheck: LLM tool for structured registration-paper checks","Human-in-the-loop tool checks registrations vs papers","RegCheck: AI-assisted audit of study registrations vs papers","Structured registration-paper comparisons with human verification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3264,"prompt_tokens":763,"completion_tokens":2501,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2435}},"tokens_in":507,"tokens_out":2501,"duration_ms":19386,"temperature":1.0,"reasoning_tokens":2435,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:33:25.824581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A formal evaluation on a corpus with expert-established ground-truth deviations, comparing RegCheck's agreement with expert reviewers against expert-expert agreement, would settle the claim; if RegCheck's agreement falls significantly below the human-human baseline, or if it misses injected inconsistencies in artificially generated texts at high rates, the tool's orienting value would be undermined.","supporting_citations":[],"review_version":1}