{"id":"73b0168e-be7c-4f9e-98d4-c61043a66c16","arxiv_id":"2607.27845","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a new 8,790-instance benchmark built from Nature Communications review records, LLMs characterize reviewer concerns well (GPT-5.5: 0.754) but verify evidence-backed revision resolution poorly (best 0.501).","lead":"A new benchmark tests whether AI systems can tell if a revised scientific manuscript actually fixes what reviewers asked for, using the revised text as evidence. Modern LLMs can summarize reviewer concerns well, but the best system still fails verification half the time, scoring 0.501.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold labels are LLM-generated without human validation; low inter-model agreement on resolution evidence threatens the central verification-bottleneck claim.","rationale":"The reader identifies the same load-bearing concern: the gold labels are LLM-generated without human validation, and the low inter-model agreement on resolution evidence threatens the validity of the verification scores. This is indeed the most critical assumption behind the paper's central claim. My review of the manuscript confirms the reader's reading: Section 4.1 describes GPT-5.5 as the sole annotator of gold 'resolution' and 'evidence' fields; Appendix B.2 reports only inter-LLM agreement, not human agreement; and the verification component includes resolution evidence block IDs, which are exactly the field with lowest agreement. The paper's limitation section does not mention this issue, which makes it a genuine soft spot. The proposed concrete test—human annotation of a random subset and recomputation of verification scores—would directly settle whether the bottleneck claim is an artifact of label noise. Because the reader's verdict is already CONDITIONAL and this concern is already the basis for that verdict, no change is needed; the condition should be explicitly resolved before the quantitative bottleneck claim is fully trusted.","tokens_in":14825,"tokens_out":2480,"duration_ms":23262,"concrete_test":"Randomly sample 200 episode-level instances from the test split. Have two domain-expert human annotators independently assign the resolution label and the resolution evidence block IDs for each instance, using the same schema and manuscript/review context as the LLM annotators. Measure human-human agreement and human-vs-GPT-5.5 agreement. Then recompute the verification scores for Claude-Opus-4.8, GPT-5.5, and one open-source model (e.g., GLM-5.2) using an adjudicated human-labeled gold set for those 200 instances. If the verification scores change by more than 5 points on the subset, or if the gap between characterization and verification narrows materially, the central bottleneck claim is not robust to label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that evidence-based verification is the primary bottleneck (best verification score 0.501 vs. characterization 0.754)—rests entirely on the correctness of the gold labels, especially resolution labels and resolution evidence block IDs. These labels are produced by GPT-5.5 with no human validation (Section 4.1, Appendix B.2). The only quality check is agreement with Claude Opus 4.8, and on the key resolution-evidence signal the agreement is low: micro-F1 0.441, mean IoU 0.308, and Cohen's kappa for resolution label only 0.261. If the gold labels are noisy or systematically biased, the reported verification scores are not a faithful measure of model capability; they could be artificially depressed (if gold evidence blocks are inconsistently chosen) or artificially inflated (if labels are easy to guess). Since verification is the dimension on which the authors base their main qualitative conclusion, the lack of human validation is the single most load-bearing weakness. The paper does report a thoughtful agreement experiment, but agreement between two LLMs does not establish ground truth; it only shows the annotation task is not fully deterministic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AutoSupervision, a benchmark for concern-level, evidence-grounded verification of scientific manuscript revisions. Each instance contains a reviewer concern, the corresponding author response, and the revised manuscript; models must characterize the concern, determine whether it was resolved, and cite the supporting manuscript evidence. The benchmark is constructed from 56,000 Nature Communications transparent-review records, yielding 8,790 episode-level concern instances with resolution labels and evidence annotations. The authors evaluate a range of LLMs, an agentic pipeline, and a supervised fine-tuned model, reporting that characterization is relatively strong (GPT-5.5: 0.754) while verification remains the main bottleneck (best verification score 0.501, Claude-Opus-4.8). The paper concludes that current models struggle with evidence-based verification and grounding.","tokens_in":15146,"tokens_out":3406,"duration_ms":36154,"significance":"If the benchmark is valid, it addresses a genuinely underexplored capability in AI-assisted science: verifying whether feedback has been substantively incorporated into a revised manuscript. The task formulation is clean and the scale (56,000 source papers, 8,790 instances) is substantial. The paper provides a broad model comparison, an ablation study, an agentic baseline, and a supervised fine-tuning experiment, and it states a commitment to releasing data and code. The central weakness is the validity of the gold standard: labels and evidence annotations are produced by GPT-5.5 with no human validation, and inter-model agreement on the key evidence signal is low. Because the main conclusion—verification is the bottleneck—rests on these labels, the benchmark's usefulness depends on additional validation.","major_comments":[{"comment":"The gold-standard resolution labels and resolution evidence block IDs are generated by GPT-5.5 without human validation. The only quality check is agreement with Claude Opus 4.8, and this agreement is low for the most load-bearing signal: resolution evidence micro-F1 is 0.441 and mean IoU is 0.308; resolution-label Cohen's kappa is 0.261 (although exact agreement is 0.848, which suggests the low kappa is partly an imbalance artifact). Since the paper's central claim—that evidence-based verification is the primary bottleneck (best verification score 0.501 vs. characterization 0.754)—is computed against these labels, noisy or systematically biased gold evidence would directly undermine that conclusion. The authors should provide human validation on a random subset, or at least analyze how label noise bounds the reported verification scores.","section":"§4.1, Appendix B.2, Table 8"},{"comment":"There is a circularity risk: GPT-5.5 is used to construct the gold labels (concern extraction, resolution labeling, evidence annotation) and is then evaluated against those same labels. Its high characterization score (0.754) and grounding score (0.702) may partly reflect self-consistency with its own annotation behavior rather than true capability. The agreement study with Claude Opus 4.8 mitigates this concern but does not remove it, because both models may share systematic biases. The concern set is also fixed to GPT-5.5's output, so the coverage and characterization results are conditioned on that model's definition of what counts as a concern. Releasing the annotation prompts and adding at least a small human-annotated verification set would address this.","section":"§4.1, Table 2"},{"comment":"The dataset is constructed exclusively from accepted papers and further filtered to instances with auditable evidence, which likely overrepresents resolved concerns (8,265 of 8,790 instances are labeled resolved) and clean evidence structures. The paper acknowledges this limitation qualitatively, but it does not quantify how the filtering affects the difficulty or the reported bottleneck. For example, if the retained instances are those with explicit follow-up confirmations, the verification task may be artificially restricted, and the low verification scores could partly reflect label ambiguity rather than model deficiency. The authors should report the distribution of follow-up evidence types and assess whether the bottleneck conclusion is robust to excluding easy-to-ground instances.","section":"§6, §4.1, Table 6"}],"minor_comments":[{"comment":"Typo: 'Natural Communication' should be 'Nature Communications'.","section":"§6"},{"comment":"The term 'ReviseBench-style supervisor' is not defined or referenced; please clarify the protocol or provide a citation.","section":"§5.7"},{"comment":"The evaluation protocol states that concerns with the 'unverifiable' resolution label are excluded, but 'unverifiable' is not listed in Table 5's label options. Please align the schema and the protocol.","section":"Appendix A"},{"comment":"The characterization stage predicts 'response status' while the scored verification fields include 'paper status'; the relationship between these two fields should be clarified.","section":"Appendix C"},{"comment":"The reproducibility statement says the dataset and code are publicly released but no URL is provided in this version. Please include the link.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The central idea is timely and the resource could be valuable, but the validity of the gold standard is the core issue. The low inter-model agreement on resolution evidence, combined with zero human validation, makes the main bottleneck claim currently unsupported. I would accept a revised version that adds human validation on a sample, quantifies label noise, and re-checks the main conclusions under alternative gold-label treatments. The paper otherwise has solid experimental breadth and clear writing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the headline: this is a genuinely new benchmark task—concern-level, evidence-grounded verification of manuscript revisions—built from Nature Communications transparent review records. The dataset is large (8,790 instances), the evaluation is serious (frontier and open-weight LLMs, agentic pipeline, SFT, ablations, a closed-loop case study), and the paper is candid about most design choices. The clear finding that models characterize concerns much better than they verify resolutions is plausible and consistent with intuition. You should read it if you care about AutoResearch/AutoReview evaluation.\n\nWhat the paper does well: it defines a clean task that existing datasets (PeerRead, NLPeer, MOPRD, DISAPERE) don't cover—the full feedback loop from reviewer concern, through author response, to actual manuscript evidence. The grounding annotation is an advance over prior resources. The ablation study in Table 3 is informative: verification without the manuscript is nearly impossible, and grounding without the concern/response loses accuracy. The agentic and SFT results show concrete improvements. The authors also report an inter-model agreement experiment, which is more than many benchmark papers do.\n\nThe soft spots are real though. The gold labels are produced by GPT-5.5 with no human validation. The only quality check—agreement with Claude Opus 4.8—is weak on the exact signal that carries the central claim: resolution evidence block IDs (micro-F1 0.441, IoU 0.308) and resolution label (kappa 0.261). That means the 'verification is the primary bottleneck' conclusion rests on labels that are not stable across models. If the gold evidence IDs are inconsistently chosen, verification scores are depressed for reasons that have nothing to do with model capability. The authors deserve credit for running the agreement study, but agreement between two LLMs does not establish ground truth. The dataset is also filtered to accepted papers, with 94% resolved labels, which limits diversity—though the authors note this in the Limitations section.\n\nOne more practical note: the reproducibility statement claims dataset and code release, but the preprint contains no link. That makes the results non-verifiable as currently presented.\n\nBottom line: the task and dataset are worth taking seriously, and the paper deserves a real referee. But the authors should provide a human-validated subset (even a few hundred instances), release the data and code, and re-report the key numbers on that subset. The central quantitative claim is conditional until that is done. For a reading group, it's a useful case study in LLM-generated benchmarks—just bring the agreement table along.","headline":"New, useful benchmark for revision verification, but the core finding rests on LLM-generated gold labels with weak agreement—human validation needed before the bottleneck claim holds.","tokens_in":15614,"tokens_out":2080,"would_cite":true,"duration_ms":19084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoSupervision, a new benchmark built from 56,000 transparent peer-review records, argues that LLMs can characterize reviewer concerns well but fail at the critical step of verifying, with manuscript evidence, that a revision resolved them","keywords":["AutoSupervision","scientific revision verification","peer review","LLM evaluation","evidence grounding","revision episodes","language models","benchmark"],"falsifier":"Take a random sample of the benchmark's 8,790 concern instances and have human experts independently label resolution status and evidence blocks; if human labels agree with the LLM-generated labels at low rates, especially on 'resolved' versus 'partially resolved', the central bottleneck conclusion is not a valid measure of model ability.","tokens_in":14738,"feed_emoji":"🔁","tokens_out":5038,"duration_ms":46880,"temperature":0.7,"pith_summary":"The paper tries to establish that a new benchmark, AutoSupervision, can measure whether AI systems can verify that scientific manuscript revisions genuinely address reviewer concerns, using grounded evidence from the revised text. It argues that modern LLMs are already good at extracting and characterizing reviewer concerns (best score 0.754), but fail at the harder capability of verifying resolution and pointing to the exact evidence (best verification score 0.501). The authors build the benchmark from 56,000 article-review records, yielding 8,790 concern instances with resolution labels and evidence block annotations. If the benchmark is trustworthy, the practical consequence is that current AI research assistants can tell you what reviewers want but cannot yet close the feedback loop: they cannot reliably confirm that a revision actually fixed the problem.","feed_headline":"AI can name review concerns but not verify revision fixes","feed_subtitle":"The best model scores 0.754 on finding concerns but only 0.501 on confirming fixes with evidence.","key_machinery":"The central object is the revision episode: a pair containing the revised manuscript and the set of review-response records for that round. The benchmark defines three scored capabilities: characterization (comment kind, point type, target scope), verification (resolution label, paper status, resolution evidence blocks), and grounding (exact manuscript blocks supporting the concern). The load-bearing design choice is that resolution labels and evidence block IDs are treated as gold, produced by an LLM-assisted pipeline and checked only by inter-model agreement.","core_discovery":"The central claim is that concern-level, evidence-grounded revision verification is a distinct and currently unsolved capability for LLMs. The paper's main empirical discovery is that the best model reaches only 0.501 on verification while characterization reaches 0.754, and that adding manuscript context and retrieval improves grounding but not verification to the same degree. The benchmark treats a revision episode—a post-revision manuscript paired with its review-response records—as the unit of evaluation, requiring models to output structured predictions for characterization, verification, and grounding.","pith_inferences":["If the benchmark is valid, a natural next step not spelled out in the paper is to use the verification and grounding scores as training signals for research agents, turning public peer-review records into self-supervision for revision quality.","The paper's own agreement check shows low inter-model agreement on the key resolution-evidence signal (micro-F1 0.441, kappa 0.261 for the resolution label); if human annotation were added, the definition of 'resolved' might need to be refined before verification scores can be trusted as absolute measures.","Because the corpus comes from accepted papers, unresolved concerns are rare (only 91 of 8,790 instances); a testable extension would sample papers with outcome diversity or include rejected manuscripts to see whether the verification bottleneck persists on harder revision trajectories.","One could directly test the benchmark's utility by comparing downstream manuscript quality after feeding models different verifier outputs, using human judges; the paper's case study is a small version of this."],"forward_implications":["Current LLMs can be used to extract and characterize reviewer concerns at high accuracy in this setup, with the best model reaching 0.754.","Evidence-based verification is the primary bottleneck: even the strongest model scores only 0.501, well below the characterization level.","Supervised fine-tuning on the benchmark substantially improves grounding (from 0.451 to 0.643) and overall performance (from 0.463 to 0.614).","Adding retrieval and agentic decomposition improves grounding for a small model (from 0.085 to 0.336), but verification remains low.","A closed-loop case study suggests that stronger verifiers provide more useful author-facing revision feedback in a two-revision cycle."],"fun_headline_variants":["AI names review concerns but can't confirm evidence of fixes","LLMs excel at spotting issues, lag on verifying revisions","Grounded verification gap: AI knows what's wrong, not if it's fixed","AI spots review issues, but revision proof fails at 0.501","Concern spotting strong, fix verification weak for AI"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the AI-generated resolution labels and evidence block IDs used as gold are correct; the paper's own agreement check shows low agreement on the key resolution-evidence signal, so if those labels are wrong, the measured 'bottleneck' is an artifact of the annotation, not of the models.","fun_headline_variants_meta":{"raw":{"variants":["AI names review concerns but can't confirm evidence of fixes","LLMs excel at spotting issues, lag on verifying revisions","Grounded verification gap: AI knows what's wrong, not if it's fixed","AI spots review issues, but revision proof fails at 0.501","Concern spotting strong, fix verification weak for AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":2907,"prompt_tokens":691,"completion_tokens":2216,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":2128}},"tokens_in":435,"tokens_out":2216,"duration_ms":14025,"temperature":1.0,"reasoning_tokens":2128,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:20:47.436204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the benchmark's 8,790 concern instances and have human experts independently label resolution status and evidence blocks; if human labels agree with the LLM-generated labels at low rates, especially on 'resolved' versus 'partially resolved', the central bottleneck conclusion is not a valid measure of model ability.","supporting_citations":[],"review_version":1}