{"id":"2a985c75-0dc2-44df-9248-d712efc30127","arxiv_id":"2508.08876","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A span-level model scores radiology report quality by weighting the revised spans between junior and senior reports and merging those weights into one QA score.","lead":"This paper introduces Sqator, a model that scores the quality of Chinese radiology reports by measuring the importance of the text spans a senior doctor changed in a junior doctor's report. If it works, it could save senior radiologists time by automating report quality review.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sqator requires the senior report as an input to extract revised spans; without it, the automatic QA claim is unsupported.","rationale":"The reader's weakest assumption correctly identified that the method depends on senior reports at inference. My stress test agrees, and I found no additional load-bearing concern beyond what can be gleaned from the abstract. The paper is abstract-only, so the verdict remains UNVERDICTED; the concern reinforces the need for the full text to clarify the inference setting and the circularity risk. If the full paper shows that senior reports are not required at test time (e.g., via a generated reference), the concern would be resolved. If it confirms that senior reports are required, the central claim should be revised to describe a retrospective auditing task, not automatic QA. Until then, no change to the reader's verdict is warranted.","tokens_in":734,"tokens_out":2554,"duration_ms":28620,"concrete_test":"Retrieve the full paper and check the system architecture and evaluation protocol. Specifically: (1) Does the model take the senior report as an input feature at inference? If yes, run a test where senior reports are withheld and replaced by an independent model's predicted senior report; if performance drops significantly, the 'automatic' claim fails. (2) Examine the importance-score consistency experiment: were the senior doctors who rated span importance blinded to the QA scores they had assigned? If not, re-run with separation to test circularity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract defines the task as automatically scoring junior reports, yet the method 'measures QA scores by measuring the importance of revised spans between junior and senior reports.' This implies that at inference the senior report — the document that encodes the expert correction — must be available as a reference. If it is not available, which is the normal prospective QA setting, the method cannot be applied. If it is available, the task shifts to retrospective auditing, which is hardly 'automatic QA.' The abstract gives no indication of how senior reports are obtained at test time. Additionally, the claim that importance scores are consistent with senior judgments risks circularity, because the same senior doctors who author the senior reports (from which revised spans are derived) also assign the QA scores and presumably provide the importance judgments. Without a clear statement of the inference-time inputs and a blinded evaluation of the consistency claim, the central result is uninterpretable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Sqator, a span-level framework for automatically scoring the quality of junior radiology reports. The method first identifies text spans that were revised when a senior report is compared with the corresponding junior report, assigns importance weights to those revised spans, and merges the span-level scores into a final QA score. The authors claim to evaluate on 12,013 radiology reports and report 'competitive QA scores' and that the importance scores are 'consistent with the judgments of senior doctors.' The abstract contains no quantitative results, no baseline comparisons, no description of the weak supervision signal, and no statement about the inputs required at inference time.","tokens_in":882,"tokens_out":1955,"duration_ms":20641,"significance":"If the method works as stated and the consistency claim is not circular, fine-grained span-level QA could provide interpretable, automatic feedback that reduces the workload of senior radiologists and improves training. The focus on revised spans between junior and senior reports is a plausible and potentially useful idea for retrospective auditing and educational feedback. However, the abstract, which is the only provided manuscript text, does not provide enough evidence to assess the technical validity or practical applicability of the method. The lack of any quantitative result or baseline makes the significance claim unverifiable from the submitted material.","major_comments":[{"comment":"The abstract states that Sqator 'measures QA scores by measuring the importance of revised spans between junior and senior reports.' This implies that the senior report, which is the product of expert review, is an input to the method. The abstract also claims the system can 'mark QA scores automatically.' If the senior report is not available at test time, the method cannot be applied in the prospective QA setting. If it is available, the task is retrospective auditing, not automatic QA. The abstract must clarify the inference-time inputs and the intended deployment scenario. This is load-bearing for the central claim.","section":"Abstract"},{"comment":"The claim that Sqator 'can achieve competitive QA scores' is unsupported by any quantitative result, baseline name, or measure of variance. The abstract mentions a collection of 12,013 reports but gives no metric values, no comparison methods, and no significance tests. Without these, the claim is uninterpretable. The manuscript should either include concrete results in the abstract (if the venue permits) or, if this is an abstract of a longer paper, the full paper must present the actual numbers, baselines, and error analyses.","section":"Abstract"},{"comment":"The statement that 'the importance scores of revised spans can be also consistent with the judgments of senior doctors' risks circularity. The senior doctors write the senior reports from which revised spans are derived, and they also provide the QA scores used for evaluation. The abstract does not describe any blinding or held-out procedure that would break this dependency. If the importance model is trained or validated using the same senior doctors' judgments, then consistency is expected by construction. The authors should explain the provenance of each supervision signal and any steps taken to avoid circular evaluation.","section":"Abstract"},{"comment":"The title announces 'Weakly Supervised' learning, but the abstract never identifies the weak supervision signal. Is it the revision difference between junior and senior reports? Is it the aggregate QA score? Without specifying what is weakly supervised and how the weak labels are generated, the methodological contribution is unclear. This is a central omission because the proposed framework's novelty appears to lie in the span-level supervision strategy.","section":"Title/Abstract"}],"minor_comments":[{"comment":"The acronym 'Sqator' appears without a clear definition of what the 'S' stands for (Span-level? Quality Assurance EvaluaTOR?). Consider spelling out the name in the abstract or defining it more clearly.","section":"Abstract"},{"comment":"The phrase 'the QA scores may be inaccurate for reasons like diagnosis bias, the ability of senior doctors, and so on' is vague. 'Ability' is presumably 'skill' or 'expertise'; the sentence structure is awkward. This is a presentation issue, not a substantive one.","section":"Abstract"},{"comment":"No mention of related work or the specific weaknesses of 'common document-level semantic comparison method.' Without a baseline description, the abstract does not convey the state of the art or the claimed improvement.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The submission is an abstract-only manuscript, which makes full technical evaluation impossible. The central claims are not supported by the material provided. I would encourage the editor to require the full paper or an extended abstract with experimental details before further review. The concern about the inference-time availability of senior reports is not merely a presentation issue; if the method requires the senior report as input, the 'automatic QA' framing is misleading. The circularity concern about senior doctors' consistency is also substantive and should be addressed explicitly in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know going in. First, this is an abstract-only review, so the verdict rests on what the abstract shows and doesn't show. Second, the abstract itself reveals a structural ambiguity: Sqator scores a junior report by measuring the importance of revised spans between the junior and senior reports. That phrasing says the senior report is an input at scoring time. If that's true, you aren't building an automatic prospective QA grader; you're building a retrospective auditor that needs the gold-standard senior report in hand. The abstract doesn't tell us how senior reports are supplied at inference, and that's not a minor footnote—it determines whether the method addresses the workload problem it claims.\n\nCredit where due: moving from whole-document semantic comparison to span-level importance weighting is a sensible shift for this task, and the idea that revised spans carry explainable QA signal is plausible. The 12,013-report corpus is a reasonable size. The claim that importance scores match senior judgments is testable and would be useful if shown cleanly.\n\nBut the evidence in the abstract is thin. No numbers, no baselines, no variance measures, no error analysis, and no description of the weak supervision signal. 'Competitive' is doing no work. There's also a real circularity risk: senior doctors write the senior reports, provide the QA scores, and presumably supply the importance judgments. If the consistency evaluation isn't blinded, agreement with senior doctors is partly by construction. That concern isn't a stretch from the abstract; it's the natural reading.\n\nI'm not saying the paper is unsound—the full text could address all of this. But right now the central claim is uninterpretable until the inference-time inputs are specified and the consistency claim is shown to be independent of the label source.\n\nThis is for people in clinical NLP and radiology QA who want a finer-grained alternative to document similarity. I would send it to peer review—the idea is worth referee time—but the authors need to clarify whether they're building an automated grader or an auditing tool, and they need to show that the importance scores aren't just reflecting the same doctors' judgments in a closed loop.","headline":"The abstract sells an automatic QA grader, but the method as described needs the senior report at inference, so the real contribution is a retrospective span-importance audit.","tokens_in":1371,"tokens_out":2457,"would_cite":false,"duration_ms":23220,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tries to establish that a junior radiology report's QA score can be computed from the importance-weighted text spans revised in the corresponding senior report, and that the proposed Sqator system does this competitively on 12,01","keywords":["radiology report QA","span-level evaluation","weakly supervised learning","revised spans","Chinese radiology","clinical NLP","quality assurance","text span importance"],"falsifier":"Take junior reports where the senior correction consists of adding a single critical finding (e.g., 'right pneumothorax') with no other edits; if the aggregated span-importance score does not classify these as severely unqualified, the decomposition fails. More directly, score junior reports with the senior reference withheld; if senior-doctor QA scores do not correlate with Sqator's revision-based predictions, the central assumption is falsified.","tokens_in":625,"feed_emoji":"🩻","tokens_out":4794,"duration_ms":46914,"temperature":0.7,"pith_summary":"This paper tries to show that the quality-assurance (QA) score a senior radiologist would give to a junior-written radiology report can be computed automatically from the specific text spans the senior would revise. The proposed system, Sqator, finds those revised spans between the junior report and a senior reference report, assigns an importance weight to each span, and merges the weights into a final QA score. The authors report that on 12,013 Chinese radiology reports this span-level approach matches the accuracy of document-level semantic comparison while producing interpretable, span-level importance scores that line up with senior doctors' judgments. If this holds, radiology QA could shift from labor-intensive full-report reviewing to targeted, weighted edit detection, making scoring faster, more consistent, and more instructive.","feed_headline":"Revised spans predict junior radiology report scores","feed_subtitle":"A weakly supervised model matches senior doctors on 12,013 Chinese reports and shows exactly what to fix.","key_machinery":"The central mechanism is the revised-span decomposition: a junior report's QA quality is represented by the set of textual spans that a senior's corrected version changes, each weighted by an importance score. Sqator is the named system: it detects these spans, estimates their importance in a weakly supervised way, and merges the weighted span scores into a single QA score. The revised span carries the argument because it is the unit that connects a textual difference to a clinical judgment—bypassing document-level similarity, which can miss small but critical omissions.","core_discovery":"Sqator's central claim is that a report's QA score can be decomposed into the importance-weighted contributions of the fine-grained spans that differ between the junior report and the senior reference. Rather than comparing whole documents, Sqator first identifies revised spans—short phrase-level segments where the senior text diverges from the junior text—then scores the importance of each revision and aggregates these scores into the final QA grade. The method is weakly supervised: it does not require manual per-span importance labels for every revision. Evaluated on a collection of 12,013 radiology reports, Sqator achieves QA scores competitive with existing approaches, and the per-span i","pith_inferences":["The method's reliance on a senior reference report means it measures 'revision quality' rather than standalone report quality; a natural but unstated extension is to train a reference-free scorer using the learned span-importance statistics.","Span-importance scores could double as an error taxonomy: linking revised spans to structured findings (missed finding, wrong descriptor, typo) could turn QA into automatic error categorization.","The same span-revision idea may transfer to other clinical documents (discharge summaries, operative notes) that have gold-standard revisions, though the paper only evaluates Chinese radiology reports.","A testable extension: feed the importance-weighted spans into other downstream tasks (report retrieval, auto-completion) to see whether span importance captures clinically meaningful salience beyond QA."],"forward_implications":["If QA score is determined by importance-weighted revised spans, then quality issues can be localized to specific phrases, giving junior doctors concrete feedback instead of a single score.","Automated QA on 12,013 reports at competitive accuracy suggests senior-radiologist workload in routine QA could be reduced, freeing time for difficult cases.","The consistency of span importance scores with senior judgments implies the method could serve as a second reader or calibration tool for human scoring.","Because the approach needs only the junior/senior report pair, it can exploit existing revision histories for training without expensive manual annotation."],"supporting_citations":[],"fun_headline_variants":["Revised spans predict junior report scores","Weakly supervised span scoring matches senior docs","Pinpoint report flaws via revised span scoring","How junior reports are graded by revised spans","Edit-span importance predicts radiology QA scores"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"A junior report's QA score is fully determined by the importance-weighted set of text spans revised in the corresponding senior report, and the senior report is available at scoring time.","fun_headline_variants_meta":{"raw":{"variants":["Revised spans predict junior report scores","Weakly supervised span scoring matches senior docs","Pinpoint report flaws via revised span scoring","How junior reports are graded by revised spans","Edit-span importance predicts radiology QA scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00082,"raw_usage":{"total_tokens":3404,"prompt_tokens":698,"completion_tokens":2706,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":2642}},"tokens_in":442,"tokens_out":2706,"duration_ms":21950,"temperature":1.0,"reasoning_tokens":2642,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:19:30.122361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take junior reports where the senior correction consists of adding a single critical finding (e.g., 'right pneumothorax') with no other edits; if the aggregated span-importance score does not classify these as severely unqualified, the decomposition fails. More directly, score junior reports with the senior reference withheld; if senior-doctor QA scores do not correlate with Sqator's revision-based predictions, the central assumption is falsified.","supporting_citations":[],"review_version":1}