{"id":"bfb0ec88-41c1-4056-ba36-4adf0767a4e7","arxiv_id":"2606.31292","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"AtomiMed is a new modality-agnostic evaluation framework for medical report generation that decomposes reports into hierarchical atomic clinical facts and applies agentic cross-verification to achieve higher correlation with radiologist judgments than n-gram metrics.","lead":"AtomiMed proposes a hierarchical framework that breaks medical reports into atomic clinical facts at disease and attribute levels and uses an agentic verification process to check consistency between predicted and ground-truth reports. A smart generalist might read it because better evaluation metrics could improve the reliability of AI systems that generate diagnostic reports from medical images.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Accuracy of automated hierarchical decomposition into Atomic Clinical Facts is assumed without reported expert validation of completeness or fidelity","rationale":"The reader's weakest_assumption directly identifies the same extraction-fidelity risk that would falsify the correlation results. Because the provided abstract supplies no inter-annotator or completeness metrics for the hierarchy, the concern remains load-bearing; the full-text experiments cannot be assessed for this gap without those numbers.","tokens_in":1734,"tokens_out":318,"duration_ms":30787,"concrete_test":"Sample 100 reports from OmniMRG-Bench (balanced across X-ray/CT/MRI/US); have two board-certified radiologists independently re-extract the same Disease+Attribute hierarchy using only the paper's published definition; compute token-level and fact-level F1 against the toolkit output. If mean agreement <0.85 on attribute descriptors, the decomposition step introduces measurable information loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (higher correlation with radiologist judgment) rests on the decomposition step producing a faithful multi-level hierarchy of Disease-level entities and Attribute-level descriptors. The abstract describes an Agentic Cross-Verification loop and MRGEvalKit for extraction, yet provides no quantitative check (e.g., inter-rater agreement between automated facts and independent radiologist re-annotation) that critical diagnostic information is neither omitted nor fabricated. If the hierarchy systematically drops severity modifiers or location details on certain modalities, the decoupled detection/descriptive scores and subsequent correlation numbers become unreliable even if the verification loop runs cleanly.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes AtomiMed, a modality-agnostic framework for evaluating medical report generation (MRG) that decomposes reports into a multi-level hierarchy of Atomic Clinical Facts (Disease-level entities and Attribute-level descriptors for location, morphology, severity). It employs an Agentic Cross-Verification loop to simulate multi-radiologist review for decoupled assessment of diagnostic detection and descriptive accuracy. The work also introduces the open-source MRGEvalKit toolkit for hierarchical extraction and the OmniMRG-Bench multi-modal benchmark (X-ray, CT, MRI, Ultrasound), claiming via experiments on expert-annotated reader studies that AtomiMed yields significantly higher correlation with radiologist judgments than traditional n-gram or model-based metrics.","tokens_in":1843,"tokens_out":525,"duration_ms":33073,"significance":"If the central claims hold after validation, AtomiMed could meaningfully advance MRG evaluation by moving beyond surface-level overlap to clinically grounded fact-checking, with the open-source toolkit and benchmark offering reusable infrastructure for the community. The decoupled scoring approach addresses a recognized gap in detecting catastrophic diagnostic errors.","major_comments":[{"comment":"Abstract: The central claim that 'AtomiMed achieves significantly higher correlation with human radiologist judgment compared to traditional and model-based metrics' supplies no quantitative results, dataset sizes, statistical tests (e.g., Pearson/Spearman coefficients, p-values), or implementation details, making it impossible to judge whether the data support the claim.","section":"Abstract"},{"comment":"Decomposition step and Agentic Cross-Verification loop: The framework's validity rests on the assumption that automated extraction into the Atomic Clinical Facts hierarchy faithfully captures clinical meaning without omitting or fabricating critical information (e.g., severity modifiers or location details). No quantitative check such as inter-rater agreement between the automated facts and independent radiologist re-annotation is reported; this is load-bearing for the subsequent decoupled scores and correlation results.","section":"Abstract / decomposition description"}],"minor_comments":[{"comment":"The abstract lists modalities covered by OmniMRG-Bench but provides no information on the number of reports, readers, or annotation protocol used in the expert studies.","section":"Abstract"},{"comment":"The GitHub link is provided, but the manuscript does not specify the exact versions of any underlying LLMs or extraction models used in MRGEvalKit.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback on our manuscript. We address each major comment below with clarifications and commitments to revision where the points identify gaps in the current presentation. Our responses focus on strengthening the evidence for the framework's validity without altering the core claims.","responses":[{"response":"We agree that the abstract should include key quantitative support for the central claim to allow readers to assess the strength of the results immediately. The body of the manuscript reports these details from the expert-annotated reader studies (including Pearson and Spearman coefficients, dataset sizes across modalities, and statistical significance), but they are not summarized in the abstract. In the revised version, we will update the abstract to incorporate the specific correlation values, sample sizes, and p-values.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central claim that 'AtomiMed achieves significantly higher correlation with human radiologist judgment compared to traditional and model-based metrics' supplies no quantitative results, dataset sizes, statistical tests (e.g., Pearson/Spearman coefficients, p-values), or implementation details, making it impossible to judge whether the data support the claim."},{"response":"We acknowledge that a direct quantitative validation of the automated hierarchical extraction (e.g., inter-rater agreement between the LLM-based decomposition and independent radiologist re-annotations on attributes such as severity and location) is not reported in the current manuscript. The primary validation comes from the end-to-end correlation of AtomiMed scores with radiologist judgments in the reader studies, which indirectly supports the decomposition quality. However, we agree this is a load-bearing assumption and will add a dedicated analysis in the revision, including agreement metrics (e.g., Cohen's kappa or F1 on fact extraction) computed on a held-out subset of reports re-annotated by radiologists.","revision_made":"yes","referee_comment":"[Abstract / decomposition description] Decomposition step and Agentic Cross-Verification loop: The framework's validity rests on the assumption that automated extraction into the Atomic Clinical Facts hierarchy faithfully captures clinical meaning without omitting or fabricating critical information (e.g., severity modifiers or location details). No quantitative check such as inter-rater agreement between the automated facts and independent radiologist re-annotation is reported; this is load-bearing for the subsequent decoupled scores and correlation results."}],"tokens_in":1427,"tokens_out":501,"duration_ms":26995,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper tries to improve evaluation of AI-generated medical reports by decomposing them into a multi-level hierarchy of atomic clinical facts and running an agentic loop to score detection and description separately.\n\nWhat is new is the standardized hierarchy that splits disease-level entities from attribute-level details like location, morphology, and severity, combined with the cross-verification step that treats the process like multi-radiologist review. They also ship MRGEvalKit for extraction and OmniMRG-Bench covering several modalities. Releasing code and a benchmark is useful even if the core idea needs testing.\n\nThe approach makes sense on paper because n-gram metrics really do miss clinically important errors, and decoupling the two scoring dimensions could give more actionable signals for model training. The claim of higher correlation with radiologist judgment comes from expert reader studies, which is the right kind of evidence to collect.\n\nThe soft spot is exactly the one the stress-test note flags. The whole system depends on the automated decomposition producing a faithful set of facts without dropping or adding critical information. The abstract does not report any quantitative check, such as inter-rater agreement between the extracted facts and independent radiologist annotations, on completeness or fidelity. If that step systematically misses severity or location details on certain modalities, the downstream correlation numbers lose their meaning. The reader studies are mentioned but without numbers, dataset sizes, or statistical tests visible here, it is difficult to judge how strong the evidence actually is.\n\nThis work is for researchers building or evaluating medical report generation models. A reader who wants concrete alternatives to surface metrics would get value from the framework description and the released toolkit. It is coherent enough on its own terms to deserve a serious referee who can examine the full experiments and the decomposition validation.","headline":"AtomiMed's hierarchical atomic facts plus agentic verification is a reasonable attempt to move past n-gram metrics for medical reports, but the missing validation on whether the decomposition itself is accurate is a real problem for the central claim.","tokens_in":2378,"tokens_out":447,"would_cite":false,"duration_ms":37228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"AtomiMed improves medical report evaluation by decomposing narratives into hierarchical atomic clinical facts and cross-verifying them to separate detection from description accuracy.","keywords":["medical report evaluation","atomic fact checking","clinical accuracy","hierarchical decomposition","agentic verification","radiologist correlation","multi-modal benchmark"],"falsifier":"A reader study on new reports where n-gram metrics or existing model-based scores show higher correlation with radiologist ratings than AtomiMed scores.","tokens_in":2617,"feed_emoji":"🩺","tokens_out":582,"duration_ms":30775,"temperature":0.7,"pith_summary":"Traditional metrics for medical report generation use n-gram overlap and miss clinical factual errors. AtomiMed decomposes reports into a standardized hierarchy of atomic facts at the disease level and attribute level, including location, morphology, and severity. An agentic cross-verification loop then checks consistency between ground-truth and predicted reports, simulating peer review. Experiments across expert reader studies show this yields higher correlation with radiologist judgments than prior metrics, and the method works across X-ray, CT, MRI, and ultrasound.","feed_headline":"Atomic fact hierarchy lifts medical report evaluation","feed_subtitle":"Decomposing reports into diseases and attributes catches diagnostic errors that n-gram metrics overlook and aligns better with radiologists.","key_machinery":"Hierarchical Atomic Clinical Facts (disease entities plus attribute descriptors) that enable decoupled scoring via an agentic cross-verification loop between reports.","core_discovery":"The paper establishes that breaking medical narratives into a multi-level hierarchy of Atomic Clinical Facts, with disease-level entities and attribute-level descriptors, followed by agentic cross-verification, decouples diagnostic detection accuracy from descriptive accuracy and produces evaluation scores that align more closely with human radiologist judgment than n-gram or model-based alternatives.","pith_inferences":["The atomic-fact breakdown could supply fine-grained reward signals for training report-generation models to reduce specific clinical errors.","The same decomposition approach might transfer to factual verification in other technical narrative domains such as legal summaries or engineering logs.","Automated extraction of the atomic facts from raw text could be tested for reliability as a standalone preprocessing step."],"forward_implications":["Allows separate measurement of whether a finding is present versus how accurately its attributes are described.","Applies the same evaluation protocol to reports from X-ray, CT, MRI, and ultrasound without modality-specific changes.","Produces scores that correlate more strongly with expert human judgment than surface-level overlap metrics.","Supplies an open toolkit and benchmark to run the evaluation consistently across research groups."],"fun_headline_variants":["Atomic clinical facts enable precise medical report evaluation","AtomiMed decomposes reports into disease and attribute facts","Cross-verification of atomic facts matches radiologist scores","Hierarchy of facts separates diagnostic and descriptive accuracy"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Breaking medical narratives into a fixed multi-level hierarchy of atomic facts captures all critical clinical information without omissions or inventions.","fun_headline_variants_meta":{"raw":{"variants":["Atomic clinical facts enable precise medical report evaluation","AtomiMed decomposes reports into disease and attribute facts","Cross-verification of atomic facts matches radiologist scores","Hierarchy of facts separates diagnostic and descriptive accuracy"]},"model":"grok-4.3","cost_usd":0.004419,"raw_usage":{"total_tokens":2200,"prompt_tokens":650,"num_sources_used":0,"completion_tokens":58,"cost_in_usd_ticks":44187000,"prompt_tokens_details":{"text_tokens":650,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1492,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":650,"tokens_out":58,"duration_ms":19369,"temperature":1.0,"reasoning_tokens":1492,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-01T03:24:27.951026+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A reader study on new reports where n-gram metrics or existing model-based scores show higher correlation with radiologist ratings than AtomiMed scores.","supporting_citations":[],"review_version":1}