{"id":"89a79311-57ab-4e09-a82f-9c1aea855c11","arxiv_id":"2508.19319","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MedVQA-TREE fuses three levels of ultrasound image features with UMLS-guided PubMed retrieval to predict sarcopenia, reporting 99% accuracy on a 24-patient proprietary dataset.","lead":"A new multimodal system combines hierarchical ultrasound image analysis with PubMed knowledge retrieval to diagnose sarcopenia, reporting 99% accuracy on a small private dataset. The result is notable if valid, but the paper has unresolved data inconsistencies and no public code or external validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"99% accuracy claim rests on unverified image-level k-fold splitting with only 24 patients; without patient-grouped folds the result cannot be distinguished from patient-identity leakage.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing issue: the proprietary sarcopenia evaluation uses image-level statistics on a 24-patient dataset without evidence of patient-level splitting. If that concern lands, the headline 99% figure collapses because the model may be recognizing patients, not sarcopenia. The paper never supplies the necessary detail, and the dataset is not public, so the result cannot be independently audited. I agree with the reader's REJECT verdict: this is a critical, unresolved methodological gap. The public benchmark results (Table 2) actually undercut the 'outperformed previous SOTA by over 10%' claim—MedVQA-TREE scores 0.84 on PathVQA versus 0.91 for several baselines—so the claim rests almost entirely on the proprietary dataset, making the leakage concern even more consequential. The concrete test I propose is minimal: publish patient-level fold assignments or rerun with grouped folds. This single check would settle whether the near-perfect accuracy reflects generalization or leakage. Until then, the central claim is unsupported.","tokens_in":23301,"tokens_out":2282,"duration_ms":22589,"concrete_test":"Obtain the patient-to-fold assignment (or rerun the 5-fold cross-validation with folds defined by patient ID, ensuring no patient appears in more than one fold). If grouped accuracy remains ~0.99 across folds, the central claim withstands this critique. If it drops substantially (e.g., to the 0.70–0.80 range seen in Tables 3 and 7), leakage is confirmed. Additionally, rerun the grouped k-fold with the RAG component disabled to check whether per-patient retrieval features inflate performance independently of visual features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims 'up to 99% diagnostic accuracy' on a custom sarcopenia ultrasound dataset. Section 2.1 reports 3,474 images from only 24 patients, and Section 3.2.1 reports k-fold cross-validation results without ever stating that folds are grouped by patient. With ~145 images per patient on average, an image-level split places images from the same patient in both training and test folds, allowing the model to memorize patient-specific appearance and per-patient clinical variables (age, BMI, SPPB) rather than learning generalizable sarcopenia signs. The RAG module compounds this risk: retrieved text is generated from the same per-patient structured variables, so if split at image level, the retrieval features themselves leak patient identity. Table 11's 0.99 accuracy is therefore not evidence of diagnostic generalization unless patient-level grouping is explicitly confirmed. The concern is reinforced by internal data inconsistencies: transverse (1,350) + longitudinal (962) = 2,312, not 3,474, and the sex-stratified counts sum to 3,472, not 3,474, making the dataset description unreliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MedVQA-TREE, a multimodal framework for sarcopenia diagnosis from ultrasound images combined with structured clinical data and retrieved medical text. The architecture combines a three-level hierarchical visual encoder (global features, SAM-based regional features, superpixel spatial-graph features), a question-guided gated fusion mechanism with LoRA, and a UMLS-guided multi-query/multi-hop retrieval pipeline. The authors report experiments on VQA-RAD, PathVQA, and a proprietary sarcopenia ultrasound dataset, claiming up to 99% diagnostic accuracy and improvements over state-of-the-art medical VQA methods by more than 10%. Ablation tables show a progression from individual visual levels (accuracies roughly 0.52–0.70) to combined hierarchical features (0.90) and finally to RAG-enhanced fusion (0.99).","tokens_in":23601,"tokens_out":7230,"duration_ms":66319,"significance":"If the 99% accuracy result were valid, the paper would be a meaningful contribution to low-resource sarcopenia screening and to medical VQA in small-data regimes. The hierarchical visual reasoning, SAM-based region selection, and lightweight gated fusion are reasonable and potentially useful ideas. However, the central quantitative claim is not supported by the evaluation as presented: the k-fold cross-validation is performed on 3,474 images from only 24 patients with no described patient-level split, the dataset description is internally inconsistent, and the RAG module retrieves from the training set at test time. These issues mean the main empirical result cannot currently be distinguished from leakage or circularity, and the comparison claims are contradicted by the paper's own Table 2.","major_comments":[{"comment":"The cross-validation is applied to 3,474 images from only 24 patients, and the paper never states that folds are grouped by patient. With an average of roughly 145 images per patient, an image-level split places images from the same patient in both training and test folds, allowing the model to memorize patient-specific appearance and patient-level clinical variables (age, BMI, SPPB). Table 11's 0.99 accuracy is therefore indistinguishable from patient-identity leakage. The authors must report patient-exclusive folds, per-patient predictions, and fold-wise patient counts; all k-fold results in Tables 3–11 need to be re-run under that protocol or explicitly justified.","section":"§2.1, §3.2.1, Tables 3–11"},{"comment":"The dataset description is numerically inconsistent. Transverse (1,350) and longitudinal (962) image counts sum to 2,312, not 3,474. The sex-stratified counts sum to 3,472 (male: 763+1,240=2,003; female: 297+1,172=1,469). The plane-level sarcopenic/non-sarcopenic counts total 776/1,536=2,312, whereas the stated 30.53%/69.47% proportions applied to 3,474 images correspond to approximately 1,060/2,414 labels. Please correct these counts; the class-imbalance description and all reported metrics depend on them.","section":"§2.1"},{"comment":"The test-time RAG procedure is described as generating a query from the patient record and matching it with chunks 'aligned with the training set.' This is not external knowledge; it is retrieval from the training data. When combined with image-level splitting, the retrieved chunks can include the exact same patients' clinical variables. Even under patient-level splitting, matching test queries to training-derived topic categories creates a circularity in which predictions are informed by the training distribution. The near-perfect 0.99 result is a fitted outcome under this protocol. The authors should replace this with a genuinely external, label-independent knowledge base, or demonstrate that no information about training labels flows through the retrieval step.","section":"§3.2.1, Fusion Strategies with RAG"},{"comment":"The claim of outperforming previous state-of-the-art methods by over 10% is not supported by the reported numbers. On PathVQA, MedVQA-TREE achieves 0.84, below all listed baselines (0.90–0.91). On VQA-RAD it achieves 0.88 versus 0.87 for PeFoMed, a difference of about one point. The only >10% margin is on the sarcopenia dataset, whose validity is undermined by the issues above. Please restrict the claims to actual comparisons and provide confidence intervals or proper statistical tests for each dataset.","section":"Abstract, Table 2"}],"minor_comments":[{"comment":"The row labels '1-fold' through '5-fold' are ambiguous. Clarify whether these are the five folds of a 5-fold cross-validation and report the number of patients in each fold.","section":"§3.2.1"},{"comment":"These tables show high accuracy with very low F1/precision/recall (e.g., Table 12: accuracy 0.70–0.75, F1 0.12–0.19). Explain how this pattern arises and why accuracy alone is a meaningful headline metric in this setting.","section":"Tables 12–15"},{"comment":"The text mentions T5 as the underlying language model, but the formal model in Eqs. (1)–(8) does not include a T5 component. Clarify where T5 enters the architecture.","section":"§2.4.6"},{"comment":"The citation for PubMedBERT appears to point to a NEJM AI paper on BiomedCLIP rather than the original PubMedBERT publication. Please check the reference list and correct it.","section":"Table 1, References"},{"comment":"Figure 2's caption says 'image selected of VQA-RAD dataset', while the surrounding text describes the sarcopenia ultrasound pipeline. Clarify which dataset is actually used for the segmentation illustrations.","section":"Figures 2–3"},{"comment":"The paper refers to the sarcopenia task as 'VQA', but the sarcopenia experiments are binary image classification with clinical features and no natural-language question. Clarify how the gate uses the 'textual query' and whether any free-form question is posed to the model.","section":"§2.1, §3.1"}],"recommendation":"reject","confidential_remarks":"The empirical core of the paper cannot be trusted as written. The 99% accuracy claim is likely explained by image-level leakage over 24 patients, the dataset counts are self-contradictory, and the RAG design retrieves from the training set at test time. These are not mere presentation issues; they invalidate the main contribution. I would not be confident in any of the reported numbers without access to the code and a patient-level evaluation. A rejection is appropriate, though the authors could resubmit a substantially revised version with a fully corrected evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper is a plausible engineering assembly, but the headline result doesn't survive contact with the evaluation setup. The 99% accuracy on the sarcopenia dataset is almost certainly inflated by patient-level leakage: 24 patients, 3,474 images, and k-fold CV that never states folds are grouped by patient. With ~145 images per patient, image-level folds put the same patient in train and test, so the model can memorize identity. The RAG module makes it worse, since test queries are matched to \"training-aligned categories\" built from the training data.\n\nWhat's genuinely decent: the three-level visual hierarchy (global features, SAM regions, superpixel graphs), the question-guided gate with LoRA, and the UMLS-driven retrieval are sensible and clearly described. The VQA-RAD result (0.88) is competitive, and the LoRA ablation is a real check. The architecture is reasonable for low-data medical VQA.\n\nThe soft spots are large. The dataset description is internally inconsistent: transverse + longitudinal = 2,312, not 3,474, and the sex-stratified counts sum to 3,472. No baselines are run on the proprietary dataset, so the \"over 10%\" gain is only a comparison with prior art on public benchmarks, not on the same data. On PathVQA the model gets 0.84, below the 0.91 baselines listed. The RAG \"multi-hop\" is really multi-query expansion; there is no iterative reasoning loop. No code or data.\n\nNet: the central claim is not supported as reported. That said, the task is clinically relevant, the pipeline is plausible, and the evaluation problems are addressable in revision. A serious referee could ask for patient-grouped folds, corrected numbers, same-data baselines, and external validation. I would not trust the 99% figure, but I wouldn't desk-reject the paper either; it needs major revision with real evaluation hygiene.\n\nFor you: maybe bring to reading group as a leakage case study. I would not cite it.\n\nRecommendation: send to peer review with the expectation of heavy revision.","headline":"Plausible architecture but the 99% sarcopenia claim rests on image-level leakage and inconsistent dataset counts; needs major revision with patient-grouped evaluation.","tokens_in":24081,"tokens_out":2361,"would_cite":false,"duration_ms":20506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-level visual tree plus retrieval-augmented clinical knowledge pushes sarcopenia ultrasound diagnosis to 99% accuracy.","keywords":["MedVQA","Multimodal Reasoning","Sarcopenia Prediction","Retrieval Framework","Ultrasound diagnosis","Hierarchical visual features","Visual Question Answering","Retrieval-Augmented Generation"],"falsifier":"Re-run the full pipeline with patient-disjoint cross-validation—train on a subset of the 24 patients and test on the remaining patients, or leave one patient out at a time—and compare accuracy to the reported 0.99. If accuracy drops substantially, the near-perfect figure is explained by image-level leakage rather than by the hierarchical architecture or retrieval pipeline.","tokens_in":23249,"feed_emoji":"🩺","tokens_out":12414,"duration_ms":106099,"temperature":0.7,"pith_summary":"MedVQA-TREE is built around a clinical intuition: to diagnose sarcopenia from ultrasound, a model should look the way a clinician does—global view first, then the relevant muscle region, then the spatial relationships among small structures—and should be able to consult external medical knowledge when the image alone is ambiguous. The paper reports that this combination reaches up to 99% diagnostic accuracy on a proprietary sarcopenia ultrasound dataset (3,474 images from 24 patients) and 88% on the public VQA-RAD benchmark, beating the listed baselines by more than 10 points on the sarcopenia task. The mechanism is a question-guided gate that selects which of three visual levels to trust, a low-rank fusion module that combines that level with patient demographics and test scores, and a retrieval pipeline that expands patient variables into standardized clinical concepts via UMLS and filters external literature before fusing it into the prediction. The claim matters because sarcopenia is underdiagnosed and ultrasound is portable and cheap; if accurate, this design would make AI-assisted screening feasible outside CT and MRI settings.","feed_headline":"Three-level ultrasound model hits 99% accuracy in sarcopenia screening","feed_subtitle":"It fuses coarse-to-fine image features with patient data and external medical knowledge to beat prior medical image-question models.","key_machinery":"The load-bearing object is the three-level 'tree' of visual representations coupled to a question-guided gate and a retrieval-augmented fusion block. Level 1 is a global feature vector; Level 2 uses a pretrained segmentation model to isolate anatomically relevant muscle regions and extracts regional features; Level 3 builds a spatial graph whose nodes are centroids of superpixel-derived regions and whose edges encode proximity and anatomical adjacency. The gate is a softmax over the projected question text that selects exactly one level, and a low-rank adaptation module transforms the selected vector before it is combined with text and numeric features for classification. The retrieval side","core_discovery":"The paper's central claim is that sarcopenia can be classified from rectus femoris ultrasound with near-perfect accuracy when image interpretation is structured hierarchically and augmented by retrieved clinical knowledge. The vision module extracts three complementary representations: coarse global features, region features from a segmentation model, and node embeddings from a superpixel-based spatial graph that encodes anatomical adjacency. A soft gating function, computed from the projected question text, selects the most relevant level; the chosen features pass through a low-rank adapter and are added to the projected text and numeric representations before classification. In parallel, t","pith_inferences":["Editorial extension: the sharp jump from 0.93 to 0.99 when retrieval is added should be tested for leakage—since queries are generated from each patient's own clinical variables, matching could occur between a test patient's profile and similar text seen in training, independent of the image.","Editorial extension: the paper never states whether its k-fold splits group images by patient; with 24 patients and 3,474 images, patient-disjoint cross-validation is the single most informative next experiment for separating memorization from generalization.","Editorial extension: the coarse-to-fine visual tree plus ontology-guided retrieval could transfer to other under-labeled musculoskeletal ultrasound tasks, such as frailty or muscle-quality assessment, where cues are subtle and clinical context is essential.","Editorial extension: a reader should read the 99% as domain-specific; the same architecture reports 0.84 on PathVQA, below several listed baselines, so the contribution is best understood as a low-data sarcopenia-ultrasound method rather than a universal medical-VQA winner."],"forward_implications":["The reported 99% on the custom ultrasound dataset points toward bedside screening: a portable acquisition device plus this lightweight model could reduce reliance on costly CT or MRI for sarcopenia assessment.","The ablation progression (coarse ~0.70, fine ~0.68, fused hierarchy ~0.90, fused with clinical data ~0.93, with retrieval ~0.99) implies that no single visual level is sufficient and that external knowledge retrieval is a central ingredient, not a marginal add-on.","Because the gate processes only the selected visual level, the framework could adapt its computation to the question, an efficiency property that matters for community clinics with limited hardware.","On VQA-RAD the model reaches 0.88, beating the listed baselines by a small margin; the same hierarchy may transfer to other low-data medical VQA tasks, though the PathVQA result of 0.84 shows the transfer is not uniform."],"supporting_citations":[{"why":"Supplies the sarcopenia definition and criteria used to label patients in the custom ultrasound dataset.","marker":"[1]"},{"why":"Provides VQA-RAD, the public radiology question-answering benchmark used for the first generalization test.","marker":"[31]"},{"why":"Provides PathVQA, the public pathology question-answering benchmark used as a second external generalization test.","marker":"[32]"},{"why":"Supplies the API used to fetch external biomedical abstracts for the patient-derived queries.","marker":"[33]"},{"why":"Supplies the segmentation model used to produce Level 2 regional features and to help define the superpixel graph at Level 3.","marker":"[38]"},{"why":"PubMedBERT is one of the biomedical text encoders used to embed and rank semantically filtered sentences before fusion.","marker":"[40]"},{"why":"BioBERT is another biomedical encoder used in the semantic filtering stage and included in the model-selection comparison.","marker":"[41]"},{"why":"BioLinkBERT, pretrained with citation links, supports the multi-hop and concept-connecting queries the paper emphasizes.","marker":"[42]"}],"fun_headline_variants":["Sarcopenia ultrasound model hits 99% with hierarchical visual reasoning","Hierarchical vision boosts sarcopenia ultrasound accuracy to 99%","Ultrasound sarcopenia AI: coarse-to-fine features plus knowledge hits 99%","99% sarcopenia detection via multimodal retrieval and hierarchical vision","Sarcopenia AI: gated fusion and UMLS retrieval achieve 99% accuracy"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported accuracy treats each ultrasound image as an independent sample even though the 3,474 images come from 24 patients; if the cross-validation folds are not grouped by patient, images from the same person can appear in both training and test data, and the model can memorize patient appearance instead of learning generalizable signs of sarcopenia.","fun_headline_variants_meta":{"raw":{"variants":["Sarcopenia ultrasound model hits 99% with hierarchical visual reasoning","Hierarchical vision boosts sarcopenia ultrasound accuracy to 99%","Ultrasound sarcopenia AI: coarse-to-fine features plus knowledge hits 99%","99% sarcopenia detection via multimodal retrieval and hierarchical vision","Sarcopenia AI: gated fusion and UMLS retrieval achieve 99% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000354,"raw_usage":{"total_tokens":1740,"prompt_tokens":697,"completion_tokens":1043,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":946}},"tokens_in":441,"tokens_out":1043,"duration_ms":7532,"temperature":1.0,"reasoning_tokens":946,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:00:47.351477+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full pipeline with patient-disjoint cross-validation—train on a subset of the 24 patients and test on the remaining patients, or leave one patient out at a time—and compare accuracy to the reported 0.99. If accuracy drops substantially, the near-perfect figure is explained by image-level leakage rather than by the hierarchical architecture or retrieval pipeline.","supporting_citations":[{"cited_title":"Dupont, et al., Revised sarcopenia consensus: are we missing the preclinical stage? online comment on, Age And Ageing 48 (1) (2018) 1–1","cited_arxiv_id":null,"evidence_quote":"Supplies the sarcopenia definition and criteria used to label patients in the custom ultrasound dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides VQA-RAD, the public radiology question-answering benchmark used for the first generalization test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the API used to fetch external biomedical abstracts for the patient-derived queries."},{"cited_title":"Kirillov, E","cited_arxiv_id":null,"evidence_quote":"Supplies the segmentation model used to produce Level 2 regional features and to help define the superpixel graph at Level 3."},{"cited_title":"Zhang, Y","cited_arxiv_id":null,"evidence_quote":"PubMedBERT is one of the biomedical text encoders used to embed and rank semantically filtered sentences before fusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BioBERT is another biomedical encoder used in the semantic filtering stage and included in the model-selection comparison."}],"review_version":1}