{"id":"f2d9b64c-0867-417b-b077-258162ed67f9","arxiv_id":"2505.17619","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CAS-IQA predicts three quality scores for synthetic X-ray angiographies using a vision-language model with multi-image fusion and routing, and outperforms prior IQA methods on the new CAS-3K benchmark.","lead":"A new vision-language model, CAS-IQA, scores the quality of synthetic X-ray angiographies by comparing them against real contrast and non-contrast images. The same work introduces CAS-3K, a 3,565-image benchmark with three clinically motivated quality scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'significantly outperforms' claim rests on PLCC gaps of 0.003–0.029 without confidence intervals, and the OQ gain is computed against MA-AGIQA, which is not the second-best baseline in Table 1; these gaps may be within noise.","rationale":"The reader correctly flags the ground-truth MOS as a validity risk, and I agree that clinician-rated subsets would strengthen the paper. However, the most load-bearing concern for the central claim 'CAS-IQA significantly outperforms all baselines' is that the reported numeric margins are tiny and presented without any uncertainty quantification. With 715 test images, a PLCC difference of 0.0033 (VMC) or even 0.0289 (OQ) is plausibly within sampling noise; no confidence intervals, significance tests, or repeated splits are provided. The internal inconsistency in Section 5.2 - identifying MA-AGIQA as the second-best method for OQ when Table 1 shows LoDa is better - further undermines the claimed 'considerable margin'. The statistical concern is more direct than the rater-expertise concern because it attacks the existence of the reported superiority, not just its clinical interpretation. I found no critical technical error in the method itself: the MUST module is well-defined, the ablation supports its contribution, and the train/test split sizes (2850 and 715, multiples of 5) are consistent with a clean pair-level split, avoiding obvious content leakage. Given these issues, the paper's verdict should remain CONDITIONAL, requiring bootstrap confidence intervals and corrected, consistent reporting of the gains before the superiority claim is accepted.","tokens_in":10255,"tokens_out":12002,"duration_ms":108196,"concrete_test":"Bootstrap the test set: for each of 10,000 resamples of the 715 test images, compute PLCC/SRCC for CAS-IQA and for the actual best baseline per metric (LoDa for OQ, MA-AGIQA for VMC/VBD) and the differences; report 95% percentile CIs. If any CI includes 0, the 'significantly outperforms' claim is unsupported. Also recompute the Section 5.2 percentage improvements against the correct second-best baseline and verify them against Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 5.2) is that CAS-IQA significantly outperforms all baselines on CAS-3K. The supporting evidence is fragile in two concrete ways. First, the reported improvements are small in absolute terms: VMC PLCC 0.6925 vs MA-AGIQA 0.6892 (delta=0.0033), VBD delta=0.0132, OQ delta=0.0289 vs MA-AGIQA. With n=715, the approximate 95% CI for a single correlation is +/-0.07, so differences of 0.003-0.029 are within sampling noise; the paper reports neither confidence intervals nor significance tests. Second, the paper states PLCC improvements of 0.5%, 2.0%, and 4.1% over 'the second-best method, MA-AGIQA', but in Table 1 the second-best OQ PLCC is LoDa (0.6899), not MA-AGIQA (0.6696). Using the paper's own relative formula, the gain over LoDa is ~1.2%, not 4.1%, so the headline margin for OQ is overstated. The MOS-rater validity concern (Section 3.2) is important for clinical relevance, but it is not the most direct threat: even if the ground truth were perfect, the current analysis does not demonstrate that the performance gaps are statistically reliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CAS-IQA, a vision-language-model-based framework for assessing the quality of synthetic X-ray angiographies, and CAS-3K, a dataset of 3,565 synthetic angiographies generated from 713 Mask-Contrast pairs by five generative models. Three task-specific metrics are defined (VMC, VBD, OQ), and a Multi-path feature fusion and routing (MUST) module is proposed to combine Mask, Generated, and Contrast visual tokens and route them to metric-specific branches. On a single train/test split of CAS-3K, CAS-IQA reports the highest PLCC and SRCC among the listed baselines on all three metrics, and ablation results attribute part of the gain to MUST. The paper claims this is the first IQA dataset for medical AIGC.","tokens_in":10618,"tokens_out":9989,"duration_ms":111218,"significance":"If the results hold, the paper would fill a real gap: there is currently no standard benchmark for quality assessment of synthetic angiography, and the proposed VMC/VBD/OQ metrics are clinically motivated. The use of auxiliary Mask and Contrast images as references is a plausible and useful departure from single-image IQA, and the evaluation covers a broad set of handcrafted, DNN-based, and VLM-based baselines. However, the core empirical claims rest on a single split with no confidence intervals or significance tests, the ground-truth MOS comes from three student raters without expert validation, and the dataset/model are not yet publicly available. These issues currently limit the strength of the claims to conditional rather than established.","major_comments":[{"comment":"The paper claims PLCC improvements of 0.5%, 2.0%, and 4.1% for VMC, VBD, and OQ, respectively, over the second-best method, MA-AGIQA. This is inconsistent with Table 1: for OQ, the second-best PLCC is LoDa (0.6899), not MA-AGIQA (0.6696), and the relative gain over LoDa is roughly 1.2%, not 4.1%. Moreover, the raw deltas are 0.0033 (VMC), 0.0132 (VBD), and 0.0289 (OQ vs MA-AGIQA), and no confidence intervals, significance tests, or repeated-seed results are provided. With n=715, the sampling uncertainty of a correlation is on the order of 0.04 in Pearson units, so the smaller gaps may be within noise. The central claim of significant superiority needs bootstrap or permutation evidence, and the baseline identification must be corrected.","section":"§5.2, Table 1"},{"comment":"The ground-truth MOS are produced by three graduate students in biomedical engineering after a short training session, with board-certified radiologists involved only in defining the metrics. The 0.7 inter-rater correlation threshold measures agreement among the students, not agreement with expert clinical judgment. Since the value of the dataset and model is explicitly stated as clinical relevance, the paper should validate a subset of student ratings against ratings by interventional radiologists, or at least report the students' agreement with the radiologists who helped define the metrics. Without such validation, every model comparison inherits unvalidated labels.","section":"§3.2"},{"comment":"The train/test split is described only as 2,850 training images and 715 test images, with no statement about whether the split is disjoint at the level of the 713 source Mask-Contrast pairs. Because each source pair generates five synthetic images, an image-level random split can place near-duplicate content in both training and test, inflating measured performance, especially for a method that uses Mask and Contrast as auxiliary inputs. The authors should report a pair-disjoint (or at least generator-stratified) split and, if the current results were obtained with overlapping source pairs, re-run the comparisons under a clean split.","section":"§5.1"},{"comment":"The ablation of MUST is confounded with the prompt/inference format. The w/o MUST condition uses a single prompt asking for all three quality levels at once, while the w/ MUST condition uses three separate metric-specific prompts and branch-specific tokens. The consistent improvements in Table 2 could therefore come from the multi-turn or multi-branch prompting scheme rather than from the fusion and routing operations attributed to MUST. An additional control, such as three separate prompts without the routing/fusion mechanism, is needed to isolate the effect of the MUST module.","section":"§5.3, Table 2"}],"minor_comments":[{"comment":"There are several typos in the text: 'sore estimation' should be 'score estimation' in Fig. 2(b), 'collaborat' should be 'collaborate' in §3.1, and 'imgae' should be 'image' in the prompt shown in §5.3.","section":"Fig. 2 and §3.1"},{"comment":"The claim that CAS-3K is 'the first IQA dataset for medical AIGC' should be softened to 'to the best of our knowledge' and supported by a more systematic literature search, since the current statement is difficult to verify from the cited references alone.","section":"§2.3"},{"comment":"The manual screening of the 713 high-quality Mask-Contrast pairs is not documented. The authors should state the inclusion/exclusion criteria and, ideally, report how many candidate pairs were rejected and why.","section":"§3.1"},{"comment":"The manuscript does not provide an availability statement for CAS-3K or the trained CAS-IQA model, nor does it report random seeds, the number of runs, or whether the split was stratified by generative model. Such details are needed for reproducibility of a dataset-plus-model paper.","section":"§5.1"},{"comment":"The statement that the w/o MUST baseline requires about 15 epochs to converge is not supported by learning curves. Either provide the curves or remove the claim.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a genuinely novel application area, but the empirical evidence is currently not strong enough for the strength of the claims. The most important fixes are statistical inference for the reported gains, a clean pair-disjoint split description, and expert validation of the rating scale. I see no circularity problem; the evaluation is standard supervised learning on a held-out split. The absence of public data/code is a further concern for a paper whose main contribution is a new dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first IQA dataset for synthetic angiographies, and the three-image, three-metric VLM setup is a sensible departure from single-image IQA. But the performance margins the paper headlines are within sampling noise, and the MOS ground truth comes from three graduate students, not radiologists. The core idea is worth engaging; the evidence as reported is not.\n\nWhat is genuinely new: CAS-3K, 3,565 synthetic angiographies from five generative models, each scored on three clinically defined metrics (VMC, VBD, OQ). The three metrics were worked out with board-certified radiologists, which gives the benchmark face validity. The model takes the mask, generated, and contrast images as input and routes fused tokens to metric-specific branches—a real change from methods that score a single image. The ablation shows adding the fusion module improves all three metrics, even if the gains are not tested for significance.\n\nThe soft spots are concentrated in the evaluation. First, the 'significantly outperforms' claim rests on PLCC differences of 0.003 to 0.029 on a test set of 715. With n=715, the 95% CI on a correlation alone is about ±0.07, so those gaps are within noise. No confidence intervals or significance tests are reported. Second, the paper says the improvements are '0.5%, 2.0%, and 4.1% over the second-best method, MA-AGIQA,' but in Table 1 the second-best OQ is LoDa (0.6899), not MA-AGIQA (0.6696). The 4.1% figure is overstated; the real edge over LoDa is about 1.2%. That is a concrete internal inconsistency in the central claim. Third, the MOS comes from three biomedical engineering graduate students after a short training session; the radiologists only helped define the metrics. Student ratings may track radiologist judgments, but the paper provides no evidence that they do, which is a problem because the whole point is clinical usefulness. The dataset is also not public, so the benchmark cannot be stress-tested by others.\n\nThe reader's stress-test note is on target; the rater issue is secondary to the statistical one. Even with perfect labels, the method comparisons would still be statistically weak.\n\nBottom line: this deserves a serious referee, but the authors need to release the dataset, add a clinician-validated subset, fix the second-best comparison, and report intervals or explicit significance tests. As written, the ordering of methods is plausible but not demonstrated.","headline":"Real first dataset for angiography IQA, but the performance claims need better statistics and the ground truth needs clinician validation.","tokens_in":11197,"tokens_out":3142,"would_cite":false,"duration_ms":25300,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model that compares a synthetic angiogram against its non-contrast mask and real contrast reference can score its quality more accurately than existing single-image IQA methods, and the authors build the CAS-3K dataset…","keywords":["synthetic angiography","image quality assessment","vision-language model","medical AIGC","CAS-3K dataset","multi-source feature fusion","X-ray angiography","quality scoring"],"falsifier":"Ask a panel of board-certified interventional radiologists to independently score a stratified sample of CAS-3K images on the same three metrics, then compute the PLCC between their mean ratings and the student-derived MOS; if that correlation is below roughly 0.7, or if CAS-IQA's advantage over the second-best method disappears when evaluated against the radiologist ratings, the paper's claim of clinically meaningful quality scores is falsified.","tokens_in":10035,"feed_emoji":"🩻","tokens_out":9716,"duration_ms":71042,"temperature":0.7,"pith_summary":"CAS-IQA is built around a simple premise: a synthetic X-ray angiogram cannot be judged in isolation, because its quality is defined by how faithfully it preserves vessel structure relative to a real contrast image and a non-contrast mask. To test this, the authors built CAS-3K, a dataset of 3,565 synthetic angiographies from five generative models, annotated by three raters on three clinically defined metrics: vessel morphology consistency, vessel branch detection, and overall quality. They then trained a vision-language model with a Multi-path feature fusion and routing (MUST) module that fuses visual tokens from the three image sources and routes them to metric-specific branches. On CAS-3K, the model reports the best PLCC and SRCC on all three metrics, with PLCC gains of 0.5%, 2.0%, and 4.1% over the second-best method. If these results are right, CAS-IQA offers a clinically grounded alternative to single-score IQA for medical AIGC.","feed_headline":"New model tops prior IQA methods on synthetic angiograms","feed_subtitle":"Fusing mask and contrast references boosts synthetic-angiography quality scoring by up to 4.1% PLCC.","key_machinery":"The load-bearing mechanism is the MUST (Multi-path featUre fuSion and rouTing) module, a token-level fusion-and-routing block. It computes vessel tokens by deformably aligning Contrast tokens to Mask tokens and subtracting them, then creates three branch representations: VMC uses self-attention over vessel tokens as queries into Generated tokens; VBD uses a convolutional block to extract local bifurcation-level tokens before cross-attention; OQ combines the Generated self-attention map with the VMC and VBD attention maps using learnable weights $\\alpha$, $\\beta$, $\\gamma$ summing to one. A vision abstractor and LLM, fine-tuned with low-rank adaptation, take the branch-specific tokens and prompts to emit five-level quality tokens, and a softmax-weighted sum of level midpoints converts the logits into 0–100 scores.","core_discovery":"The central discovery the paper tries to establish is that vision-language models can produce fine-grained, reference-aware quality scores for synthetic angiographies that align with human ratings better than existing IQA methods, provided the model is given the generated image together with its non-contrast mask and its real contrast-filled counterpart. The authors claim CAS-3K is the first IQA dataset for medical AIGC, and that CAS-IQA, built on a VLM with the MUST module, achieves the best PLCC and SRCC on all three metrics, with the largest gain in overall quality (4.1% PLCC over the second-best method). The paper attributes this to the MUST module's ability to isolate vessel tokens via deformable subtraction of mask from contrast, apply global self-attention for morphology, local convolution for branch details, and weighted attention-map fusion for overall quality, then feed branch-specific tokens to the LLM through metric-specific prompts.","pith_inferences":["An implication the paper leaves implicit is that if the student-based MOS is validated against radiologist ratings, CAS-IQA's scores could serve as an automated acceptance gate for synthetic angiography before clinical use.","The same mask–contrast–generated triple-input design should transfer to other paired medical synthesis tasks, such as contrast-enhanced CT or MRI, where reference-aware quality scoring is also missing.","Because the ground truth comes from only three raters, the reported 0.5–4.1% PLCC margins may lie within the noise of subjective labelling; a multi-rater replication with expert adjudication would show whether the ranking is stable."],"forward_implications":["Reference-aware scoring becomes a practical option for medical AIGC: generated angiograms can be audited against their mask and contrast reference instead of being judged as isolated images.","The three task-specific metrics give generative-model developers concrete optimization targets: improving vessel morphology, branch detection, and overall realism should translate into higher quality scores.","The MUST module's fusion-and-routing design is claimed to accelerate convergence, reaching optimal performance in about five epochs versus fifteen without it, which matters in small medical datasets.","The VBD analysis suggests that branch-level artifacts are visually salient and may be detected with less contextual reference, so branch screening could potentially be handled by simpler single-image models."],"supporting_citations":[{"why":"Supplies the XCAD source angiography pairs from which mask–contrast pairs are selected.","marker":"[14]"},{"why":"Supplies the CADICA source angiography pairs used alongside XCAD to build CAS-3K.","marker":"[7]"},{"why":"One of the GAN-based generators producing the synthetic angiographies in CAS-3K.","marker":"[35]"},{"why":"One of the GAN-based generators contributing synthetic angiographies to CAS-3K.","marker":"[17]"},{"why":"One of the diffusion-based generators producing synthetic angiographies for CAS-3K.","marker":"[33]"},{"why":"One of the diffusion-based generators producing synthetic angiographies for CAS-3K.","marker":"[34]"},{"why":"Defines the five-level discrete scoring scheme and the logit-to-score inference that CAS-IQA adapts.","marker":"[25]"},{"why":"Provides the vision-language backbone whose visual abstractor and LLM are fine-tuned with low-rank adaptation.","marker":"[29]"},{"why":"Supplies the subjective rating protocol (outlier rejection, score normalization) used to compute MOS.","marker":"[19]"},{"why":"The second-best baseline whose per-metric PLCC/SRCC results set the margin CAS-IQA claims to beat.","marker":"[24]"}],"fun_headline_variants":["VLM with auxiliary refs improves synthetic angiography scoring","Reference-aware VLM beats prior IQA on synthetic angiograms","CAS-IQA: fusing mask and contrast boosts angio quality scores","Fine-grained VLM scoring for synthetic angiography outperforms IQA","MUST module elevates VLM-based IQA on synthetic X-rays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the mean opinion scores from just three biomedical-engineering graduate students, who were trained briefly and admitted to annotation only after their pairwise correlations exceeded 0.7, are a valid proxy for what board-certified interventional radiologists would judge, since no radiologist actually rated the images.","fun_headline_variants_meta":{"raw":{"variants":["VLM with auxiliary refs improves synthetic angiography scoring","Reference-aware VLM beats prior IQA on synthetic angiograms","CAS-IQA: fusing mask and contrast boosts angio quality scores","Fine-grained VLM scoring for synthetic angiography outperforms IQA","MUST module elevates VLM-based IQA on synthetic X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3083,"prompt_tokens":934,"completion_tokens":2149,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2061}},"tokens_in":550,"tokens_out":2149,"duration_ms":12741,"temperature":1.0,"reasoning_tokens":2061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:43:20.329257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a panel of board-certified interventional radiologists to independently score a stratified sample of CAS-3K images on the same three metrics, then compute the PLCC between their mean ratings and the student-derived MOS; if that correlation is below roughly 0.7, or if CAS-IQA's advantage over the second-best method disappears when evaluated against the radiologist ratings, the paper's claim of clinically meaningful quality scores is falsified.","supporting_citations":[{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Supplies the XCAD source angiography pairs from which mask–contrast pairs are selected."},{"cited_title":"Expert Systems41(12), e13708 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the CADICA source angiography pairs used alongside XCAD to build CAS-3K."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"One of the GAN-based generators producing the synthetic angiographies in CAS-3K."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"One of the GAN-based generators contributing synthetic angiographies to CAS-3K."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"One of the diffusion-based generators producing synthetic angiographies for CAS-3K."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"One of the diffusion-based generators producing synthetic angiographies for CAS-3K."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Defines the five-level discrete scoring scheme and the logit-to-score inference that CAS-IQA adapts."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides the vision-language backbone whose visual abstractor and LLM are fine-tuned with low-rank adaptation."},{"cited_title":"Recommendation ITU-R BT500(13) (2012)","cited_arxiv_id":null,"evidence_quote":"Supplies the subjective rating protocol (outlier rejection, score normalization) used to compute MOS."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"The second-best baseline whose per-metric PLCC/SRCC results set the margin CAS-IQA claims to beat."}],"review_version":1}