{"id":"31b945ba-8cbf-404c-9f2c-6431db40ed20","arxiv_id":"2607.07673","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"An automated curation pipeline extracts 11M high-fidelity medical image-text pairs from PMC, yielding CLIP and MLLM vision encoders that outperform baselines on 26 benchmarks and a clinical dermatology retrieval task.","lead":"MedPMC is a five-stage automated pipeline that extracts 11 million medically relevant image-text pairs from PubMed Central articles, filtering out non-medical images and splitting compound figures into aligned subfigure-subcaption pairs. A CLIP model trained on this curated corpus outperforms a baseline trained on 2x more but unfiltered pairs by 7.1 AUC points across 26 medical benchmarks.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The controlled comparison attributes the 7.1 pp AUC gain to curation quality, but does not disentangle medical-image filtering (which roughly doubles effective medical training pairs) from the higher-fidelity pipeline stages (decomposition, caption alignment) that constitute the paper's distinctive","rationale":"The reader identified GPT-4T synthetic label noise as the weakest assumption. This is a legitimate concern but is mitigated by the fact that pipeline components are independently validated on human-labeled test sets, and GPT-4T's influence on the final CLIP training data is indirect (through trained classifiers, not direct label injection). The more load-bearing concern is the confound in the controlled comparison: MedPMC-CLIP trains on roughly 2.2× more medical image-text pairs than BMC-CLIP, and the paper does not ablate whether the improvement comes from simple filtering (more medical examples) or from the distinctive high-fidelity pipeline stages (decomposition, alignment). However, this is an attribution gap, not a validity challenge. The paper's central claim — that the curation framework as a whole produces better training data and stronger models — is well-supported by the controlled experiment, the breadth of evaluation (26 benchmarks, 11 specialties), the downstream QA results, and the independent clinical dermatology transfer experiment. The statistical analysis is rigorous (paired bootstrap, 95% CIs), and the public release of data, code, models, and benchmarks enhances reproducibility. The concern would strengthen the paper if addressed but does not invalidate the central finding. The verdict of ACCEPT is appropriate, though correctness_risk could be nudged from 'unknown' toward 'low-moderate' given the attribution gap.","tokens_in":26302,"tokens_out":5169,"duration_ms":253798,"concrete_test":"Train a CLIP model on a medical-filtered subset of BIOMEDICA (applying only MedPMC's medical figure classifier to retain ~medical images, without multi-panel decomposition or caption alignment), subsampled to approximately 11M pairs to match MedPMC's scale. If this filtered-but-not-decomposed baseline approaches MedPMC-CLIP's average AUC (within ~2 pp), the gain is primarily from filtering rather than high-fidelity curation. If it remains substantially below MedPMC-CLIP, the decomposition and alignment stages are load-bearing for the claimed improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central controlled experiment replaces BIOMEDICA 24M with MedPMC 11M while holding architecture, initialization, and training protocol fixed. The paper interprets the resulting 7.1 pp AUC improvement as evidence that 'high-fidelity curation' strengthens medical foundation models. However, the two corpora differ not only in curation fidelity but also in medical image density: MedPMC is 95.3% medical (~10.5M medical pairs) while BIOMEDICA is 19.7% medical (~4.7M medical pairs). This means MedPMC-CLIP effectively trains on approximately 2.2× more medical image-text pairs than BMC-CLIP, despite fewer total pairs. The improvement could be primarily driven by this higher count of medical training examples — a simple filtering effect — rather than by the paper's more distinctive curation contributions (multi-panel figure decomposition, joint caption separation and alignment, medical figure classification of subfigures). The paper does not provide an ablation that controls for the number of medical training pairs, such as training CLIP on a medical-filtered subset of BIOMEDICA without decomposition or alignment. Without such an ablation, the evidence supports the broad claim that 'curation helps' but does not specifically validate that the high-fidelity aspects of the pipeline (beyond filtering) are what drive the gains. This is an attribution gap rather than a validity challenge — the end-to-end result is real, but the mechanism is under-specified. The reader's concern about GPT-4T synthetic label noise is valid but secondary: pipeline components are evaluated on human-labeled held-out test sets (F1 scores in the 81–97 range), and GPT-4T's influence on the final training data is indirect through trained classifier models rather than direct label injection.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This manuscript introduces MedPMC, a five-stage modular pipeline for curating medical image-text pairs from permissively licensed PubMed Central articles. The pipeline performs initial screening, multi-panel figure detection, figure separation, joint caption separation and alignment, and medical figure classification, yielding 11M pairs from 6.1M articles. The authors train MedPMC-CLIP using the exact architecture and training protocol of BMC-CLIP (which trains on the 24M-pair BIOMEDICA corpus) to isolate the effect of the dataset. They report a 7.1 pp improvement in average zero-shot AUC across 26 benchmarks spanning 11 specialties, gains in two MLLM QA benchmarks when MedPMC-CLIP replaces the vision encoder in LLaVA-Med, and an 11.7 pp improvement in Recall@5 on a morphology-to-image retrieval task using 10,524 internal clinical dermatology photographs. The framework, corpus, benchmarks, and model checkpoints are publicly released.","tokens_in":26514,"tokens_out":3305,"duration_ms":126350,"significance":"The paper makes a substantial contribution as a resource: a reproducible, continuously updatable curation pipeline with component-level benchmarks, a large curated medical image-text corpus, and pretrained models all publicly released. The controlled CLIP experiment—matching architecture, initialization, hyperparameters, and training schedule to BMC-CLIP—is a well-designed end-to-end test of whether curation quality translates into downstream gains. The inclusion of 95% confidence intervals via paired bootstrap across benchmarks, the independent clinical dermatology evaluation on internal patient data, and the embedding-space analysis of distributional alignment with clinical images are all commendable. The component-level benchmark suite comparing against prior pipeline approaches (PMC-OA, MedICaT, GPT-4T zero-shot) adds engineering value and positions the framework for community refinement.","major_comments":[{"comment":"§2.4, paragraph on training details: The paper states that the head-to-head comparison with BMC-CLIP 'isolated the effect of dataset quality.' However, the two corpora differ in at least three load-bearing ways simultaneously: (1) medical image density (MedPMC is 95.3% medical vs. 19.7% for BIOMEDICA, yielding ~10.5M vs. ~4.7M medical pairs—a roughly 2.2× difference in medical training examples), (2) total pair count (11M vs. 24M), and (3) curation depth (multi-panel decomposition, joint caption alignment). The 7.1 pp AUC gain is attributed to 'high-fidelity curation,' but without an ablation that controls for the number of medical training pairs—e.g., training on a medical-filtered subset of BIOMEDICA without decomposition or alignment—the evidence supports the broad claim that curation helps but does not specifically validate that the distinctive pipeline stages (beyond filtering) are载","section":null},{"comment":"§2.1, Caption separation and alignment: The paper states that samples where the number of generated subcaptions does not match the number of input subfigures are removed to 'prioritize pair fidelity.' The pipeline goes from 29M subfigures to 12.5M subfigure-subcaption pairs at this stage—a 57% reduction. This is a substantial filter that could introduce systematic selection bias (e.g., excluding figures with many panels or complex layouts, which may represent important clinical content). The paper does not quantify the mismatch rate, analyze what types of figures are dropped, or discuss the potential impact on dataset coverage. This information is needed to assess whether the resulting corpus is representative of the medical literature or biased toward simpler figure layouts.","section":null}],"minor_comments":[{"comment":"§2.1, Initial screening: The paper reports F1=93.2 for initial screening but does not report precision and recall separately in the main text. Given that this stage gates all downstream processing, reporting these would help readers assess the false-negative rate (medically relevant figures lost) and false-positive rate (wasted downstream computation).","section":null},{"comment":"§4.1, Validation and model selection: The validation sets include synthetic annotations generated by the same GPT-4T procedure used for training data. While the test sets use manual labels, the use of synthetic labels in validation could bias model selection toward models that align with GPT-4T's labeling tendencies. A brief discussion of this potential bias would strengthen the methodology section.","section":null},{"comment":"Fig. 2a: The BIOMEDICA sample size (432 images) is much smaller than the MedPMC sample (2,906 images). The confidence intervals on the BIOMEDICA composition estimates are correspondingly wider. It would be helpful to note this asymmetry or report CIs on the category proportions.","section":null},{"comment":"§2.4, Downstream medical QA: The MMMU improvement of 1.9 pp has a 95% CI of [-4.1, 8.0], which includes zero. The text acknowledges this, but the abstract states the improvement without qualification ('improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks'). The abstract should note that the MMMU result is not statistically significant.","section":null},{"comment":"Extended Data Table 2: The caption separation and alignment training set lists '1,664 (M)' from MedICaT, but §4.1 states '1,361 manually annotated samples from the MedICaT dataset.' Please reconcile these numbers.","section":null},{"comment":"§4.4, Morphology-to-skin image retrieval: The set-based matching criterion for retrieval evaluation is described, but the specific threshold or matching rule (subset vs. exact match of concept sets) could be stated more precisely in the main text rather than requiring the reader to consult the cited references.","section":null},{"comment":"The paper uses 'GPT-4' and 'GPT-4T' somewhat interchangeably in places (e.g., §4.1 multi-panel figure detection says 'GPT-4' while other sections say 'GPT-4T'). Standardizing the terminology would improve clarity.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is a strong resource contribution with a well-designed end-to-end validation. The two major comments are about attribution precision and dataset representativeness—both are addressable through additional discussion and analysis without requiring new CLIP training runs (though a medical-filtered BIOMEDICA ablation would be ideal, it may be beyond the scope of a revision). The paper's broad claim that curation improves medical foundation models is well-supported; the issue is that the paper occasionally overstates the precision of the causal attribution. I would recommend the authors temper claims of 'isolating dataset quality' to acknowledge the confound between medical image density and curation depth, and add the requested analysis of the caption-mismatch filter. Given the public release of all resources, this work is likely to be highly impactful regardless of the attribution gap."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee raises two major comments: (1) the CLIP comparison with BMC-CLIP does not isolate which aspects of curation drive the observed gains, and (2) the caption alignment filter drops 57% of subfigures without analysis of potential selection bias. Both points are well-taken. We will address them through a combination of new analysis and revised language in the manuscript.","responses":[{"response":"The referee is correct that the three differences between MedPMC and BIOMEDICA (medical pair density, total pair count, and curation depth) are confounded in our current experimental design. We agree that the phrase 'isolated the effect of dataset quality' overstates what the comparison demonstrates. The experiment isolates the effect of the training corpus as a whole, not the individual contributions of filtering, decomposition, or alignment. We will revise the language in §2.4 and the Discussion to accurately characterize the comparison as isolating the effect of the training corpus while acknowledging that the corpora differ along multiple dimensions simultaneously. Regarding the proposed ablation—training on a medical-filtered subset of BIOMEDICA without decomposition or alignment—we agree this would be informative and will attempt it as a supplementary analysis. However, we note a practical constraint: BIOMEDICA does not provide per-image medical relevance labels, so constructing such a subset requires applying our own medical figure classifier to the BIOMEDICA corpus. This means the ablation would partially use MedPMC's own curation components, making it a cleaner test of 'filtering alone vs. filtering plus decomposition and alignment' rather than a fully independent test. We will be transparent about this limitation if the ablation is included. Regardless of whether the ablation is feasible within the revision timeline, we will soften the causal claim and explicitly enumerate the confounded factors.","revision_made":"partial","referee_comment":"§2.4: The head-to-head comparison with BMC-CLIP does not isolate the effect of curation quality because the corpora differ in medical image density, total pair count, and curation depth simultaneously. An ablation controlling for the number of medical training pairs is needed."},{"response":"This is a fair and important point. We will add analysis to the revised manuscript. Specifically, we plan to: (1) quantify the mismatch rate as a function of panel count, showing how the drop-off concentrates among figures with many subpanels; (2) sample and manually categorize a subset of dropped figures to characterize what types of content are disproportionately excluded; and (3) add a discussion paragraph acknowledging the potential for selection bias toward simpler layouts and its implications for dataset coverage. We agree that the current manuscript does not adequately address this trade-off between fidelity and coverage. The revised text will explicitly note that the filter prioritizes pair-level fidelity at the cost of excluding complex multi-panel figures, which may underrepresent certain clinical content types. We will also discuss potential future approaches to recovering some of the dropped pairs (e.g., partial alignment or relaxed matching criteria) as future work.","revision_made":"yes","referee_comment":"§2.1: The caption separation and alignment stage drops 57% of subfigures (from 29M to 12.5M) when subcaption counts do not match subfigure counts. This could introduce systematic selection bias toward simpler layouts. The paper does not quantify the mismatch rate, analyze dropped figure types, or discuss impact on coverage."}],"tokens_in":26253,"tokens_out":714,"duration_ms":74472,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this is a well-engineered data curation pipeline that produces a genuinely useful 11M-pair medical image-text corpus, and the end-to-end model gains are real. But the headline controlled comparison has a confound the paper doesn't address, which limits how strongly you can attribute the gains to the distinctive parts of the pipeline rather than to simple medical filtering.","headline":"Solid data infrastructure paper with a real attribution gap in its central experiment","tokens_in":27172,"tokens_out":643,"would_cite":true,"duration_ms":32243,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Cleaner medical data beats bigger data: 11M curated pairs top 24M","keywords":[],"falsifier":"If a CLIP model trained on a random 11M subset of BIOMEDICA (without MedPMC's curation but matched for size) achieved comparable performance to MedPMC-CLIP, the curation pipeline's contribution would be confounded with corpus size effects. Alternatively, if the pipeline's synthetic GPT-4T labels were replaced with human annotations and the final CLIP model's performance changed substantially, the gains would be attributable to label quality rather than data curation per se.","tokens_in":26602,"feed_emoji":"🔬","tokens_out":1302,"duration_ms":168504,"temperature":0.7,"pith_summary":"This paper argues that the quality of image-text pairs matters more than raw quantity when training medical vision-language models. The authors built MedPMC, a five-stage automated pipeline that mines PubMed Central's open-access biomedical literature for clinically relevant images. The pipeline screens out non-medical figures (graphs, charts, molecular diagrams), decomposes compound multi-panel figures into individual subimages, aligns each subimage with its corresponding subcaption, and filters the result for medical relevance. Applied to 6.1 million articles, it produced 11 million image-text pairs with 95.3% medical relevance, versus 19.7% in a prior 24-million-pair dataset. The central claim is tested through a controlled experiment: a CLIP-style model trained on the 11M MedPMC corpus, using identical architecture and hyperparameters to a baseline trained on the 24M BIOMEDICA corpus, improved average zero-shot AUC by 7.1 percentage points across 26 benchmarks spanning 11 medical specialties, improved medical visual question-answering by up to 16.9 percentage points when used as the vision encoder in a multimodal language model, and improved morphology-to-image retrieval by 11.7 percentage points on 10,524 internal clinical dermatology photographs. The paper frames data curation not as preprocessing but as reusable, continuously updatable infrastructure, with modular components that can be individually benchmarked, swapped, and refined as new literature is published.","feed_headline":"Curated 11M medical image pairs beat 24M uncured ones","feed_subtitle":"Five-stage pipeline filtering PubMed Central figures yields 95% medical relevance and 7-point AUC gains over twice the data, proving data c","key_machinery":"The five-stage curation pipeline (initial screening via PubMedBERT text classifier, multi-panel figure detection via Vision Transformer, figure separation via YOLOv10, caption separation and alignment via supervised InternVL-2.5-4B, medical figure classification via Vision Transformer) is the central object. The controlled comparison design—identical CLIP architecture, initialization, hyperparameters, and training schedule, with only the training corpus changed—is the mechanism that isolates dataset quality as the causal variable. The modular benchmark suite for each pipeline stage is the infrastructure that makes the framework auditable and continuously improvable.","core_discovery":"The paper's central discovery is that systematically removing non-medical images, decomposing compound figures into panel-level units, and aligning each panel with its specific subcaption yields training data that produces stronger medical AI models than datasets more than twice as large but lacking this curation. The mechanism is signal-to-noise: when 80% of a corpus consists of graphs, charts, and schematics rather than clinically relevant images, the contrastive learning objective wastes capacity on irrelevant visual content. By filtering to 95.3% medical relevance and achieving panel-level image-text correspondence, each training pair provides supervision at the level where clinical视觉特征最","pith_inferences":["The use of GPT-4 Turbo to generate synthetic training labels for three of the five pipeline stages means the curation pipeline's quality is partly bounded by GPT-4T's own medical knowledge. If GPT-4T systematically mislabels certain image types or introduces domain-specific biases, these errors propagate into the curated corpus and could be learned by downstream models. The paper does not ablate t","The 7.1-point AUC gain over BMC-CLIP is attributed solely to dataset quality, but the two corpora differ in composition (MedPMC has 11M pairs, BIOMEDICA has 24M), and the smaller corpus may benefit from a more favorable signal-to-noise ratio per gradient update rather than per pair. A scaling-curve comparison at matched corpus sizes would more cleanly isolate curation quality from dataset size eff","The embedding-space analysis showing MedPMC dermatology images overlap more with clinical photographs than existing public dermatology datasets do suggests that biomedical literature may capture visual diversity that curated clinical datasets miss, but this could also reflect publication bias toward visually distinctive or representative cases rather than the full distribution of routine clinical "],"forward_implications":["If data fidelity dominates data scale for medical multimodal pretraining, then the field's emphasis on building ever-larger corpora may be less productive than investing in curation pipelines that improve signal-to-noise ratio.","The modular, continuously updatable pipeline design means that as new medical imaging modalities, diseases, and terminology emerge in the literature, the corpus can be refreshed semiannually without rebuilding from scratch, keeping pretrained models current.","The finding that literature-derived images transfer to clinical dermatology retrieval suggests that curated biomedical publications can serve as a pretraining substrate that complements, rather than competes with, institution-specific clinical datasets.","The release of component-level benchmarks for each curation stage establishes a standardized evaluation framework that could drive competition and improvement in individual curation subtasks, much as task-specific benchmarks have advanced other areas of machine learning."],"fun_headline_variants":["MedPMC: 11M curated medical image pairs beat 24M uncurated ones","Higher-fidelity curation beats 2x larger uncured medical datasets","Filtering out non-medical images boosts medical AI over 2x larger datasets","Panel-level image-text alignment beats datasets twice as large","Curated PubMed Central data yields 7-point AUC gains with half the pairs"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The controlled comparison attributing all performance gains to dataset quality assumes that the only meaningful difference between MedPMC's 11M pairs and BIOMEDICA's 24M pairs is the curation applied. However, three of the five curation pipeline stages use GPT-4 Turbo to generate synthetic training labels, and if those synthetic labels encode systematic biases or medical knowledge that the downstream CLIP model then absorbs, the performance gain could partly reflect distilled","fun_headline_variants_meta":{"raw":{"variants":["MedPMC: 11M curated medical image pairs beat 24M uncurated ones","Higher-fidelity curation beats 2x larger uncured medical datasets","Filtering out non-medical images boosts medical AI over 2x larger datasets","Panel-level image-text alignment beats datasets twice as large","Curated PubMed Central data yields 7-point AUC gains with half the pairs"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":816,"prompt_tokens":733,"completion_tokens":83,"prompt_tokens_details":null},"tokens_in":733,"tokens_out":83,"duration_ms":32005,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T02:47:35.088331+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a CLIP model trained on a random 11M subset of BIOMEDICA (without MedPMC's curation but matched for size) achieved comparable performance to MedPMC-CLIP, the curation pipeline's contribution would be confounded with corpus size effects. Alternatively, if the pipeline's synthetic GPT-4T labels were replaced with human annotations and the final CLIP model's performance changed substantially, the gains would be attributable to label quality rather than data curation per se.","supporting_citations":[],"review_version":1}