Pith. sign in

REVIEW 4 major objections 6 minor 10 cited by

MedCoT: Medical Chain of Thought via Hierarchical Expert

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MedCoT claims a 256M-parameter hierarchical expert chain with sparse-MoE voting beats a 7B medical VQA model by up to 5.5 points.

desk verdict A useful new combination for Med-VQA—hierarchical expert rationales plus a local sparse-MoE decoder—but the headline accuracy gains are not yet trustworthy because of test-set hyperparameter tuning, missing error bars, and no contamination check on the Gemini-generated rationales. read the letter →

arxiv 2412.13736 v1 pith:PUWHQCK5 submitted 2024-12-18 cs.CV

Jiaxiang Liu , Yuan Wang , Jiawei Du , Joey Tianyi Zhou , Zuozhu Liu This is my paper · ORCID
classification cs.CV
keywords medicalvisualquestionansweringchain-of-thoughtreasoninghierarchicalexpertverificationsparsemixtureofexpertsinterpretabilityrationalegenerationself-reflectionmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedCoT is a medical visual question answering method built around a three-tier expert chain: an Initial Specialist proposes a diagnostic rationale, a Follow-up Specialist critiques and corrects that rationale and writes an image caption, and a locally hosted Diagnostic Specialist—a multimodal T5 with a sparse mixture of experts—votes on the final answer. The paper's central claim is that this hierarchy produces both more accurate answers and readable reasoning paths without any manually annotated rationales. On closed questions it reports 87.50% accuracy on VQA-RAD and 87.26% on SLAKE-EN, and with roughly 256M trainable parameters it outperforms the 7B-parameter LLaVA-Med by 5.52% and 4.09% respectively. The significance is that medical diagnosis is modeled as a multi-expert consultation rather than a single-model shortcut, making the reasoning chain inspectable.

What carries the argument

The central mechanism is the hierarchical expert verification chain. The Initial Specialist is a general-purpose multimodal LLM prompted to produce a step-by-step rationale; the Follow-up Specialist uses self-reflection to keep or revise that rationale and to generate a caption that bridges the image-text modality gap; the Diagnostic Specialist is a locally hosted multimodal T5 whose visual features come from DETR and whose text features come from Flan-T5, fused by cross-attention and a sparse top-k Mixture of Experts. The sparse MoE selects k = 2 experts per input, aggregates their outputs with a softmax-weighted feature-level majority vote, and gates the final fusion between textual and attended visual features; empirically the experts auto-specialize, with different experts handling head, chest, and abdomen questions. The chain is what carries the argument: it converts an opaque single-model answer into a traceable rationale that is checked by a second pass and then voted on by specialist modules.

What would settle it

Replace the medical image with a blank or semantically unrelated image while keeping the question and options identical; if the Initial and Follow-up Specialists still produce matching rationales and the Diagnostic Specialist still answers correctly, then the chain is relying on textual priors or memorized benchmark content rather than on visual evidence.

Watch

Extended reading notes

Core claim

MedCoT's claim is that an explicit, verifiable reasoning chain is both more accurate and more interpretable than single-shot Med-VQA. The pipeline starts with an Initial Specialist (a proprietary LLM, Gemini Pro 1.5) that proposes a rationale from the image and question; a Follow-up Specialist (the same LLM) classifies the rationale as effective or ineffective, rewrites it if needed, and adds an image caption; a local Diagnostic Specialist—a multimodal T5 with a sparse MoE and feature-level majority vote—then produces the final answer from the image, question, refined rationale, and caption. The paper reports that on closed questions this three-tier chain reaches 87.50% on VQA-RAD and 87.26% on SLAKE-EN, beating the 7B-parameter LLaVA-Med by 5.52 and 4.09 points, and that ablations attribute 6.62 points to the Follow-up Specialist and 4.78 points to the MoE on VQA-RAD.

Load-bearing premise

The reported accuracy gains assume that the rationales produced by the external language model come from genuine visual understanding of each image, not from the model having memorized the benchmark questions and answers.

Editorial extensions

If this is right

  • A roughly 256M-parameter local Diagnostic Specialist is enough to outperform a 7B end-to-end medical VQA model on VQA-RAD and SLAKE-EN closed questions, which would cut deployment cost if the result holds.
  • The Follow-up Specialist's self-reflection contributes 6.62 accuracy points on VQA-RAD (87.50% vs 80.88%), so rationale quality is a first-order accuracy lever rather than a cosmetic addition.
  • The sparse MoE contributes another 4.78 points on VQA-RAD (87.50% vs 82.72%), and its experts specialize by organ, giving per-case traceability of which expert handled a question.
  • Human rationale annotation is not required; the chain generates and verifies rationales by prompting an LLM, removing a major bottleneck for interpretable Med-VQA.
  • On open-ended questions MedCoT reports higher Rouge and BLEU scores than MedThink, indicating the reasoning chain also improves generative answer quality, not only closed-form accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's reported gains do not yet separate the architecture's contribution from the memory of the proprietary LLM; running the pipeline with the image removed or with rephrased questions would settle whether the rationales are truly vision-grounded.
  • If the result holds, the three-tier draft-verify-decide pattern transfers naturally to other high-stakes multimodal question answering settings such as pathology slides, satellite imagery, or industrial inspection, where a small local model makes the final call and an LLM supplies the rationale.
  • The appendix case where both specialists agree on a wrong pneumomediastinum diagnosis shows the chain inherits a single-point-of-failure risk in the LLM; a second, vision-only verification signal beyond the Follow-up Specialist might catch errors the current design misses.
  • Since k=2 was optimal on every dataset while the best expert count varied from 5 to 10, the computational cost is essentially fixed by the number of experts, and expert count is a dataset-level hyperparameter rather than a universal constant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MedCoT, a hierarchical expert reasoning framework for medical visual question answering. An Initial Specialist (Gemini Pro 1.5) generates a preliminary rationale; a Follow-up Specialist (also Gemini Pro 1.5) validates and refines the rationale and produces an image caption; a locally trained Diagnostic Specialist (Flan-T5 + DETR with sparse MoE and feature-level majority voting) ingests the image, question, rationale, and caption and outputs the final answer. Experiments on VQA-RAD, SLAKE-EN, Med-VQA-2019, and PathVQA report accuracy gains over previous state-of-the-art methods, including a 256M-parameter model outperforming LLaVA-Med (7B) by 5.52% and 4.09% on VQA-RAD and SLAKE-EN. The paper claims both improved accuracy and interpretability.

Significance. The core idea is plausible and practically relevant: using a proprietary LLM as a rationale generator and a small local sparse-MoE model as a diagnostic aggregator is a reasonable way to inject interpretable reasoning into Med-VQA. If the empirical claims hold, the method would be a useful lightweight alternative to large medical VLMs and would demonstrate that sparse MoE voting improves robustness. The paper releases code and presents qualitative examples. However, the central empirical claims are currently undermined by post-hoc test-set hyperparameter selection, the absence of a no-rationale baseline, potential contamination of the public benchmarks by Gemini's pretraining data, and the lack of error bars. These issues are fixable but must be addressed before the state-of-the-art claims can be accepted.

major comments (4)
  1. [Section 4.3, Grid Search] The optimal number of experts (6, 10, 5, 5 for VQA-RAD, SLAKE-EN, Med-2019, and PathVQA) and the optimal k value (2) are selected per dataset on the test sets, as described in the Grid Search paragraph. This is post-hoc selection on the test data, so the reported accuracies in Figure 4 and Table 1 are optimistic and do not estimate generalization honestly. Please select hyperparameters on a validation split (or with nested cross-validation) and report test accuracy for the selected configuration, together with error bars over multiple runs or bootstrapping.
  2. [Table 1 and Section 4.3, Ablation Study] The ablation varies only the Follow-up Specialist and the MoE components; it never removes the rationale/caption input entirely. Since the Diagnostic Specialist is trained on Gemini-generated rationales and captions, the 9.93-point gap between the no-MoE/no-Follow-up condition (77.57) and full MedCoT (87.50) on VQA-RAD could be largely due to the additional textual information rather than the Follow-up refinement or the MoE architecture. Add a baseline with no rationale/caption input and a baseline with only the Initial Specialist rationale, so that the contribution of each component is isolated.
  3. [Sections 4.1 and 4.2] Gemini Pro 1.5 is used as the Initial and Follow-up Specialist on four public benchmarks (VQA-RAD, SLAKE-EN, Med-VQA-2019, PathVQA). Because Gemini's pretraining data likely include these public datasets, the generated rationales may implicitly encode ground-truth answers, and the Diagnostic Specialist may be reading the answer out of the rationale rather than performing genuine visual reasoning. The paper provides no contamination analysis, no evaluation on a private or newly collected dataset, and no control such as comparing Gemini rationales on original versus visually perturbed images. This is load-bearing for the generalization claim in the abstract; please add such an analysis or explicitly scope the claims to settings where the rationales are independently verified.
  4. [Figure 4 and Table 1] The main results are reported as single runs without error bars or significance tests. For differences of 4-6% against LLaVA-Med, it is possible that confidence intervals overlap. Report multiple random seeds (or bootstrap confidence intervals) and, where feasible, a significance test, so that the claimed improvements can be assessed quantitatively.
minor comments (6)
  1. [Equations (4)-(6)] The notation for the MoE feature-level majority vote is under-specified: Wi is defined as softmax(Vtop k)_i over k selected experts, but the dimensions of Vtop k, Ei,Ff, and EFf are not stated. Please define these terms explicitly and clarify how the weighted average operates on feature vectors.
  2. [Equation (1)] The symbol f is used both for the rationale generator in the optimization objective and later for the fused feature Ff; this collision makes the notation hard to follow. Please rename one of these uses.
  3. [Figures 2 and 5] The captions of Figures 2 and 5 are nearly identical, and the duplicated text should be differentiated. Figure 5's caption says 'context' but the figure shows two Diagnostic Specialist boxes; please clarify the intended message.
  4. [Section 4.1] The dataset name 'Med-VQA-2019' is presumably VQA-Med 2019 from ImageCLEF; please use the standard name and cite the original source consistently throughout.
  5. [Abstract and Section 4.2] The abstract states that MedCoT 'surpasses existing state-of-the-art approaches' without noting that the main-text comparisons are for closed-end questions, with open-end results in the appendix. Please state this qualification in the abstract or in the main-text summary.
  6. [Limitation section] The limitation section acknowledges LLM hallucination but does not discuss the possibility that the public benchmark rationales are contaminated by Gemini's pretraining data; adding this discussion would strengthen the paper's treatment of external validity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MedCoT's accuracy claims rest on external benchmark evaluations; no fitted parameter is renamed as prediction, and no self-citation carries the central argument.

full rationale

MedCoT's central claim is an empirical accuracy comparison on four standard Med-VQA datasets, evaluated against external baselines and held-out benchmark splits. The Diagnostic Specialist is a locally trained multimodal T5 with a sparse MoE, trained by a standard negative log-likelihood loss against dataset labels (Section 3.4.1). The rationales produced by Gemini Pro 1.5 (Section 4.1) are inputs to the pipeline, not fitted parameters, and nothing in Equations (1)-(6) defines the target answer as a function of the rationale alone or as a self-citation. The prediction is not statistically forced: the Diagnostic Specialist must still map the image, question, and rationale to the correct label, and the ablations in Table 1 show that removing the Follow-up Specialist or the MoE changes accuracy, which would be impossible if the answer were already contained in the input by construction. Self-citations (for example, MedThink and VQA-Adapter) appear only as related-work baselines and not as load-bearing premises of the derivation. The genuine risk noted by the skeptical reading—that Gemini Pro 1.5 may have memorized answers from public benchmarks, thereby making its rationales leakage-carrying—is an external validity and data-contamination concern, not a self-referential or definitional reduction, and should be scored as a correctness risk rather than circularity. No derivation step reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical claim depends on two fitted hyperparameters (expert count and top-k, tuned per dataset on test data) and on two domain assumptions: that CoT rationales improve answers and that Gemini Pro's rationales are clean. No new physical or conceptual entities are introduced.

free parameters (2)
  • MoE expert count = 6 (VQA-RAD), 10 (SLAKE-EN), 5 (Med-2019), 5 (PathVQA)
    Grid-searched per dataset to maximize test accuracy (Section 4.3); acts as a test-set-fitted hyperparameter.
  • MoE top-k = 2 (all datasets)
    Grid-searched on test sets (Section 4.3); a fitted hyperparameter.
assumptions (3)
  • domain assumption A generated rationale R improves the final answer accuracy (Eq. 1)
    The whole method assumes the multimodal CoT decomposition in Eq. 1 helps; no comparison to a no-rationale model is provided.
  • domain assumption Gemini Pro 1.5 provides unbiased, contamination-free rationales on benchmark images
    Section 4.1 says Gemini Pro is used; the paper does not test whether the LLM has memorized public benchmark answers.
  • domain assumption Sparse MoE with top-2 experts and a weighted feature vote outperforms a single gate
    Assumed in Section 3.4.2 and validated only via ablations with single runs; not theoretically motivated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedCoT: Medical Chain of Thought via Hierarchical Expert." pith.science (2026). https://pith.science/paper/PUWHQCK5

@misc{pith2026241213736,
  author       = {Pith},
  title        = {Pith review of: MedCoT: Medical Chain of Thought via Hierarchical Expert},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUWHQCK5}},
  note         = {Machine review of arXiv:2412.13736}
}
read the original abstract

Artificial intelligence has advanced in Medical Visual Question Answering (Med-VQA), but prevalent research tends to focus on the accuracy of the answers, often overlooking the reasoning paths and interpretability, which are crucial in clinical settings. Besides, current Med-VQA algorithms, typically reliant on singular models, lack the robustness needed for real-world medical diagnostics which usually require collaborative expert evaluation. To address these shortcomings, this paper presents MedCoT, a novel hierarchical expert verification reasoning chain method designed to enhance interpretability and accuracy in biomedical imaging inquiries. MedCoT is predicated on two principles: The necessity for explicit reasoning paths in Med-VQA and the requirement for multi-expert review to formulate accurate conclusions. The methodology involves an Initial Specialist proposing diagnostic rationales, followed by a Follow-up Specialist who validates these rationales, and finally, a consensus is reached through a vote among a sparse Mixture of Experts within the locally deployed Diagnostic Specialist, which then provides the definitive diagnosis. Experimental evaluations on four standard Med-VQA datasets demonstrate that MedCoT surpasses existing state-of-the-art approaches, providing significant improvements in performance and interpretability.

Figures

Figures reproduced from arXiv: 2412.13736 by the authors.

Figure 1
Figure 1. The upper figure shows a comparison of the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The MedCoT pipeline begins with an Initial Specialist receiving a medical question and image to generate [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Diagnostic Specialist Pipeline. After passing [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: MedCoT is compared with various SoTA methods on closed questions on the VQA-RAD and SLAKE-EN datasets. MedCoT not only achieves SoTA accuracy in answers but also provides reasoning paths (rationale). The metric used is Accuracy (%). where EFf is the value of the final …
Figure 5
Figure 5. Figure 5: The MedCoT pipeline begins with an Initial Specialist receiving a medical question and image to generate [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The Diagnostic Specialist’s sparse MoE shows [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The expert number grid search on two datasets. The blue line represents the results from training with [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning

    cs.AI 2026-07 accept novelty 6.0 of 10

    A dual clinical-computational taxonomy for medical LLM reasoning plus a five-level 5k-sample benchmark showing specialists excel at diagnosis and general models at decision support/dialogue.

  2. 4KAgent: Agentic Any Image to 4K Super-Resolution

    cs.CV 2025-07 reject novelty 6.0 of 10

    An agentic pipeline that plans and executes image restoration from a toolbox of pretrained models to upscale arbitrary images to 4K, reporting state-of-the-art results on many benchmarks.

  3. Med-U1: Incentivizing Unified Medical Reasoning in LLMs via Large-scale Reinforcement Learning

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A pure reinforcement learning recipe with mixed rule-based rewards and length control improves Qwen2.5-based models across diverse medical QA formats.

  4. Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new 8-stage chest X-ray VQA benchmark and a context-aware model trained on it.

  5. Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning

    cs.CL 2025-08 conditional novelty 5.0 of 10

    RoMed and CCL: a 144k-question perturbation benchmark for medical VQA and a consistency-plus-contrastive training method that improves LLaVA-Med's accuracy and reduces answer variation.

  6. RARL: Improving Medical VLM Reasoning and Generalization with Reinforcement Learning and LoRA under Data and Hardware Constraints

    cs.CV 2025-06 reject novelty 5.0 of 10

    RARL fine-tunes Qwen2-VL-2B on 716 medical samples with GRPO, LoRA, and a vaguely defined reasoning reward, claiming gains of 7.78% over SFT on reasoning and up to 27% on unseen VQA benchmarks.

  7. Silence is Not Consensus: Disrupting Agreement Bias in Multi-Agent LLMs via Catfish Agent for Clinical Decision Making

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A "catfish" agent that injects structured dissent into multi-agent LLM teams improves clinical question-answering accuracy by reducing premature consensus.

  8. TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Without any parameter updates, a retrieval-augmented generalist-specialist agent pair with consistency-based verification raises accuracy on 862 hard medical QA questions for GPT-4o, DeepSeek-R1, and Qwen2.5-7B.

  9. Architecting Clinical Collaboration: Multi-Agent Reasoning Systems for Multimodal Medical VQA

    cs.AI 2025-07 reject novelty 4.0 of 10

    A systematic study on dermatology VQA finds that multi-agent reasoning and retrieval architectures outperform fine-tuned open-source vision-language models, maintaining 70% accuracy under distribution shift.

  10. Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects

    cs.RO 2025-05 conditional novelty 4.0 of 10

    A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.

Reference graph

Works this paper leans on

43 extracted references · 24 canonical work pages · cited by 10 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Joey Liu, Dina Demner-Fushman, and Henning M \"u ller. 2019. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. CLEF (working notes), 2(6)

    work page 2019

  4. [4]

    Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2021. Weaqa: Weak supervision via captions for visual question answering. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

    work page 2021

  5. [5]

    Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning M \"u ller. 2019. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019

    work page 2019

  6. [6]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision, pages 213--229. Springer

    2020

  7. [7]

    Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022. All you may need for vqa are image captions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1947--1963

    work page 2022

  8. [8]

    Zhihong Chen, Guanbin Li, and Xiang Wan. 2022. Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5152--5161

    work page 2022

Show all 43 references
  1. [9]

    Sedigheh Eslami, Gerard de Melo, and Christoph Meinel. 2021. Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906

    2021 arXiv

  2. [10]

    Sedigheh Eslami, Christoph Meinel, and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.88 P ub M ed CLIP : How much does CLIP benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, ...

    2023 doi

  3. [11]

    William Fedus, Jeff Dean, and Barret Zoph. 2022 a . A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667

    2022 arXiv

  4. [12]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022 b . Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  5. [13]

    Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Yang Feng, Jian Wu, and Zuozhu Liu. 2024. Medthink: Explaining medical visual question answering via multimodal decision-making rationale. arXiv preprint arXiv:2404.12372

    2024 arXiv

  6. [14]

    Haifan Gong, Guanqi Chen, Sishuo Liu, Yizhou Yu, and Guanbin Li. 2021. Cross-modal self-attention with multi-task pre-training for medical visual question answering. In Proceedings of the 2021 International Conference on Multimedia Retrieval, pages 456--460

    2021

  7. [15]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. https://arxiv.org/abs/2003.10286 Pathvqa: 30000+ questions for medical visual question answering . Preprint, arXiv:2003.10286

    2020 arXiv

  8. [16]

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79--87

    1991

  9. [17]

    Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, and CV Jawahar. 2021. Mmbert: multimodal bert pretraining for improved medical vqa. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1033--1036. IEEE

    2021

  10. [18]

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896--1907

    2020

  11. [19]

    Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. Advances in neural information processing systems, 31

    2018

  12. [20]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1--10

    2018

  13. [21]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations

    2020

  14. [22]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650--1654. IEEE

    2021

  16. [24]

    Jiaxiang Liu, Tianxiang Hu, Yan Zhang, Yang Feng, Jin Hao, Junhui Lv, and Zuozhu Liu. 2023 a . Parameter-efficient transfer learning for medical visual question answering. IEEE Transactions on Emerging Topics in Computational Intelligence

    2023

  17. [25]

    Jiaxiang Liu, Tianxiang Hu, Yan Zhang, Xiaotang Gai, YANG FENG, and Zuozhu Liu. 2023 b . A chatgpt aided explainable framework for zero-shot medical image diagnosis. In ICML 3rd Workshop on Interpretable Machine Learning in Healthcare (IMLH)

    2023

  18. [26]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2...

    2022

  19. [27]

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842

    2023 arXiv

  20. [28]

    Binh D Nguyen, Thanh-Toan Do, Binh X Nguyen, Tuong Do, Erman Tjiputra, and Quang D Tran. 2019. Overcoming data limitation in medical visual question answering. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 522--530. Springer

    2019

  21. [29]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  22. [30]

    Obioma Pelka, Sven Koitka, Johannes R \"u ckert, Felix Nensa, and Christoph M Friedrich. 2018. Radiology objects in context (roco): a multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Labe...

    2018

  23. [31]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485--5551

    2020

  24. [32]

    Fuji Ren and Yangyang Zhou. 2020. Cgmvqa: A new classification and generative model for medical visual question answering. IEEE Access, 8:50626--50636

    2020

  25. [33]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2016. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations

    2016

  26. [34]

    Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. 2022. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6088--6100

    2022

  27. [35]

    Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C.H. Hoi. 2022 a . https://aclanthology.org/2022.findings-emnlp.67 Plug-and-play VQA : Zero-shot VQA by conjoining large pretrained models with zero training . In Findings of the Association for Computa...

    2022

  28. [36]

    Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C.H. Hoi. 2022 b . https://doi.org/10.18653/v1/2022.findings-emnlp.67 Plug-and-play VQA : Zero-shot VQA by conjoining large pretrained models with zero training . In Findings of the Association for Comp...

    2022 doi

  29. [37]

    Zhecan Wang, Bin Xiao, Noel Codella, Jianwei Yang, Yen-Chun Chen, Luowei Zhou, Shih-Fu Chang, Xiyang Dai, Haoxuan You, and Lu Yuan. 2022. Clip-td: Clip targeted distillation for vision-language tasks. In International Conference on Learning Representations

    2022

  30. [38]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  31. [39]

    Li-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen, and Xiao-Ming Wu. 2020. Medical visual question answering via conditional reasoning. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2345--2354

    2020

  32. [40]

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023 a . Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199

    2023 arXiv

  33. [41]

    Ruiyuan Zhang, Jiaxiang Liu, Zexi Li, Hao Dong, Jie Fu, and Chao Wu. 2024. Scalable geometric fracture assembly via co-creation space among assemblers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7269--7277

    2024

  34. [42]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023 b . Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923

    2023 arXiv

  35. [43]

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. 2023. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.