Evaluating Multimodal Large Language Models on Educational Textbook Question Answering

Ali Alkhathlan; Amani Jamal; Anas Zafar; Areej Alhothali; Hessa A. Alawwad; Usman Naseem

REVIEW 4 major objections 4 minor 23 references

Evaluating Multimodal Large Language Models on Educational Textbook Question Answering

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding retrieved textbook context to a strong multimodal model collapses its diagram-question accuracy, and fine-tuning recovers the loss only for the model with end-to-end multimodal training.

desk verdict A real and striking empirical observation—retrieved context tanks LLaMA 3.2-Vision's diagram reasoning—but the 'catastrophic context interference' label outruns the evidence, since the paper never shows the retrieved context was actually the right context. read the letter →

arxiv 2506.21596 v2 pith:PCRQIQEX submitted 2025-06-18 cs.CL cs.AIcs.IR

Hessa A. Alawwad , Anas Zafar , Areej Alhothali , Usman Naseem , Ali Alkhathlan , Amani Jamal This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords multimodallargelanguagemodelstextbookquestionansweringretrieval-augmentedgenerationcatastrophiccontextinterferenceCK12-QAdiagram-basedquestionsLLaMA3.2-VisionVA-1.5

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates two open multimodal models, LLaVA-1.5 and LLaMA 3.2-Vision, on CK12-QA, a dataset of 26,000 curriculum-based science questions with diagrams. It sets up a retrieval-augmented pipeline that supplies each question with three relevant lesson paragraphs and three related images. The central finding is that this added context, intended to help, sharply hurts LLaMA 3.2-Vision's zero-shot accuracy on diagram-based questions, dropping it from 74.07% to 25.93% on validation and from 74.37% to 15.28% on test; the authors call this 'catastrophic context interference.' The paper further claims that fine-tuning reverses the damage for LLaMA, raising test accuracy to 71.16%, while LLaVA degrades to 27.58%, pointing to architectural differences in how the two models fuse vision and language. A sympathetic reader would care because the result challenges the default assumption that giving a large multimodal model more relevant context always improves reasoning, especially for diagram-based educational content.

What carries the argument

The load-bearing mechanism is a multimodal retrieval-augmented generation (RAG) pipeline: question text or diagram embeddings from ImageBind are matched by cosine similarity against lesson paragraphs and images indexed in a KDB.AI vector store, and the top-k items are inserted into each model's prompt before the question. The named phenomenon 'catastrophic context interference' is the sharp zero-shot accuracy collapse this pipeline causes for LLaMA 3.2-Vision on diagram-based questions when three paragraphs and three images are added. The other central device is fine-tuning with autoregressive cross-entropy loss over question, retrieved context, diagram, and answer, which is what separates LLaMA's recovery from LLaVA's decline.

What would settle it

Run the same zero-shot evaluation with a re-ranked or human-curated retrieval stage that keeps only paragraphs directly needed to answer each diagram question; if LLaMA 3.2-Vision's diagram accuracy then stays near its 74% baseline, the 'catastrophic context interference' would be shown to come from irrelevant retrieved context rather than from the model's context integration mechanism. Alternatively, annotate a stratified sample of retrieved paragraph–question pairs for answer-relevance and check whether the incorrect predictions cluster on pairs judged irrelevant.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that modern multimodal large language models do not yet reliably integrate retrieved textual and visual context when answering textbook questions. Using CK12-QA, the authors show a statistically significant drop in LLaMA 3.2-Vision's diagram-question accuracy when a standard multimodal RAG pipeline adds three paragraphs and three images as context: from 74.07% to 25.93% on validation and from 74.37% to 15.28% on test. They name this failure 'catastrophic context interference' and argue it is a boundary of current MLLM reasoning rather than a retrieval failure, since a brief inspection finds the retrieved material topically relevant. The paper also claims that fine-tuning on the dataset lets LLaMA 3.2-Vision learn to use the same context effectively, reaching 71.16% test accuracy, whereas LLaVA-1.5 falls to 27.58%, a divergence the authors attribute to architectural differences in cross-modal fusion depth.

Load-bearing premise

The load-bearing premise is that the retrieval pipeline's returned paragraphs and diagrams are genuinely relevant to each question, so the accuracy drop must be caused by the model's failure to integrate context rather than by the retriever providing useless or mismatched content; the paper supports this only with a brief inspection of retrieval results.

Editorial extensions

If this is right

Retrieval-augmented prompting should not be applied blindly to multimodal educational QA; adding topically related context can override a model's visual grounding and lower accuracy on diagram questions.
Fine-tuning on task-specific multimodal data can turn a context-interference failure into a large gain for models with end-to-end cross-modal training, restoring LLaMA 3.2-Vision to 71.16% test accuracy under the same 3p-3I input.
Evaluation of educational MLLMs should report diagram, true/false, and multiple-choice accuracy separately, since overall accuracy can hide large category-level swings such as LLaMA's diagram drop and LLaVA's text gains.
Context filtering that respects visual–textual alignment, rather than textual similarity alone, is a concrete design target suggested by these results.
The opposite fine-tuning outcomes for LLaVA and LLaMA indicate that modality-fusion architecture, not just scale, determines whether a model can learn to prioritize the right modality per question.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One testable extension would be to replace the current retrieval with a re-ranked, question-aware selector and re-run the zero-shot evaluation; if LLaMA's diagram accuracy stays near 74%, the reported interference would be better attributed to noisy context delivery than to the model's fusion mechanism.
The same experimental setup could be applied to other diagram-heavy benchmarks, not just CK12-QA, to see whether 'catastrophic context interference' is a general property of strong MLLMs or specific to this dataset and these models.
The results suggest an implicit design principle for AI tutoring systems: present text and diagrams as competing hypotheses only when the question demands cross-modal comparison, and otherwise keep irrelevant but topically related paragraphs out of the prompt.
A natural follow-up is to log per-question retrieval relevance scores and correlate them with the model's errors; if incorrect predictions track low relevance while a brief inspection only establishes topical relatedness, the claim would be sharpened.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper has a genuine empirical observation worth reporting—retrieved context collapses LLaMA 3.2-Vision on diagram questions—but the authors' label "catastrophic context interference" attributes the drop to model-side failure without ruling out the more boring explanation that the retriever returned topically related but non-useful context. Their own qualitative analysis in Section IV-D describes exactly that pattern. So treat the effect as real, the mechanism as unproven.

What's new: first benchmark of these two open MLLMs on CK12-QA with a multimodal RAG pipeline. The tables are internally consistent, the setup is clearly described, and the zero-shot drop (74% to 26% on validation, 74% to 15% on test) is stark enough to survive a close reading. Fine-tuning LLaMA back to 71% while LLaVA collapses is also interesting, though without training details it's hard to know whether the divergence is architectural or just hyperparameter luck. The authors deserve credit for explicitly noting the retriever-versus-model ambiguity later in the paper, which is more than most papers do.

Soft spots. The abstract calls the phenomenon "statistically significant" but there is no significance test, no error bars, no repeated runs. That's a straightforward overclaim. The retrieval quality check is a "brief inspection" that only establishes topical relatedness, not salience; the failure examples in IV-D are cases where the retrieved text is about the general topic but doesn't answer the specific question about the diagram. That is precisely what a mediocre retriever would produce, so the "catastrophic context interference" label is premature. A gold-context control (e.g., using the actual lesson passage that covers the question) or a quantitative retrieval metric would settle it. Fine-tuning details—epochs, learning rate, data splits, LoRA vs full—are absent, which matters because LLaVA's collapse could be a tuning artifact. No code or data released, which makes independent verification harder than it needs to be.

Net: this is a useful cautionary benchmark, not a proven mechanism. It deserves peer review because the empirical pattern is important for educational AI and multimodal RAG. I'd send it out, but insist on significance testing, retrieval quality measurement, training details, and releasing artifacts before publication.

Recommendation: engage with it—read the tables, cite the observation—but don't let the "catastrophic context interference" terminology get a free pass.

Referee Report

4 major / 4 minor

Summary. The paper benchmarks two open multimodal large language models, LLaVA-1.5 and LLaMA 3.2-Vision, on the CK12-QA textbook question answering benchmark. It introduces a multimodal retrieval-augmented generation (RAG) pipeline using ImageBind embeddings and KDB.AI to retrieve lesson paragraphs and diagrams, and evaluates the models under zero-shot and fine-tuned conditions across varying amounts of retrieved context (0p-0I, 1p-1I, 3p-3I). The central reported finding is a sharp drop in LLaMA 3.2-Vision's diagram-question accuracy when retrieved context is added (from 74.07% to 25.93% on validation, and from 74.37% to 15.28% on test), which the authors term 'catastrophic context interference.' Fine-tuning results show LLaMA recovering to 71.16% test accuracy under the 3p-3I condition, while LLaVA's fine-tuned accuracy declines to 27.58%, a divergence attributed to architectural differences in multimodal fusion.

Significance. The empirical observation that a strong vision-language model can be substantially harmed by retrieved multimodal context is practically important for retrieval-augmented generation and AI-in-education applications. The paper's use of a public benchmark (CK12-QA), open model weights, and a reproducible retrieval pipeline are strengths, as is the systematic comparison across two architectures and three context depths. The fine-tuning results suggesting a qualitative difference in how LLaMA and LLaVA adapt to context are interesting. However, the central 'catastrophic context interference' claim currently rests on a qualitative retrieval inspection rather than on quantitative retrieval-quality evidence or a control condition, and the paper's own qualitative analysis shows that retrieved passages were often topically related but not directly relevant to the specific question. The 'statistically significant' phrasing is unsupported by any statistical test, and at least one table contains an internal arithmetic inconsistency. These issues affect the main interpretation, so the central claim is defensible but not yet established.

major comments (4)

[IV-D, IV-E] The attribution of the accuracy drop to model-side 'catastrophic context interference' is not yet supported. Section IV-E supports retrieval quality with only a 'brief inspection,' while Section IV-D states that retrieved text was 'semantically related to the question's topic' but 'not directly relevant to answering the specific query' and that the model 'would often seize upon keywords in the distracting text.' That pattern is equally consistent with a retriever returning topically related but non-salient context, or with a prompt-format artifact. To isolate the model's integration failure, the authors should report quantitative retrieval quality (e.g., human/LLM relevance judgments, Precision@k, or selected examples with annotations) and include a control condition with gold or oracle-relevant context (e.g., the lesson paragraph actually associated with the question in CK12-QA). Without such a control, the drop from 74.07% to 25.93% on validation diagram questions cannot be uniquely attributed to the MLLM's inability to integrate context rather than to retrieval noise.
[Abstract, IV-C] The abstract and Section IV-C describe the observed degradation as a 'statistically significant phenomenon,' but no statistical test, confidence interval, or error bar is reported anywhere in the paper. The authors should report significance tests (e.g., McNemar's test or a bootstrap procedure) comparing accuracy between the 0p-0I and 3p-3I conditions, and ideally provide confidence intervals on the reported accuracy values. The effect sizes are large, so the significance claim may hold, but the current manuscript does not provide the evidence needed to support the phrase 'statistically significant.'
[Table I] Table I contains an internal arithmetic inconsistency. For LLaVA on the validation set, the 1p-1I and 3p-3I columns show identical accuracy for every reported category: Diagram Qs 42.68 vs. 42.68, T-T/F 50.40 vs. 50.40, T-MC 45.29 vs. 45.29, and All Non-Diagram 47.31 vs. 47.31. Yet the All row reports 44.89 for 1p-1I and 48.92 for 3p-3I. If 'All' is a weighted combination of the category accuracies (as stated), it must be identical in both columns. This suggests a data-handling error or an inconsistent definition of the 'All' row. The authors should correct the table and verify the corresponding claims about LLaVA improving with more retrieved context (e.g., the sentence in Section IV-A comparing validation accuracy from 44.45% to 48.92%).
[III-B, IV-B] The fine-tuning results are presented as central evidence for architectural differences, but the experimental setup omits essential details: learning rate, batch size, number of epochs, optimizer, whether fine-tuning was full or parameter-efficient (e.g., LoRA), the data split used for training, any regularization or hyperparameter search, and whether the same protocol was used for both models. Without these details, the observed LLaVA collapse after fine-tuning (from 48.92% to 27.78% validation overall accuracy) could be due to undertraining, overfitting, or mismatched training budgets rather than to a fundamental architectural limitation. The paper itself mentions 'inadequate regularization or hyperparameter tuning' as a possible contributing factor; this possibility needs to be controlled or at least documented before drawing conclusions about modality fusion architecture.

minor comments (4)

[III-A] The notation '0p-01, 1p-11, and 3p-3I' appears to mix the digit '0' and the letter 'I'; it should be '0p-0I, 1p-1I, and 3p-3I' consistently throughout the text and tables.
[III-C] The dataset description repeats the word 'comprising' ('comprising comprising over 26,000 questions'), and Section III-C contains grammatical errors such as 'Utilizes' at the start of a sentence and 'difficultly' in Section IV-F. A careful proofread is needed.
[Fig. 1] The caption says 'both diagram-based and non-diagram-based formats' but the parenthetical text says '(NDQ)' twice; the second occurrence should presumably be 'DQ'. Please clarify the abbreviations.
[Fig. 2] The caption says the plot compares performance 'under four input settings,' but the paper describes only three input settings (0p-0I, 1p-1I, 3p-3I). Please correct the caption or clarify what the fourth setting is.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the evaluation is empirical, uses external benchmarks and model weights, and the sole self-citation is not load-bearing.

full rationale

The paper's central claims are empirical measurements, not derivations from fitted parameters or self-referential definitions. Accuracy differences across input configurations (Tables I-II) are observed results on the external CK12-QA benchmark using external model weights (LLaVA-1.5, LLaMA 3.2-Vision), so there is no fitted quantity that is renamed as a prediction. The term 'catastrophic context interference' is a label for the observed drop in diagram-question accuracy, not a construct defined so that the drop follows by construction; the phenomenon is reported as a measurement. Fine-tuning uses standard autoregressive cross-entropy loss on CK12-QA and is evaluated on the held-out test split, which is a normal train/test evaluation and not circular. The only self-citation, [15], appears in related work as background for textual retrieval-augmented QA and is not used to justify the multimodal experimental setup or the interference finding, so it is not load-bearing. The paper itself flags a genuine validity limitation: Section IV-E supports retrieval relevance only by 'a brief inspection of retrieval results,' and Section IV-C concedes the pipeline 'may be providing context that is topically relevant but not optimally salient.' This undermines the strength of the attribution to model-side interference, but it is an empirical/internal-validity concern, not circularity: the raw accuracy numbers would stand regardless of how the drop is interpreted. No step in the paper reduces to its own inputs by definition, by fitted-input renaming, or through a self-citation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on a fixed retrieval depth (k=3) and on an implicit assumption that the retrieval is relevant enough to isolate model-side interference. No independent falsifiable entity is introduced beyond the named phenomenon 'catastrophic context interference,' which is a label for observed behavior rather than a postulated mechanism.

free parameters (2)

top-k retrieval context count (k=3) = 3 paragraphs, 3 images
The central effect (catastrophic context interference) is demonstrated at k=3; no ablation over k is reported, so the threshold at which degradation begins is unknown.
fine-tuning hyperparameters (unspecified)
LLaMA's recovery to 71.16% and LLaVA's collapse to 27.58% could depend strongly on learning rate, epochs, and data order, none of which are reported.

assumptions (3)

domain assumption Ground-truth answers in CK12-QA are correct and unambiguous
Used as labels for exact-match accuracy in Eq. (2); no human verification or error analysis of dataset labels is performed.
domain assumption ImageBind embeddings align text and images well enough that cosine-similarity retrieval returns relevant context
Retrieval relevance is only checked by 'brief inspection' (Section IV-E), yet the central interpretation of the drop as model-side interference assumes retrieval quality is adequate.
domain assumption The MLLMs' letter-only outputs are reliable and parseable; a wrong-format output is treated as an incorrect answer
The models are constrained to output a letter; any failure to comply counts as wrong, which could inflate the degradation if the context makes the model refuse or shift format.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Multimodal Large Language Models on Educational Textbook Question Answering." pith.science (2026). https://pith.science/paper/PCRQIQEX

@misc{pith2026250621596,
  author       = {Pith},
  title        = {Pith review of: Evaluating Multimodal Large Language Models on Educational Textbook Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCRQIQEX}},
  note         = {Machine review of arXiv:2506.21596}
}

read the original abstract

Multimodal large language models (MLLMs) have shown success in vision-language tasks, but their ability to reason over complex educational materials remains largely untested. This work presents the first evaluation of state-of-the-art MLLMs, including LLaVA-1.5 and LLaMA 3.2-Vision, on the textbook question answering (TQA) task using the CK12-QA dataset. We introduce a multimodal retrieval-augmented generation (RAG) pipeline to simulate real-world learning by providing relevant lesson paragraphs and diagrams as context. Our zero-shot experiments reveal a critical trade-off: while retrieved context improves LLaVA's performance on text-based questions, it significantly degrades the accuracy of the more powerful LLaMA 3.2-Vision on diagram-based tasks, dropping its validation accuracy from 74.07% to 25.93%. We term this statistically significant phenomenon "catastrophic context interference." Furthermore, fine-tuning highlights architectural differences: LLaMA 3.2-Vision's performance improves to 71.16% on the test set, demonstrating its capacity to learn multimodal integration, whereas LLaVA's performance declines, indicating challenges with generalization. Our results underscore the challenges MLLMs face in modality prioritization and context integration, providing a benchmark and pointing to key directions for developing more robust AI-driven educational tools.

Figures

Figures reproduced from arXiv: 2506.21596 by the authors.

**Figure 1.** Examples from the CK12-QA dataset [6], illustrating various question types, including both diagram-based and non-diagram-based formats. (NDQ) and diagram-based questions (NDQ), each paired with corresponding lesson texts and images to simulate realistic curriculum-based reasoning scenarios. By augmenting the questions with retrieved lesson content, we develop a retrievalaugmented generation (RAG) pipeline that more… view at source ↗

**Figure 2.** Accuracy of LLaVA and LLaMA 3.2-Vision across input configurations. The plot compares performance on DQ and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Comparison of zero-shot (3p-3I) and fine-tuned (3p [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Comparison of zero-shot (3p-3I) and fine-tuned (3p [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 18 canonical work pages

[1]

Visual instruction tuning,

H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

2023
[2]

Improved baselines with visual instruction tuning,

H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” arXiv:2310.03744, 2023

arXiv 2023
[3]

The llama 3 herd of models,

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

arXiv 2024
[4]

Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education,

A. Bewersdorff, C. Hartmann, M. Hornberger, K. Seßler, M. Bannert, E. Kasneci, G. Kasneci, X. Zhai, and C. Nerdel, “Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education,” Learn. Individ. Differ ., vol. 118, p. 102601, 2025

work page 2025
[5]

On opportunities and challenges of large multimodal foundation models in education,

S. K ¨uchemann, K. E. Avila, Y . Dinc, C. Hortmann, N. Revenga, V . Ruf, N. Stausberg, S. Steinert, F. Fischer, M. Fischer et al., “On opportunities and challenges of large multimodal foundation models in education,” NPJ Sci. Learn. , vol. 10, no. 1, p. 11, 2025

work page 2025
[6]

Are you smarter than a sixth grader? textbook question an- swering for multimodal machine comprehension,

A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Ha- jishirzi, “Are you smarter than a sixth grader? textbook question an- swering for multimodal machine comprehension,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2017, pp. 4999–5007

work page 2017
[7]

A review on vision-language-based approaches: Challenges and applications,

H.-T. Ho, L. V . Nguyen, M.-T. Pham, Q.-H. Pham, Q.-D. Tran, D. N. M. Huy, and T.-H. Nguyen, “A review on vision-language-based approaches: Challenges and applications,” Comput. Mater . Continua , vol. 82, no. 2, 2025

work page 2025
[8]

Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2023, pp. 19 730– 19 742

work page 2023

Show all 23 references

[9]

Minigpt-4: Enhancing vision-language understanding with advanced large language models,

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” in Proc. 12th Int. Conf. Learn. Represent. (ICLR) , 2024

2024
[10]

Making the v in VQA matter: Elevating the role of image understanding in visual question answering,

Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in VQA matter: Elevating the role of image understanding in visual question answering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 6904–6913

2017
[11]

Ok-VQA: A visual question answering benchmark requiring external knowledge,

K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-VQA: A visual question answering benchmark requiring external knowledge,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2019, pp. 3195– 3204

2019
[12]

Microsoft COCO: Common objects in context,

T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. 13th Eur . Conf. Comput. Vis. (ECCV) . Zurich, Switzerland: Springer, Sep 2014, pp. 740–755, part 13

2014
[13]

SPIQA: A dataset for multimodal question answering on scientific papers,

S. Pramanick, R. Chellappa, and S. Venugopalan, “SPIQA: A dataset for multimodal question answering on scientific papers,” arXiv preprint arXiv:2407.09413, 2024

2024 arXiv
[14]

Eduvqa: A multimodal visual question answering framework for smart education,

J. Xiao and Z. Zhang, “Eduvqa: A multimodal visual question answering framework for smart education,” Alex. Eng. J. , vol. 122, pp. 615–624, 2025

2025
[15]

Enhancing textual textbook question answering with large language models and retrieval augmented generation,

H. A. Alawwad, A. Alhothali, U. Naseem, A. Alkhathlan, and A. Jamal, “Enhancing textual textbook question answering with large language models and retrieval augmented generation,” Pattern Recognit., vol. 162, p. 111332, 2025

2025
[16]

Enhancing textbook question answering with knowledge graph-augmented large language models,

M. He, A. Zhou, and X. Shi, “Enhancing textbook question answering with knowledge graph-augmented large language models,” in Proc. 16th Asian Conf. Mach. Learn. (ACML) , 2024

2024
[17]

ISAAQ - mastering textbook questions with pre-trained transformers and bottom-up and top-down attention,

J. M. Gomez-Perez and R. Ortega, “ISAAQ - mastering textbook questions with pre-trained transformers and bottom-up and top-down attention,” in Proc. 2020 Conf. Empir . Methods Nat. Lang. Process. (EMNLP), B. Webber, T. Cohn, Y . He, and Y . Liu, Eds. Online: Association for Co...

2020
[18]

Imagebind: One embedding space to bind them all,

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2023, pp. 15 180–15 190

2023
[19]

KDB.AI: The scalable vector database for ai,

KX, “KDB.AI: The scalable vector database for ai,” https://kdb.ai/, 2023, accessed: 2025-05-22

2023
[20]

GPT-4v(ision) system card,

OpenAI, “GPT-4v(ision) system card,” https://openai.com/index/gpt-4v- system-card/, 2023, accessed: 2025-05-22

2023
[21]

Gemini: A family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millicanet al., “Gemini: A family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[22]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2021, pp. 8748–8763

2021
[23]

Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality,

W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al. , “Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality,” https://vicuna.lmsys.org, 2023, accessed: Apr. 14, 2023

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Visual instruction tuning,

H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

2023

[2] [2]

Improved baselines with visual instruction tuning,

H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” arXiv:2310.03744, 2023

arXiv 2023

[3] [3]

The llama 3 herd of models,

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

arXiv 2024

[4] [4]

Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education,

A. Bewersdorff, C. Hartmann, M. Hornberger, K. Seßler, M. Bannert, E. Kasneci, G. Kasneci, X. Zhai, and C. Nerdel, “Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education,” Learn. Individ. Differ ., vol. 118, p. 102601, 2025

work page 2025

[5] [5]

On opportunities and challenges of large multimodal foundation models in education,

S. K ¨uchemann, K. E. Avila, Y . Dinc, C. Hortmann, N. Revenga, V . Ruf, N. Stausberg, S. Steinert, F. Fischer, M. Fischer et al., “On opportunities and challenges of large multimodal foundation models in education,” NPJ Sci. Learn. , vol. 10, no. 1, p. 11, 2025

work page 2025

[6] [6]

Are you smarter than a sixth grader? textbook question an- swering for multimodal machine comprehension,

A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Ha- jishirzi, “Are you smarter than a sixth grader? textbook question an- swering for multimodal machine comprehension,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2017, pp. 4999–5007

work page 2017

[7] [7]

A review on vision-language-based approaches: Challenges and applications,

H.-T. Ho, L. V . Nguyen, M.-T. Pham, Q.-H. Pham, Q.-D. Tran, D. N. M. Huy, and T.-H. Nguyen, “A review on vision-language-based approaches: Challenges and applications,” Comput. Mater . Continua , vol. 82, no. 2, 2025

work page 2025

[8] [8]

Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2023, pp. 19 730– 19 742

work page 2023