Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Knowledge-graph augmentation gives no reliable accuracy gain in controlled document RAG; image evidence is necessary but not sufficient for pixel-only questions, and caption-built figure benchmarks overstate text-only visual QA.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:26 UTC pith:THIIX3OA

load-bearing objection A thoughtful controlled evaluation whose design is the contribution; treat the headline numbers as conditional until the LLM judges are validated against humans. the 3 major comments →

arxiv 2607.16604 v1 pith:THIIX3OA submitted 2026-07-18 eess.IV

When Do Multimodal and Graph-Augmented RAG Help? A Controlled Evaluation for Document Question Answering

Sokipriala Jonah This is my paper
classification eess.IV
keywords retrieval-augmented generationknowledge graphsmultimodal retrievalvision-language modelsdocument question answeringfigure-question benchmarksLLM-as-judge evaluationefficiency analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to determine when augmenting a text-based retrieval-augmented generation (RAG) system with knowledge graphs and images actually helps document question answering. It builds a late-fusion pipeline in which text passages, LLM-extracted subject–relation–object triples, and contrastive image–text retrieval of figure/table crops are retrieved independently and combined only at generation time, then ablates each evidence source across four multimodal generators and three question types on 1,000 scientific pages. Its central result: graph evidence yields no reliable accuracy gain in this corpus, image evidence is necessary but not sufficient for pixel-only figure questions (text-only systems score zero; multimodal systems score 0.057–0.114), and caption-generated questions substantially overestimate text-only visual question answering. The point of the study is that multimodal and graph augmentation are not universally beneficial — their value depends on retrieval design, corpus structure, benchmark construction, and generator cost/efficiency.

Core claim

On the paper’s own terms, the discovery is a set of conditional negative and positive results from a controlled four-way ablation. Holding the text index, visual index, knowledge graph, and retrieval parameters fixed, the authors show that adding knowledge-graph triples to the prompt changes mean accuracy by +0.028 on text questions, –0.017 on multi-hop questions, and 0.000 on figure questions, with direction varying by generator — so no reliable improvement. Adding one retrieved image moves pixel-only figure accuracy from exactly zero to 0.057–0.114 across generators, but the correct crop is ranked first only 22.9% of the time (Recall@3 = 0.371), and even when the right crop is supplied, ge

What carries the argument

The load-bearing mechanism is a late-fusion, three-branch RAG architecture: (i) text passages embedded with a text encoder and retrieved by cosine similarity; (ii) a directed knowledge graph of LLM-extracted subject–relation–object triples matched by whole-word entity overlap, with provenance restricted to pages already retrieved by text; (iii) a contrastive image–text embedding model (named CLIP in the paper) ranking figure/table crops by similarity, with scoring at depth 5 while only the top crop is passed as pixels or the top three captions as text. The branches are combined only by concatenation into a single generation prompt, so α, β ∈ {0,1} flags ablate graph and visual evidence witho

Load-bearing premise

As the paper acknowledges, all accuracy, faithfulness, and relevancy scores come from LLM judges with no human validation, inter-judge agreement, or calibration, and the 30–35 questions per condition make small accuracy deltas fragile; if judge verdicts are biased by graph triples, image tokens, or answer length, the central null results collapse.

What would settle it

Have two independent human annotators score the full set of responses across the four configurations and the caption-answerable/pixel-only question sets, then compare against the LLM judge verdicts; if human–judge agreement is low or judge scores systematically track graph-triple presence, image tokens, or answer length, the reported ablation and zero-text-accuracy results are called into question.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On pixel-only figure questions, text-only configurations always score zero, so any system intended to answer questions whose answers appear only in an image must include a visual retrieval branch.
  • Because the higher-cost diagnostic generator added only two correct answers, improving visual retrieval and document representation is likely to pay off more than scaling up the generator.
  • Benchmarks built from captions substantially overestimate text-only visual question answering; evaluations should report whether figure questions are caption-answerable and how many reference answers appear in captions.
  • Knowledge-graph augmentation is not shown to be useless in general; its lack of benefit here is tied to a corpus of unrelated pages where provenance-filtered triples mostly restate retrieved text, so graph RAG needs evaluation on collections with real cross-document relations.
  • Image tokenisation varied elevenfold across generators for the same input, so multimodal deployment cost must be measured directly rather than inferred from token prices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Recall@1 ceiling of 0.229 on pixel-only questions implies that even a perfect figure reader would cap the system around 23% accuracy; feeding the gold crop directly to each generator would separate the retrieval bottleneck from the reading bottleneck.
  • With 30–35 questions per condition and no reported judge calibration, the small deltas (e.g., +0.028 text accuracy from graph evidence) are likely within noise; larger, human-validated replication could convert the null result from suggestive to definitive.
  • The elevenfold token variation suggests 'cost per correct answer' is the decision metric that matters; recomputing the paper's cost table with cost divided by accuracy would likely reorder the recommended backends for multimodal figure QA.
  • Retrofitting existing figure-QA benchmarks with a pixel-only protocol could reveal how much of their published scores are actually caption recovery rather than visual reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a late-fusion multimodal graph-RAG pipeline in which text passages, LLM-extracted subject–relation–object triples, and CLIP-retrieved figure/table crops are retrieved independently and combined only at generation time. Using 1,000 PubLayNet pages, the authors run a four-way ablation (baseline, +KG, +multimodal, +both) across four multimodal generators and three question types, and also compare matched caption-answerable and pixel-only figure questions. The main reported findings are that knowledge-graph augmentation gives no reliable accuracy improvement; text-only systems score zero on pixel-only figure questions while multimodal systems reach 0.057–0.114; caption-derived question sets substantially overestimate text-only visual question answering; and image tokenization causes an elevenfold variation in input tokens across generators. The paper makes its code and evaluation resources public and explicitly acknowledges limitations around sample size, single-corpus scope, and LLM-based judges.

Significance. The study is potentially valuable for the RAG evaluation community because it fixes the retrieval indices, prompts, and evidence payloads across conditions, allowing evidence-configuration effects to be separated from generator choice. The matched caption-answerable versus pixel-only protocol is a useful methodological contribution, and the cost/token measurements (Section IV-I, Table V) are a practical addition. The strengths include the controlled design, the separation of judge models from generators, the manually reviewed question sets, and the public release of code and configurations. However, the quantitative conclusions rest on small per-cell samples (n=30–35) and on LLM-judge verdicts with no reported human validation or calibration. Since the headline effects are often only two to four questions, the central claims are currently conditional on judge reliability and on absence of sampling noise; the paper itself acknowledges this in Section V-A but the abstract states the findings in stronger terms.

major comments (3)
  1. [Section IV-D; Tables II and III] All accuracy, faithfulness, and relevancy scores are binary verdicts produced by DeepSeek-Chat and Claude Haiku 4.5 following RAGAS, with no human validation, inter-judge agreement, or calibration reported. This is load-bearing because the effects are small: on pixel-only figure questions the entire multimodal accuracy range of 0.057–0.114 corresponds to 2–4 correct answers out of 35, and the KG deltas in Section IV-F are also 1–3 questions. A modest judge bias—toward longer answers, toward answers containing image tokens, or against abstentions—could flip the reported zero-versus-nonzero gap or erase the claimed lack of KG benefit. Treating unparseable judge outputs as zero makes this worse. Please report a human-validated subset with per-metric agreement, or at minimum a sensitivity analysis that re-scores a random sample with an alternative protocol and shows the conclusions are stabl
  2. [Section IV-E/F; Table II] No confidence intervals, significance tests, or effect-size bounds are reported for any accuracy, faithfulness, or relevancy comparison. With n=35 for text/figure questions and n=30 for multi-hop questions, the statement that KG augmentation provides 'no reliable accuracy improvement' is inferred from changes as small as +0.028 or −0.017, i.e., roughly one question. A bootstrap interval or exact test would show whether the observed ordering is within sampling noise. Without this, the negative KG result is not established as a stable null; it is at best a descriptive observation for this corpus. Please add uncertainty quantification and state the minimum effect the design can detect.
  3. [Section IV-H; Figure 2] The paper correctly notes that pixel-based generators receive only the top-ranked crop and that Recall@1=0.229 (8 of 35 questions) is the operational retrieval ceiling. But the conclusion that accuracy is 'constrained by both image retrieval and the generators' ability to interpret dense scientific figures' would be more directly supported by conditioning: report multimodal accuracy on the subset where the gold crop is retrieved (n=8), and compare it with accuracy on non-retrieved cases. With only 35 questions and LLM judges, the current decomposition is suggestive rather than demonstrated.
minor comments (5)
  1. [Section IV-G] The explanation that text-only systems answer caption-answerable questions because 'captions can be retrieved as text' is ambiguous. Section III-B describes GPT-4o-generated captions used by the caption-mediated path and question protocol, but does not say those generated captions are in the text index. If text-only accuracy instead comes from original figure captions appearing in OCR text, please say so explicitly to avoid the impression that generated captions are available to the baseline.
  2. [Section III-B] Please clarify how the 1,000 pages are sampled from small-publaynet-wds and whether any pages are shared across the text, multi-hop, and figure question sets. The figure-question pool is described as 598 crops; stating the overlap and selection process would help assess independence.
  3. [Table V] The cost and token figures would benefit from a short note on whether failed requests, retries, and Gemini's billed reasoning tokens are included in the per-question cost, and whether the token counts refer to billed or observed tokens.
  4. [References] Reference [4] contains a placeholder 'tODO: add arXiv identifier / final venue once confirmed'. This must be resolved before publication.
  5. [Section III-C3] The text says the two visual-input variants use the same CLIP ranking, but pixel input is used only for figure questions while text and multi-hop questions use captions. This gating is stated later but could be noted in Section III-C3 as well to avoid confusion.

Circularity Check

0 steps flagged

No significant circularity: the paper is a controlled experimental ablation with external question authors/judges and a fixed retrieval stack; the acknowledged LLM-judge limitation is a validity concern, not a circular step.

full rationale

The paper's central claims are empirical ablations, not derivations. The four configurations differ only by α, β in Eq. (4), and the retrieval indices are constructed once and reused, so the +KG and +multimodal comparisons are not fitted to their outcomes. Question authors (DeepSeek-Chat, Claude Haiku 4.5) and the LLM judges are external to the four production generators, and RAGAS-based binary verdicts are a measurement choice rather than a fitted parameter renamed as a prediction. The caption-answerable protocol deliberately conditions question generation on GPT-4o captions; the paper explicitly presents this as the tested intervention ('This design distinguishes caption-based answer recovery from question answering that requires direct interpretation'), so it is not a hidden dependency. No load-bearing self-citation or imported uniqueness theorem appears; all cited methods (CLIP, RAGAS, PubLayNet, GraphRAG) are external. The paper's own limitation section (V-A) states that 'Answer quality was assessed using binary LLM judges rather than human annotation'—this is a measurement-validity caveat, not circularity. The small per-cell samples (30–35 questions) and unvalidated judges could weaken the reported comparisons, but they do not make any result equivalent to its input by definition. Reference [4] contains a TODO for an arXiv identifier, but that is a bibliographic artifact, not an argumentative dependency. Hence no circular step can be exhibited.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claims rest on several hand-set design choices and assumed instrument validities rather than on derived parameters. The hyperparameters (k=3, m=5, KG size, chunk size) are fixed by the authors, so the findings about KG and visual evidence are conditional on these choices and on the corpus. The paper itself acknowledges this conditionality, but the ledger makes the dependency explicit.

free parameters (5)
  • Top-k text chunks (k) = 3
    Design choice for text retrieval depth; affects how much context the generator receives. Fixed across all conditions, not fitted to outcomes.
  • Visual scoring depth (m) = 5
    CLIP retrieves and scores at depth 5, but the generator receives only the top-1 crop (pixel) or top-3 captions. Hand-set design choice.
  • KG neighborhood size = 8
    Up to eight neighboring triples are added to the prompt. Arbitrary design choice that affects the KG branch's contribution.
  • Entity minimum length = 5 characters
    Filter to reduce incidental entity matches; changes which graph facts are retrieved.
  • Text chunk size / overlap = 500 / 50 chars
    Chunking parameters for OCR text, set because extracted text has inconsistent punctuation. Affects retrieval granularity.
axioms (5)
  • domain assumption The small-publaynet-wds 1,000-page subset is representative of PubLayNet and of scientific document QA generally.
    Used to draw conclusions about multimodal/graph RAG beyond the specific corpus; the paper acknowledges this limitation in Section V-A.
  • domain assumption LLM judges (RAGAS framework) provide valid binary verdicts for accuracy, faithfulness, and relevancy.
    All quantitative results rest on judge outputs; no human validation or inter-judge agreement is reported (Section IV-D).
  • domain assumption CLIP ViT-B/32 text-image similarity is an adequate measure of figure relevance for document QA.
    The visual retriever relies on this; the paper's recall results show limitations, but it assumes the embedding captures enough semantics (Sections III-B, III-C).
  • domain assumption GPT-4o-mini triple extraction produces a knowledge graph whose edges are correct and useful for QA.
    The KG branch depends on LLM-extracted triples; no manual verification of triple correctness is reported (Section III-C.2).
  • domain assumption Tesseract OCR preserves tabular and textual content accurately enough for text-only baselines.
    Tables are OCR'd into text; imperfect OCR could confound the comparison between text-only and multimodal systems (Section III-B-a).

pith-pipeline@v1.3.0-alltime-deepseek · 11016 in / 14751 out tokens · 141054 ms · 2026-08-01T20:26:49.140792+00:00 · methodology

0 comments
read the original abstract

Retrieval-augmented generation (RAG) systems commonly operate on text extracted from documents, potentially losing information contained in figures, tables, layout, and relationships distributed across passages. We present an explainable multimodal graph-RAG architecture that augments a text-only baseline with LLM-extracted subject--relation--object triples and CLIP-based retrieval of figures and tables. The three evidence sources are retrieved independently and fused only at generation time, allowing the effects of graph evidence, visual evidence, and generator choice to be evaluated separately. We conduct a controlled four-way ablation on 1,000 PubLayNet pages across single-passage, multi-hop, and figure questions using two closed-weight and two open-weight multimodal generators. We also compare matched caption-answerable and pixel-only figure question sets to distinguish caption recovery from genuine visual question answering. Knowledge-graph augmentation provides no reliable accuracy improvement across generators or question types in this corpus. On pixel-only questions, text-only systems achieve zero accuracy, while multimodal systems reach 0.057--0.114 and are constrained by both image retrieval (Recall@3 = 0.371) and the generators' ability to interpret dense scientific figures and tables. Caption-derived questions substantially overestimate text-only visual question-answering capability. Processing the same image also produces an elevenfold variation in input tokens across generators, showing that image tokenisation can dominate deployment cost. Source code, experiment configurations, and evaluation resources are publicly available at https://github.com/sokistar24/multimodal-graph-rag. These findings show that the value of multimodal and graph augmentation depends on retrieval design, corpus structure, benchmark construction, and generator efficiency.

Figures

Figures reproduced from arXiv: 2607.16604 by Sokipriala Jonah.

Figure 1
Figure 1. Figure 1: Multimodal graph-RAG architecture. Three evidence sources are retrieved independently and fused late into a single generation prompt. The knowledge [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cost and accuracy on pixel-only figure questions for the multimodal [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Input tokens per question for the multimodal configuration. All models [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 8 linked inside Pith

  1. [1]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. tau Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459–9474

    2020

  2. [2]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    CRAG – comprehensive RAG benchmark,

    X. Yang, K. Sun, H. Xin, Y . Sun, N. Bhalla, X. Chen, S. Choudhary, R. D. Gui, Z. W. Jiang, Z. Jiang, L. Kong, B. Moran, J. Wang, Y . E. Xu, A. Yan, C. Yang, E. Yuan, H. Zha, N. Tang, L. Chen, N. Scheffer, Y . Liu, N. Shah, R. Wanga, A. Kumar, W. tau Yih, and X. L. Dong, “CRAG – comprehensive RAG benchmark,” inAdvances in Neural Information Processing Sys...

    2024

  4. [4]

    PixelRAG: Web screenshots beat text for retrieval-augmented generation,

    Y . Wang, Z. Li, Z. Wang, P. Teiletche, L. Jin, M. Zaharia, J. E. Gonzalez, and S. Min, “PixelRAG: Web screenshots beat text for retrieval-augmented generation,”arXiv preprint, 2025, tODO: add arXiv identifier / final venue once confirmed

    2025

  5. [5]

    MMed-RAG: Versatile multimodal RAG system for medical vision language models,

    P. Xia, K. Zhu, H. Li, T. Wang, W. Shi, S. Wang, L. Zhang, J. Zou, and H. Yao, “MMed-RAG: Versatile multimodal RAG system for medical vision language models,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  6. [6]

    ColPali: Efficient document retrieval with vision language models,

    M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo, “ColPali: Efficient document retrieval with vision language models,”arXiv preprint arXiv:2407.01449, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    From local to global: A graph RAG approach to query- focused summarization,

    D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, and J. Larson, “From local to global: A graph RAG approach to query- focused summarization,”arXiv preprint arXiv:2404.16130, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    M3DocRAG: Multi-modal retrieval is what you need for multi-page multi-document understanding,

    J. Cho, D. Mahata, O. Irsoy, Y . He, and M. Bansal, “M3DocRAG: Multi-modal retrieval is what you need for multi-page multi-document understanding,”arXiv preprint arXiv:2411.04952, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Bench- marking retrieval-augmented multimodal generation for document ques- tion answering,

    K. Dong, Y . Chang, S. Huang, Y . Wang, R. Tang, and Y . Liu, “Bench- marking retrieval-augmented multimodal generation for document ques- tion answering,”arXiv preprint arXiv:2505.16470, 2025, accepted to the NeurIPS 2025 Datasets and Benchmarks Track

    arXiv 2025

  10. [10]

    MKG-RAG-Bench: Benchmarking retrieval in multimodal knowledge graph-augmented gen- eration,

    X. Wang, B. Hoang, H. Liu, T. Wang, and F. Ma, “MKG-RAG-Bench: Benchmarking retrieval in multimodal knowledge graph-augmented gen- eration,”arXiv preprint arXiv:2606.26458, 2026, accepted at KDD 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Multimodal graph RAG for long-range vi- sually rich document understanding,

    Y .-C. Wang and C.-S. Chen, “Multimodal graph RAG for long-range vi- sually rich document understanding,”arXiv preprint arXiv:2606.28780, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    PubLayNet: Largest dataset ever for document layout analysis,

    X. Zhong, J. Tang, and A. J. Yepes, “PubLayNet: Largest dataset ever for document layout analysis,” inProceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 1015–1022

    2019

  13. [13]

    Dense passage retrieval for open-domain question an- swering,

    V . Karpukhin, B. O ˘guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. tau Yih, “Dense passage retrieval for open-domain question an- swering,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6769–6781

    2020

  14. [14]

    RAGAS: Au- tomated evaluation of retrieval augmented generation,

    S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “RAGAS: Au- tomated evaluation of retrieval augmented generation,” inProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2024, pp. 150–158

    2024

  15. [15]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inProceedings of the 38th International Conference on Machine Learning (ICML), 2021, pp. 8748–8763

    2021

  16. [16]

    Exploring network structure, dynamics, and function using NetworkX,

    A. A. Hagberg, D. A. Schult, and P. J. Swart, “Exploring network structure, dynamics, and function using NetworkX,” inProceedings of the 7th Python in Science Conference (SciPy), 2008, pp. 11–15

    2008

  17. [17]

    GPT-4o system card,

    OpenAI, “GPT-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Gemini: A family of highly capable multimodal models,

    Gemini Team, Google, “Gemini: A family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,

    Meta AI, “The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,” https://ai.meta.com/blog/ llama-4-multimodal-intelligence/, 2025, accessed 18 July 2026

    2025

  20. [20]

    Llama 4 Scout 17B 16E Instruct model card,

    DeepInfra, “Llama 4 Scout 17B 16E Instruct model card,” https://deepinfra.com/meta-llama/Llama-4-Scout-17B-16E-Instruct, 2025, hosted inference model, accessed 18 July 2026

    2025

  21. [21]

    Llama 4 Maverick 17B 128E Instruct FP8 model card,

    DeepInfra, “Llama 4 Maverick 17B 128E Instruct FP8 model card,” https://deepinfra.com/meta-llama/ Llama-4-Maverick-17B-128E-Instruct-FP8, 2025, hosted inference model, accessed 18 July 2026

    2025

  22. [22]

    small-publaynet-wds,

    lhoestq, “small-publaynet-wds,” https://huggingface.co/datasets/lhoestq/ small-publaynet-wds, hugging Face dataset, accessed 18 July 2026

    2026

  23. [23]

    New embedding models and api updates,

    OpenAI, “New embedding models and api updates,” https://openai.com/ index/new-embedding-models-and-api-updates/, Jan. 2024, accessed 18 July 2026

    2024

  24. [24]

    Billion-scale similarity search with GPUs,

    J. Johnson, M. Douze, and H. J ´egou, “Billion-scale similarity search with GPUs,”IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535– 547, 2019

    2019

  25. [25]

    GPT-4o mini: Advancing cost-efficient intelligence,

    OpenAI, “GPT-4o mini: Advancing cost-efficient intelligence,” https: //openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/, Jul. 2024, accessed 18 July 2026

    2024