Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models

Farhad Imani; Hongyi Xu; Kiarash Naghavi Khanghah; Lela Romeo; Qian Yang; Rajiv Malhotra; Zhiling Chen

REVIEW 3 major objections 4 minor 5 references

Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A retrieval-augmented pipeline can classify manufacturing anomalies from literature alone, without training data.

desk verdict Genuinely new multimodal RAG pipeline for AM anomaly classification, but zero-shot claim unestablished because the retrieval corpus includes the test set. read the letter →

arxiv 2505.13828 v1 pith:SABEMODD submitted 2025-05-20 cs.AI

Kiarash Naghavi Khanghah , Zhiling Chen , Lela Romeo , Qian Yang , Rajiv Malhotra , Farhad Imani , Hongyi Xu This is my paper · ORCID

classification cs.AI

keywords multimodalRAGlaserpowderbedfusionanomalydetectionzero-shotclassificationlargelanguagemodelsmanufacturingdefectsretrieval-augmentedgenerationvisualdocumentretrieval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a fully literature-driven system can detect, classify, and explain defects in laser powder bed fusion images without any labeled training data. The system retrieves relevant images and text from published papers—images through ColPali and text through embedding-based search—then feeds a retrieved reference image, its description, and scientific context to a multimodal large language model that judges each candidate anomaly. On four L-PBF datasets, the framework reports that the stronger model, GPT-4o-mini, beats both a smaller open model and a proportional-random baseline, and that adding retrieval raises average classification accuracy from about 50% to 62%. If true, the approach means anomaly-detection capability can be updated by indexing new research rather than retraining models on new defect examples.

What carries the argument

The load-bearing mechanism is the multimodal RAG loop: a dual retriever—ColPali, a visual document retriever that indexes PDF pages as images rather than OCR text, for images, and text-embedding-ada-002 with GPT-4o-mini for text—composes a prompt pairing each test image with a retrieved exemplar image and retrieved scientific context, and an MLLM makes a per-anomaly binary decision. The per-anomaly prompting and one-hot output module turn a generative model into a measurable classifier. Retrieval is what supplies the 'training' content, so no model weights are updated.

What would settle it

Check the retrieval corpus for overlap with the test images: run the same image-retrieval queries with a test image inserted and see whether the top-ranked retrieval is that image or its ground-truth label; then rerun Table 3 with all documents that contain test images or labels removed. If accuracy falls back to the no-retrieval numbers, the 12-point improvement is leakage rather than generalization.

Watch

Extended reading notes

Core claim

The paper's central claim is that retrieval-augmented generation can replace training data for visual anomaly classification in additive manufacturing. In the proposed pipeline, text and image retrievers pull anomaly-specific passages and example images from a corpus of L-PBF papers; the retrieval output is packaged into a structured prompt with a reference image and visual description; and an MLLM answers, for each anomaly type, whether that anomaly is present in a test image. The output module converts the per-anomaly judgments into a one-hot vector so accuracy can be scored against ground truth. The paper reports that GPT-4o-mini used this way outperforms Qwen2-VL-2B by an average margin of 34.6% and outperforms the proportional random baseline, and that including the retrieval step improves average accuracy by about 12 percentage points over the same model without retrieval (Table 3). The conclusion drawn is that literature alone can support zero-shot anomaly identification, classification, and explanation.

Load-bearing premise

The result assumes that the paper database searched by the retrieval system does not already contain the test images or their ground-truth labels; if it does, the zero-shot generalization claim is just re-finding the answer in the source material.

Editorial extensions

If this is right

If correct, anomaly-classification capability can be extended to new processes or defect types by indexing new papers rather than collecting labeled images.
The accuracy comparison must account for class imbalance, because a model that simply predicts every anomaly can appear strong on F1 while failing on the accuracy metric used here.
The framework outputs root-cause and prevention explanations tied to retrieved literature, not just labels, making the results actionable for process control.
Because retrieval is the mechanism, improvements in visual document retrieval should directly raise detection accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors do not verify that the test images and ground-truth labels are absent from the retrieval corpus; a reader could test this directly by removing the dataset and taxonomy documents from the corpus and rerunning Table 3.
GPT-4o-mini's own pretrained knowledge likely already encodes some manufacturing anomaly patterns, so the reported 12-point retrieval gain may understate retrieval's true contribution for genuinely novel anomaly types; a controlled study using newly published papers and models frozen before those papers appeared would isolate the retrieval effect.
The explanation quality is not quantitatively evaluated in the paper, only illustrated with one example; a reader could test whether generated explanations cite retrieved evidence or simply reproduce the model's internal knowledge.
The per-anomaly prompting strategy means cost and latency scale with the number of anomaly types, so a production version might need retrieval-based ranking of candidate anomalies before classification.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper reports a genuinely new integration—multimodal RAG for zero-shot L-PBF anomaly classification, using ColPali for image retrieval and GPT-4o-mini/Qwen2-VL for classification—but the central claim is undercut by an evaluation setup that allows the retrieval corpus to contain the test set itself.

What's actually new: the specific combination of image retrieval from scientific literature plus text RAG, feeding a reference image and description to an MLLM for anomaly classification. Prior work is either text-only (RAAD-LLM) or few-shot (AnomalyGPT); this particular pipeline is not in the cited literature. The authors also report an honest comparison of two backbones, including the failure mode of Qwen2-VL-2B, which tends to flag everything as anomalous, and they use accuracy rather than F1 in an imbalanced setting, which is a reasonable choice.

The soft spot is load-bearing. Section 3 states that the test images come from ORNL’s Peregrine dataset [41] and are used 'exclusively for testing.' But Table 2 lists both [41] and Scime et al. [6]—the paper that defined the ORNL anomaly taxonomy—as documents in the RAG corpus, with the anomaly labels used as ground truth. The retriever can therefore supply the MLLM with the test set's own anomaly names, descriptions, and possibly the same images. That means the zero-shot, literature-only claim is not established, and the asserted +12% accuracy gain from retrieval in Table 3 may simply be retrieval of test-set metadata. The paper never checks whether the retrieved top-k items overlap the evaluation images.

Secondary issues are smaller: no error bars (three repeated detections are averaged but variance is not reported), no comparison against the cited zero-shot baselines (AnomalyCLIP, WinCLIP, VMAD), and no code or data artifacts for reproduction. In the appendix, GPT-4o-mini's accuracy is below the random baseline for some anomalies (e.g., Recoater Hopping in AddUp at 0.19 vs 0.96), which the paper notes but does not reconcile with the overall positive claim. Those are fixable and don't change my bottom line.

The citation pattern looks fine and the math is just accuracy arithmetic. This is a genuine system contribution with a fixable evaluation flaw. I'd send it to peer review, with the requirement that the authors remove overlapping sources from the retrieval corpus, report the retrieved items, and re-run the with/without-RAG comparison. If the gain survives that, it's a solid systems paper. If not, the contribution is a well-documented pipeline description rather than evidence for zero-shot generalization.

Referee Report

3 major / 4 minor

Summary. The paper proposes a multimodal retrieval-augmented generation (RAG) framework for zero-shot anomaly detection, classification, and explanation in laser powder bed fusion (L-PBF) manufacturing. The framework retrieves relevant images and text from a corpus of scientific documents, feeds the retrieved information to a multimodal large language model, and aggregates the per-anomaly binary predictions through an LLM into one-hot encoded classification outputs. The method is evaluated on four sub-datasets from the ORNL Peregrine dataset (54 visible-light images) using Qwen2-VL-2B and GPT-4o-mini as the detection MLLMs. The paper reports that GPT-4o-mini outperforms Qwen2-VL-2B and a proportional random baseline, and that adding retrieval improves average accuracy by about 12% (Table 3). The central claim is that literature-based retrieval alone enables zero-shot anomaly classification without requiring training data.

Significance. If the central claim were established, the work would provide a practically attractive approach: a continuously updatable, literature-only RAG pipeline that can classify manufacturing anomalies without supervised training data. The paper describes a concrete end-to-end architecture (ColPali for image retrieval, embedding-based text retrieval, MLLM prompting, and an LLM-based output module), compares two MLLMs under fixed prompts, and uses a real industrial dataset with per-anomaly and per-dataset accuracy tables. The explicit ablation with and without retrieval is good practice, and the paper openly acknowledges that some anomalies are poorly detected and that RAG can introduce hallucination. However, the significance is currently undermined by a load-bearing evaluation flaw: the retrieval corpus overlaps with the test data source, so the zero-shot and accuracy-gain claims are not established.

major comments (3)

[Section 3, Table 2] The RAG corpus in Table 2 explicitly includes Refs [6] and [41], while Section 3 states that the test images come from the ORNL Peregrine dataset [41] and are used 'exclusively for testing.' Because [41] is the source of both the test images and the ground-truth labels, the retrieval phase can supply the MLLM with the test set's own anomaly names, descriptions, and potentially same-distribution images. This violates the disjointness required for the 'literature-only, zero-shot' claim and means the with/without retrieval comparison in Table 3 does not measure the benefit of external literature retrieval. The authors must either remove Refs [6,41] from the retrieval corpus and rerun the evaluation, or demonstrate that no retrieved item originates from the test set.
[Section 4, Figure 4] The paper never reports the actual top-k retrieved items for any query. Without this information, the reader cannot rule out direct leakage of test images into the prompt. The authors should provide retrieval traces (e.g., retrieved image thumbnails and text snippets) for a few representative cases, and verify that the retrieved images are not identical or near-duplicates of the test images. This is essential evidence for the 'literature-based' claim, not an optional detail.
[Abstract, Table 3] The central claim of a 12% average accuracy gain from retrieval is not robustly supported by the per-class results: for AddUp FormUp 350 the gain is 1 percentage point (0.620 vs. 0.610), and Table A1 shows GPT-4o-mini scoring 0.19 on Recoater Hopping and 0.23 on Soot, far below the majority-class baseline values of 0.96 and 0.85, respectively. The paper should report confidence intervals or significance tests for the accuracy differences, and the abstract's blanket statement about a 12% improvement should be tempered to reflect the per-dataset variability.

minor comments (4)

[Section 4] In the paragraph discussing the RAG ablation, the sentence 'The results in Table 2 show that incorporating RAG systems enhances prediction accuracy' refers to Table 3, not Table 2, which lists documents in the retrieval corpus.
[Section 4, Tables A1–A4] The 'proportional random baseline' appears to be computed as the proportion of the majority class (i.e., always predicting the most frequent outcome), not as the expected accuracy of a random classifier that predicts classes with their prior probabilities. Please define the baseline precisely, as the term 'proportional random' usually denotes a different calculation.
[Figure 8 caption] There is a typo in the caption: 'AddUp FromUp 350' should be 'AddUp FormUp 350'. The same typo appears in Table 3.
[Section 3] The total test set is small (26 + 14 + 9 + 5 = 54 images), and no confidence intervals or statistical tests are reported for any of the accuracy comparisons. This should be acknowledged as a limitation, particularly for per-anomaly rows with very few positive or negative cases.

Circularity Check

2 steps flagged · score 6.0 of 10

RAG index includes the ORNL test dataset (Table 2, refs [6,41]); the zero-shot 'literature-only' claim and Table 3's +12% RAG gain are not established until that overlap is removed.

self definitional [Section 3 (RAG document dataset, Table 2) and Section 4 (Table 3)]
""Since the dataset includes ground truth files and our proposed model does not require training data, we used these images exclusively for testing. ... The dataset of document information for RAG (Table 2) consists of scientific papers that investigate various defect types and anomalies within the L-PBF process [6, 42-52]. ... Scime, L., et al., Additive Manufacturing, 2020. - Oak Ridge National Laboratory (ORNL), 2023. Layer-wise anomaly detection in PBF [6, 41]""

Table 2 explicitly includes [41] (the ORNL dataset) and [6] (the paper that defined the ORNL anomaly taxonomy) in the RAG corpus, while Section 3 states that the ORNL images and ground-truth labels are used 'exclusively for testing.' The retrieval corpus is therefore not disjoint from the evaluation set: a query for each test anomaly can retrieve the test set's own anomaly names, descriptions, and same-source images. The central claim of zero-shot, literature-only detection, and Table 3's with-retrieval accuracy gain, are not established because the 'literature' input can contain the ground-truth output by construction. The paper does not report retrieved top-k items or exclude [41]/[6] from retrieval, so this overlap is not ruled out.
other [Section 2.2, Figure 5 (AnomalyDetectionPrompt)]
""Analyze the test image carefully and determine if {anomaly_name} is possible. Use the information provided in the reference image and additional scientific information to support your assessment. ... The reference image shows an example of {anomaly_name}: {reference_image}+{reference_image_description}. Use it for comparison.""

The detection prompt injects a retrieved reference image and retrieved text about the anomaly into the MLLM before asking whether the anomaly exists in the test image. Because the retrieval corpus includes the ORNL dataset that supplied the test images and their ground-truth labels, the 'additional scientific information' and 'reference image' can be exactly the test set's own anomaly annotations. Consequently, the with-retrieval condition can copy the answer from the retrieval index rather than generalize from independent literature; the reported 12% average improvement (Table 3) may be an artifact of retrieving from the evaluation distribution.

full rationale

The paper contains no fitted equations or formal derivation, so equation-level circularity is absent. However, the evaluation's central claim is circular as designed. Table 2 places the ORNL test dataset [41] and the paper that defined its anomaly taxonomy [6] inside the RAG corpus, while Section 3 uses those same ORNL images as the test set; the detection prompt then feeds the retrieved reference image and text to the MLLM. This means 'retrieved literature' can contain the test set's own ground-truth labels, and the with-RAG vs without-RAG comparison in Table 3 cannot distinguish external knowledge augmentation from retrieval of the evaluation distribution. The paper does not report the retrieved top-k items or verify disjointness. Apart from this evaluation-set leakage, no load-bearing self-citation chain or ansatz-smuggling was found. Score 6: the central empirical prediction partially reduces by construction, but the framework itself (ColPali retrieval, Qwen2-VL/GPT-4o-mini prompts) is independently implemented.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central evaluation depends on retrieval quality from a small hand-picked corpus and on the correctness of annotations. No new physical or model entities are introduced. The most serious ledger entry is the unstated overlap between the RAG corpus and the evaluation set.

free parameters (3)

RAG top-k retrieval count for images and text = not reported
Section 2.1 says images are saved at top-k and text retrieval returns relevant segments, but k is never specified. Classification accuracy likely depends on this hyperparameter.
Number and composition of RAG source documents = about 12 documents from Table 2
The corpus was assembled by 'manual review assisted by an LLM' in Section 3. Inclusion criteria are not systematic, and the corpus determines what information the MLLM receives.
Number of repeated detections per anomaly = 3
Section 2.2 runs detection three times per anomaly and averages the results. This choice affects variance, but no sensitivity analysis is given.

assumptions (4)

domain assumption An anomaly class can be recognized from a single retrieved reference image plus textual description, without any training examples.
The generation prompt in Figure 5 treats one reference image as sufficient for comparison. This is a strong assumption for defects that vary in appearance across machines, materials, and imaging conditions.
domain assumption ORNL ground-truth anomaly annotations are correct and complete.
Section 3 converts the annotation file into ground truth for accuracy. The paper does not audit label quality or annotation disagreements.
domain assumption Visual characteristics of anomalies in the retrieved literature transfer across different printers and materials.
Section 3 evaluates on four printer/material datasets with the same prompts, implicitly assuming the retrieved reference images are representative across machines.
ad hoc to paper The RAG corpus and the evaluation dataset are disjoint.
Table 2 lists ORNL Peregrine [41] and Scime [6] as RAG documents, while Section 3 uses ORNL Peregrine images for testing. The paper never states that retrieved reference images and test images are disjoint.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models." pith.science (2026). https://pith.science/paper/SABEMODD

@misc{pith2026250513828,
  author       = {Pith},
  title        = {Pith review of: Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SABEMODD}},
  note         = {Machine review of arXiv:2505.13828}
}

read the original abstract

Additive manufacturing enables the fabrication of complex designs while minimizing waste, but faces challenges related to defects and process anomalies. This study presents a novel multimodal Retrieval-Augmented Generation-based framework that automates anomaly detection across various Additive Manufacturing processes leveraging retrieved information from literature, including images and descriptive text, rather than training datasets. This framework integrates text and image retrieval from scientific literature and multimodal generation models to perform zero-shot anomaly identification, classification, and explanation generation in a Laser Powder Bed Fusion setting. The proposed framework is evaluated on four L-PBF manufacturing datasets from Oak Ridge National Laboratory, featuring various printer makes, models, and materials. This evaluation demonstrates the framework's adaptability and generalizability across diverse images without requiring additional training. Comparative analysis using Qwen2-VL-2B and GPT-4o-mini as MLLM within the proposed framework highlights that GPT-4o-mini outperforms Qwen2-VL-2B and proportional random baseline in manufacturing anomalies classification. Additionally, the evaluation of the RAG system confirms that incorporating retrieval mechanisms improves average accuracy by 12% by reducing the risk of hallucination and providing additional information. The proposed framework can be continuously updated by integrating emerging research, allowing seamless adaptation to the evolving landscape of AM technologies. This scalable, automated, and zero-shot-capable framework streamlines AM anomaly analysis, enhancing efficiency and accuracy.

Figures

Figures reproduced from arXiv: 2505.13828 by the authors.

**Figure 1.** (a) Various AM processes, each with unique characteristics. (b) The overwhelming influx of data from AM research. (c) Diverse defects across different AM processes. (d) Challenges of manual review in handling large datasets. (e) The effectiveness of our proposed RAG-based method in addressing these challenges. Additive manufacturing (AM) has transformed the manufacturing industry by allowing the production of intric… view at source ↗

**Figure 3.** Comprehensive workflow of the proposed framework – The framework consists of three main phases: (a) Retrieval Phase, which retrieves relevant images, image descriptions, and textual information related to anomalies detection, root causes, and prevention strategies; (b) Generation Phase, which utilizes the retrieved information to provide the MLLM with sufficient context to detect anomalies, and (c) Output Module, wh… view at source ↗

**Figure 5.** Generation prompts for anomaly detection and explanation 2.2 Generation Phase In the generation phase (Figure 3b), the multimodal data obtained in the earlier retrieval phase is synthesized and analyzed in detail through both visual recognition and textual analysis. This phase addresses two key objectives: anomaly detection and classification with explanation. Anomaly Detection: Using retrieved anomaly images and te… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Retrieval query for gathering text and image information related to the target anomaly [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 9.** Figure 9: Predicted anomalies and explanation given test image from EOS M290 – 17-4 PH stainless Steel Sometimes, adding an RAG system can introduce more problems than it solves regarding hallucination [61]. The RAG system design, which bases outputs on retrieved information fro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 1 canonical work pages

[3]

EOS M290

Highlighted the adaptability of the framework, demonstrating its ability to support evolving AM research and emerging processes by applying it to different datasets. 4) Conducted a comparative analysis of the classification capabilities of a small closed-source model and a large open-source model to assess their effectiveness and underlying reasoning in a...

work page 2020
[4]

RESULTS The dataset introduced in Section 3 is used to evaluate the performance of the proposed method in anomaly detection and classification. Several metrics can be employed to evaluate the accuracy of generated response, including Recall@K , Precision@K, F1 score@K [53-55], Bilingual Evaluation Understudy (BLEU) [56], Recall-Oriented Understudy for Gis...

arXiv 2020
[14]

Large Language Models as Few-Shot Defect Detectors for Additive Manufacturing

Fang, Q., et al. Large Language Models as Few-Shot Defect Detectors for Additive Manufacturing. in 2024 China Automation Congress (CAC). 2024. IEEE. 15. Yang, T., et al., Ad-llm: Benchmarking large language models for anomaly detection. arXiv preprint arXiv:2412.11142, 2024. 16. Li, Y., et al., Myriad: Large multimodal model by applying vision experts for...

arXiv 2024
[47]

2023, Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); RTX …

Snow, Z., et al., Machine Learning Enabled Sensor Fusion for In-Situ Defect Detection in Laser Powder Bed Fusion. 2023, Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); RTX …. 48. Cannizzaro, D., et al. Image analytics and machine learning for in-situ defects detection in Additive Manufacturing. in 2021 Design, Automation & Test in Eur...

arXiv 2023
[62]

Lin, and Y

Liu, J., J. Lin, and Y. Liu, How Much Can RAG Help the Reasoning of LLM? arXiv preprint arXiv:2410.02338, 2024. 63. Marvin, G., et al. Prompt engineering in large language models. in International conference on data intelligence and cognitive informatics. 2023. Springer. 64. Espenes, A.L. and A. Trøan, Implementing prompt engineering and retrieval augment...

arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [3]

EOS M290

Highlighted the adaptability of the framework, demonstrating its ability to support evolving AM research and emerging processes by applying it to different datasets. 4) Conducted a comparative analysis of the classification capabilities of a small closed-source model and a large open-source model to assess their effectiveness and underlying reasoning in a...

work page 2020

[2] [4]

RESULTS The dataset introduced in Section 3 is used to evaluate the performance of the proposed method in anomaly detection and classification. Several metrics can be employed to evaluate the accuracy of generated response, including Recall@K , Precision@K, F1 score@K [53-55], Bilingual Evaluation Understudy (BLEU) [56], Recall-Oriented Understudy for Gis...

arXiv 2020

[3] [14]

Large Language Models as Few-Shot Defect Detectors for Additive Manufacturing

Fang, Q., et al. Large Language Models as Few-Shot Defect Detectors for Additive Manufacturing. in 2024 China Automation Congress (CAC). 2024. IEEE. 15. Yang, T., et al., Ad-llm: Benchmarking large language models for anomaly detection. arXiv preprint arXiv:2412.11142, 2024. 16. Li, Y., et al., Myriad: Large multimodal model by applying vision experts for...

arXiv 2024

[4] [47]

2023, Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); RTX …

Snow, Z., et al., Machine Learning Enabled Sensor Fusion for In-Situ Defect Detection in Laser Powder Bed Fusion. 2023, Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); RTX …. 48. Cannizzaro, D., et al. Image analytics and machine learning for in-situ defects detection in Additive Manufacturing. in 2021 Design, Automation & Test in Eur...

arXiv 2023

[5] [62]

Lin, and Y

Liu, J., J. Lin, and Y. Liu, How Much Can RAG Help the Reasoning of LLM? arXiv preprint arXiv:2410.02338, 2024. 63. Marvin, G., et al. Prompt engineering in large language models. in International conference on data intelligence and cognitive informatics. 2023. Springer. 64. Espenes, A.L. and A. Trøan, Implementing prompt engineering and retrieval augment...

arXiv 2024