REVIEW 3 major objections 4 minor 1 cited by
Application Of Large Language Models For The Extraction Of Information From Particle Accelerator Technical Documentation
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A locally run language-model chatbot can answer technical questions from decades-old accelerator documentation, and 800-character chunks with Top-5 retrieval produce the most accurate answers.
desk verdict Useful corpus-specific RAG application, but the headline chunk-size recommendation is confounded by inconsistent context windows and a same-family LLM judge. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is retrieval-augmented generation: documents are parsed, split into chunks, embedded into a vector database, and at query time the top-k most similar chunks are appended to the prompt of a locally hosted, instruction-tuned LLM. The same LLM then generates the answer and, in a separate evaluation prompt, judges whether the generated answer matches a gold answer. Three knobs drive the results: chunk size (800 versus 1600/2000 characters or paragraphs), retrieval depth k (3 or 5), and the effective context window of the LLM runtime, whose truncation at 2048 tokens produces hallucinations.
What would settle it
Re-run the 1600-character, Top-5 configuration with the context window raised from 2048 to 6000 tokens; if the hallucinations do not disappear, the paper's truncation explanation is wrong. Separately, have two independent accelerator engineers grade the 100 generated answers by hand and compare their labels with the model-as-judge labels; low agreement would invalidate the reported accuracy and confidence.
Extended reading notes
Core claim
The paper claims that retrieval-augmented generation is not just feasible but effective for question answering over legacy accelerator documentation, and it identifies the settings that make it work. On a corpus of 58 technical PDFs in English and German, the best configuration is 800-character chunks with Top-5 retrieval, maximising both answer accuracy and the model's self-reported confidence. Translating German chunks into English before embedding substantially improves retrieval for German queries and slightly helps English queries; larger chunks and paragraph units do not help. The only configuration that hallucinated (1600-character chunks, Top-5) failed because the runtime's default 2
Load-bearing premise
The reported accuracy numbers rest entirely on the 100 expert-written question-answer pairs being correct, complete, and representative—and on the same language model that writes the answers being a trustworthy judge of whether those answers are right.
Editorial extensions
If this is right
- A local RAG chatbot can serve as a practical knowledge-preservation tool for legacy accelerator facilities, letting new specialists query decades of documentation without exposing internal files.
- For similar technical corpora, 800-character chunks with Top-5 retrieval is a defensible default; matching retrieval recall with larger chunks did not translate into better answers.
- German-language documentation can be folded into a mostly English system by translating chunks before embedding, which raises retrieval quality for both German and English queries.
- Engineers building similar systems should verify that the context window comfortably exceeds the largest retrieved input, since truncation (not retrieval failure) was the observed cause of hallucination.
- Tables, figures and schematics remain outside the current capability; the documented gains apply to textual content only.
Reading between the lines
- The narrow 0.90-0.93 band of model-judged confidence suggests self-reported confidence is not a fine-grained signal; real deployments should track human-verified correctness on a sample before trusting the confidence scores.
- If the chunk-size finding generalizes, legacy engineering documentation outside accelerators (power plants, refineries, spacecraft) may show the same pattern: small chunks plus limited retrieval depth beats semantically whole paragraphs.
- A direct extension would be to build a second benchmark from tables and figures and test the proposed automatic captioning; captions may preserve existence of a quantity but not exact values, so numeric QA over plots is the harder test.
- The fact that adding surrounding paragraphs slightly degraded accuracy hints that contextual redundancy can mislead generation; prompt designers should test minimal context rather than assume richer context helps.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a retrieval-augmented generation (RAG) system for question answering over particle accelerator technical documentation at PSI. The authors compare several chunking strategies (character windows of 800/1600/2000 characters, paragraph-based splits), retrieval top-k values, and prompt variants (with/without translation and chunk scores), using a 100-question bilingual benchmark built by two domain experts. They report retrieval metrics (recall@k, MRR@k) and generation accuracy/confidence judged by the same Gemma model family, concluding that 800-character chunks with Top-5 retrieval give the best answer accuracy and confidence, and that chunk size plays a critical role.
Significance. If the central claim is supported, the paper offers useful practical guidance for deploying RAG on legacy technical documentation in specialized scientific facilities, a genuine and timely problem. The authors provide a real corpus, a realistic task, and an end-to-end local pipeline. However, the evidence base is thin: a single 100-question test set, no held-out split, no uncertainty quantification, and a model-based judge from the same family as the generator. The headline conclusion about chunk size is further confounded by an inconsistent context-window setting between experimental conditions. The paper's contribution is therefore preliminary and would need a controlled evaluation to substantiate the central recommendation.
major comments (3)
- [Generation performance / Paragraph-Level Chunking and Context Window] The central claim that 800-char chunks with Top-5 deliver the highest accuracy is confounded by an uncontrolled change in the LLM context window. The text states that only the 1600-char Top-5 setup showed hallucinations and attributes this to Ollama's default 2048-token context, while the paragraph experiments explicitly increase the context window to 6000 tokens. Thus the 1600-char condition was run under truncation-induced hallucination, while the 800-char condition was not (or at least not equivalently). The observed accuracy gap between 800- and 1600-char chunks may therefore be an artifact of token truncation rather than a genuine chunk-size effect. The authors should re-run all generation conditions under a fixed, sufficiently large context window and report actual input token lengths for each configuration.
- [Methods, Evaluation; Results, Generation performance] Answer accuracy is judged by the same Gemma model family that generates the answers. Without validation of the judge against human labels, the reported accuracy and confidence scores may reflect the model's own stylistic preferences rather than factual correctness. The paper provides no human agreement study, no independent judge, and no analysis of judge failures. Since the entire generation evaluation rests on this metric, the authors should validate the judge on a human-annotated subset (e.g., 50–100 answers) and report agreement, or use an independent judge.
- [Results, Generation performance; Recommendation] The 100-question benchmark is used both to select the best configuration and to evaluate it; no held-out split or cross-validation is reported. Figure 4 shows only point estimates with no confidence intervals or statistical tests, so the reported advantage of 800-char/Top-5 over other settings could be noise. Post-hoc selection on the test set inflates the apparent performance of the chosen configuration. The authors should use a train/dev/test split (or bootstrap resampling) and report uncertainty, e.g., confidence intervals for accuracy, MRR, and recall.
minor comments (4)
- [Abstract / Introduction] There are several typographical spacing errors, e.g., 'explorestheapplication' and 'Top-3forrecall'. Please proofread the text.
- [Results, Chunking strategies] The generation experiments include only 800- and 1600-char chunks, while retrieval tests also included 2000-char windows. Clarify why 2000-char chunks were not tested in generation.
- [Results, Generation performance] The per-language breakdown of generation accuracy is not reported, despite the benchmark having 70 English and 30 German questions. Given the observed gap in retrieval performance between English and German, it would be informative to report generation accuracy separately for each language.
- [References] The citation for 'gemma2:27b-instruct-q4_K_M' is [14], but reference [14] is Gemma (2024), while reference [15] is Gemma 2. Please correct the citation.
Circularity Check
No significant circularity; central claims are empirical, with only a minor non-load-bearing self-citation.
full rationale
The paper's core derivation chain—document chunking, embedding, retrieval, and generation—is externally grounded: 100 QA pairs were created by two domain experts and linked to gold reference files, and retrieval is scored with standard recall@k and MRR@k. The central recommendation (800-char chunks with Top-5) comes from comparing 800- and 1600-char conditions under the same context window, so it is not forced by construction. The only co-author citation ([5] M. Sapinski et al.) is an example input document, not a load-bearing premise, so it does not create circularity. Two passages assert limitations that affect validity but not circularity: (1) the evaluation uses 'the same Gemma model as a judge' for answer accuracy, which is a potential bias in the metric, but because human gold answers are provided, the judge is not defined in terms of the generated answer; and (2) the context window was increased to 6000 tokens for paragraph experiments after 1600-char Top-5 suffered truncation at the default 2048 window, which is an experimental confound for cross-condition comparisons, but not a circular reduction. Neither passage makes a prediction equivalent to its inputs by definition. Therefore no circular step is exhibited.
Assumptions & free parameters
free parameters (3)
- Retrieval top-k =
5
- Chunk size =
800 characters
- Translation of German chunks to English =
on
assumptions (3)
- domain assumption The Gemma model's judgments of answer correctness are accurate and unbiased.
- domain assumption The 58 PDFs and supplemental text cover the relevant knowledge needed to answer the 100 benchmark questions.
- domain assumption MinerU and BGE-M3 preserve the semantic content of technical documents sufficiently for retrieval.
Cite this review
Pith. "Pith review of Application Of Large Language Models For The Extraction Of Information From Particle Accelerator Technical Documentation." pith.science (2026). https://pith.science/paper/45EW7OXL
@misc{pith2026250902227,
author = {Pith},
title = {Pith review of: Application Of Large Language Models For The Extraction Of Information From Particle Accelerator Technical Documentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/45EW7OXL}},
note = {Machine review of arXiv:2509.02227}
}
read the original abstract
The large set of technical documentation of legacy accelerator systems, coupled with the retirement of experienced personnel, underscores the urgent need for efficient methods to preserve and transfer specialized knowledge. This paper explores the application of large language models (LLMs), to automate and enhance the extraction of information from particle accelerator technical documents. By exploiting LLMs, we aim to address the challenges of knowledge retention, enabling the retrieval of domain expertise embedded in legacy documentation. We present initial results of adapting LLMs to this specialized domain. Our evaluation demonstrates the effectiveness of LLMs in extracting, summarizing, and organizing knowledge, significantly reducing the risk of losing valuable insights as personnel retire. Furthermore, we discuss the limitations of current LLMs, such as interpretability and handling of rare domain-specific terms, and propose strategies for improvement. This work highlights the potential of LLMs to play a pivotal role in preserving institutional knowledge and ensuring continuity in highly specialized fields.
Figures
Forward citations
Cited by 1 Pith paper
-
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
A deployed hybrid RAG for APS operations improves vital-nugget recall over BM25 mainly via cross-encoder reranking; graph and corrective loops help only marginally on a 50-question facility benchmark.
Reference graph
Works this paper leans on
- [1]
-
[2]
F. Mayet. GAIA: A General AI Assistant for Intelligent Accelerator Operations. arXiv preprint arXiv:2405.01359, 2024
arXiv 2024
-
[3]
A. Sulc, A. Bien, A. Eichler, D. Ratner, F. Rehm, F. Mayet, G. Hartmann, H. Hoschouer, H. Tuennermann, J. Kaiser, et al. Towards unlocking insights from logbooks us- ing AI. In Proceedings of IPAC 2024 , THPR37, 2024. doi:10.18429/JACoW-IPAC2024-THPR37. Also available as arXiv:2406.12881 [physics.acc-ph]
arXiv 2024
-
[4]
R. Dolling. Diagnostics of the PROSCAN proton-therapy beam lines. InProceedings of DIPAC 2003, 2003
work page 2003
-
[5]
M. Sapinski, R. Dölling, and M. Rohrer. Commission- ing of the Renewed Long Radial Probe in PSI Ring Cy- clotron. In Proceedings of IBIC 2022 , MOP19, 2022. doi:10.18429/JACoW-IBIC2022-MOP19
-
[6]
P. Forck. JUAS Lecture Notes on Beam Instru- mentation and Diagnostics. 2011. Available at https://www.gsi.de/work/gesamtprojektleitung_ fair/commons/beam_instrumentation/research_ and_development_rd/veroeffentlichungen.htm
work page 2011
-
[7]
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang. Retrieval-Augmented Gener- ation for Large Language Models: A Survey.arXiv preprint arXiv:2312.10997, 2023
arXiv 2023
- [8]
Show all 15 references
-
[9]
B. Wang, C. Xu, X. Zhao, L. Ouyang, F. Wu, Z. Zhao, R. Xu, K. Liu, Y. Qu, F. Shang, B. Zhang, L. Wei, Z. Sui, W. Li, B.Shi,Y.Qiao,D.Lin,andC.He. MinerU:AnOpen-Source Solution for Precise Document Content Extraction.arXiv preprint arXiv:2409.18839, 2024
2024 arXiv
-
[10]
BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi- Granularity Text Embeddings Through Self-Knowledge Dis- tillation
J.Chen,S.Xiao,P.Zhang,K.Luo,D.Lian,andZ.Liu. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi- Granularity Text Embeddings Through Self-Knowledge Dis- tillation. arXiv preprint arXiv:2402.03216, 2024
2024 arXiv
-
[11]
Evaluation Measures in Information Re- trieval
Pinecone. Evaluation Measures in Information Re- trieval. 2023. https://www.pinecone.io/learn/ offline-evaluation/
2023
-
[12]
Questions about context size (GitHub issue #2204)
Ollama. Questions about context size (GitHub issue #2204). 2024. https://github.com/ollama/ollama/ issues/2204
2024
-
[13]
Gemma Team. Gemma 3. 2025. Kaggle.https://goo. gle/Gemma3Report
2025
-
[14]
Gemma Team. Gemma. 2024. Kaggle. doi:10.34740/KAGGLE/M/3301. https://www.kaggle. com/m/3301
2024 doi
-
[15]
I don’t know
M. Rivière et al. Gemma 2: Improving Open Language Models at a Practical Size.arXiv preprint arXiv:2408.00118, 2024. APPENDIX Generation Prompt Listing 1: Prompt without chunk scores You are a beam-accelerator Q&A assistant. Answer the user’s question using ONLY the provided c...
2024 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.