{"id":"fa734577-2688-4a82-a8af-7f9c479d6a83","arxiv_id":"2509.02227","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A RAG pipeline for accelerator documentation works best with 800-character chunks and top-5 retrieval, and translating German documents helps retrieval.","lead":"The paper tests a question-answer chatbot that reads old accelerator documents with a large language model and a retrieval system. It finds that cutting documents into 800-character chunks and retrieving five pieces gives the most accurate answers on a small test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Chunk-size recommendation may be an artifact of an inconsistent context-window setting: 1600-char Top-5 tests ran at default 2048 tokens and hallucinated; paragraph tests used 6000 tokens.","rationale":"The reader's weakest assumption concerned the ground-truth QA pairs and the LLM judge. While these are valid concerns, the most load-bearing issue is the context-window confound in the generation experiments. The paper's own hallucination analysis reveals that the 1600-char Top-5 condition was run at the default 2048-token context and suffered from truncation, whereas paragraph experiments were run at 6000 tokens. The comparison between 800- and 1600-char chunks is therefore not a clean test of chunk size. This directly undermines the paper's central recommendation. The reader did not identify this confound, so agreement is 'disagree'. The verdict remains CONDITIONAL (UNCHANGED) because the flaw is correctable: re-running with a fixed context window would settle it. If the confound is not addressed, the recommendation should be rejected; but the paper is an initial study, and CONDITIONAL is the appropriate judgment pending that test.","tokens_in":4426,"tokens_out":3865,"duration_ms":36505,"concrete_test":"Re-run the generation evaluation for all chunk/k/prompt combinations (800-char, 1600-char, paragraph, paragraph+context; k=3,5; k-N, k-T, k-S) using a single context window of at least 6000 tokens for every condition. Compare answer accuracy. If 1600-char Top-5 no longer hallucinates and matches or exceeds 800-char Top-5 accuracy, the recommendation is invalidated. If 800-char still wins, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation that 800-char chunks with Top-5 retrieval deliver the highest answer accuracy is confounded by an inconsistent context-window setting across generation experiments. The generation tests for 800- and 1600-char chunks used Ollama's default 2048-token context (as the paper itself notes when explaining hallucinations in the 1600-char Top-5 setup), while the later paragraph-based experiments explicitly increased the context to 6000 tokens. Thus the 1600-char Top-5 condition was disadvantaged by truncation-induced hallucination, and the observed superiority of 800-char chunks may simply reflect this truncation rather than a genuine chunk-size effect. The retrieval results (recall@k, MRR@k) are unaffected, but the headline generation claim depends critically on a fair comparison of chunk sizes. Without re-running all chunk conditions under a fixed, adequate context window, the claim that chunk size is 'critical' is not supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a retrieval-augmented generation (RAG) system for question answering over particle accelerator technical documentation at PSI. The authors compare several chunking strategies (character windows of 800/1600/2000 characters, paragraph-based splits), retrieval top-k values, and prompt variants (with/without translation and chunk scores), using a 100-question bilingual benchmark built by two domain experts. They report retrieval metrics (recall@k, MRR@k) and generation accuracy/confidence judged by the same Gemma model family, concluding that 800-character chunks with Top-5 retrieval give the best answer accuracy and confidence, and that chunk size plays a critical role.","tokens_in":4626,"tokens_out":3560,"duration_ms":42306,"significance":"If the central claim is supported, the paper offers useful practical guidance for deploying RAG on legacy technical documentation in specialized scientific facilities, a genuine and timely problem. The authors provide a real corpus, a realistic task, and an end-to-end local pipeline. However, the evidence base is thin: a single 100-question test set, no held-out split, no uncertainty quantification, and a model-based judge from the same family as the generator. The headline conclusion about chunk size is further confounded by an inconsistent context-window setting between experimental conditions. The paper's contribution is therefore preliminary and would need a controlled evaluation to substantiate the central recommendation.","major_comments":[{"comment":"The central claim that 800-char chunks with Top-5 deliver the highest accuracy is confounded by an uncontrolled change in the LLM context window. The text states that only the 1600-char Top-5 setup showed hallucinations and attributes this to Ollama's default 2048-token context, while the paragraph experiments explicitly increase the context window to 6000 tokens. Thus the 1600-char condition was run under truncation-induced hallucination, while the 800-char condition was not (or at least not equivalently). The observed accuracy gap between 800- and 1600-char chunks may therefore be an artifact of token truncation rather than a genuine chunk-size effect. The authors should re-run all generation conditions under a fixed, sufficiently large context window and report actual input token lengths for each configuration.","section":"Generation performance / Paragraph-Level Chunking and Context Window"},{"comment":"Answer accuracy is judged by the same Gemma model family that generates the answers. Without validation of the judge against human labels, the reported accuracy and confidence scores may reflect the model's own stylistic preferences rather than factual correctness. The paper provides no human agreement study, no independent judge, and no analysis of judge failures. Since the entire generation evaluation rests on this metric, the authors should validate the judge on a human-annotated subset (e.g., 50–100 answers) and report agreement, or use an independent judge.","section":"Methods, Evaluation; Results, Generation performance"},{"comment":"The 100-question benchmark is used both to select the best configuration and to evaluate it; no held-out split or cross-validation is reported. Figure 4 shows only point estimates with no confidence intervals or statistical tests, so the reported advantage of 800-char/Top-5 over other settings could be noise. Post-hoc selection on the test set inflates the apparent performance of the chosen configuration. The authors should use a train/dev/test split (or bootstrap resampling) and report uncertainty, e.g., confidence intervals for accuracy, MRR, and recall.","section":"Results, Generation performance; Recommendation"}],"minor_comments":[{"comment":"There are several typographical spacing errors, e.g., 'explorestheapplication' and 'Top-3forrecall'. Please proofread the text.","section":"Abstract / Introduction"},{"comment":"The generation experiments include only 800- and 1600-char chunks, while retrieval tests also included 2000-char windows. Clarify why 2000-char chunks were not tested in generation.","section":"Results, Chunking strategies"},{"comment":"The per-language breakdown of generation accuracy is not reported, despite the benchmark having 70 English and 30 German questions. Given the observed gap in retrieval performance between English and German, it would be informative to report generation accuracy separately for each language.","section":"Results, Generation performance"},{"comment":"The citation for 'gemma2:27b-instruct-q4_K_M' is [14], but reference [14] is Gemma (2024), while reference [15] is Gemma 2. Please correct the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of an engineering experience report than a methodological contribution. The core recommendation is plausible but rests on a confounded comparison and a circular judge. If the authors can re-run the chunk-size comparison with a fixed context window, validate the judge on human labels, and add basic uncertainty quantification, the revised paper would be suitable for publication as a short/practice paper. The lack of novelty in the RAG methodology itself is not the main issue; the evaluation rigor is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straightforward, honest RAG application paper for accelerator documentation. The retrieval results are plausible and the authors are transparent about limitations. But the key generation finding—800-char chunks beat 1600-char—rests on an unfair comparison. The stress-test note is right: 1600-char Top-5 ran at Ollama's default 2048-token context and hallucinated; the paragraph experiments used 6000 tokens. They never re-ran 800-char under the wider window. So the observed generation gap could be a truncation artifact, not a chunk-size effect. The retrieval metrics (recall@k, MRR@k) aren't affected, and they do support smaller chunks, but the abstract and recommendation overstate the case.\n\nWhat's actually new: they validate a full RAG pipeline on a real, mixed-language (English/German) corpus of technical documents, and show that translating German chunks to English improves retrieval. That's a useful data point for anyone building similar systems. The paper is refreshingly explicit about its own limitations—tables and figures not handled, missing data sources, and the context-window issue—so it earns some trust.\n\nSoft spots beyond the context-window confound: the evaluation uses Gemma to judge answers generated by Gemma. There's no human validation of the judge, so the reported accuracy and 'confidence' reflect one model family's preferences. The benchmark is 100 expert-written QA pairs, with no baseline or error bars—post-hoc selection of the best configuration on the test set. And the abstract's claim that this 'significantly reduces the risk of losing valuable insights' is not measured by any of their metrics.\n\nWho should read this: people working on corpus-specific RAG for institutional knowledge, especially in physics or engineering. It's an application note, not a methodological contribution. With the context-window fix, a human-evaluated judge, and more careful claims, it could be a solid workshop or short-conference paper. As submitted, the central recommendation is too fragile to accept as-is.\n\nRecommendation: send it to peer review, but with the expectation of major revision. The reviewer should demand a consistent context window across all chunk conditions, human evaluation of judge reliability, and a toned-down abstract.","headline":"Useful corpus-specific RAG application, but the headline chunk-size recommendation is confounded by inconsistent context windows and a same-family LLM judge.","tokens_in":5107,"tokens_out":4636,"would_cite":false,"duration_ms":46776,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A locally run language-model chatbot can answer technical questions from decades-old accelerator documentation, and 800-character chunks with Top-5 retrieval produce the most accurate answers.","keywords":["retrieval-augmented generation","large language models","particle accelerator documentation","knowledge preservation","chunking strategy","multilingual retrieval","model-as-judge evaluation","RAG hallucination"],"falsifier":"Re-run the 1600-character, Top-5 configuration with the context window raised from 2048 to 6000 tokens; if the hallucinations do not disappear, the paper's truncation explanation is wrong. Separately, have two independent accelerator engineers grade the 100 generated answers by hand and compare their labels with the model-as-judge labels; low agreement would invalidate the reported accuracy and confidence.","tokens_in":4308,"feed_emoji":"⚛️","tokens_out":8310,"duration_ms":83305,"temperature":0.7,"pith_summary":"The paper tries to establish that a locally run retrieval-augmented generation pipeline can extract reliable, source-grounded answers from the messy legacy documentation of a 50-year-old proton accelerator, addressing the loss of expertise as experienced staff retire. Using 58 PDFs in English and German, two domain experts built 100 question-answer pairs, and the authors tested chunk sizes, retrieval depths, and prompt variants. The central result is a practical configuration: split documents into 800-character chunks, retrieve the top five chunks, translate German chunks to English, and feed them to an instruction-tuned LLM; this yields the highest answer accuracy and confidence among the options tested. A sympathetic reader would care because the recipe is simple, runs entirely on a local machine, and directly targets a knowledge-retention problem common to large experimental facilities.","feed_headline":"Smaller text chunks give better AI answers from old accelerator docs","feed_subtitle":"A local retrieval-augmented chatbot tops out at 800-character chunks with five retrieved sources, a recipe others can copy.","key_machinery":"The load-bearing mechanism is retrieval-augmented generation: documents are parsed, split into chunks, embedded into a vector database, and at query time the top-k most similar chunks are appended to the prompt of a locally hosted, instruction-tuned LLM. The same LLM then generates the answer and, in a separate evaluation prompt, judges whether the generated answer matches a gold answer. Three knobs drive the results: chunk size (800 versus 1600/2000 characters or paragraphs), retrieval depth k (3 or 5), and the effective context window of the LLM runtime, whose truncation at 2048 tokens produces hallucinations.","core_discovery":"The paper claims that retrieval-augmented generation is not just feasible but effective for question answering over legacy accelerator documentation, and it identifies the settings that make it work. On a corpus of 58 technical PDFs in English and German, the best configuration is 800-character chunks with Top-5 retrieval, maximising both answer accuracy and the model's self-reported confidence. Translating German chunks into English before embedding substantially improves retrieval for German queries and slightly helps English queries; larger chunks and paragraph units do not help. The only configuration that hallucinated (1600-character chunks, Top-5) failed because the runtime's default 2","pith_inferences":["The narrow 0.90-0.93 band of model-judged confidence suggests self-reported confidence is not a fine-grained signal; real deployments should track human-verified correctness on a sample before trusting the confidence scores.","If the chunk-size finding generalizes, legacy engineering documentation outside accelerators (power plants, refineries, spacecraft) may show the same pattern: small chunks plus limited retrieval depth beats semantically whole paragraphs.","A direct extension would be to build a second benchmark from tables and figures and test the proposed automatic captioning; captions may preserve existence of a quantity but not exact values, so numeric QA over plots is the harder test.","The fact that adding surrounding paragraphs slightly degraded accuracy hints that contextual redundancy can mislead generation; prompt designers should test minimal context rather than assume richer context helps."],"forward_implications":["A local RAG chatbot can serve as a practical knowledge-preservation tool for legacy accelerator facilities, letting new specialists query decades of documentation without exposing internal files.","For similar technical corpora, 800-character chunks with Top-5 retrieval is a defensible default; matching retrieval recall with larger chunks did not translate into better answers.","German-language documentation can be folded into a mostly English system by translating chunks before embedding, which raises retrieval quality for both German and English queries.","Engineers building similar systems should verify that the context window comfortably exceeds the largest retrieved input, since truncation (not retrieval failure) was the observed cause of hallucination.","Tables, figures and schematics remain outside the current capability; the documented gains apply to textual content only."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation framework and the grounding rationale that motivates combining an LLM with an external knowledge base.","marker":"[7]"},{"why":"Provides the local model runtime; its default context-window setting becomes the explanatory variable in the hallucination analysis.","marker":"[8]"},{"why":"Parses the PDF corpus into text, equations, and tables, defining what the chunker can work with.","marker":"[9]"},{"why":"Embeds chunks and queries in a multilingual space, making the German-English translation results possible.","marker":"[10]"},{"why":"Defines the recall@k and MRR@k measures used for every retrieval comparison.","marker":"[11]"},{"why":"Documents the 2048-token default context window that the paper blames for truncation-induced hallucinations.","marker":"[12]"},{"why":"The instruction-tuned model that generates answers and also evaluates them, so its behaviour sets both the accuracy and the confidence numbers.","marker":"[13]"},{"why":"The smaller instruction-tuned model used to translate German chunks to English in the translation prompt variants.","marker":"[14]"}],"fun_headline_variants":["800-character chunks best for AI answers from accelerator docs","RAG on legacy accelerator docs: 800-char chunks, top-5 retrieval wins","Optimal settings for LLM extraction from accelerator manuals revealed","German-to-English translation boosts RAG on accelerator docs","LLM recipe: 800-char chunks, top-5 sources for accelerator Q&A"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported accuracy numbers rest entirely on the 100 expert-written question-answer pairs being correct, complete, and representative—and on the same language model that writes the answers being a trustworthy judge of whether those answers are right.","fun_headline_variants_meta":{"raw":{"variants":["800-character chunks best for AI answers from accelerator docs","RAG on legacy accelerator docs: 800-char chunks, top-5 retrieval wins","Optimal settings for LLM extraction from accelerator manuals revealed","German-to-English translation boosts RAG on accelerator docs","LLM recipe: 800-char chunks, top-5 sources for accelerator Q&A"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":2947,"prompt_tokens":673,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":417,"completion_tokens_details":{"reasoning_tokens":2181}},"tokens_in":417,"tokens_out":2274,"duration_ms":17226,"temperature":1.0,"reasoning_tokens":2181,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:43:35.134977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 1600-character, Top-5 configuration with the context window raised from 2048 to 6000 tokens; if the hallucinations do not disappear, the paper's truncation explanation is wrong. Separately, have two independent accelerator engineers grade the 100 generated answers by hand and compare their labels with the model-as-judge labels; low agreement would invalidate the reported accuracy and confidence.","supporting_citations":[{"cited_title":"Ollamaprojecthomepage","cited_arxiv_id":null,"evidence_quote":"Provides the local model runtime; its default context-window setting becomes the explanatory variable in the hallucination analysis."},{"cited_title":"Evaluation Measures in Information Re- trieval","cited_arxiv_id":null,"evidence_quote":"Defines the recall@k and MRR@k measures used for every retrieval comparison."},{"cited_title":"Questions about context size (GitHub issue #2204)","cited_arxiv_id":null,"evidence_quote":"Documents the 2048-token default context window that the paper blames for truncation-induced hallucinations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The instruction-tuned model that generates answers and also evaluates them, so its behaviour sets both the accuracy and the confidence numbers."}],"review_version":1}