{"id":"c0a1764c-5097-4d2a-a6b8-efa13bb9ae18","arxiv_id":"2509.09688","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A RAG-based AI assistant with a custom web crawler provides natural-language access to RHIC's internal documentation, and initial qualitative tests suggest it outperforms public LLMs on proprietary, unpublished knowledge.","lead":"Researchers built a chatbot that answers questions about RHIC's 25 years of experimental documentation, using retrieval-augmented generation over archived collaboration notes and mailing lists. Its reported advantage over public AI tools is access to proprietary, unpublished knowledge, a model that could generalize to other legacy scientific facilities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of accurate, superior RAG answers is unsupported: Section 4 says the formal benchmark is in progress, yet Section 5 asserts 'substantial inter-rater reliability' and 'significant differences' without any statistics.","rationale":"The reader's verdict is CONDITIONAL, and the reader's rationale already notes that the formal benchmark is in progress while the conclusion asserts reliability and significance without statistics. However, the reader's stated weakest_assumption is corpus completeness/fidelity, whereas I identify the missing evaluation as the more load-bearing concern for the central claim. The corpus issue matters for the system's practical coverage, but the claim that the system provides accurate, superior answers is directly undermined by the absence of the promised benchmark data and by the internal contradiction between Section 4 ('benchmark in progress') and Section 5 ('substantial inter-rater reliability... significant differences'). Therefore the verdict should remain CONDITIONAL, but the condition should explicitly require releasing or supplying a quantitative, expert-scored evaluation with reported inter-rater reliability and significance testing.","tokens_in":5512,"tokens_out":5114,"duration_ms":61201,"concrete_test":"Independently run a controlled evaluation: take the two example questions from Section 4 (space charge effects; TOF resolution in STAR) plus at least 18 additional expert-authored questions, generate answers with the described RAG pipeline (Llama3.3 or Mistral + ChromaDB) and with ChatGPT o3, have three domain experts blinded to system identity score correctness and completeness on a predefined 1–5 rubric, then compute inter-rater reliability (e.g., Cohen's kappa or Krippendorff's alpha) and a paired significance test. If no significant difference is found, or if the authors cannot supply the benchmark data at all, the conclusion's statistical claims fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's load-bearing claim is that the locally deployed RAG system 'effectively preserves and serves' RHIC knowledge and outperforms public LLMs on proprietary-data questions. The only evidence in Section 4 is a qualitative comparison of Llama3.3-70B, Mistral-Large-2411 (RAG-augmented) and ChatGPT o3 on an unspecified number of domain questions. That section explicitly states: 'The development of a formal, multi-dimensional evaluation benchmark to systematically quantify these observations is currently in progress.' No benchmark questions, no scoring rubric, no raw ratings, no counts, no effect sizes are presented. Nevertheless, Section 5 concludes that the evaluation 'established expert-validated reference answers with substantial inter-rater reliability, revealing significant differences between RAG-enhanced and baseline models.' No reliability coefficient, significance test, or effect size appears anywhere in the manuscript. This is an internal inconsistency, not a stylistic weakness: the central empirical assertion is unverified by the paper's own account of its evidence. A corpus coverage audit would also be valuable, but it is secondary; even a perfect corpus does not establish accurate answers without a valid evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an AI-powered assistant for preserving and querying RHIC knowledge. It presents a recursive multi-format web crawler that harvests HTML, PDF/PS, and Office documents from collaboration sites; a Retrieval-Augmented Generation system with ChromaDB and an MCP-based orchestration layer; an empirical comparison of three inference engines (vLLM, LlamaCpp, Ollama) on GPU throughput; and a qualitative evaluation comparing RAG-augmented Llama3.3-70B and Mistral-Large-2411 with ChatGPT o3 on a small set of domain questions. The paper concludes that the assistant 'effectively preserves and serves nuclear physics knowledge' and that RAG-based access to proprietary data gives it a decisive advantage over public LLMs.","tokens_in":5781,"tokens_out":5436,"duration_ms":61557,"significance":"If the central claims were fully supported, this would be a useful experience report for scientific knowledge preservation: the scraper pipeline addresses real format heterogeneity and provenance tracking, and the MCP/RAG deployment is a practical template for future facilities such as the EIC. The paper's strengths are the concrete crawling architecture in Section 2, the attention to provenance metadata, and the practical inference-engine throughput measurements. However, the empirical evaluation is not yet at the level needed to support the conclusions: Section 4 explicitly says the formal benchmark is in progress, while Section 5 makes strong statistical claims with no supporting statistics. The paper would be publishable as an early-deployment/architecture report once the evaluation claims are either supplied or appropriately qualified.","major_comments":[{"comment":"Section 4 states that 'the development of a formal, multi-dimensional evaluation benchmark... is currently in progress' and reports only qualitative impressions for two illustrative questions. Section 5 nevertheless concludes that the evaluation 'established expert-validated reference answers with substantial inter-rater reliability, revealing significant differences between RAG-enhanced and baseline models.' No number of questions, scoring rubric, raw ratings, reliability coefficient, significance test, or effect size is provided. This is an internal inconsistency: the paper's own account of its evidence does not support the conclusion. Since the central claim is that the assistant 'effectively preserves and serves nuclear physics knowledge' and is superior on proprietary-data queries, this issue is load-bearing. The authors should either remove the statistical language and clearly labe","section":"Section 4 vs. Section 5"},{"comment":"The preservation claim depends on the completeness and fidelity of the harvested corpus, but no coverage audit is reported. The crawler filters out external links, blacklists login pages and calendars, logs unsupported extensions, and converts legacy formats via external tools; Section 5 acknowledges that authentication support for protected institutional resources is future work. Thus it is unknown whether the corpus actually contains the tacit knowledge the paper aims to preserve. Please report corpus statistics (number of documents, tokens, source types, per-format extraction success/failure rates) and a coverage audit against a known inventory such as internal wikis, mailing-list archives, and detector documentation. Without this, the 'key advantage' of private/unpublished information is supported only by anecdote.","section":"Section 2 / Fig. 2.1"},{"comment":"The inference-engine comparison is presented as a 'fair comparison,' but the experimental protocol is not specified. The reader is not told which model was served, quantization level, batch size, concurrency, prompt and generation lengths, CPU/RAM configuration, or software versions beyond GPU names, and no repeated trials or error bars are shown. The conclusion that vLLM scales best is therefore not reproducible. Since this section motivates the system's deployment choices, the missing setup details should be added or the figure should be explicitly labeled as illustrative.","section":"Section 3.2 / Fig. 3.1"}],"minor_comments":[{"comment":"The abstract says the system 'effectively preserves and serves nuclear physics knowledge,' but the body reports only initial deployment and an ongoing benchmark; the wording should be qualified to match the evidence.","section":"Abstract"},{"comment":"The paper mentions 'many PB of heterogeneous data' in Section 1 and 'thousands of documents' in Section 3.1; please give actual count/volume after scraping so readers can judge corpus scale.","section":"Section 1 / Section 3.1"},{"comment":"The first paragraph contains grammatical issues ('many generic LLMs... exist and available') and the sentence beginning 'Our approach is tailored...' is difficult to parse; careful language editing is needed.","section":"Section 4"},{"comment":"Figure 3.1 needs full axis labels, a legend, and panel identifiers; also, reference [14] is incomplete (missing journal/arXiv identifier).","section":"Fig. 3.1 / References"},{"comment":"The terms 'baseline' and 'RAG-enhanced' should be defined explicitly: state that the baseline is ChatGPT o3 (or whatever was used) and give exact model versions for Llama3.3 and Mistral.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the mismatch between Section 4 and Section 5: the former admits the benchmark is in progress while the latter asserts statistical reliability and significant differences. If the authors replace those unsupported claims with their actual protocol and results, or explicitly restrict conclusions to qualitative observations, the paper could become acceptable as an experience/architecture report. I would not reject solely because the benchmark is preliminary, but the current over-claiming is load-bearing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2509.09688. The genuinely new thing here is not the components—RAG, ChromaDB, MCP, vLLM are all established and cited—but the specific integration for RHIC: a custom multi-format crawler that handles HTML, PDF/PS, and legacy Office docs, feeding a locally deployed RAG system over internal archives, with MCP orchestration. The observation that internal mailing lists are a differentiator against public chatbots is reasonable and, if true, is the paper's main contribution: a concrete account of why a locally deployed assistant can answer questions commercial LLMs cannot. That is worth taking seriously for data preservation at RHIC and for the EIC.\n\nThe paper does some things well. The inference engine comparison (vLLM vs LlamaCpp vs Ollama across GPU counts) is a useful operational data point. Section 2's crawler description is detailed enough to reproduce the approach. The authors are reporting on a real deployed system, not a toy.\n\nNow the soft spots. The load-bearing problem is the evaluation. Section 4 explicitly says the formal benchmark is 'currently in progress' and presents only qualitative impressions from two sample questions. Yet Section 5 concludes that the evaluation 'established expert-validated reference answers with substantial inter-rater reliability, revealing significant differences' with no reliability coefficient, no significance test, no effect size, not even a question count. That is an internal inconsistency, not a stylistic lapse: the conclusion claims statistics the paper admits it has not collected. The stress-test note is right on this.\n\nThe other soft spot is corpus completeness. The crawler deliberately filters external links, skips login pages, logs unsupported extensions, and relies on external converters. If significant tacit knowledge lives only in authenticated wikis or binary formats, the assistant's answers will be silently gappy. The paper gives no coverage audit, no corpus statistics beyond anecdotes. That is secondary to the evaluation problem but worth flagging.\n\nAlso, no artifacts are shipped: no benchmark questions, no reference answers, no code repo. For a systems paper, that limits external verification.\n\nNone of this is fatal. The architecture is plausible, the central claim is conditional on proper evaluation, and the paper is honest enough to state its own limitation. The problems are addressable with a released benchmark, real inter-rater statistics, and a corpus coverage audit.\n\nWho is this for? Anyone working on preserving scientific institutional knowledge, especially in nuclear/particle physics, and anyone building RAG over proprietary archives. It deserves a serious referee—the systems work is real and the evaluation gap is fixable in revision. If I were the editor, I'd send it out, but I'd explicitly ask the reviewers to verify the match between Section 4's evidence and Section 5's claims.","headline":"A plausible, honestly written systems report for preserving RHIC institutional knowledge, whose evaluation section undercuts its own conclusion—worth engaging, but only with a clear request for real metrics.","tokens_in":6259,"tokens_out":2403,"would_cite":false,"duration_ms":23526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents a deployed assistant that preserves RHIC's tacit knowledge by answering natural-language questions from internal archives that public chatbots cannot access.","keywords":["retrieval-augmented generation","knowledge preservation","RHIC","tacit knowledge","Model Context Protocol","scientific legacy data","web content extraction","question answering"],"falsifier":"Build a test set of insider-level questions whose answers exist only in sources the crawler deliberately excludes, such as authenticated wikis or mailing-list attachments in unsupported formats, then query the deployed assistant. If it consistently fails or gives plausible but wrong answers on those questions, the claim that internal proprietary data access is the decisive advantage would not be supported.","tokens_in":5413,"feed_emoji":"🔬","tokens_out":6982,"duration_ms":72834,"temperature":0.7,"pith_summary":"The paper describes an AI assistant built to keep the Relativistic Heavy Ion Collider's knowledge usable after the collider stops taking data. It argues that a locally deployed retrieval-augmented generation system, fed by a custom crawler over internal documentation and mailing lists, can answer insider-level nuclear physics questions more usefully than public chatbots that only see the open web. The system preserves tacit knowledge—analysis techniques, detector know-how, and troubleshooting—that otherwise lives in documents and conversations inaccessible to commercial LLMs. This matters because the future value of RHIC's roughly exabyte-scale data depends on researchers being able to understand how it was produced.","feed_headline":"RHIC AI assistant answers from archives public chatbots cannot see","feed_subtitle":"It uses internal STAR documents and mailing lists to keep 25 years of physics know-how usable.","key_machinery":"The load-bearing mechanism is retrieval-augmented generation orchestrated by the Model Context Protocol. A bespoke crawler turns heterogeneous archived documents into searchable text with provenance metadata, a vector store makes that text semantically retrievable, and an MCP dispatcher builds an orchestration graph that separates what to compute (retrieval, summarization, inference, evaluation) from where to run it (local GPU, cloud API, container). That separation lets the assistant swap inference engines while keeping the chain of evidence behind each answer reproducible and traceable.","core_discovery":"The central claim is that a RAG architecture wrapped in the Model Context Protocol gives a scientific collaboration a practical way to serve its own historical knowledge through natural language. The assistant harvests RHIC collaboration websites with a recursive, multi-format crawler; converts HTML, PDF, PostScript, and legacy Office files into clean text with provenance headers; embeds the text in a vector database; and answers questions by retrieving relevant passages and passing them to a decoder model. On expert-validated questions about STAR, open models augmented with the local corpus matched a commercial model on accuracy while citing internal sources and retrieving context from coll","pith_inferences":["Beyond the paper: the decisive role of internal data could be tested directly—exclude all mailing-list content from the corpus and measure how much answer quality drops on questions known to be answered there; the paper's claim predicts a sharp drop.","Beyond the paper: merging the local archive with permissioned live web search would let the assistant combine private institutional memory with the newest open literature, covering questions neither source answers alone.","Beyond the paper: the crawler's filtering and format-conversion choices imply that a corpus-fidelity audit—comparing extracted text against a sample of original legacy files—would quantify how much knowledge survives the pipeline."],"forward_implications":["STAR's analysis notes, software documentation, and technical discussions can remain queryable after the collider closes, so future students can reconstruct how results were obtained.","Queries that depend on unpublished collaboration knowledge—mailing-list troubleshooting, detector-specific know-how—can be answered with citations, something public web models cannot do because they never see that content.","The same architecture can be extended to other RHIC experiments and to future facilities, making knowledge preservation an explicit design goal rather than an afterthought.","Separating execution logic from deployment target lets the assistant adopt newer models without rebuilding the pipeline, protecting long-term access as model generations change.","Open models combined with a local vector store can deliver the needed answers at minimal operational cost, reducing dependence on per-query commercial APIs."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation formulation the assistant uses to ground answers in retrieved documents.","marker":"[11]"},{"why":"Defines the Model Context Protocol that lets each reasoning step be exposed as a composable, monitorable context.","marker":"[7]"},{"why":"Provides the landscape and security analysis of MCP referenced for the orchestration layer.","marker":"[9]"},{"why":"Supplies the vector database used to embed and search the harvested document corpus.","marker":"[8]"},{"why":"Supplies the machine-learning PDF-to-Markdown converter used to extract text from legacy PDF and PostScript files.","marker":"[1]"},{"why":"Supplies the HTML parsing library used by the crawler to extract page text and discover hyperlinks.","marker":"[13]"},{"why":"Survey of inference engines used as the basis for the deployment throughput and utilization comparison.","marker":"[12]"}],"fun_headline_variants":["AI assistant serves RHIC's 25-year knowledge via plain language","RHIC's legacy data get a chat interface for science","RAG-based AI opens RHIC's internal docs to natural queries","Model Context Protocol powers RHIC's historical knowledge assistant","Ask RHIC's AI: 25 years of physics know-how, instantly searchable"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The argument assumes the harvested archive contains the knowledge that matters: if significant know-how lives only in pages the crawler excludes—login-protected wikis, calendars, unsupported attachments, or off-site links—the assistant will answer confidently with gaps it cannot see.","fun_headline_variants_meta":{"raw":{"variants":["AI assistant serves RHIC's 25-year knowledge via plain language","RHIC's legacy data get a chat interface for science","RAG-based AI opens RHIC's internal docs to natural queries","Model Context Protocol powers RHIC's historical knowledge assistant","Ask RHIC's AI: 25 years of physics know-how, instantly searchable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1284,"prompt_tokens":659,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":535}},"tokens_in":403,"tokens_out":625,"duration_ms":7504,"temperature":1.0,"reasoning_tokens":535,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:09:22.460731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of insider-level questions whose answers exist only in sources the crawler deliberately excludes, such as authenticated wikis or mailing-list attachments in unsupported formats, then query the deployed assistant. If it consistently fails or gives plausible but wrong answers on those questions, the claim that internal proprietary data access is the decisive advantage would not be supported.","supporting_citations":[{"cited_title":"Lewis, E","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-augmented generation formulation the assistant uses to ground answers in retrieved documents."},{"cited_title":"https://www.anthropic.com/news/ model-context-protocol, Nov","cited_arxiv_id":null,"evidence_quote":"Defines the Model Context Protocol that lets each reasoning step be exposed as a composable, monitorable context."},{"cited_title":"https://www.trychroma.com/, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the vector database used to embed and search the harvested document corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the machine-learning PDF-to-Markdown converter used to extract text from legacy PDF and PostScript files."},{"cited_title":", Beautiful soup documentation","cited_arxiv_id":null,"evidence_quote":"Supplies the HTML parsing library used by the crawler to extract page text and discover hyperlinks."}],"review_version":1}