{"id":"f1bb2c71-4154-432b-9af1-617bf7b6e322","arxiv_id":"2608.08883","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AquiLLM presents an updated local RAG-LLM architecture with memory, skills, multimodal support, and local retrieval, aimed at preserving research group knowledge.","lead":"This paper describes AquiLLM, an open-source, locally hosted AI assistant for research groups, and the new features its authors added to it. They argue that private deployment, conversation memory, and team-specific instruction prompts can help groups preserve knowledge that usually goes undocumented.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that AquiLLM supports tacit knowledge capture is explicitly deferred: Section 4 admits no benchmarking on informal artifacts, leaving the claimed outcome unevaluated.","rationale":"The reader's weakest assumption identifies the unmeasured quality of the new local embedding/reranking models and memory layer. That is a real component-level risk, but the more load-bearing concern is broader: the paper's headline capability—tacit knowledge capture—is explicitly unbenchmarked, per Section 4. Even if the new retrieval components were perfect, the paper provides no evidence that capturing and reusing tacit knowledge actually occurs. The only prior evaluation [6] concerns formal astronomy documents and is not included, so it cannot independently support the central claim. The architecture description is clear and internally plausible, and the paper is honest about its limitations, but the central claim fails the evidence test. This supports the reader's REJECT verdict, so no change is recommended. I mark partial agreement because the reader emphasizes component-level parity while I emphasize the whole-chain absence of evaluation, though both point in the same direction.","tokens_in":9030,"tokens_out":3296,"duration_ms":31614,"concrete_test":"Re-run the evaluation protocol from [6] (five domain experts, 141 questions, faithfulness ratings) on a corpus of the informal artifacts named in Section 4 (meeting minutes, Slack messages, code comments, playbooks), using the new local embedding/reranker and memory components. Compare per-category faithfulness scores against the formal-corpus baseline in [6]. If the comparative-synthesis category (already the lowest in [6]) degrades further, or if responses requiring episodic memory fail to retrieve relevant prior context, then the tacit-knowledge capture claim fails this direct test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that AquiLLM supports the capture and reuse of tacit knowledge in research groups, is not supported by any evaluation reported in this manuscript. Section 4 states: 'we have not yet benchmarked our current approach on informal knowledge artifacts produced collaboratively by research groups such as meeting minutes, Slack messages, code comments, or collection-specific playbooks. Therefore, rigorously evaluating the capture and reuse of tacit knowledge remains to be done.' The only evaluation cited, [6], is an accepted paper not included here; it assessed faithfulness on 141 responses over 31 formal astronomy documents, not informal or tacit artifacts. Section 3 also concedes that the new local embedding and reranking models (Qwen3-VL-Embedding-2B and Qwen3-VL-Reranker-2B) 'have not yet been fully evaluated in this deployment,' and the memory layer is optional, disabled by default, and 'introduces additional complexity around privacy and correctness, such as stale memories.' Consequently, every mechanism claimed to enable tacit knowledge capture—local retrieval, memory, skills—is either unevaluated or evaluated only on formal corpora. The architecture description may be internally consistent, but the load-bearing empirical claim is deferred, not demonstrated. This is not a disagreement with external consensus; it is an internal admission in the manuscript itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AquiLLM, an open-source, modular RAG-LLM framework intended for research groups, and describes a set of architectural and feature enhancements: local embedding and reranking with open-weight models, multimodal ingestion, vLLM-based OpenAI-compatible inference, a configurable semantic/episodic memory layer, collection-scoped skills, salience-aware context packing, and UI improvements. The authors argue that these enhancements support the capture and reuse of tacit knowledge in collaborative scientific settings while maintaining local control over data and infrastructure. No new quantitative evaluation is reported; the only evaluation cited is the authors' prior accepted paper [6], which assessed faithfulness on formal astronomy documents. Section 4 explicitly states that benchmarking on informal knowledge artifacts, such as meeting minutes, Slack messages, code comments, and playbooks, has not been done and that rigorously evaluating the capture and reuse of tacit knowledge remains to be done.","tokens_in":9188,"tokens_out":4919,"duration_ms":45523,"significance":"If the central claim were validated, AquiLLM would be a valuable contribution: a fully local, open-source RAG-LLM system with memory and skills could help small research groups retain informal expertise, protect sensitive data, and avoid vendor lock-in. The paper provides a detailed, internally consistent architecture description, a thoughtful comparison with related tools in Table 1, and honest discussion of failure modes from a prior evaluation. However, the significance is currently limited because the paper's headline claim—supporting tacit knowledge capture—is explicitly unevaluated, and the new retrieval and memory components are not benchmarked. The contribution is thus an architecture proposal with plausible design rationale rather than a demonstrated capability.","major_comments":[{"comment":"The central claim that AquiLLM supports tacit knowledge capture is explicitly deferred. The paper states: \"we have not yet benchmarked our current approach on informal knowledge artifacts produced collaboratively by research groups such as meeting minutes, Slack messages, code comments, or collection-specific playbooks. Therefore, rigorously evaluating the capture and reuse of tacit knowledge remains to be done.\" Since the title, abstract, and Section 1 assert that AquiLLM supports tacit knowledge capture, the manuscript currently provides no evidence for its primary claim. The authors should either add a rigorous evaluation on informal artifacts or reframe the paper as a system-description/position paper in which tacit knowledge capture is a design goal, not an established property.","section":"Section 4"},{"comment":"The new multimodal embedding and reranking models (Qwen3-VL-Embedding-2B and Qwen3-VL-Reranker-2B) are described as central improvements, but the manuscript states they \"have not yet been fully evaluated in this deployment.\" No retrieval quality metrics, comparisons against the previous external services (e.g., Cohere), or ablation results are provided. Because retrieval quality is foundational to the claimed RAG improvements and to any downstream tacit knowledge capture, this missing evaluation is load-bearing.","section":"Section 3, local embedding and reranking"},{"comment":"The memory layer is a key mechanism for accumulating tacit knowledge, yet it is disabled by default, optional, and the authors note that it \"introduces additional complexity around privacy and correctness, such as stale memories.\" No experiments, case studies, or user demonstrations show that semantic and episodic memory improves knowledge capture or retrieval. As described, the memory component remains a speculative design feature rather than a validated capability.","section":"Section 3, memory layer"},{"comment":"The only prior evaluation cited to support the system's effectiveness is [6], an accepted paper not included in this preprint and with overlapping authorship. That study assessed faithfulness on 141 responses over 31 formal astronomy documents, not on informal or tacit artifacts. Thus [6] cannot serve as evidence for the current paper's claim of supporting tacit knowledge capture, and its exclusion makes verification of even the formal-document results impossible for readers.","section":"Section 2 and reference [6]"}],"minor_comments":[{"comment":"The abstract contains \"userinterfaceimprovements\" with missing spaces; it should read \"user interface improvements.\"","section":"Abstract"},{"comment":"The phrase \"hybrid gated delta network\" is used without a citation or explanation; either define the architecture or add a reference to a source that describes it.","section":"Section 3, chat model paragraph"},{"comment":"Figure 1 is referenced in the text but is not visible in the provided manuscript; ensure the figure is included and legible, with a caption that names all labeled components.","section":"Figure 1"},{"comment":"Table 1 gives a qualitative comparison with Open WebUI and NotebookLM, but no criteria or methodology are stated; clarify whether the comparison is based on documentation, hands-on testing, or authors' design intent.","section":"Table 1"},{"comment":"The Limitations section conflates known limitations with future work; consider separating unresolved limitations from planned extensions and adding a dedicated evaluation section if any benchmarking is included.","section":"Section 4"},{"comment":"Several references are dated 2026 and may be future or in-press works; verify that all cited works (including the accepted paper [6]) are publicly available and correctly dated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an architecture description with no new evaluation, and its central claim is explicitly deferred to future work. It may be better suited to a workshop or system-demo venue than to a full journal article in its current form. The reliance on an accepted paper [6] by overlapping authors, not included in the preprint, makes the evidence base inaccessible; the editor may wish to request a copy for review. If the authors can add a focused evaluation of informal knowledge artifacts or substantially soften the tacit-knowledge claims, the paper could become viable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this after seeing your take, and I largely agree. The paper is an honest, clearly structured description of an updated RAG-LLM system for research groups, and it gets real credit for naming its own limits. But the title and framing promise that AquiLLM supports tacit knowledge capture, and the manuscript itself admits in Section 4 that rigorous evaluation of that capture and reuse remains to be done. That admission is the load-bearing issue: every mechanism claimed to enable tacit knowledge capture—local retrieval, memory, skills—is either unevaluated or evaluated only on formal astronomy documents.\n\nWhat's genuinely new here is the specific integration of existing components: local open-weight embeddings and rerankers (Qwen3-VL-Embedding-2B, Qwen3-VL-Reranker-2B), vLLM serving, LMCache, Mem0-backed memory, and the procedural-memory framing of \"skills\" as collection-scoped instructions. The salience-aware context packing with pinned sections is a concrete idea, and the modular architecture is described in enough detail that another group could reproduce the setup. The authors also deserve credit for the limitations section, which names real issues (asynchronous memory writes, provider asymmetries, memory disabled by default) rather than hiding them.\n\nThe soft spots are proportional to the gap. The prior faithfulness evaluation [6] covered 141 responses over 31 formal documents, not the informal artifacts—meeting minutes, Slack messages, playbooks—that tacit knowledge capture actually concerns. The new embedding and reranker models are explicitly \"not yet fully evaluated in this deployment.\" Memory is optional and off by default. So the evidence does not support the headline claim, and the paper's own text confirms this. I see no technical absurdity in the architecture; it is plausible, but a plausible architecture without measurements is a technical report, not a validated research contribution.\n\nWho gets value? Groups building locally hosted RAG stacks will find the component choices and the modular design a useful reference. For a systems or workshop track, this could be a fine paper once the overreach is trimmed. For a venue expecting empirical support for the title's claim, it should be rejected or returned for major revision. I would desk reject it for a top-tier AI venue, but I'd suggest the authors submit the architecture as a demonstration and pair it with the deferred evaluation as soon as it exists.\n\nMy recommendation: don't send this to peer review as-is, but keep an eye on the follow-up.","headline":"A well-written architecture report that is honest about its own central gap: the tacit knowledge capture claim is explicitly deferred, so the paper is more of a systems description than a validated contribution.","tokens_in":9795,"tokens_out":2241,"would_cite":false,"duration_ms":22281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a locally hosted, open-weight RAG architecture organized around persistent project collections can capture and reuse a research group's tacit knowledge.","keywords":["retrieval-augmented generation","tacit knowledge","open-weight models","multimodal retrieval","episodic memory","procedural memory","research cyberinfrastructure","local deployment"],"falsifier":"Re-run the prior expert-rating study (141 responses over the 31-document astronomy corpus) with the new local retrieval stack and no external services; if faithfulness ratings, especially on comparative questions, fall below the earlier results, the claim that local control preserves answer quality is refuted. In a second check, ask new group members an onboarding question with and without the collection's skills loaded and look for the group's standards in the answers; no difference would show procedural memory is inert.","tokens_in":8784,"feed_emoji":"🧠","tokens_out":7485,"duration_ms":71221,"temperature":0.7,"pith_summary":"The paper sets out to show that a research group's tacit knowledge—the unwritten how-to that lives in meeting notes, plotting standards, and analysis workflows—can be captured and reused by a locally hosted retrieval-augmented generation system built from open-weight models rather than commercial APIs. It presents architectural improvements to AquiLLM that organize interaction around a persistent project collection, add configurable semantic and episodic memory, load collection-scoped workflow instructions (called skills) into the chat prompt, and use salience-aware context packing to stay within limited GPU memory. If the approach works, research groups could retain expertise when members leave, keep sensitive data on their own infrastructure, and make informal practices inspectable and reproducible. The authors state plainly that the new retrieval components and the capture of tacit knowledge itself have not yet been rigorously evaluated.","feed_headline":"Local AI targets a research lab's unwritten know-how","feed_subtitle":"AquiLLM keeps documents, memory, and workflow skills on local hardware, so expertise stays in the group.","key_machinery":"The load-bearing mechanism is skills, a form of procedural memory implemented as Markdown workflow instructions attached to a project collection and loaded into the session prompt only when that collection is active and its access permissions are verified. Around that sits a configurable memory layer that separates stable facts about the user from episodic semantic memories indexed for similarity search against prior message history. Salience-aware context packing carries the argument over the hardware constraint: deterministic token budgets, pinned sections, and staged pruning—tool-header deduplication, boilerplate collapse, extractive shortening, optional prompt compression, then salience-ordered dropping—keep prompts within GPU limits while guaranteeing inclusion of the latest turn, contiguous tool-call chains, and the last user turns.","core_discovery":"On its own terms, the paper's central claim is that the tacit-knowledge problem in small research groups can be treated as an architecture problem rather than just a model problem. AquiLLM makes the research group, not an individual notebook or shared knowledge base, the unit of shared context: project collections define document boundaries, carry their own workflow instructions, and persist user-specific facts and episodic conversation memories that can be retrieved by similarity search. The system layers procedural memory on top of the corpus so that how a group wants its documents read is bundled with the documents themselves. It replaces external retrieval services with local embedding and reranking, adds multimodal ingestion for plots and figures, and applies deterministic token budgets with staged pruning so long contexts fit in GPU memory. A prior expert evaluation of the system on astronomy material found that responses grounded in strongly localized evidence scored highest, while questions requiring synthesis across documents scored lowest, which motivates the memory and context-packing mechanisms introduced here.","pith_inferences":["If the architecture works as intended, the unit a system answers from shifts from isolated documents to a living group corpus plus its unwritten rules; evaluation should then measure onboarding speed or answer fidelity for new group members rather than standard RAG benchmarks.","A direct test would compare answers with skills enabled versus disabled: if procedural memory is active, the system should measurably favor the group's own plot formats, reduction steps, and checklists over generic advice.","The paper's reported weakness in cross-document synthesis suggests that memory and context packing alone may not fix multi-source questions; a natural extension is to test whether explicit synthesis tooling closes that gap on informal artifacts such as Slack threads and meeting minutes.","Because local embedding and reranking are unmeasured, the pragmatic check is an A/B comparison against the previous external retrieval service on the same astronomy corpus; equal faithfulness would establish that local control costs nothing in answer quality."],"forward_implications":["A research group can run the full pipeline—chat, embedding, reranking, transcription, and memory—on its own hardware, so papers, figures, and meeting minutes never leave local control.","Collection-scoped skills make onboarding documents, plotting standards, and reduction checklists the default frame for any chat over that project's corpus, while staying invisible to other projects.","Multimodal ingestion lets the system read plots and figures from slides or publications, so visual results enter the same retrieval and memory loop as text.","The memory layer carries user preferences and prior conversations across sessions, so follow-up work accumulates rather than restarting from scratch.","Deterministic token budgets and staged pruning keep long-context and multi-tool conversations within GPU memory, which is what makes local deployment on modest hardware feasible."],"supporting_citations":[{"why":"Supplies the expert evaluation of 141 responses over 31 astronomy documents that motivates the new memory and context-packing mechanisms.","marker":"[6]"},{"why":"Defines the original AquiLLM system and its tacit-knowledge goal, which this paper extends.","marker":"[7]"},{"why":"Gives the account of tacit versus explicit knowledge that frames the system's purpose.","marker":"[9]"},{"why":"Provides the classical source for the concept of tacit knowledge that the architecture targets.","marker":"[23]"},{"why":"Underlies the local inference engine used to serve open-weight models with efficient GPU memory management.","marker":"[17]"},{"why":"Motivates the episodic memory layer by studying very long-term conversational memory in agents.","marker":"[19]"},{"why":"Supplies the idea of procedural memory that the skills mechanism implements.","marker":"[27]"},{"why":"The open-weight chat model selected for reasoning, coding, tool use, and multimodal input.","marker":"[24]"}],"fun_headline_variants":["Open-source RAG captures lab's tacit knowledge on local hardware","AquiLLM turns research groups into shared, local memory for AI","Local LLM pipeline preserves lab's unwritten know-how with memory","Open-source AquiLLM embeds procedural memory into research AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing external retrieval services with local embedding, reranking, and memory models does not silently degrade answer quality, and that the system actually captures tacit knowledge; neither has been measured.","fun_headline_variants_meta":{"raw":{"variants":["Open-source RAG captures lab's tacit knowledge on local hardware","AquiLLM turns research groups into shared, local memory for AI","Local LLM pipeline preserves lab's unwritten know-how with memory","Open-source AquiLLM embeds procedural memory into research AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000434,"raw_usage":{"total_tokens":2178,"prompt_tokens":878,"completion_tokens":1300,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1225}},"tokens_in":494,"tokens_out":1300,"duration_ms":9184,"temperature":1.0,"reasoning_tokens":1225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:20:53.738183+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the prior expert-rating study (141 responses over the 31-document astronomy corpus) with the new local retrieval stack and no external services; if faithfulness ratings, especially on comparative questions, fall below the earlier results, the claim that local control preserves answer quality is refuted. In a second check, ask new group members an onboarding question with and without the collection's skills loaded and look for the group's standards in the answers; no difference would show procedural memory is inert.","supporting_citations":[{"cited_title":"AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research","cited_arxiv_id":null,"evidence_quote":"Supplies the expert evaluation of 141 responses over 31 astronomy documents that motivates the new memory and context-packing mechanisms."},{"cited_title":"AquiLLM: a RAG Tool for Capturing Tacit Knowledge in Research Groups","cited_arxiv_id":"2508.05648","evidence_quote":"Defines the original AquiLLM system and its tacit-knowledge goal, which this paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the account of tacit versus explicit knowledge that frames the system's purpose."},{"cited_title":"Polanyi.The Tacit Dimension","cited_arxiv_id":null,"evidence_quote":"Provides the classical source for the concept of tacit knowledge that the architecture targets."},{"cited_title":"Model weights and configuration files for the post-trainedQwen3.6-27Bmodel.Apr.2026.url:https://huggingface","cited_arxiv_id":null,"evidence_quote":"The open-weight chat model selected for reasoning, coding, tool use, and multimodal input."}],"review_version":1}